Information processing device, information processing method, and program

Abstract

To provide an information processing device that enables a moving body to resume autonomous movement.SOLUTION: The information processing device is an information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, and having: a first acquisition unit that acquires a malfunction factor of the first moving body; a second acquisition unit that acquires information about each of the plurality of moving bodies; and a selection unit that selects a second moving body that guides the autonomous movement of the first moving body based on the malfunction factor and the information about each of the plurality of moving bodies.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device that controls the autonomous movement of a moving object. [Background technology]

[0002] Mobile bodies such as mobile robots and autonomous guided vehicles (AGVs) that move autonomously based on measurement information such as image information and distance information may suspend their autonomous movement due to a failure in position measurement processing, an emergency stop to avoid a collision, etc. Patent Document 1 discloses a configuration in which a host management system gives an operation command to a mobile body that is unable to move autonomously based on information acquired from multiple cameras attached to a structure such as a building, thereby causing the mobile body to resume autonomous movement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-179921 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, if a camera is not installed or if the moving object is not within the area photographed by the camera, the moving object cannot resume autonomous movement.

[0005] An object of the present invention is to provide an information processing device that enables a moving body to resume autonomous movement. [Means for solving the problem]

[0006] An information processing device as one aspect of the present invention is an information processing device used to resume autonomous movement of a first moving body among a plurality of moving bodies, each of which moves autonomously, that is unable to move autonomously due to a malfunction, and is characterized by having a first acquisition unit that acquires the cause of the malfunction of the first moving body, a second acquisition unit that acquires information relating to each of the plurality of moving bodies, and a selection unit that selects a second moving body to guide the autonomous movement of the first moving body based on the cause of the malfunction and the information relating to each of the plurality of moving bodies. Effect of the Invention

[0007] According to the present invention, it is possible to provide an information processing device that enables a moving body to resume autonomous movement. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a system configuration according to first and second embodiments. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of an information processing apparatus according to first to fourth embodiments. [Diagram 3] 1 is a flowchart showing the process of the information processing device according to the first and second embodiments. [Figure 4] 11 is a flowchart showing a selection process of a second moving object in the first embodiment. [Diagram 5] 13 is a flowchart showing a selection process of a second moving object in the second embodiment. [Figure 6] FIG. 11 is a diagram illustrating a system configuration of a third embodiment. [Figure 7] 13 is a flowchart showing the process of the information processing device according to the third and fourth embodiments. [Figure 8] 13 is a flowchart showing a process for determining a control condition of a second moving object according to the third embodiment. [Figure 9] 13 is a flowchart showing a process for determining a control condition of a second moving object in the fourth embodiment. [Figure 10] FIG. 13 is a diagram illustrating the configuration of a system according to a fifth embodiment. [Figure 11] 13 is a flowchart showing the process of an information processing device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same components, and duplicated descriptions will be omitted.

[0010] In each embodiment, it is assumed that a plurality of moving bodies are operating simultaneously. Each moving body performs position and orientation measurement and map generation for position and orientation measurement by SLAM (Simultaneous Localization And Mapping) technology described in Non-Patent Document 1, and moves autonomously based on the generated map. Note that in each embodiment, an autonomous guided vehicle (AGV) is assumed as the moving body, but the present invention is not limited to this. For example, it may be a moving robot. Note that the method of position and orientation measurement is not limited to the method of calculating the position and orientation based on the detected position of a feature point on a 2D image described in Non-Patent Document 1. For example, a method of calculating the position and orientation based on three-dimensional point cloud information such as LiDAR (light detection and ranging) or a method of measuring the position and orientation using GPS may be used.

[0011] Non-patent document 1: Raul Mur-Artal et.al, ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics It is assumed that optical markers with different IDs are attached to each of the multiple moving bodies. Each moving body has a sensor, and the multiple moving bodies include moving bodies having different sensor fields of view and moving bodies equipped with different sensors. The field of view is expressed as the volume of a quadrangular pyramid with a plane at a predetermined distance as the base and the optical center of the sensor as the apex. In each embodiment, the moving body is recognized by detecting the optical marker attached to the moving body, but the present invention is not limited to this as long as the moving body can be recognized. For example, if the sensor is a sensor capable of measuring distance such as ToF or LIDAR, a three-dimensional model of the moving body may be held in advance, and the moving body may be recognized by model fitting using the measured point cloud and an ICP (Iterative Closest Point) algorithm. In addition, a moving body classifier may be created by deep learning to identify moving bodies appearing in an image.

[0012] In each embodiment, a camera is used as a sensor, but the present invention is not limited to this. For example, a LiDAR or ToF may be used as a sensor, and a combination of multiple cameras or multiple types of sensors may be used instead of a single camera.

[0013] In each embodiment, a moving body (first moving body) that cannot move autonomously due to a malfunction selects another moving body (second moving body) to resume autonomous movement, and the selected second moving body guides the return of the first moving body. Guiding the return means generating a movement path for the first moving body to return and giving an operation instruction, or making the first moving body follow the movement of the second moving body 2 and guiding it to a place where it can return. In addition, malfunctions include malfunctions in position measurement, malfunctions due to detection of an obstacle, malfunctions in communication with a server, etc. EXAMPLES

[0014] 1 is a diagram for explaining the configuration of a system according to the present embodiment. The system according to the present embodiment includes a plurality of moving bodies including a moving body 1 (first moving body) and a moving body 2 (second moving body), and a moving body management device 3 that centrally manages them. Each moving body receives a destination and a moving route from the moving body management device 3, and moves autonomously to the destination by estimating its own position using SLAM.

[0015] In this embodiment, it is important to select a second moving body that is less likely to experience the malfunction that occurs in the first moving body. In other words, it is necessary to select a second moving body that has a large difference from the moving conditions of the first moving body. The moving conditions of the moving body (information about the moving body) include sensing conditions such as the field of view of the sensor, the type of the sensor, and the setting values ​​of various parameters of the sensor, conditions of the moving path of the moving body, and communication conditions that indicate the communication method with the server, etc.

[0016] In this embodiment, a method for selecting a second moving body when a position measurement defect occurs in a first moving body will be described. In this embodiment, the causes of position measurement defect are a shortage of the number of corresponding points between feature points on a map and feature points during measurement, an increase in the number of erroneous corresponding points, a shortage of the number of feature points during measurement, and a change in brightness during map generation. When the number of corresponding points between feature points on a map and feature points during measurement is insufficient, a moving body having a sensor field of view different from that of the first moving body or a sensor field of view wider than that of the first moving body is selected as the second moving body. When the number of erroneous corresponding points increases, a moving body having a sensor with a noise resistance higher than that of the first moving body (for example, a high-image-quality camera or a high-sensitivity-resistant camera) or a moving body having a sensor field of view wider than that of the first moving body is selected as the second moving body. When the number of feature points during measurement is insufficient, a moving body having a sensor field of view different from that of the first moving body or a moving body having a measurement method and a sensor different from that of the first moving body is selected as the second moving body. If the brightness changes from when the map was generated, a moving object having a sensor that is resistant to the change in brightness (for example, LiDAR or a high dynamic range camera) is selected as the second moving object.

[0017] When a malfunction occurs, the moving body 1 transmits to the moving body management device 3 the position at the time when the moving body 1 was moving stably before the malfunction occurred, the cause of the malfunction, and the moving conditions, and waits for an operation instruction for returning guidance. When the moving body 1 receives the operation instruction, it recognizes the optical marker attached to the moving body 2 from the image acquired by the camera. The moving body 1 calculates the relative position and orientation of the moving bodies 1 and 2 by solving a PnP problem (Perspective-n-point problem) using the correspondence between the three-dimensional coordinates of the recognized optical marker and the two-dimensional coordinates of the optical marker on the image. After calculating the relative position and orientation, the moving body 1 sets a following movement flag to 1 and transmits it to the moving body 2. The moving body 1 moves by following the moving body 2 while maintaining a certain distance from the moving body 2 based on the calculated relative position and orientation. While moving by following the moving body 2, the moving body 1 detects feature points in the image, and when a sufficient number of feature points matching the map are secured, it stops the following movement, sets a following flag to 0, and transmits it to the moving body 2. Thereafter, the moving object 1 starts autonomous movement, notifies the moving object management device 3 of the success of the autonomous movement, receives the destination and the movement route, and returns to a normal state.

[0018] When the moving body 2 receives an instruction from the moving body management device 3 to guide the moving body 1 in which a malfunction has occurred to return, the moving body 2 receives the position of the moving body 1 and the moving route to the moving body 1 from the moving body management device 3, temporarily sets the destination to the position of the moving body 1, and moves autonomously to the destination. After arriving at the position of the moving body 1, the moving body 2 recognizes the optical marker attached to the moving body 1 from the image acquired from the camera, and calculates the relative position and orientation of the moving bodies 1 and 2 by solving the PnP problem. Based on the calculated relative position and orientation, the moving body 2 calculates the position where the moving body 1 can recognize the optical marker attached to the housing of the moving body 2, and the moving body 2 moves to the calculated position. After that, the moving body 2 receives a following movement flag from the moving body 1, and if the value is 1, it moves at a slower speed so that the moving body 1 can stably follow. Also, the moving body 2 receives a following movement flag from the moving body 1, and if the value is 0, it returns its speed to the normal state (returns to the original state).

[0019] The mobile object management device 3 centrally manages the operating status of multiple mobile objects, generates a movement route based on the item transport task of each mobile object, and transmits it to each mobile object. When a malfunction occurs in the mobile object 1, the mobile object management device 3 first acquires the malfunction location, the malfunction cause, and the movement conditions from the mobile object 1. Next, the mobile object management device 3 selects a mobile object 2 that guides the mobile object 1 to return based on the difference with the movement conditions of the mobile object 1. Next, the mobile object management device 3 acquires the position of the selected mobile object 2, and generates a movement route for the mobile object 2 with the position of the mobile object 2 as the start point and the position of the mobile object 1 as the goal point. The mobile object management device 3 then transmits the generated movement route to the mobile object 2.

[0020] The mobile object management device 3 includes a malfunction occurrence position acquisition unit (position acquisition unit) 31, a return guidance route planning unit (generation unit) 32, a return guidance instruction transmission unit (transmission unit) 33, and an information processing device 300. The information processing device 300 includes a malfunction cause acquisition unit (first acquisition unit) 310, a movement condition acquisition unit (second acquisition unit) 311, and a return guidance mobile object selection unit (selection unit) 312. The information processing device 300 may be configured to include the functions of the malfunction occurrence position acquisition unit 31, the return guidance route planning unit (generation unit) 32, and the return guidance instruction transmission unit 33.

[0021] The malfunction occurrence position acquisition unit 31 acquires information on the malfunction occurrence position output by the moving body 1 and outputs it to the return guidance path planning unit 32. The malfunction factor acquisition unit 310 acquires information on the malfunction factor output by the moving body 1 and outputs it to the return guidance moving body selection unit 312. The movement condition acquisition unit 311 acquires information on the movement condition output by the moving body 1 and outputs it to the return guidance moving body selection unit 312. The movement condition acquisition unit 311 acquires information on the movement condition output by the moving body 2 and outputs it to the return guidance moving body selection unit 312. The return guidance moving body selection unit 312 acquires information on the malfunction factor, the first movement condition, and the second movement condition, selects the moving body 2 to be guided to return the moving body 1, and outputs the individual identification ID of the selected moving body 2 to the return guidance path planning unit 32. The return guidance path planning unit 32 acquires information on the position of the moving body 1 and the position of the moving body 2, generates a path for the moving body 2 to move to the position of the moving body 1, and outputs it to the return guidance instruction transmission unit 33. The return guidance instruction transmitting unit 33 acquires the information on the moving route output by the return guidance route planning unit 32, and transmits the moving route to the moving object 2.

[0022] FIG. 2 is a diagram showing a hardware configuration of the information processing device 300. The CPU 11 controls various devices connected to the system bus 21 and executes programs. The ROM 12 stores a BIOS program and a boot program. The RAM 13 is used as a main storage device of the CPU 11. The external memory 14 stores programs processed by the information processing device 300. The input unit 15 performs processing related to input of information from a keyboard, a mouse, etc. The display unit 16 outputs the calculation results of the information processing device 300 to a display device such as a liquid crystal display device, a projector, and an LED indicator according to an instruction from the CPU 11. The communication interface 17 is used to perform information communication via a network. The information processing device 300 exchanges calculated position and orientation information with a control unit of a moving object via the communication interface 17. The I / O 18 acquires images from a sensor.

[0023] Fig. 3 is a flowchart showing the processing of the information processing device 300. The flow in Fig. 3 is realized by the CPU executing a control program, and starts when an instruction to start moving is output to a moving object.

[0024] In step S301, the malfunction cause acquisition unit 310 acquires malfunction causes of the moving object 1. In this embodiment, the malfunction causes are classified into four types: cause A (insufficient number of corresponding points with the map), cause B (increase in number of mismatched points), cause C (insufficient number of feature points during measurement), and cause D (change in brightness from when the map was generated), and are held as flags.

[0025] In step S302, the movement condition acquisition unit 311 acquires a plurality of movement conditions including the moving object 1. In this embodiment, the movement conditions are two conditions, condition A (sensor field of view) and condition B (sensor type), which are stored as flags. In step S303, the return guiding moving object selection unit 312 first obtains a difference between the moving conditions of the moving object 1 used for selecting the moving object and other moving objects based on the malfunction factor acquired in step S301. Next, the return guiding moving object selection unit 312 assigns priorities based on the differences and selects the moving object with the highest priority as the moving object 2.

[0026] FIG. 4 is a flowchart showing the selection process of the moving object 2 by the return guiding moving object selection unit 312 in step S303 of FIG.

[0027] In step S401, the return guiding moving object selection unit 312 judges whether the malfunction cause acquired in step S301 is the cause A (insufficient number of corresponding points on the map). If the return guiding moving object selection unit 312 judges that the malfunction cause is the cause A, it executes the process of step S402, and if it judges that the malfunction cause is not the cause A, it executes the process of step S403.

[0028] In step S402, the return guiding moving object selection unit 312 calculates the field of view of the common sensor of the moving objects 1 and 2.

[0029] In step S403, the return guiding moving object selection unit 312 selects a moving object with the smallest common sensor field of view calculated in step S402, that is, with the largest difference in the sensor field of view, as the moving object 2. In this embodiment, the return guiding moving object selection unit 312 selects a moving object with the largest difference in the sensor field of view as the moving object 2, but the present invention is not limited to this. For example, the return guiding moving object selection unit 312 may select a moving object with a difference in the sensor field of view equal to or larger than a predetermined value, as the moving object 2.

[0030] In step S404, the return guiding moving object selection unit 312 judges whether the malfunction factor acquired in step S301 is factor B (increase in the number of erroneous correspondence points). If the return guiding moving object selection unit 312 judges that the malfunction factor is factor B, it executes the process of step S405, and if it judges that the malfunction factor is not factor B, it executes the process of step S406.

[0031] In step S405, the return guidance moving object selection unit 312 selects as moving object 2 a moving object equipped with a camera with higher image quality or a sensor with a larger image pickup element than moving object 1, that is, a moving object that is resistant to noise.

[0032] In step S406, the return guiding moving object selection unit 312 judges whether the malfunction factor acquired in step S301 is factor C (lack of feature points during measurement). If the return guiding moving object selection unit 312 judges that the malfunction factor is factor C, it executes the process of step S407, and if it judges that the malfunction factor is not factor C, it executes the process of step S408.

[0033] In step S407, the return guiding moving object selection unit 312 selects, as the moving object 2, a moving object that has a wider field of view than the moving object 1 and has the largest field of view among the moving objects other than the moving object 1.

[0034] In step S408, the return guiding moving object selection unit 312 judges whether the malfunction factor acquired in the return step S301 is factor D (change in brightness from the time of map generation). If the return guiding moving object selection unit 312 judges that the malfunction factor is factor D, it executes the process of step S409, and if it judges that the malfunction factor is not factor D, it ends this flow.

[0035] In step S409, the return guiding moving object selection unit 312 selects as the moving object 2 a moving object equipped with a sensor having a wider dynamic range than the moving object 1 or a sensor that does not depend on brightness, such as LiDAR or ToF.

[0036] As described above, according to the configuration of this embodiment, a moving body 2 that is likely to be able to guide the moving body 1, which has a problem with its position measurement, to return is selected, and the moving body 2 guides the moving body 1 to return. This allows the moving body 1 to resume autonomous movement even outside the shooting area of ​​a camera (environmental camera) attached to a structure such as a building. In other words, it is possible to expand the environment in which autonomous movement can be resumed.

[0037] In this embodiment, the information processing device 300 is provided in the mobile object management device 3, but the present invention is not limited to this. For example, the information processing device 300 may be provided in the mobile object, or may be provided on a server separate from the mobile object or the mobile object management device 3.

[0038] In addition, in steps S402 and S403, the return guidance moving object selection unit 312 selects a moving object with the smallest common field of view as the moving object 2 in this embodiment when the cause of the malfunction is a lack of corresponding points with the map, but the present invention is not limited to this. For example, a moving object with a wider field of view than the moving object 1 may be selected as the moving object 2. By selecting a moving object with a wider field of view as the moving object 2, the number of corresponding feature points on the map and the feature points being measured increases compared to the moving object 1, so that the possibility of guiding the moving object 1 to return can be improved. In addition, the moving object 2 may be selected based on the movement history of each moving object. Specifically, a moving object that has a track record of successfully moving along the same route at the time closest to the time when the malfunction occurred in the moving object 1 may be selected as the moving object 2. By selecting a moving object that has a track record of successfully moving as the moving object 2, the probability of malfunction occurring in the moving object 2 can be reduced, so that the possibility of guiding the moving object 1 to return can be improved. In addition, the moving object 2 may be selected based on the position measurement method. Specifically, a moving object that uses a different position measurement method from the moving object 1 may be selected as the moving object 2. For example, in this embodiment, the position of the moving body 1 is measured using the feature-based SLAM described in Non-Patent Document 1, but a moving body whose position is measured using a point cloud-based SLAM such as LiDAR may be selected as the moving body 2. This makes it possible to suppress the possibility of occurrence of the same problem as with the moving body 1. EXAMPLES

[0039] In this embodiment, a method for selecting a second moving body when a malfunction occurs in a first moving body due to an obstacle will be described. In this embodiment, the causes of malfunctions due to an obstacle are "stopping in front of the obstacle without being able to avoid it," "not being able to return after avoiding contact with the obstacle," and "crash and stop due to collision with the obstacle without being able to avoid it." When the moving body 1 is unable to avoid an obstacle and stops in front of it, a moving body equipped with an obstacle avoidance function is selected as the moving body 2 from among moving bodies other than the moving body 1. The moving body 2 sends an operation instruction to the moving body 1, so that the moving body 1 can avoid the obstacle and resume autonomous movement. When the moving body 1 is unable to return after avoiding contact with the obstacle, a moving body that has a track record of moving along the same route at a time close to the time when the malfunction occurred is selected as the moving body 2. The moving body 2 is made to follow the moving body 1, and the moving body 2 guides the moving body 1 to a position where it can return, so that the moving body 1 can resume autonomous movement. When the moving body 1 is unable to avoid an obstacle and stops due to a collision, a moving body with a wider field of view than the moving body 1 and with the largest field of view is selected as the moving body 2. Moving body 2 detects information about obstacles around moving body 1 and sends operational instructions to moving body 1, enabling moving body 1 to safely avoid the obstacles and resume autonomous movement.

[0040] The basic configuration of this embodiment is similar to that of the first embodiment, and only the configuration different from the first embodiment will be described in this embodiment, and the description of the same configuration will be omitted.

[0041] The information processing device 300 executes the flow of Fig. 3. The processes from step S301 to step S303 are similar to those described in the first embodiment, and therefore the description will be omitted. Note that the malfunction factors in this embodiment are classified into three types: factor A (unable to avoid an obstacle and stops in front of the obstacle), factor B (unable to return after avoiding contact with the obstacle), and factor C (unable to avoid an obstacle and stops due to collision), and are stored as flags.

[0042] The process of step S304 in this embodiment is different from that in embodiment 1. Hereinafter, the process of step S304 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the selection process of the second moving object by the return guiding moving object selection unit 312 in step S304 in Fig. 3 in this embodiment.

[0043] In step S501, the return guiding moving object selection unit 312 judges whether the malfunction cause acquired in step S301 is the cause A (stopping in front of the obstacle without being able to avoid it). If the return guiding moving object selection unit 312 judges that the malfunction cause is the cause A, it executes the process of step S502, and if it judges that the malfunction cause is not the cause A, it executes the process of step S503.

[0044] In step S502, the return guiding moving object selection unit 312 selects a moving object that is capable of avoiding obstacles (has a function of avoiding obstacles) different from the moving object 1 as the moving object 2.

[0045] In step S503, the return guiding moving object selection unit 312 judges whether the malfunction factor acquired in step S301 is factor B (unable to return after avoiding contact with an obstacle). If the return guiding moving object selection unit 312 judges that the malfunction factor is factor B, it executes the process of step S504, and if it judges that the malfunction factor is not factor B, it executes the process of step S505.

[0046] In step S504, the return guiding moving object selection unit 312 selects as the moving object 2 a moving object that has a track record of traveling along the same route as the moving object 1 at a time closest to the time when the malfunction occurred in the moving object 1.

[0047] In step S505, the return guiding moving object selection unit 312 judges whether the malfunction factor acquired in step S301 is factor C (crash into an obstacle without being able to avoid it and stop). If the return guiding moving object selection unit 312 judges that the malfunction factor is factor C, it executes the process of step S506, and if it judges that the malfunction factor is not factor C, it ends this flow.

[0048] In step S506, the return guiding moving object selection unit 312 selects a moving object having a wider field of view than the moving object 1 and having the largest field of view as the moving object 2.

[0049] As described above, according to the configuration of this embodiment, a moving body 2 that is likely to be able to guide the moving body 1 that has encountered a malfunction due to an obstacle to return is selected, and the moving body 2 guides the moving body 1 to return. This allows the moving body 1 to resume autonomous movement even outside the shooting area of ​​a camera (environmental camera) attached to a structure such as a building. In other words, it is possible to expand the environment in which autonomous movement can be resumed.

[0050] In the first and second embodiments, the mobile body management device 3 manages information on a plurality of mobile bodies, but a problem may occur in communication between the mobile body management device 3 and the plurality of mobile bodies, causing the mobile body 1 to stop. In such a case, a mobile body having a communication means different from that of the mobile body 1 (e.g., a different frequency band, different communication strength, different communication standard, and different communication area) may be selected as the mobile body 2. In this way, the mobile body 2 guides the mobile body 1 to a position where communication is possible, and the mobile body 1 can resume autonomous movement.

[0051] Furthermore, when the battery of the moving body 1 is insufficient, a moving body having a power supply function and a large remaining battery charge or a moving body having another battery for power supply may be selected as the moving body 2. This allows the battery of the moving body 1 to be powered and the moving body 1 to resume autonomous movement. Also, a moving body that can physically tow the moving body 1 may be selected as the moving body 2, and the battery shortage of the moving body 1 may be resolved by towing the moving body 1 to a charging point. EXAMPLES

[0052] In the first and second embodiments, examples have been described in which an appropriate moving body 2 is selected as the moving body to guide to return based on the difference between the moving conditions of the moving bodies 1 and 2, thereby improving the probability that the moving body 2 can guide the moving body 1 to return. In this embodiment, an example will be described in which the control conditions (second control conditions) of the moving body 2 are set so as to differ from the control conditions (first control conditions) of the moving body 1, thereby improving the probability that the moving body 2 can guide the moving body 1 to return. The control conditions are settings related to the speed and traveling direction of the moving body, and can be determined by inputting a control amount to the moving body.

[0053] The configuration of this embodiment other than the information processing device 300 is the same as that of the first embodiment, and only the configuration different from that of the first embodiment will be described in this embodiment, and the description of the same configuration will be omitted.

[0054] 6 is a diagram for explaining the configuration of the system of this embodiment. The information processing device 300 includes a malfunction cause acquisition unit 310, a control condition acquisition unit (third acquisition unit) 314, and a control condition determination unit (determination unit) 315. The control condition acquisition unit 314 acquires the control conditions from the moving object 1 and outputs them to the control condition determination unit 315. The control condition determination unit 315 acquires the malfunction cause from the malfunction cause acquisition unit 310 and the control conditions output by the control condition acquisition unit 314, and outputs the control conditions of the moving object 2 to the return guidance instruction transmission unit 33.

[0055] In this embodiment, a method for controlling a second moving body when a position measurement defect occurs in a first moving body is described. In this embodiment, the causes of position measurement defect are a shortage of the number of corresponding points between feature points on a map and feature points during measurement, an increase in the number of erroneous corresponding points, and a shortage of the number of feature points during measurement. When the number of corresponding points between feature points on a map and feature points during measurement is insufficient, a control condition is determined so that the moving body 2 moves in a direction with more feature points than the direction in which the moving body 1 was facing. This can reduce the probability of position measurement defect occurring in the moving body 2, and can improve the possibility of the moving body 1 resuming autonomous movement. When the number of erroneous corresponding points increases, a control condition is determined so that the moving speed of the moving body 2 is slowed down, and feature points are matched with higher accuracy by taking more processing time than the moving body 1. For example, when the moving body 1 performs matching using ORB features, it is sufficient to perform matching using SIFT features or SURF features, which take more processing time but are more accurate. This can reduce the probability of position measurement defect occurring in the moving body 2, and can improve the possibility of the moving body 1 resuming autonomous movement. When the number of feature points being measured is insufficient, the control conditions are determined so that the moving body 2 moves while looking around the surroundings. This makes it possible to detect feature points with a wider field of view than the moving body 1, reduces the probability of position measurement errors occurring in the moving body 2, and improves the possibility that the moving body 1 will resume autonomous movement. Fig. 7 is a flowchart showing the processing of the information processing device 300. The flow in Fig. 7 is realized by the CPU executing a control program, and starts when an instruction to start moving is output to a moving object.

[0056] In step S701, the malfunction cause acquisition unit 310 acquires malfunction causes of the moving object 1. In this embodiment, the malfunction causes are classified into three types: cause A (insufficient number of corresponding points with the map), cause B (increase in the number of mismatched points), and cause C (insufficient number of feature points during measurement), and are held as flags.

[0057] In step S702, the control condition acquisition unit 314 acquires the control conditions of the moving object 1.

[0058] In step S703, the control condition determination unit 315 determines the control condition for the moving body 2 based on the malfunction factor acquired in step S701 and the control condition acquired in step S702. In this embodiment, the moving body 2 is assumed to be randomly selected from moving bodies different from the moving body 1.

[0059] FIG. 8 is a flowchart showing the process of determining the control conditions for the second moving object by the control condition determining unit 315 in step S703 of FIG.

[0060] In step S801, the control condition determination unit 315 determines whether the malfunction cause acquired in step S701 is factor A (insufficient number of corresponding points on the map). If the control condition determination unit 315 determines that the malfunction cause is factor A, it executes the process of step S802, and if it determines that the malfunction cause is not factor A, it executes the process of step S803.

[0061] In step S802, the control condition determination unit 315 obtains the direction of the moving body 1 from the control conditions obtained in step S702, and determines the control conditions for the moving body 2 to move in a direction in which more feature points move into the field of view than in the direction of the moving body 1 when it moves. Note that in the process of this step, the control conditions for the moving body 2 to move in a direction in which more feature points move into the field of view than in the direction of the moving body 1 when it moves are determined, but the present invention is not limited to this. For example, the control conditions may be determined so that the moving body 2 moves while looking around or moves at a slower speed so that more feature points can be detected.

[0062] In step S803, the control condition determination unit 315 determines whether the malfunction cause acquired in step S702 is factor B (increase in the number of mismatch points). If the control condition determination unit 315 determines that the malfunction cause is factor B, it executes the process of step S804, and if it determines that the malfunction cause is not factor B, it executes the process of step S805.

[0063] In step S804, the control condition determination unit 315 obtains the movement speed of the moving body 1 from the control conditions obtained in step S702, and determines the control conditions for the moving body 2 so that the moving body 2 moves at a slower speed than the moving body 1 to perform feature point matching with high accuracy.

[0064] In step S805, the control condition determination unit 315 determines whether the malfunction cause acquired in the return step S301 is factor C (lack of feature points during measurement). If the control condition determination unit 315 determines that the malfunction cause is factor C, it executes the process of step S806, and if it determines that the malfunction cause is not factor C, it ends this flow.

[0065] In step S806, the control condition determination unit 315 first obtains the posture of the moving body 1 during movement from the control conditions obtained in step S702. Next, the control condition determination unit 315 determines the control conditions for the moving body 2 that moves while changing its posture by a predetermined amount or more based on the posture of the moving body 1 during movement (while looking around the surroundings) at a point where the number of feature points is a predetermined number or less. Note that in the processing of this step, the control conditions are determined so that the moving body 2 moves while looking around the surroundings, but the present invention is not limited to this. The control conditions may also be determined so that the moving body 2 moves while facing in a direction with many features or moves at a slower speed.

[0066] As described above, according to the configuration of this embodiment, a control method for the moving body 2 is determined so that the moving body 1, which has a problem with its position measurement, can be guided to return, and the moving body 2 guides the moving body 1 to return. This allows the moving body 1 to resume autonomous movement even outside the shooting area of ​​a camera (environmental camera) attached to a structure such as a building. In other words, it is possible to expand the environment in which autonomous movement can be resumed.

[0067] In this embodiment, a case where a malfunction occurs in position measurement has been described, but the cause of the malfunction may be "a fast moving speed and a lack of measurable feature points" or "few feature points in the traveling direction." When the moving speed is fast and there are a lack of measurable feature points, the control conditions may be determined so that the moving body 2 moves at a slower speed than the moving body 1 and moves toward guiding the moving body 1 to return. This can reduce the probability that a malfunction occurs in the moving body 2 and improve the possibility that the moving body 1 will return.

[0068] Also, when there are few characteristic points in the traveling direction, the control conditions may be determined so that the moving body 2 changes the angle from the traveling direction of the moving body 1 by a predetermined amount or more, and moves in the direction with many characteristic points as the traveling direction. This can reduce the probability of malfunction of the moving body 2 and improve the possibility of the moving body 1 returning to normal. EXAMPLES

[0069] In this embodiment, a method for controlling a second moving body when a malfunction occurs in a first moving body due to an obstacle will be described. In this embodiment, the causes of malfunctions due to an obstacle are "stopping in front of an obstacle without being able to avoid the obstacle" and "crash and stop due to collision with an obstacle without being able to avoid the obstacle". When stopping in front of an obstacle without being able to avoid the obstacle, a control condition is determined so that the moving body 2 moves in a direction different from the direction in which the moving body 1 moved. This can reduce the probability of a malfunction in position measurement occurring in the moving body 2, and can improve the possibility that the moving body 1 resumes autonomous movement. When stopping due to collision with an obstacle without being able to avoid the obstacle, a control condition is determined so that the moving body 2 moves while looking around. This allows the moving body 2 to detect obstacles around the moving body 1 and send an operation instruction to the moving body 1, allowing the moving body 1 to resume returning.

[0070] The basic configuration of this embodiment is similar to that of the third embodiment, and only the configuration different from the third embodiment will be described in this embodiment, and the description of the same configuration will be omitted.

[0071] The information processing device 300 executes the flow of Fig. 7. The processes from step S701 to step S702 are similar to those described in the third embodiment, and therefore the description will be omitted. Note that there are two malfunction factors in this embodiment: factor A (unable to avoid an obstacle and stopping in front of the obstacle) and factor B (unable to avoid an obstacle and colliding with it and stopping), which are stored as flags.

[0072] The process of step S703 in this embodiment is different from that in embodiment 3. Hereinafter, the process of step S703 in this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the process of determining the control condition for the second moving object by the control condition determination unit 315 in step S703 in Fig. 7 in this embodiment.

[0073] In step S901, the control condition determination unit 315 determines whether the malfunction cause acquired in step S701 is factor A (stopping in front of an obstacle without being able to avoid the obstacle). If the control condition determination unit 315 determines that the malfunction cause is factor A, it executes the process of step S902, and if it determines that the malfunction cause is not factor A, it executes the process of step S903.

[0074] In step S902, the control condition determination unit 315 obtains the movement direction of the moving body 1 from the control conditions obtained in step S702, and determines the control conditions for the moving body 2 so that the moving body 2 moves toward the position of the moving body 1 from a movement direction that differs from that of the moving body 1 by a predetermined angle or more.

[0075] In step S903, the control condition determination unit 315 determines whether the malfunction cause acquired in step S701 is factor B (crash into an obstacle without being able to avoid it, resulting in a stop). If the control condition determination unit 315 determines that the malfunction cause is factor B, it executes the process of step S904, and if it determines that the malfunction cause is not factor B, it ends this flow.

[0076] In step S904, the control condition determination unit 315 obtains the posture of the moving body 1 during movement from the control conditions obtained in step S702, and determines the control conditions such that the moving body 2 moves while looking around the surroundings a predetermined amount or more based on the obtained posture at a point where the number of feature points is a predetermined number or less.

[0077] As described above, according to the configuration of this embodiment, a control method for the moving body 2 is determined so that the moving body 1, which has experienced a malfunction due to an obstacle, can be guided to return, and the moving body 2 guides the moving body 1 to return. This allows the moving body 1 to resume autonomous movement even outside the shooting area of ​​a camera (environmental camera) attached to a structure such as a building. In other words, the environment in which autonomous movement can be resumed can be expanded.

[0078] In addition, as a cause of a malfunction due to an obstacle, "the obstacle cannot be avoided because the moving speed is fast" or "there are many obstacles in the moving direction" may be mentioned. When the moving speed is fast and the obstacle cannot be avoided, the control conditions may be determined so that the moving body 2 moves at a slower speed than the moving body 1 and moves toward the return guidance of the moving body 1. This can reduce the probability of a malfunction in the position measurement of the moving body 2, and can improve the possibility that the moving body 1 will resume autonomous movement. In addition, when there are many obstacles in the moving direction, the control conditions may be determined so that the moving body 2 moves at an angle that is changed by a predetermined amount or more from the traveling direction of the moving body 1. This can reduce the probability of a malfunction in the position measurement of the moving body 2, and can improve the possibility that the moving body 1 will resume autonomous movement.

[0079] Although this embodiment does not mention a method for selecting the moving object 2, the method for selecting the moving object 2 described in the embodiment 1 may be used. By determining a control condition for the moving object 2 selected by the selection method described in the embodiment 1, it is possible to further increase the possibility that the moving object 1 will resume autonomous movement. EXAMPLES

[0080] In the first and second embodiments, the moving body 2 guides the moving body 1 to return, and the moving body 1 is made to follow the moving body 2, thereby guiding the moving body 1 to return. However, the moving body 2 does not necessarily need to move voluntarily to return. For example, a method may be used in which the moving body 1 recognizes the moving body 2 and follows it voluntarily. In this embodiment, an example will be described in which the moving body 1 voluntarily follows the moving body 2 and resumes autonomous movement, even if the moving body 2 does not receive an instruction for return guidance.

[0081] 10 is a diagram illustrating the configuration of a system according to this embodiment. The system according to this embodiment has a plurality of moving objects including a moving object 1 (first moving object) and a moving object 2 (second moving object). The moving object 1 includes an information processing device 100, a sensor 111, a self-position estimation unit 112, and a following notification unit 113. In this embodiment, the sensor 111 is a camera.

[0082] The information processing device 100 includes a moving object recognition unit (recognition unit) 101, a tracking position determination unit (first determination unit) 102, and a control condition determination unit (second determination unit) 103. The moving object recognition unit 101 acquires an image output by a sensor 111, recognizes a moving object 2 from the acquired image, outputs a relative position and orientation of the moving objects 1 and 2 to the tracking position determination unit 102, and outputs an individual identification ID of the moving object 2 to the tracking notification unit 113. The tracking position determination unit 102 acquires the relative position and orientation output by the moving object recognition unit 101, determines a tracking position based on the relative position and orientation, and outputs it to the control condition determination unit 103 and the tracking notification unit 113. The control condition determination unit 103 acquires the tracking position output by the tracking position determination unit 102, determines a control condition for the moving object 2 based on the tracking position, and outputs it to the tracking notification unit 113.

[0083] Fig. 11 is a flowchart showing the processing of the information processing device 100. The flow of Fig. 11 is realized by the CPU executing a control program. Here, it is assumed that the moving object 1 has a malfunction and is in a state where it cannot move autonomously, and the moving object 2 is moving autonomously to a target position.

[0084] In step S1101, the moving object recognition unit 101 acquires an individual identification ID of the moving object 2 by recognizing an optical marker attached to the moving object 2 from an image output by the sensor 111. The moving object recognition unit 101 also calculates the relative position and orientation of the moving objects 1 and 2 by solving a PnP problem from the correspondence between the three-dimensional coordinates of the optical marker and the two-dimensional coordinates of the optical marker obtained from the image.

[0085] In step S1102, the tracking position determination unit 102 determines a position where the relative position and orientation calculated in step S1101 and the area of ​​the optical marker shown in the image output by the sensor 111 are within a predetermined range as the tracking position of the moving object 1.

[0086] In step S1103, the control condition determination unit 103 determines the control condition of the moving body 2 based on the tracking position determined in step S1102. Specifically, when the moving body 1 deviates from the tracking position determined in step S1102 by a predetermined amount, the moving body 2 moves at a reduced speed so that the moving body 1 can return to the tracking position.

[0087] As described above, according to the configuration of this embodiment, the moving body 1 in which a malfunction has occurred detects the moving body 2 which is moving autonomously and follows it spontaneously. This allows the moving body 1 to resume autonomous movement not only near the place where the malfunction has occurred, but also at any place where the moving body 2 can move, thereby expanding the environment in which autonomous movement can be resumed.

[0088] In this embodiment, the tracking position is determined based on the area of ​​the optical marker, but when recognizing a moving object by model fitting, the tracking position may be determined so that the residual of ICP falls within a predetermined range. When recognizing a moving object using deep learning, the tracking position may be determined so that the area of ​​the moving object in the image falls within a predetermined range. [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0089] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A first acquisition unit that acquires a cause of a defect in the first moving object; A second acquisition unit that acquires information about each of the plurality of moving objects; and a selection unit that selects a second moving body to guide the return of the first moving body based on information about the malfunction cause and each of the plurality of moving bodies. (Configuration 2) The information processing device described in configuration 1, characterized in that the selection unit selects the second moving body based on the malfunction factor and a difference between the field of view of a sensor included in the first moving body and the field of view of a sensor included in the moving body different from the first moving body. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the selection unit selects the second moving object based on the cause of the malfunction and the type of a sensor included in each of the plurality of moving objects. (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the selection unit selects the second moving object based on the cause of the malfunction and a movement history of the plurality of moving objects. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the selection unit selects the second moving object based on the cause of the malfunction and a method for measuring the positions of the multiple moving objects. (Configuration 6) The information processing device according to any one of configurations 1 to 5, characterized in that the selection unit selects the second moving body based on the cause of the malfunction and an obstacle avoidance function of a moving body different from the first moving body. (Configuration 7) A location acquisition unit that acquires a location where a malfunction has occurred in the first moving object; a generation unit that generates a path for moving the second moving object to a position where a malfunction has occurred in the first moving object; 7. The information processing device according to any one of configurations 1 to 6, further comprising a transmitting unit that transmits the route to the second moving object. (Configuration 8) An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A first acquisition unit that acquires a cause of a defect in the first moving object; a third acquisition unit that acquires a first control condition of the first moving object; and a determination unit that determines a second control condition of a second moving object that induces the return of the first moving object based on the cause of the malfunction and the first control condition. (Configuration 9) 9. The information processing apparatus according to configuration 8, wherein the determination unit determines, as the second control condition, that the second moving body moves at a moving speed slower than a moving speed of the first moving body. (Configuration 10) The information processing device described in configuration 8 or 9, characterized in that the determination unit determines the second control condition so that the second moving body moves while changing its attitude by more than a predetermined amount based on the attitude of the first moving body during movement. (Configuration 11) An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A recognition unit for recognizing a second moving object that guides the return of the first moving object; a first determination unit that determines a position of the first moving body relative to the second moving body so that the first moving body follows the second moving body; and a second determination unit that determines a control condition for the second moving object based on a position of the first moving object relative to the second moving object. (Method 1) An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: acquiring a cause of the malfunction of the first moving object; acquiring information about each of the plurality of moving objects; and selecting a second moving body to guide the autonomous movement of the first moving body based on the malfunction cause and information about each of the plurality of moving bodies. (Method 2) An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: acquiring a cause of the malfunction of the first moving object; obtaining a first control condition of the first moving object; and determining a second control condition for a second moving body that guides the autonomous movement of the first moving body based on the malfunction factor and the first control condition. (Method 3) An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: A step of recognizing a second moving object that guides the autonomous movement of the first moving object; determining a position of the first moving body relative to the second moving body so that the first moving body follows the second moving body; and determining a control condition for the second moving object based on a position of the first moving object relative to the second moving object. (Configuration 12) A program for causing a computer to execute the information processing method according to any one of Methods 1 to 3.

[0090] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0091] 1. First Mobile Unit 2. Second Mobile Unit 300 Information processing device 310 Malfunction cause acquisition unit (first acquisition unit) 311 Movement condition acquisition unit (second acquisition unit) 312 Return guidance moving object selection unit (selection unit)

Claims

1. An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A first acquisition unit that acquires a cause of a malfunction of the first moving object; A second acquisition unit that acquires information about each of the plurality of moving objects; and a selection unit that selects a second moving object to guide the return of the first moving object based on information about the malfunction cause and each of the plurality of moving objects.

2. The information processing device according to claim 1, characterized in that the selection unit selects the second moving body based on the cause of the malfunction and a difference between the field of view of a sensor included in the first moving body and the field of view of a sensor included in the moving body different from the first moving body.

3. The information processing apparatus according to claim 1 , wherein the selection unit selects the second moving object based on a cause of the malfunction and a type of a sensor included in each of the plurality of moving objects.

4. The information processing apparatus according to claim 1 , wherein the selection unit selects the second moving object based on the cause of the malfunction and a movement history of the plurality of moving objects.

5. The information processing apparatus according to claim 1 , wherein the selection unit selects the second moving object based on a cause of the malfunction and a method of measuring the positions of the plurality of moving objects.

6. The information processing apparatus according to claim 1 , wherein the selection unit selects the second moving object based on a cause of the malfunction and an obstacle avoidance function of a moving object different from the first moving object.

7. a location acquisition unit that acquires a location where a malfunction has occurred in the first moving object; a generation unit that generates a path for moving the second moving object to a position where a malfunction has occurred in the first moving object; The information processing apparatus according to claim 1 , further comprising a transmitting unit that transmits the route to the second mobile unit.

8. An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A first acquisition unit that acquires a cause of a malfunction of the first moving object; a third acquisition unit that acquires a first control condition of the first moving object; a determination unit that determines a second control condition of a second moving object that induces the return of the first moving object based on the cause of the malfunction and the first control condition.

9. The information processing apparatus according to claim 8 , wherein the determination unit determines, as the second control condition, that the second moving object moves at a moving speed slower than a moving speed of the first moving object.

10. The information processing apparatus according to claim 8 , wherein the determination unit determines the second control condition such that the second moving body moves while changing its attitude by a predetermined amount or more based on the attitude of the first moving body during the movement.

11. An information processing device used to resume autonomous movement of a first moving body that cannot move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, A recognition unit for recognizing a second moving object that guides the return of the first moving object; a first determination unit that determines a position of the first moving body relative to the second moving body so that the first moving body follows the second moving body; and a second determination unit that determines a control condition for the second moving object based on a position of the first moving object relative to the second moving object.

12. 1. An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: acquiring a cause of a malfunction of the first moving object; acquiring information about each of the plurality of moving objects; and selecting a second moving body to guide the return of the first moving body based on the cause of the malfunction and information about each of the plurality of moving bodies.

13. 1. An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: acquiring a cause of a malfunction of the first moving object; obtaining a first control condition of the first moving object; determining a second control condition for a second moving object that induces the return of the first moving object based on the cause of the malfunction and the first control condition.

14. 1. An information processing method used for a first moving body, which is unable to move autonomously due to a malfunction among a plurality of moving bodies each of which moves autonomously, to resume autonomous movement, comprising: A step of recognizing a second moving object that guides the return of the first moving object; determining a position of the first moving body relative to the second moving body so that the first moving body follows the second moving body; determining a control condition for the second moving object based on a position of the first moving object relative to the second moving object.

15. A program for causing a computer to execute the information processing method according to any one of claims 12 to 14.

Images