Information processing device, moving body, information processing method, and computer program

Abstract

To provide an information processing device capable of restarting the update of an environmental map.SOLUTION: The information processing device includes an environmental map holding part for holding the environmental map, a captured image input part for inputting a captured image obtained by capturing an environment by an imaging device, a first position attitude measurement part for measuring the first position and attitude of a moving body on which the imaging device is mounted on the basis of the environmental map and the captured image, a second position attitude measurement part for measuring the second position and attitude of the moving body on the basis of sensor data acquired by a sensor other than the imaging device, and an environmental map update part for updating the environmental map on the basis of the first position and attitude. The environmental map update part restarts the update of the environmental map by using the second position and attitude, when the update of the environmental map based on the first position and attitude is interrupted.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, a mobile object, an information processing method, and a computer program. [Background technology]

[0002] There is a demand to quickly create a map of the environment in advance for autonomous movement of moving bodies such as trackless AGVs (Automated Guided Vehicles) and AMRs (Autonomous Mobile Robots).

[0003] Simultaneous Localization and Mapping (SLAM) technology is known as a method for simultaneously creating a map of the environment and estimating the position and orientation of a moving object using information obtained by sensors such as cameras and distance sensors mounted on the moving object.

[0004] The map of the environment created by SLAM contains information on the feature points in the environment as well as information on the intermittent positions and orientations of the imaging device during map creation (hereafter referred to as map elements). However, if the estimation of the vehicle's own position and orientation fails during map creation, the vehicle loses its own position and map creation is interrupted.

[0005] In order to resume map creation, it is necessary to return to a location where a map of the environment has already been created and where the self-location can be estimated. Patent Document 1 describes a method for recovering the lost self-location based on other sensor information when the estimation of the self-location or orientation fails. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] WO2019-111701 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method of Patent Document 1 has a problem in that map creation cannot be resumed even if the self-position is restored.

[0008] In view of the above, an object of the present invention is to provide an information processing device capable of resuming updating of an environmental map. [Means for solving the problem]

[0009] The information processing device according to the present invention comprises: an environmental map storage unit for storing an environmental map; a captured image input unit for inputting a captured image of an environment captured by an imaging device; a first position and orientation measurement unit that measures a first position and orientation of a moving object on which the imaging device is mounted, based on the environmental map and the captured image; a second position and orientation measurement unit that measures a second position and orientation of the moving object based on sensor data acquired by a sensor other than the imaging device; an environment map update unit that updates the environment map based on the first position and orientation; The environmental map update unit is characterized in that when the update of the environmental map based on the first position and orientation is interrupted, the environmental map update unit resumes the update of the environmental map using the second position and orientation. Effect of the Invention

[0010] According to the present invention, it is possible to provide an information processing device capable of restarting updating of an environmental map. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an example of use of a moving object in the first embodiment. FIG. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device 100 in FIG. [Diagram 3] 2 is a functional block diagram showing an example of a data flow configuration of the information processing device according to the first embodiment. FIG. [Figure 4]4 is a flowchart showing an example of an information processing method executed by the information processing device 100 according to the first embodiment. [Diagram 5] 5 is a flowchart showing the procedure of a process for updating the environment map in step S407 of FIG. [Figure 6] 5 is an image diagram showing an example of the map of the environment stored in the environment map storage unit 305 of FIG. 4 being displayed on, for example, the tablet 102 of the user 101 via the environment map display unit 307. FIG. [Figure 7] 13 is a flowchart showing an update processing procedure of the information processing method according to the second embodiment. [Figure 8] 13 is a flowchart showing a procedure for displaying a map of an environment in an information processing method according to a third embodiment. [Figure 9] FIG. 9 is a conceptual diagram of a visualized map of the environment obtained by executing the flowchart of FIG. 8. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0013] <Embodiment 1> 1 is a schematic diagram showing a usage example of a moving body in embodiment 1. A moving body 1 equipped with an information processing device 100, a first sensor 10 as an imaging device, and a second sensor 11 travels autonomously based on a map of the environment (environmental map) created in advance.

[0014] A user 101 uses a mobile terminal such as a tablet 102 to instruct the mobile object 1 to create a map of the environment. The user 101 can also use the tablet 102 to check the created map of the environment and instruct the mobile object 1 to start autonomous driving, and the mobile object 1 has a drive control unit such as an ECU (Electronic Control Unit) (not shown) that performs autonomous driving based on an environmental map updated by an environmental map update unit described below.

[0015] Fig. 2 is a diagram showing an example of a hardware configuration of the information processing device 100 shown in Fig. 1. Reference numeral 21 denotes a CPU, which controls various devices connected to a system bus 29 and executes programs. Reference numeral 22 denotes a ROM, which stores a BIOS program and a boot program.

[0016] Reference numeral 23 denotes a RAM, which is used as a main storage device for the CPU 21. Reference numeral 24 denotes an external memory, which stores computer programs to be processed by the CPU 21 as a computer. An input unit 25 performs processing related to input of information, etc., via a keyboard, mouse, etc.

[0017] The display unit 26 generates display control signals for the calculation results, maps, images, etc. of the information processing device 100 in accordance with instructions from 21, and displays them on a display device such as a tablet 102 via the communication I / F. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator.

[0018] Reference numeral 27 denotes a communication interface, which communicates information via a network, and the communication interface may be Ethernet, or any type of interface such as USB, serial communication, wireless communication, etc. Reference numeral 28 denotes an I / O, which inputs data from a sensor.

[0019] Fig. 3 is a functional block diagram showing an example of the functional configuration of the information processing device in embodiment 1. Note that some of the functional blocks shown in Fig. 3 are realized by causing a CPU serving as a computer included in the information processing device to execute a computer program stored in a memory serving as a storage medium.

[0020] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, the functional blocks shown in Fig. 3 do not have to be built into the same housing, and may be configured by separate devices connected to each other via signal paths.

[0021] Reference numeral 302 denotes a captured image input unit that executes a captured image input step of inputting a captured image of an environment from a first sensor 10 serving as an imaging device in Fig. 1. In this embodiment, the first sensor is a stereo camera.

[0022] A first position and orientation measurement unit 303 estimates the position and / or orientation of the moving body 1 on which the imaging device is mounted, based on a captured image and an environmental map. If the first position and orientation measurement unit 303 can estimate its own position and / or orientation, it outputs the position and orientation information of the moving body 1, and if it cannot estimate its own position and / or orientation, it does not output the position and orientation information of the moving body 1.

[0023] Reference numeral 304 denotes an environment map updating unit that updates the environment map using the first position and orientation position obtained by the first position and orientation measurement unit 303 and the captured image. Reference numeral 305 denotes an environment map holding unit that executes an environment map holding step that holds the environment map updated by the environment map updating unit 304.

[0024] Reference numeral 306 denotes an environment map update resumption determination unit that determines whether or not updating of the environment map may be resumed when the moving object 1 has lost its own position (hereinafter, referred to as a lost state).

[0025] The environment map update resumption determination unit 306 functions as an update resumption determination unit that determines whether or not the update of the environment map based on the first position and attitude is interrupted. 307 is an environment map display unit that generates and outputs a display control signal for displaying the creation status and creation results of the environment map on the tablet 102 of the user 101 in Fig. 1.

[0026] Reference numeral 308 denotes a second position and / or orientation measurement unit that estimates a second position and / or orientation of the moving object 1 based on sensor data acquired by a second sensor 11 that is a sensor other than an imaging device. The second sensor 11 is, for example, a 2D LiDAR. Reference numeral 309 denotes a position and orientation input unit that inputs the position and orientation information acquired from the second position and orientation measurement unit 308 to the information processing device 100.

[0027] Fig. 4 is a flowchart showing an example of an information processing method executed by the information processing device 100 in embodiment 1. Note that the operation of each step in the flowchart in Fig. 4 is performed sequentially by the CPU as a computer in the information processing device 100 executing a computer program stored in the memory.

[0028] In the following description, the processing procedures according to the flowcharts are not limited to those shown in the drawings, and any combination of procedures, combining of multiple processes, or subdivision of processes is possible as long as the results of the embodiment are achieved. Also, each process can be extracted individually and function as a single functional element, and can be used in combination with processes other than those shown.

[0029] The flow described in Fig. 4 starts when an instruction is given to start traveling of the moving object 1 in Fig. 1. In step S401, the information processing device 100 in Fig. 3 is initialized. That is, a program is read from the external memory 24 in Fig. 2, and the information processing device 100 is made operable. Similarly, state variables managed by the environment map update unit 304 are initialized.

[0030] Next, in step S402, the captured image input unit 302 performs a process of inputting a captured image from the first sensor 10 in Fig. 1. In step S403, the first position and orientation measurement unit 303 performs a process of estimating its own position and / or orientation from the captured image using a SLAM technique, and performs a process of measuring first position and orientation information.

[0031] Here, step S403 functions as a first position and orientation measurement step for measuring a first position and orientation of a moving object on which an image capturing device is mounted, based on an environmental map and a captured image.

[0032] In step S404, the position and orientation input unit 309 performs a process of acquiring second position and orientation information. Here, step S404 functions as a second position and orientation measurement step of measuring a second position and orientation of the moving object based on sensor data acquired by a sensor other than the imaging device.

[0033] The second position and attitude information is not limited to a particular sensor as long as it is measured based on the measurement values ​​of a sensor, such as a 2D LiDAR or wheel odometry, different from the first sensor 10. That is, the sensor (second sensor 11) other than the imaging device may include a LiDAR or an odometry sensor.

[0034] In step S405, the environment map update resumption determination unit 306 performs processing to determine whether or not the environment map can be resumed after recovery from the lost state. If it is determined in step S405 that the environment map can be resumed, the process proceeds to step S406.

[0035] If it is determined in step S405 that resumption is not possible, the process proceeds to step S407. Whether or not resumption is possible is determined based on whether or not the amount of feature points in the captured image acquired in step S402 is equal to or greater than a predetermined threshold value.

[0036] The above-mentioned predetermined threshold value can be set as a parameter in a configuration file or a program. In determining whether or not resumption is possible, not only the amount of feature points but also the distribution of feature points on a captured image or the distribution in a three-dimensional space may be added as conditions. Furthermore, the threshold value may be automatically updated taking into account the amount of feature points obtained while the moving object 1 is traveling.

[0037] That is, the update resumption determination unit determines whether or not to resume updating based on at least one of the number of image features detected in a captured image, or their distribution on the image, and their distribution in a three-dimensional space.

[0038] The determination of whether or not the moving object 1 is in a lost state is performed by the environment map updating unit 304. The environment map update resumption determining unit 306 receives information in advance as to whether or not the moving object 1 is in a lost state from the environment map updating unit 304. If the moving object 1 is not in a lost state, the environment map update resumption determining unit 306 may skip the process of determining whether or not updating of the environment map can be resumed.

[0039] In step S406, the environmental map update unit 304 performs a process of determining the position and attitude information to be used when the moving object 1 returns from the lost state. The position and attitude information to be used at the time of return is, for example, the second position and attitude information acquired in step S104, and is closer in time to the first position and attitude information acquired in step S403.

[0040] The position and orientation information used at the time of return may be not only the second position and orientation information but also information stored in a map of the environment, or may be estimated by combining the above two pieces of information. Furthermore, if there is no second position and orientation information close to the time, the values ​​may be estimated from the second position and orientation information at another time.

[0041] Furthermore, the position and orientation information used during return may be estimated from a combination of not only the second position and orientation information but also the first position and orientation information.

[0042] In step S407, the environment map update unit 304 performs processing to update the environment map based on the first position and orientation. Here, step S407 functions as an environment map update step for updating the environment map based on the first position and orientation.

[0043] A method for updating the environmental map will be described later. In step S408, the environmental map update unit 304 performs processing to determine whether or not to end the creation of the environmental map. Specifically, if an end instruction from the user 101 is received by the tablet 102, the flow of Fig. 4 ends. If not, the process returns to step S402 and continues.

[0044] Fig. 5 is a flowchart showing the processing procedure for updating the environmental map in step S407 in Fig. 4. Note that the CPU as a computer in the information processing device 100 executes a computer program stored in the memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 5.

[0045] In step S501, the environment map update unit 304 performs processing to determine whether or not the moving object 1 is in a lost state. The environment map update unit 304 holds a state variable indicating whether or not the moving object 1 is in a lost state, and makes the determination by referring to the state variable.

[0046] That is, for example, if the value falls below a predetermined state variable, it is determined to be in a lost state. If it is determined to be in a lost state in step S501, the process proceeds to step S502.

[0047] In step S502, the environment map update resumption determination unit 306 determines whether or not the creation of the map of the environment can be resumed. Here, step S502 functions as an update resumption determination step for determining whether or not the update can be resumed when the update of the environment map based on the first position and orientation is interrupted. Specifically, the same determination as that in the case where recovery from the lost state is possible in step S405 in Fig. 4 may be made, but the determination may also be made by combining multiple other pieces of information.

[0048] For example, a method may be used in which map creation can be resumed if recovery is possible for a certain period of time. If it is determined in step S502 that map creation cannot be resumed, the sequence of FIG.

[0049] On the other hand, if it is determined in step S502 that map creation can be resumed, the process proceeds to step S506. In step S506, the environment map update unit 304 performs a process of adding the position and orientation information to be used upon return, determined in step S406 of FIG. 4, to the environment map as a map element.

[0050] Then, in step S507, the environment map updating unit 304 sets the state variable indicating the lost state to OFF, the environment map updating unit 304 cancels the lost state, and resumes creating a map of the environment. In this way, when the updating of the environment map based on the first position and orientation is interrupted in steps S506 and S507, the environment map updating unit 304 resumes updating of the environment map using the second position and orientation. After the processing of step S507, the sequence of FIG. 5 ends.

[0051] If it is not determined that the state is lost in step S501, the environment map update unit 304 determines in step S503 whether the first position and orientation information was acquired in step S403 of FIG.

[0052] If it is determined in step S503 that the first position and orientation information has been acquired, the process proceeds to step S504. In step S504, the environment map update unit 304 performs a process of adding the first position and orientation information acquired in S403 of Fig. 4 to the environment map as a map element, and ends the sequence in Fig. 5.

[0053] On the other hand, if the first position and orientation information cannot be acquired in step S503, the process proceeds to step S505. In step S505, the environment map update unit 304 performs a process of turning on a state variable indicating a lost state, and interrupts the creation of the environment map. When the process of step S505 is completed, the sequence of FIG. 5 is terminated.

[0054] Fig. 6 is an image diagram of an environment map stored in the environment map storage unit 305 in Fig. 4 being displayed, for example, on the tablet 102 of the user 101 via the environment map display unit 307. In Fig. 6, 601 indicates an area of ​​walls where the moving object 1 in Fig. 1 cannot travel.

[0055] 602 shows the route traveled by the moving object 1 when it was not lost, and 603 shows the route traveled by the moving object 1 when it was lost. 604 shows the route traveled by the moving object 1 when it recovered from being lost and was not lost. In other words, it shows that the moving object 1 traveled along the trajectory from 602 to 604.

[0056] Map elements 605, 606, and 607 are added to the map of the environment. After that, the moving object 1 is in a lost state, and it is shown that there are no map elements on the route 603 in the lost state.

[0057] Numeral 608 indicates a map element that is added to the map of the environment after recovery from a lost state. Numeral 609, like 605 to 607, is a map element that is added to the map of the environment when the moving object 1 travels, and the map element is added to the map of the environment for the subsequent trajectory unless the moving object 1 is in a lost state. In this way, when the moving object 1 recovers from a lost state, it can resume creating the map of the environment.

[0058] <Variation 1> When adding a map element, the environment map update unit 304 may set and output a reliability indicating the accuracy of the map element.

[0059] Then, in step S506, the reliability of map elements in the lost state and for a certain period of time after recovery from the lost state and restarting map creation may be set to a lower level. That is, the environment map update unit may temporarily set the reliability of the environment map to a lower level after restarting updating of the environment map.

[0060] Furthermore, the moving body 1 may refer to a map of the environment and, if the reliability of a map element is low, may reduce the traveling speed of the autonomous traveling, thereby enabling the moving body 1 to travel safely.

[0061] <Variation 2> In step S502 in Fig. 5, the environment map update resumption determination unit 306 may add the distribution state of the feature points to the determination conditions when determining whether the creation of the environment map can be resumed. That is, for example, the covariance matrix of the coordinates of the feature points on the image may be calculated, and it may be determined that the creation can be resumed if the minimum eigenvalue is equal to or greater than a predetermined threshold. Alternatively, it may be determined that the creation cannot be resumed if the distribution of the feature points in the three-dimensional space is concentrated on a straight line.

[0062] Also, in step S502, when determining whether creation of the map of the environment can be resumed, the image may be divided into grids, the number of feature points in each grid may be counted, and it may be determined that creation of the map of the environment can be resumed if the number is equal to or greater than a predetermined threshold.

[0063] Also, in step S502, when determining whether or not the creation of the map of the environment can be resumed, if there is a large change in brightness or a large change in layout on the images in successive captured images, it may be determined not to resume. That is, the update resumption determination unit may determine whether or not the update can be resumed based on at least one of the change in brightness and the change in layout of the captured images in successive captured images.

[0064] <Variation 3> If it is determined that the answer is No in step S502 of FIG. 5, the environment map update unit 304 may perform a process of adding a map element using the second position and orientation information to the environment map.

[0065] Also, in step S504 in FIG. 5, the environment map update unit 304 may perform a process of adding map elements to the environment map using both the first position and orientation information as well as the second position and orientation information.

[0066] Furthermore, map elements may be assigned a type indicating whether first position and orientation information (first information), or second position and orientation information (second information), or both (first information and second information) are used.

[0067] That is, the environmental map update unit may assign to the environmental map any one of the types of information based on the first position and attitude, or based on the second position and attitude, or information including both the information based on the first position and attitude and the information based on the second position and attitude. Also, the reliability of the environmental map may be set using the type of map element to be useful for the autonomous driving of the mobile body 1.

[0068] <Modification 4> The environment map updating unit 304 may perform a process of deleting the map element added immediately before when the environment map is set to the lost state in step S505 in Fig. 5. That is, when the update of the environment map is interrupted, the environment map updating unit 304 may delete the first position and orientation used for the update immediately before the interruption from the map of the environment. At that time, the environment map updating unit 304 may delete up to a preset number of N map elements (N is a value equal to or greater than 0).

[0069] <Variation 5> In step S502, it may be determined whether map creation can be resumed, taking into consideration the reliability of the measurement result of the second position and orientation measurement unit acquired in step S405 of Fig. 4. In other words, the update resumption determination unit may determine whether update can be resumed, based on the reliability of sensor data acquired by a sensor (second sensor 11) other than the imaging device.

[0070] For example, when a LiDAR is used as the second sensor 11, if the distance between the moving body 1 and the wall is long or there is a lot of rainfall, the sensor information obtained from the LiDAR cannot be acquired stably, the measurement results of the second position and orientation measurement unit become unstable, and the reliability decreases.

[0071] Therefore, when a LiDAR or the like is used as the second sensor 11, in step S502, if the distance to the subject is equal to or greater than a predetermined value or if the amount of rainfall is equal to or greater than a predetermined value, it may be determined that map creation cannot be resumed.

[0072] Also, in step S502, if resumption is possible based on the information obtained from the second sensor 12, but the position and attitude information at the time of return in step S506 is inappropriate, it may be determined as a final decision that resumption is not possible.

[0073] <Variation 6> The environment map update unit 304 performs the process of updating the environment map in step S407 in FIG. 4, but it is also possible to temporarily store the captured images and sensor data required for the update and perform the process of updating the environment map all at once after step S408.

[0074] <Embodiment 2> In the first embodiment, a configuration is shown in which the moving body 1 recovers from the lost state and can resume creating a map of the environment when the moving body 1 becomes lost. In the second embodiment, an example is described in which, when recovering from the lost state and resuming updating of the map of the environment, the map of the environment is optimized so that the moving body 1 can travel autonomously with high accuracy.

[0075] Fig. 7 is a flowchart showing the update processing procedure of the information processing method in the second embodiment, and shows another example of the environmental map update processing in step S407 in Fig. 4. Note that the CPU as a computer in the information processing device 100 executes a computer program stored in the memory, thereby sequentially performing the operations of each step in the flowchart in Fig. 7. Note that the steps in Fig. 7 with the same reference numbers as those in Fig. 5 are the same processes, and therefore their explanations will be omitted.

[0076] In FIG. 7, in step S506, the environment map update unit 304 performs a process of adding the position and orientation information to be used upon return, determined in step S406 in FIG. 4, to the environment map as a map element, and then the process proceeds to step S701. In step S701, the environmental map update unit 304 sets a constraint condition for the map element added in step S506. That is, the second position and orientation information obtained by 2D LiDAR or the like is set as a constraint condition for optimization. That is, the constraint condition is set for the map element so that the amount of rotation around the normal to the plane on which the moving body moves and the position on the plane, among the second position and orientation information, become the position and orientation of the corresponding map element.

[0077] After the process of step S701, in step S507, the environment map update unit 304 turns off the state variable indicating the lost state, and then proceeds to step S702. Also, if it is determined in step S502 that map creation cannot be resumed, or after the processes of steps S504 and S505, the process proceeds to step S702.

[0078] In step S702, the environment map update unit 304 performs a process of optimizing the map of the environment. That is, the environment map update unit 304 fixes the map elements for which the constraint conditions were set in step S701, for example.

[0079] On the other hand, for map elements for which no constraints are set, the system performs optimization processing of the map of the environment by correcting the position and orientation information, for example, by performing pose graph optimization and bundle adjustment using SLAM technology.

[0080] Here, step S702 functions as an optimization step (optimization unit) that optimizes at least a part of the environmental map based on the second position and orientation. After the process of step S702, the sequence of FIG.

[0081] As described above, in the second embodiment, when the moving object 1 recovers from a lost state and resumes creating a map of the environment, the map of the environment is optimized.

[0082] <Embodiment 3> In the first embodiment, a configuration has been described in which the moving body 1 recovers from the lost state and can resume creating a map of the environment when the moving body 1 becomes lost. In the third embodiment, an example is shown in which the created map of the environment is displayed to the user 101, and an area in which the moving body 1 can start moving autonomously is also displayed.

[0083] Fig. 8 is a flowchart showing a procedure for displaying an environmental map in the information processing method according to embodiment 3. Note that the CPU as a computer in information processing device 100 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 8.

[0084] The flow described in Fig. 8 is started when an instruction is given to start preparation for traveling of the moving object 1 in Fig. 1. First, in step S801, the environmental map display unit 307 performs processing to acquire the map of the environment held in the environmental map holding unit 305.

[0085] In step S802, the environment map display unit 307 analyzes map elements included in the map of the environment and performs processing to convert them into display information for the environment map display unit 307. That is, the environment map display unit 307 creates rectangle data from which autonomous movement can be started, based on the position information of the map elements, and creates data indicating the direction from which autonomous movement can be started, based on the attitude information of the map elements.

[0086] The data is then sent to, for example, the tablet 102 of the user 101 to be displayed. After the process of step S802, the flow of FIG.

[0087] Fig. 9 is an image diagram of a visualized map of the environment obtained by executing the flowchart of Fig. 8. 901 indicates a walled area where the moving object 1 of Fig. 1 cannot travel. 902 and 903 indicate areas of the map of the environment where map elements exist, and indicate that the moving object 1 can resume autonomous movement in the direction of the arrow.

[0088] That is, in the third embodiment, the environment map display unit 307 generates a display control signal for displaying the environment map, and the area and direction in which the first position and orientation information can be used.

[0089] It is shown that it is difficult for the moving object 1 to resume autonomous movement in places other than the areas 902 and 903. The user 101 can start autonomous movement after manually moving the moving object 1 to either the place 902 or 903.

[0090] 9, the area and direction in which autonomous movement can be started are shown using rectangles and arrows, but they may be shown in other ways, such as colors or ellipses. Also, the area in which it is difficult to start autonomous movement may be displayed using rectangles or the like, or may be displayed using colors or density according to the degree of difficulty in starting autonomous movement.

[0091] Furthermore, when the user 101 specifies the resumption position of the moving object 1 on the screen displayed on the tablet 102, it is also possible to notify the user in advance, or when the moving object 1 is lost, of whether or not it is possible to start autonomous movement.

[0092] As described above, in the third embodiment, by displaying to the user 101 in advance a location where the autonomous movement can be started, the user 101 can easily resume the autonomous movement of the moving object 1 through an operation by the user 101.

[0093] In the above embodiment, the present invention is applied to an autonomous moving body. However, the moving body in the present embodiment is not limited to an autonomous moving body such as an AGV (Automated Guided Vehicle) or an AMR (Autonomous Mobile Robot).

[0094] In addition, the moving object may be used as a driving assistance device even if it does not move completely autonomously. In addition, the moving object may be any moving device that moves, such as a car, a train, a ship, an airplane, a robot, or a drone.

[0095] Furthermore, at least a part of the information processing system of the embodiment may or may not be mounted on a mobile object. The present invention can also be applied to a case where a mobile object is remotely controlled.

[0096] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. Note that the present invention includes combinations of the above embodiments, and also includes the following combinations.

[0097] (Configuration 1) An information processing device comprising: an environmental map holding unit that holds an environmental map; an captured image input unit that inputs an image of an environment captured by an imaging device; a first position and orientation measurement unit that measures a first position and orientation of a moving body on which the imaging device is mounted based on the environmental map and the captured image; a second position and orientation measurement unit that measures a second position and orientation of the moving body based on sensor data acquired by a sensor other than the imaging device; and an environmental map update unit that updates the environmental map based on the first position and orientation, wherein the environmental map update unit resumes the updating of the environmental map using the second position and orientation when the updating of the environmental map based on the first position and orientation is interrupted.

[0098] (Configuration 2) The information processing device according to configuration 1, further comprising an update resumption determination unit that determines whether or not the update can be resumed when the update of the environmental map based on the first position and orientation is interrupted.

[0099] (Configuration 3) The information processing device according to Configuration 2, wherein the update resumption determination unit determines whether or not to resume the update based on at least one of the number of image features detected on the captured image or their distribution on the image, or their distribution in three-dimensional space.

[0100] (Configuration 4) The information processing device according to configuration 2 or 3, wherein the update resumption determination unit determines whether or not to resume the update based on at least one of a change in brightness of the consecutive captured images and a change in layout of the consecutive captured images.

[0101] (Configuration 5) The information processing device according to any one of configurations 2 to 4, wherein the update resumption determination unit determines whether or not to resume the update based on the reliability of the sensor data acquired by a sensor other than the imaging device.

[0102] (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the sensor other than the imaging device includes a LiDAR.

[0103] (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the sensor other than the imaging device includes an odometry sensor.

[0104] (Configuration 8) The information processing device described in any one of configurations 1 to 7, characterized in that the environmental map update unit temporarily sets the reliability of the environmental map to a low level after resuming the update of the environmental map.

[0105] (Configuration 9) An information processing device described in any one of configurations 1 to 8, characterized in that when the update of the environmental map is interrupted, the environmental map update unit deletes from the map of the environment the first position and orientation used in the update immediately before the interruption.

[0106] (Configuration 10) An information processing device described in any one of configurations 1 to 9, characterized in that the environmental map update unit assigns to the map of the environment one of the following types of information: information based on the first position and orientation, or information based on the second position and orientation, or information including both information based on the first position and orientation and information based on the second position and orientation.

[0107] (Configuration 11) The information processing device according to any one of configurations 1 to 10, further comprising an optimization unit that optimizes at least a part of the environmental map based on the second position and orientation.

[0108] (Configuration 12) An information processing device according to any one of configurations 1 to 11, further comprising an environmental map display unit that generates a display signal for displaying the environmental map and the area and direction in which the first position and attitude information can be used.

[0109] (Configuration 13) A moving body comprising: a drive control unit that performs autonomous traveling based on the environmental map updated by the environmental map update unit of the information processing device according to any one of configurations 1 to 12.

[0110] (Method 7) An information processing method comprising: an environmental map holding step of holding an environmental map; an captured image input step of inputting an captured image of an environment captured by an imaging device; a first position and orientation measurement step of measuring a first position and orientation of a moving body mounted with the imaging device based on the environmental map and the captured image; a second position and orientation measurement step of measuring a second position and orientation of the moving body based on sensor data acquired by a sensor other than the imaging device; and an environmental map update step of updating the environmental map based on the first position and orientation, wherein the environmental map update step resumes the updating of the environmental map using the second position and orientation when the updating of the environmental map based on the first position and orientation is interrupted.

[0111] (Program) A computer program for controlling each part of the information processing device according to any one of configurations 1 to 12 or the moving object according to configuration 13 by a computer.

[0112] In order to realize a part or all of the control in the above-mentioned embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0113] 1: Moving object 10: First sensor 11: Second sensor 100: Information processing device

Claims

1. an environmental map storage unit for storing an environmental map; a captured image input unit for inputting a captured image of an environment captured by an imaging device; a first position and orientation measurement unit that measures a first position and orientation of a moving object on which the imaging device is mounted, based on the environmental map and the captured image; a second position and orientation measurement unit that measures a second position and orientation of the moving object based on sensor data acquired by a sensor other than the imaging device; an environment map update unit that updates the environment map based on the first position and orientation; The information processing device is characterized in that the environmental map update unit resumes the updating of the environmental map using the second position and orientation when the updating of the environmental map based on the first position and orientation is interrupted.

2. 2 . The information processing apparatus according to claim 1 , further comprising an update resumption determination unit that determines whether or not the update can be resumed when the update of the environmental map based on the first position and orientation is interrupted.

3. The information processing device according to claim 2 , wherein the update resumption determination unit determines whether or not to resume the update based on at least one of the number of image features detected in the captured image, or their distribution on the image, and their distribution in three-dimensional space.

4. The information processing apparatus according to claim 2 , wherein the update resumption determination unit determines whether or not the update can be resumed based on at least one of a change in brightness of the consecutive captured images and a change in layout of the consecutive captured images.

5. The information processing apparatus according to claim 2 , wherein the update resumption determination unit determines whether or not to resume the update based on reliability of the sensor data acquired by a sensor other than the imaging device.

6. The information processing device according to claim 1 , wherein the sensor other than the imaging device includes a LiDAR.

7. 2. The information processing apparatus according to claim 1, wherein the sensor other than the imaging device includes an odometry sensor.

8. The information processing apparatus according to claim 1 , wherein the environment map update unit temporarily sets the reliability of the environment map to a low level after restarting the update of the environment map.

9. The information processing device according to claim 1, characterized in that, when the update of the environmental map is interrupted, the environmental map update unit deletes from the map of the environment the first position and orientation used in the update immediately before the interruption.

10. The information processing device according to claim 1, characterized in that the environmental map update unit assigns to the map of the environment one of the following types of information: information based on the first position and orientation, information based on the second position and orientation, or information including both information based on the first position and orientation and information based on the second position and orientation.

11. The information processing apparatus according to claim 1 , further comprising an optimization unit that optimizes at least a part of the environmental map based on the second position and orientation.

12. 2. The information processing apparatus according to claim 1, further comprising an environmental map display unit that generates a display signal for displaying the environmental map and an area and a direction in which the first position and orientation information can be used.

13. A mobile body comprising a drive control unit that performs autonomous traveling based on the environmental map updated by the environmental map update unit of the information processing device according to any one of claims 1 to 12.

14. an environment map maintaining step of maintaining an environment map; a captured image input step of inputting a captured image of an environment captured by an imaging device; a first position and orientation measurement step of measuring a first position and orientation of a moving object on which the imaging device is mounted, based on the environmental map and the captured image; a second position and orientation measurement step of measuring a second position and orientation of the moving object based on sensor data acquired by a sensor other than the imaging device; an environment map updating step of updating the environment map based on the first position and orientation; The information processing method is characterized in that the environmental map update step resumes the updating of the environmental map using the second position and orientation when the updating of the environmental map based on the first position and orientation is interrupted.

15. A computer program for controlling each part of the information processing device according to any one of claims 1 to 12 or the moving body according to claim 13 by a computer.

Images