Information processing device, information processing method, and program

Abstract

To reduce a processing load in the association between map elements already held in a three-dimensional map and a new map element.SOLUTION: An information processing apparatus: selects at least one map element from first three-dimensional map information comprising a plurality of map elements, the map elements including the position and posture of an imaging apparatus mounted on a movable body, the position and posture being estimated by photographing an environment with the imaging apparatus, three-dimensional position information in the environment on feature points included in an image obtained by photographing the environment; selects, from map elements of second three-dimensional map information the environment of which is already measured, a map element including a feature element having three-dimensional position information within a predetermined distance from a position indicated by three-dimensional information on a feature point included in the selected map element, and associates the map element selected from the first three-dimensional map information and the map element selected from the second three-dimensional map information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for creating three-dimensional map information of an environment in which a mobile object moves. [Background technology]

[0002] Position and orientation estimation processing of an imaging device using image information captured by the imaging device is used for various purposes, such as self-position estimation for autonomous driving of mobile bodies such as robots, 3D modeling of objects and spaces, etc. Position and orientation estimation uses a 3D map of the environment in which image features detected from the image information and 3D position information of the image features calculated from them are used as map elements.

[0003] Non-Patent Document 1 describes that when a new map element not held in the 3D map is observed during position and orientation estimation, the new map element is associated with map elements already held in the 3D map, and the position and orientation information of the new map element is corrected. By correcting the position and orientation information, accumulated errors in position and orientation estimation are reduced. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Raul Mur-Artal, JMM Montiel and Juan D. Tardos, “ORB-SLAM2:an Open-Source SLAM System for Monocular,Stereo and RGB-D Cameras”IEEE Transactions on Robotics,vol.33,no.5,pp.1255-1262,2017. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method of Non-Patent Document 1, map elements of already stored 3D maps that are unlikely to match are checked to see whether they match with new map elements, which imposes a high processing load on matching map elements.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to reduce the processing load involved in associating new map elements with map elements of already-held three-dimensional maps. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to the present invention has the following configuration: A first selection means selects at least one map element from first three-dimensional map information composed of a first map element group, using as map elements position and orientation information including an accumulated error of an imaging device mounted on a moving body, which is estimated by capturing an image of an environment with the imaging device, and three-dimensional position information in the environment of a feature point included in an image obtained by capturing the image of the environment; Within the scope the map element selected by the first selection means is obtained based on an environment in which the map element selected by the first selection means is obtained; a second selection means for selecting a map element from the second group of map elements constituting the second three-dimensional map information, the map element being within a range acquired by the acquisition means, the distance between a position indicated by three-dimensional position information of a feature point included in the map element from the second group of map elements constituting the second three-dimensional map information and a position indicated by three-dimensional position information of a feature point included in the map element selected by the first selection means; and a matching means for matching the map element selected by the first selection means with the map element selected by the second selection means. Effect of the Invention

[0008] The present invention can reduce the processing load required for associating new map elements with map elements of already stored three-dimensional maps. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an overall configuration diagram of a mobile robot including an information processing device. [Diagram 2] 4 is a flowchart illustrating a process according to an embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of a second map element selection process in the embodiment. [Figure 4] 1 shows an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In this embodiment, an information processing device that performs position and orientation estimation of a camera mounted on a mobile robot moving along a predetermined route by SLAM (Simultaneous Localization and Mapping) will be described. When the mobile robot changes its route to avoid an obstacle, or when the arrangement of surrounding objects changes, a new map element is added to three-dimensional map information (hereinafter, also simply referred to as "map"). In order to reduce an error included in the camera position and orientation of the added map element, the device of this embodiment associates the added map element with an existing map element. At that time, the range for searching the existing map element is narrowed down to within a predetermined range from the estimated position of the camera included in the added map element, thereby reducing the number of map elements to be targeted for optimization processing, and thus reducing the processing load. The contents of the optimization processing will be described later.

[0011] It should be noted that the configuration shown in this embodiment is merely an example, and the present invention is not limited to the illustrated configuration.

[0012] FIG. 1 shows the functional configuration of the information processing device.

[0013] The information processing device 100 is composed of a map holding unit 110, a first map element selection unit 120, a second map element selection unit 130, and a matching unit 140. The map holding unit 110 holds a map created by a preliminary map creation unit 160. The preliminary map creation unit 160 measures the environment in which the mobile robot moves before executing the process described in this embodiment, and creates a three-dimensional map of the environment. The map is composed of a plurality of map elements, and the map elements hold three-dimensional position information (observed feature point information) of feature points of an observation target observed, for example, using a stereo camera at a position and orientation estimation point of the camera, and position and orientation information of the camera at each position and orientation estimation point. Hereinafter, the observed feature point information will be described as including three-dimensional position information of the observed feature points. The map element adding unit 170 adds the newly measured map elements to the map held by the map holding unit 110. The map elements include both map elements (existing map elements) that have been optimized by the correction unit 150 and are already held in the map, and map elements (additional map elements) that have been added by the map element adding unit 170 and have not yet been optimized. To distinguish between these two, the map elements also hold information on whether they have been optimized. The first map element selecting unit 120 selects one map element (additional map element) that is not optimized and is included in the map. In this embodiment, it is assumed that one of the added map elements is selected. The second map element selecting unit 130 selects a map element (existing map element) that has observation feature point information at a position that is close to the observation feature point information included in the added map element selected by the first map element selecting unit 120 from the pre-measured map. The matching unit 140 estimates the correspondence relationship between the observation feature point information included in the added map element selected by the first map element selecting unit 120 and the observation feature point information included in the existing map element selected by the second map element selecting unit 130.

[0014] The correspondence relationship between each observed feature point obtained by the matching unit 140 is used in the optimization process of the position and orientation information of the camera of the map element by the correction unit 150. In the optimization process by the correction unit 150, a deviation between the position and orientation information of the camera included in the additional map element selected by the first map element selection unit 120 and the position and orientation information of the camera included in the existing map element selected by the second map element selection unit 130 is obtained from the correspondence relationship between each observed feature point. To eliminate this deviation, the correction unit 150 corrects the position and orientation information of the camera included in the additional map element selected by the first map element selection unit 120. In this embodiment, this optimization process is performed as a bundle adjustment process. In addition to the bundle adjustment, optimization processes such as a reprojection error minimization process and an ICP (Interactive Closet Point) process can be used.

[0015] FIG. 4 is a diagram showing a hardware configuration of the information processing device 100. H11 is a CPU, which controls various devices connected to the system bus H21. H12 is a ROM, which stores a BIOS program and a boot program. H13 is a RAM, which is used as a main storage device of the CPU H11. H14 is an external memory, which stores programs processed by the information processing device 1. An input unit H15 performs processing related to input of information, etc., using a keyboard or mouse. A display unit H16 outputs the calculation result of the information processing device 1 to a display device according to an instruction from H11. The display device may be a liquid crystal display device, a projector, an LED indicator, or any other type. H17 is a communication interface, which performs information communication via a network. The communication interface may be Ethernet, or any type such as USB, serial communication, or wireless communication. The calculated position and orientation information is exchanged with a control unit of a moving body (not shown) via the communication interface 17. H18 is an I / O, which inputs an image from the imaging device 10.

[0016] 2 is a flowchart explaining the operation of the information processing device 100. Hereinafter, it is assumed that the flowchart is realized by the CPU executing a control program. The operation of the information processing device starts when an instruction is given to start moving a moving object. Although a case where all steps are executed within the information processing device will be explained, steps S200 to S202 and S207 can also be executed by a device external to the information processing device 100, and the information processing device 100 can start processing from S203.

[0017] In step S200, the advance map creation unit 160 creates a three-dimensional map of the environment in which the mobile robot will move in advance. Step S200 can be executed by the information processing device 100, or the map can be created by another device and the information processing device 100 can skip step S200.

[0018] In step S201, the map holding unit 110 holds the map created by the advance map creation unit 160 in step S200. All of the maps created by the advance map creation unit 160 can be held in the map holding unit 100. Alternatively, the advance map can be held in an external server connected via a network, and maps required for processing by the information processing device 100 can be downloaded and held in the map holding unit 110.

[0019] Thereafter, when a new map element is measured by the position and orientation estimation process, in step S202, the map element adding unit 170 adds the map element to the map held by the map holding unit 110. Step S202 can be executed by the information processing device 100, or the information processing device 100 can not execute step S202.

[0020] In step S203, it is determined whether there is a map element that the first map element selection unit 120 will select as the first map element in the map held by the map holding unit 110. In other words, the first map element selection unit 120 determines whether there is a map element that will be subjected to optimization processing by the correction unit 150 and has not yet been selected by the first map element selection unit 120. At this time, it is not necessary to select as the first map element all map elements that have not been subjected to optimization processing by the correction unit 150, and the selection may be limited to map elements that are the targets of optimization processing. For example, if the most recently added map element is selected as the first map element, the correction unit 150 will execute optimization processing every time a map element is added.

[0021] If step S203 is true, the first map element selection unit 120 selects a first map element in step S204. In this embodiment, one additional map element is selected.

[0022] In step S205, an existing map element having observed feature point information at a position close to the observed feature point information included in the additional map element selected by the first map element selection unit 120 in step S204 is selected. To this end, the second map element selection unit 130 first determines a search range for the map elements to be selected. The method of determining the search range will be described later. After determining the search range, the second map element selection unit 130 selects existing map elements included within the search range.

[0023] Next, in step S206, the association unit 140 associates the first map element with the second map element. This association method is the same as the method described in Non-Patent Document 1. That is, a pair of observed feature points at approximate positions is searched for among the map element selected in step S204 and the observed feature point group included in the map element selected in step S205.

[0024] After that, the process returns to step S203, and if there is a first map element to select, steps S204 to S206 are repeated. If there is no first map element to select, in step S207, the correction unit 150 corrects the camera position and orientation information of the additional map element selected by the first map element selection unit 120, using the correspondence relationship determined by the correspondence unit 140. The method of this correction is also the same as the method described in Non-Patent Document 1.

[0025] The above is the process flow executed in the information processing device 100.

[0026] In this embodiment, when searching for a map element to be selected by the second map element selection unit 130, a specified distance is determined, and an existing map element in which the camera position is located within a search range having a radius of the distance is selected. This is because an existing map element within a specified distance from an additional map element is highly likely to have common feature points, that is, is highly likely to be associated with the additional map element. In this embodiment, the radius distance is determined based on the area in which the mobile robot can move. In this embodiment, the size of the search range to be searched is determined based on the width of the passage through which the mobile robot passes. Specifically, planes corresponding to floors and walls are assigned to the observed feature point group linked to the additional map element selected by the first map element selection unit 120, and the passage width is calculated from the positional relationship (between the walls) of those planes. This makes it possible to set a search range that matches the environment even if the area in which the mobile robot can move changes. A range that is sufficient as existing map elements that may be associated by the association unit 140 can be set as the search range.

[0027] 3 shows an example of application of the method of determining the search range of the map element selected by the second map element selection unit 130 in this embodiment. First, the passage width 310 of the passage 320 in which the additional map element 300 selected by the first map element selection unit 120 exists is calculated. A circle 340 having the additional map element 300 selected by the first map element selection unit 120 as its center and the calculated passage width 310 as its radius 330 is set as the search range of the map element selected by the second map element selection unit 130. The second map element selection unit 130 selects existing map elements 350, 360 included in the circle 340 as existing map elements to be associated with the additional map element 300 by the association unit 140.

[0028] In addition to the method of calculating the aisle width from the observed feature point information, the aisle width on the route may be measured in advance, a reference table in which the aisle width corresponding to the position on the route is recorded may be created, and the search range may be determined by referring to the reference table based on the position of the mobile robot. A fixed aisle width may be determined for all the aisles. Also, instead of a camera that outputs brightness, a separate sensor such as an infrared sensor, a TOF sensor, or a range image sensor may be prepared, and the aisle width may be calculated from the results measured by this sensor.

[0029] Also, instead of performing the optimization process each time a map element is added, a 3D map composed of map elements whose camera positions and orientations and feature points are misaligned can be created once, and the optimization process can be performed later. In this case, the process of the information processing device 100 starts from S203 in FIG. 2.

[0030] The search range may be determined not only according to the movable area of ​​the mobile robot, but also according to the distance from the point where the addition of the map element to the map is started to the sensor measurement position of the additional map element selected by the first map element selection unit 120. This is a method of determining the search range taking into consideration an environment without walls, a case where the mobile body returns to the original route by avoiding an obstacle, etc. In this case, the search range is set so that the search range becomes wider as the distance from the point where the addition of the map element to the map is started to the camera measurement position of the additional map element selected by the first map element selection unit 120 increases. For example, the search range is determined by the horizontal movement distance with respect to the moving direction of the mobile body from the point where the addition of the map element to the map is started to the camera measurement position of the additional map element selected by the first map element selection unit 120 using the position and orientation estimation information of the camera included in the additional map element. This horizontal movement distance is set as the radius of the circle that is the search range of the map element selected by the second map element selection unit 130. The horizontal movement distance corresponds to, for example, the distance that the mobile body returns to the original route after avoiding an obstacle. Alternatively, the size of the search range may be the sine of the following angle relative to the distance from the point where adding of the map element to the map started to the camera measurement position of the added map element selected by the first map element selection unit 120. The angle is the angle formed by the orientation of the camera included in the added map element at the point where adding of the map element to the map started and the orientation of the camera included in the added map element selected by the first map element selection unit 120.

[0031] In this embodiment, the map element selected by the first map element selection unit 120 is one of the additional map elements, but an existing map element may be selected. In this case, the map held by the map holding unit 110 may or may not have the additional map element.

[0032] In this way, in the first embodiment, the search range of the map elements to be selected by the second map element selection unit 130 is determined taking into consideration distances in the physical environment, such as passage width and travel distances to avoid obstacles. The processing load for associating map elements can be reduced. Furthermore, if the map elements are optimized, the position and orientation of the moving object can be measured more accurately. As a result, the moving object can move stably.

[0033] [Variation 1] In this modification, a modification of step S205 will be described. A method for reducing the processing load by narrowing the search range of map elements selected by the second map element selection unit 130 to map elements including observed feature point information whose positions are similar to those of observed feature point information included in the map element selected by the first map element selection unit 120 will be described.

[0034] The functional configuration diagram and processing flow of the information processing device 100 are the same as those of the first embodiment, so the explanation will be omitted. Only the points that differ from the first embodiment will be explained.

[0035] In this modified example, each map element holds identification information of map elements that contain features whose positions are similar to those of features contained in the map element. That is, similar to the method of Non-Patent Document 1, a common feature point number graph is held, which indicates the number of common observation feature points between map elements, for observation feature points contained in each map element. The larger the value of the common feature point number graph, the more common observation feature points the map elements connected by the graph have, and the map element is the one that should be selected by the second map element selection unit 130.

[0036] The detailed processing content of step S205 in this modified example will be described below.

[0037] First, a criterion for determining the number of common feature points in the graph is determined, which is that the number of common feature points must be equal to or greater than a predetermined number.

[0038] Next, the second map element selection unit 130 selects map elements whose common feature point count graphs with the map elements selected by the first map element selection unit 120 are equal to or greater than a criterion. Of the selected map elements, those selected from the existing map elements are regarded as the map elements selected by the second map element selection unit 130.

[0039] The above is the method of searching for map elements selected by the second map element selection unit 130 in this modified example.

[0040] The criteria for determining the common feature point number graph may be the same for all map elements selected by the first map element selection unit 120, or different criteria may be set for each map element.

[0041] The judgment criterion is not limited to the number of common feature points. There is also a method in which, even if the number of common feature points is not equal to or greater than the threshold, if there are common feature points, the map elements may be included in the search range. The measured order of the map elements may be held as map element IDs, and the map element IDs may be used as the judgment criterion. In this case, the number of map element IDs going back from the map element selected by the first map element selection unit 120 may be set as the judgment criterion. The number of map element IDs going back may be the same value for all map elements, or different values ​​may be set. In addition, all map elements from the point where the addition of map elements to the map is started to the additional map element already selected by the first map element selection unit 120, or multiple map elements going back in map element IDs from the additional map element selected by the first map element selection unit 120 may be selected. Then, the existing map elements that satisfy the judgment criterion for the selected map element may be selected as the map elements to be selected by the second map element selection unit 130.

[0042] [Variation 2] In this modified example, the range in which the second map element search range 130 searches for existing map elements is narrowed down to a predetermined range from the camera measurement position of the added map element, and the search range is further narrowed down to map elements whose common feature point count graph is equal to or greater than a judgment criterion.

[0043] As described in the first embodiment, the search range of existing map elements selected by the second map element selection unit 130 is narrowed down in consideration of the physical environment, and further narrowed down to existing map elements that have common feature points as described in the first modification. This reduces the number of map elements to be associated, further reducing the processing load for associating map elements.

[0044] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable recording medium. [Explanation of symbols]

[0045] 100 Information processing device 110 Map holder 120 First map element selection section 130 Second map element selection section 140 Mapping section 150 Correction section 160 Advance Mapping Department 170 Map Element Addition Section

Claims

1. a first selection means for selecting at least one map element from first three-dimensional map information composed of a first group of map elements, the map elements being position and orientation information including accumulated errors of an imaging device mounted on a moving body and estimated by capturing an image of the environment with the imaging device, and three-dimensional position information in the environment of feature points included in an image obtained by capturing the image of the environment; an acquisition means for acquiring, based on the environment in which the map element selected by the first selection means is obtained, a range of distance from the estimated position of the imaging device when an image corresponding to the map element selected by the first selection means is captured, the range being included in second three-dimensional map information created in advance for the environment and used in the process of reducing the accumulated error; a second selection means for selecting a map element from the second map element group, the map element being selected by the first selection means, such that the distance between the position indicated by the three-dimensional position information of a feature point included in the map element from the second map element group constituting the second three-dimensional map information is within the range acquired by the acquisition means; and a correlation unit for correlating the map element selected by the first selection unit with the map element selected by the second selection unit.

2. The map elements further include at least one piece of identification information representing the number of common feature points between the map elements; 2. The information processing apparatus according to claim 1, wherein the second selection means selects the map element using identification information included in the map element selected by the first selection means.

3. further comprising a map element adding means for adding map elements to the three-dimensional map information held by the map holding means for holding the first and second three-dimensional map information; 2. The information processing apparatus according to claim 1, wherein the first selection means selects the map element added by the map element adding means.

4. 4. The information processing apparatus according to claim 1, wherein the second selection means acquires a range based on a width of a passage through which the moving body equipped with the imaging device can move.

5. further comprising a map element adding means for adding map elements to the three-dimensional map information held by the map holding means for holding the first and second three-dimensional map information; 2. The information processing apparatus according to claim 1, wherein the first selection means selects map elements from the map element to which the map element addition means has started adding the map element to a map element that has already been selected by the map element addition means.

6. 3. The information processing apparatus according to claim 2, wherein the second selection means selects map elements having a predetermined number of common feature points or more.

7. a first selection means for selecting at least one map element from first three-dimensional map information composed of a first group of map elements, the map elements being position and orientation information including accumulated errors of an imaging device mounted on a moving body and estimated by capturing an image of the environment with the imaging device, and three-dimensional position information in the environment of feature points included in an image obtained by capturing the image of the environment; an acquisition means for acquiring, based on the environment in which the map element selected by the first selection means is obtained, a range of distance from the estimated position of the imaging device when an image corresponding to the map element selected by the first selection means is captured, the range being included in second three-dimensional map information created in advance for the environment and used in the process of reducing the accumulated error; a second selection means for selecting a map element from the second map element group, the map element being selected by the first selection means, such that the distance between the position indicated by the three-dimensional position information of a feature point included in the map element from the second map element group constituting the second three-dimensional map information is within the range acquired by the acquisition means; a correspondence determining means for determining a correspondence between the map element selected by the first selecting means and the map element selected by the second selecting means; and a correction means for correcting, based on the correspondence relationship, position and orientation information of the imaging device included in the map element selected by the first selection means, using position and orientation information of the imaging device included in the map element selected by the second selection means.

8. a first selection step of selecting at least one map element from first three-dimensional map information composed of a first group of map elements, the map elements being position and orientation information including accumulated errors of an imaging device mounted on a moving body and estimated by capturing an image of the environment with the imaging device, and three-dimensional position information in the environment of feature points included in an image obtained by capturing the image of the environment; an acquisition step of searching for map elements included in second three-dimensional map information created in advance for the environment and to be used in the process of reducing the accumulated error, and acquiring a range of distances from an estimated position of the imaging device when an image corresponding to the map elements selected in the first selection step was taken, based on the environment in which the map elements selected in the first selection step are obtained; a second selection step of selecting a map element from the second map element group, the map element being selected in the first selection step, such that the distance between the position indicated by the three-dimensional position information of a feature point included in the map element from the second map element group constituting the second three-dimensional map information is within the range acquired in the acquisition step; an associating step of associating the map element selected in the first selecting step with the map element selected in the second selecting step.

9. A program that causes a computer to function as each of the means of the information processing device according to any one of claims 1 to 7.

Images