Information processing device, information processing method, and program

Abstract

To reduce the storage capacity for storing map elements of a three-dimensional map of an environment in which a movable body moves.SOLUTION: An information processing apparatus acquires three-dimensional map information comprising a plurality of map elements, the map elements including the position and posture of an imaging apparatus mounted on a movable body, the position and posture being estimated by photographing an environment with the imaging apparatus, three-dimensional position information in the environment on feature points included in an image obtained by photographing the environment, and the image. In two map elements where the positions and postures of the imaging apparatus are similar to each other, when the size of areas not similar to each other between images included in the map element is equal to or larger than a threshold, the information processing apparatus determines one of the map elements as an object of deletion.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for creating three-dimensional map information of an environment in which a mobile object moves. [Background technology]

[0002] There are methods for measuring the position and orientation of moving objects such as autonomous mobile robots, and for creating electronic map data to be used for position and orientation measurement. A well-known technology is Simultaneous Localization and Mapping (SLAM), which uses cameras and laser range scanners (laser range finders, Laser Imaging Detection and Ranging (LIDAR)) as sensors. When the path of a moving object changes, or when the location of surrounding objects changes, new map elements are added to the map data.

[0003] Non-patent document 1 discloses a technique for reducing the memory capacity required to store map data (three-dimensional map information) by deleting an image contained in a map element when the image has similar characteristics to an image contained in another map element. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Raul Mur-Artal, JMM Montiel,Member, IEEE,and Juan D. Tardos,Member, IEEE, “ORB-SLAM: a Versatile and AccurateMonocular SLAM System” 2015 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of Non-Patent Document 1, if a map element cannot be complemented by another map element, it cannot be deleted, and therefore, there are cases where the storage capacity is insufficient.

[0006] The present invention has been made in consideration of the above problems, and has as its object to reduce the memory capacity required to store map elements of a three-dimensional map of an environment in which a mobile object moves. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to the present invention has the following configuration: A position and orientation of an imaging device mounted on a moving body that is estimated by capturing an image of an environment with the imaging device, three-dimensional position information in the environment of feature points included in an image obtained by capturing the image of the environment, and the image are set as map elements, and the three-dimensional map information is acquired from a storage unit that stores three-dimensional map information composed of a plurality of the map elements, and in two map elements in which the position and orientation of the imaging device included in the map elements are similar, When the size of a region having dissimilar features in a common region captured in a common field of view among the images is equal to or larger than a threshold value, and a determining means for determining one of the map elements as a deletion target to be deleted from the storage means. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the memory capacity required to store map elements of a three-dimensional map of an environment in which a mobile object moves. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Diagram 2] 1 is a functional block diagram of an information processing device according to a first embodiment. [Diagram 3] FIG. 2 is a functional block diagram of a mobile body system and a map creation system according to the first embodiment. [Figure 4] 4 is an operation flowchart of the information processing device according to the first embodiment. [Diagram 5] 4 is an operation flowchart of the information processing device according to the first embodiment. [Figure 6] FIG. 13 is a diagram showing an example of a comparison of map elements. [Figure 7]10 is an operation flowchart of the information processing device according to the second embodiment. [Figure 8] FIG. 11 is a functional block diagram of an information processing device according to a third embodiment. [Figure 9] 13 is an operation flowchart of the information processing device according to the third embodiment. [Figure 10] FIG. 13 is a diagram showing an example of the selection of a map element. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. Note that the embodiments are merely examples, and are not intended to limit the scope of the present invention.

[0011] (First embodiment) As a first embodiment of an information processing device according to the present invention, an information processing device that determines map elements to be deleted based on the degree of change in the object arrangement around the sensor will be described below. Specifically, when the object arrangement around the sensor is changed, different map elements before and after the change in arrangement are stored, which requires storage capacity. Therefore, in the case of a temporary change in arrangement, the map elements after the change in arrangement are deleted because they are unlikely to be used again.

[0012] In this embodiment, the sensor is a camera that outputs a two-dimensional array of brightness value data. The camera may be a monochrome camera, a color camera, or a stereo camera. The internal parameters of the camera are assumed to be known through prior calibration.

[0013] 1 is a diagram showing the hardware configuration of an information processing device according to the first embodiment. The information processing device 100 includes a CPU 101, a memory 102, a storage device 103, an input device 104, and an output device 105, which are connected to each other via a bus 106.

[0014] The CPU 101 executes various functions, which will be described later, by executing a determination program 107 for map elements to be deleted, which is stored in the storage device 103. The memory 102 temporarily stores programs and data that the CPU 101 reads from the storage device 103. The memory 102 is also used as an area for the CPU 101 to execute various programs. The storage device 103 stores an operating system (OS), map elements for traveling along a route, and map elements obtained during traveling.

[0015] The input device 104 is a functional unit that accepts input from a user. Specifically, a keyboard and a mouse are used. The output device 105 outputs information input by the input device 104 and the execution results of a program executed by the CPU 101. The communication interface 107 communicates information via a network, and the communication interface may be Ethernet, or any type of interface such as USB, serial communication, or wireless communication.

[0016] 2 is a functional block diagram of the information processing device 100 according to the first embodiment. The information processing device 100 includes an acquisition unit 201, a decision unit 202, and a move / delete unit 203.

[0017] The acquisition unit 201 acquires and outputs, as environmental condition information, images captured by a camera from a position and orientation measurement unit 301 of a mobile system 300 described later, three-dimensional position information of feature points detected from the images, and map elements linked to the shooting time and the camera position and orientation information.

[0018] The determining unit 202 determines map elements to be deleted or moved from among the map elements input from the acquiring unit 201, and outputs the determined map elements.

[0019] The move / delete unit 203 moves or deletes the map element input by the determination unit 202 to a specified location.

[0020] 3 is a functional block diagram of a mobile body system 300 and a map creation system 310 according to the first embodiment. Information processing of the mobile body system 300 and the map creation system 310 is executed by the same hardware configuration as that of the information processing device 100. The mobile body system 300 includes a position and orientation measurement unit 301, a mobile body control unit 302, and a mobile body system communication unit 303. The map creation system 310 includes a map generation unit 311, a map storage unit 312, an environment measurement unit 313, and a map creation system communication unit 314. The mobile body system 300, the map creation system 310, and the information processing device 100 can all be mounted on a mobile body. The information processing device 100 is located on a server outside the mobile body, and the mobile body can be controlled by communication between the information processing device 100 and the mobile body via a network.

[0021] Examples of the mobile object include an autonomous mobile robot (AMR), an automatic guided vehicle (AGV), an autonomous vehicle, a cleaning robot, a delivery robot, a drone, etc. The mobile object is equipped with a camera as an imaging device.

[0022] The position and orientation measurement unit 301 measures the position and orientation of the mobile system 300 from the surrounding environment, and outputs the results as map elements. Although the measurement method is not shown, it uses an image captured by a camera mounted on the mobile system 300, and calculates using the distance from the camera to an object and position information of image feature points. As long as the features of the image feature points are features obtained from the gradient value of brightness, various types of features such as SIFT, SURF, and FAST can be used.

[0023] The mobile body control unit 302 controls the driving of the mobile body system 300 based on the results of the position and orientation measurement unit 301 and information and instructions received from a mobile body system communication unit 303 described later, and outputs the results of the position and orientation measurement unit 301 to the mobile body system communication unit 303.

[0024] A mobile system communication unit 303 outputs the results of the position and orientation measurement unit 301 to a map creation system 310 which is an external module, and receives information from the map creation system 310 .

[0025] The map generation unit 311 generates three-dimensional map information (hereinafter, "map" will be explained as "three-dimensional map") from the position and orientation measurement results measured by the mobile system 300 received from the map creation system communication unit 314 described later. The three-dimensional map information is composed of multiple map elements.

[0026] The map storage unit 312 stores the map generated by the map generation unit 311 and information received from an environmental measurement unit (described later), and outputs the stored information to a map creation system communication unit 314 .

[0027] The environment measurement unit 313 uses a sensor to output the information, specifically, the image and time when the map element was created.

[0028] The map creation system communication unit 314 is connected to the mobile system communication unit 303 of the mobile system 300 and the acquisition unit 201 of the information processing device 100, and inputs and outputs information related to maps and map elements.

[0029] 4 and 5 are operation flowcharts of the information processing device 100 according to the first embodiment. Hereinafter, it is assumed that the flowcharts are realized by the CPU executing a control program. The operation of the information processing device is started when the storage capacity of the map storage unit 312 reaches a predetermined capacity, when a predetermined time has elapsed since the moving object started to move, etc.

[0030] In step S401, a group of map elements is obtained from the map creation system 310.

[0031] In step S402, a map element to be deleted is determined from the map elements acquired in step S401.

[0032] In step S403, the map element determined in step S402 is deleted.

[0033] Here, the determining step S402 will be described in detail.

[0034] In step S501, an unselected map element is selected from the map element group acquired in step S401.

[0035] In step S502, among the map elements acquired in step S401, map elements other than the map element selected in S501 are targeted to extract map elements with camera positions and orientations similar to the camera position and orientation associated with the map element selected in S501. Map elements with similar camera positions and orientations (differences within a predetermined range) are extracted. Then, the map element with the most recent shooting time is extracted.

[0036] In step S503, a common area captured in a common field of view between the map element selected in step S501 and the map element extracted in step S502 is obtained using the features of the image feature points associated with each map element.

[0037] In step S504, two image areas, the common area found in S503 of the image linked to the map element selected in S501, and the common area found in S503 of the image linked to the map element extracted in S502, are divided into grid areas with the same number of divisions.

[0038] In step S505, the average brightness value is calculated within each lattice region of the two image regions, and the number of lattice regions or the ratio to the image of lattice regions where the absolute value of the difference in the average brightness value for each lattice region is equal to or greater than a predetermined threshold value P is calculated.

[0039] In step S506, it is determined whether the number of regions found in step S505 is equal to or greater than a predetermined threshold value R, and if so, the process proceeds to step S507, otherwise the process ends. If the number of regions found in step S505 is less than the predetermined threshold value R, the image is important for improving the accuracy of the position and orientation in the future, and is likely to be used in the future, so it is not subject to deletion.

[0040] In step S507, it is determined that the map element selected in step S501 is to be deleted.

[0041] In step S508, it is determined whether or not to continue the process of determining the deletion target. If there are unselected map elements remaining from the map element group acquired in step S401, the process returns to step S501 and continues. If not, the process proceeds to step S403 and the deletion process is performed.

[0042] FIG. 6 is a diagram showing a schematic representation of map elements to which environmental condition information and position and orientation information are linked. 601 and 602 are examples of images linked to map elements selected / extracted in steps S501 and S502, respectively. In the latest captured image 602, an object 603 has been added since the image 601 was captured. Area 604 is the common area found in step S503. On the other hand, area 605 is not a common area and cannot be compared. Within the common area 604, locations where object placement has changed are calculated. The image from step S504 is divided as shown in 606, and locations where object placement has changed in step S505 are marked as partial area 607. The partial area 607 where object placement has changed is an area where the features are not similar between images 601 and 602. In this case, if the condition for deletion is that a grid area of ​​20% or more of the image area has changed, then the partial area 607 has changed by 6 / 28 = approximately 21%, and the captured image 602 is determined to be a map element to be deleted.

[0043] As described above, according to the information processing device 100 according to the first embodiment, it is possible to reduce the storage capacity by deleting map elements based on the degree of change in the object arrangement around the sensor.

[0044] <Variation 1-1> In the first embodiment, map elements whose object location has changed are to be deleted, but map elements to be deleted may be determined based on the usage history of the map elements. When a moving body moves autonomously, it estimates its own position and attitude by referring to map elements that have been created in advance or map elements that have been created during previous travel. For this reason, the use time and number of times of use of the map elements used in the estimation may be stored, and map elements that have been used a long time ago or have been used infrequently may be determined to be deleted.

[0045] <Variation 1-2> Map elements may be assigned a degree of importance, and map elements with a high degree of importance may be excluded from the determination of map elements to be deleted as shown in the first embodiment and the modified examples. This degree of importance is set based on a user's instruction or the history of map usage. If the environment in which a moving object moves is measured before executing the process described in this embodiment and a three-dimensional map of the environment is created in advance, the importance of the map elements of the three-dimensional map can be set high and excluded from the determination of map elements to be deleted.

[0047] <Variation 1-4> In the first embodiment and the modified example, the map element to be deleted is determined from the difference in the numerical value obtained from the image that is the environmental condition information, but the map element to be deleted may be determined from the contents of the image. Specifically, an object present in the image is detected by machine learning, image recognition, or the like. It is determined whether or not the detected object is likely to move. If a moving object is present in the image and its proportion in the image is small, the object area is treated as an area 408 that cannot be compared, and it is determined whether or not there is a change in the object arrangement in other parts. Also, if the moving object occupies a large proportion of the image, it is determined as a map element to be deleted. In addition to the size of the object, the range, speed, and frequency of the movement may be taken into consideration to determine the map element to be deleted. For example, in the case of a scene where the object passes by quickly, it may not be present the next time it is photographed, so it can be determined as a deletion target.

[0048] <Variation 1-5> Instead of recognizing objects that may move as in Modification 1-4, an object may be recognized as having actually moved and determined to be a deletion target. Also, a threshold may be used to determine the amount of movement or the number of objects that have moved to select deletion targets.

[0049] Second embodiment In the first embodiment, an example of determining a map element to be deleted based on the degree of change in the object arrangement around the sensor was shown. In addition, when the brightness around the sensor is changed, the information obtained by the sensor is different even in the same arrangement, so that different map elements before and after the brightness change are stored, which may increase the storage capacity. Therefore, in the second embodiment, an example of determining a map element to be deleted based on the degree of change in brightness around the sensor will be described. Specifically, the brightness changes gradually with time, and map elements with a large difference in brightness that are unlikely to be used are deleted. In addition, when a temporary brightness change occurs, the map element is determined to be the map element to be deleted. In this embodiment, the sensor is a camera that outputs a two-dimensional array of brightness value data.

[0050] The hardware configuration of the information processing device according to the second embodiment is the same as that in FIG. 1, and therefore a description thereof will be omitted.

[0051] The functional block diagram of the information processing device 100 according to the second embodiment is the same as that of Fig. 2, and the acquisition unit 201 also acquires an image as environmental condition information and a map element to which position and orientation information is linked. The mobile system 300 and the map creation system 310 according to the second embodiment are also the same as those of Fig. 3, but input and output information on brightness at the time when the map element was created as information measured by the environment measurement unit 313.

[0052] Fig. 7 is a flowchart showing the operation of step S402 of the information processing device 100 according to the second embodiment. The process in Fig. 4 is the same as that in the first embodiment.

[0053] Steps S701 to S703 and steps S706 to S707 are the same as steps S501 to S503 and steps S507 to S508 in FIG. 5, so the description thereof will be omitted.

[0054] In step S704, the average pixel value of each of the two image regions determined in step S703 is determined.

[0055] In step S705, the average brightness value is calculated for each image of the two image regions, and it is determined whether the absolute value of the brightness difference is equal to or greater than a predetermined threshold value P.

[0056] As described above, according to the information processing device 100 according to the second embodiment, it is possible to reduce the storage capacity by deleting map elements based on the degree of variation in brightness around the sensor.

[0057] <Variation 2-1> In the first embodiment, an example was described where the brightness fluctuations were gradual. However, in the case of temporary brightness fluctuations, the fluctuation in the average brightness value of the image in the time direction may be obtained, and map elements at points in time with high kurtosis may be determined to be targets for deletion.

[0058] <Variation 2-2> In the first embodiment, an example was described in which brightness information obtained from an image was used, but the comparison may also be performed using brightness information obtained from a sensor other than a camera. Specifically, a luminance meter is provided that is fixed in a sensor while maintaining a predetermined relative position and orientation relationship, and illuminance measured in synchronization with the sensor measurement is stored as environmental condition information. The illuminance is compared with the illuminance at the time of the map element to be compared, and if the absolute value of the illuminance difference is equal to or greater than a predetermined threshold, the map element may be determined to be deleted.

[0059] <Modification 2-3> In the first embodiment and the modified example, the determination is made using direct brightness information obtained from an image or a sensor, but the determination may be made from other brightness information. Specifically, weather forecasts and weather result information provided on the Internet are received and used. This weather forecast and result information are linked to the map element creation position and stored. If the map element to be compared is a current one, the change in brightness may be determined based on the change in weather using weather forecast information, or if the map element to be compared is a past map element, the weather, cloud cover, sunrise, and sunset information from the weather result information.

[0060] Third embodiment In the first and second embodiments, the information processing device 100 determines the map elements to be deleted, but in the third embodiment, an example will be described in which the map elements to be deleted are selected in the first and second embodiments, and the information is presented to the user to delete the map elements. Here, the method of the first embodiment will be used for explanation, but there is no problem if the method of the second embodiment is used.

[0061] The hardware configuration of the information processing device according to the third embodiment is the same as that in FIG. 1, and therefore a description thereof will be omitted.

[0062] Fig. 8 is a functional block diagram of an information processing device 100 according to the third embodiment. In addition to Fig. 2, a display unit 204 is added. The display unit 204 displays the information selected by the determination unit 202, and receives instructions from the user.

[0063] Fig. 9 is a flowchart of the information processing device 100 according to the third embodiment. Steps S901 to S907 are the same as steps S501 to S507 in Fig. 5, and therefore a description thereof will be omitted.

[0064] In step S908, information on the map element to be deleted determined in step S907 is output to the output device 105.

[0065] In step S909, an input from the user is awaited, and if a deletion instruction is received from the user, the process proceeds to the next step S910, where the target map element is deleted.

[0066] In step S911, it is determined whether or not to continue the selection process. If the selection process is to be continued, the process returns to step S901 and is repeated. If the selection process is not to be continued, the process ends.

[0067] Figure 10 shows the output in step S908 of the flowchart in Figure 9. Information 1002 of the map element to be deleted is displayed in a display area 1001, along with information 1003 of map elements that are frequently used while driving and that are photographed in a similar location and with a similar angle of view compared to the map element to be deleted. The user selects the map element to be deleted from this information, and deletes the selected map element in step S910.

[0068] As described above, according to the information processing device 100 according to the third embodiment, the user can select map elements to be deleted, and the selected map elements can be deleted, thereby reducing the storage capacity.

[0069] (Other embodiments) In the above embodiment, an example of deleting a selected map element has been described, but the map element may be moved to a predetermined location instead of being deleted. Specifically, the storage capacity of the map storage unit 312 can be reduced by moving the map element from the memory 102 in the information processing device 100 to the storage device 103, or by moving the map element from the storage device 103 in the information processing device 100 to a storage device of an external system.

[0070] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable recording medium.

Claims

1. an acquisition means for acquiring three-dimensional map information from a storage means for storing three-dimensional map information composed of a plurality of map elements, the three-dimensional map information being defined as map elements including a position and orientation of an imaging device mounted on a moving body that is estimated by capturing an image of an environment with the imaging device, three-dimensional position information in the environment of feature points included in the image obtained by capturing the image of the environment, and the image; and a determination means for determining that one of the map elements is to be deleted from the storage means when, in two map elements in which the position and orientation of the imaging device contained in the map element are similar, the size of an area in the images in which the common area photographed in a common field of view has dissimilar features is equal to or greater than a threshold value.

2. The information processing device according to claim 1, characterized in that the determination means determines a first map element of first three-dimensional map information that has been created in advance for the environment and a second map element in second three-dimensional map information that is created in conjunction with the movement of the moving body and has a position and orientation similar to that included in the first map element, when a size of an area that is not similar between images included in the first map element and the second map element is equal to or larger than a threshold value.

3. 3. The information processing apparatus according to claim 1, wherein the region having dissimilar features between the images is a partial region of the images.

4. 3. The information processing device according to claim 1, wherein the determining means determines one of the map elements to be deleted from the storage means when a size of an area in which the luminance characteristics between the images are not similar is equal to or larger than a threshold value.

5. 5. The information processing apparatus according to claim 1, wherein the determining unit determines the map element to be deleted from among two map elements in which a difference in position and orientation of the imaging device included in the map element is within a predetermined range.

6. 6. The information processing apparatus according to claim 1, wherein a difference in average luminance values ​​of pixels in an area having dissimilar features between the images is equal to or greater than a threshold value.

7. The information processing device according to claim 1 or 2, characterized in that the determination means determines one of the map elements to be deleted from the storage means when the size of an area in which the features obtained from the brightness gradient values ​​between the images are not similar is equal to or greater than a threshold value.

8. an acquisition step of acquiring three-dimensional map information composed of a plurality of map elements, the map elements being the position and orientation of an imaging device mounted on a moving body estimated by capturing an image of an environment with the imaging device, three-dimensional position information in the environment of feature points included in the image obtained by capturing the image of the environment, and the image; and a determining step of determining, for two map elements having similar positions and orientations of the imaging devices contained in the map elements, that one of the map elements is to be deleted when the size of an area having dissimilar features in a common area captured in a common field of view in the images is equal to or larger than a threshold.

9. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 7.

Images