Information processing device, mobile device, computer program, and information processing method.

Abstract

To provide an information processor capable of reducing memory usage for holding a map element.SOLUTION: In a mobile robot 200 as a moving body, an information processor 100 for estimating a position and attitude of the moving body includes: a first sensor output information acquisition section 110 that acquires first sensor output information from a first sensor 170; a second sensor output information acquisition section 120 that acquires second sensor output information from a second sensor 180; a position attitude information acquisition section 130 that acquires position attitude information on the moving body from a position attitude estimation section 190 for estimating the position and attitude of the moving body using the first sensor output information and the second sensor output information; a map element creation section 140 that creates a map element including the first sensor output information, the second sensor output information, and the position attitude information; a redundant data determination section 150 that determines redundant data among data held by the map element; and a redundant data erasure means 160 that erases the data determined to be redundant from the map element.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a moving body, a computer program, an information processing method, and the like.

Background Art

[0002] Pose measurement processing using information obtained from various sensors such as a photographing device and an IMU (Inertial Measurement Unit) sensor is used for self-position estimation for autonomous driving of a moving body such as a robot, and is also used for various other purposes such as three-dimensional modeling of an object or space. For pose estimation using both a photographing device and an IMU sensor, a three-dimensional map of the environment using acceleration and angular velocity obtained from the IMU sensor, image features detected from image information obtained from the photographing device, and three-dimensional position information of the image features calculated therefrom as map elements is used.

[0003] During pose estimation, when a new map element not held in the three-dimensional map is observed, the map element is added to the map and referred to during pose estimation. Since the IMU sensor acquires information more frequently than the photographing device, the frequency of adding and adjusting map elements also increases. Therefore, there is a concern that the memory usage amount of the map elements held by the map becomes extremely large compared to the case of using only the photographing device.

[0004] In Patent Document 1, a method for deleting redundant data possessed by a map when using a photographing device has been proposed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the method described in Patent Document 1 has the problem that it is time-consuming to process, making it difficult to efficiently remove redundant data from map elements that combine different sensors.

[0007] This invention has been made in view of the above problems, and one of its objectives is to provide an information processing device that can reduce the amount of memory used to hold map elements. [Means for solving the problem]

[0008] To solve the above problems, one aspect of the present invention is: In an information processing device that estimates the position and orientation of a moving object, A means for acquiring first sensor output information from a first sensor, A means for acquiring second sensor output information from a second sensor, A position and attitude information acquisition means that uses the first sensor output information and the second sensor output information to estimate the position and attitude of the moving body and acquire position and attitude information, A map element creation means for creating a map element that includes the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination means for determining redundant data among the data contained in the map element, A redundant data deletion means that deletes data that the redundant data determination means has determined to be redundant from the map element, to have death, The redundant data determination means determines that some of the map elements are redundant based on the accuracy of the position and orientation estimation and the timing of creating the map elements. It is characterized by doing so. [Effects of the Invention]

[0009] The present invention enables the realization of an information processing device that can reduce the amount of memory used to hold map elements. [Brief explanation of the drawing]

[0010] [Figure 1]This is a functional block diagram of a mobile robot 200 including an information processing device 100 according to Example 1. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device 100 according to Example 1. [Figure 3] This is a flowchart illustrating the operation of the information processing device 100 according to Example 1. [Figure 4] This is a functional block diagram of a mobile robot 200 including an information processing device 100 according to Example 2. [Figure 5] This is a flowchart illustrating the operation of the information processing device 100 according to Example 2. [Figure 6] (A) and (B) are flowcharts illustrating the operation of the information processing device 100 according to Example 5. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0012] This embodiment describes an information processing device that mounts an imaging device and an IMU sensor on a mobile robot, which is a moving object that travels along a predetermined path, and performs position and orientation estimation of the mobile robot using SLAM. SLAM stands for Simultaneous Localization and Mapping. For position and orientation estimation using both an imaging device and an IMU sensor, the method described in the following literature, for example, is employed.

[0013] C. Campos, R. Elvira, J. J. G. Rodriguez, J. M. M. Montiel and J. D. Tardos, “ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM,” in IEEE Transactions on Robotics, doi:10.1109 / TRO.2021.3075644.

[0014] The imaging device acquires images less frequently than the IMU sensor. That is, the output information obtained from the imaging device is updated at a predetermined interval less frequently than the IMU sensor. The IMU sensor acquires acceleration and angular velocity measurement values more frequently, that is, at an interval shorter than the predetermined interval, and the position and orientation estimation is performed in accordance with the frequency at which the captured images are acquired.

[0015] At this time, the IMU sensor information integrates the data acquired during the predetermined interval of the captured image, calculates the speed and rotation amount of the moving body, and uses them. This integration process integrates the acceleration and angular velocity acquired by the IMU sensor from after the imaging device captures an image until the next capture, and is used in accordance with the position and orientation estimation when using the latter captured image among the two captured images. Note that in the embodiment, imaging and photographing are used in the same meaning.

[0016] The results obtained from the position and orientation estimation include, in addition to the position and orientation of the moving body, three-dimensional feature point information of the space observed by the imaging device, bias values of the acceleration and angular velocity observed by the IMU sensor, and the like. The values of the acceleration and angular velocity observed by the IMU sensor are values obtained by adding noise and bias values to the true values. Therefore, in order to use the correct acceleration and angular velocity for the position and orientation estimation, it is necessary to correctly estimate the bias value, and the bias value is also estimated simultaneously during the position and orientation estimation.

[0017] The results of position and orientation estimation, the captured images observed at that location, and the measured values ​​of acceleration and angular velocity are recorded, and a three-dimensional map (hereinafter simply referred to as "the map") is created so that it can be referenced as prior information when passing through the same location later. The map consists of map elements that hold the position and orientation estimation results at each point and the data acquired by the imaging device and IMU sensor.

[0018] After map creation, when passing through the same location, the position and orientation estimation process uses not only images, acceleration, and angular velocity acquired by the imaging device and IMU sensor mounted on the moving object, but also data held by map elements that have position and orientation information close to that point. This enables more accurate position and orientation estimation.

[0019] The acceleration and angular velocity measurements stored in a map element are not limited to those taken at the point where the map element is created, but also include all acceleration and angular velocity measurements taken since the creation of the previous map element. Furthermore, the integral values ​​of acceleration and angular velocity are calculated using the acceleration and angular velocity measurements from the time of the previous map element.

[0020] In the case of an information processing device that uses only an imaging device, the map elements include 3D feature point information of the space observed by the imaging device. If 3D feature point information that is not held by any of the map elements held by the map is observed, a map element that holds that information is created. Therefore, when using only an imaging device, a map element is created every time a certain distance of about 50 cm is moved. When an IMU sensor is used in combination, it is necessary to reflect the constantly changing acceleration and angular velocity measurements into the position and orientation information held by the map elements.

[0021] Therefore, when using an IMU sensor, a map element is created every so often, for example, approximately every 5 seconds. Thus, unless the moving object is moving at a very high speed, using an IMU sensor results in a higher frequency of map element creation. A higher frequency of map element creation also increases the amount of memory used to store the map elements (amount of stored data). The device of this embodiment reduces the memory usage of a map by identifying and deleting redundant data from the data within the map elements of the map.

[0022] [Example 1] Figure 1 is a functional block diagram of a mobile robot 200 including an information processing device 100 according to Embodiment 1. The mobile robot 200 shown in Figure 1 has a drive device (not shown) for movement and a mobile robot control unit for controlling the overall operation of the mobile robot 200. The mobile robot control unit includes a CPU and memory as a computer and drives the mobile robot using the drive device based on map elements, etc., acquired by the information processing device 100.

[0023] Furthermore, some of the functional blocks shown in Figure 1 are implemented by having the CPU 211, which acts as a computer included in the information processing device 100, execute computer programs stored in memory (ROM 212, external memory 214, etc.) as a storage medium. However, some or all of these may be implemented in hardware.

[0024] In terms of hardware, dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs) can be used. Furthermore, the functional blocks shown in Figure 1 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. Furthermore, the above explanation regarding Figure 1 also applies to Figure 4.

[0025] The information processing device 100 according to Embodiment 1 includes a first sensor output information acquisition unit 110, a second sensor output information acquisition unit 120, a position and orientation information acquisition unit 130, a map element creation unit 140, a redundant data determination unit 150, and a redundant data deletion unit 160. The first sensor output information acquisition unit 110 acquires the information captured and output by the imaging device, which acts as the first sensor 170, i.e., the captured image (imaging image), as first sensor output information. The imaging device includes a CMOS image sensor, etc.

[0026] The second sensor output information acquisition unit 120 acquires the information output by the IMU sensor, which is the second sensor 180, namely acceleration and angular velocity, and the integral value of acceleration and angular velocity calculated for use in position and attitude estimation, as second sensor output information. The position and orientation estimation unit 190 estimates the position and orientation of the moving object using the captured image output by the first sensor 170 and the acceleration and angular velocity output by the IMU sensor, which is the second sensor 180. This position and orientation information includes not only the position and orientation of the moving object, but also three-dimensional position information of feature points observed by the imaging device, and estimated bias values ​​for acceleration and angular velocity.

[0027] The position and orientation information acquisition unit 130 acquires the position and orientation information of the moving object that has been previously estimated by the position and orientation estimation unit 190. The map element creation unit 140 creates map elements to be held by the map. The map elements include the captured image acquired by the first sensor output information acquisition unit 110, the measured values ​​of acceleration and angular velocity acquired by the second sensor output information acquisition unit 120, the integral values ​​of acceleration and angular velocity, and the position and orientation information of the moving object acquired by the position and orientation information acquisition unit 130. In other words, the map elements include the first sensor output information, the second sensor output information, and the position and orientation information.

[0028] The map elements created here are used by the position and orientation estimation unit 190 to estimate the position and orientation of the moving object. The redundant data determination unit 150 determines which data in a map element is redundant. The redundant data deletion unit 160 deletes the data that the redundant data determination unit 150 has determined to be redundant from the map element, while leaving the data that was not determined to be redundant.

[0029] Figure 2 is a block diagram showing the hardware configuration of the information processing device 100 according to Embodiment 1. 211 is the CPU, which functions as a control means that controls the operation of each part of the device connected to the system bus 221 based on computer programs stored in memory (ROM 212, etc.) as a storage medium. 212 is ROM, which stores the BIOS program, boot program, and other computer programs. 213 is RAM, which is used as the main memory for the CPU 211.

[0030] 214 is an external memory such as an HDD or SSD, and functions as memory for storing map elements. Furthermore, the external memory 214 may also store computer programs processed by the information processing device 100. The input unit 215 processes information input such as from a keyboard or mouse. The display unit 216 outputs the calculation results of the information processing device 1 to a display device (not shown) according to instructions from 211. The display device can be any liquid crystal display, projector, LED indicator, etc.

[0031] 217 is a communication interface that communicates information over a network. The communication interface can be Ethernet, USB, serial communication, wireless communication, etc., and is not limited to any particular type. The communication interface 217 communicates with the aforementioned mobile robot control unit, such as exchanging calculated position and orientation information, and also communicates with an external server to store various data on the external server. 218 is an I / O unit that receives images and acceleration / angular velocity inputs from the first sensor 170 and the second sensor 180, respectively.

[0032] Figure 3 is a flowchart illustrating the operation of the information processing device 100 according to Embodiment 1. The flowchart in Figure 3 is realized by the CPU 211 executing a computer program stored in the memory (ROM 212, etc.) as a storage medium. The same applies to the flowcharts in Figures 5 and 6. The information processing device 100 starts operating when it becomes necessary to create map elements. Map elements are created at the frequency necessary to reflect the information acquired by the IMU sensor in position and attitude estimation. In this embodiment, for example, map elements are created at 5-second intervals.

[0033] Steps S300 to S302 are steps in which the map element creation unit 140 acquires the information necessary to create map elements in step S303. The order of these processes is not limited to the order shown in Figure 3. Regardless of the order of steps S300 to S302, each step is performed once when creating map elements. In step S300, the first sensor output information acquisition unit 110 acquires the first sensor output information, which is a captured image taken at a point where a map element is to be created.

[0034] In step S301, the second sensor output information acquisition unit 120 acquires the second sensor output information, which is the measured values ​​of acceleration and angular velocity, and the integrated values ​​of acceleration and angular velocity calculated for use in position and orientation estimation. The measured values ​​of acceleration and angular velocity to be acquired are the acceleration and angular velocity measured from the point where the map element was created in the previous instance to the point where the map element is being created in the current instance. The integrated values ​​of acceleration and angular velocity to be acquired are obtained by integrating the acceleration and angular velocity measured from the point where the map element was created in the previous instance to the point where the map element is being created in the current instance.

[0035] In step S302, the position and orientation estimation unit 190 uses the first sensor output information, which is the image captured by the imaging device, and the second sensor output information, which is the measured values ​​of acceleration and angular velocity, and the integrated values ​​of acceleration and angular velocity, to estimate the position and orientation information of the moving object. The position and orientation information of the moving object includes the position and orientation of the moving object, the three-dimensional position information of feature points observed by the imaging device, and the estimated bias values ​​of acceleration and angular velocity. The position and orientation information acquisition unit 130 then acquires the position and orientation information of the moving object estimated by the position and orientation estimation unit 190.

[0036] In step S303, the map element creation unit 140 creates map elements to be held by the map. The map elements include the captured image acquired by the first sensor output information acquisition unit 110 in step S300, and the measured values ​​of acceleration and angular velocity and the integrated values ​​of acceleration and angular velocity acquired by the second sensor output information acquisition unit 120 in step S301. The map elements also include the position and attitude information of the moving object estimated by the position and attitude estimation unit 190 acquired by the position and attitude information acquisition unit 130 in step S302.

[0037] The map elements created here are used by the position and orientation estimation unit 190 to estimate the position and orientation of the moving object. When using map element data for position and orientation estimation for the first time, all map element data is used for position and orientation estimation. When using map element data for position and orientation estimation for the next time, the measured values ​​of acceleration and angular velocity, the integrated values ​​of acceleration and angular velocity, and the estimated bias values ​​of acceleration and angular velocity are used for position and orientation estimation.

[0038] In step S304, the redundant data determination unit 150 determines that there is redundant data. In this embodiment, it determines that there is redundant data among the data contained in each map element. The data contained in the map element is used by the position and orientation estimation unit 190 for position and orientation estimation. However, in this embodiment, for map elements that have been used once for position and orientation estimation, only the measured values ​​of acceleration and angular velocity, the integrated values ​​of acceleration and angular velocity, and the estimated bias values ​​of acceleration and angular velocity are used for subsequent position and orientation estimation.

[0039] Therefore, in this embodiment, among the data held by a map element used once for position and orientation estimation, data other than measured values ​​of acceleration and angular velocity, integral values ​​of acceleration and angular velocity, and estimated bias values ​​of acceleration and angular velocity, such as captured images and 3D feature point information of space, are determined to be redundant. In other words, in step S304, among the data held by the map element, data that is not used for subsequent position and orientation estimation is determined to be redundant data.

[0040] In step S305, the redundant data deletion unit 160 deletes the data that the redundant data determination unit 150 determined to be redundant in step S301 from the map element. In this embodiment, as described above, the captured image acquired from the first sensor 170, which is an imaging device, and the three-dimensional feature point information of the space are deleted. The redundant data deletion unit 160 deletes the redundant data immediately after the redundant data determination unit 150 determines that the data is redundant.

[0041] Map elements from which redundant data has been removed are used by the position and attitude estimation unit 190 as prior information for position and attitude estimation. At this time, the position and attitude estimation unit 190 estimates the position and attitude of the moving object using the measured values ​​of acceleration and angular velocity output by the IMU sensor, which is the second sensor 180, and the integral value of acceleration and angular velocity. Furthermore, the position and attitude of the moving object is also estimated using the measured values ​​of acceleration and angular velocity held by the map elements, the integral value of acceleration and angular velocity, and the estimated bias value of acceleration and angular velocity.

[0042] In Example 1, the first and second sensors used were an imaging device and an IMU sensor, but other sensors such as Lidar (Light Detection and Ranging) and GPS (Global Positioning System) may also be used. Sensors that can be used together to estimate position and attitude can be combined. In this case, the first sensor output information acquisition unit 110 and the second sensor output information acquisition unit 120 acquire the same data from the information output from each sensor that the position and attitude estimation unit 190 uses for position and attitude estimation.

[0043] The map element includes information acquired by the first sensor output information acquisition unit 110 and the second sensor output information acquisition unit 120. The redundant data determination unit determines that data other than the data used for position and orientation estimation is redundant. In other words, it determines that data not used for position and orientation estimation is redundant. In Example 1, the data that the redundant data determination unit 150 determines to be redundant is limited to the measured values ​​of acceleration and angular velocity, the integral values ​​of acceleration and angular velocity, and the estimated bias values ​​of acceleration and angular velocity, but it is not limited to this.

[0044] For example, when using captured images and 3D feature point information in space for position and orientation estimation, any other data not used for position and orientation estimation may be considered redundant data.

[0045] The timing of redundancy data deletion by the redundancy data deletion unit 160 is set to immediately after the redundancy data determination unit 150 determines that the data is redundant, but this is not limited to this. The redundancy data determination unit 150 may delete the data after a certain amount of time has elapsed since determining that the data is redundant, or after moving a certain distance. Alternatively, the data may be deleted when the memory usage of the RAM 213 or external memory 214 used by the map exceeds a predetermined memory usage amount. Or, the user may specify the timing of deletion.

[0046] Furthermore, in Embodiment 1, after the redundant data deletion unit 160 deletes redundant data, the position and attitude estimation unit 190 uses the following data for position and attitude estimation processing. Specifically, it uses the captured image output by the first sensor 170, the measured values ​​of acceleration and angular velocity output by the second sensor 180, the integral values ​​of acceleration and angular velocity, the measured values ​​of acceleration and angular velocity held by the map elements, the integral values ​​of acceleration and angular velocity, and the estimated bias values ​​of acceleration and angular velocity. However, the data used for position and attitude estimation processing is not limited to these.

[0047] After removing redundant data, only the acceleration and angular velocity measurements output by the second sensor 180, the integral values ​​of acceleration and angular velocity, the acceleration and angular velocity measurements held by the map elements, the integral values ​​of acceleration and angular velocity, and the estimated bias values ​​of acceleration and angular velocity may be used. In Example 1, we described a method for deleting redundant data while creating a map. However, this example is not limited to this; it can also be applied when a map is created in advance, saved to a file, and then loaded and used when performing position and orientation estimation.

[0048] [Example 2] In Example 1, data held by map elements that are not used for position and orientation estimation were determined to be redundant, and this data was removed from the map elements to reduce the memory usage of the map. Example 2 describes an example of determining that data not used for adjusting map elements is redundant.

[0049] The positional orientation information of map elements in a map is obtained as a result of estimating the positional orientation of a moving object, and therefore discrepancies occur due to the effects of measurement errors, etc. To resolve these discrepancies, the data of each map element is adjusted. In this embodiment, when this adjustment of map elements is performed, data other than the data used to adjust the map elements, i.e., data not used to adjust the map elements, is determined to be redundant.

[0050] Figure 4 is a functional block diagram of a mobile robot 200 including an information processing device 100 according to Embodiment 2. In Figure 4, the same reference numerals are used for the same components or elements as in Figure 1, and redundant explanations are omitted or simplified.

[0051] The map element adjustment unit 400 is a functional block that adjusts map elements. It performs adjustments on multiple map elements, correcting not only the position and orientation information held by each map element, but also the 3D feature point information of the space and the estimated bias values ​​of acceleration and angular velocity. Map element adjustment is a so-called bundle adjustment (a process to reduce reprojection errors). Adjustments are performed only once for the same combination of map elements. When the map element creation unit 140 creates a new map element, the adjustment is performed again in combination with the existing map element.

[0052] Bundle adjustment is a process that corrects the camera position and orientation and the three-dimensional position of image features so as to minimize the difference between the position of the projection point (where the three-dimensional position of image features is projected onto map elements (keyframes)) and the position of the image feature points detected from the image. Local bundle adjustment is the process of selecting a predetermined number of map elements (keyframes) and performing map correction.

[0053] Details regarding keyframes, bundle adjustments, and local bundle adjustments can be found in the method by Raul et al. (Raul Mur-Artal et.al, ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics).

[0054] Figure 5 is a flowchart illustrating the operation of the information processing device 100 according to Example 2. Steps S500 to S503 are the same as steps S300 to S303 shown in Figure 3, which were described in Example 1, so their explanation will be omitted.

[0055] In step S504, the map element adjustment unit 400 performs adjustments to the map elements. For map elements undergoing adjustment for the first time, the system adjusts the position and orientation, spatial 3D feature point information, and deviations in the acceleration and angular velocity bias values ​​contained within that map element. Subsequently, for map elements that have already undergone adjustment, only the position and orientation and the estimated bias values ​​of acceleration and angular velocity are adjusted. The timing of the map element adjustment unit 400 is as shown in the flowchart of Figure 5, immediately after the map element creation unit 140 creates a new map element in step S503.

[0056] In step S505, the redundant data determination unit 150 determines that the data is redundant according to the map element adjustment method performed by the map element adjustment unit 400 in step S504. Once a map element has been adjusted, only the estimated bias values ​​for position, orientation, acceleration, and angular velocity are subsequently adjusted. Therefore, no adjustment is performed on the 3D feature point information in space, and the captured image and the 3D feature point information in space are determined to be redundant.

[0057] In step S506, the redundant data deletion unit 160 deletes the data that the redundant data determination unit 150 determined to be redundant in step S505 from the map elements. In this embodiment, the captured image and 3D feature point information of space acquired from the imaging device, which is the first sensor 170, are deleted. Immediately after the map element adjustment unit 400 performs the adjustment of the map elements in step S504, the determination and deletion of redundant data in steps S505 and 506 are performed.

[0058] After step S506, the position and attitude estimation unit 190 estimates the position and attitude using the same data used by the map element adjustment unit 400 for map element adjustment: measured values ​​of position and attitude and acceleration / angular velocity, integrated values ​​of acceleration / angular velocity, and estimated bias values ​​of acceleration / angular velocity. Thus, in Example 2, redundant data is identified by considering the method of adjusting map elements. This makes it possible to correct discrepancies in map element information, create a more accurate map, and reduce memory usage by deleting redundant data.

[0059] In Example 2, regarding the map element adjustment method, adjustments to map elements that have already undergone adjustment are performed using only position, orientation, and bias values ​​for acceleration and angular velocity. However, it is also possible to perform the adjustment using only position, orientation, and 3D feature point information of space. In that case, the measured values ​​of acceleration and angular velocity, and the bias values ​​of acceleration and angular velocity used to adjust the estimated bias values ​​of acceleration and angular velocity, may be deemed redundant.

[0060] Furthermore, although the map element adjustment unit 400 performed adjustments to a map element immediately after creating one map element, it is not limited to this, and the map element creation unit 140 may perform adjustments to a map element after creating new map elements multiple times.

[0061] [Example 3] In Examples 1 and 2, redundant data held by map elements was identified and removed from the map elements. This reduced the memory usage of each map element, and consequently, the memory usage of the map as a whole was reduced. Example 3 describes an example of determining whether some map elements are redundant and deleting some of the map elements that are deemed redundant.

[0062] The reason a moving object uses both the first sensor 170, an imaging device, and the second sensor, an IMU sensor, is to increase the number of environments in which the position and orientation of the moving object can be estimated with high accuracy. On the other hand, there are cases where sufficient position and orientation estimation accuracy can be ensured even with only the imaging device, such as when sufficient feature point information can be observed from the captured image or when shooting is possible under a stable lighting environment. On the other hand, if the errors in the acceleration and angular velocity obtained from the IMU sensor are small, there are cases where sufficient position and orientation estimation accuracy can be ensured even with only the IMU sensor.

[0063] In Example 3, when creating a map, the position and orientation estimation unit 190 estimates the position and orientation using both the first sensor 170, which is an imaging device, and the second sensor 180, which is an IMU sensor, and then creates a map. After redundancy data is deleted, position and orientation estimation is performed using only the imaging device at locations where sufficient position and orientation estimation accuracy can be ensured with only the imaging device; otherwise, position and orientation estimation is performed using both the imaging device and the IMU sensor.

[0064] The timing of map element creation differs depending on whether position and orientation estimation is performed using only an imaging device or in combination with an IMU sensor. When position and orientation estimation is performed using only images captured by the imaging device, map elements are created when there is a change in the 3D feature point information of the space observed from the captured images that exceeds a predetermined threshold. Therefore, map elements are created when the device moves a certain distance or changes orientation. On the other hand, when position and orientation estimation is performed based on acceleration and angular velocity using an IMU sensor in combination with an imaging device, in addition to the case when position and orientation estimation is performed using only captured images, map elements are also created after a certain period of time has elapsed in order to reflect the moment-to-moment changing acceleration and angular velocity in the position and orientation estimation.

[0065] The information processing device in Example 3 has the same configuration as in Figure 1, and the flowchart representing the processing flow in Example 3 is also the same as in Figure 3. The differences in processing content at each step compared to other examples are described below.

[0066] When creating map elements in step 303, the map element creation unit 140 determines the position and orientation estimation accuracy at the points where the map elements are to be created and adds the result of the position and orientation estimation accuracy determination to the map elements. The position and orientation estimation accuracy determination is made by determining whether sufficient position and orientation accuracy can be guaranteed at each point using only the captured images, and is determined by the position and orientation information acquisition unit 130 in step S302 from the 3D feature point information of space acquired from the position and orientation estimation unit 190.

[0067] If the spatial 3D feature point information exceeds a predetermined threshold, it is determined that sufficient positional accuracy can be ensured using only the captured image; otherwise, it is determined that acceleration and angular velocity should be used in combination. In addition, information about the timing of map element creation is added to the map element, indicating whether the map element is created when positional accuracy is estimated using only the captured image or when positional accuracy is estimated using both acceleration and angular velocity.

[0068] In step S304, the redundant data determination unit 150 checks the determination result of the position and orientation estimation accuracy of the map element and the map element creation timing information. The unit determines that the map element is redundant only if the location where the map element was created can be guaranteed to have sufficient position and orientation estimation accuracy using only the captured image, and the timing of map element creation was when position and orientation estimation was performed using both acceleration and angular velocity. In other words, the redundant data determination unit 150 determines that some of the map elements are redundant based on the position and orientation estimation accuracy and the timing of map element creation.

[0069] In step S305, the redundant data deletion unit 160 deletes map elements that the redundant data determination unit 150 has determined to be redundant. After the redundancy data is deleted, the position and orientation estimation unit 190 refers to the determination result of the position and orientation estimation accuracy held by the map elements. Then, at locations where position and orientation estimation accuracy can be guaranteed with only the captured image, position and orientation estimation is performed using only the captured image; otherwise, position and orientation estimation is performed using both acceleration and angular velocity.

[0070] The above describes a method for determining redundant map elements from the environment in which a moving object is moving. In Example 3, the map elements themselves are removed from the map, and the position and orientation estimation method is switched depending on the environment to use only the captured image or a combination of the captured image and acceleration / angular velocity. This not only reduces the memory usage of the map but also reduces the processing load.

[0071] Furthermore, in the above, redundancy was determined only when the location where the map element was created could be adequately estimated using only the captured image, and when the timing of map element creation coincided with the use of both acceleration and angular velocity for position and orientation estimation. However, Example 3 is not limited to this. For example, the result of determining position and orientation accuracy may be created based on whether sufficient position and orientation accuracy can be ensured using only acceleration and angular velocity, and whether captured images are also used.

[0072] In this case, a map element may be deemed redundant only if the location where the map element was created can be adequately positioned using only acceleration and angular velocity, and the timing of the map element creation coincides with the use of captured images for positional estimation. In this case, during positional estimation after the deletion of redundant data, the positional estimation unit 190 refers to the determination result of the positional estimation accuracy held by the map element. Then, at locations where positional estimation accuracy can be adequately ensured using only acceleration and angular velocity, positional estimation is performed using only acceleration and angular velocity; otherwise, positional estimation is performed using captured images in combination.

[0073] [Example 4] In Example 3, a method for determining whether a map element itself is redundant was described. In this example, the redundancy of a map element was determined from the result of the position and orientation estimation accuracy and the map element creation timing information. In this example, an example of determining that a map element is redundant when the position and orientation it holds are redundant is described. Note that the configuration diagram and processing flow in this example are the same as in Figure 1 and Figure 3, respectively. The differences in processing content from other examples are described below.

[0074] As explained in Example 3, when using captured images in combination with acceleration and angular velocity, map elements are created when a certain distance is moved, when the orientation changes, or at regular intervals. Therefore, for example, map elements may be created even if the position and orientation of the moving object do not change. In this example, map elements with similar positions and orientations are deleted as redundant data.

[0075] When the moving object is stationary, the acceleration and angular velocity measurements output by the IMU sensor, which is the second sensor 180, are constant, and the velocity, which is the integral of acceleration, is 0. In step S304, the redundant data determination unit 150 compares the acceleration and angular velocity measurements and velocity of the newly created map element with the acceleration and angular velocity measurements and velocity of the map element created immediately before it. If the changes in acceleration and angular velocity and the velocity values ​​are within a predetermined range, the redundant data determination unit 150 determines that there is no movement of the moving object and determines that the map element of interest is redundant.

[0076] In step S305, the redundant data deletion unit 160 deletes the map elements that the redundant data determination unit 150 has determined to be redundant. At that time, it recalculates the integral value of acceleration and angular velocity held by the map elements that follow the map element to be deleted. The integral value of acceleration and angular velocity is calculated using the acceleration and angular velocity between the map elements.

[0077] Therefore, when a map element is deleted, the integral values ​​of acceleration and angular velocity that should be held by the map elements following the deleted element must be integrated using the measured values ​​of acceleration and angular velocity from the map element preceding the deleted element. Thus, before deleting the map element, the redundant data deletion unit 160 integrates the integral values ​​of acceleration and angular velocity held by the map element to be deleted with the integral values ​​of acceleration and angular velocity held by the subsequent map elements. After that, the map element to be deleted is deleted.

[0078] The above describes the method for determining redundant data in Example 4. However, the method for determining whether there is no change in the position and orientation of the moving object may be determined not only from the measured values ​​of acceleration and angular velocity and the speed, but also from the movement instructions given to the moving object. In that case, map elements may be determined to be redundant when there are no movement instructions given to the moving object, or when the drive device of the moving object is not driving the moving object.

[0079] Furthermore, the method for determining whether the positional orientation information held by a map element is redundant is not limited to whether or not the moving object is moving. For example, if the position and orientation of the map element of interest are within a predetermined range relative to the positions and orientations of other map elements held by the map, the redundant data determination unit 150 determines that the map element of interest is redundant. Alternatively, the determination may be made based on the similarity of the three-dimensional feature point information of the space held by the map element.

[0080] In that case, a matching process is performed between the 3D feature points of the space held by the map element of interest and the 3D feature points of the space held by other map elements. If there is a map element in which the feature points that match the 3D feature points of the space held by the map element of interest are equal to or greater than a predetermined percentage, the redundant data determination unit 150 determines that the map element of interest is redundant.

[0081] Alternatively, the system may determine whether the change in the sensor output information, which is updated at a low frequency and at predetermined intervals, is below a predetermined threshold, among the first sensor output information and the second sensor output information. If it is determined to be below the predetermined threshold, the system may determine that the map element containing the sensor output information, which is updated at a high frequency, is redundant within the predetermined interval.

[0082] [Example 5] In Examples 1 to 4, the first sensor output information acquisition unit 110, the second sensor output information acquisition unit 120, the position and orientation information acquisition unit 130, the map element creation unit 140, and the redundant data determination unit 150 and redundant data deletion unit 160 were all described as performing a series of processes.

[0083] In Example 5, as shown in Figure 4, an example is described in which, in the case of an information processing device also equipped with a map element adjustment unit 400, some processing is separated from other processing and performed in separate threads. Specifically, the first sensor output information acquisition unit 110, the second sensor output information acquisition unit 120, the position and orientation information acquisition unit 130, the map element creation unit 140, the map element adjustment unit 400, and the redundant data determination unit 150 and the redundant data deletion unit 160 are processed in parallel in different threads.

[0084] Creating and adjusting map elements affects the accuracy of estimating the position and orientation of a moving object, and the processing load is high. On the other hand, determining and deleting redundant data does not affect the accuracy of estimating the position and orientation of a moving object, and it is sufficient if this is done before the map exceeds the allowable memory usage. Therefore, in this embodiment, two threads are provided: one that executes essential processing at high speed, and another that executes processing that only needs to be done when necessary, thereby speeding up the overall execution time of the position and orientation estimation process.

[0085] The configuration diagram of this embodiment is the same as in Figure 4, but the processing flow is shown in Figure 6. Figures 6(A) and 6(B) are flowcharts illustrating the operation of the information processing device 100 according to Embodiment 5. Figure 6(A) shows threads for creating map elements and adjusting map elements, and Figure 6(B) shows a thread for deleting redundant data. The differences in processing methods compared to other embodiments in Figures 6(A) and (B) are described below.

[0086] In Figure 6(A), in order to acquire the information necessary for the map elements, the first sensor output information acquisition unit 110 acquires the first sensor output information in step S600, and the second sensor output information acquisition unit 120 acquires the second sensor output information in step S601. In addition, the position and orientation information acquisition unit 130 acquires position and orientation information in step S602. Note that the order of steps S600 to S602 is not limited to this.

[0087] In step S603, the map element creation unit 140 creates map elements at the timing when the captured image and acceleration / angular velocity are used in combination, as described in Example 3. In step S604, the map element adjustment unit 400 adjusts the discrepancies in the data of each map element. Furthermore, in this embodiment, in step S604, once the map element adjustment unit 400 has performed the adjustment of the map element, it adds a flag to the map element indicating whether the adjustment has been completed or not.

[0088] In this embodiment 5, as shown in Figure 6(B), the map element adjustment unit 400 and the redundant data determination unit 150 are performed in separate threads. Therefore, when the redundant data determination unit determines whether a map element is redundant, the adjustment of the map element is not necessarily complete. Accordingly, in this embodiment, as described above, when the map element adjustment unit 400 performs adjustment of a map element, it adds a flag to the map element indicating whether the map element has been adjusted or not. Then, in step S605, the redundant data determination unit 150 checks for the flag indicating that the map element has been adjusted, and if the flag indicates that the map element has been adjusted, it performs a redundancy determination of the map element. If the flag indicates that the map element adjustment is not complete, nothing is done.

[0089] When the flag indicating that map elements have been adjusted is present, the redundant data determination unit 150 performs a determination of redundant data in step S606. The redundant data determination unit 150 determines whether a map element, whose data shifts have been adjusted by the map element adjustment unit 400, is redundant compared to other map elements. The redundancy determination method uses two methods: whether the position and orientation information held by the map element is within a predetermined range compared to other map elements, and whether the three-dimensional feature point information of the space is similar compared to other map elements. The redundant data determination unit 150 determines that a map element is redundant if it is found to be in the same condition using either one or both of these methods.

[0090] In step S607, when the redundant data deletion unit 160 deletes map elements that it has determined to be redundant, the redundant data deletion unit 160 recalculates the integral values ​​of acceleration and angular velocity held by the map elements, as described in Embodiment 4.

[0091] The above is an example of a method in which the processing in the map element creation unit 140 and the map element adjustment unit 400, and the processing in the redundant data determination unit 150 and the redundant data deletion unit 160 are executed in separate threads. The processing in the map element adjustment unit 400 is computationally intensive, and by separating the processing of the redundant data determination unit 150 and the redundant data deletion unit 160, which can be executed separately, into separate threads, the processing speed of the information processing device can be increased.

[0092] In the above, it was explained that the redundant data determination unit 150 performs redundant data determination when the map element to be determined has been adjusted. However, the redundant data determination may be performed regardless of whether the map element has been adjusted or not, and the redundant data deletion unit 160 may delete redundant map elements if the map element adjustment flag is set. In other words, steps S605 and S606 may be swapped. In that case, if the map element to be deleted has the map element adjustment incomplete flag, the redundant data deletion unit 160 does nothing. Alternatively, the map element adjustment unit 400 may perform map element adjustment processing including the map element to be deleted before deleting the map element to be deleted.

[0093] Furthermore, the redundant data determination unit 150 does not have to do anything if the map element to be determined does not have a "map element adjusted" flag, but it is not limited to this. For example, if the map element to be determined has a "map element adjustment not yet completed" flag, the map element adjustment unit 400 may be made to perform a map element adjustment process including the map element to be deleted, and then the redundant determination of the map element to be determined may be performed.

[0094] Furthermore, although it has been explained that the redundant data determination unit 150 checks whether or not there is a flag indicating that the map element has been adjusted, it is not limited to this. For example, the redundant data determination unit 150 may determine redundant map elements by adding conditions that indicate that the map element has been adjusted. That is, for example, by adding conditions such as when a certain amount of time has elapsed since the map element was created, or when a certain distance has been moved from the map element creation point, the determination of redundant data may be performed only after the adjustment of the map element has been reliably carried out.

[0095] Furthermore, the method for determining redundant data is not limited to whether the position and orientation information held by a map element is within a predetermined range compared to other map elements, or whether the 3D feature point information in space is similar to that of other map elements. As explained in Example 4, the determination may be made based on the measured values ​​of acceleration and angular velocity output by the IMU sensor, or as explained in Example 1, it may be determined whether there is redundant data among the data held by the map element. Alternatively, the method for determining redundant data in Example 3 may be used. Furthermore, in Example 5, it is sufficient if the redundancy detection and redundancy deletion are performed in a different thread from the processing for adjusting the map elements.

[0096] [Example 6] Examples 1 to 5 described the detailed processing content of the information processing device. Example 6 describes an example in which the redundant data determination unit 150 notifies the user of map elements that it has determined to be redundant. Only the differences from the other examples will be briefly explained.

[0097] The information processing device is equipped with a display unit 216 as shown in Figure 2. When the map element creation unit 140 creates a map element, it determines coordinate values ​​based on the location information held by the map element, and the display unit 16 displays the map element as an image on the display device. At that time, the display unit 16 notifies the user by displaying map elements that the redundant data determination unit 150 has determined to be redundant in a different display color from the other map elements. Furthermore, after the redundant data deletion unit 160 has deleted the redundant map elements, the display device removes the display of the redundant map elements to notify the user. This allows the user to visually confirm the results of the map element redundancy determination and deletion, and to understand which data is redundant and which data has been deleted.

[0098] Furthermore, the status of map element creation and deletion may be displayed on a display device using a GUI, or the color of map elements deemed redundant may be changed, the display of map elements to be deleted may blink for a certain period of time, and then the display of the map element may be removed. In addition to displaying through a GUI or other means, notification of deletion may also be given by voice, and the method of notifying the user is not limited.

[0099] Alternatively, the system may display the map's memory usage on the GUI and notify the user when the map's memory usage reaches a predetermined threshold, prompting them to delete redundant data. Furthermore, the system may be configured to perform redundant data determination and deletion as described in Examples 1 to 5 when the amount of memory used by the map reaches a predetermined threshold, and not perform redundant data determination and deletion as described in Examples 1 to 5 if the amount of memory used by the map does not reach a predetermined threshold.

[0100] [Example 7] In the above-described embodiment, a mobile robot 200 was used as the mobile body. However, the mobile body is not limited to a form like the mobile robot 200, and can be any mobile device such as an automobile, motorcycle, bicycle, wheelchair, ship, airplane, or drone.

[0101] Although the present invention has been described in detail above based on preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. Furthermore, Examples 1 to 7 may be combined as appropriate. Furthermore, in this embodiment, some or all of the control may be performed by supplying a computer program that realizes the functions of the embodiment described above to an information processing device, etc., via a network or various storage media. The computer (or CPU, MPU, etc.) in the information processing device, etc., may then read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0102] 100: Information Processing Device 110: First sensor output information acquisition unit 120: Second sensor output information acquisition unit 130: Position and orientation information acquisition unit 140: Map Element Creation Section 150: Redundancy Data Determination Unit 160: Redundant data removal section 170: First recovery 180: Second sensor 190:Position and orientation estimation unit 200: Mobile robot 400: Map element adjustment unit

Claims

1. In an information processing device that estimates the position and orientation of a moving object, A means for acquiring first sensor output information from a first sensor, A means for acquiring second sensor output information from a second sensor, A position and orientation information acquisition means that uses the first sensor output information and the second sensor output information to estimate the position and orientation of the moving body and acquire position and orientation information, A map element creation means for creating a map element that includes the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination means for determining redundant data among the data contained in the map element, A redundant data deletion means that deletes data that the redundant data determination means has determined to be redundant from the map element, It has, The information processing device is characterized in that the redundant data determination means determines that some of the map elements are redundant based on the accuracy of the position and orientation estimation and the timing of creating the map elements.

2. The information processing device according to claim 1, characterized in that the redundant data determination means determines that data among the data possessed by the map element that is not used for position and orientation estimation is redundant data.

3. The system includes a map element adjustment means for adjusting discrepancies in the information contained within the map elements, The information processing apparatus according to claim 1 or 2, characterized in that the redundant data determination means determines that data among the data possessed by the map element that the map element adjustment means does not use for adjusting the misalignment is redundant.

4. The information processing apparatus according to claim 3, wherein the redundant data determination means determines that the map element adjustment means does not use the data of the map element to adjust the misalignment of the map element to be redundant.

5. The information processing apparatus according to claim 1, wherein the redundant data determination means determines that a map element is redundant when the position and orientation held by the map element is redundant.

6. The information processing device according to claim 1, wherein the redundant data determination means determines that, among the first sensor output information and the second sensor output information, the change in the sensor output information that is updated at a low frequency and at a predetermined interval is below a predetermined threshold, the map element including the sensor output information that is updated at a high frequency at the predetermined interval is redundant.

7. The information processing apparatus according to claim 3 or 4, characterized in that the processing by the redundant data determination means and the redundant data deletion means is performed in a different thread from the processing by the map element adjustment means.

8. The information processing apparatus according to claim 7, characterized in that it notifies the user of the redundant data determined by the redundant data determination means.

9. The information processing apparatus according to claim 8, characterized in that when the redundant data deletion means deletes the redundant data, it notifies the user that the data has been deleted.

10. An information processing device according to any one of claims 1 to 9, A mobile body characterized by having a driving means for driving the mobile body based on the map elements acquired by the information processing device.

11. A computer program for controlling, by computer, each means of the information processing apparatus described in any one of claims 1 to 9 or the mobile body described in claim 10.

12. In an information processing method for estimating the position and orientation of a moving object, A first sensor output information acquisition step in which first sensor output information is acquired from the first sensor, A second sensor output information acquisition step, which involves acquiring second sensor output information from a second sensor, A position and attitude information acquisition step involves estimating the position and attitude of the moving body using the first sensor output information and the second sensor output information to acquire position and attitude information, A map element creation step that creates a map element including the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination step for determining redundant data among the data contained in the map element, The redundancy data deletion step includes deleting data determined to be redundant in the redundancy data determination step from the map elements. The information processing method is characterized in that the redundant data determination step determines that some of the map elements are redundant based on the accuracy of the position and orientation estimation and the timing of creating the map elements.

13. In an information processing device that estimates the position and orientation of a moving object, A means for acquiring first sensor output information from a first sensor, A means for acquiring second sensor output information from a second sensor, A position and orientation information acquisition means that uses the first sensor output information and the second sensor output information to estimate the position and orientation of the moving body and acquire position and orientation information, A map element creation means for creating a map element that includes the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination means for determining redundant data among the data contained in the map element, A redundant data deletion means that deletes data that the redundant data determination means has determined to be redundant from the map element, It has, The redundant data determination means is characterized in that, when it is determined that the change in the sensor output information, which is updated at a low frequency and at a predetermined interval, among the first sensor output information and the second sensor output information is below a predetermined threshold, the map element including the sensor output information, which is updated at a high frequency, is redundant at the predetermined interval.

14. In an information processing method for estimating the position and orientation of a moving object, A first sensor output information acquisition step in which first sensor output information is acquired from the first sensor, A second sensor output information acquisition step, which involves acquiring second sensor output information from a second sensor, A position and attitude information acquisition step involves estimating the position and attitude of the moving body using the first sensor output information and the second sensor output information to acquire position and attitude information, A map element creation step that creates a map element including the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination step for determining redundant data among the data contained in the map element, The redundancy data deletion step includes deleting data determined to be redundant in the redundancy data determination step from the map elements. The redundant data determination step is characterized in that, if it is determined that the change in the sensor output information, which is updated at a low frequency and at a predetermined interval, is below a predetermined threshold, the map element including the sensor output information, which is updated at a high frequency, is redundant at the predetermined interval.

15. In an information processing device that estimates the position and orientation of a moving object, A means for acquiring first sensor output information from a first sensor, A means for acquiring second sensor output information from a second sensor, A position and orientation information acquisition means that uses the first sensor output information and the second sensor output information to estimate the position and orientation of the moving body and acquire position and orientation information, A map element creation means for creating a map element that includes the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination means for determining redundant data among the data contained in the map element, A redundant data deletion means that deletes data that the redundant data determination means has determined to be redundant from the map element, The system includes a map element adjustment means for adjusting discrepancies in the information contained in the aforementioned map elements, The redundant data determination means determines that data from the map element that the map element adjustment means does not use to adjust the misalignment is redundant, The information processing apparatus is characterized in that the processing by the redundant data determination means and the redundant data deletion means is performed in a different thread from the processing by the map element adjustment means.

16. In an information processing method for estimating the position and orientation of a moving object, A first sensor output information acquisition step in which first sensor output information is acquired from the first sensor, A second sensor output information acquisition step, which involves acquiring second sensor output information from a second sensor, A position and attitude information acquisition step involves estimating the position and attitude of the moving body using the first sensor output information and the second sensor output information to acquire position and attitude information, A map element creation step that creates a map element including the first sensor output information, the second sensor output information, and the position and orientation information, A redundant data determination step for determining redundant data among the data contained in the map element, A redundant data deletion step is performed to delete the data determined to be redundant in the redundant data determination step from the map element. The system includes a map element adjustment step for adjusting discrepancies in the information contained in the map elements, The redundant data determination step determines that data from the map element that the map element adjustment step does not use to adjust the misalignment is redundant, The information processing method is characterized in that the processing by the redundancy data determination step and the redundancy data deletion step is performed in a different thread from the processing by the map element adjustment step.

Images