Information processing device, information processing method, and mobile robot

Abstract

To provide a technology to obtain highly precise map information.SOLUTION: Sensor information that is a sensing result on an actual space by a sensor during a movement, and a positional attitude of the sensor during the sensing are obtained. The sensor information and / or the positional attitude is taken as an index. The progress state of a work to obtain a sensor information set including a closed path is estimated on the basis of a recent index and a past index with a predefined relation with the recent index, and a notification of the estimated progress state is made.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for generating map information. [Background technology]

[0002] SLAM (Simultaneous Localization and Mapping) is a technology for estimating the position and orientation of a sensor such as a camera and map information of the surrounding environment by moving the sensor. Non-Patent Document 1 discloses a technology called loop-closing as a method for estimating highly accurate map information. In loop-closing, a loop-shaped section (closed route) is recognized on the route along which the sensor is moved, and the map continuity on the closed route is added as a constraint to estimate highly accurate map information. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] MA Raul, JMM Montiel and JD Tardos., ORB-SLAM: A Versatile and Accurate Monocular SLAM System, Trans. Robotics vol. 31, 2015 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to estimate map information of the surrounding environment of a sensor, an operator who takes photographs while moving the sensor cannot be sure whether the photographic data including a closed path is being acquired during the photographing process, and therefore the execution of loop closure is uncertain. As a result, it is difficult to obtain highly accurate map information. The present invention provides a technology for obtaining highly accurate map information. [Means for solving the problem]

[0005] One aspect of the present invention is a system including: an acquisition unit mounted on a moving body, the acquisition unit acquiring a sensing result of a real space by a sensor that senses the real space; While the moving body is moving along the moving path an estimation means for estimating a degree of progress until the movement path forms a closed path as a degree of completion of a task for acquiring a group of captured images to be used in closing a loop, based on the sensing result acquired by the acquisition means; a notification means for notifying the user of information regarding the achievement level; The present invention is characterized by comprising: [Effects of the Invention]

[0006] According to the configuration of the present invention, highly accurate map information can be obtained. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 shows the system configuration. [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of a mobile robot. [Figure 3] FIG. 5 is a block diagram showing an example of the functional configuration of an information processing device 500. [Figure 4] 10 is a flowchart showing the operation of the information processing device 500. [Figure 5] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] [First embodiment] Generally, SLAM involves continuously acquiring 2D images and 3D data from a sensor to estimate the position and orientation of the sensor and map information about the surrounding environment.One of the challenges of SLAM is a phenomenon known as drift, whereby the error in the map information increases depending on the amount of movement of the sensor.

[0010] To mitigate the effects of this drift, there is a function called loop closure. Loop closure recognizes a closed path on the route along which the sensor is moved, and estimates highly accurate map information by adding map continuity on the closed path as a constraint. The above-mentioned Non-Patent Document 1 describes a method for executing loop closure from photographic data that includes a closed path.

[0011] To create highly accurate map information, it is important to be able to perform loop closure to reduce the effects of drift. However, it is difficult to determine whether the captured image data including the closed path has been acquired at the time the user takes the image.

[0012] Therefore, in this embodiment, the progress of the work of acquiring a group of captured images including a closed route is estimated, and the estimated progress is notified to the user. The progress of the work indicates how far the work has come to completion, and may hereinafter be referred to as the degree of completion. The user determines whether to end the image capturing process by looking at the notified progress. This allows the group of captured images including the closed route to be reliably acquired and loop closure to be performed, thereby obtaining highly accurate map information. The specific configuration and procedure are described below.

[0013] First, the configuration of the system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a terminal device 400 and a mobile robot 200 that moves in response to operations of a user 300 on the terminal device 400, and the mobile robot 200 is further provided with an imaging device 100 that continuously captures images. In this embodiment, map information of the environment is estimated (generated) based on a group of images captured by the imaging device 100 while the mobile robot 200 is moving.

[0014] Next, an example of the configuration of a mobile robot will be described with reference to the block diagram of Fig. 2. As described above, the imaging device 100 continuously captures images, and each of the continuously captured images is input to the information processing device 500.

[0015] The information processing device 500 estimates (generates) map information of the environment based on a group of captured images continuously input from the imaging device 100, and estimates the progress of the work of acquiring a group of captured images including a closed path.

[0016] The signal transmitting / receiving unit 220 performs data communication with the terminal device 400. For example, when the user 300 operates the terminal device 400 to input a movement instruction to move the mobile robot 200, the terminal device 400 transmits a signal (movement instruction signal) including the movement instruction to the mobile robot 200 via wireless communication. The signal transmitting / receiving unit 220 receives the movement instruction signal transmitted from the terminal device 400 via wireless communication and outputs the received movement instruction signal to the motor control unit 210. The signal transmitting / receiving unit 220 also transmits a signal (notification signal) including a progress state estimated by the information processing device 500 to the terminal device 400. The terminal device 400 receives the notification signal and displays a screen based on the progress state included in the notification signal. The signal transmitting / receiving unit 220 also transmits a signal (map information signal) including map information generated by the information processing device 500 to the terminal device 400. The terminal device 400 receives the map information signal and displays a screen based on the map information included in the map information signal.

[0017] The motor control unit 210 controls the drive of the motor that controls the wheels 250, 251 of the mobile robot 200 based on the movement instruction signal output from the signal transmission / reception unit 220, thereby controlling the movement speed and movement direction of the mobile robot 200.

[0018] Next, an example of the functional configuration of the information processing device 500 will be described with reference to the block diagram of Fig. 3. The acquisition unit 510 acquires each captured image output from the imaging device 100 as the imaging device 100 continuously captures images. The position and orientation estimation unit 520 uses the captured images acquired by the acquisition unit 510 to estimate the position and orientation of the imaging device 100 at the time of capturing the captured images.

[0019] The progress state estimation unit 530 uses the captured images acquired by the acquisition unit 510 and / or the position and orientation estimated by the position and orientation estimation unit 520 as indices, and estimates the progress state of the work of acquiring a group of captured images including a closed path based on the latest indices and past indices that have a specified relationship with the latest indices.

[0020] The map information estimation unit 550 estimates map information of the environment based on the captured images acquired by the acquisition unit 510 and the position and orientation of the imaging device 100 estimated by the position and orientation estimation unit 520 based on the captured images. The map information is information in which feature points detected from captured images already acquired by the acquisition unit 510 are registered in association with the position and orientation of the imaging device 100 at the time of capturing the captured images. Note that, after estimating the map information, when estimating the position and orientation of the imaging device 100 from a newly captured image, corresponding feature points corresponding to the feature points detected from the newly captured image are identified from among the feature points included in the map information. Then, the position and orientation of the imaging device 100 at the time of capturing the newly captured image are estimated based on the position and orientation of the imaging device 100 associated with the corresponding feature points in the map information.

[0021] The notification unit 540 outputs the progress state estimated by the progress state estimation unit 530 and the map information estimated by the map information estimation unit 550 to the signal transmission / reception unit 220. As a result, the signal transmission / reception unit 220 generates a signal including the progress state and a signal including the map information, and transmits them to the terminal device 400.

[0022] The control unit 580 controls the overall operation of the information processing device 500. The storage unit 590 stores computer programs and data related to the various processes described below. The computer programs and data stored in the storage unit 590 are used by the other functional units to execute the various processes described below.

[0023] Next, the operation of the information processing device 500 will be described with reference to the flowchart of Fig. 4. The process according to the flowchart of Fig. 4 is executed while the mobile robot 200 is moving.

[0024] In step S700, control unit 580 performs initialization processing. In the initialization processing, computer programs and data stored in storage unit 590 are read out. The data stored in storage unit 590 includes camera parameters of imaging device 100, etc.

[0025] In step S710, the acquisition unit 510 acquires the captured image output from the imaging device 100 and stores the acquired captured image in the storage unit 590.

[0026] In step S720, the position and orientation estimation unit 520 estimates the position and orientation of the image capturing device 100 at the time of capturing the captured image based on the captured image acquired by the acquisition unit 510 in step S710, and stores the estimated position and orientation of the image capturing device 100 in the storage unit 590. Techniques for estimating the position and orientation of the image capturing device 100 that captured the captured image from the captured image are well known, and for example, the method described in Non-Patent Document 1 above can be used. In Non-Patent Document 1, the position and orientation of a sensor (the image capturing device 100 in this embodiment) is estimated by detecting feature points from the images and associating the images with each other.

[0027] In step S730, the progress state estimation unit 530 estimates the progress state of the work of acquiring a group of captured images including a closed path, based on the “group of positions of the imaging device 100 previously estimated by the position and orientation estimation unit 520” stored in the memory unit 590.

[0028] For example, the progress state estimation unit 530 determines as Qs the most recently estimated position among the "group of positions of the image capture device 100 estimated in the past by the position and orientation estimation unit 520" stored in the storage unit 590. The progress state estimation unit 530 also determines as Qt the position that is the shortest distance from the position Qs among the "group of positions of the image capture device 100 estimated in the past by the position and orientation estimation unit 520" stored in the storage unit 590. The progress state estimation unit 530 then calculates the distance D between the positions Qs and Qt, and estimates the "progress state (degree of accomplishment) X of the task of acquiring a group of captured images including a closed path" based on the calculated distance D. The progress state (degree of accomplishment) X can be calculated so that it decreases as the distance D increases and increases as the distance D decreases. For example, the progress state X is calculated based on an equation such as X = -D or X = 1 / D. In other words, the shorter the distance D, the higher the task progress state (degree of accomplishment) X is estimated to be.

[0029] In step S740, the notification unit 540 outputs the progress state (achievement level) X estimated in step S730 to the signal transmission / reception unit 220. As described above, the signal transmission / reception unit 220 transmits a notification signal including the progress state (achievement level) X to the terminal device 400.

[0030] When the terminal device 400 receives a notification signal from the mobile robot 200, it displays a screen based on the progress state (achievement level) X included in the communication signal. On the screen based on the progress state (achievement level) X, for example, the progress state (achievement level) X (numerical value) may be displayed as a character string, or a graph such as a bar graph or pie chart representing the numerical value may be displayed. Furthermore, the progress state (achievement level) X may be subjected to threshold processing to display a corresponding evaluation result, such as "GOOD" if the progress state (achievement level) X is equal to or greater than a threshold, or "NP_LOOP" if the progress state (achievement level) X is less than the threshold.

[0031] In step S750, the control unit 580 determines whether or not a termination condition for terminating imaging by the imaging device 100 has been satisfied. For example, when the user 300, upon viewing the progress status displayed on the terminal device 400, inputs an instruction to terminate imaging, the terminal device 400 transmits a signal including the instruction to terminate imaging (an end instruction signal) to the mobile robot 200. Upon receiving the end instruction signal, the signal transmitting / receiving unit 220 outputs the end instruction signal to the control unit 580, so that the control unit 580 determines that the termination condition has been satisfied and controls the imaging device 100 to stop imaging.

[0032] If the end condition is met as a result of the above determination, the process proceeds to step S760, and if the end condition is not met, the process proceeds to step S710, where imaging is continued.

[0033] In step S760, the map information estimation unit 550 estimates map information based on the captured images and the position and orientation of the imaging device 100 stored in the storage unit 590. Methods for estimating map information are well known, and for example, Non-Patent Document 1 discloses a method for estimating map information based on feature points on images captured at each time point.

[0034] In this way, in this embodiment, the progress of the work of acquiring captured images including a closed route is estimated based on the position of the imaging device 100 on the route, and the estimated progress is notified to the user. The user checks the notified progress and decides whether to end the image capturing. This makes it possible to reliably acquire captured images including a closed route and obtain highly accurate map information with loop closure.

[0035] [Second embodiment] In the following embodiments and modifications including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, they will be considered to be the same as the first embodiment. In the first embodiment, the progress state estimation unit 530 estimated the progress state based on the distance D between the image capture positions on the route. However, any method for estimating the progress state can be used as long as it can use either the image capture positions or the captured images to determine the degree to which the route along which the image capture device 100 has moved is closed, and notify the user of the determined degree as the progress state of task completion.

[0036] In this embodiment, as an example of a method for calculating the progress state based on captured images, a method for calculating the progress state based on the similarity of captured images on a route will be described. More specifically, in step S730, the progress state estimation unit 530 estimates the progress state of the work for acquiring a group of captured images including a closed route based on the “group of captured images previously acquired by the acquisition unit 510” stored in the storage unit 590.

[0037] For example, the progress state estimation unit 530 determines the most recently captured image Ps from among the "group of captured images previously acquired by the acquisition unit 510" stored in the storage unit 590. The progress state estimation unit 530 also determines the captured image Pt that has the highest similarity to the captured image Ps from among the "group of captured images previously acquired by the acquisition unit 510" stored in the storage unit 590. This "similarity" may be defined based on the SSD (see below) or the SAD (see below), or the similarity between images may be calculated in any manner. The progress state estimation unit 530 then calculates the SSD (sum of squared difference) between the captured image Ps and the captured image Pt, and calculates the similarity M between the captured image Ps and the captured image Pt based on the calculated SSD. The higher the similarity between images, the smaller the SSD value. Therefore, for example, the progress state estimation unit 530 calculates the similarity M by calculating M = -SSD. The progress state estimation unit 530 then calculates the progress state (degree of achievement) X so that it increases as the similarity M increases and decreases as the similarity M decreases. For example, the progress state estimation unit 530 sets the progress state (degree of achievement) X as X = M. In other words, the higher the similarity M, the higher the degree of progress (degree of achievement) X of the task is estimated to be, since it is determined that "the robot 200 has approached a route that it has taken in the past and formed a closed route."

[0038] In this embodiment, the similarity M is calculated based on the SSD between the captured image Ps and the captured image Pt. However, the method for calculating the similarity M between the captured image Ps and the captured image Pt is not limited to this. For example, the similarity M may be calculated based on the Sum of Absolute Difference (SAD) between the captured image Ps and the captured image Pt. Furthermore, two similar images tend to have corresponding feature points (feature points with similar feature amounts) detected between the two images. Taking advantage of this tendency, feature points may be detected from the captured image Ps and the captured image Pt, and the greater the number of corresponding feature points between the captured image Ps and the captured image Pt, the higher the similarity M may be. In this case, for example, when the number of corresponding feature points is N, the similarity M is set to N. Furthermore, the similarity between the captured images may be calculated based on the similarity of the arrangement of the feature points, etc., in addition to the number of corresponding feature points. In either case, the progress (achievement level) X is set to X=M.

[0039] In this manner, in this embodiment, the progress of the work of acquiring captured images including a closed route is estimated based on captured images along the route, and the estimated progress is notified to the user. The user checks the notified progress and decides whether to end the image capturing. This ensures that captured images including the closed route are acquired, and highly accurate map information with loop closure can be obtained.

[0040] The progress state may be calculated based on both the captured image and the position of the imaging device 100. That is, the progress state (achievement level) X may be calculated by calculating X=f(D, M)=(-D)+k×M, where k is a coefficient that determines the ratio between D and M, and is a preset value. As described above, various equations are possible for expressing the relationship between the progress state (achievement level) X and the distance D, and for expressing the relationship between the progress state (achievement level) X and the similarity M, and therefore functions that can be applied to the function f are not limited to the specific functions described above.

[0041] [Third embodiment] An autonomous mobile robot can be configured by configuring the mobile robot 200 as follows. The map information estimated by the map information estimation unit 550 is stored in the memory unit 590. The position and orientation estimation unit 520 estimates the position and orientation of the mobile robot 200 in real space based on the captured image acquired by the acquisition unit 510 and the map information stored in the memory unit 590. This estimation method is known, and the method described in Non-Patent Document 1, for example, can be used. A path planning unit is then newly added to the information processing device 500. The path planning unit calculates the next control value based on the position and orientation estimated by the position and orientation estimation unit 520 so that the mobile robot 200 can move along a predetermined target route, and outputs the control value to the motor control unit 210. This control value is the control amount of the motor for moving to the next destination, and the motor control unit 210 moves the mobile robot 200 by controlling the motor based on the control value.

[0042] Here, the accuracy of the estimated position and orientation depends on the accuracy of the map information. When the end of imaging is determined using the notification method of this embodiment, captured images that allow loop closure can be acquired, and therefore highly accurate map information can be obtained. In other words, when the map information is used to estimate the position and orientation of the imaging device 100, the accuracy can be improved. Furthermore, if the position and orientation can be estimated with high accuracy, the autonomous driving of the mobile robot can also be performed with high accuracy.

[0043] [Fourth embodiment] In the first embodiment, the progress status is estimated in step S730, the progress status is notified in step S740, it is determined whether the imaging termination condition is satisfied in step S750, and after imaging is terminated, the map information is estimated in step S760. However, the timing for estimating the map information is not limited to this estimation timing, and may be any timing after the value represented by the progress status becomes equal to or greater than a reference value and a closed path is formed, for example.

[0044] For example, after the progress state is estimated in step S730, if the value represented by the progress state is equal to or greater than a reference value, the map information may be estimated in step S760. Furthermore, instead of generating map information immediately after the image capture is completed, the map information may be generated when the user instructs generation of map information after the process according to the flowchart in Fig. 4 (excluding the process of step S760) is completed.

[0045] In response to this, the timing of notifying the progress status may be, as described in the first embodiment, immediately after estimating the progress status, or after estimating the map information, and the timing is not limited to a specific timing.

[0046] <First Modification> Depending on the movement of the mobile robot 200 (for example, when the moving speed of the mobile robot 200 is slow), similar captured images may be successively output from the imaging device 100. In such a case, the similarity M between the captured images becomes high, and the user may be notified of a high progress state even though the mobile robot 200 has not yet moved along the closed loop path.

[0047] Similarly, depending on the movement of the mobile robot 200 (for example, when the moving speed of the mobile robot 200 is slow), the position and orientation estimation unit 520 may repeatedly estimate similar positions and orientations. In such a case, the distance between the image capture positions becomes small, and the user may be notified that the mobile robot 200 is progressing well even though it has not yet traveled a closed loop path.

[0048] To avoid this situation, the progress status is calculated only when the distance traveled by the mobile robot 200 exceeds a predetermined reference value B; the progress status is not calculated unless the distance traveled by the mobile robot 200 exceeds the reference value B. The distance traveled by the mobile robot 200 can be calculated from the group of positions of the image capturing device 100 stored in the storage unit 590. For example, if position 1, position 2, ... position i (i is an integer equal to or greater than 3) are stored in the storage unit 590 in the order of movement (position i is the most recent position), then the distance L can be calculated as the sum of the results of calculating |position x - position (x - 1)| for each of x = 2 to i.

[0049] <Variation 2> In the above embodiment and modified examples, the imaging device 100 has been described as an example of a sensor that captures an image of real space, but any sensor may be used as long as it is a device that collects two-dimensional sensing results of real space.

[0050] Furthermore, the above-described embodiments and modifications can be implemented in the same way by using three-dimensional sensing results of real space (three-dimensional point clouds of real space) instead of two-dimensional sensing results of real space. When three-dimensional sensing results of real space are used instead of two-dimensional sensing results of real space, the image capture device 100 may be a stereo camera or a sensor that collects a three-dimensional point cloud of real space. For example, a sensor such as a range sensor or LiDAR can be used as a sensor that collects a three-dimensional point cloud of real space. Methods for estimating position and orientation and methods for estimating map information using three-dimensional point clouds are well known, as disclosed in the following documents, for example.

[0051] M.Keller, D.Lefloch, M.Lambers, S.Izadi, T.Weyrich, A.Kolb, "Real-time 3D Reconstruction in Dynamic Scenes using Point-based Fusion", International Conference on 3D Vision (3DV), 2013 In this document, the position and orientation of a sensor are estimated by matching 3D point clouds acquired at each time point. Then, the 3D point clouds acquired at each time point are integrated according to the position and orientation of the sensor to estimate map information of the environment. As described above, the position and orientation of the sensor at each time point can be estimated using the 3D point clouds acquired at each time point, so it is possible to estimate the progress state based on the position of the imaging device 100, as shown in the first embodiment. Furthermore, the progress state can also be estimated based on the similarity of 3D point clouds instead of captured images. The similarity of 3D point clouds can be determined, for example, by matching 3D point clouds acquired at adjacent times and determining the ratio of the number of matching 3D points between one 3D point cloud and another 3D point cloud. Methods for matching between 3D point clouds include, for example, ICP (Iterative Closest Point).

[0052] In other words, the information processing device 500 acquires sensor information, which is the result of sensing real space using a moving sensor, and the position and orientation of the sensor during the sensing, and estimates the progress of the work of acquiring a group of sensor information including a closed path based on the sensor information and / or the position and orientation as indices, the most recent indices, and past indices that have a specified relationship with the most recent indices.

[0053] Map information includes geometric information of the surrounding environment estimated from sensor information. Specifically, if the sensor information is a 2D image, the map information includes the position information of feature points of objects detected from the captured image at each point in time. If the sensor information is a 3D point cloud, the map information includes the 3D point cloud at each point in time or a 3D point cloud of the entire environment that is an integration of multiple 3D point clouds.

[0054] In the third embodiment, the position and orientation of the imaging device 100 mounted on the mobile robot 200 is estimated based on this map information, and the mobile robot 200 travels autonomously. If the imaging device 100 is a camera that captures two-dimensional images and the map information is information on feature points of the two-dimensional images, the position and orientation of the mobile robot 200 in real space is estimated by associating the feature points between the map information and the two-dimensional images. On the other hand, if the imaging device 100 is a sensor that collects three-dimensional sensing results and the map information is a three-dimensional point cloud of the environment, the position and orientation of the mobile robot 200 in real space is estimated by matching the three-dimensional point clouds.

[0055] <Variation 3> The method for estimating the position and orientation of the imaging device 100 is not limited to a specific estimation method. For example, the position and orientation of the imaging device 100 that captured the captured image may be estimated from feature points detected from the captured image. Furthermore, if a sensor that collects a 3D point cloud in real space is used as the imaging device 100, the position and orientation of the imaging device 100 may be estimated based on matching between 3D point clouds. Furthermore, the imaging device 100 may be equipped with an IMU (Inertial Measurement Unit) and the position and orientation of the imaging device 100 may be estimated using the IMU. Furthermore, the position and orientation of the imaging device 100 may be estimated based on GPS. Furthermore, an index with a known shape, such as an AR marker, may be placed in the environment, and the position and orientation of the imaging device 100 that captured the captured image may be estimated based on the captured image of the index.

[0056] <Variation 4> In the above-described embodiment and modified examples, the progress status is notified to the terminal device 400, thereby notifying the user 300 of the terminal device 400, but the destination of the progress status notification is not limited to the terminal device 400. In addition, the method of notifying the progress status is not limited to a specific notification method.

[0057] For example, the progress status may be notified to a notification device that can generate stimuli that allow the degree of progress (degree of achievement) to be detected by the senses of hearing, sight, touch, etc. Such a notification device may be a speaker that outputs sound, a monitor that outputs text and images, or a device that notifies the progress status by switching an LED on and off.

[0058] Furthermore, the notification may be given by changing the level of the notification, such as the volume of the output sound, the size of the displayed characters, the color of the characters, the type of characters, the intensity of the displayed light, etc.

[0059] <Example 5> In the above-described embodiment and modified examples, the progress state is estimated without being limited to a specific location, but the location for estimating the progress state may be determined in advance. For example, consider a case where the mobile robot 200 is moved along a path that starts at location F and returns to location F, while capturing images using the imaging device 100. Here, location F is set as the target location for progress state estimation.

[0060] In such a case, the progress state estimation unit 530 determines the position of the image capture device 100 in location F as Q0, and determines the most recently estimated position Qs from among "a group of positions of the image capture device 100 previously estimated by the position and orientation estimation unit 520" stored in the storage unit 590. Then, the progress state estimation unit 530 calculates the distance D between the position Q0 and the position Qs, and estimates "a progress state (degree of completion) X of the task of acquiring a group of captured images including a closed path" based on the calculated distance D, as in the first embodiment.

[0061] As another example, the progress state estimation unit 530 sets the captured image captured by the imaging device 100 at location F as P0, and the captured image captured most recently as Ps from the "group of captured images previously acquired by the acquisition unit 510" stored in the storage unit 590. Then, the progress state estimation unit 530 calculates the similarity between the captured image P0 and the captured image Ps in the same manner as in the first embodiment, and calculates the progress state (degree of achievement) X based on the calculated similarity.

[0062] Also in this embodiment, similar to the second embodiment, the progress status may be calculated based on both the distance D between the position Q0 and the position Qs and the similarity between the captured image P0 and the captured image Ps.

[0063] This method of obtaining the progress state (achievement level) X does not require searching for pairs of captured images and positions on the route (pairs for calculating the distance D or image similarity), so the calculation cost for obtaining the progress state (achievement level) X can be reduced.

[0064] [Fifth embodiment] 3 may be implemented as hardware, or each functional unit except for the storage unit 590 may be implemented as software (computer program). In the latter case, a computer device capable of executing this computer program is applicable to the information processing device 500. An example of the hardware configuration of a computer device applicable to the information processing device 500 will be described using the block diagram of FIG.

[0065] The CPU 501 executes various processes using computer programs and data stored in the RAM 502 and the ROM 503. As a result, the CPU 501 controls the overall operation of the information processing device 500, and executes or controls the various processes described above as being executed by the information processing device 500.

[0066] The RAM 502 has areas for storing computer programs and data loaded from the ROM 503 or nonvolatile memory 504, data received from the imaging device 100 or the signal transmitting / receiving unit 220 via the I / F 505, etc. The RAM 502 also has a work area used when the CPU 501 executes various processes. In this way, the RAM 502 can provide various areas as needed.

[0067] The ROM 503 stores setting data for the information processing device 500, computer programs and data related to the basic operation of the information processing device 500, computer programs and data related to the startup of the information processing device 500, and the like.

[0068] The nonvolatile memory 504 stores an OS (operating system), as well as computer programs and data for causing the CPU 501 to execute or control the processes described above as those performed by the information processing device 500. The computer programs and data stored in the nonvolatile memory 504 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and become targets for processing by the CPU 501. The storage unit 590 can be implemented in the RAM 502 or the nonvolatile memory 504.

[0069] The I / F 505 functions as a communication interface for performing data communication with the imaging device 100 and the signal transmitting / receiving unit 220. The CPU 501, RAM 502, ROM 503, nonvolatile memory 504, and I / F 505 are all connected to a bus 506.

[0070] Furthermore, the numerical values, processing timing, processing order, etc. used in the above explanation are given as examples to provide a concrete explanation, and are not intended to be limiting to these numerical values, processing timing, processing order, etc.

[0071] Furthermore, some or all of the above-described embodiments and modifications may be used in appropriate combination, and some or all of the above-described embodiments and modifications may be used selectively.

[0072] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0073] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0074] 100: Imaging device 220: Signal transmitting / receiving unit 500: Information processing device 510: Acquisition unit 520: Position and orientation estimation unit 530: Progress state estimation unit 540: Notification unit 550: Map information estimation unit 580: Control unit 590: Storage unit

Claims

1. an acquisition means for acquiring a sensing result of a real space by a sensor mounted on the moving body and sensing the real space; an estimation means for estimating a degree of progress until the movement path forms a closed path as a degree of completion of a task for acquiring a group of captured images to be used for closing a loop, based on the sensing results acquired by the acquisition means while the moving object is moving along the movement path; a notification means for notifying the user of information regarding the achievement level; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the estimation means estimates the degree of achievement based on the distance between a first position and a second position closest to the first position among a group of past positions of the sensor obtained based on sensing of real space by the sensor.

3. 3. The information processing apparatus according to claim 2, wherein the estimation means estimates the achievement level when the distance between the first position and the second position is equal to or greater than a predetermined distance.

4. The information processing apparatus according to claim 2 , wherein the degree of achievement is smaller as the distance is larger and is larger as the distance is smaller.

5. The information processing device according to claim 1, characterized in that the estimation means estimates the degree of achievement based on the distance between a most recent position and a position closest to the most recent position among a group of past positions of the sensor obtained based on sensing of real space by the sensor, and based on the image among a group of past captured images obtained based on sensing of real space by the sensor that has the highest similarity between the most recent captured image and the captured image.

6. The information processing apparatus according to claim 5 , wherein the degree of achievement is larger as the degree of similarity is higher and is smaller as the degree of similarity is lower.

7. the sensor is an imaging device that captures an image of a real space, 6. The information processing device according to claim 5, wherein the similarity includes at least one of a sum of squared differences (SSD) between the captured images, a sum of absolute differences (SAD) between the captured images, the number of corresponding feature points between the captured images, and a similarity in arrangement of corresponding feature points between the captured images.

8. the sensor is a device that collects a three-dimensional point cloud of a real space; 6. The information processing apparatus according to claim 5, wherein the similarity is a result of matching between three-dimensional point groups.

9. The information processing device according to any one of claims 1 to 8, characterized in that the estimation means calculates the movement distance of the sensor based on the position of the sensor obtained by sensing the real space by the sensor, estimates the degree of achievement if the movement distance exceeds a reference value, and does not estimate the degree of achievement if the movement distance does not exceed the reference value.

10. Furthermore, 10. The information processing apparatus according to claim 1, further comprising: a generating unit that generates map information of an environment based on a result of sensing the real space by the sensor.

11. 11. The information processing apparatus according to claim 10, wherein said generating means generates said map information if said achievement level is equal to or greater than a reference value.

12. 12. The information processing apparatus according to claim 1, wherein the notification means notifies information indicating that the task has been completed when the degree of completion is equal to or greater than a predetermined threshold value.

13. A mobile robot comprising: the information processing device according to claim 1; and the sensor.

14. An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires a sensing result of the real space by a sensor that senses the real space and is mounted on a moving body; an estimation step in which an estimation means of the information processing device estimates a degree of progress until the movement path forms a closed path as a degree of completion of a task for acquiring a group of captured images to be used for loop closing, based on the sensing results acquired in the acquisition step while the moving object is moving along the movement path; a notification step in which a notification means of the information processing device notifies a user of information regarding the achievement level; An information processing method comprising:

15. A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images