Estimation device, estimation method, and program

Abstract

To increase the accuracy of estimating the motion of a mobile body such as an autonomous mobile robot including a drone using a monocular SLAM.SOLUTION: An estimation device according to an embodiment includes an estimation unit, a control unit, and an SLAM unit. The estimation unit estimates whether or not a motion of a mobile body is an on-site rotation from a plurality of first monocular images acquired from a monocular camera equipped in the mobile body. The control unit determines a translation amount for which the monocular camera is to be translated when the motion of the mobile body is an on-site rotation. The SLAM unit estimates at least one of a position and an attitude of the monocular camera and a three-dimensional point of a subject of the monocular camera by a monocular SLAM (Monocular Simultaneous Localization and Mapping) from the plurality of first monocular images and a second monocular image acquired by the monocular camera translated on the basis of the translation amount.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an estimation device, an estimation method, and a program. [Background technology]

[0002] There are many well-known techniques for estimating the motion of moving objects, such as autonomous mobile robots like drones. For example, motion estimation is one of the important issues for autonomous mobile robots. When performing motion estimation using a monocular camera, the monocular SLAM (Monocular Simultaneous Localization and Mapping) technique is commonly used, which simultaneously estimates the self-position (position and orientation) and three-dimensional points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-526313 [Non-patent literature]

[0004] [Non-Patent Document 1] ORB-SLAM, [online], [Retrieved June 21, 2022], Internet〈URL: https: / / arxiv.org / abs / 1502.00956〉 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, it has been difficult to improve the accuracy of motion estimation for moving objects such as autonomous mobile robots like drones using monocular SLAM. [Means for solving the problem]

[0006] An estimation device according to an embodiment includes an estimation unit, a control unit, and a SLAM unit. The estimation unit estimates whether the motion of a moving object is a rotation on the spot based on a plurality of first monocular images acquired by a monocular camera mounted on the moving object. If the motion of the moving object is a rotation on the spot, the control unit determines a translation amount for translating the monocular camera. The SLAM unit estimates at least one of the position and orientation of the monocular camera and a three-dimensional point of a subject of the monocular camera by monocular SLAM (Monocular Simultaneous Localization and Mapping) based on the plurality of first monocular images and a second monocular image acquired by the monocular camera translated based on the translation amount. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a bird's-eye view showing an example of the autonomous mobile robot of the first embodiment and its movement to a destination. [Figure 2] FIG. 10 is a diagram for explaining an example of a process for estimating three-dimensional points using monocular SLAM. [Figure 3] FIG. 10 is a diagram for explaining an example of optical flow. [Figure 4A] This figure shows Example 1, where it is impossible to distinguish between in-place rotation and translation from optical flow. [Figure 4B] This figure shows Example 2, where it is impossible to distinguish between in-place rotation and translation from optical flow. [Figure 5A] FIG. 10 is a diagram showing Example 1 of a case where a three-dimensional point can be estimated. [Figure 5B] FIG. 10 is a diagram showing an example 1 of a case where estimation of a three-dimensional point fails. [Figure 6A] FIG. 10 is a diagram showing Example 2 of a case where a three-dimensional point can be estimated. [Figure 6B] FIG. 10 is a diagram showing Example 2 of a case where estimation of a 3D point fails. [Figure 7] An overhead view showing an example of a situation where a turn in place is required. [Figure 8] 10A and 10B are diagrams for explaining an example of a method for estimating translation based on three-dimensional points estimated at a finite depth. [Figure 9]FIG. 2 is a diagram showing an example of the functional configuration of the estimation device according to the first embodiment. [Figure 10A] This figure shows an example of optical flow when the monocular camera is translated 40 cm horizontally. [Figure 10B] A diagram showing an example of the depth of a 3D point when the monocular camera is translated 40 cm horizontally. [Figure 11A] This figure shows an example of optical flow when the monocular camera is rotated in place by 1.5 degrees in yaw only. [Figure 11B] A diagram showing an example of the depth of a 3D point when the monocular camera is rotated in place by 1.5 degrees in yaw only. [Figure 12] 4 is a flowchart showing an example of an estimation method according to the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of the functional configuration of an estimation apparatus according to a second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a moving unit according to the second embodiment. [Figure 15] FIG. 2 is a diagram showing an example of the hardware configuration of the estimation device according to the first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of an estimation device, an estimation method, and a program will be described in detail with reference to the accompanying drawings.

[0009] (First embodiment) In the first embodiment, an estimation device mounted on an autonomous mobile robot (an example of a mobile body) such as the following will be described as an example.

[0010] An autonomous mobile robot is given a destination and moves to it. The location of the autonomous mobile robot will change during its travel depending on the obstacle situation. An example of an autonomous mobile robot is a drone. Figure 1 shows an example of a drone being given a destination and moving to that destination.

[0011] Fig. 1 is a bird's-eye view showing an example of autonomous mobile robot 100 according to the first embodiment and its movement to a destination. In the example of Fig. 1, the autonomous mobile robot 100 is a drone equipped with a monocular camera 1. In the example of Fig. 1, the route to a given destination is a narrow space surrounded by walls on both sides.

[0012] In the first embodiment, we will explain a case where motion estimation is performed using monocular SLAM. Monocular SLAM is a technology that estimates the self-position and 3D points of the surrounding environment based on the positions where the same object (for example, corner points of a subject in 3D space) appears in an image. As an example, Figure 2 shows a case where two frames are used for motion estimation.

[0013] FIG. 2 is a diagram illustrating an example of a process for estimating 3D points 104 using monocular SLAM. In FIG. 2, corner points of an object 102 are detected from an image of a past frame 101a and an image of a current frame 101b using, for example, Shi-Tomasi corner point detection. Note that the object 102 may be any object. In this embodiment, positions on the image that are characteristic of the object 102, such as corner points, are referred to as feature points 103.

[0014] After detecting the feature points 103, similar feature point pairs are found between the feature points 103 in the past frame 101a and the feature points 103 in the current frame 101b. These feature point pairs are assumed to be in correspondence with each other. In monocular SLAM, a 3D point 104 is estimated based on the correspondence between the feature point pairs. Furthermore, the self-position 105 of the monocular camera 1 is estimated based on the estimated 3D point 104 (see FIG. 8 for details, which will be described later).

[0015] Monocular SLAM encounters problems when performing pure rotational motion (referred to as "pure rotation" in this embodiment). First, optical flow will be described.

[0016] 3 is a diagram illustrating an example of the optical flow 106. The optical flow 106 is a vector that indicates the movement of corresponding feature points 103 in each frame 101. The optical flow 106 is used to estimate the self-position 105 of the monocular camera 1.

[0017] Fig. 4A is a diagram showing Example 1 in which it is impossible to distinguish between in-situ rotation and translation from the optical flow 106. Fig. 4B is a diagram showing Example 2 in which it is impossible to distinguish between in-situ rotation and translation from the optical flow 106. First, we will explain the problem that occurs at the start of motion estimation (a stage when there are no estimated three-dimensional points 104 and the self-position 105 of the monocular camera 1 is not known).

[0018] The motion estimation at the start of the motion estimation is performed using a common method such as a five-point algorithm. In the five-point algorithm, the motion of the monocular camera 1 (i.e., its own position) at the start of the motion estimation is estimated based on the input of five optical flows 106.

[0019] However, as shown in Figures 4A and 4B, there are cases where it is difficult to distinguish between rotation and translation based on the optical flow 106 alone. Figures 4A and 4B show an example of motion estimation when the optical flows 106 have approximately the same orientation and approximately the same length.

[0020] The example in Figure 4A shows a case in which it is difficult to distinguish between a rotation in place (clockwise) when the distance between each 3D point 104 of the subject 102 and the monocular camera 1 is different and a translation (movement to the right) when the distance between each 3D point 104 of the subject 102 and the monocular camera 1 is approximately the same.

[0021] The example in Figure 4B shows a case in which it is difficult to distinguish between a rotation in place (clockwise) when the distance between each 3D point 104 of the subject 102 and the monocular camera 1 is the same and a translation (movement to the right) when the distance between each 3D point 104 of the subject 102 and the monocular camera 1 is approximately the same.

[0022] 4A and 4B, it is indistinguishable between a case where the subject 102 rotates on the spot and a case where the subject 102 moves in a translational manner while the distance between the subject 102 and the monocular camera 1 is substantially constant. In motion estimation using the optical flow 106, if estimation is performed based on the indistinguishable optical flow 106 described above, there is a possibility that both rotation and translation will be erroneously estimated. Therefore, if the autonomous mobile robot 100 equipped with the monocular camera 1 rotates on the spot at the start of motion estimation, the accuracy of the motion estimation will deteriorate.

[0023] Next, a problem that occurs during the continuation of motion estimation (at the stage when there are estimated three-dimensional points 104 and the self-position 105 of the monocular camera 1 in the past frame 101a is also known) will be described.

[0024] Fig. 5A is a diagram showing an example of a case where it is possible to estimate a 3D point 104. Fig. 5B is a diagram showing an example of a case where it is not possible to estimate a 3D point 104. The position of the optical axis in Figs. 5A and 5B is the position of the optical axis of the actual monocular camera 1.

[0025] When the monocular camera 1 rotates in place during continuous motion estimation, the baseline becomes smaller as shown in Fig. 5B, compared to when the monocular camera 1 moves to obtain a sufficient baseline (Fig. 5A). If the three-dimensional position of the three-dimensional point 104 is estimated in a situation where the baseline is small, an incorrect depth (infinitely far away) may be estimated, as shown in Fig. 5B.

[0026] FIG. 6A is a diagram showing Example 2 of a case where a 3D point 104 can be estimated. FIG. 6B is a diagram showing Example 2 of a case where estimation of a 3D point 104 fails. The position of the optical axis in FIGS. 6A and 6B is the position of the optical axis of the actual monocular camera 1. After rotation in place, there may be no 3D point 104 estimated at a finite depth (the 3D position of a 3D point estimated in a situation where the baseline is large, as shown in FIG. 5A). Of the movements to be estimated, translational movements in particular require a 3D point 104 estimated at a finite depth, as shown in FIG. 6A. Therefore, if there is no 3D point 104 estimated at such a finite depth (FIG. 6B), it will be impossible to estimate correct translational movement.

[0027] FIG. 7 is an overhead view showing an example of a situation where a turn on the spot is required. The example in FIG. 7 shows a case where the autonomous mobile robot 100 equipped with the monocular camera 1 moves in translation up to the wall ahead, and then continues to turn on the spot until it turns 90 degrees to the right at a corner. In this way, the autonomous mobile robot 100 may turn on the spot depending on the obstacle situation. Due to the above-mentioned problem, motion estimation of the autonomous mobile robot 100 fails after the turn on the spot.

[0028] For example, in conventional technologies such as Patent Document 1, if the subject 102 continues to rotate in place (for example, if it continues to rotate 180 degrees) during continued motion estimation, most of the three-dimensional points 104 of the subject 102 become three-dimensional points 104 at infinity, making it impossible to estimate correct translational motion.

[0029] Here, an example of a method for estimating translation based on 3D points 104 estimated at finite depth will be described.

[0030] FIG. 8 is a diagram illustrating an example of a method for estimating translation based on a 3D point 104 estimated at a finite depth. When a 3D point 104 estimated at a finite depth is obtained, for example, as shown in FIG. 8, it is assumed that the self-position 105 of the monocular camera 1 is estimated in real time using monocular SLAM. First, the 3D point 104 estimated at a finite depth is estimated based on the correspondence between pairs of feature points 103. Next, the 3D point 104 estimated at a finite depth is projected onto a point 107 on the current camera image. Finally, the position and orientation (rotation) of the monocular camera 1 are optimized so that the difference between the position indicated by the projected point 107 and the position on the image corresponding to the pair of feature points 103 is minimized.

[0031] Here, the 3D points 104 at finite depths are used as inputs for motion estimation to optimize translation and rotation, while the 3D points 104 at infinity are used as inputs for motion estimation to optimize only rotation. If there are several 3D points 104 estimated at finite depths among the 3D points 104 input during optimization, translation can be estimated by using only the 3D points 104 estimated at finite depths. Generally, the more 3D points 104 input, the higher the accuracy of motion estimation. Therefore, the more 3D points 104 estimated at finite depths there are, the more accurate the translation estimation can be expected. Conversely, the fewer 3D points 104 estimated at finite depths used during motion estimation, the lower the accuracy of translation estimation, and if there are no 3D points 104 estimated at finite depths, translation estimation is not possible.

[0032] As described above, the autonomous mobile robot 100 faces the following problems when starting to estimate motion and when continuing to estimate motion.

[0033] <At the start of motion estimation> At the start of motion estimation, the optical flows 106 may have almost the same length and orientation. In this case, as shown in Figures 4A and 4B above, it is impossible to distinguish between rotation and translation, which results in a decrease in accuracy of motion estimation at the start of motion estimation.

[0034] <During continuous motion estimation> Even when motion estimation continues, if rotation continues on the spot (for example, if rotation continues 180 degrees), most of the 3D points 104 of the subject 102 will become 3D points 104 at infinity. Therefore, if rotation on the spot continues, the accuracy of translation estimation will deteriorate.

[0035] Hereinafter, an embodiment of an estimation device that can improve the accuracy of estimating the movement of the autonomous mobile robot 100 even in the above-mentioned case will be described.

[0036] In order to solve the above problem, the estimation device 10 of the first embodiment translates the autonomous mobile robot 100 (monocular camera 1) so that the movement of the monocular camera 1 becomes a sufficient baseline when the movement is a rotation on the spot, thereby preventing a deterioration in the accuracy of the movement estimation or a failure of the movement estimation.

[0037] Here, the definitions of the degradation of estimation accuracy and failure in the description of the first embodiment are as follows: Specific examples of the degradation of accuracy and failure of motion estimation are shown below.

[0038] "Deterioration of estimation accuracy" means that the difference between the estimation result and the true value is large (greater than a predetermined threshold). "Failure of estimation" means that estimation cannot be performed because there is no data to be used for motion estimation performed by the estimation device 10.

[0039] Specific examples of "deterioration in motion estimation accuracy" At the start of motion estimation, it may be difficult to distinguish between rotation and translation using only the optical flow 106, resulting in a decrease in the accuracy of estimation of both rotation and translation. If the rotation in place continues, the number of 3D points 104 estimated at a finite depth among the 3D points 104 input during optimization decreases, and the accuracy of estimating translation deteriorates.

[0040] Specific examples of "failure in motion estimation" During motion estimation, if there is no input of the estimated 3D points 104 (if both the 3D points 104 at finite depth and the 3D points 104 at infinity cannot be used as input for motion estimation), it is not possible to estimate both rotation and translation. During motion estimation, if there is no input of the 3D point 104 estimated at a finite depth, translation cannot be estimated.

[0041] The characteristics of the estimation device 10 of the first embodiment are as follows. When rotating on the spot, the autonomous mobile robot 100 (monocular camera 1) is translated. The amount of translation (baseline) of the autonomous mobile robot 100 (monocular camera 1) is changed depending on the distance to the subject 102. The farther the distance to the subject 102, the smaller the change in appearance (parallax). Monocular SLAM obtains information about the distance between the subject 102 and the monocular camera 1 based on changes in appearance, so the smaller the change in appearance, the less accurate the motion estimation. Rotation in place is detected from the optical flow 106. Specifically, the following criteria are used for the determination: If there is variation in the length of the optical flow 106, it is determined to be translation; If there is no variation in the length of the optical flow 106, it is determined to be rotation in place.

[0042] An example of the functional configuration of the estimation device 10 of the first embodiment will be described in detail below. The estimation device 10 of the first embodiment estimates a self-position 105 (position and orientation) and a three-dimensional point 104 of an autonomous mobile robot 100 based on a monocular image input from a monocular camera 1 in real time.

[0043] [Example of functional configuration] 9 is a diagram showing an example of the functional configuration of the estimation device 10 of the first embodiment. The estimation device 10 of the first embodiment is connected to an autonomous mobile robot 100. The estimation device 10 is mounted on the autonomous mobile robot 100. Note that the estimation device 10 may be a server device including components other than the monocular camera 1 (a SLAM unit 2, an estimation unit 3, and a control unit 4), and the server device may be remotely connected to the autonomous mobile robot 100 via a wireless network or the like.

[0044] The estimation device 10 of the first embodiment includes a monocular camera 1, a SLAM unit 2, an estimation unit 3, and a control unit 4.

[0045] The monocular camera 1 is mounted on the autonomous mobile robot 100 and captures monocular images in real time. The autonomous mobile robot 100 is, for example, a drone. The autonomous mobile robot 100 is a robot that can move to a given destination. Furthermore, the movement path of the autonomous mobile robot can be changed while traveling depending on the status of obstacles estimated by the SLAM unit 2.

[0046] The SLAM unit 2 uses SLAM technology to estimate the self-position 105 of the monocular camera 1 (autonomous mobile robot 100) and the three-dimensional point 104 of the subject 102 from the monocular image acquired by the monocular camera 1. Examples of SLAM technology that can be used include Non-Patent Document 1 and PTAM (Parallel Tracking and Mapping). PTAM is a monocular SLAM that estimates the camera pose from the correspondence between feature points.

[0047] The estimation unit 3 estimates whether the movement of the autonomous mobile robot 100 is a rotation on the spot from the optical flow 106 that indicates the movement of the feature points of the subject 102. For example, if there is variation in the length of the optical flow (if the variation in the length is equal to or greater than a threshold), the estimation unit 3 estimates that the movement is a translation. Also, for example, if there is no variation in the length of the optical flow (if the variation in the length is less than a threshold), the estimation unit 3 estimates that the movement is a rotation on the spot.

[0048] However, as shown in FIGS. 4A and 4B above, when the optical flows 106 are in approximately the same direction and approximately the same length, it may be difficult to distinguish between translation and rotation. In such cases, the estimation unit 3 of the first embodiment can safely estimate in-place rotation. This is because, when in-place rotation is estimated, even if the autonomous mobile robot 100 (monocular camera 1) is translated to move the monocular camera 1 so as to obtain a baseline, this will only result in an increase in new 3D points 104. Therefore, the estimation unit 3 of the first embodiment estimates in-place rotation when there is no variation in the length of the optical flows 106.

[0049] The control unit 4 translates the autonomous mobile robot 100 equipped with the monocular camera 1. For example, in the example of Fig. 1, since there are walls nearby in the horizontal left and right directions, the translation direction is the vertical up and down direction. The amount of translation (translation amount) is determined by the control unit 4.

[0050] For example, the control unit 4 calculates the amount b (baseline) of translation of the monocular camera 1 using the following equations (1) and (2).

[0051] <At the start of motion estimation> b = upper limit of the movable area (1) <During continuous motion estimation> b = F(d) (2) However, if F(d) exceeds the movable area, b is set to the upper limit of the movable area.

[0052] Here, d is the distance between the monocular camera 1 and the subject 102, and F is a function that determines the amount by which the monocular camera 1 is translated.

[0053] That is, the control unit 4 determines the upper limit of the movable area of ​​the monocular camera 1 when the estimation of the movement starts, and determines the translation amount b within a range below the upper limit depending on the distance to the subject 102 while the estimation of the movement continues.

[0054] When translating the monocular camera 1, the control unit 4 translates the autonomous mobile robot 100 in a predetermined direction, thereby translating the monocular camera 1. When starting to estimate the movement, the control unit 4 determines the direction with the largest upper limit of the movable area of ​​the monocular camera 1 as the predetermined direction, based on the upper limit of the movable area of ​​the monocular camera 1. For example, in the example of FIG. 1, the movable area is the height from the ground to the ceiling.

[0055] For example, if the autonomous mobile robot 100 or the estimation device 10 is equipped with an ultrasonic sensor, the distance d may be a depth statistic (e.g., average or median) of the three-dimensional point 104 acquired by the ultrasonic sensor. Alternatively, the distance d may be a depth statistic of a target object such as a specific pedestrian or an oncoming vehicle. The target object can be detected by the estimation unit 3 using a neural network obtained by a machine learning technique such as a convolutional neural network (CNN).

[0056] The greater the distance d, the smaller the change in appearance, so accuracy in motion estimation will be poor unless the autonomous mobile robot 100 equipped with the monocular camera 1 is translated by a large amount. Therefore, the greater the distance d, the greater the translational movement of the autonomous mobile robot 100 equipped with the monocular camera 1. In other words, F is a function such that the greater the distance d, the greater b=F(d) becomes.

[0057] Here, we will explain why the change in appearance is small as the distance d increases, unless the autonomous mobile robot 100 equipped with the monocular camera 1 is translated to a greater extent. If the focal length (constant) of the monocular camera 1 is f and the change in appearance (parallax) is s, the following relationship exists between d, b, s, and f. d = b × (f / s) (3)

[0058] By transforming equation (4), the following equation (4) is obtained. s = f × (b / d) (4)

[0059] Therefore, according to equation (4), unless the autonomous mobile robot 100 equipped with the monocular camera 1 is translated so that the translation amount (baseline) b becomes larger as the distance d becomes larger, the parallax s, i.e., the change in appearance, becomes smaller.

[0060] Next, an example of a simulation result of the optical flow 106 when the autonomous mobile robot 100 (monocular camera 1) performs translational motion, and an example of a simulation result of the optical flow 106 when the autonomous mobile robot 100 performs rotational motion on the spot will be shown.

[0061] Fig. 10A is a diagram showing an example of optical flow 106 when monocular camera 1 is translated 40 cm in the horizontal direction. Fig. 10B is a diagram showing an example of the depth of 3D point 104 when monocular camera 1 is translated 40 cm in the horizontal direction.

[0062] Fig. 11A is a diagram showing an example of optical flow 106 when monocular camera 1 is rotated in place by 1.5 degrees in yaw only. Fig. 11B is a diagram showing an example of the depth of 3D point 104 when monocular camera 1 is rotated in place by 1.5 degrees in yaw only.

[0063] 10A and 11A, the number of valid flows is the number of samples of the optical flow 106. The flow length is the statistical quantity (maximum, minimum, and average) of the length of the optical flow 106. The difference between the maximum, minimum, and average values ​​of the flow length in FIG. 11A is smaller than the difference between the maximum, minimum, and average values ​​of the flow length in FIG. 10A.

[0064] Differences in the minimum, maximum, and average values ​​indicate variations in the optical flow 106. Here, variations mean that the lengths of the vectors of the optical flow 106 differ depending on the depth (foreground and background) within the image. For example, the estimation unit 3 calculates the variance of the flow length of the optical flow 106, and determines the magnitude of the variations using the variance and a threshold value that is set in advance to an appropriate value depending on the operating environment of the autonomous mobile robot 100, etc.

[0065] Specifically, if the variance of the flow length is equal to or greater than a threshold, the estimation unit 3 determines that the variation is large (estimated as translation).If the variance of the flow length is less than the threshold, the estimation unit 3 determines that the variation is small (estimated as in-place rotation).

[0066] The estimation device 10 may be equipped with multiple monocular cameras 1. For example, when multiple monocular cameras 1 with different performance are equipped, a monocular image captured by any of the monocular cameras 1 may be used in the estimation process.

[0067] [Example of estimation method] Fig. 12 is a flowchart showing an example of the estimation method of the first embodiment. The example of Fig. 12 shows a flowchart for the case where the movement of the autonomous mobile robot 100 is estimated to be a turn on the spot.

[0068] First, the estimation unit 3 estimates that the movement of the autonomous mobile robot 100 (an example of a moving body) is a rotation on the spot from a plurality of first monocular images acquired by the monocular camera 1 mounted on the autonomous mobile robot 100 (an example of a moving body) (step S11).

[0069] Next, if the movement of the autonomous mobile robot 100 is estimated to be a rotation on the spot by the process of step S11, the control unit 4 determines the translation amount by which the monocular camera 1 is translated (step S12).

[0070] Next, the SLAM unit 2 estimates the position and orientation of the monocular camera 1 and the three-dimensional point 104 of the subject 102 of the monocular camera 1 using monocular SLAM from the multiple first monocular images and the second monocular image acquired by the monocular camera 1 translated based on the translation amount determined by the processing of step S12 (step S13).

[0071] In step S13, the estimation of the position and orientation of the monocular camera 1 may be an estimation of at least one of the position and orientation of the monocular camera 1. For example, an estimation result of only the position of the monocular camera 1 may be used.

[0072] As a result, the estimation device 10 of the first embodiment can improve the accuracy of motion estimation of the autonomous mobile robot 100 (an example of a moving object) using monocular SLAM. Specifically, the estimation device 10 of the first embodiment can achieve the following effects (1) to (3), for example.

[0073] (1) Effect of translating the monocular camera 1 when rotating on the spot Even if the object rotates in place when starting motion estimation, a sufficient baseline can be obtained, resulting in a finite depth 3D point 104. This prevents a deterioration in motion estimation accuracy or a failure of motion estimation. Even if the object continues to rotate on the spot (for example, if it rotates 180 degrees), a sufficient baseline can be obtained, resulting in a 3D point 104 with a finite depth. This prevents a deterioration in the accuracy of motion estimation or a failure of motion estimation.

[0074] (2) Effect of changing the amount of translation depending on the distance to the subject 102 If the amount of translation of the monocular camera 1 is small relative to the distance of the subject 102, the accuracy of the motion estimation and the accuracy of the estimation of the 3D points may deteriorate. The estimation device 10 of the first embodiment can prevent deterioration of the motion estimation accuracy by increasing the amount of translation of the monocular camera 1 so that the baseline becomes larger as the subject 102 becomes farther away (by changing the amount of translation of the monocular camera 1 depending on the distance of the subject 102). The amount of translation of the monocular camera 1 can be reduced compared to when the monocular camera 1 is unconditionally translated by a large amount to accommodate a distant subject 102.

[0075] (3) The effect of detecting in-situ rotation from the variation of optical flow Since the motion estimation history is not used, it is possible to detect whether or not the rotation is on the spot even at the start of motion estimation (there is no motion estimation history at the start). Rotation on the spot can be detected using only the monocular camera 1. Even in applications that previously required an IMU (Inertial Measurement Unit) to detect rotation on the spot, there is no need to install any sensors other than the monocular camera 1, which reduces product costs. For autonomous mobile robots 100, such as drones, that are expected to operate for long periods of time, the method for detecting in-place rotation using an IMU poses a problem. Specifically, when rotational and translational motions are estimated from the IMU, errors accumulate in the velocity estimation, resulting in a significant degradation in the accuracy of the in-place rotation estimation. The in-place rotation detection method using the estimation device 10 of the first embodiment does not encounter this problem. Below, we will describe a specific example of the accumulation of error in acceleration. Assume that an initial velocity is given at a certain time. Furthermore, suppose that velocity is continuously integrated from the acceleration to determine the velocity from a certain time onward. In this case, if the acceleration obtained from the IMU contains an error, the acceleration error accumulates in the velocity with each integration. Therefore, the amount of error in the velocity increases over time.

[0076] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0077] [Example of functional configuration] 13 is a diagram showing an example of the functional configuration of an estimation device 10-2 according to the second embodiment. The estimation device 10-2 according to the second embodiment includes a monocular camera 1, a SLAM unit 2, an estimation unit 3, a control unit 4, and a moving unit 5. The difference from the configuration according to the first embodiment is that the estimation device 10-2 further includes a moving unit 5.

[0078] Based on the translational movement amount b (baseline) determined by the control unit 4, the movement unit 5 changes the position of the monocular camera 1 mounted on the autonomous mobile robot 100 by translating it.

[0079] FIG. 14 is a diagram showing an example of the moving unit 5 of the second embodiment. The autonomous mobile robot 100 of the second embodiment is, for example, an industrial robot or an automobile. The moving unit 5 translates the monocular camera 1, for example, as shown in FIG. 14. In the example of FIG. 14, the range of motion of the monocular camera 1 corresponds to the upper limit of the movable area (formula (1)) of the first embodiment. In the example of FIG. 14, the direction in which the monocular camera 1 is translated is perpendicular to the optical axis direction of the camera. The translation amount b (baseline) is determined by the control unit 4.

[0080] As described above, the estimation device 10-2 of the second embodiment further includes a movement unit 5 that translates the position of the monocular camera 1 mounted on a moving body such as an industrial robot or an automobile based on the amount of translation. When translating the monocular camera 1, the control unit 4 translates the monocular camera 1 by controlling the movement unit 5.

[0081] As a result, the estimation device 10-2 of the second embodiment can achieve the same effects as the estimation device 10 of the first embodiment.

[0082] Finally, an example of the hardware configuration of the estimation device 10 (10-2) according to the first and second embodiments will be described. The estimation device 10 (10-2) according to the first and second embodiments can be realized by using, for example, any computer device as basic hardware.

[0083] [Example of hardware configuration] 15 is a diagram illustrating an example of the hardware configuration of the estimation device 10 (10-2) according to the first and second embodiments. The estimation device 10 (10-2) according to the first and second embodiments includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0084] The estimation device 10 (10-2) may not be provided with some of the above components. For example, if the estimation device 10 (10-2) can use the input function and display function of an external device, the estimation device 10 (10-2) may not be provided with the display device 204 and the input device 205.

[0085] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a ROM and a RAM. The auxiliary storage device 203 is a hard disk drive (HDD), a memory card, or the like.

[0086] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the estimation device 10 (10-2). The display device 204 and the input device 205 may be realized by a touch panel or the like having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0087] For example, the program executed by the estimation device 10 (10-2) is provided as a computer program product in the form of an installable or executable file recorded on a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0088] Also, for example, the program executed by the estimation device 10 (10-2) may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0089] Furthermore, for example, the program executed by the estimation device 10 (10-2) may be provided via a network such as the Internet without being downloaded. Specifically, the estimation process may be executed by a cloud service of, for example, an ASP (Application Service Provider) type.

[0090] Furthermore, for example, the program for the estimation device 10 (10-2) may be provided by being pre-installed in a ROM or the like.

[0091] The program executed by the estimation device 10 (10-2) has a modular configuration including functions that can be realized by the program among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 201 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 202. In other words, the above-mentioned functional blocks are generated on the main storage device 202.

[0092] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) rather than by software.

[0093] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0094] 1 monocular camera 2 SLAM section 3 Estimation part 4. Control Unit 5. Moving section 10 Estimation device 100 Autonomous Mobile Robot 201 processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input Device 206 Communication Equipment 210 Bus

Claims

1. an estimation unit that estimates whether or not the movement of the moving object is a rotation on the spot from a plurality of first monocular images acquired by a monocular camera mounted on the moving object; a control unit that determines a translation amount for translating the monocular camera when the movement of the moving object is estimated to be a rotation on the spot from an optical flow that indicates a movement of a feature point of a subject of the monocular camera; a monocular SLAM (Monocular Simultaneous Localization and Mapping) unit that estimates at least one of a position and an orientation of the monocular camera and a three-dimensional point of a subject of the monocular camera from the plurality of first monocular images and a second monocular image acquired by the monocular camera translated based on the translation amount, The estimation unit specifies the variation in the length of the optical flow based on a variance value of the length of the optical flow, and if the variance value is less than a threshold, estimates that the movement is an in-place rotation. Estimation device.

2. the control unit determines an upper limit of a movable area of ​​the monocular camera at the start of estimating the motion, and determines the translation amount within a range equal to or less than the upper limit according to a distance to the subject while estimating the motion continues. The estimation device according to claim 1 .

3. When the control unit translates the monocular camera, the control unit translates the monocular camera by translating the moving body in a predetermined direction. The estimation device according to claim 2 .

4. the control unit, at the start of estimating the movement, determines, based on an upper limit of a movable area of ​​the monocular camera, a direction with a larger upper limit as the predetermined direction; The estimation device according to claim 3 .

5. a moving unit that translates a position of the monocular camera mounted on the moving body based on the translation amount, When the control unit translates the monocular camera, the control unit translates the monocular camera by controlling the movement unit. The estimation device according to claim 2 .

6. The estimation unit estimates that the movement is translational when the variance value is equal to or greater than a threshold. The estimation device according to claim 1 .

7. an estimation device estimating whether or not the movement of the moving object is a rotation in place from a plurality of first monocular images acquired by a monocular camera mounted on the moving object; determining a translation amount by which the monocular camera is translated when the estimation device estimates that the movement of the moving object is an in-place rotation from an optical flow that indicates a movement of a feature point of a subject of the monocular camera; the estimation device estimates at least one of a position and an orientation of the monocular camera and a three-dimensional point of a subject of the monocular camera by monocular SLAM (Monocular Simultaneous Localization and Mapping) from the plurality of first monocular images and a second monocular image acquired by the monocular camera translated based on the translation amount; The step of estimating whether or not the rotation is in place includes a step of identifying a variation in the length of the optical flow by a variance value of the length of the optical flow; if the variance is less than a threshold, estimating that the motion is an in-place rotation; Estimation methods including:

8. Computer, an estimation unit that estimates whether or not the movement of the moving object is a rotation on the spot from a plurality of first monocular images acquired by a monocular camera mounted on the moving object; a control unit that determines a translation amount for translating the monocular camera when the movement of the moving object is estimated to be a rotation on the spot from an optical flow that indicates a movement of a feature point of a subject of the monocular camera; and functioning as an SLAM unit that estimates at least one of the position and orientation of the monocular camera and a three-dimensional point of a subject of the monocular camera by monocular SLAM (Monocular Simultaneous Localization and Mapping) from the plurality of first monocular images and a second monocular image acquired by the monocular camera translated based on the translation amount; The estimation unit specifies the variation in the length of the optical flow based on a variance value of the length of the optical flow, and if the variance value is less than a threshold, estimates that the movement is an in-place rotation. program.

Images