Information processing device, information processing method, and program

Abstract

[Problem] To make it possible to reduce the uncertainty of a distance measurement result of a sensor even if the sensor moves. [Solution] Provided is an information processing device comprising a candidate position calculation unit for obtaining a plurality of candidate positions on the basis of first three-dimensional position measurement data obtained using a sensor and a determination unit for using the candidate positions and second three-dimensional position measurement data obtained using the sensor to determine one of the candidate positions to be a determined position.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, sensors (hereinafter also referred to as "distance measurement sensors") capable of measuring the distance to a subject (object surface) (hereinafter also referred to as "distance measurement") have become known. Some distance measurement sensors may produce uncertainty in their distance measurement results due to their measurement principles. As an example, if the distance interval that cannot be distinguished by the distance measurement sensor (i.e., the interval at which uncertainty occurs in the distance measurement result) is 10 m, the distance measurement results of subjects located at distances of 1 m, 11 m, 21 m, and so on from the distance measurement sensor cannot be distinguished, and the distances to these subjects will all be measured as 1 m.

[0003] An example of a distance measurement sensor that generates uncertainty in distance measurement results is the iToF (indirect time-of-flight) camera. The iToF camera performs distance measurement by intensity-modulating light emission, irradiating the subject with the intensity-modulated light, and utilizing the fact that the phase shift between the irradiated light and the reflected light is proportional to the distance to the subject. The phase shift returns to its original state every 360 degrees, which can cause the above-mentioned uncertainty in distance measurement results. The interval between distances that cannot be distinguished is determined by the modulation frequency of the light emission. Various technologies are known to resolve the uncertainty in distance measurement results that occurs in this way (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jongenelen, Adrian PP, et al. "Analysis of errors in time-of-flight range imaging with dual-frequency modulation." [online], IEEE transactions on instrumentation and measurement VOL.60, NO.5. (2011): 1861-1868. [Retrieved October 2, 2020], Internet<https: / / sprg.massey.ac.nz / pdfs / 2011_tIM_1861.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] However, it would be desirable to provide a technique that can reduce the uncertainty in the distance measurement results of a sensor even when the sensor is moving. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided an information processing device comprising: a candidate position calculation unit that obtains a plurality of candidate positions based on first measurement data of a three-dimensional position obtained by a sensor; and a determination unit that determines one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional position obtained by the sensor.

[0007] According to another aspect of the present disclosure, there is provided an information processing method comprising: a processor obtaining a plurality of candidate positions based on first measurement data of a three-dimensional position obtained by a sensor; and determining one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional position obtained by the sensor.

[0008] According to another aspect of the present disclosure, there is provided a program that causes a computer to function as an information processing device that includes a candidate position calculation unit that obtains multiple candidate positions based on first measurement data of three-dimensional positions obtained by a sensor, and a determination unit that determines one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional positions obtained by the sensor. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an information processing system according to a first embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating pose estimation using SLAM. [Figure 3] FIG. 1 is a diagram illustrating an overview of the P3P-RANSAC process. [Figure 4] FIG. 1 is a diagram illustrating an example of the operation of P3P-RANSAC. [Figure 5] 10A and 10B are diagrams for explaining a method for reducing uncertainty in a distance measurement result according to the embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a 3D / 2D list after outlier rejection. [Figure 7] FIG. 10 is a diagram illustrating an example of an operation for resolving ambiguity in ranging according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment of the present disclosure. [Figure 9] 10A and 10B are diagrams for explaining a position determination method according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the operation of the information processing system according to the embodiment. [Figure 11] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] Furthermore, in this specification and drawings, multiple components having substantially the same or similar functional configurations may be distinguished by adding different numbers after the same reference numeral. However, if there is no need to particularly distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numeral will be used. Furthermore, similar components in different embodiments may be distinguished by adding different letters after the same reference numeral. However, if there is no need to particularly distinguish between similar components, only the same reference numeral will be used.

[0012] The explanation will be given in the following order. 0. Overview 1. First embodiment 1.1. Example of functional configuration 1.2.SLAM Pose Estimation 1.3.Resolving Ranging Ambiguities 1.4. Range measurement ambiguity resolution operation 2. Second embodiment 2.1. Example of functional configuration 2.2. Example of operation 3. Hardware configuration example 4. Summary

[0013] <0. Overview> First, an overview of an embodiment of the present disclosure will be described. Distance measurement sensors have become known in recent years. Some distance measurement sensors may cause uncertainty in the distance measurement results due to their measurement principles. Various techniques are known to eliminate uncertainty in the distance measurement results (see, for example, Non-Patent Document 1). According to such techniques, by using two modulation frequencies for intensity modulation instead of one, it is possible to widen the interval between indistinguishable distances (the interval between which uncertainty occurs in the distance measurement results).

[0014] However, this technology requires superimposing multiple images acquired by the iToF camera based on different modulation frequencies. Therefore, the iToF camera must be kept stationary. If the iToF camera is moving (i.e., if at least one of the position and orientation of the iToF camera is changing), the multiple images cannot be superimposed, and the ranging results may become unstable (such as motion blur).

[0015] In the embodiments of the present disclosure, it is assumed that the iToF camera is moving. Therefore, if such a technology is applied when the iToF camera itself is moving, the distance measurement accuracy will decrease. Therefore, in the embodiments of the present disclosure, a technology that can reduce the uncertainty of the distance measurement results by the iToF camera even when the iToF camera is moving will be mainly described.

[0016] More specifically, according to an embodiment of the present disclosure, an information processing device is provided, which includes a candidate position calculation unit that obtains multiple candidate positions based on first measurement data of three-dimensional positions obtained by an iToF camera, and a determination unit that determines one of the multiple candidate positions calculated by the candidate position calculation unit as a determined position based on the multiple candidate positions calculated by the candidate position calculation unit and second measurement data of three-dimensional positions obtained by the iToF camera.

[0017] According to this configuration, even when the iToF camera moves, it is possible to reduce the uncertainty of the distance measurement results by the iToF camera. Hereinafter, a first example of this configuration will be described as a "first embodiment," and a second example of this configuration will be described as a "second embodiment." Note that the iToF camera is an example of a sensor that measures the distance to a subject (object surface). Therefore, as will be described later, other sensors capable of measuring the distance to a subject may be used instead of the iToF camera.

[0018] The outline of the embodiments of the present disclosure has been described above.

[0019] <1. First embodiment> Next, a first embodiment of the present disclosure will be described.

[0020] (1.1. Example of functional configuration) First, a functional configuration example of an information processing system according to a first embodiment of the present disclosure will be described. Fig. 1 is a diagram showing a functional configuration example of the information processing system according to the first embodiment of the present disclosure. As shown in Fig. 1, the information processing system 1 according to the first embodiment of the present disclosure includes an information processing device 10, an iToF camera 20, a pose observation utilization unit 30, and a ranging observation utilization unit 40.

[0021] (iToF camera 20) The iToF camera 20 is a distance measurement sensor capable of measuring (ranging) the distance to a subject (object surface) and obtaining a ranging result. The iToF camera 20 intensity-modulates light emission, irradiates the intensity-modulated light, and measures the distance to the subject (the three-dimensional position of the object surface) based on the phase shift between the irradiated light and the light reflected by the object surface, which is proportional to the distance to the subject. However, as mentioned above, the phase shift returns to its original state every 360 degrees, which can cause uncertainty in the ranging result. The interval of indistinguishable distances is determined by the modulation frequency of the light emission.

[0022] The iToF camera 20 outputs the distance measurement results to the information processing device 10. More specifically, the distance measurement results output from the iToF camera 20 to the information processing device 10 may be a two-dimensional image in which the distance measurement results to the subject are arranged for each pixel. As described above, the iToF camera 20 is an example of a sensor that measures the distance to the subject. Therefore, instead of the iToF camera 20, another sensor that can measure the distance to the subject (which may cause uncertainty in the distance measurement results) may be used. Furthermore, the iToF camera 20 outputs the brightness of the reflected light as an image (brightness image) to the information processing device 10. As will be described later, the brightness image can be used to obtain two-dimensional positions that are observation positions of feature points. Note that the iToF camera 20 may be incorporated into the information processing device 10.

[0023] (Information processing device 10) The information processing device 10 estimates the position and attitude of the iToF camera 20 based on the ranging results output by the iToF camera 20. The position and attitude of the iToF camera 20 may correspond to the pose of the iToF camera 20. The information processing device 10 outputs the estimated position and attitude (pose) of the iToF camera 20 to the pose observation utilization unit 30. Furthermore, the information processing device 10 reduces the uncertainty of the ranging results output by the iToF camera 20. The information processing device 10 outputs the ranging results with reduced uncertainty to the ranging observation utilization unit 40.

[0024] The information processing device 10 includes a candidate position calculation unit 12, a motion estimation unit 13 (position and orientation estimation unit), and a position determination unit 14. The motion estimation unit 13 and the position determination unit 14 may constitute the determination unit described above. Note that detailed functions of the candidate position calculation unit 12, the motion estimation unit 13, and the position determination unit 14 will be described later.

[0025] The information processing device 10 may be configured with, for example, one or more CPUs (Central Processing Units). When the information processing device 10 is configured with a processor such as a CPU, the processor may be configured with an electronic circuit. ,Ko This can be realized by executing a program that causes a computer to function as the information processing device 10.

[0026] Additionally, the information processing device 10 includes a memory (not shown). The memory (not shown) is a recording medium that stores programs executed by the information processing device 10 and stores data necessary for executing these programs. The memory (not shown) also temporarily stores data for calculations by the information processing device 10. The memory (not shown) is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0027] (Pose Observation Utilization Section 30) The pose observation utilization unit 30 utilizes the position and posture (pose) of the iToF camera 20 output by the information processing device 10. More specifically, the pose observation utilization unit 30 utilizes the position and posture (pose) of the iToF camera 20 estimated by the motion estimation unit 13 in the information processing device 10. Note that the pose observation utilization unit 30 may be incorporated into the information processing device 10.

[0028] (Ranging Observation and Utilization Division 40) The ranging observation utilization unit 40 utilizes the ranging results with reduced uncertainty output by the information processing device 10. More specifically, the ranging observation utilization unit 40 utilizes the ranging results with reduced uncertainty output by the position determination unit 14 in the information processing device 10. The ranging observation utilization unit 40 may be incorporated into the information processing device 10.

[0029] An example of the functional configuration of the information processing system 1 according to the first embodiment of the present disclosure has been described above.

[0030] (1.2. Pose estimation using SLAM) The information processing device 10 according to the first embodiment of the present disclosure reduces uncertainty in the distance measurement results based on a combination of the distance measurement results obtained by the iToF camera 20 and a technology called SLAM (Simultaneous Localization and Mapping). SLAM estimates the position and orientation of the camera in a global coordinate system linked to the real space and creates an environmental map of the camera's surroundings in parallel.

[0031] More specifically, SLAM sequentially estimates the 3D shape of a subject based on images acquired by a camera. At the same time, SLAM estimates information indicating relative changes in the camera's position and orientation (camera motion) as self-position information (translation component) and self-orientation information (rotation component) based on the images acquired by the camera. By associating the 3D shape with the self-position information and self-orientation information, SLAM can simultaneously create a map of the surrounding environment and estimate the camera's position and orientation (pose) in that environment.

[0032] An overview of pose estimation using SLAM will be described below with reference to FIGS.

[0033] Figure 2 is a diagram for explaining pose estimation using SLAM. Referring to Figure 2, an object exists in real space, and the three-dimensional positions (x1, y1, z1) to (x7, y7, z7) of multiple feature points of the object are shown. (x1, y1, z1) to (x7, y7, z7) are three-dimensional positions in the global coordinate system and do not change with camera movement.

[0034] Furthermore, referring to FIG. 2, two-dimensional images G0 to G2 obtained in time series by a camera are shown. Each of the two-dimensional images G0 to G2 captures a feature point. Note that, although the example shown in FIG. 2 shows seven feature points, the number of feature points captured in each of the two-dimensional images G0 to G2 is not limited. In the two-dimensional image G2, the observation positions at which each feature point is captured are shown as two-dimensional positions (u1, v1) to (u7, v7). The two-dimensional positions (u1, v1) to (u7, v7), which are the observation positions, can change depending on the movement of the camera.

[0035] The three-dimensional position (x1, y1, z1) and the two-dimensional position (u1, v1) correspond to each other because they are the positions of the same feature point. Similarly, the three-dimensional position (x2, y2, z2) and the two-dimensional position (u2, v2) correspond to each other, and so do the three-dimensional positions (x7, y7, z7) and the two-dimensional position (u7, v7). Based on a 3D / 2D list in which such three-dimensional positions are associated with two-dimensional positions, SLAM estimates the camera's motion (translation component t and rotation component r) from a certain reference point in time to the point in time when the two-dimensional image G2 is obtained as the camera's position and orientation (pose).

[0036] The problem of estimating the position and orientation of a camera based on the three-dimensional positions of n points in a global coordinate system and the two-dimensional positions of those points in an observed image is known as the PnP problem (Perspective-n-Points Problem).

[0037] Hereinafter, in the 3D / 2D list, a pair of associated three-dimensional positions and two-dimensional positions will also be referred to as an "entry." Herein, the 3D / 2D list may include entries in which the association between the three-dimensional position and the two-dimensional position is correct (hereinafter also referred to as "inliers"), but may also include entries in which the association between the three-dimensional position and the two-dimensional position is incorrect (hereinafter also referred to as "outliers"). SLAM may also perform a process of rejecting outliers from the 3D / 2D list.

[0038] P3P-RANSAC (Perspective 3 Point Random Sample Concensus) is known as an example of an algorithm for discarding outliers from a 3D / 2D list. An overview of the P3P-RANSAC process will be explained with reference to Fig. 3.

[0039] (P3P-RANSAC processing overview) FIG. 3 is a diagram for explaining an overview of the P3P-RANSAC process. Referring to FIG. 3, a 3D / 2D list is shown. In the P3P-RANSAC process, the 3D / 2D list is acquired. Then, a selection process is executed to randomly select three entries from the 3D / 2D list, and a generation process is executed to generate a motion hypothesis (translation component t and rotation component r) based on the three-dimensional positions included in the three entries. This generates a motion hypothesis.

[0040] 3, lines connect the motion hypotheses and the three-dimensional positions used to generate them. As an example, the example shows a motion hypothesis (t1, r1) being generated based on a three-dimensional position (x1, y1, z1) and a two-dimensional position (u1, v1), a three-dimensional position (x3, y3, z3) and a two-dimensional position (u3, v3), and a three-dimensional position (x6, y6, z6) and a two-dimensional position (u6, v6). Next, the projection position of the three-dimensional position (x1, y1, z1) included in the 3D / 2D list onto the two-dimensional image corresponding to the motion hypothesis (t1, r1) is calculated, and the distance between each projection position and the two-dimensional position (u1, v1) (observation position) corresponding to the three-dimensional position (x1, y1, z1) is calculated.If the distance is below a threshold, a vote (predetermined vote) indicating a circle is cast for the motion hypothesis (t1, r1), and if the distance is greater than or equal to the threshold, a vote indicating an cross is cast for the motion hypothesis (t1, r1).

[0041] Such voting is performed for all entries included in the 3D / 2D list. As an example, for a motion hypothesis (t1, r1), a vote indicating "yes" is made from entries consisting of a three-dimensional position (x1, y1, z1) and a two-dimensional position (u1, v1), a vote indicating "x" is made from entries consisting of a three-dimensional position (x2, y2, z2) and a two-dimensional position (u2, v2), and a vote indicating "yes" is made from entries consisting of a three-dimensional position (x7, y7, z7) and a two-dimensional position (u7, v7). In a similar manner, an example is shown in which motion hypotheses (t2, r2) to (t100, r100) are generated. Here, the upper limit of the number of motion hypotheses to be generated is set to 100.

[0042] Next, the motion hypothesis with the largest number of votes indicating "yes" (number of votes) is selected from among the motion hypotheses (t1, r1) to (t100, r100). In the example shown in Figure 3, the number of votes indicating "yes" for the motion hypothesis (t1, r1) was six, and the motion hypothesis (t1, r1) is the motion hypothesis with the largest number of votes. Therefore, the motion hypothesis (t1, r1) is selected (shown as "Winner" in Figure 3).

[0043] An entry that votes "x" for the motion hypothesis (t1, r1) selected in this way is determined to be an outlier. As an example, an entry determined to be an outlier is rejected from the 3D / 2D list. On the other hand, an entry that votes "y" for the motion hypothesis (t1, r1) selected in this way is determined to be an inlier and remains in the 3D / 2D list.

[0044] In the example shown in Figure 3, the only entry that voted "x" for the selected motion hypothesis (t1, r1) is the entry consisting of a 3D position (x2, y2, z2) and a 2D position (u2, v2). Therefore, only this entry is rejected from the 3D / 2D list as an outlier, and the other entries remain in the 3D / 2D list as inliers.

[0045] As an example, the selected motion hypothesis (t1, r1) is output as the position and posture (pose) of the camera 20. Furthermore, the 3D / 2D list in which the outliers are discarded and the inliers are left is output as information in which the 3D position of each feature point is associated with the observed position in the 2D image.

[0046] (Example of P3P-RANSAC in action) Figure 4 shows the operation of P3P-RANSAC. For example: 4. As shown in FIG. 4, in P3P-RANSAC, a 3D / 2D list is obtained. The 3D / 2D list includes three-dimensional positions (x1, y1, z1) for which ambiguity is to be resolved. Next, three entries are randomly selected from the 3D / 2D list (S11). Then, motion hypotheses are generated based on the three selected entries (S12). First, a motion hypothesis (t1, r1) is generated.

[0047] Next, the projection position of the three-dimensional position included in the 3D / 2D list onto the two-dimensional image corresponding to the motion hypothesis is calculated (S13). First, the projection position of the three-dimensional position (x1, y1, z1) included in the 3D / 2D list onto the two-dimensional image corresponding to the motion hypothesis (t1, r1) is calculated. Next, the distance between the two-dimensional position (observation position) corresponding to the three-dimensional position and the projection position is calculated, and if the distance is below a threshold, a vote (predetermined vote) indicating a ◯ is made for the motion hypothesis, and if the distance is equal to or greater than the threshold, a vote indicating an × is made for the motion hypothesis (S14, S15).

[0048] First, the distance between the two-dimensional position (u1, v1) (observation position) corresponding to the three-dimensional position (x1, y1, z1) and the projection position is calculated, and if it is determined that this distance is below a threshold, a vote (predetermined vote) indicating "O" is made for the motion hypothesis (t1, r1). If there are entries for which voting has not been completed ("NO" in S16), the operation proceeds to S13. On the other hand, if voting has been completed for all entries ("YES" in S16), the operation proceeds to S17. If voting has been completed for the motion hypothesis (t1, r1) from all entries included in the 3D / 2D list, the operation proceeds to S17.

[0049] If the number of motion hypotheses for which voting has been completed has not reached the upper limit ("NO" in S17), the operation proceeds to S11. On the other hand, if the number of motion hypotheses for which voting has been completed has reached the upper limit ("NO" in S17), the operation proceeds to S11. YES "), the operation proceeds to S18. Specifically, when voting for the motion hypotheses (t1, r1) to (t100, r100) is completed, the operation proceeds to S18.

[0050] Next, among the motion hypotheses (t1, r1) to (t100, r100), the motion hypothesis with the largest number of votes indicating ◯ (number of votes) is adopted (S18). In the example shown in FIG. 3, the motion hypothesis (t1, r1) has the largest number of votes, so the motion hypothesis (t1, r1) is adopted. As an example, the selected motion hypothesis (t1, r1) is output to the pose observation utilization unit 30 as the position and attitude (pose) of the iToF camera 20.

[0051] Then, an entry that voted "x" for the motion hypothesis (t1, r1) selected in this way is determined to be an outlier. As an example, an entry determined to be an outlier is rejected from the 3D / 2D list. On the other hand, an entry that voted "o" for the motion hypothesis (t1, r1) selected in this way is determined to be an inlier and remains in the 3D / 2D list (S19).

[0052] The outline of pose estimation using SLAM has been explained above with reference to FIGS.

[0053] (1.3. Resolution of Ranging Ambiguity) As described above, the information processing device 10 according to the first embodiment of the present disclosure reduces the uncertainty of the ranging result based on a combination of the ranging result obtained by the iToF camera 20 and SLAM. More specifically, the information processing device 10 according to the first embodiment of the present disclosure reduces the uncertainty of the ranging result obtained by the iToF camera 20 in the above-described P3P-RANSAC processing. A method for reducing the uncertainty of the ranging result will be described below.

[0054] In the first embodiment of the present disclosure, an iToF camera 20 is used as the camera. At this time, the distance measurement results (measurement data) of each feature point obtained by the iToF camera 20 may be used as the three-dimensional positions (x1, y1, z1) to (x7, y7, z7) of each feature point. Meanwhile, the two-dimensional positions (u1, v1) to (u7, v7), which are the observation positions of each feature point, are obtained from the luminance image output from the iToF camera 20. A 3D / 2D list is created in which the three-dimensional positions and two-dimensional positions obtained in this way are associated with each other.

[0055] As an example, in the first embodiment of the present disclosure, consider resolving the uncertainty of a three-dimensional position (x1, y1, z1). The three-dimensional position (x1, y1, z1) may be an example of first measurement data. At this time, the candidate position calculation unit 12 acquires the modulation frequency of the irradiated light from the iToF camera 20. Then, the candidate position calculation unit 12 obtains multiple candidate positions based on the modulation frequency of the irradiated light and the three-dimensional position (x1, y1, z1).

[0056] More specifically, the candidate position calculation unit 12 divides the speed of light by the modulation frequency of the irradiated light to calculate the interval d1 of distances that cannot be distinguished by the iToF camera 20 (i.e., the interval at which uncertainty occurs in the distance measurement results). The candidate position calculation unit 12 calculates a candidate position other than (x1, y1, z1) by adding the unit vector of (x1, y1, z1) × the interval d1 × n (n is an integer equal to or greater than 1) to the three-dimensional position (x1, y1, z1).

[0057] Here, it is assumed that n = 1 or 2. That is, it is assumed that the candidate position calculation unit 12 calculates the candidate position (x1', y1', z1') by adding the unit vector of (x1, y1, z1) times the interval d1 × 1 to the three-dimensional position (x1, y1, z1), and calculates the candidate position (x1'', y1'', z1'') by adding the unit vector of (x1, y1, z1) times the interval d1 × 2 to the three-dimensional position (x1, y1, z1).

[0058] As a result, in addition to the candidate position (x1, y1, z1), candidate positions (x1', y1', z1') and (x1'', y1'', z1'') are calculated, resulting in three candidate positions being calculated. However, the number of candidate positions calculated by the candidate position calculation unit 12 is not limited as long as there is more than one. The motion estimation unit 13 adds entries to the 3D / 2D list in which each of the candidate positions (x1, y1, z1), (x1', y1', z1'), (x1'', y1'', z1'') calculated by the candidate position calculation unit 12 is associated with the two-dimensional position (u1, v1) that is the observation position.

[0059] 5 is a diagram for explaining a method for reducing uncertainty in a distance measurement result according to the first embodiment of the present disclosure. Referring to FIG. 5, Place(Entries that associate each of (x1', y1', z1') (x1'', y1'', z1'') with the two-dimensional position (u1, v1) that is the observation position are added to the 3D / 2D list. The 3D / 2D list also includes other entries. The measurement positions (x2, y2, z2) to (x7, y7, z7) may be examples of second measurement data. Note that the second measurement data may include one or more measurement positions (three-dimensional positions).

[0060] In an embodiment of the present disclosure, the motion estimation unit 13 estimates the position and orientation (pose) of the iToF camera 20 based on the candidate positions (x1, y1, z1), (x1', y1', z1'), (x1'', y1'', z1'') and the measurement positions (x2, y2, z2) to (x7, y7, z7) to obtain an estimation result (position and orientation estimation information). Then, the position determination unit 14 determines one of the candidate positions (x1, y1, z1), (x1', y1', z1'), (x1'', y1'', z1'') as a determined position based on the estimation result. This can resolve the uncertainty of the three-dimensional position (x1, y1, z1).

[0061] The motion estimation unit 13 performs a selection process to randomly select a predetermined number of entries from the 3D / 2D list. The predetermined number is not limited to three or more, but the following assumes that the predetermined number is three. As an example, the motion estimation unit 13 may estimate the position and attitude (pose) of the iToF camera 20 based on the three-dimensional positions constituting the selected three entries to obtain an estimation result.

[0062] However, in the first embodiment of the present disclosure, similar to the P3P-RANSAC process, it is mainly assumed that the motion estimation unit 13 executes a selection process of selecting three entries and a generation process of generating motion hypotheses (position and posture generation information) based on the three-dimensional positions constituting the selected three entries multiple times. This generates multiple motion hypotheses. Then, the motion estimation unit 13 selects one motion hypothesis from the multiple motion hypotheses as an estimation result.

[0063] At this time, it is desirable that the motion estimation unit 13 does not select two or more of the three entries constituting the candidate positions (x1, y1, z1) (x1', y1', z1') (x1'', y1'', z1'') as three entries in one selection process. This means that at most one candidate position is used to generate one motion hypothesis, and as will be explained later, it becomes easier to determine one candidate position from the candidate positions (x1, y1, z1) (x1', y1', z1') (x1'', y1'', z1'').

[0064] In the example shown in Figure 5, candidate position (x1, y1, z1) is used to generate motion hypothesis (t1, r1), candidate position (x1', y1', z1') is used to generate motion hypothesis (t2, r2), and candidate position (x1'', y1'', z1'') is used to generate motion hypothesis (t3, r3). That is, candidate positions (x1, y1, z1), (x1', y1', z1'), and (x1'', y1'', z1'') are each used to generate a different motion hypothesis.

[0065] There is no upper limit to the number of motion hypotheses generated by the motion estimation unit 13. Here, it is assumed that the upper limit to the number of motion hypotheses generated by the motion estimation unit 13 is 100. Referring to FIG. 5, an example is shown in which motion hypotheses (t1, r1) to (t100, r100) are generated. For each motion hypothesis, the motion estimation unit 13 calculates the distance between the observation position shown in the two-dimensional image of each of the candidate position and the measurement position and the projected position onto the two-dimensional image corresponding to the motion hypothesis. Then, the motion estimation unit 13 selects a motion hypothesis based on the distance between the observation position and the projected position for each motion hypothesis.

[0066] More specifically, the motion estimation unit 13 calculates a projection position of a three-dimensional position (x1, y1, z1) included in the 3D / 2D list onto a two-dimensional image corresponding to the motion hypothesis (t1, r1). Then, the motion estimation unit 13 calculates the distance between the calculated projection position and a two-dimensional position (u1, v1) (observation position) corresponding to the three-dimensional position (x1, y1, z1). If the distance is below a threshold, the motion estimation unit 13 votes (predetermined vote) to indicate a circle for the motion hypothesis (t1, r1), and if the distance is equal to or greater than the threshold, it votes to indicate an error for the motion hypothesis (t1, r1). This type of voting is performed for all entries included in the 3D / 2D list.

[0067] Next, the motion estimation unit 13 selects the motion hypothesis with the largest number of votes indicating "o" (number of votes) from among the motion hypotheses (t1, r1) to (t100, r100). In the example shown in FIG. 5, the number of votes indicating "o" for the motion hypothesis (t2, r2) is five, and the motion hypothesis (t2, r2) is the motion hypothesis with the largest number of votes. Therefore, the motion hypothesis (t2, r2) is selected (shown as "Winner" in FIG. 5).

[0068] The motion estimation unit 13 then determines entries that have voted "×" for the selected motion hypothesis (t2, r2) as outliers and discards them from the 3D / 2D list. On the other hand, the motion estimation unit 13 determines entries that have voted "〇" for the selected motion hypothesis (t2, r2) as inliers and leaves them in the 3D / 2D list. Furthermore, the motion estimation unit 13 outputs the position and orientation (pose) of the iToF camera 20 to the pose observation utilization unit 30. At this time, the motion estimation unit 13 may output the selected motion hypothesis (t2, r2) itself to the pose observation utilization unit 30. Alternatively, the motion estimation unit 13 may re-estimate the motion hypothesis based on the entry determined to be an inlier and output the resulting pose to the pose observation utilization unit 30. This allows a more accurate pose to be output to the pose observation utilization unit 30.

[0069] In the example shown in FIG. 5, the entries that voted "x" for the selected motion hypothesis (t2, r2) are an entry consisting of a three-dimensional position (x1, y1, z1) and a two-dimensional position (u1, v1), an entry consisting of a three-dimensional position (x1", y1", z1") and a two-dimensional position (u1, v1), and an entry consisting of a three-dimensional position (x6, y6, z6) and a two-dimensional position (u6, v6). Therefore, the motion estimation unit 13 discards these entries as outliers from the 3D / 2D list, and leaves the other entries as inliers in the 3D / 2D list. Note that the motion estimation unit 13 does not need to immediately discard an entry determined as an outlier from the 3D / 2D list. For example, the motion estimation unit 13 may discard an entry from the 3D / 2D list if the number of times it has been determined as an outlier has reached a threshold.

[0070] 6 is a diagram showing an example of a 3D / 2D list after outlier rejection. Referring to FIG. 6, in the 3D / 2D list after outlier rejection, an entry consisting of a three-dimensional position (x1, y1, z1) and a two-dimensional position (u1, v1), an entry consisting of a three-dimensional position (x1'', y1'', z1'') and a two-dimensional position (u1, v1), and an entry consisting of a three-dimensional position (x6, y6, z6) and a two-dimensional position (u6, v6) have been rejected. On the other hand, the determined position (x1', y1', z1') remains in the 3D / 2D list after outlier rejection as an inlier.

[0071] In the example shown in FIG. 5, among the candidate positions (x1, y1, z1), (x1', y1', z1'), and (x1'', y1'', z1''), only the entry constituted by the candidate position (x1', y1', z1') is determined to be an inlier. Therefore, the position determiner 14 may determine the candidate position (x1', y1', z1') constituting the entry determined to be an inlier (i.e., the entry that voted ◯ for the selected motion hypothesis) as the determined position. This can eliminate the uncertainty of the three-dimensional position (x1, y1, z1) obtained by the iToF camera 20. However, the method of determining one candidate position from the candidate positions (x1, y1, z1), (x1', y1', z1'), and (x1'', y1'', z1'') is not limited to this example.

[0072] That is, the position determination unit 14 may determine, as the determined position, a candidate position that satisfies a predetermined condition (hereinafter also referred to as a "determination condition") from among the candidate positions (x1, y1, z1), (x1', y1', z1'), (x1'', y1'', z1''). In this case, the determination condition may include a first condition that the candidate position is included in the three positions used to generate the selected motion hypothesis. Alternatively, the determination condition may include a second condition that a vote indicating "o" is cast for the selected motion hypothesis. Alternatively, the determination condition may include a third condition that the distance between the observed position and the projected position in the selected motion hypothesis is the smallest.

[0073] Alternatively, the decision condition may be the logical product of two or more of these first to third conditions, or the logical sum of two or more of these first to third conditions.

[0074] For example, the position determination unit 14 may determine whether three candidate positions can be narrowed down to one candidate position that satisfies the first condition, and if no candidate position that satisfies the first condition exists, may determine whether three candidate positions can be narrowed down to one candidate position that satisfies the second condition. Note that, as described above, if two or more candidate positions are not used to generate one movement hypothesis, there is no possibility that multiple candidate positions that satisfy the first condition exist.

[0075] Furthermore, if there is no candidate position that satisfies the second condition, the position determination unit 14 may determine, as the determined position, a candidate position that satisfies a third condition from among the three candidate positions. Alternatively, if there are multiple candidate positions that satisfy the second condition, the position determination unit 14 may determine, as the determined position, a candidate position that satisfies the third condition from among the multiple candidate positions that satisfy the second condition.

[0076] The position determination unit 14 outputs the ranging result with reduced uncertainty to the ranging observation utilization unit 40. More specifically, the position determination unit 14 determines the ranging result of the projection position corresponding to the three-dimensional position (x1, y1, z1) for which uncertainty is to be eliminated in the two-dimensional image obtained by the iToF camera 20 as the distance corresponding to the determined position (x1', y1', z1'), and outputs the determined two-dimensional image to the ranging observation utilization unit 40. Note that the distance corresponding to the determined position (x1', y1', z1') is the result of adding the interval d1 × 1 to the length of (x1, y1, z1).

[0077] The determined position (x1', y1', z1') selected from the plurality of candidates as described above may be used by the motion estimation unit 13 to re-estimate the position and attitude (pose) of the iToF camera 20. By using the three-dimensional position (x1', y1', z1') with the uncertainty resolved to re-estimate the pose of the iToF camera 20, the pose of the iToF camera 20 can be estimated with higher accuracy.

[0078] 6 , when the pose of the iToF camera 20 is re-estimated, (x1′, y1′, z1′) remaining in the 3D / 2D list may be reused. Meanwhile, the two-dimensional position (u1, v1) corresponding to (x1′, y1′, z1′) may be updated based on a two-dimensional image re-acquired by the iToF camera 20. Similarly, when the pose of the iToF camera 20 is re-estimated, (x2, y2, z2) to (x5, y5, z5) (x7, y7, z7) remaining in the 3D / 2D list may be reused. Meanwhile, the corresponding two-dimensional positions (u2, v2) to (u5, v5) (u7, v7) may be updated based on a two-dimensional image re-acquired by the iToF camera 20.

[0079] Instead of the entry in which (x6, y6, z6) and (u6, v6) are associated and rejected as an outlier, the reacquired 3D position and 2D position may be added to the 3D / 2D list based on the 2D image reacquired by the iToF camera 20. Based on the 3D / 2D list updated in this way, the pose of the iToF camera 20 may be re-estimated. The pose re-estimation based on the updated 3D / 2D list may be performed in the same manner as the pose estimation described above.

[0080] (1.4. Range measurement ambiguity resolution operation) FIG. 7 is a diagram illustrating an example of an operation for resolving ambiguity in ranging according to the first embodiment of the present disclosure. As illustrated in FIG. 7, the motion estimation unit 13 acquires a 3D / 2D list. The 3D / 2D list includes a three-dimensional position (x1, y1, z1) for which ambiguity is to be resolved. The candidate position calculation unit 12 acquires the modulation frequency of the irradiated light from the iToF camera 20. Then, the candidate position calculation unit 12 acquires candidate positions (x1, y1, z1) (x1', y1', z1') (x1'', y1'', z1'') based on the modulation frequency of the irradiated light and the three-dimensional position (x1, y1, z1).

[0081] The motion estimation unit 13 adds entries to the 3D / 2D list, which are entries based on the uncertainty of the distance measurement by the iToF camera 20, in which each of the candidate positions (x1, y1, z1), (x1', y1', z1'), and (x1'', y1'', z1'') is associated with a two-dimensional position (u1, v1) that is the observation position (S31). The motion estimation unit 13 randomly selects three entries from the 3D / 2D list (S11). The motion estimation unit 13 generates motion hypotheses based on the three selected entries (S12). First, a motion hypothesis (t1, r1) is generated.

[0082] The motion estimation unit 13 calculates the projection position of the three-dimensional position included in the 3D / 2D list onto the two-dimensional image corresponding to the motion hypothesis (S13). First, the projection position of the three-dimensional position (x1, y1, z1) included in the 3D / 2D list onto the two-dimensional image corresponding to the motion hypothesis (t1, r1) is calculated. The motion estimation unit 13 calculates the distance between the two-dimensional position (observation position) corresponding to the three-dimensional position and the projection position, and if the distance is below a threshold, votes (predetermined votes) indicating ◯ for the motion hypothesis, and if the distance is equal to or greater than the threshold, votes indicating × for the motion hypothesis (S14, S15).

[0083] First, the distance between the two-dimensional position (u1, v1) (observation position) corresponding to the three-dimensional position (x1, y1, z1) and the projection position is calculated, and if it is determined that this distance is below a threshold, a vote (predetermined vote) indicating "O" is made for the motion hypothesis (t1, r1). If there are entries for which voting has not been completed ("NO" in S16), the operation proceeds to S13. On the other hand, if voting has been completed for all entries ("YES" in S16), the operation proceeds to S17. If voting has been completed for the motion hypothesis (t1, r1) from all entries included in the 3D / 2D list, the operation proceeds to S17.

[0084] If the number of motion hypotheses for which voting has been completed has not reached the upper limit ("NO" in S17), the operation proceeds to S11. On the other hand, if the number of motion hypotheses for which voting has been completed has reached the upper limit ("NO" in S17), the operation proceeds to S18. Specifically, if voting for motion hypotheses (t1, r1) to (t100, r100) has been completed, the operation proceeds to S18.

[0085] The motion estimation unit 13 adopts the motion hypothesis with the largest number of votes indicating ◯ (number of votes) among the motion hypotheses (t1, r1) to (t100, r100) (S18). In the example shown in Fig. 5, the motion hypothesis (t2, r2) has the largest number of votes, so the motion hypothesis (t2, r2) is adopted.

[0086] The motion estimation unit 13 then determines entries that have voted "×" for the selected motion hypothesis (t2, r2) as outliers and discards them from the 3D / 2D list. Meanwhile, the motion estimation unit 13 determines entries that have voted "〇" for the selected motion hypothesis (t2, r2) as inliers and leaves them in the 3D / 2D list (S19). Furthermore, the motion estimation unit 13 outputs the position and orientation (pose) of the iToF camera 20 to the pose observation utilization unit 30. At this time, the motion estimation unit 13 may output the selected motion hypothesis (t2, r2) itself to the pose observation utilization unit 30. Alternatively, the motion estimation unit 13 may re-estimate the motion hypothesis based on the entry determined to be an inlier and output the resulting pose to the pose observation utilization unit 30. This allows a more accurate pose to be output to the pose observation utilization unit 30.

[0087] The position determination unit 14 determines a candidate position that satisfies the determination conditions from among the candidate positions (x1, y1, z1), (x1', y1', z1'), and (x1'', y1'', z1'') as the determined position. This can eliminate the uncertainty of the three-dimensional position (x1, y1, z1) (S32). The position determination unit 14 outputs the ranging result with reduced uncertainty to the ranging observation utilization unit 40.

[0088] The first embodiment of the present disclosure has been described above.

[0089] 2. Second embodiment Next, a second embodiment of the present disclosure will be described.

[0090] (2.1. Example of functional configuration) First, a functional configuration example of an information processing system according to a second embodiment of the present disclosure will be described. FIG. 8 is a diagram illustrating a functional configuration example of an information processing system according to the second embodiment of the present disclosure. As shown in FIG. 8, the information processing system 2 according to the second embodiment of the present disclosure includes an information processing device 50, a rigid structure 60, a pose observation utilization unit 30, and a ranging observation utilization unit 40. The rigid structure 60 includes an RGB camera 70 and an iToF camera 20. Note that instead of the RGB camera 70, the rigid structure 60 may include another camera (e.g., a grayscale camera) configured to be able to acquire its own position and orientation.

[0091] Here, the iToF camera 20, pose observation utilization unit 30, and ranging observation utilization unit 40 according to the second embodiment of the present disclosure have the same functions as the iToF camera 20, pose observation utilization unit 30, and ranging observation utilization unit 40 according to the first embodiment of the present disclosure. Therefore, in the second embodiment of the present disclosure, detailed description of these will be omitted, and the RGB camera 70 and the information processing device 50 will be mainly described.

[0092] (RGB camera 70) The RGB camera 70 is configured to be able to acquire its own position and orientation. Here, the RGB camera 70 and the iToF camera 20 are included in the same rigid body structure. Therefore, the position and orientation of the RGB camera 70 have a fixed, constant relationship with the position and orientation of the iToF camera 20. In other words, the position and orientation of the RGB camera 70 and the position and orientation of the iToF camera 20 have a relationship in which one position and orientation can be easily calculated from the other position and orientation. As an example, the RGB camera 70 may output its own position and orientation to the information processing device 50. At this time, the information processing device 50 may calculate the position and orientation of the iToF camera 20 based on the position and orientation of the RGB camera 70. Alternatively, the position and orientation of the iToF camera 20 calculated based on the position and orientation of the RGB camera 70 by the rigid body structure 60 may be output to the information processing device 50.

[0093] Here, it is mainly assumed that the RGB camera 70 outputs the position and orientation (first position and orientation information) of the iToF camera 20 at time 1 (first time) to the information processing device 50. It is also mainly assumed that the RGB camera 70 outputs the position and orientation (second position and orientation information) of the iToF camera 20 at time 0 (second time), which is different from time 1 (first time), to the information processing device 50. Here, it is assumed that time 0 (second time) is earlier than time 1 (first time).

[0094] (information processing device 50) The information processing device 50 includes a candidate position calculation unit 52, a motion estimation unit 53 (position and orientation acquisition unit), and a position determination unit 54. The detailed functions of the candidate position calculation unit 52, the motion estimation unit 53, and the position determination unit 54 will be described later.

[0095] The information processing device 50 may be configured with, for example, one or more CPUs (Central Processing Units). When the information processing device 50 is configured with a processor such as a CPU, the processor may be configured with an electronic circuit. ,Ko This can be realized by executing a program that causes a computer to function as the information processing device 50.

[0096] Additionally, the information processing device 50 includes a memory (not shown). The memory (not shown) is a recording medium that stores programs executed by the information processing device 50 and stores data necessary for executing these programs. The memory (not shown) also temporarily stores data for calculations by the information processing device 50. The memory (not shown) is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0097] The motion estimation unit 53 acquires the position and orientation (pose) of the iToF camera 20 at time 1 (first time) from the RGB camera 70, and also acquires the position and orientation of the iToF camera 20 at time 0 (second time). The motion estimation unit 53 outputs the position and orientation of the iToF camera 20 at each time to the pose observation utilization unit 30. Note that here, it is mainly assumed that the motion estimation unit 53 acquires the position and orientation of the iToF camera 20 from an external source. However, the method by which the motion estimation unit 53 acquires the position and orientation of the iToF camera 20 is not limited to this example.

[0098] For example, similar to the motion estimation unit 13 according to the first embodiment of the present disclosure, the motion estimation unit 53 may estimate the position and orientation of the iToF camera 20 by combining the ranging results of the iToF camera 20 with SLAM, or may estimate the position and orientation of the iToF camera 20 by SLAM using another method. Alternatively, the motion estimation unit 53 may estimate the position and orientation of the iToF camera 20 by a method other than SLAM.

[0099] The candidate position calculation unit 52 acquires a ranging result (two-dimensional image) obtained by the iToF camera 20 at time 1 (first time). The candidate position calculation unit 52 also acquires a three-dimensional position (first measurement data) of a certain point from the ranging result obtained at time 1 (first time). The candidate position calculation unit 52 acquires multiple candidate positions at time 1 (first time) based on the three-dimensional position of the point. The method for acquiring multiple candidate positions according to the second embodiment of the present disclosure is the same as the method for acquiring multiple candidate positions according to the first embodiment of the present disclosure. The candidate position calculation unit 52 also acquires a ranging result (second measurement data) obtained by the iToF camera 20 at time 0 (second time).

[0100] The position determination unit 54 determines one candidate position from the plurality of candidate positions as a determined position based on the plurality of candidate positions at time 1 (first time), the position and orientation of the iToF camera 20 at time 1 (first time) acquired by the motion estimation unit 53, the ranging result obtained at time 0 (second time), and the position and orientation of the iToF camera 20 at time 0 (second time) acquired by the motion estimation unit 53. Hereinafter, a position determination method according to the second embodiment of the present disclosure will be described with reference to FIG.

[0101] 9 is a diagram for explaining a position determination method according to the second embodiment of the present disclosure. Referring to FIG. 9, the position (translation component) and orientation (rotation component) of the iToF camera 20 at time 0 (second time) are shown as (t0, r0). Also, the position (translation component) and orientation (rotation component) of the iToF camera 20 at time 1 (first time) are shown as (t1, r1). Also referring to FIG. 9, objects B1 and B2 exist in real space. Object B1 is a pillar, and object B2 is a wall, but the types of objects are not limited.

[0102] When the pose of the iToF camera 20 is (t1, r1), the three-dimensional position C1 of a point on the surface of the object B1 is obtained as the distance measurement result. On the other hand, when the pose of the iToF camera 20 is (t0, r0), the three-dimensional position E11 of a point on the surface of the object B1 is obtained as the distance measurement result. Furthermore, when the pose of the iToF camera 20 is (t0, r0), the three-dimensional positions E31 and E21, which are closer than the three-dimensional positions E22 and E32 of a point on the surface of the object B2, are obtained as the distance measurement result.

[0103] The candidate position calculation unit 52 obtains this three-dimensional position C1 as one of the candidate positions (first candidate position), and also obtains three-dimensional positions C2 and C3 as other candidate positions (first candidate positions) based on this three-dimensional position C1. That is, the candidate position calculation unit 52 obtains the candidate positions C1 to C3 (first candidate positions).

[0104] The position determination unit 54 calculates a projection position m1 onto a two-dimensional image corresponding to the pose (t0, r0) of the iToF camera 20 at the candidate position C1. Then, the position determination unit 54 obtains a distance measurement result E11 at the projection position m1 of the iToF camera 20 at the pose (t0, r0). The candidate position calculation unit 52 obtains candidate positions E11 to E13 (second candidate positions) using a similar method based on the distance measurement result E11.

[0105] The position determination unit 54 calculates a projection position m2 onto a two-dimensional image corresponding to the pose (t0, r0) of the iToF camera 20 at candidate position C2. Then, the position determination unit 54 obtains a distance measurement result E21 at the projection position m2 of the iToF camera 20 at the pose (t0, r0). The candidate position calculation unit 52 obtains candidate positions E21 to E23 (second candidate positions) using a similar method based on the distance measurement result E21.

[0106] The position determination unit 54 calculates a projection position m3 onto a two-dimensional image corresponding to the pose (t0, r0) of the iToF camera 20 at candidate position C3. Then, the position determination unit 54 obtains a distance measurement result E31 at the projection position m3 of the iToF camera 20 at the pose (t0, r0). The candidate position calculation unit 52 obtains candidate positions E31 to E33 (second candidate positions) using a similar method based on the distance measurement result E31.

[0107] The position determination unit 54 determines one candidate position from candidate positions C1 to C3 as a determined position based on candidate positions C1 to C3 and candidate positions E11 to E13, E21 to E23, and E31 to E33. More specifically, the position determination unit 54 calculates the distance between candidate position C1 and each of candidate positions E11 to E13, calculates the distance between candidate position C2 and each of candidate positions E21 to E23, and calculates the distance between candidate position C3 and each of candidate positions E31 to E33. The position determination unit 54 determines one candidate position from candidate positions C1 to C3 as a determined position based on these distances.

[0108] More specifically, the position determination unit 54 may determine, as the determined position, the candidate position from among the candidate positions C1 to C3 that has the smallest calculated distance. In the example shown in FIG. 9, the candidate position E11 has the smallest distance from the candidate position C1, the candidate position E22 has the smallest distance from the candidate position C2, and the candidate position E33 has the smallest distance from the candidate position C3. Of these, the distance between the candidate position C1 and the candidate position E11 is the smallest. Therefore, the position determination unit 54 may determine, as the determined position, the candidate position C1 that has the smallest calculated distance. This may eliminate the uncertainty of the three-dimensional position C1.

[0109] The position determination unit 54 outputs the ranging result with reduced uncertainty to the ranging observation utilization unit 40. More specifically, the position determination unit 54 determines the ranging result of the projection position m1 corresponding to the three-dimensional position C1, which is the target of uncertainty elimination in the two-dimensional image obtained by the iToF camera 20, to the distance corresponding to the determined position C1, and outputs the determined two-dimensional image to the ranging observation utilization unit 40. Note that the distance corresponding to the determined position C1 is the length of (x1, y1, z1) itself, so there is no need to particularly change the ranging result of the projection position m1.

[0110] However, if all combinations of uncertain candidate positions are covered, the amount of calculation becomes enormous. Therefore, the space is divided into multiple voxels, and the distance measurement by the iToF camera 20 is performed. and ,The computational complexity can be reduced by combining occupancy ,map techniques such as voting on a voxel grid.

[0111] (2.2. Example of operation) Next, an operation example of the information processing system 2 according to the second embodiment of the present disclosure will be described. Fig. 10 is a diagram showing an operation example of the information processing system 2 according to the second embodiment of the present disclosure. As shown in Fig. 10, in the information processing system 2 according to the second embodiment of the present disclosure, the candidate position calculation unit 52 acquires a ranging result (two-dimensional image) by the iToF camera 20, and the motion estimator 53 acquires a pose of the iToF camera 20.

[0112] The candidate position calculation unit 52 obtains multiple candidate positions based on the ranging uncertainty at the pose (t1, r1) of the iToF camera 20 (i.e., at time 1). The position determination unit 54 selects one candidate position from the multiple candidate positions C1 to C3 (S41). First, candidate position C1 is selected. The position determination unit 54 calculates the projection position of the selected candidate position onto the two-dimensional image corresponding to the pose (t0, r0) of the iToF camera 20 (S42). First, projection position m1 is calculated.

[0113] The candidate position calculation unit 52 obtains a plurality of candidate positions corresponding to the pose (t0, r0) of the iToF camera 20 at the projection position (i.e., corresponding to time 0). Initially, candidate positions E11 to E13 corresponding to the pose (t0, r0) of the iToF camera 20 at the projection position m1 are obtained. Then, the position determination unit 54 selects one candidate position from the plurality of candidate positions (S43). Initially, candidate position E11 is selected.

[0114] The position determination unit 54 calculates the degree of match (i.e., distance) between the selected candidate positions (S44). First, the degree of match between candidate position C1 and candidate position E11 is calculated. If the calculation of the degree of match for all candidate positions corresponding to the pose (t0, r0) of the iToF camera 20 (i.e., corresponding to time 0) has not been completed ("NO" in S45), the operation proceeds to S43. On the other hand, if the position determination unit 54 has finished calculating the degree of match for all candidate positions corresponding to the pose (t0, r0) of the iToF camera 20 ("YES" in S45), the operation proceeds to S46.

[0115] Specifically, when the calculation of the degree of match between candidate position C1 and candidate position E11 is completed, the calculation of the degree of match between candidate position C1 and candidate position E12 is completed, and the calculation of the degree of match between candidate position C1 and candidate position E13 is completed, the operation proceeds to S46. Next, the position determination unit 54 determines the pair of candidate positions with the highest degree of match (i.e., the pair of candidate positions with the shortest distance) (S46). Initially, the pair of candidate position C1 and candidate position E11 is determined as the pair with the highest degree of match.

[0116] If the calculation of the degree of coincidence for all candidate positions in the pose (t1, r1) of the iToF camera 20 (i.e., at time 1) has not been completed (NO in S47), the operation proceeds to S41. On the other hand, if the position determination unit 54 has completed the calculation of the degree of coincidence for all candidate positions corresponding to the pose (t1, r1) of the iToF camera 20 (YES in S47), the operation proceeds to S48.

[0117] Specifically, when the pair of candidate positions C2 and E22 is determined as the pair with the highest degree of match, and the pair of candidate positions C3 and E33 is determined as the pair with the highest degree of match, the operation proceeds to S48. The position determination unit 54 determines the pair of candidate positions with the highest degree of match (i.e., the pair of candidate positions with the smallest distance) (S46).

[0118] Specifically, among the pair of candidate positions C1 and E11, the pair of candidate positions C2 and E22, and the pair of candidate positions C3 and E33, the pair of candidate positions C1 and E11 is determined as the pair of candidate positions with the highest degree of matching.

[0119] If processing has not been completed for all pixels of the two-dimensional image ("NO" in S49), the operations from S41 onwards are executed again for the next pixel. On the other hand, if processing has been completed for all pixels of the two-dimensional image ("YES" in S49), the ranging result with the uncertainty resolved is output to the ranging observation utilization unit 40.

[0120] An example of the operation of the information processing system 2 according to the second embodiment of the present disclosure has been described above.

[0121] <3. Hardware configuration example> Next, a hardware configuration example of an information processing device 900 as an example of the information processing device 10 according to the first embodiment of the present disclosure and the information processing device 50 according to the second embodiment of the present disclosure will be described with reference to Fig. 11. Fig. 11 is a block diagram showing a hardware configuration example of the information processing device 900. Note that the information processing device 10 and the information processing device 50 do not necessarily have to have all of the hardware configuration shown in Fig. 11, and some of the hardware configuration shown in Fig. 11 may not be present in the information processing device 10 and the information processing device 50.

[0122] 11, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. The information processing device 900 may include a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

[0123] The CPU 901 functions as an arithmetic processing unit and control unit, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. The CPU 901, the ROM 903, and the RAM 905 are interconnected by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0124] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929, such as a mobile phone, that operates the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data and instruct the information processing device 900 to perform processing operations. The imaging device 933, described below, may also function as an input device by capturing images of the user's hand movements, fingers, etc. In this case, the pointing position may be determined based on the hand movements and finger orientations.

[0125] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a light for illuminating the surroundings.

[0126] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0127] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0128] The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0129] The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB). The communication device 925 may also be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0130] <4. Summary> According to the embodiments of the present disclosure, it is expected that the availability of SLAM using distance measurement results obtained by an iToF camera as input will be improved. As an example, it is expected that constraints imposed on the operating environment of the iToF camera will be relaxed. For example, a constraint imposed on the operating environment of the iToF camera is that an object to be measured by the iToF camera must be located within a certain distance from the iToF camera.

[0131] It is also expected that the success rate of pose estimation using SLAM, which uses the ranging results obtained by the iToF camera as input, will increase, and the accuracy of pose estimation will improve. Furthermore, it is expected that the robustness of pose estimation for iToF cameras that perform high-speed movement will improve (compared to the Dual-modulation iToF described in Non-Patent Document 1 above).

[0132] Furthermore, according to the embodiments of the present disclosure, it is expected that the accuracy of distance measurement by the iToF camera will be improved. For example, by eliminating the uncertainty of distance measurement by the iToF camera, it is expected that the distance measurement range by the iToF camera will be expanded. Furthermore, it is expected that the robustness of distance measurement by the iToF camera, which moves at high speeds, will be improved (compared to the dual-modulation iToF described in Non-Patent Document 1 mentioned above).

[0133] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0134] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0135] The first embodiment of the present disclosure and the second embodiment of the present disclosure have been described separately above. However, the first embodiment of the present disclosure and the second embodiment of the present disclosure may be combined as appropriate. More specifically, the elimination of uncertainty in the distance measurement result by the information processing device 10 according to the first embodiment of the present disclosure and the elimination of uncertainty in the distance measurement result by the information processing device 50 according to the second embodiment of the present disclosure may be performed in combination.

[0136] The following configurations also fall within the technical scope of the present disclosure. (1) a candidate position calculation unit that obtains a plurality of candidate positions based on first measurement data of the three-dimensional position obtained by the sensor; a determination unit that determines one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional position obtained by the sensor; An information processing device comprising: (2) The determination unit a position and orientation estimation unit that estimates the position and orientation of the sensor based on the candidate position and the second measurement data to obtain position and orientation estimation information; a position determination unit that determines the determined position from the candidate positions based on the position and orientation estimation information, The information processing device according to (1) above. (3) the second measurement data includes one or more measurement locations; the position and orientation estimation unit performs a selection process of selecting a predetermined number of positions from the candidate positions and the measurement positions, and generates the position and orientation estimation information based on the predetermined number of positions. The information processing device according to (2) above. (4) the position and orientation estimation unit generates a plurality of pieces of position and orientation generation information by executing the selection process and the generation process of generating position and orientation generation information based on the predetermined number of positions a plurality of times, and selects the position and orientation estimation information from the plurality of pieces of position and orientation generation information. The information processing device according to (3) above. (5) the position and orientation estimation unit prevents two or more of the candidate positions from being selected as the predetermined number of positions in one selection process. The information processing device according to (3) or (4). (6) the position and orientation estimation unit calculates, for each of the position and orientation generation information, a distance between an observation position of each of the candidate positions and the measurement positions that appears in a two-dimensional image obtained by the sensor and a projection position onto the two-dimensional image that corresponds to the position and orientation generation information, and selects the position and orientation estimation information based on the distance between the observation position and the projection position for each of the position and orientation generation information. The information processing device according to (4) or (5). (7) the position and orientation estimation unit performs a predetermined vote on position and orientation generation information for which the distance between the observation position and the projection position is less than a threshold, and selects the position and orientation generation information for which the number of predetermined votes is the largest as the position and orientation estimation information. The information processing device according to (6) above. (8) the position determination unit determines a candidate position that satisfies a predetermined condition from among the plurality of candidate positions as the determined position; The information processing device according to (7) above. (9) the predetermined conditions include a first condition that the position and orientation estimation information is included in the predetermined number of positions used to generate the position and orientation estimation information. The information processing device according to (8). (10) the predetermined condition includes a second condition that the predetermined vote has been cast for the position and orientation estimation information. The information processing device according to (8) or (9). (11) the predetermined conditions include a third condition that the distance between the observation position and the projection position in the position and orientation estimation information is minimum. The information processing device according to any one of (8) to (10) above. (12) The determined position is used to re-estimate the position and orientation of the sensor. The information processing device according to any one of (2) to (11) above. (13) The determination unit a position and orientation acquisition unit that acquires a position and orientation of the sensor at a first time as first position and orientation information, and acquires a position and orientation of the sensor at a second time that is different from the first time as second position and orientation information; a position determination unit that determines the determined position from the candidate positions based on the candidate positions obtained based on the first measurement data obtained at the first time, the first position and orientation information, the second measurement data obtained at the second time, and the second position and orientation information, The information processing device according to (1) above. (14) the candidate locations include a plurality of first candidate locations; the position determination unit calculates a projection position of the first candidate position onto a two-dimensional image corresponding to the second position and orientation information, and determines the determined position based on the first candidate position and a plurality of second candidate positions obtained by the candidate position calculation unit based on the second measurement data at the projection position. The information processing device according to (13) above. (15) the position determination unit calculates a distance between the first candidate position and each of the plurality of second candidate positions for each of the first candidate positions, and determines the determined position based on the distance; The information processing device according to (14) above. (16) the position determination unit determines the first candidate position with the shortest distance as the determined position; The information processing device according to (15) above. (17) the sensor measures the three-dimensional position of the object surface based on a phase shift between irradiated light and light reflected by the object surface; The information processing device according to any one of (1) to (16) above. (18) the candidate position calculation unit obtains the plurality of candidate positions based on the modulation frequency of the irradiated light and the first measurement data. The information processing device according to (17) above. (19) a processor obtaining a plurality of candidate positions based on first measurement data of three-dimensional positions obtained by the sensor; determining one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional position obtained by the sensor; An information processing method comprising: (20) Computer, a candidate position calculation unit that obtains a plurality of candidate positions based on first measurement data of the three-dimensional position obtained by the sensor; a determination unit that determines one of the candidate positions as a determined position based on the candidate positions and second measurement data of the three-dimensional position obtained by the sensor; A program that causes the information processing device to function as an information processing device having the above. [Explanation of symbols]

[0137] 1, 2 Information Processing System 10, 50 Information processing equipment 12 Candidate position calculation unit 13 Motion estimation section 14 Positioning section 20 iToF cameras 30 Pose Observation and Use Section 40 Ranging Observation and Utilization Department 52 Candidate position calculation unit 53 Motion estimation section 54 Positioning section 60 Rigid Structure 70 RGB cameras

Claims

1. A candidate position calculation unit that obtains multiple candidate positions based on first measurement data of three-dimensional positions obtained by a sensor that is an iTOF camera; a determination unit that determines one of the plurality of candidate positions as a determined position based on the plurality of candidate positions and second measurement data of the three-dimensional position obtained by the sensor; Equipped with a first list is created containing entries in which, at a time when a two-dimensional image is obtained by the sensor, three-dimensional positions of feature points obtained by the sensor are associated with two-dimensional positions that are observation positions of the feature points obtained from a luminance image output from the sensor, and the three-dimensional positions of the feature points include the first measurement data and the second measurement data; the candidate position calculation unit calculates an interval of a distance that cannot be distinguished by the sensor by dividing the speed of light by a modulation frequency of the light irradiated by the sensor, and calculates a new three-dimensional position by adding a result of multiplying a unit vector of the first measurement data, the interval, and an integer of 1 or more to the first measurement data; the plurality of candidate positions include the first measurement data and the new three-dimensional position; The determination unit a position and orientation estimation unit that generates a second list by adding entries to the first list, each of which associates the new three-dimensional positions with a two-dimensional position associated with the first measurement data, and generates a plurality of motion hypotheses by repeatedly executing a selection process of selecting three entries from the second list and a generation process of generating a motion hypothesis of the sensor from a certain reference time point to a time point at which the two-dimensional image is obtained, based on the three-dimensional positions included in the three entries; calculates, for each of the plurality of motion hypotheses, a distance between the two-dimensional position and a projection position of the three-dimensional position onto the two-dimensional image corresponding to the motion hypothesis; and selects one motion hypothesis from the plurality of motion hypotheses based on the distance between the two-dimensional position and the projection position for each of the plurality of motion hypotheses; a position determination unit that determines, as a determined position, a candidate position that satisfies a predetermined condition related to the one motion hypothesis from among the plurality of candidate positions; An information processing device comprising:

2. the position and orientation estimation unit prevents two or more of the plurality of candidate positions from being selected as the three entries in one selection process. The information processing device according to claim 1 .

3. the position and orientation estimation unit performs a predetermined vote for a motion hypothesis in which a distance between the two-dimensional position and the projection position is less than a threshold, and selects, from the plurality of motion hypotheses, the motion hypothesis that has received the largest number of predetermined votes as the single motion hypothesis. The information processing device according to claim 1 .

4. the predetermined condition includes a first condition that the predetermined condition is included in the three entries used to generate the one motion hypothesis; The information processing device according to claim 1 .

5. the predetermined condition includes a second condition that the predetermined vote is cast for the one motion hypothesis; The information processing device according to claim 3 .

6. the predetermined condition includes a third condition that a distance between the two-dimensional position and the projected position in the one motion hypothesis is minimum. The information processing device according to claim 1 .

7. the sensor measures the three-dimensional position of the object surface based on a phase shift between irradiated light and light reflected by the object surface; The information processing device according to claim 1 .

8. a processor obtaining a plurality of candidate positions based on first measurement data of three-dimensional positions obtained by a sensor that is an iTOF camera; determining one of the plurality of candidate positions as a determined position based on the plurality of candidate positions and second measurement data of the three-dimensional position obtained by the sensor; Equipped with a first list is created containing entries in which, at a time when a two-dimensional image is obtained by the sensor, three-dimensional positions of feature points obtained by the sensor are associated with two-dimensional positions that are observation positions of the feature points obtained from a luminance image output from the sensor, and the three-dimensional positions of the feature points include the first measurement data and the second measurement data; the processor calculates an interval of a distance that cannot be distinguished by the sensor by dividing the speed of light by a modulation frequency of the light irradiated by the sensor, and calculates a new three-dimensional position by adding a result of multiplying the unit vector of the first measurement data, the interval, and an integer of 1 or more to the first measurement data; the plurality of candidate positions include the first measurement data and the new three-dimensional position; the processor: generating a second list by adding entries to the first list, each of which associates the new three-dimensional positions with a two-dimensional position associated with the first measurement data; generating a plurality of motion hypotheses by repeatedly executing a selection process of selecting three entries from the second list and a generation process of generating a motion hypothesis of the sensor from a certain reference time point to a time point at which the two-dimensional image is obtained, based on the three-dimensional positions included in the three entries; calculating, for each of the plurality of motion hypotheses, a distance between the two-dimensional position and a projection position of the three-dimensional position onto the two-dimensional image corresponding to the motion hypothesis; and selecting one motion hypothesis from the plurality of motion hypotheses based on the distance between the two-dimensional position and the projection position for each of the plurality of motion hypotheses; determining, as a determined position, a candidate position that satisfies a predetermined condition related to the one motion hypothesis from among the plurality of candidate positions; An information processing method comprising:

9. Computer, a candidate position calculation unit that obtains a plurality of candidate positions based on first measurement data of three-dimensional positions obtained by a sensor that is an iTOF camera; a determination unit that determines one of the plurality of candidate positions as a determined position based on the plurality of candidate positions and second measurement data of the three-dimensional position obtained by the sensor; Equipped with a first list is created containing entries in which, at a time when a two-dimensional image is obtained by the sensor, three-dimensional positions of feature points obtained by the sensor are associated with two-dimensional positions that are observation positions of the feature points obtained from a luminance image output from the sensor, and the three-dimensional positions of the feature points include the first measurement data and the second measurement data; the candidate position calculation unit calculates an interval of a distance that cannot be distinguished by the sensor by dividing the speed of light by a modulation frequency of the light irradiated by the sensor, and calculates a new three-dimensional position by adding a result of multiplying a unit vector of the first measurement data, the interval, and an integer of 1 or more to the first measurement data; the plurality of candidate positions include the first measurement data and the new three-dimensional position; The determination unit a position and orientation estimation unit that generates a second list by adding entries to the first list, each of which associates the new three-dimensional positions with a two-dimensional position associated with the first measurement data, and generates a plurality of motion hypotheses by repeatedly executing a selection process of selecting three entries from the second list and a generation process of generating a motion hypothesis of the sensor from a certain reference time point to a time point at which the two-dimensional image is obtained, based on the three-dimensional positions included in the three entries; calculates, for each of the plurality of motion hypotheses, a distance between the two-dimensional position and a projection position of the three-dimensional position onto the two-dimensional image corresponding to the motion hypothesis; and selects one motion hypothesis from the plurality of motion hypotheses based on the distance between the two-dimensional position and the projection position for each of the plurality of motion hypotheses; a position determination unit that determines, as a determined position, a candidate position that satisfies a predetermined condition related to the one motion hypothesis from among the plurality of candidate positions; A program that causes the information processing device to function as an information processing device having the above.

Images