Information processing device, information processing method, and program

Abstract

This information processing device (30) comprises a signal acquisition unit (31), a passive depth estimation unit (34), and a fusion unit (35). The signal acquisition unit (31) extracts, from light reception data, active information indicating the time of flight of reference light (PL) and passive information (PI) indicating two-dimensional image information for a subject (SU) obtained using ambient light (EL). The passive depth estimation unit (34) generates passive depth information (PDI) on the basis of the passive information (PI). The fusion unit (35) generates depth information (DI) for the subject (SU) by fusing active depth information (ADI) generated on the basis of the active information with the passive depth information (PDI).

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A 3D measurement technology using the ToF (Time of Flight) method is known, in which a reference light such as an infrared pulse is projected onto a subject, and the depth of the subject is detected based on information on the time it takes for the reflected light to be received. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-071976 [Patent Document 2] International Publication No. 2010 / 021090 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-233983 [Patent Document 4] Japanese Patent Application Laid-Open No. 2001-264033 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described method, if ambient light having the same wavelength as the reference light is present in the measurement environment, the ambient light acts as noise, reducing the accuracy of depth detection.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can improve the accuracy of depth detection. [Means for solving the problem]

[0006] According to the present disclosure, there is provided an information processing device having a signal acquisition unit that extracts active information indicating the time of flight of a reference light and passive information indicating two-dimensional image information of a subject caused by ambient light from received light data, a passive depth estimation unit that generates passive depth information based on the passive information, and a fusion unit that generates depth information of the subject by fusing the passive depth information with the active depth information generated based on the active information. Also, according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a program that causes a computer to realize the information processing of the information processing device. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of a distance measurement method using a ToF camera. [Figure 2] FIG. 10 is a diagram illustrating an example in which an ambient light source exists in the shooting environment of a subject. [Figure 3] FIG. 10 is a diagram showing an example of light reception data of a ToF camera. [Figure 4] FIG. 10 is a diagram illustrating an example of active information and passive information. [Figure 5] FIG. 10 is a diagram illustrating the data structure of LiDAR input data. [Figure 6] FIG. 1 is a diagram illustrating a schematic configuration of a ToF camera. [Figure 7] 10A and 10B are diagrams illustrating an example of a correction process of active depth information performed by a filter unit. [Figure 8] 10A and 10B are diagrams illustrating an example of a correction process of active depth information performed by a filter unit. [Figure 9] 10A and 10B are diagrams illustrating an example of waveform separation processing performed by a signal acquisition unit. [Figure 10] 10A and 10B are diagrams illustrating an example of waveform separation processing performed by a signal acquisition unit. [Figure 11] 10A and 10B are diagrams illustrating an example of waveform separation processing performed by a signal acquisition unit. [Figure 12] 10A and 10B are diagrams illustrating an example of waveform separation processing performed by a signal acquisition unit. [Figure 13] FIG. 10 is a diagram illustrating an example of a method for setting a measurement period. [Figure 14] 10 is a flowchart illustrating an example of information processing related to depth estimation. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The explanation will be given in the following order. [1. Distance measurement method using ToF camera] [2. LiDAR input data] [3. ToF camera configuration] [4. Correction of active depth information using passive information] [5. Waveform separation processing] [6. Fusion ratio setting process] [7. Setting the measurement period and sampling cycle] [8. Information Processing Methods] [9. Effects]

[0010] [1. Distance measurement method using ToF camera] FIG. 1 is an explanatory diagram of a distance measurement method using a ToF camera 1.

[0011] The ToF camera 1 has a light projecting unit 10 and a light receiving unit 20. The light projecting unit 10 projects reference light PL toward the subject SU. The reference light PL is, for example, infrared pulsed light. The light receiving unit 20 receives the reference light PL (reflected light RL) reflected by the subject SU. The ToF camera 1 detects the depth d of the subject SU based on the time t from when the reference light PL is projected until it is reflected by the subject SU and received by the light receiving unit 20. The depth d can be expressed as d=t×c / 2, where c is the speed of light.

[0012] FIG. 2 is a diagram showing an example in which an environment light source ELS exists in the shooting environment of a subject SU.

[0013] When an ambient light source ELS, such as the sun or an electric lamp, is present in the shooting environment, ambient light EL emitted from the ambient light source ELS as well as reference light PL reflected from the subject SU enters the light receiving unit 20. If ambient light EL with the same wavelength as the reference light PL is present in the shooting environment, the ambient light PL may act as noise, reducing the accuracy of depth detection. While Patent Documents 2 to 4 eliminate the ambient light PL that causes noise, the present disclosure actively utilizes the ambient light PL to improve the accuracy of depth detection. This point will be discussed later.

[0014] FIG. 3 is a diagram showing an example of light reception data LRD of the ToF camera 1.

[0015] The light receiving unit 20 has a plurality of pixels arranged in the X and Y directions. The Z direction, which is orthogonal to the X and Y directions, is the depth direction. Each pixel is provided with an infrared sensor that detects infrared rays. The infrared sensor receives infrared rays at a preset sampling period. The infrared sensor detects the number of photons received within one sampling period (sampling period) as brightness.

[0016] The light receiving unit 20 repeatedly measures the luminance within a preset measurement period (one frame). The ToF camera 1 stores the light receiving data LRD measured by the light receiving unit 20 during one measurement period as LiDAR input data ID. The LiDAR input data ID is time-series luminance data for each pixel measured during one measurement period. The ToF camera 1 converts the time from the measurement start time into distance (depth). The ToF camera 1 generates a depth map DM of the subject SU based on the LiDAR input data ID.

[0017] The lower part of Figure 3 shows the luminance signal BS of one pixel. The horizontal axis indicates time from the measurement start time. The vertical axis indicates luminance. The luminance signal BS is a signal that indicates the temporal change in luminance extracted from the LiDAR input data ID. The luminance signal BS includes a reference light component RC due to the reference light PL and an ambient light component EC due to the ambient light EL. The ambient light component EC includes an offset component and a noise component.

[0018] The position of the reference light component RC in the time axis direction reflects the distance to the object SU. The average signal value of the ambient light component EC indicates the brightness of the pixel. The brightness of each pixel provides information about a two-dimensional image of the object SU (two-dimensional image information). In the present disclosure, active information indicating the time of flight of the reference light PL and passive information indicating two-dimensional image information of the object SU due to the ambient light EL are extracted from the LiDAR input data ID. For example, the active information includes information indicating the position and magnitude of the peak at each pixel due to the reference light PL. The passive information includes information indicating the average signal value at each pixel due to the ambient light EL.

[0019] FIG. 4 is a diagram showing an example of active information AI and passive information PI.

[0020] The left side of Figure 4 shows active information AI extracted from the reference light component RC of each pixel. The active information AI includes information about the position of the reference light component RC of each pixel in the time axis direction (time of flight of the reference light PL). The active information AI directly estimates the depth of each pixel. The right side of Figure 4 shows passive information PI extracted from the ambient light component EC of each pixel. The passive information PI includes information about the average signal value of the ambient light component EC of each pixel. The passive information PI allows the accurate two-dimensional shape of the subject SU to be detected. Furthermore, by using known object recognition technology, it is possible to estimate the type of subject SU from the two-dimensional image IM of the subject SU.

[0021] For example, Figure 4 shows a state in which a doll OB1 is placed on a table OB2. Using only active information AI, it is not possible to accurately detect the depth of the boundary between the doll OB1 and the table OB2. However, if the position of the boundary between the doll OB1 and the table OB2 is detected based on passive information PI and the depth is corrected based on the detected position of the boundary, an accurate depth can be obtained. Furthermore, if the passive information PI estimates that the surface of the table OB2 is flat, the depth obtained from active information AI can be corrected based on the inclination and depth length of this flat surface.

[0022] In addition, depth can be estimated using stereo imaging with multiple 2D images (IM) taken from different viewpoints. By fusing the depth information estimated by stereo imaging with the depth information estimated based on active information (AI), highly accurate depth information can be obtained. Combining an active ranging method using active information (AI) with a passive ranging method using passive information (PI) improves the accuracy of depth detection.

[0023] [2. LiDAR input data] FIG. 5 is a diagram showing the data structure of the LiDAR input data ID.

[0024] The LiDAR input data ID includes ToF data MD measured at multiple times over a sampling period SP. The ToF data MD includes brightness data for each pixel measured at the same time. The number of samples N of the LiDAR input data ID (the number of measurements that can be performed in one measurement period FR) is determined by the memory size of the ToF camera 1 and the capacity of the ToF data MD. The ranging range (depth measurement interval), which indicates the depth resolution, varies depending on the sampling period SP. The shorter the sampling period SP, the shorter the ranging range, and as a result, the higher the depth resolution.

[0025] [3. ToF camera configuration] FIG. 6 is a diagram showing a schematic configuration of the ToF camera 1.

[0026] The ToF camera 1 includes a light projecting unit 10, a light receiving unit 20, a processing unit 30, a motion detecting unit 60, and a storage unit .

[0027] The light projecting unit 10 is, for example, a laser or a projector that projects reference light PL. The light receiving unit 20 is, for example, an image sensor having a plurality of pixels for infrared detection. The processing unit 30 is an information processing device that processes various information. The processing unit 30 generates a depth map DM based on the LiDAR input data ID acquired from the light receiving unit 20. The motion detecting unit 60 detects motion information of the light receiving unit 20. The motion detecting unit 60 includes, for example, an IMU (Inertial Measurement Unit) and a GPS sensor. The storage unit 70 stores a waveform separation model 71, setting information 72, and a program 79 required for information processing by the processing unit 30.

[0028] The processing unit 30 includes a signal acquisition unit 31, a filter unit 32, a position and orientation estimation unit 33, a passive depth estimation unit , a fusion unit 35, and a plane angle estimation unit .

[0029] The signal acquisition unit 31 acquires LiDAR input data ID from the light receiving unit 20. The signal acquisition unit 31 extracts a luminance signal BS of each pixel from the LiDAR input data ID. The signal acquisition unit 31 extracts active information AI and passive information PI from the luminance signal BS of each pixel.

[0030] The signal acquisition unit 31 generates active depth information ADI indicating an estimated depth value for each pixel from the active information AI. The signal acquisition unit 31 supplies the active depth information ADI to the filter unit 32 and the plane angle estimation unit 36. The signal acquisition unit 31 supplies passive information PI to the filter unit 32, the position and orientation estimation unit 33, the passive depth estimation unit 34, and the plane angle estimation unit 36.

[0031] The filter unit 32 corrects the active depth information ADI (first active depth information ADI1) acquired from the signal acquisition unit 31 based on the passive information PI. For example, the filter unit 32 estimates the shape of the subject SU from two-dimensional image information of the subject SU extracted from the passive information PI. The filter unit 32 corrects the first active depth information ADI1 so that the depth of the subject SU matches the shape of the subject SU estimated based on the passive information PI. The filter unit 32 supplies the corrected active depth information ADI (second active depth information ADI2) to the position and orientation estimation unit 33 and the fusion unit 35.

[0032] [4. Correction of active depth information using passive information] 7 and 8 are diagrams showing an example of the correction process of the active depth information ADI by the filter unit 32. FIG.

[0033] The filter unit 32 generates a two-dimensional image IM of the subject SU based on, for example, the passive information PI. The filter unit 32 extracts a plane portion PN of the subject SU from the two-dimensional image IM using a known object recognition technique. The plane angle estimation unit 36 ​​estimates the angle θ of the plane portion PN of the subject SU based on the first active depth information ADI1.

[0034] As shown on the left side of Fig. 8, when the planar portion PN is perpendicular to the depth direction (when the subject SU faces the light receiving unit 20), the filter unit 32 corrects the first active depth information ADI1 so that the planar portion PN has a constant depth. As shown on the right side of Fig. 8, when the planar portion PN is inclined with respect to the depth direction (when the subject SU does not face the light receiving unit 20), the filter unit 32 corrects the first active depth information ADI1 so that the depth of the planar portion PN changes depending on the angle θ.

[0035] Returning to FIG. 6 , the position and orientation estimation unit 33 estimates the position and orientation of the light receiving unit 20 during the measurement period FR. For example, the position and orientation estimation unit 33 estimates the amount of movement of the light receiving unit 20 based on depth information (second active depth information ADI2 or depth information DI) at multiple times acquired for each measurement period FR. The amount of movement may be corrected based on the movement information acquired from the motion detection unit 60 and two-dimensional image information (passive information PI) of the subject SU at multiple times acquired for each measurement period FR. The position and orientation estimation unit 33 estimates the position and orientation of the light receiving unit 20 based on the amount of movement of the light receiving unit 20. The position and orientation estimation unit 33 supplies position and orientation information PPI relating to the position and orientation of the light receiving unit 20 to the passive depth estimation unit 34.

[0036] The passive depth estimation unit 34 generates passive depth information PDI based on the passive information PI. For example, the passive depth estimation unit 34 extracts, from the passive information PI, two-dimensional images IM of a plurality of subjects SU acquired from different viewpoints at different times. The passive depth estimation unit 34 detects multiple viewpoints corresponding to the multiple two-dimensional images IM based on the position and orientation information PPI of the light receiving unit 20 estimated by the position and orientation estimation unit 33. The passive depth estimation unit 34 generates passive depth information PDI from the multiple two-dimensional images IM using a stereo imaging method.

[0037] The fusion unit 35 generates depth information DI of the subject SU by fusing the passive depth information PDI with the second active depth information ADI2 generated based on the active information AI. The fusion ratio between the second active depth information ADI2 and the passive depth information PDI is set based on, for example, the energy ratio between the active signal and the passive signal.

[0038] [5. Waveform separation processing] 9 to 12 are diagrams illustrating an example of the waveform separation process performed by the signal acquisition unit 31. FIG.

[0039] As shown in FIG. 9, the signal acquiring section 31 includes a waveform converting section 41, a peak acquiring section 42, a section averaging section 43, a fusion ratio setting section 44, a measurement period resetting section 45, and a sampling period resetting section 46.

[0040] The waveform converter 41 converts the peaks in the luminance signal BS caused by the reference light PL into a pulse waveform. The waveform converter 41 supplies a corrected luminance signal MBS obtained by waveform conversion processing of the luminance signal BS to a peak acquirer 42.

[0041] The waveform conversion process is performed based on a waveform separation model 71. The waveform separation model 71 is, for example, a trained neural network obtained by machine learning the relationship between the reference light PL and the luminance signal BS. The input data for the machine learning is the luminance signal BS detected for the object SU, the distance of which is known in advance. As shown in FIG. 10, the correct answer data is a signal obtained by converting the waveform of the luminance signal BS so that, instead of the reference light component RC, a pulse-shaped waveform (active signal AS) identical to that of the reference light PL appears at a time corresponding to the distance of the object SU. When the luminance signal BS is input to the waveform separation model 71 that has undergone such machine learning, a corrected luminance signal MBS is obtained that includes an active signal AS that accurately reflects the time of flight.

[0042] The peak acquisition unit 42 acquires a portion of the signal corresponding to the pulse waveform from the luminance signal BS (corrected luminance signal MBS) whose waveform has been converted by the waveform conversion unit 41, as an active signal AS resulting from the reference light PL. The peak acquisition unit 42 calculates the depth based on the position of the active signal AS on the time axis. The peak acquisition unit 42 generates first active depth information ADI1 based on the depth calculated for each pixel.

[0043] The section averaging unit 43 calculates an average signal value of the corrected luminance signal MBS of the portion caused by the ambient light EL. The section averaging unit 43 acquires the calculated average signal value as the signal value of the passive signal PS caused by the ambient light EL.

[0044] The method for calculating the signal value of the passive signal PS varies depending on, for example, the ratio of the energy values ​​of the active signal AS and the passive signal PS. The section averaging unit 43 varies the section on the time axis over which the signal values ​​of the corrected luminance signal MBS are averaged based on a preset criterion. The criterion is set, for example, as a condition that the energy value of the passive signal PS is greater than the energy value of the active signal AS and the difference in energy values ​​is significant.

[0045] For example, as shown in Fig. 11, if the ratio of the energy values ​​of the passive signal PS satisfies the determination criterion, the section averaging unit 43 calculates the signal value of the passive signal PS as the signal value obtained by averaging the signal values ​​of the corrected luminance signal MBS over the entire measurement time FR. As shown in Fig. 12, if the ratio of the energy values ​​of the passive signal PS and the active signal AS does not satisfy the determination criterion, the section averaging unit 43 calculates the signal value of the passive signal PS as the signal value obtained by averaging the signal values ​​of the corrected luminance signal MBS in a section on the time axis excluding the active signal AS.

[0046] [6. Fusion ratio setting process] Returning to FIG. 9, the fusion ratio setting unit 44 sets a fusion ratio FUR between the second active depth information ADI2 and the passive depth information PDI for each pixel based on the ratio between the energy value of the active signal AS and the energy value of the passive signal PS. The greater the ratio of the energy value of the active signal AS, the greater the fusion ratio FUR of the second active depth information ADI2. The shorter the distance to the subject SU, the greater the energy value of the active signal AS. Therefore, the shorter the distance to the subject SU, the greater the contribution of the second active depth information ADI2 to the generated depth information DI. When the subject SU is far away, the greater the contribution of the passive depth information PDI to the generated depth information DI.

[0047] For example, a pixel whose fusion ratio FUR of the second active depth information ADI2 is greater than that of the passive depth information PDI is defined as an active-dominant pixel in which the active signal AS is dominant, and a pixel whose fusion ratio FUR of the passive depth information PDI is greater than that of the second active depth information ADI2 is defined as a passive-dominant pixel in which the passive signal PS is dominant.

[0048] The fusion unit 35 extracts a group of active-dominant pixels in which the active signal AS is dominant from the multiple pixels included in the light receiving unit 20. The fusion unit 35 calculates, for each pixel, the difference between the passive depth estimated from the passive information PI and the active depth estimated from the active information AI. The fusion unit 35 bundle adjusts the passive depths of all pixels in the light receiving unit 20 so that the difference between the passive depth and the active depth satisfies a predetermined criterion in the active-dominant pixel group.

[0049] For example, bundle adjustment is performed to satisfy the criteria shown in the following formula (1): Formula (1) indicates the condition under which the sum of the differences between the passive depth and the active depth for all pixels in the active dominant pixel group is minimized.

[0050]

number

[0051] The fusion unit 35 determines the depth of the passive-dominant pixel group in which the passive signal PS is dominant as the passive depth after bundle adjustment, and determines the depth of the pixel group other than the passive-dominant pixel group as the active depth.

[0052] [7. Setting the measurement period and sampling cycle] The measurement period resetting unit 45 calculates the minimum measurement period FR such that the ratio of the energy values ​​of the passive signal PS satisfies a preset high ratio condition. The sampling period resetting unit 46 sets the sampling period SP of the light receiving unit 20 based on the maximum depth value of the subject SU, the number of measurable samples N, and the minimum measurement period FR that satisfies the high ratio condition.

[0053] FIG. 13 is a diagram illustrating an example of a method for setting the measurement period FR.

[0054] In Figure 13, the variable T Psv indicates the length of the measurement period SP. The active signal AS has a signal width of T Act , the signal value is P Act The passive signal PS is a pulse-shaped signal that is a time domain section (T Psv -T Act ) and the signal value is P Psv The high ratio condition is set, for example, as a condition that satisfies the following formula (2). Formula (2) indicates the condition that the energy value of the passive signal PS is 100 times or more the energy value of the active signal AS. The condition of formula (2) can be satisfied by adjusting the width and height (signal value) of the active signal AS.

[0055]

number

[0056] If the measurement period FR is extended (T Psv However, since there is a fixed upper limit to the number of samples N that can be acquired as LiDAR input data ID, extending the measurement period FR also extends the sampling period. As the sampling period becomes longer, the ranging range becomes longer and the depth resolution becomes smaller. Therefore, the measurement period resetting unit 45 adjusts the length T of the measurement period FR. Psv is set as the minimum length that satisfies equation (2).

[0057] The sampling period resetting unit 46 adaptively determines the sampling period SP for each frame according to the distance to the subject SU so as to maximize the depth resolution. The sampling period resetting unit 46 extracts the maximum depth values ​​(maximum depth values) from both the second active depth information ADI2 and the passive depth information PDI, and calculates the sampling period SP using the maximum depth value with the higher reliability.

[0058] For example, in an active distance measurement method using reference light PL, the received light intensity of the reference light PL (reflected light) changes depending on the distance to the subject SU. As the distance becomes shorter, the received light intensity increases, improving depth detection accuracy, but as the distance becomes longer, the received light intensity decreases, reducing depth detection accuracy. In an active distance measurement method, the measurable distance is shorter than in a passive measurement method using stereo imaging. Therefore, the sampling period resetting unit 46 determines the priority of the second active depth information ADI2 and the passive depth information PDI based on the distance to the subject SU. The sampling period resetting unit 46 sets the sampling period SP using the maximum depth value extracted from the one determined to have the higher priority.

[0059] For example, the sampling period resetting unit 46 extracts the maximum depth (maximum depth value) of the subject SU from both the second active depth information ADI2 and the passive depth information PDI. The sampling period resetting unit 46 determines the priority of the passive depth information PDI based on the maximum depth value (first maximum depth value) extracted from the second active depth information ADI2 and the maximum depth value (second maximum depth value) extracted from the passive depth information PDI.

[0060] If the sampling period resetting unit 46 determines that the passive depth information PDI should be prioritized, it sets the sampling period SP using the second maximum depth value. If the sampling period resetting unit 46 determines that the passive depth information PDI should not be prioritized, it sets the sampling period SP using the maximum depth value (third maximum depth value) extracted from the depth information DI.

[0061] For example, the magnitude of the first maximum depth value is max(depth Active ), the magnitude of the second maximum depth value is max(depth passive ), the length of the sampling period SP is ΔT, the number of measurable samples is N, the speed of light is c, and the number of buffer samples is buffer. As shown in the following equation (3), if it is determined that the second maximum depth value is sufficiently larger than the first maximum depth value and the depth cannot be measured accurately using the active measurement method, the sampling period resetting unit 46 sets the sampling period SP using measurement information (second maximum depth value) from a passive measurement method that can measure distant objects accurately.

[0062]

number

[0063] As shown in the following formula (4), when it is determined that the second maximum depth value and the first maximum depth value are equivalent and that the depth can be measured with high accuracy by the active measurement method, the sampling period resetting unit 46 uses the highly accurate measurement information obtained by the fusion (third maximum depth value=max(depth Fusion )) to set the sampling period SP.

[0064]

number

[0065] However, as shown in the following equation (5), when the sampling period SP becomes shorter, the measurement period FR calculated from the sampling period SP and the number of samples N may become shorter than the minimum measurement period FR calculated by equation (2). In this case, the sampling period resetting unit 46 sets the sampling period SP based on the minimum measurement period SP calculated by equation (2).

[0066]

number

[0067] Based on the calculation results, the measurement period resetting unit 45 and the sampling period resetting unit 46 control the measurement period FR and sampling period SP of the light receiving unit 20. The light receiving unit 20 performs the next measurement (next frame) based on the set measurement period FR and sampling period SP.

[0068] Information relating to the various conditions and criteria described above is included in the setting information 72. The waveform separation model 71, setting information 72, and program 79 used in the above-described processing are stored in the storage unit 70. The program 79 is a program that causes a computer to execute information processing according to this embodiment. The processing unit 30 performs various processes in accordance with the program 79 stored in the storage unit 70. The storage unit 70 may be used as a working area for temporarily storing processing results of the processing unit 30. The storage unit 70 includes any non-transitory storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage unit 70 is configured to include, for example, an optical disk, a magneto-optical disk, or a flash memory. The program 79 is stored, for example, in a non-transitory storage medium readable by a computer.

[0069] The processing unit 30 is, for example, a computer configured with a processor and a memory. The memory of the processing unit 30 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). By executing the program 79, the processing unit 30 functions as a signal acquisition unit 31, a filter unit 32, a position and orientation estimation unit 33, a passive depth estimation unit 34, a fusion unit 35, a plane angle estimation unit 36, a waveform conversion unit 41, a peak acquisition unit 42, an interval averaging unit 43, a fusion ratio setting unit 44, a measurement period resetting unit 45, and a sampling period resetting unit 46.

[0070] [8. Information Processing Methods] FIG. 14 is a flowchart showing an example of information processing related to depth estimation.

[0071] In step S1, the light projecting unit 10 projects reference light PL onto the subject SU. In step S2, the light receiving unit 20 receives reflected light RL. In step S3, the processing unit 30 acquires LiDAR input data ID from the light receiving unit 20 as received light data LRD.

[0072] In step S4, the processing unit 30 performs processing to improve the quality of the active depth information ADI. For example, the processing unit 30 extracts active information AI indicating the time of flight of the reference light PL and passive information PI indicating two-dimensional image information of the subject SU due to the ambient light EL from the LiDAR input data ID. The processing unit 30 corrects the active depth information ADI so that the depth of the subject SU is consistent with the shape of the subject SU estimated based on the passive information PI.

[0073] In parallel with step S4, the processing unit 30 estimates the depth of the subject SU from the passive information PI using a passive ranging technique. For example, in step S5, the processing unit 30 estimates the position and orientation of the light receiving unit 20 based on the two-dimensional image information of the subject SU extracted from the passive information PI, the active depth information ADI, and the motion information acquired from the motion detection unit 60.

[0074] In step S6, the processing unit 30 extracts, from the passive information PI, two-dimensional images IM of the subject SU acquired at different times and from different viewpoints. For each two-dimensional image IM, the processing unit 30 detects the viewpoint of the light receiving unit 20 from which the two-dimensional image IM was acquired, based on the position and orientation information of the light receiving unit 20. The processing unit 30 generates passive depth information PDI from the two-dimensional images IM acquired from different viewpoints using a stereo imaging method.

[0075] In step S7, the processing unit 30 fuses the active depth information ADI with the passive depth information PDI, thereby combining the active depth information ADI and the passive depth information PDI.

[0076] In step S8, the processing unit 30 determines whether to end the processing. For example, if the processing unit 30 receives an operation such as an operation of the shooting end button from the user operating the ToF camera 1, the processing unit 30 determines to end the processing. If it is determined in step S8 that the processing is to end (step S8: Yes), the depth estimation processing is ended. If it is determined in step S8 that the processing is not to end (step S8: No), the processing returns to step S1, and the above-described processing is repeated until it is determined that the processing is to end.

[0077] [9. Effects] The processing unit 30 includes a signal acquisition unit 31, a passive depth estimation unit 34, and a fusion unit 35. The signal acquisition unit 31 extracts active information AI indicating the time of flight of the reference light PL and passive information PI indicating two-dimensional image information of the subject SU due to the ambient light EL from the received light data LDR. The passive depth estimation unit 34 generates passive depth information PDI based on the passive information PI. The fusion unit 35 generates depth information DI of the subject SU by fusing the passive depth information PDI with the active depth information ADI generated based on the active information AI. In the information processing method of this embodiment, the processing of the processing unit 30 described above is executed by a computer. A program 79 of this embodiment causes a computer to realize the processing of the processing unit 30 described above.

[0078] According to this configuration, the accuracy of depth detection can be improved by utilizing the ambient light EL, so that highly accurate depth detection can be performed even in an environment with strong ambient light, such as outdoors.

[0079] The passive depth estimation unit 34 extracts, from the passive information PI, two-dimensional images IM of a plurality of subjects SU acquired from different viewpoints at different times, and generates passive depth information PDI from the plurality of two-dimensional images IM using a stereo imaging method.

[0080] According to this configuration, passive depth information PDI is generated without the need to prepare different light receiving units 20 for each viewpoint.

[0081] The passive depth estimation unit 34 detects a plurality of viewpoints corresponding to a plurality of two-dimensional images IM based on the position and orientation information PPI estimated using the active depth information ADI.

[0082] This configuration makes it possible to obtain passive depth information PDI with high depth estimation accuracy.

[0083] The processing unit 30 includes a filter unit 32. The filter unit 32 corrects the active depth information ADI so that the depth of the subject SU matches the shape of the subject SU estimated based on the passive information PI.

[0084] This configuration makes it possible to obtain active depth information with high depth estimation accuracy.

[0085] The processing unit 30 includes a plane angle estimation unit 36. The plane angle estimation unit 36 ​​estimates the angle θ of the plane portion PN of the subject SU based on the pre-correction active depth information ADI (first active depth information ADI1). When the plane portion PN is orthogonal to the depth direction, the filter unit 32 corrects the active depth information ADI so that the plane portion PN has the same depth. When the plane portion PN is inclined with respect to the depth direction, the filter unit 32 corrects the active depth information ADI so that the depth of the plane portion PN changes according to the angle θ.

[0086] According to this configuration, the depth of the planar portion PN is estimated with high accuracy based on the angle θ of the planar portion PN.

[0087] The signal acquisition unit 31 has a waveform conversion unit 41, a peak acquisition unit 42, and a section averaging unit 43. The waveform conversion unit 41 converts peaks caused by the reference light PL into a pulsed waveform. The peak acquisition unit 42 acquires a portion of the signal corresponding to the pulsed waveform as an active signal AS caused by the reference light PL. The section averaging unit 43 calculates an average signal value of the portion caused by the ambient light EL. The section averaging unit 43 acquires the calculated average signal value as the signal value of a passive signal PS caused by the ambient light EL.

[0088] According to this configuration, the active signal AS and the passive signal PS are separated with high precision.

[0089] The signal acquisition unit 31 includes a fusion ratio setting unit 44. The fusion ratio setting unit 44 sets a fusion ratio FUR between the active depth information ADI and the passive depth information PDI based on the ratio between the energy value of the active signal AS and the energy value of the passive signal PS.

[0090] According to this configuration, an appropriate fusion ratio FUR is set according to the reliability of the active depth information ADI.

[0091] The fusion unit 35 extracts an active-dominant pixel group in which the active signal AS is dominant. The fusion unit 35 bundle adjusts the passive depths of all pixels so that the difference between the passive depth estimated from the passive information PI and the active depth estimated from the active information AI satisfies a predetermined standard for the active-dominant pixel group. The fusion unit 35 sets the depth of the passive-dominant pixel group in which the passive signal PS is dominant as the passive depth after bundle adjustment, and sets the depth of the pixel group other than the passive-dominant pixel group as the active depth.

[0092] This configuration improves the accuracy of passive depth estimation.

[0093] The signal acquisition unit 31 includes a measurement period resetting unit 45. The measurement period resetting unit 45 calculates the minimum measurement period FR such that the ratio of the energy value of the passive signal PS satisfies a preset high ratio condition.

[0094] According to this configuration, the light reception data LDR that satisfies the high ratio condition is stably acquired.

[0095] The signal acquisition unit 31 has a sampling period resetting unit 46. The sampling period resetting unit 46 sets the sampling period SP based on the maximum depth value of the subject SU, the number of measurable samples N, and the minimum measurement period FR that satisfies the high ratio condition.

[0096] This configuration allows for measurements with high depth resolution.

[0097] The sampling period resetting unit 46 determines the priority of the active depth information ADI and the passive depth information PDI based on the distance to the subject SU, and sets the sampling period SP using the maximum depth value extracted from the information determined to have the higher priority.

[0098] According to this configuration, the maximum depth value that is the basis for calculating the sampling period SP is appropriately determined based on the reliability of the measurement information according to the distance to the subject SU.

[0099] The sampling period resetting unit 46 determines the priority of the passive depth information PDI based on the first maximum depth value extracted from the active depth information ADI and the second maximum depth value extracted from the passive depth information PDI. If the sampling period resetting unit 46 determines that the passive depth information PDI should be prioritized, it sets the sampling period SP using the second maximum depth value. If the sampling period resetting unit 46 determines that the passive depth information PDI should not be prioritized, it sets the sampling period SP using the third maximum depth value extracted from the depth information DI.

[0100] According to this configuration, an appropriate sampling period is calculated based on a highly accurate maximum depth value.

[0101] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0102] [Note] The present technology can also be configured as follows. (1) a signal acquisition unit that extracts active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; a passive depth estimation unit that generates passive depth information based on the passive information; a fusion unit that generates depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; An information processing device having the above. (2) the passive depth estimation unit extracts, from the passive information, a plurality of two-dimensional images of the subject acquired from different viewpoints at different times, and generates the passive depth information from the plurality of two-dimensional images by a stereo imaging method. The information processing device according to (1) above. (3) the passive depth estimation unit detects a plurality of viewpoints corresponding to the plurality of two-dimensional images based on position and orientation information estimated using the active depth information. The information processing device according to (2) above. (4) a filter unit that corrects the active depth information so that the depth of the subject matches the shape of the subject estimated based on the passive information; The information processing device according to any one of (1) to (3) above. (5) a plane angle estimation unit that estimates an angle of a plane portion of the subject based on the active depth information before correction; When the planar portion is orthogonal to the depth direction, the filter unit corrects the active depth information so that the planar portion has a constant depth, and when the planar portion is inclined with respect to the depth direction, the filter unit corrects the active depth information so that the depth of the planar portion changes according to the angle. The information processing device according to (4) above. (6) the signal acquisition unit includes a waveform conversion unit that converts a peak caused by the reference light into a pulsed waveform, a peak acquisition unit that acquires a portion of a signal corresponding to the pulsed waveform as an active signal caused by the reference light, and a section averaging unit that calculates an average signal value of a portion caused by the ambient light and acquires the calculated average signal value as a signal value of a passive signal caused by the ambient light. The information processing device according to any one of (1) to (5) above. (7) the signal acquisition unit includes a fusion ratio setting unit that sets a fusion ratio of the active depth information and the passive depth information based on a ratio between an energy value of the active signal and an energy value of the passive signal; The information processing device according to (6) above. (8) The fusion unit includes: extracting a group of active pixels in which the active signals are dominant; bundle adjusting the passive depths of all pixels so that a difference between a passive depth estimated from the passive information and an active depth estimated from the active information satisfies a predetermined criterion in the active dominant pixel group; The depth of a passive dominant pixel group in which the passive signal is dominant is set as the passive depth after bundle adjustment, and the depth of a pixel group other than the passive dominant pixel group is set as the active depth. The information processing device according to (6) or (7) above. (9) the signal acquisition unit includes a measurement period resetting unit that calculates a minimum measurement period in which a ratio of the energy value of the passive signal satisfies a preset high ratio condition; The information processing device according to any one of (6) to (8) above. (10) the signal acquisition unit has a sampling period resetting unit that sets a sampling period based on the maximum depth value of the subject, the number of measurable samples, and the minimum measurement period; The information processing device according to (9) above. (11) the sampling period resetting unit determines the priority of the active depth information and the passive depth information based on the distance to the subject, and sets the sampling period using the maximum depth value extracted from the information determined to have a higher priority. The information processing device according to (10) above. (12) the sampling period resetting unit determines the priority based on a first maximum depth value extracted from the active depth information and a second maximum depth value extracted from the passive depth information, and if it determines that the passive depth information should be prioritized, sets the sampling period using the second maximum depth value, and if it determines that the passive depth information should not be prioritized, sets the sampling period using a third maximum depth value extracted from the depth information. The information processing device according to (11) above. (13) extracting active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; generating passive depth information based on the passive information; generating depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; 10. A computer-implemented information processing method comprising: (14) extracting active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; generating passive depth information based on the passive information; generating depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; A program that makes a computer do something. [Explanation of symbols]

[0103] 30 Processing unit (information processing device) 31 Signal acquisition unit 32 Filter section 34 Passive depth estimation unit 35 Fusion Club 36 Plane angle estimator 41 Waveform conversion section 42 Peak acquisition section 43 Section Average Section 44 Fusion ratio setting section 45 Measurement period resetting unit 46 Sampling period resetting unit 79 Programs ADI Active Depth Information AI Active Information AS active signal EL ambient light FR Measurement Period IM 2D image LDR light reception data PDI Passive Depth Information PI Passive Information PL reference light PN flat part PPI position and orientation information PS Passive Signal SP Sampling period SU Subject DI Depth Information θ angle

Claims

1. a signal acquisition unit that extracts active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; a passive depth estimation unit that generates passive depth information based on the passive information; a fusion unit that generates depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; and The signal acquisition unit a waveform converter for converting a peak due to the reference light into a pulse waveform; a peak acquisition unit that acquires a portion of a signal corresponding to the pulse waveform as an active signal caused by the reference light; an interval averaging unit that calculates an average signal value of the portion caused by the ambient light and acquires the calculated average signal value as a signal value of the passive signal caused by the ambient light; having Information processing device.

2. the passive depth estimation unit extracts, from the passive information, a plurality of two-dimensional images of the subject acquired at different times and from different viewpoints, and generates the passive depth information from the plurality of two-dimensional images by a stereo imaging method; The information processing device according to claim 1 .

3. the passive depth estimation unit detects a plurality of viewpoints corresponding to the plurality of two-dimensional images based on position and orientation information estimated using the active depth information; The information processing device according to claim 2 .

4. a filter unit that corrects the active depth information so that the depth of the subject matches the shape of the subject estimated based on the passive information; The information processing device according to claim 1 .

5. a plane angle estimation unit that estimates an angle of a plane portion of the subject based on the active depth information before correction; When the planar portion is orthogonal to the depth direction, the filter unit corrects the active depth information so that the planar portion has a constant depth, and when the planar portion is inclined with respect to the depth direction, the filter unit corrects the active depth information so that the depth of the planar portion changes according to the angle. The information processing device according to claim 4 .

6. the signal acquisition unit includes a fusion ratio setting unit that sets a fusion ratio of the active depth information and the passive depth information based on a ratio between an energy value of the active signal and an energy value of the passive signal; The information processing device according to claim 1 .

7. The fusion unit includes: extracting a group of active pixels in which the active signals are dominant; bundle adjusting the passive depths of all pixels so that a difference between a passive depth estimated from the passive information and an active depth estimated from the active information satisfies a predetermined criterion in the active dominant pixel group; The depth of a passive dominant pixel group in which the passive signal is dominant is set as the passive depth after bundle adjustment, and the depth of a pixel group other than the passive dominant pixel group is set as the active depth. The information processing device according to claim 1 .

8. the signal acquisition unit includes a measurement period resetting unit that calculates a minimum measurement period in which a ratio of the energy value of the passive signal satisfies a preset high ratio condition; The information processing device according to claim 1 .

9. the signal acquisition unit has a sampling period resetting unit that sets a sampling period based on the maximum depth value of the subject, the number of measurable samples, and the minimum measurement period; The information processing device according to claim 8 .

10. the sampling period resetting unit determines the priority of the active depth information and the passive depth information based on the distance to the subject, and sets the sampling period using the maximum depth value extracted from the information determined to have a higher priority. The information processing device according to claim 9 .

11. the sampling period resetting unit determines the priority based on a first maximum depth value extracted from the active depth information and a second maximum depth value extracted from the passive depth information, and if it determines that the passive depth information should be prioritized, sets the sampling period using the second maximum depth value, and if it determines that the passive depth information should not be prioritized, sets the sampling period using a third maximum depth value extracted from the depth information. The information processing device according to claim 10.

12. extracting active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; generating passive depth information based on the passive information; generating depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; converting a peak due to the reference light into a pulse-like waveform; acquiring a signal corresponding to the pulse waveform as an active signal resulting from the reference light; Calculating an average signal value of the portion caused by the ambient light; acquiring the calculated average signal value as a signal value of the passive signal caused by the ambient light; 10. A computer-implemented information processing method comprising:

13. extracting active information indicating the time of flight of the reference light and passive information indicating two-dimensional image information of the subject due to ambient light from the received light data; generating passive depth information based on the passive information; generating depth information of the subject by fusing the passive depth information with active depth information generated based on the active information; converting a peak due to the reference light into a pulse-like waveform; acquiring a signal corresponding to the pulse waveform as an active signal resulting from the reference light; Calculating an average signal value of the portion caused by the ambient light; acquiring the calculated average signal value as a signal value of the passive signal caused by the ambient light; A program that makes a computer do something.

Images