Distance measurement processing device, distance measurement module and distance measurement processing method

Abstract

The present application relates to a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a computer-readable medium storing a computer program. The distance measurement processing device includes: a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between a first valve and a second valve using a predetermined number of detection signals, each of which is detected individually for each of two or more kinds of irradiation light having a predetermined phase difference, the two or more kinds of irradiation light being irradiated on an object, and an electric charge generated by reflected light received from the object being divided into the first valve and the second valve according to a distance to the object; and a distance measurement section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals. The present application can realize a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a computer-readable medium with higher performance.

Description

TECHNICAL FIELD

[0001] The present application relates to a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a computer-readable medium storing a computer program, and particularly to a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a computer-readable medium storing a computer program capable of achieving higher performance. BACKGROUND

[0002] In recent years, distance measurement modules for measuring a distance to an object have become smaller with the development of semiconductor technology. Accordingly, for example, a distance measurement module can be mounted to a mobile terminal such as a so-called smart phone, which is a small information processing device having a communication function.

[0003] Generally, distance measurement methods in a distance measurement module are classified into two types: an indirect Time of Flight (TOF) scheme and a structured light scheme. In the indirect TOF scheme, light is irradiated to an object, the light reflected on a surface of the object is detected, and a distance to the object is calculated based on a measurement value obtained by measuring a time of flight of the light. In the structured light scheme, structured light is irradiated to an object, and a distance to the object is calculated based on an image obtained by capturing a structure distortion on a surface of the object.

[0004] For example, Patent Document JP 2017-150893 A discloses a technique for accurately measuring a distance in a distance detection system using a TOF scheme by determining movement of an object within a detection period. SUMMARY

[0005] In addition, as described above, in the case where a distance measurement module is used in a mobile terminal, it is desirable to improve performance of a frame rate, power consumption, data transmission bandwidth, and the like.

[0006] Therefore, it is desirable to achieve higher performance.

[0007] According to an embodiment of the present application, there is provided a distance measurement processing device including: a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, each of two of the detection signals being detected with respect to each of two or more kinds of irradiation light having a predetermined phase difference, the two or more kinds of irradiation light being irradiated to an object, and a charge generated by reflected light reflected by the object being divided into the first gate and the second gate according to a distance to the object; and a distance measurement section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of the detection signals.

[0008] According to an embodiment of the present application, there is provided a distance measuring module including: an emission section that emits two kinds of emission light having a predetermined phase difference toward an object; a light receiving section that outputs a predetermined number of detection signals that respectively detect two detection signals for each of the two kinds of emission light, a charge generated by received reflection light reflected by the object being divided into a first gate and a second gate according to a distance to the object; a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between the first gate and the second gate using the predetermined number of detection signals; and a distance measuring section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals.

[0009] According to an embodiment of the present application, there is provided a distance measuring processing method implemented by a distance measuring processing apparatus for performing distance measuring processing, the distance measuring processing method including: calculating a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, respectively detecting two detection signals for each of two kinds of emission light having a predetermined phase difference, the two kinds of emission light being emitted toward an object, and a charge generated by received reflection light reflected by the object being divided into the first gate and the second gate according to a distance to the object; and obtaining a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals.

[0010] According to an embodiment of the present application, there is provided a computer readable medium storing a computer program for causing a computer of a distance measuring processing apparatus for performing distance measuring processing to execute distance measuring processing, the program including: calculating a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, respectively detecting two detection signals for each of two kinds of emission light having a predetermined phase difference, the two kinds of emission light being emitted toward an object, and a charge generated by received reflection light reflected by the object being divided into the first gate and the second gate according to a distance to the object; and obtaining a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals.

[0011] According to an embodiment of the present application, two or more kinds of emission light having a predetermined phase difference are emitted toward an object, a charge generated by received reflection light reflected by the object is divided into a first gate and a second gate according to a distance to the object, and a predetermined number of detection signals are respectively detected for each of the two or more kinds of emission light. Thereafter, a correction parameter for correcting a characteristic deviation between the first gate and the second gate is calculated using the predetermined number of detection signals, and a depth indicating the distance to the object is obtained based on the correction parameter and the predetermined number of detection signals.

[0012] According to the embodiment of the present application, higher performance can be achieved.

[0013] The effects described in this specification are not limiting. That is, the present application can achieve any of the effects described in this specification. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a block diagram showing a structural example of an embodiment of a distance measurement module employing the present technology;

[0015] Figure 2 is a diagram illustrating a charge species in a pixel circuit;

[0016] Figure 3 is a diagram showing an example of four kinds of irradiation light each of which is phase delayed by 90 degrees;

[0017] Figure 4 is a diagram illustrating distance measurement by four detection periods using four kinds of irradiation light each of which is phase delayed by 90 degrees;

[0018] Figure 5 is a diagram showing an example of a detection signal in a detection period using irradiation light which is phase delayed by 0 degrees;

[0019] Figure 6 is a diagram showing an example of a detection signal in a detection period using irradiation light which is phase delayed by 90 degrees;

[0020] Figure 7 is a diagram showing an example of a detection signal in a detection period using irradiation light which is phase delayed by 180 degrees;

[0021] Figure 8 is a diagram showing an example of a detection signal in a detection period using irradiation light which is phase delayed by 270 degrees;

[0022] Figure 9 is a diagram illustrating a relationship between detection signals A0 to A270 and detection signals B0 to B270;

[0023] Figure 10 is a diagram illustrating a correction operation;

[0024] Figure 11 is a diagram illustrating distance measurement using two detection periods;

[0025] Figure 12 is a block diagram showing a first structural example of a distance measurement operation processing section;

[0026] Figure 13 is a flowchart illustrating a first processing example of distance measurement operation processing;

[0027] Figure 14 is a block diagram showing a second example of the distance measurement operation processing section;

[0028] Figure 15 is a graph showing frame rate improvement using synthesis of distance measurement results;

[0029] Figure 16 is a graph showing power consumption reduction using synthesis of distance measurement results;

[0030] Figure 17 is a flowchart showing a second example of the distance measurement operation processing;

[0031] Figure 18 is a graph showing an example of timing of emitted light and received light for outputting one depth map;

[0032] Figure 19 is a graph showing a change in light emission pattern;

[0033] Figure 20 is a graph showing a change in light emission pattern;

[0034] Figure 21 is a graph showing a change in light emission pattern;

[0035] Figure 22 is a block diagram showing a third example of the distance measurement operation processing section;

[0036] Figure 23 is a graph showing synthesis of distance measurement results based on movement detection;

[0037] Figure 24 is a flowchart showing a third example of the distance measurement operation processing;

[0038] Figure 25 is a block diagram showing an example of the structure of an electronic device in which a distance measurement module is installed;

[0039] Figure 26 is a block diagram showing an example of the structure of an embodiment of a computer to which the technology according to the present disclosure is applied;

[0040] Figure 27 is a block diagram showing an example of the schematic structure of a vehicle control system; and

[0041] Figure 28 is a graph that assists in explaining an example of the installation positions of the outside-vehicle information detection section and the imaging section. DETAILED DESCRIPTION

[0042] Specific embodiments to which the technology according to the present disclosure is applied will be explained in detail below with reference to the accompanying drawings.

[0043] <Structure Example of Distance Measurement Module>

[0044] Figure 1 is a block diagram showing a structural example of an embodiment of a distance measurement module employing the present technology.

[0045] As shown in Figure 1 , the distance measurement module 11 includes a light emitting section 12, a light emitting control section 13, a light receiving section 14, and a distance measurement operation processing section 15. The distance measurement module 11, for example, irradiates light to an object, receives light (reflected light) that is light (irradiated light) reflected by the object, and measures a depth indicating a distance to the object.

[0046] The light emitting section 12 emits light while modulating the light according to a timing of a light emitting control signal provided from the light emitting control section 13 based on a control of the light emitting control section 13, and irradiates the irradiated light to the object.

[0047] The light emitting control section 13 provides the light emitting control section 12 with a light emitting control signal of a predetermined frequency (for example, 20 MHz or the like) and controls the light emitting section 12 to emit light. Also, the light emitting control section 13 provides the light receiving section 14 with a light emitting control signal according to a light emitting timing of the light emitting section 12 to drive the light receiving section 14.

[0048] The light receiving section 14 receives reflected light from the object on a sensor surface on which a plurality of pixels are arranged in an array form. Thereafter, the light receiving section 14 provides the distance measurement operation processing section 15 with image data constituted with detection signals based on light receiving amounts of the reflected light received by the respective pixels.

[0049] The distance measurement operation processing section 15 performs an operation of acquiring a depth from the distance measurement module 11 to the object based on the image data provided from the light receiving section 14. Thereafter, the distance measurement operation processing section 15 generates a depth map in which depths to the object are shown for the respective pixels and a reliability map in which reliabilities of the respective depths are shown for the respective pixels, and outputs the depth map and the reliability map to a subsequent control unit (for example, an application processing section 121 and an operating system processing section 122 or the like) not shown. Note that a specific structure of the distance measurement operation processing section 15 will be described later with reference to Figure 25 Figure 12

[0050] ​​Moreover, in the light-receiving section 14, a pixel array section 22 in which a plurality of pixel circuits 21 are arranged in an array form and a drive control circuit 23 are provided, and the drive control circuit 23 is arranged in a peripheral region of the pixel array section 22. The pixel array section 22 is a sensor surface that receives reflected light. The drive control circuit 23 outputs control signals (for example, a classification signal DIMIX, a selection signal ADDRESS DECODE, a reset signal RST, and the like, which will be described later) for controlling the drive of the pixel circuit 21, for example, based on a light emission control signal supplied from the light emission control section 13.

[0051] The pixel circuit 21 is configured so that the electric charge generated in one photodiode 31 is divided into taps 32A and 32B. Thereafter, of the electric charge generated in the photodiode 31, the electric charge divided into the tap 32A is read out from a signal line 33A and used as a detection signal A, and the electric charge divided into the tap 32B is read out from a signal line 33B and used as a detection signal B.

[0052] The tap 32A includes a transfer transistor 41A, a floating diffusion (FD) section 42A, a selection transistor 43A, and a reset transistor 44A. Similarly, the tap 32B includes a transfer transistor 41B, an FD section 42B, a selection transistor 43B, and a reset transistor 44B.

[0053] Reference will be made to Figure 2 the classification of electric charge in the pixel circuit 21.

[0054] As Figure 2 indicated, irradiation light is output from the light emission section 12, the irradiation light is modulated (1 cycle = 2T) so that ON and OFF of irradiation are repeated with an irradiation time T, and the photodiode 31 receives reflected light with a delay of a delay time T RT based on a distance to an object. Moreover, a classification signal DIMIX_A controls ON and OFF of the transfer transistor 41A, and a classification signal DIMIX_B controls ON and OFF of the transfer transistor 41B. As shown, when the classification signal DIMIX_A has the same phase as that of the irradiation light, the classification signal DIMIX_B has a phase opposite to that of DIMIX_A.

[0055] Therefore, when the transfer transistor 41A is in the ON state based on the classification signal DIMIX_A, the electric charge generated by the photodiode 31 that receives the reflected light is transferred to the FD section 42A, and when the transfer transistor 41B is in the ON state based on the classification signal DIMIX_B, the electric charge generated by the photodiode 31 that receives the reflected light is transferred to the FD section 42B. In this way, during the predetermined period in which the irradiation light is periodically irradiated within the irradiation time T, the electric charge transferred via the transfer transistor 41A is sequentially accumulated in the FD section 42A, and the electric charge transferred via the transfer transistor 41B is sequentially accumulated in the FD section 42B.

[0056] Next, after the period of charge accumulation ends, if the selection transistor 43A is in the ON state based on the selection signal ADDRESSDECODE_A, the electric charge accumulated in the FD section 42A is read out via the signal line 33A, and a detection signal A based on the amount of electric charge is output from the light-receiving section 14. Similarly, if the selection transistor 43B is in the ON state based on the selection signal ADDRESSDECODE_B, the electric charge accumulated in the FD section 42B is read out via the signal line 33B, and a detection signal B based on the amount of electric charge is output from the light-receiving section 14. Also, in the case where the reset transistor 44A is in the ON state based on the reset signal RST_A, the electric charge accumulated in the FD section 42A is discharged, and in the case where the reset transistor 44B is in the ON state based on the reset signal RST_B, the electric charge accumulated in the FD section 42B is discharged.

[0057] In this way, the pixel circuit 21 is able to obtain the distance to the object according to the delay time T RT The electric charge generated by the reflected light received by the photodiode 31 is divided into the valve 32A and the valve 32B, and the detection signal A and the detection signal B can be output. Then the delay time T RT is the time based on the period in which the light emitted by the light-emitting section 12 reaches the object and, after being reflected by the object, reaches the light-receiving section 14, that is, is the time based on the distance to the object. Therefore, the distance measurement module 11 is able to obtain the distance (depth) to the object according to the delay time T RT .

[0058] In addition, in the distance measurement module 11, the detection signal A and the detection signal B are affected differently for each pixel circuit 21 according to characteristic deviations of the respective elements such as the photodiode 31 provided in each pixel circuit 21. Therefore, generally, irradiation light having different phases is used, and the operation of eliminating the effects caused by the respective characteristic deviations is performed a plurality of times based on the detection signal A and the detection signal B detected according to the reflected light of the irradiation light having the respective phases.

[0059] Here, the operation of eliminating the effects caused by the respective characteristic deviations will be described with reference to Figures 3 to 9The detection signal is expected to eliminate the influence of the characteristic deviation between the valve 32A and the valve 32B in the related art.

[0060] For example, as shown in Figure 3 , four kinds of irradiation light whose phases are delayed by 90 degrees are used. That is, four periods (four times) for detecting the detection signal A and the detection signal B, respectively, are set using the irradiation light whose phase is delayed by 90 degrees based on the irradiation light whose phase is delayed by 0 degrees, the irradiation light whose phase is delayed by 180 degrees, and the irradiation light whose phase is delayed by 270 degrees.

[0061] That is, as shown in Figure 4 , for example, the detection period Q0 for detecting the reflected light of the irradiation light whose phase is delayed by 0 degrees, the detection period Q1 for detecting the reflected light of the irradiation light whose phase is delayed by 90 degrees, the detection period Q2 for detecting the reflected light of the irradiation light whose phase is delayed by 180 degrees, and the detection period Q3 for detecting the reflected light of the irradiation light whose phase is delayed by 270 degrees are sequentially set. Also, in the detection period Q0, the detection period Q1, the detection period Q2, and the detection period Q3, a reset period in which charges are reset, an integration period in which charges are integrated, and a read period in which charges are read out are respectively provided.

[0062] In this way, one depth frame for outputting one depth map is constituted by the detection periods including the detection period Q0, the detection period Q1, the detection period Q2, and the detection period Q3, and the subsequent waiting period (dead time / idle time). Such one depth frame is repeated, and the depth frames are continuously outputted at a predetermined frame rate, that is, the depth frame of the frame number t, the depth frame of the frame number t+1, and the depth frame of the frame number t+2.

[0063] Figure 5 An example of the irradiation light, the reflected light, the classification signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q0 is shown. As shown in Figure 5 , the charges are divided into the charges whose amounts are based on the delay time T RT , the valve 32A, and the valve 32B, and the charges are respectively integrated in the integration period. After that, in the read period, the charges whose amounts are integrated in the integration period are read out, and the detection signal A0 and the detection signal B0 in the detection period Q0 are outputted.

[0064] Figure 6 An example of the irradiation light, the reflected light, the classification signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q1 is shown. As shown in Figure 6 , the charges are divided into the charges whose amounts are based on the delay time T RTThe charges are divided into the shutter 32A and the shutter 32B whose charge amounts are based on the delay time T RT and the charges are accumulated in the integration period, respectively. Thereafter, in the read period, the charges of the amounts of charges accumulated in the integration period are read out and the detection signal A90 and the detection signal B90 in the detection period Q1 are output.

[0065] Figure 7 An example of the irradiation light, the reflected light, the classification signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q2 is shown. As shown in Figure 7 The charges are divided into the shutter 32A and the shutter 32B whose charge amounts are based on the delay time T RT and the charges are accumulated in the integration period, respectively. Thereafter, in the read period, the charges of the amounts of charges accumulated in the integration period are read out and the detection signal A180 and the detection signal B180 in the detection period Q2 are output.

[0066] Figure 8 An example of the irradiation light, the reflected light, the classification signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q3 is shown. As shown in Figure 8 The charges are divided into the shutter 32A and the shutter 32B whose charge amounts are based on the delay time T RT and the charges are accumulated in the integration period, respectively. Thereafter, in the read period, the charges of the amounts of charges accumulated in the integration period are read out and the detection signal A270 and the detection signal B270 in the detection period Q3 are output.

[0067] In this way, the detection signal A0 and the detection signal B0 are detected with the irradiation light of the phase delay of 0 degree in the detection period Q0, the detection signal A90 and the detection signal B90 are detected with the irradiation light of the phase delay of 90 degrees in the detection period Q1. Similarly, the detection signal A180 and the detection signal B180 are detected with the irradiation light of the phase delay of 180 degrees in the detection period Q2, the detection signal A270 and the detection signal B270 are detected with the irradiation light of the phase delay of 270 degrees in the detection period Q3.

[0068] Here, Figure 9 A relationship between the detection signals A0 to A270 and the detection signals B0 to B270 is shown in a case where the horizontal axis indicates the phase delay and the vertical axis indicates the signal intensity.

[0069] Thereafter, a relationship between the detection signal A0 and the detection signal B0, a relationship between the detection signal A90 and the detection signal B90, a relationship between the detection signal A180 and the detection signal B180, and a relationship between the detection signal A270 and the detection signal B270 are modeled by the following expression (1).

[0070]

[0071] By executing this model to obtain Offset, Gain, and angle θ according to expression (1), distance measurement can be achieved, for example, by eliminating the influence of characteristic deviations between valves 32A and 32B. That is, in order to eliminate the difference in Offset and Gain between valves 32A and 32B, it is desirable to utilize eight detection signals (detection signals A0 to A270 and detection signals B0 to B270) detected in four detection periods Q0 to Q3.

[0072] Thus, in related technologies, in order to achieve distance measurement that eliminates the influence of characteristic deviations between valves 32A and 32B, it has become necessary to detect detection signals A0 to A270 and detection signals B0 to B270.

[0073] Simultaneously, the distance measurement module 11 obtains the compensation and gain of valve 32A and valve 32B, and compensates for the deviation between them. In this way, by detecting detection signals A and B separately in two detection periods Q0 and Q1 (or detection periods Q2 and Q3), the distance measurement module 11 can achieve distance measurement that eliminates the influence of characteristic deviations between valves 32A and 32B.

[0074] For example, there is a relationship between Offset_A and Gain_A of valve 32A and Offset_B and Gain_B of valve 32B as shown in the following expression (2).

[0075]

[0076] Here, Offset_A and Offset_B are fixed values ​​for each pixel circuit 21 and can be obtained in advance. At the same time, since Gain_A and Gain_B fluctuate with the incident angle of light depending on the structure of the pixel circuit 21, it is necessary to calculate Gain_A and Gain_B for each depth frame.

[0077] That is, in the distance measurement module 11, detection signals A0 to A270 and detection signals B0 to B270 are detected in advance or in the initial processing of distance measurement, and Offset_A and Offset_B are obtained by solving the simultaneous equations shown in the following expression (3).

[0078]

[0079] Therefore, in the distance measurement module 11, Offset_A and Offset_B are stored as compensation parameters.

[0080] After that, in the distance measurement module 11, the gain parameter (Gain_A / Gain_B) shown by the following expression (4) can be obtained at the time points at which the detection signal A0, the detection signal B0, the detection signal A90, and the detection signal B90 are detected.

[0081]

[0082] Also, in the distance measurement module 11, the gain parameter (Gain_A / Gain_B) shown by the following expression (5) can be obtained at the time points at which the detection signal A180 and A270 and the detection signal B180 and B270 are detected.

[0083]

[0084] Therefore, in the distance measurement module 11, the correction can be performed according to the following expression (6) using the offset parameter (Offset_A, Offset_B) and the gain parameter (Gain_A / Gain_B) at the time points at which the detection signal A0, the detection signal B0, the detection signal A90, and the detection signal B90 are detected.

[0085]

[0086] Or

[0087]

[0088] In this way, in the distance measurement module 11, the corrected detection signal A'180 and A'270 can be obtained on the basis of the detection signal A, and the corrected detection signal B'180 and B'270 can be obtained on the basis of the detection signal B.

[0089] That is, as shown in Figure 10 the corrected detection signal A'180 can be obtained by the correction of the detection signal B0, and the corrected detection signal A'270 can be obtained by the correction of the detection signal B90. Or, the corrected detection signal B'180 can be obtained by the correction of the detection signal A0, and the corrected detection signal B'270 can be obtained by the correction of the detection signal A90.

[0090] Therefore, the distance measurement module 11 can achieve depth and reliability by utilizing the detection signal A0, detection signal A90, calibrated detection signal A'180, and calibrated detection signal A'270 to eliminate the influence of characteristic deviations between valves 32A and 32B. Alternatively, the distance measurement module 11 can achieve depth and reliability by utilizing the detection signal B0, detection signal B90, calibrated detection signal B'180, and calibrated detection signal B'270 to eliminate the influence of characteristic deviations between valves 32A and 32B.

[0091] Similarly, in the distance measurement module 11, at the time points when the detection signals A180, B270, and A180 and B270 are detected, the compensation parameters (Offset_A, Offset_B) and gain parameters (Gain_A / Gain_B) can be corrected according to the following expression (7).

[0092]

[0093] or

[0094]

[0095] In this way, in the distance measurement module 11, when using the detection signal A as a basis, it is possible to obtain the corrected detection signals A'0 and A'90, and when using the detection signal B as a basis, it is possible to obtain the corrected detection signals B'0 and B'90.

[0096] Therefore, the distance measurement module 11 can achieve depth and reliability by utilizing the calibrated detection signals A'0, A'90, A180, and A270 to eliminate the influence of characteristic deviations between valves 32A and 32B. Alternatively, the distance measurement module 11 can achieve depth and reliability by utilizing the calibrated detection signals B'0, B'90, B180, and B270 to eliminate the influence of characteristic deviations between valves 32A and 32B.

[0097] In this way, the distance measurement module 11 achieves distance measurement by pre-obtaining compensation parameters (Offset_A, Offset_B) and gain parameters (Gain_A / Gain_B) for each depth frame, thereby eliminating the influence of the characteristic deviation between valve 32A and valve 32B.

[0098] For example, such as Figure 11As shown, the distance measurement module 11 detects four detection signals (detection signal A0, detection signal B0, detection signal A90, and detection signal B90) in the two detection periods Q0 and Q1 and outputs a depth frame of frame number t. Next, the distance measurement module 11 detects four detection signals (detection signal A180, detection signal B180, detection signal A270, and detection signal B270) in the two detection periods Q2 and Q3 and outputs a depth frame of frame number t+1.

[0099] Therefore, compared to the distance measurement method described above with reference to Figure 4 which outputs one depth frame in the four detection periods Q0 to Q3, the distance measurement module 11 is able to reduce the time required to output one depth frame to half. That is, compared to the related art, the distance measurement module 11 is able to double the frame rate.

[0100] <Structure Example of Distance Measurement Operation Processing Section>

[0101] Figure 12 is a block diagram showing a first structure example of the distance measurement operation processing section 15.

[0102] The distance measurement operation processing section 15 outputs a depth d(t) constituting a depth map of frame number t and a reliability c(t) constituting a reliability map of frame number t using the detection signal A(t) and the detection signal B(t) provided from the light receiving section 14 as image data.

[0103] First, in a case where four detection signals (detection signal A0(t), detection signal B0(t), detection signal A90(t), and detection signal B90(t)) detected by the irradiation light of phase delay 0 degrees and 90 degrees are provided, the distance measurement operation processing section 15 outputs a depth d(t) and a reliability c(t) of frame number t. Next, in a case where four detection signals (detection signal A180(t+1), detection signal B180(t+1), detection signal A270(t+1), and detection signal B270(t+1)) detected by the irradiation light of phase delay 180 degrees and 270 degrees are provided, the distance measurement operation processing section 15 outputs a depth d(t+1) and a reliability c(t+1) constituting a depth frame of frame number t+1.

[0104] As shown in Figure 12 , the distance measurement operation processing section 15 includes a correction parameter calculation section 51 and a distance measurement section 52. The correction parameter calculation section 51 includes a bias correction parameter calculation section 61 and a bias correction parameter storage section 62 and calculates a gain parameter and a compensation parameter. The distance measurement section 52 includes a correction operation section 71 and a distance measurement operation section 72 and corrects a detection signal based on the gain parameter and the compensation parameter and obtains a depth.

[0105] For example, the deviation correction parameter calculation section 61 solves the following expression (8) for Offset_A and Offset_B in several frames at the start of distance measurement.

[0106]

[0107] In this way, the deviation correction parameter calculation section 61 obtains Offset_A and Offset_B and stores Offset_A and Offset_B in the deviation correction parameter storage section 62. Note that, for example, Offset_A and Offset_B can be obtained in advance when the distance measurement module 11 is tested, and Offset_A and Offset_B can be stored in the deviation correction parameter storage section 62 when the distance measurement module 11 is shipped.

[0108] Then, in a case where four detection signals (detection signal A0(t), detection signal B0(t), detection signal A90(t), and detection signal B90(t)) detected by irradiation light with phase delays of 0 degrees and 90 degrees are provided, the deviation correction parameter calculation section 61 calculates the following expression (9). In this way, the deviation correction parameter calculation section 61 obtains a gain parameter (Gain_A / Gain_B(t)) and provides the gain parameter to the correction operation section 71 of the distance measurement section 52.

[0109]

[0110] Next, in a case where four detection signals (detection signal A180(t+1), detection signal B180(t+1), detection signal A270(t+1), and detection signal B270(t+1)) detected by irradiation light with phase delays of 180 degrees and 270 degrees are provided, the deviation correction parameter calculation section 61 calculates the following expression (10). In this way, the deviation correction parameter calculation section 61 obtains a gain parameter (Gain_A / Gain_B(t+1)) and provides the gain parameter to the correction operation section 71 of the distance measurement section 52.

[0111]

[0112] The deviation correction parameter storage section 62 stores the compensation parameters (Offset_A, Offset_B) calculated by the deviation correction parameter calculation section 61 and provides the compensation parameters to the correction operation section 71 of the distance measurement section 52. Note that the deviation correction parameter calculation section 61 obtains the gain parameters and the compensation parameters for each pixel circuit 21, and the deviation correction parameter storage section 62 holds the compensation parameters for each pixel circuit 21.

[0113] At the time points at which the four detection signals (detection signal A0(t), detection signal B0(t), detection signal A90(t), and detection signal B90(t)) detected by the irradiation light of the phase delay of 0 degree and 90 degrees are provided, the gain parameters (Gain_A / Gain_B(t)) are supplied from the deviation correction parameter calculation section 61 to the correction operation section 71. Therefore, the correction operation section 71 can obtain the corrected detection signals A'180(t) and A'270(t) or the corrected detection signals B'180(t) and B'270(t) by performing the operation shown in the following expression (11) at these time points.

[0114]

[0115] or

[0116]

[0117] In this way, the correction operation section 71 supplies the corrected detection signals A'180(t) and A'270(t) or the corrected detection signals B'180(t) and B'270(t) at the time points at which the four detection signals detected by the irradiation light of the phase delay of 180 degrees and 270 degrees are provided to the distance measurement operation section 72.

[0118] Next, at the time points at which the four detection signals (detection signal A180(t+1), detection signal B180(t+1), detection signal A270(t+1), and detection signal B270(t+1)) detected by the irradiation light of the phase delay of 180 degrees and 270 degrees are provided, the gain parameters (Gain_A / Gain_B(t+1)) are supplied from the deviation correction parameter calculation section 61 to the correction operation section 71. Therefore, the correction operation section 71 can obtain the corrected detection signals A'0(t+1) and A'90(t+1) or the corrected detection signals B'0(t+1) and B'90(t+1) by performing the operation shown in the following expression (12) at these time points.

[0119]

[0120] or

[0121]

[0122] In this way, the correction operation section 71 supplies the corrected detection signals A'0(t+1) and A'90(t+1) or the corrected detection signals B'0(t+1) and B'90(t+1) at the time points at which the four detection signals detected by the irradiation light of the phase delay of 180 degrees and 270 degrees are provided to the distance measurement operation section 72.

[0123] At the time point at which the four detection signals (detection signal A0(t), detection signal B0(t), detection signal A90(t), and detection signal B90(t)) detected by the irradiation light of the phase delay of 0 degree and 90 degrees are provided, the corrected detection signals A'180(t) and A'270(t) or the corrected detection signals B'180(t) and B'270(t) are provided from the correction operation section 71 to the distance measurement operation section 72. Then, the distance measurement operation section 72 can obtain the depth d(t) and the reliability c(t) of the depth frame of the frame number t by performing the operation shown in the following expression (13).

[0124]

[0125] Q(t) = D1(t) - D3(t)

[0126] I(t) = D0(t) - D2(t)

[0127] However, the distance measurement operation section 72 can use one of D0(t) = A0(t), D2(t) = A'180(t), D1(t) = A90(t), D3(t) = A'270(t) and D2(t) = B'180(t), D0(t) = B0(t), D1(t) = B'270(t), D3(t) = B90(t) in the expression (13). Alternatively, the distance measurement operation section 72 can use the average of D0(t) = A0(t), D2(t) = A'180(t), D1(t) = A90(t), D3(t) = A'270(t) and D2(t) = B'180(t), D0(t) = B0(t), D1(t) = B'270(t), D3(t) = B90(t) in the expression (13).

[0128] Next, at the time point at which the four detection signals (detection signal A180(t+1), detection signal B180(t+1), detection signal A270(t+1), and detection signal B270(t+1)) detected by the irradiation light of the phase delay of 180 degrees and 270 degrees are provided, the corrected detection signals A'0(t+1) and A'90(t+1) or the corrected detection signals B'0(t+1) and B'90(t+1) are provided from the correction operation section 71 to the distance measurement operation section 72. Then, the distance measurement operation section 72 can obtain the depth d(t+1) and the reliability c(t+1) of the depth frame of the frame number t+1 by performing the operation shown in the following expression (14).

[0129]

[0130] Q(t+1) = D1(t+1) - D3(t+1)

[0131] I(t+1) = D0(t+1) - D2(t+1)

[0132] However, the distance measurement operation section 72 can use one of D2(t+1) = A180(t+1), D0(t+1) = A'0(t+1), D3(t+1) = A270(t+1), D1(t+1) = A'90(t+1), and D0(t+1) = B'0(t+1), D2(t+1) = B180(t+1), D1(t+1) = B'90(t+1), D3(t+1) = B270(t+1) in the expression (14). Alternatively, the distance measurement operation section 72 can use the average of D0(t+1) = A0(t+1), D2(t+1) = A'180(t+1), D1(t+1) = A90(t+1), D3(t+1) = A'270(t+1) and D2(t+1) = B'180(t+1), D0(t+1) = B0(t+1), D1(t+1) = B'270(t+1), D3(t+1) = B90(t+1) in the expression (14).

[0133] The distance measurement operation processing section 15 having the structure as described above can obtain a depth from four detection signals detected by irradiation light of phase delay 0 degree and 90 degrees, or can obtain a depth from four detection signals detected by irradiation light of phase delay 180 degrees and 270 degrees. Therefore, for example, compared to a case where a depth is obtained from eight detection signals in the related art, it is possible to double the frame rate.

[0134] Further, with this distance measurement operation processing section 15, since it is only necessary to emit irradiation light twice without increasing the frame rate, it is possible to reduce the power consumption compared to a case where irradiation light is emitted four times in the related art. Further, with this distance measurement operation processing section 15, since it is possible to reduce the detection signals required for outputting one depth frame to half of the related art, it is possible to reduce the data transmission bandwidth.

[0135] Therefore, the distance measurement module 11 including the distance measurement operation processing section 15 can achieve higher performance than the related art.

[0136] <First processing example of distance measurement operation processing>

[0137] Figure 13 is a flowchart illustrating a first processing example of distance measurement operation processing performed in the distance measurement operation processing section 15.

[0138] For example, in a case where the control is performed by the unillustrated higher-level control unit, the processing is started to perform the distance measurement operation processing. In step Sll, the distance measurement operation processing section 15 acquires two detection signals each from each of the two kinds of irradiation light different in phase delay. That is, for example, the distance measurement operation processing section 15 acquires two detection signals A0 and detection signal B0 detected by the irradiation light of 0-degree phase delay, and two detection signals A90 and detection signal B90 detected by the irradiation light of 90-degree phase delay. Or, for example, the distance measurement operation processing section 15 acquires two detection signals A180 and detection signal B180 detected by the irradiation light of 180-degree phase delay, and two detection signals A270 and detection signal B270 detected by the irradiation light of 270-degree phase delay.

[0139] In step S12, the deviation correction parameter calculating section 61 judges whether or not the compensation parameters (Offset_A, Offset_B) are already stored in the deviation correction parameter storage section 62.

[0140] When the deviation correction parameter calculating section 61 judges in step S12 that the compensation parameters (Offset_A, Offset_B) are not stored in the deviation correction parameter storage section 62, the processing proceeds to step S13.

[0141] In step S13, the deviation correction parameter calculating section 61 judges whether or not two detection signals each from each of the four kinds of irradiation light different in phase delay are acquired, which are expected to be used for the calculation of the compensation parameters (Offset_A, Offset_B). For example, in a case where eight detection signals of detection signals A0 to A270 and detection signals B0 to B270 are acquired, the deviation correction parameter calculating section 61 judges that two detection signals each from each of the four kinds of irradiation light different in phase delay are acquired.

[0142] When the deviation correction parameter calculating section 61 judges in step S13 that two detection signals each from each of the four kinds of irradiation light different in phase delay are not acquired, the processing returns to step Sll. For example, in this case, detection signals A0 and A90 and detection signals B0 and B90 are acquired, and the deviation correction parameter calculating section 61 acquires detection signals A180 and A270 and detection signals B180 and B270 in the next step Sll.

[0143] Meanwhile, when the deviation correction parameter calculating section 61 judges in step S13 that two detection signals each from each of the four kinds of irradiation light different in phase delay are acquired, the processing proceeds to step S14.

[0144] In step S14, the deviation correction parameter calculating section 61 calculates Offset_A and Offset_B by solving the simultaneous equations shown in the above expression (3).

[0145] Then, after the deviation correction parameter calculating section 61 stores Offset_A and Offset_B in the deviation correction parameter storage section 62, the process proceeds to step S15. Meanwhile, when the deviation correction parameter calculating section 61 determines in step S12 that the compensation parameters (Offset_A, Offset_B) are already stored in the deviation correction parameter storage section 62, the process proceeds to step S15.

[0146] In step S15, the deviation correction parameter calculating section 61 calculates the gain parameters (Gain_A / Gain_B) in accordance with the above expression (4) or expression (5). Then, the deviation correction parameter calculating section 61 supplies the calculated gain parameters (Gain_A / Gain_B) to the correction operation section 71, and the deviation correction parameter storage section 62 supplies the stored compensation parameters (Offset_A, Offset_B) to the correction operation section 71.

[0147] In step S16, the correction operation section 71 performs a correction operation on the four detection signals acquired in step S11 to obtain four corrected detection signals, and supplies the four corrected detection signals to the distance measurement operation section 72.

[0148] For example, when the detection signals A0 and A90 and the detection signals B0 and B90 are acquired in step S11, the correction operation section 71 performs a correction operation in accordance with the above expression (6) to obtain the corrected detection signals A'180 and A'270 or the corrected detection signals B'180 and B'270. Meanwhile, when the detection signals A180 and A270 and the detection signals B180 and B270 are acquired in step S11, the correction operation section 71 performs a correction operation in accordance with the above expression (7) to obtain the corrected detection signals A'0 and A'90 or the corrected detection signals B'0 and B'90.

[0149] In step S17, the distance measurement operation section 72 calculates the depth and the reliability using the four detection signals acquired in step S11 and the four corrected detection signals obtained by the correction operation in step S16.

[0150] For example, assume that the detection signals A0 and A90 and the detection signals B0 and B90 are obtained in step Sll, and the corrected detection signals A'180 and A'270 or the corrected detection signals B'180 and B'270 are obtained in step S16. In this case, the distance measurement operation section 72 calculates the depth and the reliability by performing the operation shown in the above expression (13). Also, assume that the detection signals A180 and A270 and the detection signals B180 and B270 are obtained in step Sll, and the corrected detection signals A'0 and A'90 or the corrected detection signals B'0 and B'90 are obtained in step S16. In this case, the distance measurement operation section 72 calculates the depth and the reliability by performing the operation shown in the above expression (14).

[0151] In step S18, the distance measurement operation processing section 15 determines whether to continue the distance measurement, for example, based on the control of the distance measurement operation processing by an unillustrated higher-level control unit.

[0152] In step S18, in the case where the distance measurement operation processing section 15 determines to continue the distance measurement, the processing returns to step Sll, and the similar processing is repeated thereafter. Meanwhile, in the case where the distance measurement operation processing section 15 determines not to continue the distance measurement in step S18, the distance measurement operation processing ends.

[0153] As described above, the distance measurement operation processing section 15 can calculate the depth and the reliability by obtaining the detection signals A0 and A90 and the detection signals B0 and B90 or the detection signals A180 and A270 and the detection signals B180 and B270. Therefore, the distance measurement operation processing section 15 can shorten the time required for detecting the detection signals necessary for calculating the depth and the reliability, and can improve the robustness, for example.

[0154] <Second example of distance measurement operation processing section>

[0155] Figure 14 is a block diagram showing a second example of the distance measurement operation processing section 15. Here, it is noted that, as for the same components as those of the distance measurement operation processing section 15 shown in Figure 14 , the same reference numerals are used to denote the same components, and detailed description thereof is omitted. Figure 12

[0156] That is, the distance measurement operation processing section 15A includes the correction parameter calculation section 51 and the distance measurement section 52A, and is similar to the distance measurement operation processing section 15 in Figure 12 , the correction parameter calculation section 51 includes the deviation correction parameter calculation section 61 and the deviation correction parameter storage section 62.

[0157] ​Although similar to the distance measurement operation processing section 15 in Figure 12 The distance measurement section 52A includes a correction operation section 71 and a distance measurement operation section 72, but differs from the distance measurement operation processing section 15 in Figure 12

[0158] Moreover, in the distance measurement section 52A, the depth d(t) and the reliability c(t) obtained by the distance measurement operation section 72 as described above are supplied to the distance measurement result storage section 73 and the result synthesis section 74 as distance measurement results. Thereafter, in the distance measurement section 52A, the depth d(t-1) and the reliability c(t-1) of the frame one frame before the frame are supplied from the distance measurement result storage section 73 to the result synthesis section 74 as distance measurement results.

[0159] The distance measurement result storage section 73 is capable of storing the depth d(t) and the reliability c(t) corresponding to one frame supplied from the distance measurement operation section 72, and supplying the depth d(t-1) and the reliability c(t-1) of the frame one frame before the frame to the result synthesis section 74.

[0160] The result synthesis section 74 synthesizes the depth d(t) and the reliability c(t) supplied from the distance measurement operation section 72 and the depth d(t-1) and the reliability c(t-1) supplied from the distance measurement result storage section 73, and outputs the resultant depth d(t) and reliability c(t) as a synthesis result.

[0161] Here, the depth d(t) and the reliability c(t) supplied from the distance measurement operation section 72 to the distance measurement result storage section 73 and the result synthesis section 74 will be denoted as a depth d'(t) and a reliability c'(t), and the synthesis result synthesized by the result synthesis section 74 will be denoted as a depth d(t) and a reliability c(t). In this case, the result synthesis section 74 is capable of synthesizing the distance measurement results using a weighted operation shown by the following expression (15) using a weight g based on the reliability c'(t).

[0162]

[0163]

[0164] Thus, in the distance measurement operation processing section 15A, by the synthesis of the distance measurement result of the current frame and the distance measurement result of the frame one frame before the current frame (hereinafter also referred to as a slide window), it is possible to improve the signal-to-noise (SN) ratio and reduce the noise in the synthesis result.

[0165] ​For example, when the detection periods Q0 to Q3 are the same as those without a sliding window, the SN ratio of the distance measurement results using the four detection signals detected in the two detection periods Q0 and Q1 is lower than the SN ratio of the distance measurement results using the eight detection signals detected in the four detection periods Q0 to Q3. Therefore, in the distance measurement calculation processing unit 15A, since distance measurement is performed using eight detection signals, including the detection signal from the previous frame, by executing a sliding window, the decrease in the SN ratio can be suppressed.

[0166] Furthermore, in the distance measurement processing unit 15A, even if the detection period in a depth frame is shortened, the SN ratio (frame×SNR / power) of the detection signal obtained in a depth frame can be improved by executing a sliding window.

[0167] Therefore, since the distance measurement calculation processing unit 15A can reduce noise by executing a sliding window, thus... Figure 15 As shown, the detection period from Q0 to Q3 can be shortened to... Figure 4 The detection time is half of the time. That is, the distance measurement processing unit 15A can double the acquisition speed of detection signals A and B and double the frame rate.

[0168] Here, for example, without executing a sliding window, if the power required to acquire the detection signal is not changed for a depth frame and the frame rate is doubled, the SN ratio decreases according to the degree to which the detection period Q0 to Q3 is shortened. In contrast, in the distance measurement calculation processing unit 15A, even if the frame rate is doubled without changing the power required to acquire the detection signal in a depth frame, the decrease in the SN ratio can be avoided by executing a sliding window.

[0169] Or, such as Figure 16 As shown, the frame rate remained unchanged during the detection period Q0 to Q3 (compared to...). Figure 4 Under the condition that the time period is the same and the SN ratio is not changed, the power required to acquire the detection signal in a depth frame can be reduced. That is, the distance measurement arithmetic processing unit 15A can achieve low power consumption by executing a sliding window.

[0170] It should be noted that in the distance measurement calculation processing unit 15A, for example, the result synthesis unit 74 can synthesize the distance measurement result by simple averaging, or by weighting based on a reference other than reliability and by performing weighting calculation based on reliability.

[0171] Furthermore, for example, the processing of distance measurement results synthesized by the result synthesis unit 74 can also be applied to the above-mentioned reference. Figure 4The structure of outputting one depth frame by four detection periods Q0 to Q3. That is, the application of the processing is not limited to the structure of outputting one depth frame based on four detection signals detected by two detection periods Q0 and Q1 or four detection signals detected by two detection periods Q2 and Q3. Further, for example, a sliding window can be executed so as to synthesize the acquired three detection signals and one detection signal of a newly acquired frame for each of the four detection periods Q0 to Q3.

[0172] <Second processing example of distance measurement operation processing>

[0173] Figure 17 is a flowchart illustrating a second processing example of the distance measurement operation processing executed in the distance measurement operation processing section 15A.

[0174] In steps S21 to S27, similar processing to steps S11 to S17 in Figure 13 is performed.

[0175] Then, in step S27, the calculated depth and reliability are supplied to the distance measurement result storage section 73 and the result synthesizing section 74, and in step S28, the result synthesizing section 74 judges whether or not a distance measurement result is stored in the distance measurement result storage section 73.

[0176] When the result synthesizing section 74 judges in step S28 that a distance measurement result is not stored in the distance measurement result storage section 73, the processing returns to step S21. That is, in this case, the depth and reliability of the previous frame are not stored in the distance measurement result storage section 73, and the result synthesizing section 74 does not perform the processing of synthesizing distance measurement results.

[0177] Meanwhile, when the result synthesizing section 74 judges in step S28 that a distance measurement result is stored in the distance measurement result storage section 73, the processing proceeds to step S29.

[0178] In step S29, the result synthesizing section 74 reads out the depth and reliability of the previous frame from the distance measurement result storage section 73. Thereafter, the result synthesizing section 74 outputs a synthesized distance measurement result obtained by synthesizing the depth and reliability of the previous frame read out from the distance measurement result storage section 73 and the measurement result of the depth and reliability supplied in step S27 by a weighted operation based on the reliability.

[0179] Thereafter, in step S30, similar processing to step S18 in Figure 13 is performed, and in the case where it is judged that the distance measurement is not continued, the distance measurement operation processing is ended.

[0180] As described above, the distance measurement operation processing section 15A can suppress a decrease in the SN ratio of the measurement result by synthesizing the measurement results through the weighting operation based on reliability, and can perform distance measurement with higher accuracy. Also, the distance measurement operation processing section 15A can increase the frame rate (see Figure 15 ) or reduce power consumption (see Figure 16 ).

[0181] <Operation of the light emitting section and the light receiving section>

[0182] The operation of the light emitting section 12 and the light receiving section 14 will be described with reference to Figures 18 to 21

[0183] Figure 18 An example of the timing of the light emission and the light reception for outputting one depth map is shown.

[0184] For example, the distance measurement module 11 can set one frame for outputting one depth map as one subframe, and divide one subframe into four detection periods of the detection period Q0, the detection period Ql, the detection period Q2, and the detection period Q3. Also, in the respective integration periods of the detection period Q0, the detection period Ql, the detection period Q2, and the detection period Q3, the light emitting section 12 emits irradiation light in accordance with the timing of the modulation signal, and the light receiving section 14 receives reflected light. As described with reference to Figure 1 , the electric charges generated in one photodiode 31 are divided into the valve 32A and the valve 32B in accordance with the classification signals DIMIX_A and DIMIX_B, and the electric charges based on the amount of light received in the integration period are accumulated.

[0185] Here, in the example shown in Figure 4 , a waiting period corresponding to one depth frame is provided after the detection period Q0, the detection period Ql, the detection period Q2, and the detection period Q3. In contrast, in the example shown in Figure 18 , a waiting period divided into four periods is provided after the detection period Q0, the detection period Ql, the detection period Q2, and the detection period Q3, respectively.

[0186] In this way, by providing a waiting period for each of the detection period Q0, the detection period Ql, the detection period Q2, and the detection period Q3, it is possible to make the intervals of the respective integration periods equal.

[0187] That is, as Figure 19 ​As shown, the light emission timing of the irradiation light of the phase delay 0 degrees, the light emission timing of the irradiation light of the phase delay 90 degrees, the light emission timing of the irradiation light of the phase delay 180 degrees, and the light emission timing of the irradiation light of the phase delay 270 degrees are arranged at equal intervals. In this way, by adopting the light emission timings at equal intervals, for example, when a sliding window is executed in the distance measurement operation processing section 15A, it is possible to suppress the negative effects due to the difference in the light emission timings.

[0188] Moreover, as shown in FIG. 17, the light emission timing of the irradiation light of the phase delay 0 degrees, the light emission timing of the irradiation light of the phase delay 90 degrees, the light emission timing of the irradiation light of the phase delay 180 degrees, and the light emission timing of the irradiation light of the phase delay 270 degrees can be adopted. As described above, the distance measurement operation processing section 15 acquires one depth frame from the four detection signals of the detection signal A0, the detection signal B0, the detection signal A90, and the detection signal B90, and acquires one depth frame from the four detection signals of the detection signal A180, the detection signal B180, the detection signal A270, and the detection signal B270. Figure 20

[0189] Therefore, as shown in FIG. 16, preferably, the light emission timing of the irradiation light of the phase delay 0 degrees and the light emission timing of the irradiation light of the phase delay 90 degrees for acquiring one depth frame are close to each other, and the light emission timing of the irradiation light of the phase delay 180 degrees and the light emission timing of the irradiation light of the phase delay 270 degrees for acquiring the next depth frame are close to each other. For example, by making the light emission timings for acquiring one depth frame close to each other, in the case where the object moves, it is possible to suppress the influence of the longer interval between the light emission timings due to the movement. Figure 20

[0190] Moreover, when a sliding window is executed in the distance measurement operation processing section 15A, since the light emission timings for acquiring one depth frame and the light emission timings for acquiring the next depth frame are arranged at equal intervals, it is possible to suppress the negative effects due to the difference in the intervals.

[0191] Moreover, as shown in FIG. 17, the light emission timing of the irradiation light of the phase delay 0 degrees, the light emission timing of the irradiation light of the phase delay 90 degrees, the light emission timing of the irradiation light of the phase delay 180 degrees, and the light emission timing of the irradiation light of the phase delay 270 degrees can be adopted. As described above, the distance measurement operation processing section 15 acquires one depth frame from the four detection signals of the detection signal A0, the detection signal B0, the detection signal A90, and the detection signal B90, and acquires one depth frame from the four detection signals of the detection signal A180, the detection signal B180, the detection signal A270, and the detection signal B270. Figure 21

[0192] Note that the light emission timings of the light emission section 12 are not limited to the example shown in FIG. 16, and various other light emission timings can be adopted. Figures 18 to 21

[0193] <Third structure example of distance measurement operation processing section>

[0194] Figure 22 is a block diagram showing a third structure example of the distance measurement operation processing section 15.

[0195] Figure 22 ​​​​The distance measurement processing unit 15B shown includes a detection signal storage unit 81, a motion detection unit 82, a four-phase distance measurement processing unit 83, a two-phase distance measurement processing unit 84, a distance measurement result storage unit 85, and a result synthesis unit 86.

[0196] Moreover, similarly, as referenced Figure 12 The distance measurement processing unit 15B is provided with four detection signals detected by illumination light with a phase delay of 0 degrees and illumination light with a phase delay of 90 degrees, and four detection signals detected by illumination light with a phase delay of 180 degrees and illumination light with a phase delay of 270 degrees. That is, detection signals A0(t), B0(t), A90(t), and B90(t) are provided to the distance measurement processing unit 15B, and detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1) are continuously provided to the distance measurement processing unit 15B.

[0197] The detection signal storage unit 81 can store four detection signals and provide the previously stored four detection signals to the motion detection unit 82, providing four detection signals each time.

[0198] That is, the detection signal storage unit 81 stores detection signals A180(t-1), B180(t-1), A270(t-1), and B270(t-1), and provides these detection signals to the motion detection unit 82 at the time points when detection signals A0(t), B0(t), A90(t), and B90(t) are provided. Furthermore, the detection signal storage unit 81 stores detection signals A0(t), B0(t), A90(t), and B90(t), and provides these detection signals to the motion detection unit 82 at the time points when detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1) are provided.

[0199] The motion detection unit 82 detects the movement of an object for each pixel of the light receiving unit 14 and determines whether a moving object has appeared based on a predetermined threshold th.

[0200] That is, the motion detection unit 82 makes a judgment based on the judgment condition shown in the following expression (16) at the time points when the detection signal A0(t), detection signal B0(t), detection signal A90(t) and detection signal B90(t) are provided.

[0201]

[0202] For example, in a case where the determination condition of the expression (16) is satisfied, the movement detection section 82 determines that a moving object does not appear in the depth frame acquired on the basis of the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t). In this case, the movement detection section 82 outputs the moving object detection signal M(t) = 0 indicating that the moving object does not appear, and supplies the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t) to the four-phase distance measurement operation section 83. Also, the movement detection section 82 supplies the detection signal A180(t - 1), the detection signal B180(t - 1), the detection signal A270(t - 1), and the detection signal B270(t - 1) supplied from the detection signal storage section 81 to the four-phase distance measurement operation section 83.

[0203] Meanwhile, in a case where the determination condition of the expression (16) is not satisfied, the movement detection section 82 determines that a moving object appears in the depth frame acquired on the basis of the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t). In this case, the movement detection section 82 outputs the moving object detection signal M(t) = 1 indicating that the moving object appears, and supplies the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t) to the two-phase distance measurement operation section 84.

[0204] Similarly, the movement detection section 82 makes a determination in accordance with a determination condition shown in the following expression (17) at a time point at which the detection signal A180(t + 1), the detection signal B180(t + 1), the detection signal A270(t + 1), and the detection signal B270(t + 1) are supplied.

[0205]

[0206] For example, in a case where the determination condition of the expression (17) is satisfied, the movement detection section 82 determines that a moving object does not appear in the depth frame acquired on the basis of the detection signal A180(t + 1), the detection signal B180(t + 1), the detection signal A270(t + 1), and the detection signal B270(t + 1). In this case, the movement detection section 82 outputs the moving object detection signal M(t) = 0 indicating that the moving object does not appear, and supplies the detection signal A180(t + 1), the detection signal B180(t + 1), the detection signal A270(t + 1), and the detection signal B270(t + 1) to the four-phase distance measurement operation section 83. Also, the movement detection section 82 supplies the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t) supplied from the detection signal storage section 81 to the four-phase distance measurement operation section 83.

[0207] Meanwhile, in a case where the determination condition of the expression (17) is not satisfied, the movement detection section 82 determines that a moving object appears in the depth frame acquired based on the detection signal A180(t+1), the detection signal B180(t+1), the detection signal A270(t+1), and the detection signal B270(t+1). In this case, the movement detection section 82 outputs the moving object detection signal M(t) = 1 indicating that the moving object appears, and supplies the detection signal A180(t+1), the detection signal B180(t+1), the detection signal A270(t+1), and the detection signal B270(t+1) to the two-phase distance measurement operation section 84.

[0208] In a case where it is determined by the movement detection section 82 that the moving object does not appear, the four-phase distance measurement operation section 83 performs the distance measurement processing (hereinafter referred to as four-phase distance measurement operation processing) by an operation using eight detection signals of the irradiation light of the phase delay 0 degree, the irradiation light of the phase delay 90 degrees, the irradiation light of the phase delay 180 degrees, and the irradiation light of the phase delay 270 degrees.

[0209] For example, in this case, the detection signal A180(t-1), the detection signal B180(t-1), the detection signal A270(t-1), the detection signal B270(t-1), and the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), and the detection signal B90(t) are supplied from the movement detection section 82 to the four-phase distance measurement operation section 83.

[0210] Therefore, the four-phase distance measurement operation section 83 acquires the depth d(t) and the reliability c(t) by an operation according to the following expression (18), and supplies the depth d(t) and the reliability c(t) to the distance measurement result storage section 85 and the result synthesis section 86.

[0211]

[0212] Q(t) = D1(t) - D3(t)

[0213] I(t) = D0(t) - D2(t)

[0214] D0(t) = A0(t) - B0(t)

[0215] D1(t) = A90(t) - B90(t)

[0216] D2(t) = A180(t-1) - B180(t-1)

[0217] D3(t) = A270(t-1) - B270(t-1)

[0218] Similarly, the four-phase distance measurement operation section 83 is able to obtain the depth d(t+1) and the reliability c(t+1) using the detection signal A0(t), the detection signal B0(t), the detection signal A90(t), the detection signal B90(t), the detection signal A180(t+1), the detection signal B180(t+1), the detection signal A270(t+1), and the detection signal B270(t+1).

[0219] The two-phase distance measurement operation section 84 has the same function as the distance measurement operation processing section 15 in Figure 12 and includes the correction parameter calculation section 51 and the distance measurement section 52 shown in Figure 12

[0220] That is, in a case where the moving object is determined to be present by the movement detection section 82, the two-phase distance measurement operation section 84 performs the distance measurement processing by the operation of four detection signals detected by the irradiation light of the phase delay 0 degrees and the irradiation light of the phase delay 90 degrees or four detection signals detected by the irradiation light of the phase delay 180 degrees and the irradiation light of the phase delay 270 degrees (hereinafter referred to as two-phase distance measurement operation processing). Thereafter, the two-phase distance measurement operation section 84 supplies the depth d and the reliability c obtained by the two-phase distance measurement operation processing to the distance measurement result storage section 85 and the result synthesis section 86.

[0221] The distance measurement result storage section 85 and the result synthesis section 86 have the same function as the distance measurement result storage section 73 and the result synthesis section 74 in Figure 14 . That is, the distance measurement result storage section 85 supplies the distance measurement result of the previous frame to the result synthesis section 86, and the result synthesis section 86 is able to synthesize the distance measurement result of the current frame with the distance measurement result of the previous frame of the current frame.

[0222] Thus, as shown in Figure 23 , the distance measurement operation processing section 15B is able to synthesize two continuous depth frames according to the result of the movement detection of each frame and output the synthesized frame as one depth frame.

[0223] ​For example, in a case where it is determined that a moving object is present at the time point at which the depth frame of frame number t is output, based on the distance measurement result of the frame of frame number t-1 as the moving detection result before the distance measurement results are synthesized, the distance measurement operation processing section 15B outputs the measurement result of the frame of frame number t as is as the depth frame. Meanwhile, in a case where it is determined that a moving object is not present at the time point at which the depth frame of frame number t is output, based on the depth frame of frame number t-1 as the moving detection result before the distance measurement results are synthesized, the distance measurement operation processing section 15B outputs the synthesized distance measurement result obtained by synthesizing the distance measurement results of the frame of frame number t-1 as the depth frame of frame number t. Note that the moving detection section 82 can perform moving detection for each pixel, and can switch the processing between the four-phase distance measurement operation section 83 and the two-phase distance measurement operation section 84 for each pixel, and switch the processing in this manner between the four-phase distance measurement operation section 83 and the two-phase distance measurement operation section 84.

[0224] As described above, the distance measurement operation processing section 15B can switch the four-phase distance measurement operation processing and the two-phase distance measurement operation processing according to the result of the moving detection. Therefore, the distance measurement operation processing section 15B can improve the measurement accuracy for a moving object by obtaining a depth frame at a higher frame rate (for example, in a case where the two-phase distance measurement operation processing is performed when a moving object is present). In this way, the distance measurement operation processing section 15B can improve the robustness to a moving object. Also, in a case where a moving object is not present, the distance measurement operation processing section 15B can achieve low noise by performing the four-phase distance measurement operation processing. Note that the moving detection section 82 can switch between the four-phase distance measurement operation section 83 and the two-phase distance measurement operation section 84 by performing conditional determination based on the luminance obtained from the detection signals or by performing conditional determination based on the reliability of the previous frame of the frame.

[0225] <Third Processing Example of Distance Measurement Operation Processing>

[0226] Figure 24 is a flowchart illustrating a third processing example of the distance measurement operation processing performed in the distance measurement operation processing section 15B.

[0227] In step S41, processing similar to step S11 in Figure 13 In step S41, the distance measurement operation processing section 15B acquires two detection signals using each of the two kinds of irradiation light different in phase delay.

[0228] In step S41, the moving detection section 82 determines whether or not a detection signal has been stored in the detection signal storage section 81.

[0229] In step S41, in a case where the movement detection section 82 determines that the detection signal is not stored in the detection signal storage section 81, the process returns to step S41. That is, in this case, the detection signal of the previous frame is not stored in the detection signal storage section 81, and the movement detection section 82 does not perform the process of detecting movement.

[0230] Meanwhile, in step S41, in a case where the movement detection section 82 determines that the detection signal is stored in the detection signal storage section 81, the process proceeds to step S43. In step S43, the movement detection section 82 determines whether a moving object is present or not in accordance with the determination condition shown in Expression (16) or Expression (17) described above.

[0231] In step S43, in a case where the movement detection section 82 determines that a moving object is not present, the process proceeds to step S44. In step S44, as described above, the quad-phase distance measurement operation section 83 obtains a depth and a reliability by performing the quad-phase distance measurement operation processing, and supplies the depth and the reliability to the distance measurement result storage section 85 and the result synthesizing section 86 as a distance measurement result, and the process proceeds to step S46.

[0232] Meanwhile, in step S43, in a case where the movement detection section 82 determines that a moving object is present, the process proceeds to step S45. In step S45, as described above, the dual-phase distance measurement operation section 84 obtains a depth and a reliability by performing the dual-phase distance measurement operation processing, and supplies the depth and the reliability to the distance measurement result storage section 85 and the result synthesizing section 86 as a distance measurement result, and the process proceeds to step S46.

[0233] In steps S46 to S48, a process similar to steps S28 to S30 in Figure 17 is performed, and in a case where it is determined in step S48 that the distance measurement is not to be continued, the distance measurement operation processing is ended.

[0234] As described above, by switching between the quad-phase distance measurement operation processing and the dual-phase distance measurement operation processing in accordance with the result of movement detection, the distance measurement operation processing section 15B is able to perform appropriate distance measurement on a moving object.

[0235] Note that the present technology can be applied to a scheme for adjusting the amplitude of light projected onto an object, which is called a Continuous-Wave scheme in an indirect ToF scheme. Also, the structure of the photodiode 31 of the light receiving section 14 can be applied to a depth sensor having a structure in which charges are divided into two gates (gate 32A and gate 32B), and to a depth sensor having a current assisted photonic demodulator (CAPD) structure.

[0236] Also, as the irradiation light irradiated from the distance measurement module 11 toward an object, irradiation light other than the four kinds of irradiation light whose phases are each delayed by 90 degrees as described above can be used, and any number of detection signals other than four can be used for distance measurement depending on the type of irradiation light. Also, if the influence of the characteristic deviation between the gate 32A and the gate 32B can be eliminated, a parameter other than the compensation parameter and the gain parameter can be used as a parameter for correction operation.

[0237] <Structure example of electronic device>

[0238] For example, the distance measurement module 11 as described above can be mounted on an electronic device such as a smartphone.

[0239] Figure 25 is a block diagram showing a structure example of a camera mounted on an electronic device.

[0240] As shown in Figure 25 , in the electronic device 101, the distance measurement module 102, the camera 103, the display 104, the speaker 105, the microphone 106, the communication module 107, the sensor unit 108, the touch panel 109, and the control unit 110 are connected via a bus 111. Also, the control unit 110 has a function as an application processing section 121 and an operating system processing section 122 by executing a program by a CPU.

[0241] Figure 1 The distance measurement module 11 in the electronic device 101 can be used as the distance measurement module 102. For example, the distance measurement module 102 is arranged on the front side of the electronic device 101, and by performing distance measurement targeting a user of the electronic device 101, the depth of the surface shape of the face, the hand, the finger, or the like of the user can be output.

[0242] The camera 103 is arranged on the front side of the electronic device 101, and acquires an image in which a user appears by capturing an image of the user of the electronic device 101 as a target. Note that although not shown, a structure in which the camera 103 is arranged on the back side of the electronic device 101 can also be adopted.

[0243] The display 104 displays an operation screen processed by the application processing section 121 and the operating system processing section 122, an image acquired by the imaging device 103, and the like. The speaker 105 and the microphone 106 output, for example, a speech of the other end and collect a speech of the user when a call is made using the electronic device 101.

[0244] The communication module 107 communicates via a communication network. The sensor unit 108 senses a speed, an acceleration, a proximity, and the like, and the touch panel 109 acquires a touch operation of the user on an operation screen displayed on the display 104.

[0245] The application processing section 121 performs processing for providing various services by the electronic device 101. For example, the application processing section 121 can perform processing of forming a face using a computer graphic in which an expression of the user is virtually reproduced based on the depth provided from the distance measuring module 102 and processing of displaying the face on the display 104. Also, the application processing section 121 can perform, for example, processing of forming three-dimensional shape data of an arbitrary three-dimensional object based on the depth provided from the distance measuring module 102.

[0246] The operating system processing section 122 performs processing for realizing a basic function and an operation of the electronic device 101. For example, the operating system processing section 122 can perform processing of recognizing a face of the user and processing of releasing a lock of the electronic device 101 based on the depth provided from the distance measuring module 102. Also, the operating system processing section 122 can perform, for example, processing of recognizing a posture of the user based on the depth provided from the distance measuring module 102 and processing of inputting various operations according to the posture.

[0247] In the electronic device 101 having such a structure, for example, improvement of a frame rate, reduction of power consumption, and reduction of a data transmission bandwidth can be realized by employing the distance measuring module 11 described above. In this way, the electronic device 101 can form a face moving more smoothly using a computer graphic, can recognize the face with high precision, can suppress power consumption of a battery, or can perform data transmission with a narrower bandwidth.

[0248] <Structure Example of Computer>

[0249] The series of processes described above can be executed by hardware or software. In the case where the series of processes is executed by software, a program constituting the software is installed on a general-purpose computer or the like.

[0250] Figure 26 is a block diagram showing a structure example of an embodiment of a computer on which a program that performs the series of processes described above is installed.

[0251] In the computer, a central processing unit (CPU) 201, a read only memory (ROM) 202, a random access memory (RAM) 203, and an electrically erasable programmable read only memory (EEPROM) 204 are connected to each other through a bus 205. An input / output interface 206 is also connected to the bus 205, and the input / output interface 206 is connected to the outside.

[0252] In the computer having the above structure, the above series of processes are performed by, for example, the CPU 201 loading a program stored in the ROM 202 and the EEPROM 204 onto the RAM 203 via the bus 205 and executing the program. Also, the program executed by the computer (CPU 201) can be installed in the EEPROM 204 from the outside via the input / output interface 206, or can be uploaded and written into the ROM 202 in advance.

[0253] In this way, the CPU 201 performs processing according to the above flow or performs processing performed by the structure of the above block diagram. Thereafter, for example, the CPU 201 can output a processing result to the outside as needed via the input / output interface 206.

[0254] Here, in the present specification, processing steps performed by a computer according to a program are not always necessarily executed chronologically in the order described in the flow. That is, processing performed by a computer according to a program includes processing in parallel or individually (for example, parallel processing or object-based processing).

[0255] Also, as for the program, processing can be executed by one computer (one processor), or processing can be distributed and executed by a plurality of computers. In addition, the program can be transferred to a remote computer and executed.

[0256] Also, in the present specification, the system means a group of a plurality of constituent elements (such as devices or modules (parts)) and does not consider whether all the constituent elements are in the same housing. Therefore, the system is either a plurality of devices stored in separate housings and connected through a network, or a device in which a plurality of modules are stored in a single housing.

[0257] Also, for example, an element described as a single device (or processing unit) can be divided or configured into a plurality of devices (or processing units). Conversely, elements described as a plurality of devices (or processing units) above can be configured to be integrated as a single device (or processing unit). Also, elements other than those described above can be added to the structure of each device (or processing unit). Also, a part of the structure of a given device (or processing unit) can be included in the structure of another device (or another processing unit) as long as the structure or the operation of the system as a whole is substantially the same.

[0258] Further, for example, the technology of the present application can employ a cloud computing structure that processes by distributing and sharing one function using a plurality of devices through a network.

[0259] Further, for example, the above-described program can be executed in any device. In this case, it is enough if the device has the necessary function (functional module or the like) and can obtain the necessary information.

[0260] Further, for example, each step described by the above-described flowchart can be executed by one device or by being distributed to a plurality of devices. Also, in a case where a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or by being distributed to a plurality of devices. In other words, the plurality of processes included in one step can be executed as processes of a plurality of steps. Conversely, processes described as a plurality of steps can be executed collectively as one step.

[0261] Note that, in a program executed by a computer, the processes in the steps used to describe the program can be executed chronologically in the order described in this specification, or can be executed simultaneously or individually as necessary, such as when a call is made. In other words, the processes in the steps can be executed in an order different from the order described above, as long as there is no contradiction. Also, the processes in the steps used to describe the program can be executed simultaneously by the processes of another program or collectively by the processes of another program.

[0262] Note that, the various technologies described in this specification can be implemented independently of one another, as long as there is no contradiction. Of course, any of the various technologies can also be implemented in combination. For example, part or all of the technology described in any one embodiment can be implemented in combination with part or all of the technology described in another embodiment. Further, part or all of any one of the technologies described above can be implemented in combination with another technology not described above.

[0263] <Application Example of Mobile Object>

[0264] The technology according to an embodiment of the present application (technology of the present application) is applicable to various products. For example, the technology according to an embodiment of the present application is realized as a device mounted on any type of mobile object such as a car, an electric vehicle, a hybrid vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot.

[0265] Figure 27 is a block diagram showing an example of the schematic structure of a vehicle control system that is an example of a mobile body control system that can employ the technology according to an embodiment of the present application.

[0266] The vehicle control system 12000 includes a plurality of electronic control units that are connected to one another via a communication network 12001. In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detecting unit 12030, an inside information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are shown as a functional structure of the integrated control unit 12050. Figure 27 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detecting unit 12030, an inside information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are shown as a functional structure of the integrated control unit 12050.

[0267] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle in accordance with various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generation device (for generating a drive force of the vehicle such as an internal combustion engine, a drive motor, or the like), a drive force transmission mechanism (for transmitting a drive force to a wheel), a steering mechanism (for adjusting a steering angle of the vehicle), a brake device (for generating a braking force of the vehicle), or the like.

[0268] The body system control unit 12020 controls the operation of various devices provided on the vehicle body in accordance with various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves emitted from a mobile device can be input to the body system control unit 12020 as a substitute for a key or a signal of various switches. The body system control unit 12020 receives these input radio waves or signals and controls a door lock device, a power window device, or a lamp, or the like of the vehicle.

[0269] The outside information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside information detecting unit 12030 is connected to an imaging section 12031. The outside information detecting unit 12030 causes the imaging section 12031 to capture an image of the outside of the vehicle and receives the captured image. Based on the received image, the outside information detecting unit 12030 can perform a process of detecting an object such as a person, a vehicle, an obstacle, a sign, a symbol on a road surface, or the like, or a process of detecting a distance to the object.

[0270] The imaging section 12031 is an optical sensor that can receive light and outputs an electric signal corresponding to the amount of light of the received light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 can be visible light, or can be invisible light such as infrared rays.

[0271] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The in-vehicle information detecting unit 12040 is connected to, for example, a driver state detecting portion 12041 that detects the state of the driver. The driver state detecting portion 12041 includes, for example, a camera that captures the driver. On the basis of detection information input from the driver state detecting portion 12041, the in-vehicle information detecting unit 12040 can calculate the degree of fatigue of the driver or the degree of concentration of the driver, or can determine whether the driver is dozing off.

[0272] The microcomputer 12051 is capable of calculating control target values for driving force generation devices, steering mechanisms, or braking devices on the basis of information about the inside or outside of the vehicle obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and outputting control commands to the drive system control unit 12010. The microcomputer 12051 is capable of, for example, performing cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS) including collision avoidance or mitigation of the vehicle, follow-up driving based on a following distance, vehicle speed maintenance driving, vehicle collision warning, warning of vehicle lane departure, and the like.

[0273] In addition, the microcomputer 12051 is capable of performing cooperative control for automatic driving that enables the vehicle to travel automatically without relying on the operation of the driver or the like by controlling driving force generation devices, steering devices, braking devices, and the like on the basis of information about the outside or inside of the vehicle obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

[0274] In addition, the microcomputer 12051 is capable of outputting control instructions to the body system control unit 12020 on the basis of information about the outside of the vehicle obtained by the outside-vehicle information detecting unit 12030. The microcomputer 12051 is capable of, for example, performing cooperative control for preventing dazzling light by controlling a headlamp to change high beam to low beam in accordance with the position of an oncoming vehicle or an approaching vehicle detected by the outside-vehicle information detecting unit 12030.

[0275] The sound / image output portion 12052 transmits an output signal of at least one of sound and images to an output device capable of visually or aurally indicating information to the user of the vehicle or outside the vehicle. In Figure 27 In the example, an audio speaker 12061, a display portion 12062, and a device panel 12063 are shown as the output device. The display portion 12062 can include, for example, at least one of a vehicle-mounted display and a head-up display.

[0276] Figure 28 is a view showing an example of a mounting position of the imaging portion 12031.

[0277] In Figure 28 , the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.

[0278] The imaging sections 12101, 12102, 12103, 12104, and 12105 are arranged, for example, at positions on the front nose, side mirrors, rear bumper, and rear door of the vehicle 12100, and at a position on the upper portion of the interior windshield. The imaging section 12101 provided at the front nose and the imaging section 12105 provided on the upper portion of the interior windshield mainly obtain images of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided at the side mirrors mainly obtain images of the sides of the vehicle 12100. The imaging section 12104 provided at the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The imaging section 12105 provided on the upper portion of the interior windshield is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, and the like.

[0279] In addition, Figure 28 Examples of the imaging ranges of the imaging sections 12101 to 12104 are shown. The imaging range 12111 represents the imaging range of the imaging section 12101 provided at the front nose. The imaging ranges 12112 and 12113 represent the imaging ranges of the imaging sections 12102 and 12103 provided at the side mirrors, respectively. The imaging range 12114 represents the imaging range of the imaging section 12104 provided at the rear bumper or rear door. For example, a bird's-eye image of the vehicle 12100 viewed from above is obtained by superimposing image data captured by the imaging sections 12101 to 12104.

[0280] At least one of the imaging sections 12101 to 12104 can have a function of acquiring distance information. For example, at least one of the imaging sections 12101 to 12104 can be a stereo camera constituted by a plurality of imaging elements, or can be an imaging element having pixels for phase difference detection.

[0281] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, thereby selecting the closest three-dimensional object (in particular, one appearing on the travel path of the vehicle 12100 and traveling in substantially the same direction as the vehicle 12100 at a predetermined speed (e.g., equal to or greater than 0 km / hour)) as the preceding vehicle. Also, the microcomputer 12051 can set in advance the following distance to the preceding vehicle to be maintained, and perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. Thus, cooperative control for automatic driving (autonomous travel of the vehicle without relying on the operation of the driver or the like) can be performed.

[0282] For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data on two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, select the classified three-dimensional object data, and use the selected three-dimensional object data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as ones that the driver of the vehicle 12100 can visually recognize and ones that the driver of the vehicle 12100 cannot visually recognize. In addition, the microcomputer 12051 judges the collision risk indicating the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than a set value and thus collision is likely, the microcomputer 12051 outputs a warning to the driver through the audio speaker 12061 or the display section 12062, and performs forced deceleration or evasive steering through the drive system control unit 12010. The microcomputer 12051 can thereby assist the driver to avoid collision.

[0283] At least one of the imaging sections 12101 to 12104 can be an infrared camera that detects infrared rays. The microcomputer 12051 can recognize a pedestrian, for example, by determining whether there is a pedestrian in the captured images of the imaging sections 12101 to 12104. This recognition of a pedestrian is achieved, for example, by a step of extracting feature points in the captured images of the imaging sections 12101 to 12104 as infrared cameras and a step of determining whether it is a pedestrian by performing pattern matching processing on a series of feature points representing the outline of an object. When the microcomputer 12051 determines that there is a pedestrian in the captured images of the imaging sections 12101 to 12104 and thereby recognizes a pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square outline for emphasis is displayed to be superimposed on the recognized pedestrian. The sound / image output section 12052 can also control the display section 12062 so that an icon or the like representing a pedestrian is displayed at a desired position.

[0284] An example of a vehicle control system employing the technology according to the present application has been described above. The technology according to the present application can be applied to the in-vehicle information detection unit 12040 in the above-described structure. Specifically, the state of the driver can be more accurately detected with distance measurement by the distance measurement module 11. Also, processing of recognizing the posture of the driver can be performed with distance measurement by the distance measurement module 11 and various operations can be performed in accordance with the posture.

[0285] Examples of combinations of structures

[0286] In addition, the present technology can also be configured in the following manner.

[0287] (1) A distance measurement processing apparatus comprising:

[0288] a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between a first valve and a second valve using a predetermined number of detection signals, each of the two detection signals being detected for each of two or more kinds of irradiation light having a predetermined phase difference, the two or more kinds of irradiation light being irradiated on an object, and the electric charges generated by reflected light reflected by the object being divided into the first valve and the second valve in accordance with the distance to the object; and

[0289] a distance measurement section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals.

[0290] (2) The distance measurement processing apparatus according to (1),

[0291] wherein, for a first detection period in which the reflected light having the first phase of the two kinds of the irradiation light is received and a second detection period in which the reflected light having the second phase of the two kinds of the irradiation light is received,

[0292] In the first detection period, a plurality of electric charges are alternately divided into the first gate and the second gate, a first detection signal based on electric charges divided into the first gate and accumulated in the first gate and a second detection signal based on electric charges divided into the second gate and accumulated in the second gate are detected, and,

[0293] In the second detection period, a plurality of electric charges are alternately divided into the first gate and the second gate, a third detection signal based on electric charges divided into the first gate and accumulated in the first gate and a fourth detection signal based on electric charges divided into the second gate and accumulated in the second gate are detected.

[0294] (3) The distance measurement processing device according to (2),

[0295] wherein the correction parameter calculation section calculates the two kinds of the correction parameters using the first detection signal, the second detection signal, the third detection signal, and the fourth detection signal.

[0296] (4) The distance measurement processing device according to (3),

[0297] wherein the correction parameter calculation section includes:

[0298] a calculation section for calculating the two kinds of the correction parameters; and

[0299] a storage section for storing one of the correction parameters calculated by the calculation section.

[0300] (5) The distance measurement processing device according to (4),

[0301] wherein the calculation section calculates one of the correction parameters to be stored in the storage section at the start of the process of obtaining the depth by the distance measurement section and stores the one of the correction parameters in the storage section.

[0302] (6) The distance measurement processing device according to (5),

[0303] wherein the storage section holds the correction parameters for each pixel of a light-receiving section for receiving the reflected light.

[0304] (7) The distance measurement processing device according to (5) or (6),

[0305] The calculation section obtains a compensation parameter for correcting a characteristic deviation between the first and second valves by compensation, as one of the correction parameters stored in the storage section.

[0306] (8) The distance measurement processing apparatus according to (7),

[0307] The calculation section obtains the compensation parameter using, in addition to the first, second, third, and fourth detection signals, a signal obtained by combining the fifth and sixth detection signals.

[0308] In a third detection period in which the reflected light of the illumination light having a third phase is received, among a plurality of electric charges that are alternately divided into the first and second valves, a fifth detection signal based on electric charges divided into the first valve and accumulated in the first valve and a sixth detection signal based on electric charges divided into the second valve and accumulated in the second valve are obtained.

[0309] In a fourth detection period in which the reflected light of the illumination light having a fourth phase is received, among a plurality of electric charges that are alternately divided into the first and second valves, a seventh detection signal based on electric charges divided into the first valve and accumulated in the first valve and an eighth detection signal based on electric charges divided into the second valve and accumulated in the second valve are obtained.

[0310] (9) The distance measurement processing apparatus according to any one of (5) to (8),

[0311] The calculation section obtains a gain parameter for correcting a characteristic deviation between the first and second valves by gain, as another one of the correction parameters.

[0312] (10) The distance measurement processing apparatus according to (9),

[0313] The calculation section obtains the gain parameter for each frame of the depth output at a predetermined frame rate.

[0314] (11) The distance measurement processing apparatus according to any one of (2) to (10),

[0315] The distance measurement section includes:

[0316] a correction operation section that performs an operation of obtaining a first corrected detection signal by correcting the first detection signal and a second corrected detection signal by correcting the third detection signal, or an operation of obtaining a third corrected detection signal by correcting the second detection signal and a fourth corrected detection signal by correcting the fourth detection signal, using the correction parameters calculated by the correction parameter calculation section; and

[0317] a distance measurement operation section that performs an operation of obtaining the depth using the first detection signal, the third detection signal, the third corrected detection signal, and the fourth corrected detection signal, or an operation of obtaining the depth using the second detection signal, the fourth detection signal, the first corrected detection signal, and the second corrected detection signal.

[0318] (12) The distance measurement processing apparatus according to any one of (1) to (11), further comprising:

[0319] a distance measurement result storage section for storing the depth obtained by the distance measurement section; and

[0320] a result synthesis section that synthesizes the depth of a previous frame of a current frame stored in the distance measurement result storage section and the depth of the current frame, and outputs the synthesized depth.

[0321] (13) The distance measurement processing apparatus according to (12),

[0322] wherein the distance measurement section obtains a reliability for the depth at the same time as the depth,

[0323] the reliability is stored in the distance measurement result storage section together with the depth, and

[0324] the result synthesis section synthesizes the depth of a previous frame of a current frame and the depth of the current frame by weighted addition according to the reliability.

[0325] (14) A distance measurement module comprising:

[0326] a light emission section that irradiates an object with two kinds of irradiation light having a predetermined phase difference;

[0327] a light reception section that outputs a predetermined number of detection signals each detecting two detection signals for each of the two kinds of the irradiation light, a charge generated by reception of reflected light reflected by the object being divided into a first gate and a second gate according to a distance to the object;

[0328] a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between the first valve and the second valve using a predetermined number of the detection signals; and

[0329] a distance measurement section that obtains a depth indicating a distance to the object based on the correction parameter and a predetermined number of the detection signals.

[0330] (15) A distance measurement processing method implemented by a distance measurement processing apparatus for performing distance measurement processing, the distance measurement processing method comprising:

[0331] calculating a correction parameter for correcting a characteristic deviation between a first valve and a second valve using a predetermined number of detection signals, detecting two of the detection signals each for each of two kinds of irradiation light having a predetermined phase difference, the two kinds of irradiation light being irradiated on an object, and a charge generated by received reflected light reflected by the object being divided into the first valve and the second valve according to a distance to the object; and

[0332] obtaining a depth indicating a distance to the object based on the correction parameter and a predetermined number of the detection signals.

[0333] (16) A program for causing a computer of a distance measurement processing apparatus for performing distance measurement processing to execute distance measurement processing, the program comprising:

[0334] calculating a correction parameter for correcting a characteristic deviation between a first valve and a second valve using a predetermined number of detection signals, detecting two of the detection signals each for each of two kinds of irradiation light having a predetermined phase difference, the two kinds of irradiation light being irradiated on an object, and a charge generated by received reflected light reflected by the object being divided into the first valve and the second valve according to a distance to the object; and

[0335] obtaining a depth indicating a distance to the object based on the correction parameter and a predetermined number of the detection signals.

[0336] It should be noted that embodiments of the present application are not limited to the above-described embodiments, and can be changed in various ways without departing from the gist of the present application. Also, the effects described in the present specification are merely illustrative or exemplary effects, and are not limiting, and other effects can be achieved by the technical idea of the present application.

[0337] Cross Reference to Related Applications

[0338] This application claims priority to Japanese Prior Patent Application JP 2018-087512 filed on April 27, 2018, the entire contents of which are incorporated herein by reference.

Claims

1. A distance measurement processing apparatus comprising: a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, two of the detection signals being detected respectively for each of two or more kinds of illumination light having a predetermined phase difference, the two or more kinds of illumination light being illuminated on an object, and a charge generated by a reflected light received from the object being divided into the first gate and the second gate according to a distance to the object; and a distance measurement section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of detection signals, wherein, for a first detection period in which the reflected light of the illumination light having a first phase is received and a second detection period in which the reflected light of the illumination light having a second phase is received, in the first detection period, a plurality of charges are alternately divided into the first gate and the second gate, a first detection signal based on a charge divided into the first gate and accumulated in the first gate and a second detection signal based on a charge divided into the second gate and accumulated in the second gate are detected, and in the second detection period, a plurality of charges are alternately divided into the first gate and the second gate, a third detection signal based on a charge divided into the first gate and accumulated in the first gate and a fourth detection signal based on a charge divided into the second gate and accumulated in the second gate are detected.

2. The distance measurement processing apparatus according to claim 1, wherein the correction parameter calculation section calculates two of the correction parameters using the first detection signal, the second detection signal, the third detection signal, and the fourth detection signal. the correction parameter calculation section includes:

3. The distance measurement processing apparatus according to claim 2, wherein a calculation section for calculating two of the correction parameters; and a storage section for storing one of the correction parameters calculated by the calculation section.

4. The distance measurement processing apparatus according to claim 3, wherein the calculation section calculates one of the correction parameters to be stored in the storage section at the start of a process in which the depth is obtained by the distance measurement section, and stores the one of the correction parameters in the storage section.

5. The distance measurement processing apparatus according to claim 4, wherein the storage section holds the correction parameter for each pixel of a light receiving section that receives the reflected light.

6. The distance measurement processing apparatus according to claim 4, wherein the calculation section obtains a compensation parameter for correcting a characteristic deviation between the first gate and the second gate by compensation by being one of the correction parameters stored in the storage section. The calculation section obtains the compensation parameter using the following signal in addition to the first detection signal, the second detection signal, the third detection signal, and the fourth detection signal:

7. The distance measurement processing apparatus according to claim 6, wherein a fifth detection signal based on a charge divided into the first gate and accumulated in the first gate and a sixth detection signal based on a charge divided into the second gate and accumulated in the second gate. in a third detection period in which the reflected light of the irradiation light having a third phase is received, among a plurality of electric charges which are alternately divided into the first shutter and the second shutter, a fifth detection signal based on electric charges divided into the first shutter and accumulated in the first shutter and a sixth detection signal based on electric charges divided into the second shutter and accumulated in the second shutter are obtained; and in a fourth detection period in which the reflected light of the irradiation light having a fourth phase is received, among a plurality of electric charges which are alternately divided into the first shutter and the second shutter, a seventh detection signal based on electric charges divided into the first shutter and accumulated in the first shutter and an eighth detection signal based on electric charges divided into the second shutter and accumulated in the second shutter are obtained.

8. The distance measurement processing apparatus according to claim 4, wherein the calculation section obtains a gain parameter for correcting a characteristic deviation between the first shutter and the second shutter by gain as another one of the correction parameters.

9. The distance measurement processing apparatus according to claim 8, wherein the calculation section obtains the gain parameter for each frame of the depth output at a predetermined frame rate.

10. The distance measuring processing apparatus according to claim 1, wherein the distance measurement section includes: a correction operation section which performs an operation of obtaining a first corrected detection signal by correcting the first detection signal and a second corrected detection signal by correcting the third detection signal, or an operation of obtaining a third corrected detection signal by correcting the second detection signal and a fourth corrected detection signal by correcting the fourth detection signal, using the correction parameters calculated by the correction parameter calculation section; and a distance measurement operation section which performs an operation of obtaining the depth using the first detection signal, the third detection signal, the third corrected detection signal, and the fourth corrected detection signal, or an operation of obtaining the depth using the second detection signal, the fourth detection signal, the first corrected detection signal, and the second corrected detection signal.

11. The distance measurement processing apparatus according to any one of claims 1 to 10, further comprising: a distance measurement result storage section for storing the depth obtained by the distance measurement section; and a result synthesizing section which synthesizes the depth of a previous frame of a current frame stored in the distance measurement result storage section and the depth of the current frame, and outputs the synthesized depth.

12. The distance measurement processing apparatus according to claim 11, wherein the distance measurement section obtains a reliability for the depth at the same time as the depth is obtained, the reliability is stored in the distance measurement result storage section together with the depth, and the result synthesizing section synthesizes the depth of a previous frame of a current frame and the depth of the current frame by weighted addition according to the reliability.

13. A distance measurement module comprising: an emission section which irradiates an object with two kinds of irradiation light having a predetermined phase difference; a light-receiving section that outputs a predetermined number of detection signals each detecting two detection signals for each of two kinds of the irradiation light, a plurality of charges being divided into a first gate and a second gate according to a distance to the object by charges generated by received reflected light reflected by the object; a correction parameter calculation section that calculates a correction parameter for correcting a characteristic deviation between the first gate and the second gate using the predetermined number of the detection signals; and a distance measurement section that obtains a depth indicating the distance to the object based on the correction parameter and the predetermined number of the detection signals, wherein, for a first detection period in which the reflected light of the irradiation light having a first phase is received and a second detection period in which the reflected light of the irradiation light having a second phase is received in the two kinds of the irradiation light, in the first detection period, a plurality of charges are alternately divided into the first gate and the second gate, a first detection signal based on charges divided into the first gate and accumulated in the first gate and a second detection signal based on charges divided into the second gate and accumulated in the second gate are detected, and in the second detection period, a plurality of charges are alternately divided into the first gate and the second gate, a third detection signal based on charges divided into the first gate and accumulated in the first gate and a fourth detection signal based on charges divided into the second gate and accumulated in the second gate are detected.

14. A distance measurement processing method implemented by a distance measurement processing apparatus for performing a distance measurement processing, the distance measurement processing method comprising the steps of: calculating a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, two detection signals each being detected for each of two kinds of irradiation light having a predetermined phase difference, the two kinds of the irradiation light being irradiated on an object, and a plurality of charges being divided into the first gate and the second gate according to a distance to the object by charges generated by received reflected light reflected by the object; and obtaining a depth indicating the distance to the object based on the correction parameter and the predetermined number of the detection signals, wherein, for a first detection period in which the reflected light of the irradiation light having a first phase is received and a second detection period in which the reflected light of the irradiation light having a second phase is received in the two kinds of the irradiation light, in the first detection period, a plurality of charges are alternately divided into the first gate and the second gate, a first detection signal based on charges divided into the first gate and accumulated in the first gate and a second detection signal based on charges divided into the second gate and accumulated in the second gate are detected, and in the second detection period, a plurality of charges are alternately divided into the first gate and the second gate, a third detection signal based on charges divided into the first gate and accumulated in the first gate and a fourth detection signal based on charges divided into the second gate and accumulated in the second gate are detected.

15. A computer-readable medium storing a computer program for causing a computer of a distance measurement processing apparatus for performing a distance measurement process to execute the distance measurement process, the computer program comprising: calculating a correction parameter for correcting a characteristic deviation between a first gate and a second gate using a predetermined number of detection signals, each of the two detection signals being detected with respect to each of two kinds of illumination light having a predetermined phase difference, the two kinds of illumination light being illuminated on an object, and a charge generated by received reflected light reflected by the object being divided into the first gate and the second gate according to a distance to the object; and obtaining a depth indicating the distance to the object based on the correction parameter and the predetermined number of the detection signals, wherein, for a first detection period in which the reflected light of the illumination light having a first phase is received and a second detection period in which the reflected light of the illumination light having a second phase is received, in the first detection period, a plurality of charges are alternately divided into the first gate and the second gate, a first detection signal based on a charge divided into the first gate and accumulated in the first gate and a second detection signal based on a charge divided into the second gate and accumulated in the second gate are detected, and in the second detection period, a plurality of charges are alternately divided into the first gate and the second gate, a third detection signal based on a charge divided into the first gate and accumulated in the first gate and a fourth detection signal based on a charge divided into the second gate and accumulated in the second gate are detected.

Images