Distance measurement processing device, distance measurement module, distance measurement processing method and program

Abstract

The present invention relates to a distance measurement processing device, a distance measurement module, a distance measurement processing method and a program capable of achieving higher performance. The distance measurement processing device includes: a four-phase distance measurement operation unit, which performs an operation to calculate the depth representing the distance to an object by using all eight detection signals, and can detect two detection signals among the eight detection signals for each of the first phase to the fourth phase of the irradiation light; a two-phase distance measurement operation unit, which performs an operation to calculate the depth representing the distance to the object by alternately using four detection signals based on the first phase of the irradiation light and the second phase of the irradiation light and four detection signals based on the third phase of the irradiation light and the fourth phase of the irradiation light among the eight detection signals; and a condition determination unit, which performs condition determination based on the detection signal and switches between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit to be used.

Description

Technical Field

[0001] The present technology relates to a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a program, and in particular, to a distance measurement processing device, a distance measurement module, a distance measurement processing method, and a program capable of achieving higher performance. Background Art

[0002] In recent years, with the advancement of semiconductor technology, the miniaturization of distance measurement modules for measuring the distance to an object has been progressing. Therefore, for example, it has been realized to install distance measurement modules in mobile terminals such as so-called smart phones, which are small information processing devices with communication functions.

[0003] Generally, as ranging methods in ranging modules, there are two types of ranging methods: indirect time of flight (ToF) and structured light. According to the indirect ToF method, light reflected by the surface of an object after being irradiated on the object is detected, and the distance to the object is calculated based on the measurement value obtained by measuring the flight time of the light. According to the structured light method, patterned light is irradiated onto the object, and the distance to the object is calculated based on an image obtained by imaging the deformation of the pattern on the surface of the object.

[0004] For example, Japanese Patent Application Publication No. 2017-150893 discloses a technology for determining the movement of an object during a detection period in order to accurately measure the distance in a ranging system that performs ranging according to a ToF method. Summary of the invention

[0005] Meanwhile, in order to use the ranging module in the mobile terminal as described above, it is required to improve performance of a frame rate, power consumption, or a data transmission band, etc.

[0006] The present invention has been made in view of the above circumstances, and the present invention can achieve higher performance.

[0007] An embodiment of the present invention provides a distance measurement processing device, comprising: a four-phase distance measurement operation unit, which is configured to: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating a first phase of irradiation light, a second phase of irradiation light, a third phase of irradiation light, and a fourth phase of irradiation light onto the object and receiving reflected light reflected by the object, and for each of the irradiation light of the first phase to the irradiation light of the fourth phase, a calculation unit can be performed to calculate a depth representing the distance to the object by using all eight detection signals. two detection signals among the eight detection signals are detected; a two-phase ranging operation unit, which is configured to: perform the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and a condition determination unit, which is configured to perform condition determination based on the detection signals and is configured to switch between the four-phase ranging operation unit and the two-phase ranging operation unit in use.

[0008] An embodiment of the present invention provides a distance measurement module, which includes: a light emitting unit, which is configured to irradiate a first phase of illumination light, a second phase of illumination light, a third phase of illumination light, and a fourth phase of illumination light onto an object; a light receiving unit, which is configured to output eight detection signals when allocating charges to a first tap and a second tap according to the distance to the object, the charges being generated by receiving reflected light reflected by the object, and for each of the illumination lights from the first phase to the fourth phase of illumination light, two detection signals among the eight detection signals can be detected; a four-phase distance measurement operation unit, which is configured to output eight detection signals by using all the illumination lights from the first phase to the fourth phase of illumination light. The eight detection signals are used to perform an operation for calculating the depth of the distance to the object; a two-phase ranging operation unit, which is configured to: perform the operation for calculating the depth of the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase among the eight detection signals and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase; and a condition determination unit, which is configured to perform condition determination based on the detection signals and is configured to switch between the four-phase ranging operation unit and the two-phase ranging operation unit.

[0009] An embodiment of the present invention provides a distance measurement processing method, which includes: performing a four-phase distance measurement operation processing: when a charge is allocated to a first tap and a second tap according to the distance to an object, performing an operation for calculating the depth representing the distance to the object by using all eight detection signals, the charge is generated by irradiating a first phase of illumination light, a second phase of illumination light, a third phase of illumination light, and a fourth phase of illumination light onto the object and receiving reflected light reflected by the object, and for each of the illumination light of the first phase to the fourth phase, two detection signals among the eight detection signals can be detected; performing a two-phase distance measurement operation processing: performing the operation for calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and performing condition determination based on the detection signals, and switching between the four-phase distance measurement operation processing and the two-phase distance measurement operation processing to be used.

[0010] An embodiment of the present invention provides a program for causing a computer of a distance measurement processing device that performs distance measurement processing to perform the distance measurement processing, the distance measurement processing comprising: performing a four-phase distance measurement operation processing: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase onto the object and receiving reflected light reflected by the object, for the first phase Each type of irradiation light from the irradiation light to the fourth phase can detect two of the eight detection signals; a two-phase ranging operation is performed: the operation of calculating the depth representing the distance to the object is performed by alternately using four detection signals based on the irradiation light of the first phase and the irradiation light of the second phase among the eight detection signals and four detection signals based on the irradiation light of the third phase and the irradiation light of the fourth phase; and a condition judgment is performed based on the detection signal, and switching is performed between the four-phase ranging operation and the two-phase ranging operation.

[0011] In an embodiment of the present invention, a charge is allocated to a first tap and a second tap according to the distance to an object, the charge being generated by irradiating the first phase of irradiation light, the second phase of irradiation light, the third phase of irradiation light, and the fourth phase of irradiation light onto the object and receiving reflected light reflected by the object, and eight detection signals can be detected for the irradiation light of the first phase to the fourth phase (two detection signals of the eight detection signals can be detected for each of the irradiation lights). Then, a four-phase ranging operation process is performed by using all eight detection signals, and the four-phase ranging operation process is used to perform an operation for calculating the depth representing the distance to the object. A two-phase ranging operation process is performed by alternately using four detection signals based on the irradiation light of the first phase and the irradiation light of the second phase and four detection signals based on the irradiation light of the third phase and the irradiation light of the fourth phase among the eight detection signals, and the two-phase ranging operation process is used to perform an operation for calculating the depth representing the distance to the object. Condition determination is performed based on the detection signal, and the four-phase ranging operation process and the two-phase ranging operation process used are switched.

[0012] According to the embodiments of the present invention, better performance can be achieved.

[0013] It is to be noted that the effects described here are not restrictive, and any effects described in the present invention may be produced.

[0014] These and other objects, features and advantages of the present invention will become more apparent in light of the following detailed description of best mode embodiments thereof, as illustrated in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram showing a configuration example of an embodiment of a distance measurement module to which the present technology is applied;

[0016] Figure 2 is a diagram used to describe the distribution of charge in a pixel circuit;

[0017] Figure 3 is a diagram showing an example of four types of irradiation lights each having a phase delay of 90°;

[0018] Figure 4 is a diagram for describing distance measurement using four detection periods based on four types of illumination light each having a phase delay of 90°;

[0019] Figure 5 is a diagram showing an example of a detection signal in a detection period based on irradiation light having a phase delay of 0°;

[0020] Figure 6 is a diagram showing an example of a detection signal in a detection period based on irradiation light having a phase delay of 90°;

[0021] Figure 7 is a diagram showing an example of a detection signal in a detection period based on irradiation light having a phase delay of 180°;

[0022] Figure 8 is a diagram showing an example of a detection signal in a detection period based on irradiation light having a phase delay of 270°;

[0023] Fig. 9 is a diagram for describing the relationship between detection signals A0 to A270 and detection signals B0 to B270;

[0024] Fig.10 is a diagram used to describe the correction operation;

[0025] Fig.11 is a diagram used to describe ranging using two detection periods;

[0026] Fig.12 is a block diagram showing a first configuration example of a distance measurement operation processing unit;

[0027] Fig.13 is a flowchart for describing a first processing example of ranging operation processing;

[0028] Fig.14 is a block diagram showing a second configuration example of the ranging operation processing unit;

[0029] Fig.15 It is a diagram used to describe the improvement of frame rate by synthesizing ranging results;

[0030] Fig.16 is a diagram used to describe reducing power consumption by synthesizing ranging results;

[0031] Fig.17 is a flowchart for describing a second processing example of the ranging operation processing;

[0032] Fig.18 is a diagram showing an example of the timing of emitting light and receiving light to output a depth map;

[0033] Fig.19 is a diagram showing changes in light emission patterns;

[0034] Fig. 20 is a diagram showing changes in light emission patterns;

[0035] Fig.21 is a diagram showing changes in light emission patterns;

[0036] Fig. 22 is a block diagram showing a third configuration example of the ranging operation processing unit;

[0037] Fig.23 It is a diagram used to describe the synthesis of ranging results based on motion detection;

[0038] Fig.24 is a flowchart for describing a third processing example of the ranging operation process;

[0039] Fig.25 is a block diagram showing a configuration example of an electronic device in which a distance measuring module is mounted;

[0040] Fig.26 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied;

[0041] Fig. 27 is a block diagram depicting a schematic configuration example of a vehicle control system; and

[0042] Fig.28 It is a diagram for assisting in explaining an example of the installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0043] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the accompanying drawings.

[0044] (Configuration example of distance measurement module)

[0045] Figure 1 is a block diagram showing a configuration example of an embodiment of a distance measurement module to which the present technology is applied.

[0046] like Figure 1 As shown, the distance measuring module 11 includes a light emitting unit 12, a light emitting control unit 13, a light receiving unit 14 and a distance measuring operation processing unit 15. For example, the distance measuring module 11 irradiates light onto an object and receives light (reflected light) generated when the object reflects light (irradiated light) to measure the depth representing the distance to the object.

[0047] The light emitting unit 12 emits light while modulating light according to the control of the light emitting control unit 13 at a timing corresponding to a light emitting control signal supplied from the light emitting control unit 13 , and the light emitting unit 12 irradiates the irradiation light onto the object.

[0048] The light-emitting control unit 13 supplies a light-emitting control signal having a prescribed frequency (e.g., 20 MHz, etc.) to the light-emitting unit 12 to control the light-emitting of the light-emitting unit 12. In addition, the light-emitting control unit 13 also supplies a light-emitting control signal to the light-receiving unit 14 to drive the light-receiving unit 14 in conjunction with the light-emitting timing of the light-emitting unit 12.

[0049] The light receiving unit 14 receives light reflected from an object onto a sensor surface of the light receiving unit 14 in which a plurality of pixels are arranged in an array. The light receiving unit 14 then supplies image data consisting of a detection signal corresponding to the light reception amount of the reflected light received by each pixel to the distance measurement operation processing unit 15.

[0050] The ranging operation processing unit 15 performs an operation of calculating the depth from the ranging module 11 to the object based on the image data supplied from the light receiving unit 14. Then, the ranging operation processing unit 15 generates a depth map and a confidence map, and outputs the generated maps to a subsequent control unit (e.g., Fig.25 In the depth map, the depth to the object is represented for each pixel, and in the confidence map, the confidence of each depth is represented for each pixel. Note that reference will be made later. Fig.12 The detailed configuration of the distance measurement operation processing unit 15 is described.

[0051] In addition, a pixel array unit 22 is provided in the light receiving unit 14, in which a plurality of pixel circuits 21 are arranged in an array form, and a drive control circuit 23 is arranged in the peripheral area of ​​the pixel array unit 22. The pixel array unit 22 is a sensor surface that receives reflected light. The drive control circuit 23 outputs a control signal (for example, a distribution signal DIMIX, a selection signal ADDRESS DECODE, or a reset signal RST to be described later) based on, for example, a light emission control signal supplied from the light emission control unit 13, to control the drive of the pixel circuit 21.

[0052] The pixel circuit 21 is configured so that the charge generated by one photodiode 31 is distributed to a tap 32A and a tap 32B. Then, among the charges generated by the photodiode 31, the charge distributed to the tap 32A is read out from the signal line 33A to be used as a detection signal A, and the charge distributed to the tap 32B is read out from the signal line 33B to be used as a detection signal B.

[0053] The tap 32A is composed of a transfer transistor 41A, a floating diffusion (FD) unit 42A, a selection transistor 43A, and a reset transistor 44A. Similarly, the tap 32B is composed of a transfer transistor 41B, an FD unit 42B, a selection transistor 43B, and a reset transistor 44B.

[0054] Reference Figure 2 Distribution of charges in the pixel circuit 21 is described.

[0055] like Figure 2As shown, the illumination light is output from the light emitting unit 12: the illumination light is modulated to repeatedly turn on and off the illumination with an illumination time T (1 cycle = 2T), and the reflected light is received by the photodiode 31 so as to be delayed by a delay time T corresponding to the distance to the object. RT In addition, the distribution signal DIMIX_A controls the on / off of the transfer transistor 41A, and the distribution signal DIMIX_B controls the on / off of the transfer transistor 41B. Figure 2 As shown, the distribution signal DIMIX_A has the same phase as the phase of the irradiation light, and the distribution signal DIMIX_B has a phase obtained by inverting the phase of the distribution signal DIMIX_A.

[0056] Therefore, during the period when the transfer transistor 41A is turned on according to the distribution signal DIMIX_A, the charge generated when the photodiode 31 receives the reflected light is transferred to the FD unit 42A, and during the period when the transfer transistor 41B is turned on according to the distribution signal DIMIX_B, the charge generated when the photodiode 31 receives the reflected light is transferred to the FD unit 42B. Therefore, during the prescribed period when the irradiation light is periodically irradiated with the irradiation time T, the charge transferred via the transfer transistor 41A is sequentially accumulated in the FD unit 42A, and the charge transferred via the transfer transistor 41B is sequentially accumulated in the FD unit 42B.

[0057] Then, when the selection transistor 43A is turned on according to the selection signal ADDRESSDECODE_A after the period of accumulating the charge ends, the charge accumulated in the FD unit 42A is read out via the signal line 33A, and a detection signal A corresponding to the charge amount is output from the light receiving unit 14. Similarly, when the selection transistor 43B is turned on according to the selection signal ADDRESS DECODE_B, the charge accumulated in the FD unit 42B is read out via the signal line 33B, and a detection signal B corresponding to the charge amount is output from the light receiving unit 14. Furthermore, when the reset transistor 44A is turned on according to the reset signal RST_A, the charge accumulated in the FD unit 42A is discharged, and when the reset transistor 44B is turned on according to the reset signal RST_B, the charge accumulated in the FD unit 42B is discharged.

[0058] As described above, the pixel circuit 21 can be configured to generate a pixel signal according to the delay time T RT The charge generated by the reflected light received by the photodiode 31 is distributed to the taps 32A and 32B, and the pixel circuit 21 can output the detection signals A and B. In addition, the delay time T RT corresponds to the time from when the light emitting unit 12 emits light to the object to when the light is reflected by the object and returns to the light receiving unit 14, that is, corresponds to the distance to the object. Therefore, the ranging module 11 can calculate the time depending on the delay time T based on the detection signals A and B.RT The distance (depth) to the object.

[0059] Meanwhile, in the distance measuring module 11, since the characteristics of the photodiodes 31 of the respective pixel circuits 21 are different, the influence on the detection signals A and B of the respective pixel circuits 21 is different. Therefore, generally, by irradiation lights of different phases, an operation for eliminating the influence caused by the characteristic differences is performed multiple times based on the detection signals A and B detected based on the reflected light generated by the irradiation lights of the respective phases.

[0060] For example, Figure 3 As shown, four types of irradiation lights with a phase delay of 90° are used. That is, irradiation light with a phase delay of 0°, irradiation light with a phase delay of 90°, irradiation light with a phase delay of 180°, and irradiation light with a phase delay of 270° are used as references, and four periods (quads) for detecting detection signals A and B are set.

[0061] That is, Figure 4 As shown, for example, a detection period Q0 for detecting reflected light generated by irradiation light with a phase delay of 0°, a detection period Q1 for detecting reflected light generated by irradiation light with a phase delay of 90°, a detection period Q2 for detecting reflected light generated by irradiation light with a phase delay of 180°, and a detection period Q3 for detecting reflected light generated by irradiation light with a phase delay of 270° are sequentially set. In addition, in each of the detection periods Q0, Q1, Q2, and Q3, a reset period for resetting charges, an integration period for accumulating charges, and a readout period for reading out charges are set.

[0062] A depth frame for outputting a depth map is formed by a detection period including detection periods Q0, Q1, Q2, and Q3 and a subsequent standby period (idle time). Such a depth frame is repeatedly output. Therefore, depth frames such as a depth frame with a frame number of t, a depth frame with a frame number of t+1, and a depth frame with a frame number of t+2 can be sequentially output at a prescribed frame rate.

[0063] Figure 5 2 shows an example of irradiated light, reflected light, distributed signals DIMIX_A and DIMIX_B, and detection signals A and B in the detection period Q0. Figure 5 As shown, the charge is proportional to the delay time T RT The corresponding amounts are distributed to the taps 32A and 32B, and the charges are accumulated in the integration period. Then, in the readout period, the charges respectively accumulated in the integration period are read out, and the detection signals A0 and B0 in the detection period Q0 are output.

[0064] Figure 6FIG. 2 shows an example of irradiated light, reflected light, distributed signals DIMIX_A and DIMIX_B, and detection signals A and B in the detection period Q1. Figure 6 As shown, the charge is proportional to the delay time T RT The corresponding amount is distributed to the taps 32A and 32B, and the charge is accumulated in the integration period. Then, in the readout period, the charges respectively accumulated in the integration period are read out, and the detection signals A90 and B90 in the detection period Q1 are output.

[0065] Figure 7 An example of the irradiated light, the reflected light, the distribution signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q2 is shown. Figure 7 As shown, the charge is proportional to the delay time T RT The corresponding amount is distributed to the taps 32A and 32B, and the charge is accumulated in the integration period. Then, in the readout period, the charges respectively accumulated in the integration period are read out, and the detection signals A180 and B180 in the detection period Q2 are output.

[0066] Figure 8 An example of the irradiated light, the reflected light, the distribution signals DIMIX_A and DIMIX_B, and the detection signals A and B in the detection period Q3 is shown. Figure 8 As shown, the charge is proportional to the delay time T RT The corresponding amount is distributed to the taps 32A and 32B, and the charge is accumulated in the integration period. Then, in the readout period, the charges respectively accumulated in the integration period are read out, and the detection signals A270 and B270 in the detection period Q3 are output.

[0067] As described above, in the detection period Q0, detection signals A0 and B0 are detected by the irradiation light with a phase delay of 0°, and in the detection period Q1, detection signals A90 and B90 are detected by the irradiation light with a phase delay of 90°. Similarly, in the detection period Q2, detection signals A180 and B180 are detected by the irradiation light with a phase delay of 180°, and in the detection period Q3, detection signals A270 and B270 are detected by the irradiation light with a phase delay of 270°.

[0068] here, Fig. 9 The relationship between the detection signals A0 to A270 and the detection signals B0 to B270 is shown when the horizontal axis and the vertical axis represent phase delay and signal strength, respectively.

[0069] The relationship between the detection signals A0 and B0 , the relationship between the detection signals A90 and B90 , the relationship between the detection signals A180 and B180 , and the relationship between the detection signals A270 and B270 are modeled as shown in the following equation (1).

[0070] (Formula 1)

[0071]

[0072] By performing such modeling and calculating the offset, gain, and angle θ according to formula (1), the following distance measurement can be performed: wherein, for example, the influence caused by the characteristic difference between the taps 32A and 32B can be eliminated. That is, in order to eliminate the difference in the offset Offset and the gain Gain between the taps 32A and 32B, it is desirable to use the eight detection signals (detection signals A0 to A270 and detection signals B0 to B270) detected in the four detection periods Q0 to Q3.

[0073] To this end, the ranging module 11 calculates the offset and gain of the tap 32A and the offset and gain of the tap 32B, and the ranging module 11 compensates for their differences. Therefore, the ranging module 11 can perform ranging as follows: in which the influence caused by the characteristic difference between the taps 32A and 32B can be eliminated by only detecting the detection signals A and B in each of the two detection periods Q0 and Q1 (or the detection periods Q2 and Q3).

[0074] For example, the following equation (2) shows the relationship between the offset Offset_A and the gain Gain_A of the tap 32A and the offset Offset_B and the gain Gain_B of the tap 32B.

[0075] (Formula 2)

[0076]

[0077] Here, for each pixel circuit 21, offset Offset_A and offset Offset_B have fixed values ​​and can be calculated in advance. On the other hand, gain Gain_A and gain Gain_B may fluctuate with the incident angle of light according to the structure of the pixel circuit 21, so it is desirable to calculate gain Gain_A and gain Gain_B for each depth frame.

[0078] That is, the distance measurement module 11 detects the detection signals A0 to A270 and the detection signals B0 to B270 in advance or in the initial process of performing distance measurement, and solves the simultaneous equations shown in the following equation (3) to calculate the offset Offset_A and the offset Offset_B.

[0079] (Formula 3)

[0080]

[0081] Then, the ranging module 11 stores the offset Offset_A and the offset Offset_B as offset parameters.

[0082] Subsequently, at the time of detecting the detection signals A0 and B0 and the detection signals A90 and B90, the distance measurement module 11 calculates the gain parameter (Gain_A / Gain_B) as shown in the following equation (4).

[0083] (Formula 4)

[0084]

[0085] In addition, at the time of detecting the detection signals A180 and A270 and the detection signals B180 and B270, the distance measurement module 11 calculates the gain parameter (Gain_A / Gain_B) as shown in the following equation (5).

[0086] (Formula 5)

[0087]

[0088] Therefore, at the moment when the detection signals A0, B0, A90 and B90 are detected, the ranging module 11 can apply correction using the offset parameters (Offset_A and Offset_B) and the gain parameter (Gain_A / Gain_B) according to the following equation (6).

[0089] (Formula 6)

[0090]

[0091] or

[0092]

[0093] Therefore, the distance measurement module 11 calculates the corrected detection signals A′ 180 and A′ 270 based on the detection signal A, and calculates the corrected detection signals B′ 180 and B′ 270 based on the detection signal B.

[0094] That is, Fig.10 As shown, the ranging module 11 performs correction on the detection signal B0 to calculate the corrected detection signal A'180, and performs correction on the detection signal B90 to calculate the corrected detection signal A'270. Alternatively, the ranging module 11 performs correction on the detection signal A0 to calculate the corrected detection signal B'180, and performs correction on the detection signal A90 to calculate the corrected detection signal B'270.

[0095] Therefore, the ranging module 11 can use the detection signals A0 and A90 and the correction detection signals A'180 and A'270 to eliminate the influence caused by the characteristic difference between the taps 32A and 32B in order to calculate the depth and confidence. Alternatively, the ranging module 11 can use the detection signals B0 and B90 and the correction detection signals B'180 and B'270 to eliminate the influence caused by the characteristic difference between the taps 32A and 32B.

[0096] Similarly, at the moment when the detection signals A180, B180, A270 and B270 are detected, the ranging module 11 can apply correction using the offset parameters (Offset_A and Offset_B) and the gain parameter (Gain_A / Gain_B) according to the following equation (7).

[0097] (Formula 7)

[0098]

[0099] or

[0100]

[0101] Therefore, the distance measurement module 11 calculates the corrected detection signals A′0 and A′90 based on the detection signal A, and calculates the corrected detection signals B′0 and B′90 based on the detection signal B.

[0102] Therefore, the ranging module 11 can use the corrected detection signals A'0 and A'90 and the detection signals A180 and A270 to eliminate the influence caused by the characteristic difference between the taps 32A and 32B, so as to calculate the depth and confidence. Alternatively, the ranging module 11 can use the corrected detection signals B'0 and B'90 and the detection signals B180 and B270 to eliminate the influence caused by the characteristic difference between the taps 32A and 32B, so as to calculate the depth and confidence.

[0103] As described above, the ranging module 11 calculates the offset parameters (Offset_A and Offset_B) in advance, and calculates the gain parameter (Gain_A / Gain_B) for each depth frame in order to perform ranging that can eliminate the influence caused by the characteristic difference between the taps 32A and 32B.

[0104] For example, Fig.11 As shown, the ranging module 11 detects four detection signals (detection signals A0, B0, A90 and B90) in two detection periods Q0 and Q1, so as to output a depth frame with a frame number of t. Subsequently, the ranging module 11 detects four detection signals (detection signals A180, B180, A270 and B270) in two detection periods Q2 and Q3, so as to output a depth frame with a frame number of t+1.

[0105] Therefore, as mentioned above Figure 4 Compared with the distance measurement method of outputting a depth frame within four detection periods Q0-Q3, the distance measurement module 11 can reduce the time for outputting a depth frame by half. That is, compared with the prior art, the distance measurement module 11 can double the frame rate.

[0106] (Configuration Example of Distance Measurement Operation Processing Unit)

[0107] Fig.12 : is a block diagram showing a first configuration example of the ranging operation processing unit 15 .

[0108] The ranging operation processing unit 15 uses the detection signals A(t) and B(t) supplied from the light receiving unit 14 as image data to output the depth d(t) constituting the depth map of frame number t and the confidence c(t) constituting the confidence map of frame number t.

[0109] First, when receiving four detection signals (detection signals A0(t), B0(t), A90(t), and B90(t)) detected by illumination light with a phase delay of 0° and illumination light with a phase delay of 90°, the ranging operation processing unit 15 outputs the depth d(t) and confidence c(t) of the depth frame with a frame number of t. Subsequently, when receiving four detection signals (detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1)) detected by illumination light with a phase delay of 180° and illumination light with a phase delay of 270°, the ranging operation processing unit 15 outputs the depth d(t+1) and confidence c(t+1) of the depth frame with a frame number of t+1.

[0110] like Fig.12 As shown, the distance measurement operation processing unit 15 includes a correction parameter calculation unit 51 and a distance measurement unit 52. Furthermore, the correction parameter calculation unit 51 has a difference correction parameter calculation unit 61 and a difference correction parameter storage unit 62, and the distance measurement unit 52 has a correction operation unit 71 and a distance measurement operation unit 72.

[0111] At the start of ranging, the difference correction parameter calculation unit 61 solves the following equation (8) regarding the offsets Offset_A and Offset_B in, for example, several frames.

[0112] (Formula 8)

[0113]

[0114] Therefore, the differential correction parameter calculation unit 61 calculates the offsets Offset_A and Offset_B, and stores the offsets Offset_A and Offset_B in the differential correction parameter storage unit 62. Note that the offsets Offset_A and Offset_B may be calculated in advance, for example, when the ranging module 11 is inspected, and the offsets Offset_A and Offset_B may be stored in the differential correction parameter storage unit 62 when the ranging module 11 is shipped.

[0115] Then, when receiving four detection signals (detection signals A0(t), B0(t), A90(t), and B90(t)) detected by the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90°, the differential correction parameter calculation unit 61 calculates the following equation (9). Therefore, the differential correction parameter calculation unit 61 calculates the gain parameter (Gain_A / Gain_B(t)) and supplies the gain parameter to the correction operation unit 71 of the distance measuring unit 52.

[0116] (Formula 9)

[0117]

[0118] Subsequently, when receiving four detection signals (detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1)) detected by the irradiation light with a phase delay of 180° and the irradiation light with a phase delay of 270°, the differential correction parameter calculation unit 61 calculates the following equation (10). Therefore, the differential correction parameter calculation unit 61 calculates the gain parameter (Gain_A / Gain_B(t+1)) and supplies the gain parameter to the correction operation unit 71 of the distance measuring unit 52.

[0119] (Formula 10)

[0120]

[0121] The differential correction parameter storage unit 62 stores the offset parameters (Offset_A and Offset_B) calculated by the differential correction parameter calculation unit 61, and supplies the offset parameters to the correction operation unit 71. It should be noted that the differential correction parameter calculation unit 61 calculates the gain parameter and the offset parameter of each pixel circuit 21, and the differential correction parameter storage unit 62 holds the offset parameter of each pixel circuit 21.

[0122] At the moment when four detection signals (detection signals A0(t), B0(t), A90(t), and B90(t)) detected by the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90° are supplied, the correction operation unit 71 receives the gain parameter (Gain_A / Gain_B) from the differential correction parameter calculation unit 61. Therefore, the correction operation unit 71 can perform the operation shown in the following formula (11) at this moment to calculate the correction detection signals A'180(t) and A'270(t), or the correction detection signals B'180(t) and B'270(t).

[0123] (Formula 11)

[0124]

[0125] or

[0126]

[0127] Therefore, at the moment when four detection signals detected by supplying the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90° are supplied, the correction operation unit 71 supplies the correction detection signals A'180(t) and A'270(t), or the correction detection signals B'180(t) and B'270(t) to the ranging operation unit 72.

[0128] Subsequently, at the moment when four detection signals (detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1)) detected by the irradiation light with a phase delay of 180° and the irradiation light with a phase delay of 270° are supplied, the correction operation unit 71 receives the gain parameter (Gain_A / Gain_B(t+1)) from the differential correction parameter calculation unit 61. Therefore, the correction operation unit 71 can perform the operation shown in the following formula (12) at this moment to calculate the correction detection signals A'0(t+1) and A'90(t+1), or the correction detection signals B'0(t+1) and B'90(t+1).

[0129] (Formula 12)

[0130]

[0131] or

[0132]

[0133] Therefore, at the moment of supplying four detection signals detected by irradiation light with a phase delay of 180° and irradiation light with a phase delay of 270°, the correction operation unit 71 supplies correction detection signals A'0(t+1) and A'90(t+1), or correction detection signals B'0(t+1) and B'90(t+1).

[0134] At the moment of supplying four detection signals (detection signals A0(t), B0(t), A90(t) and B90(t)) detected by the illumination light with a phase delay of 0° and the illumination light with a phase delay of 90°, the ranging operation unit 72 receives the correction detection signals A'180(t) and A'270(t), or the correction detection signals B'180(t) and B'270(t). Then, the ranging operation unit 72 can perform the operation shown in the following formula (13) to calculate the depth d(t) and confidence c(t) of the depth frame with a frame number t.

[0135] (Formula 13)

[0136]

[0137] Q(t)=D1(t)-D3(t)

[0138] [(t) = DO(t) - D2(t)

[0139] However, in the above formula (13), the ranging operation unit 72 can use one of the following two: D0(t)=A0(t), D2(t)=A'180(t), D1(t)=A90(t), D3(t)=A'270(t); and D2(t)=B'180(t), D0(t)=B0(t), D1(t)=B'270(t), D3(t)=B90(t). Alternatively, in the above formula (13), the ranging operation unit 72 can use the average value of the following two: D0(t)=A0(t), D2(t)=A'180(t), D1(t)=A90(t), D3(t)=A'270(t); and D2(t)=B'180(t), D0(t)=B0(t), D1(t)=B'270(t), D3(t)=B90(t).

[0140] Subsequently, at the moment of supplying four detection signals (detection signals A180(t+1), B180(t+1), A270(t+1) and B270(t+1)) detected by the illumination light with a phase delay of 180° and the illumination light with a phase delay of 270°, the ranging operation unit 72 receives the correction detection signals A'0(t+1) and A'90(t+1), or the correction detection signals B'0(t+1) and B'90(t+1). Then, the ranging operation unit 72 can perform the operation shown in the following formula (14) to calculate the depth d(t+1) and confidence c(t+1) of the depth frame with frame number t+1.

[0141] (Formula 14)

[0142]

[0143] Q(t+1)=D1(t+1)-D3(t+1)

[0144] I(t+1)=DO(t-t1)-D2(t-t1)

[0145] However, in the above formula (14), the ranging operation unit 72 can use one of the following two: D2(t+1)=A180(t+1), D0(t+1)=A'0(t+1), D3(t+1)=A270(t+1), D1(t+1)=A'90(t+1); and D0(t+1)=B'0(t+1), D2(t+1)=B180(t+1), D1(t+1)=B'90(t+1), D3(t+1)=B270(t+1). Alternatively, in the above formula (14), the ranging operation unit 72 can use the average value of the following two: D0(t+1)=A'0(t+1), D2(t+1)=A'180(t+1), D1(t+1)=A90(t+1), D3(t+1)=A'270(t+1); and D2(t+1)=B'180(t+1), D0(t+1)=B0(t+1), D1(t+1)=B'270(t+1), D3(t+1)=B90(t+1).

[0146] The distance measurement operation processing unit 15 configured as described above can calculate the depth based on four detection signals detected by the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90°, or can calculate the depth based on four detection signals detected by the irradiation light with a phase delay of 180° and the irradiation light with a phase delay of 270°. Therefore, for example, the frame rate can be doubled compared to the case where the depth is calculated based on eight detection signals in the prior art.

[0147] In addition, when the frame rate is not increased, it is expected that the ranging operation processing unit 15 only needs to emit illumination light twice. Therefore, compared with the case of emitting illumination light four times in the prior art, the ranging operation processing unit 15 can reduce power consumption. In addition, the ranging operation processing unit 15 can reduce the number of detection signals expected to be detected so as to output a depth frame in a manner that the number of detection signals is halved compared to the prior art. Therefore, the ranging operation processing unit 15 can reduce the data transmission band.

[0148] Therefore, compared with the prior art, the distance measurement module 11 including the distance measurement operation processing unit 15 can improve its performance.

[0149] (First Processing Example of Distance Measurement Calculation Processing)

[0150] Fig.13 : is a flowchart for describing a first processing example of the distance measurement operation processing performed by the distance measurement operation processing unit 15 .

[0151] For example, when the ranging operation processing unit 15 is controlled to perform the ranging operation processing by a high-order control unit not shown, the ranging operation processing is started. In step S11, the ranging operation processing unit 15 obtains two detection signals for each of the two types of irradiation lights having different phase delays. That is, the ranging operation processing unit 15 obtains, for example, two detection signals A0 and B0 detected by the irradiation light with a phase delay of 0° and two detection signals A90 and B90 detected by the irradiation light with a phase delay of 90°. Alternatively, the ranging operation processing unit 15 obtains, for example, two detection signals A180 and B180 detected by the irradiation light with a phase delay of 180° and two detection signals A270 and B270 detected by the irradiation light with a phase delay of 270°.

[0152] In step S12 , the difference correction parameter calculation unit 61 determines whether the offset parameters (Offset_A and Offset_B) have been stored in the difference correction parameter storage unit 62 .

[0153] When the difference correction parameter calculation unit 61 determines in step S12 that the offset parameters (Offset_A and Offset_B) have not been stored in the difference correction parameter storage unit 62 , the process proceeds to step S13 .

[0154] In step S13, the differential correction parameter calculation unit 61 determines whether two detection signals have been obtained for each of the four types of illumination lights with different phase delays desired for calculating the offset parameters (Offset_A and Offset_B). For example, when eight detection signals of the detection signals A0 to A270 and the detection signals B0 to B270 have been obtained, the differential correction parameter calculation unit 61 determines that two detection signals have been obtained for each of the four types of illumination lights with different phase delays.

[0155] When the differential correction parameter calculation unit 61 determines in step S13 that two detection signals have not been obtained for each of the four types of irradiation lights having different phase delays, the process returns to step S11. In this case, for example, the detection signals A0 and A90 and the detection signals B0 and B90 have been obtained. Therefore, the differential correction parameter calculation unit 61 obtains the detection signals A180 and A270 and the detection signals B180 and B270 in the next step S11.

[0156] On the other hand, when the differential correction parameter calculation unit 61 determines in step S13 that two detection signals have been obtained for each of the four types of illumination light having different phase delays, the process proceeds to step S14 .

[0157] In step S14 , the difference correction parameter calculation unit 61 solves the simultaneous equations shown in the above equation (3) to calculate the offsets Offset_A and Offset_B.

[0158] Then, after the differential correction parameter calculation unit 61 stores the offsets Offset_A and Offset_B in the differential correction parameter storage unit 62, the process proceeds to step S15. On the other hand, when the differential correction parameter calculation unit 61 determines in step S12 that the offset parameters (Offset_A and Offset_B) are already stored in the differential correction parameter storage unit 62, the process proceeds to step S15.

[0159] In step S15, the differential correction parameter calculation unit 61 calculates the gain parameter (Gain_A / Gain_B) according to the above formula (4) or (5). Then, the differential correction parameter calculation unit 61 supplies the calculated gain parameter (Gain_A / Gain_B) to the correction operation unit 71, and the differential correction parameter storage unit 62 supplies the stored offset parameters (Offset_A and Offset_B) to the correction operation unit 71.

[0160] In step S16 , the correction operation unit 71 performs a correction operation on the four detection signals obtained in step S11 to obtain four corrected detection signals, and the correction operation unit 71 supplies the obtained four corrected detection signals to the distance measurement operation unit 72 .

[0161] For example, when the detection signals A0 and A90 and the detection signals B0 and B90 have been obtained in step S11, the correction operation unit 71 performs a correction operation according to the above formula (6) to obtain the corrected detection signals A'180 and A'270 and the corrected detection signals B'180 and B'270. In addition, when the detection signals A180 and A270 and the detection signals B180 and B270 have been obtained in step S11, the correction operation unit 71 performs a correction operation according to the above formula (7) to obtain the corrected detection signals A'0 and A'90 and the corrected detection signals B'0 and B'90.

[0162] In step S17 , the distance measurement operation unit 72 calculates the depth and the confidence level using the four detection signals obtained in step S11 and the four corrected detection signals obtained by the correction operation in step S16 .

[0163] For example, it is assumed that: detection signals A0 and A90 and detection signals B0 and B90 have been obtained in step S11, and corrected detection signals A'180 and A'270 and corrected detection signals B'180 and B'270 have been obtained in step S16. At this time, the ranging operation unit 72 performs the operation shown in the above formula (13) to calculate the depth and confidence. In addition, it is assumed that: detection signals A180 and A270 and detection signals B180 and B270 have been obtained in step S11, and corrected detection signals A'0 and A'90 and corrected detection signals B'0 and B'90 have been obtained in step S16. At this time, the ranging operation unit 72 performs the operation shown in the above formula (14) to calculate the depth and confidence.

[0164] In step S18 , the distance measurement operation processing unit 15 determines whether to continue the distance measurement according to control of the distance measurement operation processing by a higher-order control unit not shown.

[0165] When the distance measurement operation processing unit 15 determines in step S18 that the distance measurement is to be continued, the process returns to step S11 to repeatedly perform the same process. On the other hand, when the distance measurement operation processing unit 15 determines in step S18 that the distance measurement is not to be continued, the distance measurement operation process ends.

[0166] As described above, the ranging operation processing unit 15 can obtain the detection signals A0 and A90 and the detection signals B0 and B90, or can obtain the detection signals A180 and A270 and the detection signals B180 and B270, so as to calculate the depth and the confidence. Therefore, the ranging operation processing unit 15 can reduce the time expected to be used to detect the detection signal for calculating the depth and the confidence, and can improve the robustness.

[0167] (Second Configuration Example of Distance Measurement Operation Processing Unit)

[0168] Fig.14 is a block diagram showing a second configuration example of the distance measurement operation processing unit 15. Fig.14 In the distance measurement operation processing unit 15A shown in FIG. Fig.12 The same components of the distance measurement operation processing unit 15 are denoted by the same symbols, and detailed description of these same components will be omitted.

[0169] That is, the distance measurement operation processing unit 15A includes a correction parameter calculation unit 51 and a distance measurement unit 52A. Fig.12 Like the distance measurement operation processing unit 15 , the correction parameter calculation unit 51 includes a difference correction parameter calculation unit 61 and a difference correction parameter storage unit 62 .

[0170] and Fig.12 The distance measuring unit 52A has a correction operation unit 71 and a distance measuring operation unit 72, but the distance measuring unit 52A is different from the distance measuring unit 15. Fig.12 The distance measurement unit 52 of the distance measurement operation processing unit 15 is different in that the distance measurement unit 52A has a distance measurement result storage unit 73 and a result synthesis unit 74.

[0171] In addition, the distance measurement unit 52A is configured so that the depth d(t) and the confidence c(t) calculated by the distance measurement operation unit 72 as described above are supplied as distance measurement results to the distance measurement result storage unit 73 and the result synthesis unit 74. In addition, the distance measurement unit 52A is configured so that the distance measurement result of the previous frame (that is, the depth d(t-1) and the confidence c(t-1)) is supplied from the distance measurement result storage unit 73 to the result synthesis unit 74.

[0172] The ranging result storage unit 73 can store only the depth d(t) and the confidence c(t) of one frame supplied from the ranging operation unit 72 , and can supply the depth d(t−1) and the confidence c(t−1) of the previous frame to the result synthesis unit 74 .

[0173] The result synthesis unit 74 synthesizes the depth d(t) and confidence c(t) supplied from the ranging operation unit 72 and the depth d(t-1) and confidence c(t-1) supplied from the ranging result storage unit 73, and the result synthesis unit 74 outputs the calculated depth d(t) and confidence c(t) as their synthesis result.

[0174] Here, it is assumed that the depth d(t) and the confidence c(t) supplied from the ranging operation unit 72 to the ranging result storage unit 73 and the result synthesis unit 74 are the depth d'(t) and the confidence c'(t), respectively, and the synthesis result of the result synthesis unit 74 is the depth d(t) and the confidence c(t). In this case, according to the weighted operation shown in the following formula (15), the result synthesis unit 74 can synthesize the ranging results together using the weight g based on the confidence c'(t).

[0175] (Formula 15)

[0176]

[0177]

[0178] Therefore, the ranging operation processing unit 15A can synthesize the ranging result of the current frame and the ranging result of the previous frame (hereinafter also referred to as a sliding window) to improve the signal-to-noise (SN) ratio and reduce the noise of the synthesis result.

[0179] For example, when the detection periods Q0 to Q3 are the same as those of the case where the sliding window is not performed, the SN ratio of the ranging result using four detection signals detected in two detection periods Q0 and Q1 is reduced compared to the ranging result using eight detection signals detected in four detection periods Q0 to Q3. Therefore, the ranging operation processing unit 15A performs the sliding window using eight detection signals including the ranging result of the previous frame so as to synthesize the ranging results. Therefore, the ranging operation processing unit 15A can prevent the SN ratio from being reduced.

[0180] Furthermore, even if the detection period in one depth frame is reduced, the ranging operation processing unit 15A can perform a sliding window so as to achieve an increase in the SN ratio (frame×SNR / power) per power expected to be used to obtain a detection signal for one depth frame.

[0181] Therefore, since the sliding window reduces the noise, it is similar to Figure 4 Compared with the detection period, Fig.15 As shown, the distance measurement operation processing unit 15A can reduce the detection periods Q0 to Q3 by half. That is, the distance measurement operation processing unit 15A can double the speed of obtaining the detection signals A and B to double the frame rate.

[0182] Here, for example, when the sliding window is not performed, under the condition that the power of the detection signal expected to obtain one depth frame does not change and the frame rate is doubled, the SN ratio is reduced by an amount corresponding to the reduction in the detection periods Q0 to Q3. On the other hand, under the condition that the power of the detection signal expected to obtain one depth frame does not change, the ranging operation processing unit 15A may perform the sliding window to avoid the SN ratio from decreasing even if the frame rate is doubled.

[0183] Alternatively, if Fig.16 As shown, the frame rate does not change with the change of the detection period Q0~Q3 (the detection period Q0~Q3 and Figure 4 Under the condition that the detection periods Q0 to Q3 are the same) and the SN does not change, the power of the detection signal expected to be used to obtain a depth frame can be reduced. That is, the ranging operation processing unit 15A can perform a sliding window to reduce power consumption.

[0184] Note that in addition to the processing of the result synthesis unit 74 that performs weighted operation based on the confidence level to synthesize the ranging results, for example, the ranging operation processing unit 15A may synthesize the ranging results by simple averaging or weighting based on criteria other than the confidence level.

[0185] In addition, for example, the process of combining the distance measurement results by the result combining unit 74 can be applied to the following configuration: Figure 4 The four detection periods Q0 to Q3 output one depth frame. That is, the application of the process is not limited to the following configuration: one depth frame is output based on four detection signals detected in two detection periods Q0 and Q1, or one depth frame is output based on four detection signals detected in two detection periods Q2 and Q3.

[0186] (Second Processing Example of Distance Measurement Operation Processing)

[0187] Fig.17 : is a flowchart for describing a second processing example of the distance measurement operation processing performed by the distance measurement operation processing unit 15A.

[0188] In steps S21 to S27, the Fig.13 The processing of steps S11 to S17 is the same as that of steps S11 to S17.

[0189] Then, in step S27, the calculated depth and confidence are supplied to the ranging result storage unit 73 and the result synthesis unit 74. The result synthesis unit 74 determines whether the ranging result has been stored in the ranging result storage unit 73 in step S28.

[0190] When the result synthesis unit 74 determines in step S28 that the ranging result has not been stored in the ranging result storage unit 73, the process returns to step S21. That is, in this case, the depth and confidence of the previous frame have not been stored in the ranging result storage unit 73, and therefore, the result synthesis unit 74 does not perform the process of synthesizing the ranging results together.

[0191] On the other hand, when the result synthesis unit 74 determines in step S28 that the distance measurement result has been stored in the distance measurement result storage unit 73 , the process proceeds to step S29 .

[0192] In step S29, the result synthesis unit 74 reads out the depth and confidence of the previous frame from the ranging result storage unit 73. Then, the result synthesis unit 74 performs a weighted operation on the depth and confidence supplied in step S27 and the depth and confidence of the previous frame read out from the ranging result storage unit 73 according to the confidence, so as to output a synthesized ranging result in which the measurement results are synthesized together.

[0193] Then, in step S30, the Fig.13 The same processing as the processing of step S18 is performed. When it is determined that the distance measurement is not to be continued, the distance measurement calculation processing ends.

[0194] As described above, the ranging operation processing unit 15A can combine the measurement results according to the weighted operation based on the confidence level to reduce the SN ratio of the measurement results and perform ranging more accurately. In addition, the ranging operation processing unit 15A can increase the frame rate (see Fig.15 ) or reduce power consumption (see Fig.16 ).

[0195] (Operation of the light emitting unit and the light receiving unit)

[0196] Reference Figure 18 to Figure 21 The actions of the light emitting unit 12 and the light receiving unit 14 are described.

[0197] Fig.18 An example of the timing of emitting light and receiving light to output a depth map is shown.

[0198] For example, the ranging module 11 may set one frame for outputting a depth map as one subframe, and this subframe is divided into four detection periods, namely, detection periods Q0, Q1, Q2 and Q3. In addition, during the integration period of each of the detection periods Q0, Q1, Q2 and Q3, the light emitting unit 12 emits irradiation light at a time corresponding to the modulated signal, and the light receiving unit 14 receives the reflected light generated by the irradiation light. Figure 1As described above, the charges generated by one photodiode 31 are distributed to the taps 32A and 32B according to the distribution signals DMIX_A and DMIX_B, and the charges are accumulated in a manner corresponding to the amount of light received during the integration period.

[0199] Here, in Figure 4 In the above example shown in FIG. 1 , a standby period corresponding to one depth frame is provided after the detection periods Q0, Q1, Q2, and Q3. Fig.18 In the illustrated example, a standby period divided into four is provided after each of the detection periods Q0 , Q1 , Q2 , and Q3 .

[0200] Therefore, by providing a standby period after each of the detection periods Q0, Q1, Q2, and Q3, the intervals between the respective integration periods can be made uniform.

[0201] That is, Fig.19 As shown, the emission timing of the irradiation light with a phase delay of 0°, the emission timing of the irradiation light with a phase delay of 90°, the emission timing of the irradiation light with a phase delay of 180°, and the emission timing of the irradiation light with a phase delay of 270° are uniformly set. Therefore, by adopting the emission timings set at uniform intervals, for example, when the distance measurement operation processing unit 15A performs a sliding window, adverse effects due to their different intervals can be prevented.

[0202] In addition, you can use Fig. 20 As described above, the distance measurement operation processing unit 15 obtains one depth frame from the four detection signals A0, B0, A90 and B90, and obtains one depth frame from the four detection signals A180, B180, A270 and B270.

[0203] Therefore, if Fig. 20 As shown, it is expected that the emission time of the illumination light with a phase delay of 0° and the emission time of the illumination light with a phase delay of 90° for obtaining one depth frame are close to each other, and it is expected that the emission time of the illumination light with a phase delay of 180° and the emission time of the illumination light with a phase delay of 270° for obtaining the next depth frame are close to each other. For example, by making the emission time for obtaining one depth frame close, when the object moves, the influence caused by the separated emission time can be prevented by the movement of the object.

[0204] Furthermore, by making the interval between the light emission timing for obtaining one depth frame and the light emission timing for obtaining the next depth frame uniform, when the ranging operation processing unit 15A performs a sliding window, adverse effects due to their different intervals can be prevented.

[0205] In addition, you can use Fig.21That is, if the offsets Offset_A and Offset_B are calculated in advance, the distance measurement operation processing unit 15 can obtain a depth frame using only the illumination light with a phase delay of 0° and the illumination light with a phase delay of 90°.

[0206] It should be noted that the light-emitting timing of the light-emitting unit 12 is not limited to Figure 18 to Figure 21 Examples are shown, and various other light emission timings may be employed.

[0207] (Third Configuration Example of Distance Measurement Operation Processing Unit)

[0208] Fig. 22 is a block diagram showing a third configuration example of the ranging operation processing unit 15 .

[0209] Fig. 22 The illustrated distance measurement operation processing unit 15B includes a detection signal storage unit 81 , a motion detection unit 82 , a four-phase distance measurement operation unit 83 , a two-phase distance measurement operation unit 84 , a measurement result storage unit 85 , and a result synthesis unit 86 .

[0210] In addition, with reference Fig.12 As in the above case, the distance measurement operation processing unit 15B receives four detection signals detected by the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90°, and receives four detection signals detected by the irradiation light with a phase delay of 180° and the irradiation light with a phase delay of 270°. That is, the distance measurement operation processing unit 15B receives the detection signals A0(t), B0(t), A90(t) and B90(t), and then receives the detection signals A180(t+1), B180(t+1), A270(t+1) and B270(t+1).

[0211] The detection signal storage unit 81 is capable of storing four detection signals. Whenever the detection signal storage unit 81 receives four detection signals, the detection signal storage unit 81 supplies the previous four detection signals to the motion detection unit 82.

[0212] That is, at the moment when the detection signal storage unit 81 receives the detection signals A0(t), B0(t), A90(t), and B90(t), the detection signal storage unit 81 has already stored the detection signals A180(t-1), B180(t-1), A270(t-1), and B270(t-1), and therefore, the detection signal storage unit 81 supplies the already stored detection signals to the motion detection unit 82. In addition, at the moment when the detection signal storage unit 81 receives the detection signals A180(t+1), B180(t+1), A270(t+1), and B270(t+1), the detection signal storage unit 81 has already stored the detection signals A0(t), B0(t), A90(t), and B90(t), and therefore, the detection signal storage unit 81 supplies the already stored detection signals to the motion detection unit 82.

[0213] The motion detection unit 82 detects the motion of the object for each pixel of the light receiving unit 14 , and the motion detection unit 82 determines whether the object in motion has been reflected based on a prescribed threshold value th.

[0214] That is, at the moment when the motion detection unit 82 receives the detection signals A0(t), B0(t), A90(t) and B90(t), the motion detection unit 82 makes a determination based on the determination condition shown in the following equation (16).

[0215] (Formula 16)

[0216]

[0217] For example, when the determination condition of the above formula (16) is satisfied, the motion detection unit 82 determines that the object in motion is not reflected in the depth frame obtained based on the detection signals A0(t), B0(t), A90(t) and B90(t). In this case, the motion detection unit 82 outputs a motion-object detection signal M(t)=0, M(t)=0 indicates that the object in motion is not reflected, and the motion detection unit 82 supplies the detection signals A0(t), B0(t), A90(t) and B90(t) to the four-phase ranging operation unit 83. In this case, the motion detection unit 82 also supplies the detection signals A180(t-1), B180(t-1), A270(t-1) and B270(t-1) supplied from the detection signal storage unit 81 to the four-phase ranging operation unit 83.

[0218] On the other hand, when the determination condition of the above formula (16) is not satisfied, the motion detection unit 82 determines that an object in motion has been reflected in the depth frame obtained based on the detection signals A0(t), B0(t), A90(t) and B90(t). In this case, the motion detection unit 82 outputs a motion-object detection signal M(t)=1, M(t)=1 indicating that an object in motion has been reflected, and the motion detection unit 82 supplies the detection signals A0(t), B0(t), A90(t) and B90(t) to the two-phase ranging operation unit 84.

[0219] Similarly, at the moment when the motion detection unit 82 receives the detection signals A180(t+1), B180(t+1), A270(t+1) and B270(t+1), the motion detection unit 82 makes a determination based on the determination condition shown in the following formula (17).

[0220] (Formula 17)

[0221]

[0222] For example, when the determination condition of the above formula (17) is satisfied, the motion detection unit 82 determines that the object in motion is not reflected in the depth frame obtained based on the detection signals A180 (t + 1), B180 (t + 1), A270 (t + 1) and B270 (t + 1). In this case, the motion detection unit 82 outputs a motion-object detection signal M (t) = 0 indicating that the object in motion is not reflected, and the motion detection unit 82 supplies the detection signals A180 (t + 1), B180 (t + 1), A270 (t + 1) and B270 (t + 1) to the four-phase ranging operation unit 83. In this case, the motion detection unit 82 also supplies the detection signals A0 (t), B0 (t), A90 (t) and B90 (t) supplied from the detection signal storage unit 81 to the four-phase ranging operation unit 83.

[0223] On the other hand, when the determination condition of the above formula (17) is not satisfied, the motion detection unit 82 determines that an object in motion has been reflected in the depth frame obtained based on the detection signals A180 (t + 1), B180 (t + 1), A270 (t + 1) and B270 (t + 1). In this case, the motion detection unit 82 outputs a motion-object detection signal M (t) = 1 indicating that an object in motion has been reflected, and the motion detection unit 82 supplies the detection signals A180 (t + 1), B180 (t + 1), A270 (t + 1) and B270 (t + 1) to the two-phase ranging operation unit 84.

[0224] When the motion detection unit 82 determines that a moving object has not been reflected, the four-phase ranging operation unit 83 performs the following processing (hereinafter referred to as four-phase ranging operation processing): ranging is performed by using operations on eight detection signals detected by illumination light with a phase delay of 0°, illumination light with a phase delay of 90°, illumination light with a phase delay of 180°, and illumination light with a phase delay of 270°.

[0225] For example, in this case, the motion detection unit 82 supplies detection signals A180(t-1), B180(t-1), A270(t-1), B270(t-1), A0(t), B0(t), A90(t) and B90(t) to the four-phase ranging operation unit 83.

[0226] Therefore, the four-phase ranging operation unit 83 performs operation according to the following formula (18) to calculate the depth d(t) and the confidence c(t), and supplies the calculated depth d(t) and confidence c(t) to the ranging result storage unit 85 and the result synthesis unit 86.

[0227] (Formula 18)

[0228]

[0229] Q(t)=D1(t)-D3(t)

[0230] I(t)=D0(t)-D2(t)

[0231] D0(t)=A0(t)-B0(t)

[0232] D1(t)=A90(t)-B90(t)

[0233] D2(t)=A180(t-1)-B180(t-1)

[0234] D3(t)=A270(t-1)-B270(t-1)

[0235] Similarly, the four-phase ranging operation unit 83 can use the detection signals A0(t), B0(t), A90(t), B90(t), A180(t+1), B180(t+1), A270(t+1) and B270(t+1) to calculate the depth d(t+1) and the confidence c(t+1).

[0236] The two-phase distance measurement operation unit 84 has Fig.12 The distance measurement operation processing unit 15 has the same function, and the two-phase distance measurement operation unit 84 includes Fig.12 The correction parameter calculation unit 51 and the distance measurement unit 52 are shown.

[0237] That is, when the motion detection unit 82 determines that a moving object has been reflected, the two-phase ranging operation unit 84 performs the following processing (hereinafter referred to as two-phase ranging operation processing): ranging is performed by using the operation of four detection signals detected by the irradiation light with a phase delay of 0° and the irradiation light with a phase delay of 90° or the four detection signals detected by the irradiation light with a phase delay of 180° and the irradiation light with a phase delay of 270°. Then, the two-phase ranging operation unit 84 supplies the depth d(t) and the confidence c(t) calculated by the two-phase ranging operation processing to the ranging result storage unit 85 and the result synthesis unit 86.

[0238] The distance measurement result storage unit 85 and the result synthesis unit 86 have Fig.14 The distance measurement result storage unit 85 has the same function as the distance measurement result storage unit 73 and the result synthesis unit 74. That is, the distance measurement result storage unit 85 can supply the distance measurement result of the previous frame to the result synthesis unit 86, and the result synthesis unit 86 can synthesize the distance measurement result of the current frame and the distance measurement result of the previous frame together.

[0239] like Fig.23 As shown, the distance measurement operation processing unit 15B configured in this way can synthesize two consecutive depth frames into one depth frame according to the motion detection results of each frame to be output.

[0240] For example, when the motion detection using the ranging result before the previous frame number t-1 is synthesized with other ranging results indicates that the moving object has been reflected at the time of outputting the depth frame with the frame number t, the ranging operation processing unit 15B directly outputs the ranging result with the frame number t as the depth frame. On the other hand, when the motion detection using the ranging result before the previous frame number t-1 is synthesized with other ranging results indicates that the moving object has not been reflected at the time of outputting the depth frame with the frame number t, the ranging operation processing unit 15B outputs the synthesized ranging result obtained by synthesizing with the ranging result with the frame number t-1 as the depth frame with the frame number t.

[0241] As described above, the ranging operation processing unit 15B can switch between the four-phase ranging operation processing and the two-phase ranging operation processing according to the motion detection result. Therefore, for example, when a moving object has been reflected, the ranging operation processing unit 15B can perform a two-phase ranging operation processing to calculate a depth frame at a higher frame rate and improve the measurement accuracy of the moving object. Therefore, the ranging operation processing unit 15B can improve the robustness of the moving object. In addition, when a moving object has not yet been reflected, the ranging operation processing unit 15B can perform a four-phase ranging operation processing to further reduce noise.

[0242] (Third Processing Example of Distance Measurement Operation Processing)

[0243] Fig.24 : is a flowchart for describing a third processing example of the distance measurement operation processing performed by the distance measurement operation processing unit 15B.

[0244] In step S41, the Fig.13 That is, the distance measurement operation processing unit 15B obtains two detection signals for each of the two types of irradiation lights having different phase delays.

[0245] In step S42 , the motion detection unit 82 determines whether the detection signal has been stored in the detection signal storage unit 81 .

[0246] When the motion detection unit 82 determines in step S42 that the detection signal has not been stored in the detection signal storage unit 81, the process returns to step S41. That is, the detection signal of the previous frame has not been stored in the detection signal storage unit 81, and therefore, the motion detection unit 82 does not perform processing for detecting motion.

[0247] On the other hand, when the motion detection unit 82 determines in step S42 that the detection signal has been stored in the detection signal storage unit 81, the process proceeds to step S43. In step S43, the motion detection unit 82 determines whether a moving object has been reflected according to the determination condition shown in the above formula (16) or (17).

[0248] When the motion detection unit 82 determines in step S43 that a moving object has not been reflected, the process proceeds to step S44. In step S44, the four-phase ranging operation unit 83 performs the four-phase ranging operation process as described above to calculate the depth and the confidence, and supplies the calculated depth and confidence as ranging results to the ranging result storage unit 85 and the result synthesis unit 86. Then, the process proceeds to step S46.

[0249] On the other hand, when the motion detection unit 82 determines in step S43 that a moving object has been reflected, the process proceeds to step S45. In step S45, the two-phase ranging operation unit 84 performs the two-phase ranging operation process as described above to calculate the depth and the confidence, and supplies the calculated depth and confidence as ranging results to the ranging result storage unit 85 and the result synthesis unit 86. Then, the process proceeds to step S46.

[0250] In steps S46 to S48, the Fig.17 The same processing as the processing of steps S28 to S30 is performed. If it is determined in step S48 that the distance measurement is not to be continued, the distance measurement calculation processing ends.

[0251] As described above, the ranging operation processing unit 15B can switch between the four-phase ranging operation processing and the two-phase ranging operation processing according to the motion detection result to perform appropriate ranging on the moving object.

[0252] It should be noted that the present technology is applicable to a system for modulating the amplitude of light emitted to an object in an indirect ToF system called a continuous wave system. In addition, the structure of the photodiode 31 of the light receiving unit 14 is not limited to a depth sensor having a current assisted photonic demodulator (CAPD: Current Assisted Photonic Demodulator) structure, but is applicable to a depth sensor having a structure in which charges are distributed to two taps 32A and 32B.

[0253] (Configuration example of electronic equipment)

[0254] The distance measurement module 11 as described above may be installed in an electronic device such as a smart phone.

[0255] Fig.25 is a block diagram showing a configuration example of an imaging device installed in an electronic device.

[0256] like Fig.25 As shown, the electronic device 101 includes a distance measurement module 102, a camera 103, a display 104, a speaker 105, a microphone 106, a communication module 107, a sensor unit 108, a touch panel 109, and a control unit 110. In addition, when the CPU runs the program, the control unit 110 has the functions of an application processing unit 121 and an operating system processing unit 122.

[0257] Figure 1 The ranging module 11 is used as the ranging module 102. For example, the ranging module 102 is arranged on the front surface of the electronic device 101, and the ranging module 102 can perform ranging with the user of the electronic device 101 as a target to output the depth of the surface shape of the user's face, hand or finger, etc. as a ranging result.

[0258] The camera 103 is disposed on the front surface of the electronic device 101, and performs imaging with the user of the electronic device 101 as the subject to obtain an image of the user. Note that the camera 103 may also be disposed on the rear surface of the electronic device 101, although not shown.

[0259] The display 104 displays an operation screen for executing processing of the application processing unit 121 and the operating system processing unit 122 or an image captured by the camera 103. When a call is started with the electronic device 101, the speaker 105 and the microphone 106 output the voice of the person on the other side and collect the user's voice.

[0260] The communication module 107 performs communication via a communication network. The sensor unit 108 detects speed, acceleration, proximity, or the like, and the touch panel 109 obtains a touch operation of a user on an operation screen displayed on the display 104 .

[0261] The application processing unit 121 performs processing for providing various services by the electronic device 101. For example, the application processing unit 121 may perform processing of generating a face through computer graphics in which the facial expression of the user is virtually reproduced based on the depth supplied from the ranging module 102, and displaying the generated face on the display 104. For example, the application processing unit 121 may perform processing of generating a 3D shape of any object based on the depth supplied from the ranging module 102.

[0262] The operating system processing unit 122 performs processing for implementing basic functions and actions of the electronic device 101. For example, the operating system processing unit 122 may perform processing for recognizing the face of the user based on the depth supplied from the ranging module 102 and unlocking the electronic device 101. In addition, the operating system processing unit 122 may perform processing for recognizing the gesture of the user based on the depth supplied from the ranging module 102 and inputting various operations according to the gesture.

[0263] By applying the above ranging module 11, the electronic device 101 configured in this way can achieve, for example, an increase in frame rate, a reduction in power consumption, and a reduction in data transmission band. Therefore, the electronic device 101 can produce a face that moves more smoothly through computer graphics, can recognize a face with high accuracy, can reduce battery consumption, or can transmit narrowband data.

[0264] (Computer configuration example)

[0265] Next, the above series of processing can be executed not only by hardware but also by software. When the series of processing is executed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0266] Fig.26 : is a block diagram showing a configuration example of an embodiment of a computer installed with a program for executing the above series of processes.

[0267] In the computer, a central processing unit (CPU: Central Processing Unit) 201, a read only memory (ROM: Read Only Memory) 202, a random access memory (RAM: Random Access Memory) 203, and an electrically erasable programmable read only memory (EEPROM: Electronically Erasable and Programmable ReadOnly Memory) 204 are connected to each other via a bus 205. The bus 205 is also connected to an input / output interface 206, and the input / output interface 206 is connected to the outside.

[0268] In the computer configured as described above, the CPU 201 loads the program stored in the ROM 202 and the EEPROM 204 into the RAM 203 via the bus 205 to be executed. Thus, the above series of processing is performed. In addition, in addition to being written in advance to the ROM 202, the program executed by the computer (CPU 201) can be installed or updated in the EEPROM 205 from the outside via the input / output interface 206.

[0269] Thus, the CPU 201 executes processing according to the above flowchart, or executes processing executed according to the configuration of the above block diagram. Then, if necessary, the CPU 201 can output the processing result to the outside via, for example, the input / output interface 206.

[0270] Here, in this specification, the processing performed by the computer according to the program may not necessarily be performed in time in the order described in the flowchart. That is, the processing performed by the computer according to the program includes processing performed in parallel or individually (for example, parallel processing or processing according to objects).

[0271] In addition, the program may be processed by one computer (processor), or may be processed in a distributed manner by a plurality of computers. In addition, the program may be transmitted to a remote computer and executed.

[0272] In addition, the system in this specification refers to a collection of multiple components (such as devices or modules (components)), and all components may not necessarily be housed in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network, and a device in which multiple modules are housed in one housing are both systems.

[0273] In addition, the configuration described above as one device (or one processing unit) may be divided and configured as a plurality of devices (or a plurality of processing units). Conversely, the configuration described above as a plurality of devices (or a plurality of processing units) may be uniformly configured as one device (or a processing unit). In addition, configurations other than the above configurations may naturally be added to the configurations of the above-mentioned respective devices (or respective processing units). In addition, if the entire system has substantially the same configuration or operation, a portion of the configuration of a device (or processing unit) may be included in the configuration of other devices (or other processing units).

[0274] Furthermore, for example, the present technology may adopt a cloud computing configuration in which one function is shared and cooperatively processed among a plurality of devices via a network.

[0275] In addition, for example, the above program can be executed in any device. In this case, it is expected that the device can have necessary functions (such as functional blocks) and can obtain necessary information.

[0276] In addition, the various steps described in the above flowchart can be performed not only by one device but also by multiple devices in a common manner. In addition, when one step includes multiple processes, the multiple processes included in this one step can be performed not only by one device but also by multiple devices in a common manner. In other words, the multiple processes included in one step can be performed as processes of multiple steps. Conversely, the processes described as multiple steps can be uniformly performed as one step.

[0277] It should be noted that the program executed by the computer may be a program in which the processing of the steps describing the program is performed in time in the order described in this specification, or the processing of these steps is performed in parallel or separately at an appropriate time (for example, when called). That is, unless any contradiction occurs, the processing of the individual steps may be performed in an order different from the above order. In addition, the processing of the steps describing the program may be performed in parallel with the processing of other programs, or may be performed in combination with the processing of other programs.

[0278] It should be noted that unless any contradiction occurs, the present technology described multiple times in this specification can be performed separately. Of course, any number of multiple present technologies can be performed in combination. For example, part or all of the present technology described in any embodiment can be performed in combination with part or all of the present technology described in other embodiments. In addition, part or all of the above-mentioned present technology can be performed in combination with other technologies not described above.

[0279] (Application example of mobile object)

[0280] The technology according to the embodiments of the present invention (the present technology) can be applied to various products. For example, the technology according to the embodiments of the present invention can be implemented as a device installed in any type of movable body such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobility device (personal mobility), an airplane, a drone, a ship, and a robot.

[0281] Fig. 27 : is a block diagram depicting a schematic configuration example of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present invention can be applied.

[0282] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. Fig. 27 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and an in-vehicle network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050.

[0283] The drive system control unit 12010 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for the following devices: a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine or a drive motor, etc.; a drive force transmitting mechanism for transmitting the drive force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating the braking force of the vehicle, etc.

[0284] The body system control unit 12020 controls the operation of various devices provided to the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for the following devices: a keyless entry system, a smart key system, an automatic window device, or various lights such as headlights, taillights, brake lights, turn signals, or fog lights. In this case, radio waves sent from a mobile device that replaces the key or signals of various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals and controls the door lock device, automatic window device, or lights, etc. of the vehicle.

[0285] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the camera unit 12031. The vehicle exterior information detection unit 12030 causes the camera unit 12031 to capture an image of the exterior of the vehicle, and the vehicle exterior information detection unit 12030 receives the captured image. Based on the received image, the vehicle exterior information detection unit 12030 may perform a process of detecting an object such as a pedestrian, a vehicle, an obstacle, a sign, or text on the road surface, or perform a process of detecting the distance to the above-mentioned object.

[0286] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 may output the electrical signal as an image, or may output the electrical signal as information about distance measurement. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0287] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 is connected to a driver state detection unit 12041 for detecting the state of the driver. For example, the driver state detection unit 12041 includes a camera for photographing the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off.

[0288] The microcomputer 12051 can calculate the control target value of the driving force generating device, the steering mechanism, or the braking device based on the information about the inside or outside of the vehicle (the information obtained by the outside information detection unit 12030 or the inside information detection unit 12040), and the microcomputer 12051 can output the control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform the coordinated control aimed at realizing the functions of the advanced driver assistance system (ADAS), which includes vehicle collision avoidance or vehicle impact mitigation, following driving based on the vehicle-to-vehicle distance, speed maintenance driving, vehicle collision warning, or vehicle lane departure warning.

[0289] In addition, based on information about the outside or inside of the vehicle (information obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040), the microcomputer 12051 can perform collaborative control of automatic driving aimed at enabling the vehicle to drive autonomously without relying on the driver's operation by controlling the driving force generating device, steering mechanism or braking device, etc.

[0290] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on information about the outside of the vehicle (information obtained by the vehicle exterior information detection unit 12030). For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to change from high beam to low beam according to the position of the preceding vehicle or the oncoming vehicle detected by the vehicle exterior information detection unit 12030.

[0291] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device, which can visually or auditorily notify the occupants of the vehicle or the outside of the vehicle of the information. Fig. 27 In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and a dashboard 12063 are shown as output devices. For example, the display portion 12062 may include at least one of an in-vehicle display and a head-up display.

[0292] Fig.28 This is a diagram illustrating an example of the installation position of the camera unit 12031.

[0293] exist Fig.28 In the figure, the camera unit 12031 includes camera units 12101, 12102, 12103, 12104 and 12105.

[0294] For example, the camera units 12101, 12102, 12103, 12104 and 12105 are arranged at positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 12100 and at positions on the upper part of the windshield inside the vehicle. The camera unit 12101 arranged on the front nose and the camera unit 12105 arranged on the upper part of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The camera units 12102 and 12103 arranged on the side mirrors mainly obtain images on both sides of the vehicle 12100. The camera unit 12104 arranged on the rear bumper or rear door mainly obtains images behind the vehicle 12100. The camera unit 12105 arranged on the upper part of the windshield inside the vehicle is mainly used to detect the front vehicle, pedestrians, obstacles, signal lights, traffic signs or lanes, etc.

[0295] By the way, Fig.28Examples of the imaging ranges of the imaging units 12101 to 12104 are depicted. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose. The imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors. The imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or the rear door. For example, by superimposing the image data captured by the imaging units 12101 to 12104, it is possible to obtain a bird's-eye view image of the vehicle 12100 as viewed from above.

[0296] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0297] For example, the microcomputer 12051 can determine the distance to each solid object within the imaging range 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby specifically extract the following closest solid object as the front vehicle: the solid object exists on the driving road of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in the same direction as the vehicle 12100. In addition, the microcomputer 12051 can pre-set the inter-vehicle distance to be maintained in front of the front vehicle, and can perform automatic braking control (including follow-up stop control) or automatic acceleration control (including follow-up start control) and the like. Therefore, it is possible to perform cooperative control of automatic driving that is intended to enable the vehicle to travel autonomously without relying on the operation of the driver or the like.

[0298] For example, the microcomputer 12051 can classify the three-dimensional object data of the three-dimensional object into three-dimensional object data of two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, electric poles and other three-dimensional objects based on the distance information obtained from the camera units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can visually identify and obstacles that the driver of the vehicle 12100 cannot visually identify. Then, the microcomputer 12051 determines the collision risk for indicating the danger of collision with each obstacle. When the collision risk is above the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the drive system control unit 12010. Therefore, the microcomputer 12051 can assist driving to avoid collision.

[0299] At least one of the camera units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 is capable of identifying pedestrians by determining whether there are pedestrians in the images captured by the camera units 12101 to 12104. For example, such identification of pedestrians is performed by extracting feature points in the captured images of the camera units 12101 to 12104 as infrared cameras, and determining whether the object is a pedestrian by pattern matching a series of feature points representing the contour of the object. When the microcomputer 12051 determines that there are pedestrians in the captured images of the camera units 12101 to 12104 and thus identifies the pedestrians, the sound / image output unit 12052 controls the display unit 12062 so that the square contour line used for emphasis is displayed in a manner superimposed with the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 so that an icon representing a pedestrian or the like is displayed at a desired position.

[0300] In the above, an example of a vehicle control system to which the technology according to an embodiment of the present invention can be applied has been described. The technology according to an embodiment of the present invention can be applied to the in-vehicle information detection unit 12040 in the above-mentioned configuration. Specifically, when the ranging module 11 is used for ranging, the state of the driver can be detected more accurately. In addition, when the ranging module 11 is used for ranging, a process for recognizing the driver's gesture can be performed so as to perform various operations according to the gesture.

[0301] (Example of constructed combination)

[0302] It should be noted that the present technology can adopt the following configurations.

[0303] (1) A distance measurement processing device, comprising:

[0304] a four-phase distance measurement operation unit configured to: when allocating charges to a first tap and a second tap according to a distance to an object, perform an operation for calculating a depth representing the distance to the object by using all eight detection signals, the charges being generated by irradiating irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase onto the object and receiving reflected light reflected by the object, and for each of the irradiation light of the first phase to the irradiation light of the fourth phase, two detection signals among the eight detection signals can be detected;

[0305] a two-phase distance measurement operation unit configured to perform the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and

[0306] A condition determination unit is configured to perform condition determination based on the detection signal and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit to be used.

[0307] (2) The distance measurement processing device according to (1), wherein

[0308] In a first detection period for receiving the reflected light generated by the irradiation light of the first phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a first detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a second detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected,

[0309] In a second detection period for receiving the reflected light generated by the irradiation light of the second phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a third detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a fourth detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected,

[0310] In a third detection period of receiving the reflected light generated by the irradiation light of the third phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a fifth detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a sixth detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected, and

[0311] During the fourth detection period of receiving the reflected light generated by the irradiation light of the fourth phase, the charge is configured to be alternately distributed to the first tap and the second tap multiple times, and a seventh detection signal corresponding to the charge distributed to the first tap and accumulated in the first tap, and an eighth detection signal corresponding to the charge distributed to the second tap and accumulated in the second tap are detected.

[0312] (3) The distance measurement processing device according to (2), wherein

[0313] The four-phase ranging operation unit is configured to eliminate the influence caused by the characteristic difference between the first tap and the second tap by using the difference between the first detection signal and the second detection signal, the difference between the third detection signal and the fourth detection signal, the difference between the fifth detection signal and the sixth detection signal, and the difference between the seventh detection signal and the eighth detection signal, so as to calculate the depth.

[0314] (4) The distance measurement processing device according to (3), wherein

[0315] The two-phase ranging operation unit is configured to perform the following operations alternately:

[0316] After calculating a correction parameter for correcting a characteristic difference between the first tap and the second tap by using the first to fourth detection signals, calculating the depth based on the first to fourth detection signals and the correction parameter, and

[0317] After calculating a correction parameter for correcting the characteristic difference between the first tap and the second tap by using the fifth to eighth detection signals, the depth is calculated based on the fifth to eighth detection signals and the correction parameter.

[0318] (5) The distance measurement processing device according to any one of (1) to (4), further comprising:

[0319] a measurement result storage unit configured to store the depth calculated by one of the four-phase distance measurement operation unit and the two-phase distance measurement operation unit; and

[0320] A result synthesis unit is configured to synthesize the depth of the current frame and the depth of the previous frame stored in the measurement result storage unit and output the synthesized depth.

[0321] (6) The distance measurement processing device according to (5), wherein

[0322] The four-phase distance measurement operation unit and the two-phase distance measurement operation unit are configured to calculate the confidence with respect to the depth and the depth,

[0323] The measurement result storage unit is configured to store the confidence level and the depth together, and

[0324] The result synthesis unit is configured to perform weighted addition corresponding to the confidence level to synthesize the depth of the previous frame and the depth of the current frame together.

[0325] (7) The distance measurement processing device according to (5) or (6), wherein

[0326] The result synthesis unit is configured to synthesize the depth of the previous frame calculated by the four-phase ranging operation unit using all the eight detection signals and the depth of the current frame calculated by the four-phase ranging operation unit using all the eight detection signals.

[0327] (8) The distance measurement processing device according to (5) or (6), wherein:

[0328] Among the depth calculated by the two-phase ranging operation unit using the four detection signals based on the illumination light of the first phase and the illumination light of the second phase, and the depth calculated by the two-phase ranging operation unit using the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase, the result synthesis unit is configured to identify one depth as the depth of the previous frame and the other depth as the depth of the current frame, and the result synthesis unit is configured to synthesize the two depths together.

[0329] (9) The distance measurement processing device according to any one of (1) to (8),

[0330] The condition determination unit is configured to perform the condition determination on each pixel of the light receiving unit that receives the reflected light, and the condition determination unit is configured to switch between performing the operation of calculating the depth by the four-phase ranging operation unit and performing the operation of calculating the depth by the two-phase ranging operation unit for each of the pixels.

[0331] (10) The distance measurement processing device according to any one of (1) to (9),

[0332] The condition determination unit is configured to:

[0333] The condition determination is performed based on detecting whether the object is moving by comparing the four detection signals based on the illumination light of the first phase and the illumination light of the second phase with the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase,

[0334] When the object is detected to be in motion, the two-phase ranging operation unit performs the operation of calculating the depth, and

[0335] When the object is not detected to be in motion, the operation of calculating the depth is performed by the four-phase ranging operation unit.

[0336] (11) The distance measurement processing device according to any one of (1) to (9),

[0337] The condition determination unit is configured to:

[0338] The condition determination is performed based on the brightness calculated from the four detection signals based on the illumination light of the first phase and the illumination light of the second phase and the brightness calculated from the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase, and

[0339] Switching is performed between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit according to the brightness.

[0340] (12) The distance measurement processing device according to any one of (1) to (9),

[0341] The four-phase distance measurement operation unit and the two-phase distance measurement operation unit are configured to calculate the confidence level relative to the depth and the depth, and

[0342] The condition determination unit is configured to perform the condition determination based on the confidence calculated in a previous frame, and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit according to the confidence.

[0343] (13) The distance measurement processing device according to any one of (1) to (12),

[0344] The first detection period, the second detection period, the third detection period, and the fourth detection period are arranged at substantially even intervals.

[0345] (14) A distance measurement module, comprising:

[0346] a light emitting unit configured to irradiate the illumination light of the first phase, the illumination light of the second phase, the illumination light of the third phase, and the illumination light of the fourth phase onto the object;

[0347] a light receiving unit configured to output eight detection signals when allocating charges generated by receiving reflected light reflected by the object to a first tap and a second tap according to a distance to the object, wherein two detection signals among the eight detection signals can be detected for each of the illumination light of the first phase to the illumination light of the fourth phase;

[0348] a four-phase distance measurement operation unit configured to perform an operation of calculating a depth representing the distance to the object by using all of the eight detection signals;

[0349] a two-phase distance measurement operation unit configured to perform the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and

[0350] A condition determination unit is configured to perform condition determination based on the detection signal and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit.

[0351] (15) A ranging processing method, comprising:

[0352] performing a four-phase distance measurement operation process: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating the object with irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase and receiving reflected light reflected by the object, and two detection signals among the eight detection signals can be detected for each of the irradiation light of the first phase to the irradiation light of the fourth phase;

[0353] performing two-phase distance measurement operation processing: performing the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and

[0354] Condition determination is performed based on the detection signal, and switching is performed between the four-phase distance measurement operation process and the two-phase distance measurement operation process to be used.

[0355] (16) A program for causing a computer of a distance measurement processing device that performs distance measurement processing to perform the distance measurement processing, the distance measurement processing comprising:

[0356] performing a four-phase distance measurement operation process: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating the object with irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase and receiving reflected light reflected by the object, and two detection signals among the eight detection signals can be detected for each of the irradiation light of the first phase to the irradiation light of the fourth phase;

[0357] performing two-phase distance measurement operation processing: performing the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; and

[0358] Condition determination is performed based on the detection signal, and switching is performed between the four-phase distance measurement operation process and the two-phase distance measurement operation process to be used.

[0359] It should be noted that the present embodiment is not limited to those embodiments described above, but the present invention can be modified in various ways without departing from the essence of the present invention. In addition, the effects described in this specification are only for illustration and are not restrictive. Other effects can be produced.

[0360] Those skilled in the art should understand that various modifications, combinations, sub-combinations and alterations may be made according to design requirements and other factors, as long as these modifications, combinations, sub-combinations and alterations are within the scope of the attached claims or their equivalents.

Claims

1. A distance measurement processing device, comprising: a four-phase distance measurement operation unit configured to: when allocating charges to a first tap and a second tap according to a distance to an object, perform an operation for calculating a depth representing the distance to the object by using all eight detection signals, the charges being generated by irradiating irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase onto the object and receiving reflected light reflected by the object, and for each of the irradiation light of the first phase to the irradiation light of the fourth phase, two detection signals among the eight detection signals can be detected; a two-phase distance measurement operation unit configured to perform the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; as well as A condition determination unit is configured to perform condition determination based on the detection signal and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit to be used.

2. The distance measurement processing device according to claim 1, wherein In a first detection period for receiving the reflected light generated by the irradiation light of the first phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a first detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a second detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected, In a second detection period for receiving the reflected light generated by the irradiation light of the second phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a third detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a fourth detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected, In a third detection period of receiving the reflected light generated by the irradiation light of the third phase, the charge is configured to be alternately allocated to the first tap and the second tap a plurality of times, and a fifth detection signal corresponding to the charge allocated to the first tap and accumulated in the first tap and a sixth detection signal corresponding to the charge allocated to the second tap and accumulated in the second tap are detected, and During the fourth detection period of receiving the reflected light generated by the irradiation light of the fourth phase, the charge is configured to be alternately distributed to the first tap and the second tap multiple times, and a seventh detection signal corresponding to the charge distributed to the first tap and accumulated in the first tap, and an eighth detection signal corresponding to the charge distributed to the second tap and accumulated in the second tap are detected.

3. The ranging processing device according to claim 2, wherein The four-phase ranging operation unit is configured to eliminate the influence caused by the characteristic difference between the first tap and the second tap by using the difference between the first detection signal and the second detection signal, the difference between the third detection signal and the fourth detection signal, the difference between the fifth detection signal and the sixth detection signal, and the difference between the seventh detection signal and the eighth detection signal, so as to calculate the depth.

4. The distance measurement processing device according to claim 2, wherein The two-phase ranging operation unit is configured to perform the following operations alternately: After calculating a correction parameter for correcting a characteristic difference between the first tap and the second tap by using the first to fourth detection signals, calculating the depth based on the first to fourth detection signals and the correction parameter, and After calculating a correction parameter for correcting the characteristic difference between the first tap and the second tap by using the fifth to eighth detection signals, the depth is calculated based on the fifth to eighth detection signals and the correction parameter.

5. The ranging processing device according to any one of claims 1 to 4, further comprising: a measurement result storage unit configured to store the depth calculated by one of the four-phase distance measurement operation unit and the two-phase distance measurement operation unit; as well as A result synthesis unit is configured to synthesize the depth of the current frame and the depth of the previous frame stored in the measurement result storage unit and output the synthesized depth.

6. The distance measurement processing device according to claim 5, wherein The four-phase distance measurement operation unit and the two-phase distance measurement operation unit are configured to calculate the confidence with respect to the depth and the depth, The measurement result storage unit is configured to store the confidence level and the depth together, and The result synthesis unit is configured to perform weighted addition corresponding to the confidence level to synthesize the depth of the previous frame and the depth of the current frame together.

7. The distance measurement processing device according to claim 5, wherein The result synthesis unit is configured to synthesize the depth of the previous frame calculated by the four-phase ranging operation unit using all the eight detection signals and the depth of the current frame calculated by the four-phase ranging operation unit using all the eight detection signals.

8. The distance measurement processing device according to claim 5, wherein Among the depth calculated by the two-phase ranging operation unit using the four detection signals based on the illumination light of the first phase and the illumination light of the second phase, and the depth calculated by the two-phase ranging operation unit using the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase, the result synthesis unit is configured to identify one depth as the depth of the previous frame and the other depth as the depth of the current frame, and the result synthesis unit is configured to synthesize the two depths together.

9. The ranging processing device according to any one of claims 1 to 4, wherein The condition determination unit is configured to perform the condition determination on each pixel of the light receiving unit that receives the reflected light, and the condition determination unit is configured to switch between performing the operation of calculating the depth by the four-phase ranging operation unit and performing the operation of calculating the depth by the two-phase ranging operation unit for each of the pixels.

10. The ranging processing device according to any one of claims 1 to 4, wherein The condition determination unit is configured to: The condition determination is performed based on detecting whether the object is moving by comparing the four detection signals based on the illumination light of the first phase and the illumination light of the second phase with the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase, When the object is detected to be in motion, the two-phase ranging operation unit performs the operation of calculating the depth, and When the object is not detected to be in motion, the operation of calculating the depth is performed by the four-phase ranging operation unit.

11. The ranging processing device according to any one of claims 1 to 4, wherein The condition determination unit is configured to: The condition determination is performed based on the brightness calculated from the four detection signals based on the illumination light of the first phase and the illumination light of the second phase and the brightness calculated from the four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase, and Switching is performed between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit according to the brightness.

12. The ranging processing device according to any one of claims 1 to 4, wherein The four-phase distance measurement operation unit and the two-phase distance measurement operation unit are configured to calculate the confidence level relative to the depth and the depth, and The condition determination unit is configured to perform the condition determination based on the confidence calculated in a previous frame, and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit according to the confidence.

13. The ranging processing device according to any one of claims 2 to 4, wherein The first detection period, the second detection period, the third detection period, and the fourth detection period are set at even intervals.

14. A ranging module, comprising: a light emitting unit configured to irradiate the illumination light of the first phase, the illumination light of the second phase, the illumination light of the third phase, and the illumination light of the fourth phase onto the object; a light receiving unit configured to output eight detection signals when allocating charges generated by receiving reflected light reflected by the object to a first tap and a second tap according to a distance to the object, wherein two detection signals among the eight detection signals can be detected for each of the illumination light of the first phase to the illumination light of the fourth phase; a four-phase distance measurement operation unit configured to perform an operation of calculating a depth representing the distance to the object by using all of the eight detection signals; a two-phase distance measurement operation unit configured to perform the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; as well as A condition determination unit is configured to perform condition determination based on the detection signal and is configured to switch between the four-phase distance measurement operation unit and the two-phase distance measurement operation unit.

15. A ranging processing method, comprising: performing a four-phase distance measurement operation process: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating the object with irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase and receiving reflected light reflected by the object, and two detection signals among the eight detection signals can be detected for each of the irradiation light of the first phase to the irradiation light of the fourth phase; performing two-phase distance measurement operation processing: performing the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; as well as Condition determination is performed based on the detection signal, and switching is performed between the four-phase distance measurement operation process and the two-phase distance measurement operation process to be used.

16. A computer program product, wherein a program for causing a computer of a distance measurement processing device to perform distance measurement processing is stored in the computer program product, wherein the distance measurement processing comprises: performing a four-phase distance measurement operation process: when a charge is allocated to a first tap and a second tap according to a distance to an object, an operation for calculating a depth representing the distance to the object is performed by using all eight detection signals, the charge being generated by irradiating the object with irradiation light of a first phase, irradiation light of a second phase, irradiation light of a third phase, and irradiation light of a fourth phase and receiving reflected light reflected by the object, and two detection signals among the eight detection signals can be detected for each of the irradiation light of the first phase to the irradiation light of the fourth phase; performing two-phase distance measurement operation processing: performing the operation of calculating the depth representing the distance to the object by alternately using four detection signals based on the illumination light of the first phase and the illumination light of the second phase and four detection signals based on the illumination light of the third phase and the illumination light of the fourth phase among the eight detection signals; as well as Condition determination is performed based on the detection signal, and switching is performed between the four-phase distance measurement operation process and the two-phase distance measurement operation process to be used.

Images