Signal processing device, distance measuring device, distance measuring method, image sensor

Abstract

This signal processing device comprises: a first depth information calculation unit for calculating first depth information on the basis of a light reception signal for first modulated light that has been modulated using a first frequency; a second depth information calculation unit for calculating second depth information on the basis of a light reception signal for second modulated light that has been modulated using a second frequency lower than the first frequency; a distance information calculation unit for calculating information about the distance to a subject on the basis of the first depth information and second depth information; a folding determination unit for determining whether the phase change of the second modulated light from emission to light reception is less than 2π; and a distance information output unit for outputting the distance information if the phase change of the second modulated light has been determined to be less than 2π.

Description

[Technical Field]

[0001] The present technology relates to a signal processing device that performs signal processing for distance measurement, a distance measuring device, a distance measuring method, and an image sensor. [Background technology]

[0002] Time of Flight (ToF) is a well-known distance measurement technology, and it is desirable for ToF to achieve both high distance measurement range and accuracy. For example, Patent Document 1 below proposes a technique for measuring distance by analog superimposing two modulation frequencies and performing De-Alias ​​processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-036145 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, the method described in Patent Document 1 requires that the two modulation frequencies f1 and f2 satisfy the condition that n1·f1=n2·f2, n1 and n2 are natural numbers, and n1 / n2 is not a natural number.

[0005] However, there are cases where it is not possible to accurately set a modulation frequency that satisfies the above conditions due to factors such as the accuracy of a PLL (Phase Locked Loop) and measures against EMI (Electro Magnetic Interference).

[0006] This technology was developed in consideration of the above circumstances, and aims to perform de-aliasing processing while providing flexibility in setting the modulation frequency. [Means for solving the problem]

[0007] The signal processing device according to the present technology includes a first depth information calculation unit that calculates first depth information based on a received signal of first modulated light modulated by a first frequency, a second depth information calculation unit that calculates second depth information based on a received signal of second modulated light modulated by a second frequency that is less than the first frequency, a distance information calculation unit that calculates distance information to a subject based on the first depth information and the second depth information, a folding determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π, and a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π. When the phase change of the second modulated light is less than 2π, the second depth information and the actual distance to the subject are made substantially equal.

[0008] The distance information output section in the above-described signal processing device may output an invalid value when it is determined that the phase change of the second modulated light is 2π or more. When the phase change of the second modulated light is 2π or more, the second depth information is set to be shorter than the actual distance to the subject.

[0009] The distance information calculation unit in the above-mentioned signal processing device may calculate the distance information by adding a first return distance, which is the distance to the subject when the phase change of the first modulated light is set to 2π, to the first depth information. The first return distance is the distance to the subject at which the phase change from when the first modulated light is irradiated to when it is received is exactly 2π, and is also the distance that can be measured using the first modulated light. The first depth information obtained by irradiating the first modulated light is more accurate than the second depth information obtained by irradiating the second modulated light.

[0010] The above-described signal processing device may be configured to satisfy the following conditional expression (A). (A) fH ≠ n × fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. As a result, when the actual distance to the subject is equal to or greater than one time and less than two times the distance that can be measured with the second modulated light, the calculated distance information is no longer a zero value.

[0011] The above-described signal processing device may be configured to satisfy the following conditional formula (B): (B) fH=(n+0.5)fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. As a result, when the actual distance to the subject is the same as the distance measurable with the second modulated light, the distance information calculated is a value that is approximately half the distance measurable with the first modulated light.

[0012] The aliasing determination unit in the above-described signal processing device may determine whether or not a phase change of the second modulated light is less than 2π by comparing the second depth information with the distance information. When the actual distance to the subject is equal to or greater than one time and less than two times the distance that can be measured with the second modulated light, the calculated distance information deviates from the calculated second depth information.

[0013] The aliasing determination unit in the signal processing device may determine that the phase change of the second modulated light is 2π or more when the difference between the second depth information and the distance information is equal to or greater than a threshold value. As a result, when the difference between the second depth information and the distance information is relatively large, it is determined that the phase change of the second modulated light is 2π or more.

[0014] In the above-described signal processing device, a distance image may be generated by calculating the distance information for each of two-dimensionally arranged pixels. As a result, when the phase change of the second modulated light in the distance information for each pixel in the distance image is less than 2π, the second depth information and the actual distance to the subject are made substantially equal.

[0015] a light receiving unit that receives reflected light of the first modulated light and the second modulated light reflected by a subject and outputs a light receiving signal corresponding to the light intensity of the reflected light; and a signal processing unit that performs signal processing on the light receiving signal output from the light receiving unit, wherein the signal processing unit has a first depth information calculation unit that calculates first depth information based on the light receiving signal of the first modulated light, a second depth information calculation unit that calculates second depth information based on the light receiving signal of the second modulated light, a distance information calculation unit that calculates distance information to the subject based on the first depth information and the second depth information, a folding determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π, and a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π.

[0016] The ranging method performed by the ranging device of the present technology includes a process of irradiating first modulated light modulated by a first frequency, a process of calculating first depth information based on a received light signal of the first modulated light, a process of irradiating second modulated light modulated by a second frequency less than the first frequency, a process of calculating second depth information based on a received light signal of the second modulated light, a process of calculating distance information to a subject based on the first depth information and the second depth information, a process of determining whether a phase change of the second modulated light from irradiation to reception is less than 2π, and a process of outputting the distance information when it is determined that the phase change of the second modulated light is less than 2π.

[0017] The image sensor according to the present technology is capable of receiving first modulated light modulated by a first frequency and second modulated light modulated by a second frequency that is lower than the first frequency, and includes a pixel array unit having a first tap and a second tap that detect electric charges generated by photoelectric conversion in a photoelectric conversion unit, and the pixel array unit outputs detection signals of the first tap and the second tap based on reception of the first modulated light and detection signals of the first tap and the second tap based on reception of the second modulated light as light reception signals for determining whether a phase change from irradiation of the second modulated light to reception of the second modulated light is 2π or more. Such a distance measuring device, distance measuring method, and image sensor can also provide the same effects as the signal processing device according to the present technology described above. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a distance measuring device according to the present technology. [Figure 2] 1 is a diagram illustrating an example of the configuration of each pixel included in a light receiving unit according to the present technology; [Figure 3] 10 is a timing chart of signals relating to exposure control and light receiving control according to the present technology. [Figure 4] 1 is a diagram illustrating an example of the configuration of each unit included in a signal processing unit according to the present technology. [Figure 5] 10A and 10B are diagrams illustrating an example of ON / OFF control of a first control signal and a second control signal according to the present technology and the amount of charge stored in a first FD and a second FD. [Figure 6] 10 is a diagram showing the relationship between first depth information, second depth information, and distance information calculated by a distance information calculation unit according to the present technology. FIG. [Figure 7] 10 is a flowchart of a process executed by a control unit and a signal processing unit according to the present technology. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order. <1. System configuration> <2. Calculating distance information> <3. Judgment process> <4. Distance information output processing> <5. Processing flow> <6. Summary> <7. This Technology>

[0020] <1. System configuration> The configuration of a distance measuring device 1 of the present technology will be described with reference to FIG. The distance measuring device 1 includes a light emitting unit 2, a light receiving unit 3, a control unit 4, and a signal processing unit 5.

[0021] The light emitting unit 2 is configured to include a light source such as an infrared (IR) light emitting diode (LED), and emits light based on a control signal input from the control unit 4. The light irradiating unit 2 is capable of irradiating modulated light whose light intensity is modulated based on a specific frequency. The light irradiating unit 2 is also capable of irradiating multiple types of modulated light whose specific frequencies are different. In the example shown below, it is capable of irradiating two types of modulated light modulated based on two specific frequencies. In the following description, the specific frequency will be referred to as a modulation frequency.

[0022] In each of the examples shown below, the light irradiator 2 is capable of irradiating modulated light based on two types of modulation frequencies. The two types of modulation frequencies are determined such that the first frequency fH is a relatively high modulation frequency and the second frequency fL is a relatively low modulation frequency. Specifically, the first frequency fH is set so as not to be an integer multiple of the second frequency fL. Specifically, the frequencies are set so as to satisfy the following formula (1):

[0023] fH≠n×fL...Equation (1)

[0024] In this example, the first frequency fH and the second frequency fL are set to satisfy the following equation (2).

[0025] fH=(n+0.5)fL...Equation (2)

[0026] In the following description, an example will be described in which the first frequency fH is set to 50 MHz and the second frequency fL is set to 20 MHz. Furthermore, modulated light modulated based on the first frequency fH is referred to as first modulated light MLH, and modulated light modulated based on the second frequency fL is referred to as second modulated light MLL. The light emitting unit 2 may be capable of emitting modulated light based on three or more types of modulation frequencies.

[0027] The light receiving unit 3 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type, and is configured as a sensor capable of distance measurement using an iToF (Indirect ToF) method. The light receiving unit 3 includes a pixel array unit in which pixels 31 are arranged two-dimensionally. The light receiving unit 3 controls light reception based on a control signal input from the control unit 4.

[0028] An example of the configuration of the pixel 31 of the light receiving section 3 is shown in FIG. The pixel 31 is configured to include a PD (Photodiode) 32 serving as a light receiving element, and a first tap 33 and a second tap 34 for detecting charges generated by photoelectric conversion in the PD 32. The PD 32 in this example is assumed to have sensitivity to light in the infrared region, for example.

[0029] The first tap 33 is configured to include a first FD (Floating Diffusion) 35, a first transfer transistor 36 for transferring charges from the PD 32 to the first FD 35, and a selection transistor and a reset transistor (not shown).

[0030] The second tap 34 is configured to include a second FD 37, a second transfer transistor 38 for transferring charges from the PD 32 to the second FD 37, and a selection transistor and a reset transistor (not shown).

[0031] When one of the first transfer transistor 36 and the second transfer transistor 38 is controlled to an ON state, the other is controlled to an OFF state. That is, the first transfer transistor 36 and the second transfer transistor 38 are controlled so that both are not turned ON at the same time.

[0032] The signal for controlling the ON / OFF state of the first transfer transistor 36 is synchronized with the light emission cycle of the light source provided in the light irradiation unit 2. In addition, the control signals applied to the first transfer transistor 36 and the second transfer transistor 38 have a phase difference of 180 degrees.

[0033] The charge transferred to the first FD 35 is output to the signal processing unit 5 as a first signal S1, which is a detection signal output from the first tap 33 in response to the readout signal. The charges transferred to the second FD 37 are output to the signal processing unit 5 as a second signal S2, which is a detection signal output from the second tap 34 in response to the readout signal.

[0034] In addition to the components shown in FIG. 2, the pixel 31 also includes a readout transistor and a reset transistor (not shown).

[0035] Returning to the explanation of Figure 1. As described above, the control unit 4 supplies the light emitting unit 2 with a drive signal (20 MHz or 50 MHz) according to the modulation frequency for emitting modulated light. This enables the light irradiating unit 2 to irradiate light whose intensity is modulated based on the supplied rectangular wave drive signal. Note that the light irradiating unit 2 may irradiate light whose intensity is modulated based on a sine wave drive signal.

[0036] Furthermore, the control unit 4 controls the ON / OFF of the first transfer transistor 36 and the second transfer transistor 38 at a timing synchronized with the modulation frequency.

[0037] During one exposure time, the control unit 4 performs ON / OFF control multiple times (for example, hundreds to tens of thousands of times). This will be explained in detail using Fig. 3. Fig. 3 is a diagram showing the relationship between an exposure control signal Se used for exposure control, an irradiation light signal SL provided to the light irradiation unit 2 to irradiate light whose intensity has been modulated at a predetermined modulation frequency, a first control signal St1 provided to the first transfer transistor 36, and a second control signal St2 provided to the second transfer transistor 38. The first transfer transistor 36 is ON / OFF controlled based on a first control signal St1, and the second transfer transistor 38 is ON / OFF controlled based on a second control signal St2.

[0038] As shown in the figure, light is irradiated multiple times over an irradiation period TL during the exposure time Te. The multiple light irradiations are performed by alternating between the irradiation period TL and a non-irradiation period. The non-irradiation period has the same length as the irradiation period TL. In other words, the duty ratio is set to 50%.

[0039] The control unit 4 controls the switching of the first transfer transistor 36 so that charge is accumulated in the first FD 35 over a first period T1 synchronized with the irradiation light signal SL. The control unit 4 also controls the switching of the second transfer transistor 38 so that charge is accumulated in the second FD 37 over a second period T2 synchronized with the irradiation light signal SL. The first period T1 and the second period T2 have the same time length as the irradiation period TL.

[0040] As a result, electric charges are intermittently accumulated in the first FD 35 and the second FD 37 hundreds, thousands, or tens of thousands of times. If the amount of light received for one cycle of irradiated light is very small, it may be impossible to obtain meaningful data. Therefore, by receiving light for hundreds to tens of thousands of cycles and accumulating electric charge, a sufficient amount of light can be received, making it possible to obtain meaningful information.

[0041] Note that the switches may be switched once during one exposure time so that the first FD 35 and the second FD 37 each accumulate charge once. That is, the first transfer transistor 36 may be controlled to be in the ON state so that the first FD 35 accumulates charge during the first half of the exposure time Te, and at the start of the second half of the exposure time Te, switching control may be performed to switch the first transfer transistor 36 to the OFF state and the second transfer transistor 38 to the ON state, so that charge is accumulated in the second FD 37 during the second half of the exposure time.

[0042] Returning to the explanation of Figure 1. The signal processing unit 5 performs various processes based on the first signal S1 output from the first FD 35 of the light receiving unit 3 and the second signal S2 output from the second FD 37, and outputs distance information. To this end, the signal processing unit 5 includes a first depth information calculation unit 51, a second depth information calculation unit 52, a distance information calculation unit 53, a folding determination unit 54, and a distance information output unit 55.

[0043] An example of functional blocks of the signal processing unit 5 will be described with reference to FIG. The light receiving unit 3 outputs a first signal S1 which is the output of the first FD 35 and a second signal S2 which is the output of the second FD 37. In the following description, of the first signal S1 and the second signal S2 output from the light receiving unit 3, the signals output based on the light reception in response to the irradiation of the first modulated light MLH will be referred to as a first signal S1H and a second signal S2H, and the signals output based on the light reception in response to the irradiation of the second modulated light MLL will be referred to as a first signal S1L and a second signal S2L.

[0044] The first signal S1H and the second signal S2H are input to the first depth information calculation unit 51. An example of the first signal S1H and the second signal S2H is shown in FIG. As shown in the figure, the first modulated light MLH emitted based on the irradiation light signal SL is received as reflected light RL reflected by the subject by the light receiving unit 3. The electric charges generated by photoelectric conversion are accumulated in the first FD 35 and the second FD 37 in accordance with the first control signal St1 and the second control signal St2.

[0045] 5, the first region AR1 indicates the amount of charge accumulated in the first FD 35 while the first control signal St1 is in the ON state, and the second region AR2 indicates the amount of charge accumulated in the second FD 37 while the second control signal St2 is in the ON state.

[0046] The first depth information calculation unit 51 calculates depth information of the subject based on the first area AR1 and the second area AR2. The 50 MHz first modulated light MLH reflected by the subject is received by the light receiving unit 3 with a phase θ delay. The first depth information calculation unit 51 calculates the phase θ and calculates depth information according to the phase θ.

[0047] Specifically, by repeating ON / OFF control of the first control signal St1 and the second control signal St2 several hundred to several tens of thousands of times, minute charges are accumulated several hundred to several tens of thousands of times in the first FD 35 and the second FD 37. The first depth information calculation unit 51 calculates the phase θ based on the charges accumulated in the first FD 35 and the second FD 37 in this manner.

[0048] However, the phase θ calculated by the first depth information calculation unit 51 is calculated between 0 and 2π. Therefore, even if the delay of the reflected light relative to the first modulated light MLH is set to 2π+θ or 4π+θ, the phase delay calculated by the first depth information calculation unit 51 is calculated as θ.

[0049] The depth information calculated by the first depth information calculation unit 51 is depth information calculated when the first modulated light MLH intensity-modulated at 50 MHz is irradiated, and this is referred to as first depth information Dp1.

[0050] The first depth information Dp1 calculated by the first depth information calculation unit 51 has a narrow distance measurement range but high distance measurement accuracy. To compensate for the narrow distance measurement range of the first depth information calculation unit 51, a second depth information calculation unit 52 is provided.

[0051] The second depth information calculation unit 52 receives reflected light of the second modulated light MLL intensity-modulated at 20 MHz, and calculates second depth information Dp2 based on the first signal S1L output from the first FD 35 and the second signal S2L output from the second FD 37. Specifically, similar to the first depth information Dp1, the depth information is calculated by calculating the phase delay θ of the reflected light relative to the second modulated light MLL.

[0052] The second depth information Dp2 is calculated using the second modulated light MLL intensity-modulated at 20 MHz, which is a lower frequency than 50 MHz, and therefore has a wider ranging range than the first depth information Dp1.

[0053] The first depth information Dp1 output from the first depth information calculation unit 51 and the second depth information Dp2 output from the second depth information calculation unit 52 are output to the distance information calculation unit 53.

[0054] The distance information calculation unit 53 calculates the distance information Da after De-Alias ​​processing using the first depth information Dp1 and the second depth information Dp2.

[0055] Here, the distance to the object at which the phase θ becomes 2π when the first modulated light MLH is irradiated is defined as the first measurable distance MD1. Also, the distance to the object at which the phase θ becomes 2π when the second modulated light MLL is irradiated is defined as the second measurable distance MD2. Also, the first measurable distance MD1 is defined as the first return distance, and the second measurable distance MD2 is defined as the second return distance.

[0056] Distance information Da calculated by distance information calculation unit 53 is a value equal to or greater than 0 and less than second measurable distance MD2. Therefore, if the actual distance Dr to the subject is equal to or greater than the second measurable distance MD2, inappropriate distance information Da is calculated.

[0057] The aliasing determination unit 54 determines whether the distance information Da calculated by the distance information calculation unit 53 is appropriate. As will be described in detail later, the distance information Da is calculated as an aliased value each time the actual distance Dr to the subject reaches an integer multiple of the second measurable distance MD2. The aliasing determination unit 54 determines whether the distance information Da is before aliasing or after aliasing. This determination process is synonymous with the process of determining whether the actual distance Dr to the subject is less than the second measurable distance MD2 or equal to or greater than the second measurable distance MD2.

[0058] The return determination unit 54 outputs to the distance information output unit 55 the determination result as to whether the distance is equal to or greater than the second measurable distance MD2 and the distance information Da.

[0059] Distance information output unit 55 outputs distance information Da based on the determination result output from folding determination unit 54. When a determination result indicating that the actual distance Dr to the subject is less than second measurable distance MD2 is input, distance information output unit 55 outputs distance information Da calculated by distance information calculation unit 53, and when a determination result indicating that the actual distance Dr to the subject is equal to or greater than second measurable distance MD2 is input, distance information output unit 55 outputs an invalid value without outputting distance information Da calculated by distance information calculation unit 53. The invalid value may be a value greater than the second measurable distance MD2 that is greater than the possible values ​​of distance information Da, or may be a negative value that distance information Da cannot take, or may be flag information other than a numerical value.

[0060] Although not shown in each figure, the distance measuring device 1 may also be equipped with optical components such as various lenses and aperture mechanisms for appropriately focusing the light reflected from the subject onto the light receiving unit 3, as well as drivers for driving these optical components.

[0061] <2. Calculating distance information> FIG. 6 shows the relationship between the first depth information Dp1, the second depth information Dp2, and the distance information Da calculated by the distance information calculation unit 53.

[0062] As shown in FIG. 6, if the actual distance Dr from the subject is less than the first measurable distance MD1, appropriate depth information corresponding to the actual distance Dr from the subject is output for both the first depth information Dp1 and the second depth information Dp2.

[0063] If the actual distance Dr from the subject is greater than or equal to the first measurable distance MD1 and less than the second measurable distance MD2, the second depth information Dp2 outputs appropriate depth information corresponding to the actual distance Dr from the subject. On the other hand, for the first depth information Dp1, depth information less than the first measurable distance MD1 is output as depth information different from the actual distance Dr from the subject.

[0064] Furthermore, if the actual distance Dr from the subject is greater than or equal to the second measurable distance MD2, depth information different from the actual distance Dr from the subject is output for both the first depth information Dp1 and the second depth information Dp2. Specifically, depth information less than the first measurable distance MD1 is output as the first depth information Dp1, and depth information less than the second measurable distance MD2 is output as the second depth information Dp2.

[0065] The distance information calculation unit 53 calculates distance information Da from the first depth information Dp1 and the second depth information Dp2. Specifically, first, a natural number N that satisfies the following formula (3) is obtained.

[0066] MD1 × N ≤ Dp2 < MD1 × (N + 1) ··· Formula (3)

[0067] Next, the distance information Da is calculated using the following formula (4).

[0068] Da = MD1 × N + Dp1 ··· Formula (4)

[0069] In this way, by using the first depth information Dp1 instead of the second depth information Dp2 in the formula (4) for finally calculating the distance information Da, the ranging accuracy can be ensured.

[0070] Here, the distance information Da when the actual distance Dr to the subject is the second measurable distance MD2 will be described. When the actual distance Dr to the subject is the second measurable distance MD2, if the natural number N that satisfies the above formula (3) is calculated, the natural number N=0 because Dp2=0. Therefore, from equation (4), Da=Dp.

[0071] In this way, when the actual distance Dr to the subject exceeds the second measurable distance MD2, the calculated distance information Da does not become 0. This is based on the fact that when the actual distance Dr to the subject is set to MD2, the second depth information Dp2 (= 0) does not match the first depth information Dp1 (≠ 0).

[0072] <3. Judgment process> The determination process executed by the return determination unit 54 will be described.

[0073] The aliasing determination unit 54 determines whether the distance information Da calculated by the distance information calculation unit 53 is information before aliasing or information after aliasing. This determination process is synonymous with the process of determining whether the phase change of the second modulated light MLL irradiated from the light irradiating unit 2 until it is reflected by the subject and received by the light receiving unit 3 is less than 2π.

[0074] Specifically, the difference Dt between the distance information Da and the second depth information Dp2 is calculated, and it is determined whether the difference Dt is greater than a threshold value Th, thereby determining whether the phase change is less than 2π.

[0075] The difference Dt can be calculated, for example, by the following equation (5).

[0076] Dt = abs(Da - Dp2) Equation (5)

[0077] The abs function is a function that obtains the absolute value.

[0078] As shown in FIG. 6, if the phase change is less than 2π, that is, if the actual distance Dr to the subject is less than the second measurable distance MD2, the difference Dt between the distance information Da and the second depth information Dp2 is approximately zero. On the other hand, if the phase change is 2π or more, that is, if the actual distance Dr to the subject is equal to or greater than the second measurable distance MD2, the difference Dt is set to approximately half the first measurable distance MD1.

[0079] The threshold value Th, which is the comparison target for the difference Dt, is set using, for example, the following equation (6), taking into consideration noise and the like that occurs in the actual environment.

[0080] Th=(MD1) / 3...Equation (6)

[0081] If the threshold value Th is set to a small value close to zero, there is a risk that a phase change of less than 2π will be erroneously determined to be 2π or more, and if it is set to a large value close to half the first measurable distance MD1, there is a risk that a phase change of 2π or more will be erroneously determined to be less than 2π.

[0082] It is desirable to appropriately determine the threshold Th depending on the level of noise in the measurement environment. When the noise is small, it is acceptable to set the threshold Th to a small value close to zero or to a large value close to half the first measurable distance MD1.

[0083] Furthermore, the threshold value Th may be configured to change depending on the intensity of the received light signal so as to prevent erroneous determination due to noise.

[0084] The result of the determination as to whether the phase change is less than 2π (the result of the determination as to whether the actual distance Dr to the subject is less than the second measurable distance MD2) is output from the return determination unit 54 to the distance information output unit 55.

[0085] If the actual distance Dr to the subject is two or more times but less than three times the second measurable distance MD2, the difference Dt is set to approximately zero. However, in this case, the actual distance Dr to the subject is more than twice the second measurable distance MD2, which is the distance to be measured, so it is considered that the reflected light is significantly weaker. Therefore, when the amount of received light is lower than a predetermined value, the calculated first depth information Dp1 and second depth information Dp2 can be ignored to prevent inappropriate distance information from being output in the subsequent distance information output process.

[0086] In other words, by adjusting the light emission intensity so that the amount of light received reflected from a subject at a distance more than twice the distance to be measured is lower than a predetermined value, it is possible to eliminate the possibility of erroneously determining that the distance information Da is before folding, i.e., that the phase change of the second modulated light MLL is less than 2π, when the actual distance Dr to the subject is more than twice but less than three times the second measurable distance MD2.

[0087] <4. Distance information output processing> Distance information output unit 55 receives the determination result and distance information Da from folding determination unit 54, and outputs the distance information Da as necessary. Specifically, when a determination result indicating that the phase change is less than 2π is received, distance information Da is output because it indicates the actual distance Dr to the subject.

[0088] On the other hand, if a determination result indicating that the phase change is 2π or more is received, the distance information Da is information that deviates from the actual distance Dr to the subject, and therefore the distance information Da received from the folding determination unit 54 is not output as is. Instead, information indicating an invalid value may be stored in the distance information Da and output as the distance information Da.

[0089] As described above, the information indicating an invalid value may be a value greater than the second measurable distance MD2, which is a value greater than the value that distance information Da can take, or a negative value that distance information Da cannot take, or flag information other than a numerical value.

[0090] <5. Processing flow> The flow of processing executed by the control unit 4 and the signal processing unit 5 as processing units of the distance measuring device 1 will be described with reference to FIG.

[0091] First, in step S101, the control unit 4 of the distance measuring device 1 cooperates with the light emitting unit 2 and the light receiving unit 3 to perform light emitting processing and exposure processing.

[0092] As a result, a first signal S1 and a second signal S2 for each pixel 31 are output to the subsequent signal processing unit 5. Note that the light emission process and exposure process in step S101 involve irradiation and exposure of the first modulated light MLH, and irradiation and exposure of the second modulated light MLL. Therefore, in step S101, the first signal S1 and the second signal S2 based on the reception of the first modulated light MLH, and the first signal S1 and the second signal S2 based on the reception of the second modulated light MLL are output.

[0093] In step S102, the signal processing unit 5 resets the counter cnt to a value of 0. The counter cnt is incremented each time processing for a pixel is completed, and is used to determine whether the processing required to generate one distance image has been completed.

[0094] Next, the first depth information calculation unit 51 and the second depth information calculation unit 52 of the signal processing unit 5 perform a process of calculating the first depth information Dp1 and the second depth information Dp2 in step S103.

[0095] In step S104, the distance information calculation unit 53 of the signal processing unit 5 calculates distance information Da using the first depth information Dp1 and the second depth information Dp2. The calculated distance information Da is a value equal to or greater than zero and less than the second measurable distance MD2.

[0096] In step S105, the aliasing determination unit 54 of the signal processing unit 5 performs aliasing determination. As described above, the aliasing determination determines whether the phase change of the second modulated light MLL irradiated from the light irradiating unit 2 until it is reflected by the subject and received by the light receiving unit 3 is less than 2π.

[0097] If it is determined that there is no folding, i.e., if it is determined that the phase change is less than 2π, the distance information output unit 55 of the signal processing unit 5 outputs the distance information Da calculated by the distance information calculation unit 53 in step S106.

[0098] On the other hand, if it is determined that the signal is folded back, that is, if it is determined that the phase change is 2π or more, the distance information output unit 55 of the signal processing unit 5 outputs an invalid value in step S107.

[0099] After executing the process of step S106 or step S107, the signal processing unit 5 determines in step S108 whether the counter cnt is less than the value obtained by multiplying the number of horizontal pixels Pw by the number of vertical pixels Ph. The number of horizontal pixels Pw indicates the number of pixels in the horizontal direction in the captured image, and the number of vertical pixels Ph indicates the number of pixels in the vertical direction in the captured image.

[0100] If the counter cnt is less than the product of the number of horizontal pixels Pw and the number of vertical pixels Ph, it means that processing has not been completed for the number of pixels included in the distance image. In this case, the signal processing unit 5 increments the counter cnt by 1 in step S109 and returns to the processing of step S103.

[0101] On the other hand, if the counter cnt is equal to or greater than the product of the number of horizontal pixels Pw and the number of vertical pixels Ph, it means that processing has been completed for the number of pixels required to generate one distance image. In this case, the signal processing unit 5 returns to step S101 and performs light emission and exposure processing to obtain the next frame of distance image.

[0102] The distance measuring device 1 can generate multiple distance images by executing the series of processes shown in FIG.

[0103] <6. Summary> As described above, the distance measuring device 1 as a signal processing device includes a first depth information calculation unit 51 that calculates first depth information Dp1 based on the received light signal of the first modulated light MLH modulated by a first frequency fH (e.g., 50 MHz), a second depth information calculation unit 52 that calculates second depth information Dp2 based on the received light signal of the second modulated light MLL modulated by a second frequency fL (e.g., 20 MHz) that is less than the first frequency fH, a distance information calculation unit 53 that calculates distance information Da to the subject based on the first depth information Dp1 and the second depth information Dp2, a return judgment unit 54 that judges whether the phase change of the second modulated light MLL from irradiation to reception is less than 2π, and a distance information output unit 55 that outputs distance information Da when it is judged that the phase change of the second modulated light MLL is less than 2π. When the phase change of the second modulated light MLL is less than 2π, the second depth information Dp2 and the actual distance to the subject (actual distance Dr to the subject) are made substantially equal. Therefore, appropriate distance information Da according to the actual distance to the subject can be output.

[0104] As explained using Figure 7 etc., the distance information output unit 55 in the distance measuring device 1 may also output an invalid value as the distance information Da when it is determined that the phase change of the second modulated light MLL is 2π or more. When the phase change of the second modulated light MLL is 2π or more, the second depth information Dp2 is set to be shorter than the actual distance to the subject (actual distance Dr to the subject). Therefore, it is possible to prevent distance information Da that deviates from the actual distance to the subject from being output.

[0105] As explained using equation (4), etc., the distance information calculation unit 53 in the distance measuring device 1 may calculate the distance information Da by adding the first return distance and the first depth information Dp1 when the second depth information Dp2 is greater than or equal to the first return distance (first measurable distance MD1), which is the distance to the subject when the phase change of the first modulated light MLH is set to 2π. The first return distance is the distance to the subject at which the phase change from when the first modulated light MLH is irradiated to when it is received is exactly 2π, and is also the distance that can be measured by the first modulated light MLH. The first depth information Dp1 obtained by irradiating the first modulated light MLH is more accurate than the second depth information Dp2 obtained by irradiating the second modulated light MLL. This allows the distance information Da calculated by the distance information calculation unit 53 to be highly accurate.

[0106] As explained using FIG. 6 and other figures, the distance measuring device 1 may be configured to satisfy the following conditional expression (A). (A) fH ≠ n × fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. In the above example, the first frequency fH is set to 50 MHz and the second frequency fL is set to 20 MHz, so that n=2.5 in conditional formula (A) is obtained, which does not satisfy the condition that n is a natural number. Therefore, the above example satisfies conditional formula (A). As a result, the distance information Da calculated when the actual distance to the subject (actual distance Dr to the subject) is equal to or greater than one time and less than two times the second measurable distance MD2 in the second modulated light MLL will not be a zero value. Therefore, it is possible to determine whether or not there is a deviation from the actual distance to the subject.

[0107] As explained with reference to FIG. 6 and the like, the distance measuring device 1 may be configured to satisfy the following conditional formula (B). (B) fH=(n+0.5)fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. In the above example, the first frequency fH is set to 50 MHz and the second frequency fL is set to 20 MHz, so that n=2 in conditional formula (B) satisfies the condition that n is a natural number. Therefore, the above example satisfies conditional formula (B). As a result, the distance information Da calculated when the actual distance to the subject (actual distance Dr to the subject) is the same as the second measurable distance MD2 in the second modulated light MLL is a value approximately half the value of the first measurable distance MD1 in the first modulated light MLH. Therefore, the second measurable distance MD2 in the second modulated light MLL is exactly the center value of the distance that is a constant multiple of the first measurable distance MD1 in the first modulated light MLH, which not only makes it possible to determine whether or not it deviates from the actual distance to the subject, but also increases resistance to erroneous determinations due to noise, calibration deviation, color mixing, etc.

[0108] As explained using Figure 6 etc., the aliasing determination unit 54 in the distance measuring device 1 may determine whether the phase change of the second modulated light MLL is less than 2π by comparing the second depth information Dp2 with the distance information Da. When the actual distance to the subject (actual distance Dr to the subject) is greater than or equal to one and less than two times the second measurable distance MD2 in the second modulated light MLL, the calculated distance information Da will deviate from the calculated second depth information Dp2. Therefore, by calculating the difference Dt between the second depth information Dp2 and the distance information Da, it is possible to determine whether the phase change of the second modulated light MLL is less than 2π. This makes it possible to output appropriate distance information according to the actual distance to the subject.

[0109] As explained using Figure 6, etc., the aliasing determination unit 54 in the distance measuring device 1 may determine that the phase change of the second modulated light MLL is 2π or more when the difference Dt between the second depth information Dp2 and the distance information Da is greater than or equal to the threshold Th. As a result, when the difference Dt between the second depth information Dp2 and the distance information Da is relatively large, it is determined that the phase change of the second modulated light MLL is 2π or more. Therefore, it is possible to prevent erroneous determination due to noise or the like.

[0110] As described with reference to FIG. 7 and other figures, the distance measuring device 1 may generate a distance image by calculating distance information Da for each of the two-dimensionally arranged pixels 31. As a result, in the distance information Da for each pixel 31 in the distance image, if the phase change of the second modulated light MLL is less than 2π, the second depth information Dp2 and the actual distance to the subject (actual distance Dr to the subject) are made approximately equal. Therefore, it is possible to output a distance image based on appropriate distance information Da according to the actual distance to the subject.

[0111] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0112] <7. This Technology> (1) a first depth information calculation unit that calculates first depth information based on a received light signal of the first modulated light modulated by the first frequency; a second depth information calculation unit that calculates second depth information based on a light reception signal of second modulated light modulated by a second frequency that is lower than the first frequency; a distance information calculation unit that calculates distance information to a subject based on the first depth information and the second depth information; a return determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π; a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π. Signal processing device. (2) The distance information output unit outputs an invalid value when it is determined that the phase change of the second modulated light is 2π or more. The signal processing device according to (1) above. (3) When the second depth information is equal to or greater than a first aliasing distance, which is the distance to the subject when the phase change of the first modulated light is set to 2π, the distance information calculation unit calculates the distance information by adding the first aliasing distance and the first depth information. A signal processing device according to any one of (1) and (2) above. (4) The following condition (A) is satisfied: The signal processing device according to (3) above. (A) fH ≠ n × fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. (5) The following condition (B) is satisfied: The signal processing device according to (4) above. (B) fH=(n+0.5)fL however, fH: First frequency fL: 2nd frequency n: natural number Let's say. (6) The aliasing determination unit determines whether or not a phase change of the second modulated light is less than 2π by comparing the second depth information with the distance information. A signal processing device according to any one of (4) to (5) above. (7) The aliasing determination unit determines that the phase change of the second modulated light is 2π or more when the difference between the second depth information and the distance information is equal to or greater than a threshold. The signal processing device according to (6) above. (8) A distance image is generated by calculating the distance information for each of the two-dimensionally arranged pixels. A signal processing device according to any one of (1) to (7) above. (9) a light irradiation unit capable of irradiating first modulated light modulated by a first frequency and second modulated light modulated by a second frequency lower than the first frequency; a light receiving unit that receives reflected light of the first modulated light and the second modulated light reflected by an object and outputs a light receiving signal according to the light intensity of the reflected light; a signal processing unit that performs signal processing on the received light signal output from the light receiving unit, The signal processing unit a first depth information calculation unit that calculates first depth information based on a light reception signal of the first modulated light; a second depth information calculation unit that calculates second depth information based on a light reception signal of the second modulated light; a distance information calculation unit that calculates distance information to a subject based on the first depth information and the second depth information; a return determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π; a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π. Ranging device. (10) irradiation of first modulated light modulated by a first frequency; A process of calculating first depth information based on a light reception signal of the first modulated light; an irradiation process of second modulated light modulated by a second frequency that is lower than the first frequency; A process of calculating second depth information based on a light reception signal of the second modulated light; A process of calculating distance information to a subject based on the first depth information and the second depth information; a process of determining whether a phase change of the second modulated light from irradiation to reception is less than 2π; and outputting the distance information when it is determined that the phase change of the second modulated light is less than 2π. Distance method (11) a pixel array unit that is capable of receiving first modulated light modulated by a first frequency and second modulated light modulated by a second frequency that is lower than the first frequency, and that has a first tap and a second tap that detect electric charges generated by photoelectric conversion in the photoelectric conversion unit; As light reception signals for determining whether a phase change from irradiation to reception of the second modulated light is 2π or more, the pixel array unit outputs detection signals of the first tap and the second tap based on reception of the first modulated light and detection signals of the first tap and the second tap based on reception of the second modulated light. Image sensor. [Explanation of symbols]

[0113] Da distance information Dp1 1st depth information Dp2 2nd depth information Dt difference fH First frequency fL 2nd frequency MD1 1st measurable distance MD2 2nd measurable distance MLH 1st modulated light MLL 2nd modulated light Th threshold 1 Ranging device 2 Light irradiation unit 3 Light receiving section 5. Signal Processing Section 31 pixels 33 First Tap 34 Second Tap 51 1st depth information calculation section 52 Second depth information calculation section 53 Distance information calculation unit 54 Folding judgment unit 55 Distance information output unit

Claims

1. a first depth information calculation unit that calculates first depth information based on a light reception signal of the first modulated light modulated by the first frequency; a second depth information calculation unit that calculates second depth information based on a light reception signal of second modulated light modulated by a second frequency that is lower than the first frequency; a distance information calculation unit that calculates distance information to a subject based on the first depth information and the second depth information; a folding determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π; a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π, the aliasing determination unit determines that a phase change of the second modulated light is 2π or more when a difference between the second depth information and the distance information is equal to or greater than a threshold; The threshold value is variable depending on the intensity of the received light signal. Signal processing device.

2. The distance information output unit outputs an invalid value when it is determined that the phase change of the second modulated light is 2π or more. The signal processing device according to claim 1 .

3. When the second depth information is equal to or greater than a first aliasing distance, which is the distance to the subject when the phase change of the first modulated light is set to 2π, the distance information calculation unit calculates the distance information by adding the first aliasing distance and the first depth information. The signal processing device according to claim 1 .

4. The following conditional expression (A) is satisfied: The signal processing device according to claim 3 . (A) fH ≠ n × fL however, fH: First frequency fL: second frequency n: natural number Let's say.

5. The following conditional expression (B) is satisfied: The signal processing device according to claim 4 . (B) fH=(n+0.5)fL however, fH: First frequency fL: second frequency n: natural number Let's say.

6. A distance image is generated by calculating the distance information for each of the two-dimensionally arranged pixels. The signal processing device according to claim 1 .

7. a light irradiation unit capable of irradiating first modulated light modulated by a first frequency and second modulated light modulated by a second frequency lower than the first frequency; a light receiving unit that receives reflected light of the first modulated light and the second modulated light reflected by an object and outputs a light receiving signal corresponding to the light intensity of the reflected light; a signal processing unit that performs signal processing on the received light signal output from the light receiving unit, The signal processing unit a first depth information calculation unit that calculates first depth information based on a light reception signal of the first modulated light; a second depth information calculation unit that calculates second depth information based on a light reception signal of the second modulated light; a distance information calculation unit that calculates distance information to a subject based on the first depth information and the second depth information; a folding determination unit that determines whether a phase change of the second modulated light from irradiation to reception is less than 2π; a distance information output unit that outputs the distance information when it is determined that the phase change of the second modulated light is less than 2π, the aliasing determination unit determines that a phase change of the second modulated light is 2π or more when a difference between the second depth information and the distance information is equal to or greater than a threshold; The threshold value is variable depending on the intensity of the received light signal. Ranging device.

8. irradiation of first modulated light modulated by a first frequency; a process of calculating first depth information based on a light reception signal of the first modulated light; an irradiation process of second modulated light modulated by a second frequency that is lower than the first frequency; calculating second depth information based on a light reception signal of the second modulated light; A process of calculating distance information to a subject based on the first depth information and the second depth information; a process of determining whether a phase change of the second modulated light from irradiation to reception is less than 2π; and outputting the distance information when it is determined that the phase change of the second modulated light is less than 2π; In the process of determining whether the phase change is less than 2π, if the difference between the second depth information and the distance information is equal to or greater than a threshold, it is determined that the phase change of the second modulated light is equal to or greater than 2π; The threshold value is variable depending on the intensity of the received light signal. Distance measurement method.

9. a pixel array unit that is capable of receiving first modulated light modulated by a first frequency and second modulated light modulated by a second frequency that is lower than the first frequency, and that has a first tap and a second tap that detect electric charges generated by photoelectric conversion in the photoelectric conversion unit; the pixel array unit outputs detection signals of the first tap and the second tap based on reception of the first modulated light and detection signals of the first tap and the second tap based on reception of the second modulated light as light reception signals for determining whether a phase change from irradiation to reception of the second modulated light is 2π or more; a light receiving signal of the first modulated light is used to calculate first depth information; a light receiving signal of the second modulated light is used to calculate second depth information; the first depth information and the second depth information are used to calculate distance information to a subject; In the determination of whether the phase change is 2π or greater, if a difference between the second depth information and the distance information is equal to or greater than a threshold, it is determined that the phase change of the second modulated light is 2π or greater; The threshold value is variable depending on the intensity of the received light signal. Image sensor.

Images