Electronic device and method

Abstract

An electronic device comprises a circuit configured to generate a coded modulation signal (54; 60) for modulating illumination light (16; 55) emitted by a time-of-flight camera (3).

Description

Technical Field

[0001] The present disclosure relates generally to the field of electronic devices, and more particularly to imaging devices and methods for imaging devices. Background Art

[0002] A time-of-flight camera is a range imaging camera system that determines the distance of an object measuring the time of flight (ToF) of a light signal between the object and the camera for each point of the image. A time-of-flight camera thus receives a depth map of the scene. Typically, a time-of-flight camera has an illumination unit that illuminates an area of ​​interest with modulated light, and an array of pixels that collects light reflected from the same area of ​​interest. When individual pixels collect light from certain parts of the scene, a time-of-flight camera can include a lens for imaging while maintaining a reasonable light collection area.

[0003] An indirect ToF camera (iTOF) measures, for example, the phase delay of reflected infrared (IR) light. The phase data is obtained by correlating the reflected signal with a reference signal (illumination signal). Typically four frames are used to calculate the depth image. In the so-called "2-tap / 4-frame" system, the modulator is a 2-tap electro-optic modulator. The main advantage of the "2-tap / 4-frame" system is that all photogenerated electrons are utilized, rather than dumping half of the electrons, for example in a 1-tap pixel, where the dumped electrons represent the opposite sampled signal.

[0004] Although demodulation techniques for time-of-flight cameras exist, it is generally desirable to provide better demodulation techniques for time-of-flight cameras. Summary of the invention

[0005] According to a first aspect, the present disclosure provides an electronic device comprising circuitry configured to generate a coded modulation signal for modulating illumination light emitted by a time-of-flight camera.

[0006] According to a second aspect, the present disclosure provides a system comprising: a circuit configured to generate a coded modulation signal for modulating illumination light emitted by a time-of-flight camera and / or for demodulating reflected light received by the time-of-flight camera; an image sensor configured to receive the reflected light; and an illumination light source configured to emit the modulated illumination light.

[0007] According to a third aspect, the present disclosure provides a method comprising generating a coded modulation signal for modulating illumination light emitted by a time-of-flight camera.

[0008] According to a fourth aspect, the present disclosure provides a computer program comprising instructions which, when executed on a processor, generate a control signal for generating a coded modulated signal for modulating illumination light emitted by a time-of-flight camera.

[0009] Other aspects are set forth in the following description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments are described by way of example with reference to the accompanying drawings, in which:

[0011] Figure 1 Schematically illustrates the basic operating principle of a ToF camera;

[0012] Figure 2A Shows the linear relationship between phase delay and distance in ToF distance measurement;

[0013] Figure 2B Describes the measurement ambiguity that occurs when the distance to be measured exceeds the maximum measurable distance;

[0014] Figure 3 An embodiment of coding illumination light and its return light is shown, which is intended to handle the ambiguity problem in ToF sensors;

[0015] Figure 4 Schematically illustrates the basic operating principle of a ToF camera with coded illumination light;

[0016] Figure 5 An example of a circuit for implementing on / off keying is shown;

[0017] Figure 6 As an example, an operation sequence in a 2-tap single-phase pixel for obtaining scene distance is shown;

[0018] Figure 7 describing determination of a distance based on the obtained fine phase delay and coarse phase delay;

[0019] Figure 8 As an example, the sequence of operations in an N-phase pixel is shown, where N represents the number of phases that can be stored in the pixel ("or related");

[0020] Fig. 9 As another embodiment, an on / off keyed lighting group 1 and an on / off keyed lighting group 2 are shown;

[0021] Fig. 10A As an example, the spatial arrangement of the demodulation frequencies is shown;

[0022] Fig. 10B Examples of illumination light having a high modulation frequency, illumination light having a low modulation frequency, and resulting coded illumination light are shown;

[0023] Fig.11 Another embodiment showing the grouping of coded illumination light and its return light; and

[0024] Fig.12Schematically illustrates the basic operating principle of a ToF camera with coded demodulated signal. DETAILED DESCRIPTION

[0025] In reference Figure 3 Before describing the embodiments in detail, a general description is given.

[0026] Embodiments described below provide electronic devices that include circuitry configured to generate a coded modulation signal for modulating illumination light emitted by a time-of-flight camera.

[0027] The electronic device may be, for example, an image sensor, such as an image sensor of an indirect time-of-flight camera (ToF).

[0028] A circuit may include any electronic components, semiconductor components, switches, amplifiers, transistors, processing elements, etc.

[0029] A time of flight camera can be a range imaging camera system that determines the distance of an object measuring the time of flight (ToF) of a light signal between the object and the camera for each point of the image. An indirect time of flight camera (iToF) works by measuring phase delay. The phase delay can be the result of correlation of a coded modulation signal with a modulation signal. Phase data can be used to obtain the phase delay. The phase data can include, for example, four correlation phases, such as QA0 and QB 180 , Phase QA 90 and QB 270 , Phase QA 270 and QB 90 and Phase QA 180 And QB0. Each phase data can be stored in a phase memory of the ToF camera. The phase memory can be shared and reused by multiple taps. For example, the electronic device may include two taps, each tap receiving corresponding phase data during a sub-exposure. Alternatively, the electronic device may include N taps, where N represents the number of phases that can be stored in a pixel ("or related").

[0030] In some embodiments, the coded modulated signal is generated by on / off keying of a modulation clock.

[0031] The coded modulated signal may be based on a modulation clock, wherein the modulation clock may generate a periodic signal, more preferably a square wave. The modulation clock may be a continuous wave with a constant frequency. In particular, the coded modulated signal may be generated by "turning on and off" a normal modulation signal generated by the modulation clock. The coded modulated signal may be based on a modulation signal, wherein the modulation signal may be turned on for a predetermined time interval and turned off for a predetermined time interval.

[0032] Instead of using multiple PLLs to generate illumination light with different frequencies as in the conventional method, the coded modulation signal can be generated by only a single phase-locked loop (PLL) by generating a coded modulation signal with on / off keying of a modulation clock. Therefore, the power consumption of the ToF camera can be reduced.

[0033] Compared to a conventional dual-frequency method in which two different measurements with two different modulation signals and different modulation frequencies are used, the electronic device of an embodiment can perform deblurring / dealiasing / phase unwrapping in a single snapshot instead of two (or more) snapshots. This can increase the frame rate, can reduce power consumption and can avoid possible frequency contamination problems.

[0034] In some embodiments, the coded modulated signal comprises at least two modulation frequencies.

[0035] In some embodiments, the coded modulated signal includes a high frequency and a low frequency.

[0036] The high modulation frequency may be a "normal" modulation frequency generated by a modulation clock typically used in sensors of time-of-flight cameras. The coded modulation signal may for example comprise a high frequency corresponding to a first modulation frequency (modulation clock) and a low frequency corresponding to a second modulation frequency and present in the envelope as some specific modulation code.

[0037] In some embodiments, the high frequency of the coded modulated signal is generated by a modulation clock, and the low frequency of the modulated signal corresponds to the frequency of an envelope of the coded modulated signal.

[0038] Since there are other frequency components in the envelope in addition to the higher frequency predefined by the modulation clock, distance measurement can be performed without distance ambiguity. In addition, the system complexity of the ToF camera can be smaller, and only one frame may be required for a ToF camera with N-phase pixels.

[0039] In some embodiments, the circuit is configured to generate dual-modulated illumination light based on the coded modulation signal.

[0040] For example, the first frequency and the second frequency may be created by modulating the modulated light twice to generate a double modulated illumination light.Instead of continuous normal illumination light as shown, illumination light encoded by on-off keying may be emitted.

[0041] In some embodiments, the coded illumination light is generated by modulating the illumination light twice.

[0042] In some embodiments, a first frequency component of the coded modulated signal is used to calculate a coarse phase shift and a second frequency component of the coded modulated signal is used to calculate a fine phase shift.

[0043] The first frequency may have an extended operating range compared to the second frequency without ambiguity in the measured distance (at the expense of higher distance uncertainty). The extended operating range represents the maximum distance that can be measured, and the distance uncertainty represents the probability distribution of the actual distance based on the measured distance. The coarse phase shift may represent the approximate distance of the scene, and the fine phase shift may represent the probability distribution of the true distance based on the approximate distance.

[0044] In some embodiments, the circuit is configured to generate a first coded modulated signal and a second coded modulated signal, wherein the second coded modulated signal has a phase that is completely opposite to the first coded modulated signal.

[0045] In some embodiments, the circuit is configured to provide a spatial arrangement of demodulation frequencies.

[0046] In some embodiments, a coded modulation signal is used to modulate the illumination light emitted by the time-of-flight camera.

[0047] In some embodiments, the coded modulation signal is used to demodulate reflected light received by a phase pixel of a time-of-flight camera.

[0048] A system includes: a circuit configured to generate a coded modulation signal for modulating illumination light emitted by a time-of-flight camera and / or for demodulating reflected light received by the time-of-flight camera; an image sensor configured to receive the reflected light; and an illumination light source configured to emit the modulated illumination light.

[0049] In some embodiments, the image sensor is configured to demodulate the reflected signal according to the spatial arrangement.

[0050] In some embodiments, the image sensor includes multiple types of pixel subarrays, each subarray being configured to demodulate reflected light at a separate demodulation frequency.

[0051] In some embodiments, the image sensor includes two types of pixel subarrays, wherein a first type of pixel subarray demodulates reflected illumination light at a high modulation frequency corresponding to the high modulation frequency of the coded modulation signal, and a second type of pixel subarray demodulates the reflected illumination light at a low modulation frequency corresponding to the low frequency of the coded modulation signal.

[0052] Embodiments also disclose methods that include generating a coded modulation signal for modulating illumination light emitted by a time-of-flight camera.

[0053] Embodiments also disclose a computer program comprising instructions that, when executed on a processor, generate a control signal for generating a coded modulated signal for modulating illumination light emitted by a time-of-flight camera.

[0054] Figure 1 The basic operating principle of a ToF camera is schematically shown. The ToF camera 3 captures a 3D image of a scene 15 by analyzing the flight time of light from a dedicated lighting unit 18 to an object. The ToF camera 3 comprises a camera, such as a 3D sensor 1 and a processor 4. The scene 15 is actively illuminated with modulated light 16 of a predetermined wavelength, for example, with a number of light pulses of at least one predetermined frequency generated by a modulation clock 5 acting as a timing generator, using a dedicated lighting unit 18. The modulated light 16 is reflected back from objects within the scene 15. The lens 2 collects the reflected light 17 and forms an image of the object on the imaging sensor 1 of the camera. Depending on the distance between the object and the camera, a delay is experienced between the emission of the modulated light 16 (e.g., so-called light pulses) and the reception of these reflected light pulses 17 by the camera. The distance between the reflecting object and the camera can be determined based on the observed time delay and the constant value of the speed of light.

[0055] Figure 2A The linear relationship between phase delay and distance in ToF distance measurement is shown. Figure 2A In the figure, the phase delay is shown on the abscissa and the distance is shown on the ordinate. The phase delay is obtained by cross-correlating the backscattered light of the scene with the modulated signal Based on the linear relationship between phase delay and distance, the distance D can be attributed to the phase delay Distance measurement involves distance uncertainty U1, which is Figure 2A The maximum measurable distance D of the flight time measurement is represented as a normal distribution centered at the acquired distance D. max Depends on the frequency of the modulation signal. Maximum measurable distance D max for:

[0056]

[0057] where fmod represents the modulation frequency and c is the speed of light.

[0058] Figure 2B Describes when the distance to be measured exceeds the maximum measurable distance D max The measurement ambiguity that occurs when Figure 2B As shown, multiple distances D, D', D" and the same phase delay Therefore, the distance measurement is ambiguous. That is, when the measured distance exceeds the maximum measurable distance D max When the phase shift is greater than the maximum measurable distance, the distance calculation may be wrong because it cannot distinguish between a signal reflected from a distance exceeding the maximum measurable distance and another signal reflected from a close distance with the same phase shift.

[0059] exist Figure 2B The frequency of the modulation signal used in the measurement is greater than that in Figure 2AOn the one hand, since the frequency of the modulating signal is larger, Figure 2A The maximum measurable distance D max Greater than Figure 2B The maximum measurable distance D max .on the other hand, Figure 2B The measured distance uncertainty U2 is less than Figure 2A That is, a modulated signal with a lower frequency (such as Figure 2A ) allows an extended operating range without ambiguity in the measured distance, but the distance uncertainty is increased.

[0060] In order to extend the operating range, two distance measurements with different modulation frequencies generated by two phase-locked loops (PLLs) can be combined into one joint distance measurement. A first modulation signal with a low frequency, a large maximum measurable distance, and a large distance uncertainty can be used in the first distance measurement. A second modulation signal with a larger frequency, a smaller maximum measurable distance, and a smaller distance uncertainty can be used in the second distance measurement. That is, a distance measurement with multiple modulation signals of different frequencies can be used, so that each modulation signal has a different ambiguity and distance uncertainty.

[0061] Coded Light

[0062] Figure 3 An embodiment of coded illumination light and its return light is shown, which is intended to process the above Figure 2B The ambiguity problem in ToF sensors described in . Instead of using two PLLs to generate two frequencies, as described above about Figure 2A and Figure 2B In the dual frequency method described in the embodiment of the present application, the first frequency and the second frequency are created by modulating the modulated light twice to generate the dual modulated illumination light. Figure 3 The continuous normal lighting shown in the upper part, such as Figure 3 As shown in the middle part of , illumination light coded by on-off keying is emitted (the coded illumination light is generated by "turning on and off" normal illumination light with a high frequency).

[0063] Figure 3 FIG. 4 shows a group of coded illumination lights. Figure 3The group is repeated periodically as indicated by the points in . The group includes four frames T0, T1, T2, T3, each of which has a frame time T. The first frame T1 starts with N×T and ends with (N+1)×T, the second frame T2 starts with (N+1)×T and ends with (N+2)×T, the third frame T3 starts with (N+2)×T and ends with (N+3)×T, and the fourth frame T4 starts with (N+3)×T and ends with (N+4)×T, where N represents the group number. In the first frame T0, the normal illumination light is turned off, so no signal is generated. In the second frame T1, the normal illumination light is turned off in half of the subframe time T / 2 and turned on in the remaining subframe time T / 2. Compared with the second frame T1, the third frame T2 has an inversion, that is, the normal illumination light is turned on in half of the subframe time T / 2 and turned off in the remaining subframe time T / 2. Compared with the frame T0, the fourth frame T3 has an inverted phase, that is, the normal illumination light is turned on. Therefore, the envelope of the encoded illumination light has phase data of 0°, 90°, 270°, and 180° in one packet.

[0064] The double modulated illumination light includes both a high frequency and a low frequency: a high frequency corresponding to a first modulation frequency and a low frequency corresponding to a second modulation frequency defined by the envelope. The first modulation frequency may be a modulation clock typically used in sensors of time-of-flight cameras (see below). Figure 4 The envelope of the coded illumination light represents a low frequency (also referred to herein as "code"), which is 1 / T less than the first modulation frequency of the normal illumination light.

[0065] Figure 3 An example of return light generated by the coded illumination light is also shown. The return light has a phase delay compared to the coded illumination light, which depends on the distance of the object that reflected the light back.

[0066] Figure 4 Schematically illustrated the basic operating principle of a ToF camera with coded illumination light. Figure 4 based on Figure 1, and structural elements having substantially the same function and structure are denoted by the same reference numerals, and repeated description of these structural elements is omitted. The modulation clock 5 generates a high-frequency modulation signal. The modulation signal is transmitted to the on / off keying 20. The on / off keying 20 performs a second modulation on the high-frequency modulation signal received from the modulation clock 5. The on / off keying 20 is controlled by a control signal received from the processor 4. Through the on / off keying 20, a coded modulation signal is generated based on the high-frequency modulation signal obtained from the modulation clock 5 and based on the control signal (“coding”) obtained from the processor 4. The coded modulation signal is transmitted to the lighting unit 18, which generates coded illumination light 16. The coded illumination 16 is reflected back from an object within the scene 15. The reflected light 17 is collected by the 3D sensor 1, which is demodulated with the modulation signal of the modulation clock 5.

[0067] Figure 5 Shown is a method for implementing on / off keying ( Figure 4 20) in FIG. The circuit includes a MOSFET 50 for turning on / off the lighting unit 18. The MOSFET 50 receives a control signal 53 generated by the processor 4. When the MOSFET 50 receives a positive voltage, the MOSFET 50 is turned on and a high-frequency modulation signal 52 generated by the modulation clock 5 is transmitted to the lighting unit 18 via the buffer 51. When the MOSFET 50 receives a zero-level voltage, the MOSFET 50 is turned off, thereby turning off the lighting unit 18. Therefore, a coded illumination light 55 is generated based on a coded modulation signal 54, which is generated by turning on and off the MOSFET 50. The frequency of the control signal 53 represents a low frequency (the envelope frequency of the coded illumination light 55), and the frequency of the modulation signal represents a high frequency, which is generated by the timing generator 5. The coded illumination light 55 is therefore a double-modulated illumination light.

[0068] Demodulation in 2-tap single-phase pixels

[0069] Figure 6 As an example, the operation sequence in a 2-tap single-phase pixel for obtaining the distance of a scene is shown. In a 2-tap single-phase pixel, since there is only one pair of storage nodes (two floating diffusions), four frames are required, namely phases QA0 and QB 180 , Phase QA 90 and QB 270 , Phase QA 270 and QB 90 and Phase QA 180 The phase data of the returning light is obtained by using QB0 and QB1. Each frame includes a plurality of subframes T0, T1, T2, and T3, and each subframe includes an integration phase and a readout phase.

[0070] In the integration phase of the first frame T0, by modulating the clock ( Figure 4 5) in the demodulation of the 2-tap pixel, the modulation clock is equal to Figure 3 High modulation frequency, no phase delay ( Figure 6 After the integration phase, the phase data of the 2-tap pixel is read out, thereby obtaining the phase data QA0 of the first tap (tap A) and the phase data QB0 of the second tap (tap B). 180 In the integration phase of the second frame T1, the Figure 6 After the integration phase, the phase data of the 2-tap pixel is read out, and the phase data QA of the first tap (tap A) is obtained. 90 And the phase data QB of the second tap (tap B) 270 In the integration phase of the third frame T2, the phase delay of 270° ( Figure 6 After the integration phase, the phase data of the 2-tap pixel is read out, and the phase data QA of the first tap (tap A) is obtained. 270 And the phase data QB of the second tap (tap B) 90 In the integration phase of the fourth frame T3, by having a 180° phase delay ( Figure 6 After the integration phase, the phase data of the 2-tap pixel is read out, and the phase data QA of the first tap (tap A) is obtained. 180 And the phase data QB0 of the second tap (second tap B).

[0071] The fine phase delay based on the first (high) modulation frequency is calculated as follows:

[0072]

[0073] The rough phase delay based on the envelope frequency of the code modulated signal is calculated as follows:

[0074]

[0075] By using coded illumination light comprising two modulation frequencies, both the coarse and fine phase delays can be determined from the same measurement (data set), thereby eliminating the need for additional integration and demodulation using the second frequency.

[0076] Figure 7 Determination of a distance based on the obtained fine phase delay and coarse phase delay is described. Figure 7 The upper part of shows the linear relationship between the coarse phase delay and the distance in ToF distance measurement. Figure 7 The lower part of represents the linear relationship between fine phase delay and distance in ToF distance measurement. Figure 2B As described, the high frequency based ToF distance measurement has a small maximum distance D max But it has a lower distance uncertainty U2. In order to obtain the distance of the object in the scene, the information obtained by two different modulation frequencies is combined. When the distance results between the two modulation frequencies are consistent, the distance D A is considered to be the distance of the object without distance ambiguity. The resulting distance D A The distance uncertainty corresponds to the low distance uncertainty U2 of high-frequency measurement.

[0077] Demodulation in N-phase pixels

[0078] As an example, Figure 8 The sequence of operations in an N-phase pixel is shown, where N represents the number of phases that can be stored in the pixel (“or related”). The N-phase pixel of this embodiment includes eight floating diffusions ( Figure 8 ) to store eight information voltages (phase data). Therefore, it is not necessary to read out the taps immediately after each integration. Instead, the frame includes multiple integration phases (subframes) and the taps are read out after demodulating several subframes.

[0079] In the first subframe T1, the demodulation is performed by modulating the N-phase pixels with a clock equal to a high modulation frequency and without phase delay ( Figure 8 In the second subframe T2, by having a 90° phase delay ( Figure 8 In the third subframe T3, the N-tap pixel is demodulated by a modulation clock with a phase delay of 270° ( Figure 8 In the fourth subframe T3, the N-tap pixel is demodulated by a modulation clock with a 180° phase delay ( Figure 8 The N-tap pixels are demodulated using the modulation clock “f180” in FIG.

[0080] After the integration phase of the subframe is completed, the phase data of the N-phase pixel is read out, and the phase data QA0, QA 90 , QA 180 , QA 270 and QB0, QB 90 , QB 180 , QB 270 .

[0081] The fine phase delay based on high modulation frequency is calculated as follows:

[0082]

[0083] Where atan represents the inverse tangent function.

[0084] The rough phase delay based on the envelope frequency of the code modulated signal is calculated as follows:

[0085]

[0086] Improved SNR performance

[0087] Fig. 9 As another embodiment, an on / off keyed lighting group 1 and an on / off keyed lighting group 2 are shown. Both on / off keyed lighting groups 1 and 2 have the same Figure 3 The same shape as shown, wherein the on / keyed illumination group 2 has a completely opposite phase compared to the on / keyed illumination group 1. The on / keyed illumination group 1 and the on / keyed illumination group 2 are sequentially provided to the 2-tap single-phase pixel (such as Figure 6 ) and / or N-phase pixels (as shown Figure 8 ), where the order can be organized or random. The total number of on / off keyed lighting groups 2 is 1 / N times less than the number of on / off keyed lighting groups 1. Therefore, Figure 3 Compared to the embodiment described in Figure 3 In the embodiment shown, light is emitted during the non-lighting time intervals, which can improve the SNR performance.

[0088] By demodulating the supplied coded modulated signal within an integration time, the phase data is integrated in each of the individual taps and read out in a readout phase.

[0089] According to this example, the fine phase delay based on high modulation frequency is calculated as follows:

[0090]

[0091] The rough phase delay based on the envelope frequency of the code modulated signal is calculated as follows:

[0092]

[0093] Among them, Q x,1 represents the phase data obtained by lighting group 1 based on on / off keying, Q X,2 Represents the phase data obtained for lighting group 2 based on on / off keying.

[0094] Space layout

[0095] Fig. 10A The spatial arrangement of the sensor pixel array is shown as an example. The sensor pixel array receives the above Figure 3The coded illumination light. The sensor pixel array includes two different types of pixel sub-arrays f M 、f E , where the first type of pixel subarray f m The reflected illumination light is demodulated at a high modulation frequency corresponding to the high modulation frequency of the coded modulation signal, and the pixel subarray of the second type f E The reflected illumination light is demodulated at a low modulation frequency corresponding to the low frequency (envelope) of the coded modulation signal.

[0096] According to one example of spatial arrangement, the high modulation frequency f M and low modulation frequency f E Columns C1, C2, ..., C of a pixel array of an illumination unit provided to a time-of-flight camera (ToF) N-1 , C N , where the high modulation frequency f M Applied to odd columns, and low modulation frequency f E Applies to even-numbered columns.

[0097] By modulating the frequency f M 、f E The spatial arrangement of the integrated time and the time interval between on / off keying is broken. E It can be any frequency as long as it is less than or equal to f M / 2. By using four high modulation frequencies f with phase delays of 0°, 90°, 180° and 270° M For standard with high modulation frequency f M The columns are demodulated and a low modulation frequency f is obtained with four phase delays of 0°, 90°, 180° and 270°. E For standard with low modulation frequency f E The columns are demodulated.

[0098] Fig. 10B As an example, a modulation frequency f M The illumination light has a low modulation frequency f E illumination light and the resulting coded illumination light.

[0099] As above Figure 6 or Figure 8 As mentioned in the description of , pixels can be demodulated in the integration phase and read out in the readout phase.

[0100] Based on the acquired phase data, the fine phase delay is calculated as follows:

[0101]

[0102] Among them, QA0, QA 90, QA 180 , QA 270 and QB0, QB 90 , QB 180 , QB 270 is marked with f E The columns have a low modulation frequency f E The demodulated data.

[0103] The rough phase delay is calculated as follows:

[0104]

[0105] Among them, QA0', QA 90 ', QA 180 ', QA 270 ' and QB 90 ', QB 90 ', QB 180 ', QB 270 ' is in the M The columns have high modulation frequency f M Please note that the phase data QA0', QA 90 ', QA 180 ', QA 270 'With the corresponding phase data QB 90 ', QB 90 ', QB 180 ', QB 270 'Subtract.

[0106] Embodiments are not limited to the spatial arrangement of the supply to the pixel columns, other spatial arrangements may be used, for example the high modulation frequency f M applied to even columns, and the low modulation frequency f E Applies to odd-numbered columns.

[0107] In addition, the high modulation frequency f M and low modulation frequency f E Columns of a pixel array are provided to pixel phase cells of a time-of-flight camera (ToF), wherein the pixel phase cells may be where backscattered light is collected.

[0108] Backup Code

[0109] Fig.11 Another embodiment of encoding the grouping of illumination light and its return light is shown.

[0110] like Fig.11 The grouping is repeated periodically as indicated by the dots in Figure 3 In the embodiment of the present invention, the packet includes four frames T0, T1, T2, T3. Fig.11The corresponding contents of the two taps of the 2-tap phase pixel: Tap A and Tap B are also shown.

[0111] like Fig.11 As shown in the middle part of , on-off keyed coded illumination light is emitted (the coded illumination light is generated by "turning on and off" normal illumination light with a high frequency). Fig.11 The return light generated by the coded illumination light is also shown in the lower part. Compared with the coded illumination light, the return light has a phase delay that depends on the distance of the object that reflects the light back.

[0112] In the first frame T0, the normal illumination light is turned on, so the coded illumination light has the same shape as the normal illumination light. The cross-correlation result of the coded illumination light and the return light without phase shift is stored in tap A. The cross-correlation result of the coded illumination light and the return light with a 180° phase shift is stored in tap B.

[0113] In the second frame T1, the normal illumination light is turned on, so the coded illumination light has the same shape as the normal illumination light. The cross-correlation result of the coded illumination light without a 90° phase shift and the return light is stored in tap A. The cross-correlation result of the coded illumination light with a 270° phase shift and the return light is stored in tap B.

[0114] In the third frame T2, the normal illumination light is turned off, so the coded illumination light is set to zero. The cross-correlation result between the coded illumination light without a 270° phase shift and the return light is stored in tap A. The cross-correlation result between the coded illumination light with a 90° phase shift and the return light is stored in tap B.

[0115] In the fourth frame T3, the normal illumination light is turned on, so the coded illumination light is set to zero. The cross-correlation result of the coded illumination light without 180° phase shift and the return light is stored in tap A. The cross-correlation result of the coded illumination light without phase shift and the return light is stored in tap B.

[0116] Therefore, the envelope of the encoded illumination light has phase data of 0°, 90°, 270° and 180° in one packet.

[0117] The double-modulated illumination light includes both a high frequency and a low frequency: a high frequency corresponding to a first modulation frequency and a low frequency corresponding to a second modulation frequency defined by an envelope. The first modulation frequency may be a modulation clock ( Figure 4 The envelope of the coded illumination light represents a low frequency, which is 1 / T less than the first modulation frequency of the normal illumination light.

[0118] SNR Comparison

[0119] The SNR of a time-of-flight camera is estimated as follows:

[0120]

[0121] Where Q I is the amplitude of the light with the first frequency, Q Q is the amplitude of the light with the second frequency, and Q ab is the amplitude of the ambient light.

[0122] In a conventional dual-frequency approach involving two different measurements with two different modulation clocks f1, f2, the total demodulated light amplitude Q is divided for each frequency f1, f2 sig In other words, each modulated light has The total SNR of f1 of the conventional dual-frequency is:

[0123]

[0124] And the total SNR of f2 of the conventional dual-frequency is:

[0125]

[0126] According to any of the embodiments disclosed above and according to the above Figures 3 to 11 The total amplitude of the coded modulated light can be equal to the amplitude of the conventional method, that is, Q sig Since, according to the embodiment, the coded modulated light is turned off for half the packet duration, the amplitude of the coded modulated light can be set to be twice greater than the amplitude of the first or second demodulated light of the conventional demodulated light to keep the total amplitude unchanged.

[0127] Therefore, the overall SNR of f1 according to the present disclosure is:

[0128]

[0129] And the overall SNR of f2 according to the present disclosure is:

[0130]

[0131] Therefore, since one sampling data is saved, the overall SNR of the present disclosure is more advantageous than the conventional dual-frequency method, namely, better power efficiency.

[0132] For example, based on Figure 3 Calculate the SNR, where the phase delay of the envelope is defined as ph (0 in this encoding) <ph<π / 4)。

[0133] exist Figure 3 In the first frame T0, the SNR of the current embodiment is calculated as follows:

[0134]

[0135] exist Figure 3 In the second frame T1, the SNR of the current embodiment is calculated as follows:

[0136]

[0137] exist Figure 3 In the third frame T2, the SNR of the current embodiment is calculated as follows:

[0138]

[0139] exist Figure 3 In the fourth frame T3, the SNR of the current embodiment is calculated as follows:

[0140]

[0141] Therefore, in the first frame T0 and the second frame T1, the respective SNRs of the current embodiment are always lower than the conventional dual-frequency method, while in the third frame T2 and the fourth frame T3, the respective SNRs of the current embodiment are always higher than the conventional dual-frequency method.

[0142] In a 2-tap single-phase pixel, the SNR in frames T0 and T1 can be high enough to read out the noise. One possible solution is to ensure that the SNR is high enough to use two different modulation signals, where the second modulation signal is exactly in antiphase with the first modulation signal, such as Fig. 9 shown.

[0143] Coded demodulated signal

[0144] Fig.12 Schematically illustrates the basic operating principle of a ToF camera with coded demodulated signal. Figure 4 based on Figure 1 , and structural elements having substantially the same function and structure are denoted by the same reference numerals, and repeated description of these structural elements is omitted. The modulation clock 5 generates a high-frequency modulation signal. The modulation signal is transmitted to the on / off keying 20'. The on / off keying 20' performs a second modulation on the high-frequency modulation signal received from the modulation clock 5. The on / off keying 20' is controlled by a control signal received from the processor 4. Through the on / off keying 20', an encoded demodulated signal is generated based on the high-frequency modulation signal obtained from the modulation clock 5 and based on the control signal ("encoding") obtained from the processor 4. The encoded demodulated signal is transmitted to the 3D sensor 1, which collects the reflected light 17.

[0145] The 3D sensor 1 can be a phase pixel of a ToF camera. The phase pixel can include one or more floating diffusions, also called "sensing nodes". For example, in a photogate pixel sensor, the floating diffusion stores charge for readout. For example, it can be a region in the active silicon (diffusion) region of an image sensor that is electrically isolated from all other nodes. For example, it can be a quasi-neutral region isolated from other nodes by a pn junction. The charge collected by the two floating diffusions can, for example, be electrons from electron-hole pairs generated by a photosensitive region. A photodiode can be provided to convert light into current. The photodiode empties its charge into a floating diffusion, which is not affected by light. The floating diffusion then stores the charge.

[0146] In some embodiments, the method described herein is also implemented as a computer program that causes a computer and / or processor to perform the method when executed on a computer and / or processor. In some embodiments, a non-transitory computer-readable recording medium is also provided, which stores a computer program product therein, which, when executed by a processor such as the above-mentioned processor, causes the method described herein to be performed.

[0147] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, such as on a chip, and if not stated otherwise, the functions provided by these units and entities may be implemented by software.

[0148] Insofar as the embodiments of the present disclosure described above are implemented at least in part using software-controlled data processing equipment, it will be understood that providing such software-controlled computer programs and the transmission, storage or other media through which such computer programs are provided are contemplated as aspects of the present disclosure.

[0149] Note that the present technology may also be configured as follows.

[0150] (1) An electronic device comprising circuitry configured to generate a coded modulation signal (54; 60) for modulating illumination light (16; 55) emitted by a time-of-flight camera (3).

[0151] (2) The electronic device according to (1), wherein the coded modulation signal (54; 60) is generated by on / off keying of a modulation clock (5).

[0152] (3) The electronic device according to (1) or (2), wherein the coded modulated signal (54; 60) includes at least two modulation frequencies.

[0153] (4) The electronic device according to any one of (1) to (3), wherein the coded modulated signal (54; 60) includes a high frequency and a low frequency.

[0154] (5) The electronic device according to (4), wherein the high frequency of the coded modulated signal is generated by a modulation clock (5), and the low frequency of the modulated signal corresponds to a frequency (54; 60) of an envelope of the coded modulated signal.

[0155] (6) The electronic device according to any one of (1) to (5), wherein the circuit is configured to generate dual-modulated illumination light (16; 55) based on the coded modulation signal.

[0156] (7) The electronic device according to any one of (1) to (6), wherein the coded illumination light (16; 55) is generated by modulating the illumination light twice.

[0157] (8) An electronic device according to any one of (1) to (7), wherein a first frequency component of the coded modulated signal (54; 60) is used to calculate a coarse phase shift, and a second frequency component of the coded modulated signal (54; 60) is used to calculate a fine phase shift.

[0158] (9) An electronic device according to any one of (1) to (8), wherein the circuit is configured to generate a first coded modulation signal and a second coded modulation signal, wherein the second coded modulation signal has a phase completely opposite to that of the first coded modulation signal.

[0159] (10) The electronic device according to any one of (1) to (9), wherein the coded modulation signal (54) is used to modulate the illumination light emitted by the time-of-flight camera.

[0160] (11) The electronic device according to any one of (1) to (9), wherein the coded modulation signal (60) is used to demodulate the reflected light (17) received by the phase pixel of the time-of-flight camera (3).

[0161] (12) A system comprising: a circuit configured to generate a coded modulation signal for modulating illumination light emitted by a time-of-flight camera and / or for demodulating reflected light received by the time-of-flight camera; an image sensor configured to receive the reflected light; and an illumination light source configured to emit the modulated illumination light.

[0162] (13) The system of (12), wherein the image sensor is configured to demodulate the reflection signal according to the spatial arrangement.

[0163] (14) The system according to (12) or (13), wherein the image sensor includes a plurality of types of pixel subarrays (f M , f E ), each subarray (f M , f E )) is configured to demodulate the reflected light at a separate demodulation frequency.

[0164] (15) A system according to any one of (12) to (14), wherein the image sensor includes two types of pixel subarrays (f M , f E ), the first type of pixel subarray (f M ) demodulates the reflected illumination light at a high modulation frequency corresponding to the high modulation frequency of the coded modulation signal, and the second type of pixel subarray (f E ) demodulates the reflected illumination light at a low modulation frequency corresponding to the low frequency of the coded modulation signal.

[0165] (16) A method comprising generating a coded modulation signal (54; 60) for modulating illumination light (16; 55) emitted by a time-of-flight camera (3).

[0166] (17) A computer program comprising instructions which, when executed on a processor (4), generate a control signal (52) for generating a coded modulated signal (54; 60) for modulating illumination light emitted (16; 55) by a time-of-flight camera (3).

Claims

1. An electronic device, comprising: circuitry for generating a coded modulation signal for modulating illumination light into packets of coded illumination light emitted by a time-of-flight camera, in, The first frequency component of the coded modulated signal is a low frequency component, and the second frequency component of the coded modulated signal is a high frequency component, wherein the high frequency component of the coded modulation signal is generated by a modulation clock, the low frequency component of the coded modulation signal corresponds to the frequency of an envelope of the coded modulation signal, and the envelope has phase data of the group of the coded illumination light of the low frequency component of the coded modulation signal; A first distance (D) and a first uncertainty (U1) of a scene are determined by a coarse phase shift calculated using the low-frequency component, and a second uncertainty (U2) is determined based on the first distance (D) of the scene by a fine phase shift calculated using the high-frequency component, wherein the second uncertainty is smaller than the first uncertainty.

2. The electronic device according to claim 1, wherein: The coded modulation signal is generated by on / off keying of a modulation clock.

3. The electronic device according to claim 1, wherein: The coded modulated signal includes at least two modulation frequencies.

4. The electronic device according to claim 1, wherein: The circuit is configured to generate dual-modulated illumination light based on the coded modulation signal.

5. The electronic device according to claim 1, wherein: The coded illumination light is generated by modulating the illumination light twice.

6. The electronic device according to claim 1, wherein: The circuit is configured to generate a first coded modulated signal and a second coded modulated signal, wherein the second coded modulated signal has a phase that is completely opposite to the first coded modulated signal.

7. The electronic device according to claim 1, wherein: The coded modulation signal is used to demodulate the reflected light received by the phase pixel of the time-of-flight camera.

8. An imaging system, comprising the electronic device of claim 1, the circuit being configured to generate a coded modulation signal for demodulating reflected light received by a time-of-flight camera; an image sensor configured to receive the reflected light; and an illumination light source configured to emit the modulated illumination light, in, The image sensor includes two types of pixel subarrays, wherein a first type of pixel subarray demodulates the reflected illumination light at a high modulation frequency corresponding to a high frequency component of the coded modulation signal, and a second type of pixel subarray demodulates the reflected illumination light at a low modulation frequency corresponding to a low frequency component of the coded modulation signal, Each of the pixel sub-arrays is included in a sensor pixel array.

9. The system according to claim 8, wherein: The image sensor is configured to demodulate the reflected signal according to the spatial arrangement.

10. A method for an imaging device, comprising generating a coded modulation signal for modulating illumination light into packets of coded illumination light emitted by a time-of-flight camera, in, The first frequency component of the coded modulated signal is a low frequency component, and the second frequency component of the coded modulated signal is a high frequency component, wherein the high frequency component of the coded modulation signal is generated by a modulation clock, the low frequency component of the coded modulation signal corresponds to the frequency of an envelope of the coded modulation signal, and the envelope has phase data of the group of the coded illumination light of the low frequency component of the coded modulation signal; A first distance (D) and a first uncertainty (U1) of a scene are determined by a coarse phase shift calculated using the low-frequency component, and a second uncertainty (U2) is determined based on the first distance (D) of the scene by a fine phase shift calculated using the high-frequency component, wherein the second uncertainty is smaller than the first uncertainty.

11. A computer program product comprising instructions that, when executed on a processor, generate a control signal for generating a coded modulation signal for modulating illumination light into packets of coded illumination light emitted by a time-of-flight camera, in, The first frequency component of the coded modulated signal is a low frequency component, and the second frequency component of the coded modulated signal is a high frequency component, wherein the high frequency component of the coded modulation signal is generated by a modulation clock, the low frequency component of the coded modulation signal corresponds to the frequency of an envelope of the coded modulation signal, and the envelope has phase data of the group of the coded illumination light of the low frequency component of the coded modulation signal; A first distance (D) and a first uncertainty (U1) of a scene are determined by a coarse phase shift calculated using the low-frequency component, and a second uncertainty (U2) is determined based on the first distance (D) of the scene by a fine phase shift calculated using the high-frequency component, wherein the second uncertainty is smaller than the first uncertainty.

Images