Time-of-Flight Image Sensor Resolution Enhancement and Increased Data Robustness Using Merging Modules

Abstract

Embodiments of the present disclosure relate to time-of-flight image sensor resolution enhancement and increased data robustness using a merging module. A time-of-flight (ToF) image sensor system includes: a pixel array, wherein each pixel of the pixel array is configured to receive a reflected modulated light signal and demodulate the reflected modulated light signal to generate an electrical signal; a plurality of analog-to-digital converters (ADCs), wherein each ADC is coupled to at least one assigned pixel of the pixel array and is configured to convert a corresponding electrical signal generated by the at least one assigned pixel into an actual pixel value; and a merging circuit coupled to the plurality of ADCs and configured to generate at least one interpolated pixel, wherein the merging circuit is configured to generate each of the at least one interpolated pixel based on actual pixel values ​​corresponding to different pairs of adjacent pixels of the pixel array, each of the at least one interpolated pixel having a virtual pixel value.

Description

Technical Field

[0001] The present disclosure relates generally to time-of-flight image sensors, and more particularly to pixel interpolation utilized by time-of-flight image sensors. Background Art

[0002] Time-of-Flight (ToF) cameras provide a point cloud with distance information in each pixel and address a wide range of applications in the consumer, industrial, and automotive markets. ToF cameras typically have larger pixels, which limits the practical resolution that can be achieved. Many applications require ToF image sensors to increase resolution, while there are increasing demands for data robustness and smaller form factors. These requirements are conflicting and require compromises.

[0003] Therefore, there may be a need for an improved ToF image sensor with increased resolution and / or reduced noise. Summary of the invention

[0004] One or more embodiments provide a time-of-flight (ToF) image sensor system, which includes an illumination source configured to generate a modulated light signal; a pixel array including a plurality of actual pixel lines and a plurality of actual pixel columns, wherein each pixel of the pixel array is configured to receive a modulated light signal reflected from an object as a reflected modulated light signal and demodulate the reflected modulated light signal to generate an electrical signal; a plurality of analog-to-digital converters (ADCs), wherein each ADC is coupled to at least one assigned pixel of the pixel array and is configured to convert a corresponding electrical signal generated by the at least one assigned pixel into an actual pixel value; a merging circuit coupled to the plurality of ADCs and configured to generate at least one interpolated pixel, wherein the merging circuit is configured to generate each of at least one interpolated pixel based on actual pixel values ​​corresponding to different pairs of adjacent pixels of the pixel array, each of the at least one interpolated pixel having a virtual pixel value; and an output circuit coupled to the merging circuit and configured to output a virtual pixel value of each of the at least one interpolated pixel according to an image structure.

[0005] One or more embodiments also provide a method for pixel interpolation in a time-of-flight (ToF) image sensor system, the method comprising generating a modulated light signal; generating a plurality of electrical signals based on pixels of a pixel array, the pixel array receiving the modulated light signal reflected from an object as a reflected modulated light signal; converting the plurality of electrical signals into a plurality of actual pixel values, wherein each of the plurality of actual pixel values ​​corresponds to at least one pixel of the pixel array; generating at least one interpolated pixel, comprising generating each of the at least one interpolated pixel based on actual pixel values ​​corresponding to different pairs of adjacent pixels of the pixel array, each of the at least one interpolated pixel having a virtual pixel value; and outputting the virtual pixel value of each of the at least one interpolated pixel according to an image structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments are described herein with reference to the accompanying drawings.

[0007] Figure 1 is a schematic block diagram of a three-dimensional (3D) depth camera system according to one or more embodiments;

[0008] Figure 2A-2C A pixel interpolation method according to one or more embodiments is shown;

[0009] Figure 2D The pixel structure of the final image as a result of the pixel interpolation performed by the merging module is shown.

[0010] Figure 3 A pixel interpolation method according to one or more embodiments is shown;

[0011] Figure 4 A pixel interpolation method is shown in accordance with one or more embodiments, wherein noise reduction is combined with improved resolution; and

[0012] Figure 5 A schematic block diagram of a merge pipeline according to one or more embodiments is shown. DETAILED DESCRIPTION

[0013] In the following, various embodiments will be described in detail with reference to the accompanying drawings. It should be noted that these embodiments are for illustrative purposes only and should not be interpreted as limiting. For example, although an embodiment may be described as including multiple features or elements, this should not be interpreted as indicating that all of these features or elements are required to implement the embodiment. On the contrary, in other embodiments, some features or elements may be omitted or may be replaced with alternative features or elements. In addition, other features or elements other than the features or elements explicitly shown and described may be provided, such as conventional components of a sensor device.

[0014] Unless otherwise specifically noted, the features of different embodiments may be combined to form other embodiments. Changes or modifications described in one of the embodiments may also be applicable to other embodiments. In some instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments.

[0015] Unless otherwise specified, the connection or coupling between the elements shown in the drawings or described herein may be based on a wired connection or a wireless connection. In addition, such connection or coupling may be a direct connection or coupling without additional intermediate elements, or may be an indirect connection or coupling with one or more additional intervening elements, as long as the general purpose of the connection or coupling (e.g., transmitting a certain signal or transmitting a certain information) is substantially maintained.

[0016] Embodiments relate to optical sensors and optical sensor systems, and to obtaining information about optical sensors and optical sensor systems. A sensor may refer to a component that converts a physical quantity to be measured into an electrical signal (e.g., a current signal or a voltage signal). The physical quantity may, for example, include electromagnetic radiation (such as visible light, infrared (IR) radiation, or other types of illumination signals), current, or voltage, but is not limited thereto. For example, an image sensor may be a silicon chip inside a camera that converts photons of light from a lens into a voltage. The larger the active area of ​​the sensor, the more light that can be collected to create an image.

[0017] As used herein, a sensor device may refer to a device including a sensor and other components, such as bias circuitry, an analog-to-digital converter, or a filter. The sensor device may be integrated on a single chip, although in other embodiments, the sensor device may be implemented using multiple chips or components external to the chip.

[0018] More specifically, embodiments relate to a three-dimensional (3D) image sensor configured to determine the depth (i.e., distance) to an object from a depth camera, or more specifically from a pixel array, using the time-of-flight (ToF) principle. Typically, light is transmitted from a depth camera and reflected back by an object. The reflected light is received by the pixel array, where the reflected light is demodulated during image acquisition on a pixel-by-pixel basis to generate a measurement signal. Each pixel may also be configured to perform multiple image acquisitions at different acquisition times, and thereby generate multiple measurement signals therefrom.

[0019] Indirect time-of-flight (ToF) three-dimensional image (3DI) sensors are based on continuously modulated light for scene illumination and demodulation of the received light at the pixel level during the integration phase. In particular, continuous wave modulation uses a continuous light wave instead of short light pulses, and the modulation is performed according to the frequency of a sine wave. For continuous wave modulation, the detected wave after reflection has a phase shift, and the phase shift is proportional to the distance from the reflecting object or surface. Therefore, the distance can be determined based on the measured phase shift. Depth information is obtained by calculating pixel values ​​from several image acquisitions with a predefined constant phase step between light emission and pixel modulation. For example, four depth images with discrete (consistent) phase differences of 0° / 90° / 180 / 270° can be used to estimate the depth value of each pixel. This is in contrast to pulse modulation, in which the system measures the distance to a 3D object by measuring the absolute time required for a light pulse to propagate from a light source to a 3D scene and back after reflection.

[0020] Although specific embodiments may be described in the context of a continuous wave ToF system (i.e., an indirect ToF system), the concepts described herein may also be extended to direct ToF systems, such as light detection and ranging (LIDAR) systems. LIDAR systems use pulse modulation to measure ToF and make distance measurements. In particular, in LIDAR, a light source transmits a pulse of light into the field of view, and the light is reflected from one or more objects by backscattering. Therefore, LIDAR is a direct ToF system in which a pulse of light (e.g., a laser beam of infrared light) is emitted into the field of view, and an array of pixels detects and measures the reflected beam. For example, an array of photodetectors receives reflections from an object illuminated by the light.

[0021] In LIDAR, the difference in the return time of each light pulse across multiple pixels of a pixel array can then be used to make a digital 3D representation of the environment or generate other sensor data. For example, a light source can emit a single light pulse, and a time-to-digital converter (TDC) electrically coupled to the pixel array can count the time from the time the light pulse is emitted (corresponding to a start signal) until the time the reflected light pulse is received at the receiver (i.e., the pixel array) (corresponding to a stop signal). The "time of flight" of the light pulse is then converted to a distance. In another example, an analog-to-digital converter (ADC) can be electrically coupled to the pixel array (e.g., indirectly coupled with an intermediate element therebetween) for pulse detection and ToF measurement. For example, the ADC can be used to estimate the time interval between start / stop signals by an appropriate algorithm.

[0022] Scans such as oscillating horizontal scans (e.g., from left to right and right to left of the field of view) can illuminate the scene in a continuous scanning manner. Each shot of the laser beam by the light source produces a scan line in the "field of view". By emitting continuous light pulses in different scanning directions, an area called the field of view can be scanned, and objects within the area can be detected and imaged.

[0023] In one example, for each distance sample, the microcontroller triggers a laser pulse from a light source and also starts a timer in a time-to-digital converter (TDC) integrated circuit (IC). The laser pulse propagates through the transmission optics, is reflected by the target field, and is captured by one or more receiving photodiodes of the photodetector array. Each receiving photodiode emits a short electrical pulse that is read out by an analog readout circuit. Each signal read out from the analog readout circuit can be amplified by an electrical signal amplifier.

[0024] The comparator IC recognizes the pulse and sends a digital signal to the TDC to stop the timer. The TDC uses the clock frequency to calibrate each measurement. The TDC sends serial data of the differential time between the start digital signal and the stop digital signal to the microcontroller, which filters out any false readings, averages multiple time measurements, and calculates the distance to the target at that particular field location. By emitting continuous light pulses in different directions established by the scanning mirror, an area (i.e., field of view) can be scanned, a three-dimensional image can be generated, and objects within the area can be detected.

[0025] The signal processing chain of the receiver may also include an ADC for each photodiode or for a group of photodiodes. The ADC is configured to convert an analog electrical signal from a photodiode or a group of photodiodes into a digital signal for further data processing.

[0026] In addition, instead of using the TDC method, the ADC can be used for signal detection and ToF measurement. For example, each ADC can be used to detect the analog electrical signal from one or more photodiodes to estimate the time interval between the start signal (i.e., corresponding to the timing of emitting the light pulse) and the stop signal (i.e., corresponding to the timing of receiving the analog electrical signal at the ADC) using an appropriate algorithm.

[0027] Figure 11 is a schematic block diagram of a three-dimensional (3D) depth camera system 100 according to one or more embodiments. The 3D depth camera system 100 includes a 3D depth camera 1 and an object 2. The object 2 may be one or more objects constituting a 3D scene for imaging. The 3D depth camera system 100 uses a ToF-based 3DI sensor concept (which uses indirect depth measurement, where a continuously modulated illumination signal is the light source of the illumination unit) and a modulated sensor core (wherein the received light is demodulated by a pixel modulation signal on a pixel basis).

[0028] The 3D depth camera 1 is an image sensor, which includes: a sensor chip 3 including various components for performing depth measurement, an illumination unit 4, and an imaging optical device 5 (e.g., a lens). The illumination unit 4 can be an illumination source configured to emit a modulated light signal (e.g., modulated IR light) (such as a light emitting diode (LED) or a vertical cavity surface emitting laser (VCSEL)), or other suitable light sources configured to emit a modulated illumination signal (e.g., modulated IR light or light signal). The illumination unit 4 can be configured to emit modulated IR light toward the object 2, and the imaging optical device 5 (e.g., a lens) can be configured to receive reflected modulated IR light reflected from the surface of the object 2. It should be understood that the terms "light signal", "IR light" and "illumination signal" can be used interchangeably herein.

[0029] The illumination unit 4 may also be configured to emit the modulated IR light towards the object 2 using illumination optics, such as a lens or a diffractive optical element (DOE). Thus, using the illumination optics, the illumination unit 4 may illuminate only the field of view (FOV) of the image sensor 1 .

[0030] The imaging optics 5 (which may be an imaging lens system or objective including one or more lenses or DOEs) is configured to receive reflected IR light reflected from the object 2 (i.e., an object of the 3D scene). The reflected light is directed by the imaging optics 5 to the pixel array 6 of the sensor core 7.

[0031] The sensor chip 3 comprises: a sensor core 7 including a pixel array 6, an illumination driver 8, a modulation driver 9, and a control unit 10. The control unit comprises two phase-locked loop (PLL) circuits 11 and 12, a sequence controller 13, and a processing unit 14 including a merging module configured to increase the resolution of the resulting image by using pixel interpolation. The merging module is a circuit composed entirely of hardware (e.g., logic circuitry) without involving software or program code implemented by a processor.

[0032] N readout channels 16 are used to couple the processing unit 14 to the pixel array 6 in order to read out the electrical signals generated by the pixels. The sensor core 7 includes a multiplexer 15 that couples a pixel or a group of pixels of the pixel array to the corresponding readout channel assigned thereto. The sequence controller 13 is configured to change the pixel assignment to the readout channel 16 so that different pixels can be read out at different times through different readout channels 16. This helps to reduce the complexity of the readout circuitry required to read out the entire pixel array. Each readout channel is coupled to a corresponding ADC at the processing unit 14, wherein each ADC is configured to receive an electrical signal from a pixel or a group of pixels coupled to the assigned readout channel.

[0033] The PLL circuits 11 and 12 may together form a modulator to modulate one or more driver signals. The first PLL 11 may be configured to control the modulation signal f via the illumination driver 8. Illu , the second PLL 12 may be configured to control the modulation signal f via the modulation driver 9 mod Therefore, under the control of the first PLL 11, the illumination driver 8 is configured to generate an illumination modulation signal f having a first modulation frequency. Illu Similarly, under the control of the second PLL 12, the modulation driver 9 is configured to generate a sensor core (pixel) modulation signal f having a second modulation frequency. mod The PLL 11 and the PLL 12 are configured such that for each image acquisition, during at least a portion of the image acquisition time, the first modulation frequency and the second modulation frequency may have a predetermined frequency difference greater than zero.

[0034] Therefore, the 3D depth camera 1 is configured to emit modulated IR light and measure the time it takes for the IR signal to propagate from the depth camera 1 to the 3D scene 2 and back again. The elapsed time (referred to as "time of flight") enables the 3D depth camera 1 to generate raw image data on a pixel-by-pixel basis and output the raw image data to the processing unit 14. In particular, the 3D depth camera 1 may be a continuously modulated ToF camera that uses or determines the reflected modulated light and the modulated signal f mod The flight time is measured by the phase difference between the reflected and modulated light. For example, the phase difference can be converted into a distance based on the known frequency difference and the image acquisition time. Therefore, the depth (distance) to the object can be determined based on the phase difference information determined when the reflected modulated light is received.

[0035] As will be described in more detail below, for each of a plurality of image acquisitions performed at different image acquisition times, the predetermined frequency difference may be set to be greater than zero for all or part of the integration time, all or part of the waiting time, or all or part of the image acquisition time.mod and f Illu The frequencies of can be equal so that there is no frequency difference between them.

[0036] During the depth measurement, the illumination driver 8 may be configured to receive a control signal from the first PLL 11 and modulate the modulation signal f according to the control signal. Illu The irradiation unit 4 then outputs the irradiation modulation signal f to the irradiation unit 4. Illu to generate a modulated optical signal.

[0037] During depth measurement, the modulation driver 9 may be configured to receive a control signal from the second PLL 12 and modulate the sensor core modulation signal f according to the control signal. mod The modulation driver 9 may generate a plurality of modulation signals f received by the sensor core 7, although not shown. mod , wherein each modulation signal has the same modulation frequency and phase to control the modulation frequency of the ToF pixel of the pixel array 6. Here, in this article, the reference modulation signal f mod When , it will be appreciated that this may include one or more signals having the same modulation frequency and phase for modulating the pixels of pixel array 6 .

[0038] The pixel array 6 may be a 224×172 pixel matrix, but is not limited thereto. The pixel array 6 may be configured to receive the reflected IR signal from the imaging optical device 5 and modulate the IR signal using the modulation signal f mod The reflected IR signal is demodulated so that the modulated (reflected) illumination signal and the modulated pixel array 6 (eg, signal f mod ) between the phase difference (ie, phase shift) between the reflected light and the reflected light. In particular, the pixel array 6 can be configured to directly demodulate the reflected light.

[0039] In other words, the sensor core 7 includes a pixel modulation signal f mod The pixel array 6 is a modulated pixel array, and the pixels of the pixel array 6 are configured to receive the modulated light signal reflected from the object 2 as the reflected modulated light signal. The pixels of the pixel array 6 are also configured to use the pixel modulation signal f during image acquisition. mod The reflected modulated light signal is demodulated to generate a measurement signal. The pixel is configured to generate a plurality of measurement signals based on a plurality of image acquisitions acquired at different image acquisition times. A similar process may be performed for each pixel or a subset of pixels of the pixel array 6.

[0040] The processing unit 14 may be configured to receive multiple measurement signals from each pixel (e.g., receive multiple measurement signals from each pixel), and calculate the depth of the object 2 on a pixel-by-pixel basis based on the multiple measurement signals using the phase difference information embedded in each measurement signal. In particular, the processing unit 14 may calculate an absolute phase difference based on a phase difference value associated with the measurement signal for each pixel, and convert the absolute phase difference into depth (distance) information. Thus, a 3D image of the object 2 may be generated, output, and / or displayed.

[0041] The sequence controller 13 may include one or more processors and a clock source, and may be configured to control each of the PLLs 11 and 12. That is, the sequence controller 13 may be configured to control the modulation frequency implemented by the PLLs 11 and 12 to control the two modulation signals f mod and f Illu and the frequency difference between them. The sequence controller 13 may be an internal controller logic with which the processing unit 14 may communicate, or through which the processing unit 14 may parameterize the depth camera 1 to control one or more of the following: f mod and f mod The definition of the corresponding pixel modulation and illumination modulation frequencies, the duration and timing of the introduction of the frequency difference during each image acquisition provided by the PLLs 11 and 12, or an image sequence with a defined frame rate.

[0042] The first PLL circuit 11 and the second PLL circuit 12 may be coupled to each other so that they share a common clock source (eg, provided by the sequence controller 13). Illu and pixel modulation signal f mod The two frequencies of are coupled to each other and can be derived from the same clock frequency. This allows the frequency difference between the two frequencies to be constant and precise based on the desired frequency difference. In the case where the frequency difference is greater than zero, it can be ensured that the modulated signal f mod and f Illu The phase difference between them grows continuously and linearly with time in a consistent and expected manner. In the absence of a frequency difference, it is ensured that the modulated signal f mod and f Illu The phase difference between them does not change with time and remains constant.

[0043] Therefore, the two PLLs 11 and 12 can be programmed by the sequence controller 13 so that their frequencies are slightly different (e.g., for a modulation frequency of 80 MHz, the frequency difference is 1 Hz or 0.1 Hz). Synchronous start-up of the two PLLs 11 and 12 for sensor pixel modulation and illumination signal modulation can be achieved to start at a phase difference of 0° and that the two modulation signals f Illu and f mod The phase difference between the two modulation signals f is constant and increases continuously. Illu and f mod The frequency difference between φ and φ introduces a linear phase shift that grows linearly with time. In principle, phase differences (phase ambiguity) greater than 360° can also be used. By programming a specific frame rate, integration time and waiting time, for example by the processing unit 14, an arbitrary but very precise and easily calculable phase difference can be selected or determined for depth measurement.

[0044] By irradiating the modulated signal f Illu The frequency is set to be the same as the pixel modulation signal f mod The phase difference may also be set by the sequence controller 13 to be constant for at least a portion of the image acquisition. In this case, the frequency difference is equal to zero (ie, there is no frequency difference), and the modulation signal f mod and f Illu The phase difference between them does not change with time.

[0045] This programming of the set frequency difference allows the pixel modulation signal f mod and the illumination modulation signal f Illu The frequency difference may be precisely controllable between φ and φ and continuously variable between φ when required. At other times, there may be no frequency difference, which provides a constant phase difference for a specific period of time during the image acquisition time.

[0046] It should be understood that although the use of two PLLs is described, other equivalent structures are possible. For example, a structure with a combined PLL or modulator structure can be implemented, in which two different frequencies can be obtained. For example, a combined PLL structure can be implemented, in which the modulated signal f mod and f Illu For example, the combined PLL structure may be a dedicated PLL circuit that enables output of two stable signals having coupled frequencies.

[0047] Other embodiments herein relate to using a binning module of the sensor chip 3 (ie, the image sensor 1 ) to increase the resolution of the resulting image by using pixel interpolation, and may be applied independently of the initial image sensor resolution defined by the pixel array 6 .

[0048] By using a merging module consisting only of hardware to perform pixel interpolation and thereby generate interpolated pixels (i.e., virtual pixels), the following advantages can be achieved. First, the computational load on the application processor can be reduced and memory requirements can be relaxed. Second, the pixel resulting from the merging operation can be treated as a single pixel with an area equal to the combined area of ​​the merged pixel. This means that the merged pixel provides better sensitivity and lower noise, which in turn provides a sub-image with improved dynamic range and lower noise. The sub-image can be used as a reference image to make pixel validity / confidence decisions for the constituent pixels in post-processing. The end result will be an increase in the resolution of the original data (up to 3 times), with the additional benefit of higher confidence and improved signal-to-noise ratio (SNR) for the embedded image.

[0049] Figure 2A-2C A pixel interpolation method according to one or more embodiments is shown. In particular, Figure 2A A 2×4 pixel array 200 is shown including two pixel lines L0 and L1 (also referred to as pixel rows) and N pixel columns, where N is equal to 4 in this example. Each pixel P is identified in a matrix format representing its pixel line and pixel row. The pixel array 200 represents a simplified version of the pixel array 6. Therefore, each pixel P is an "actual", "real" or "physical" pixel configured to detect light and generate an electrical signal in response thereto.

[0050] Merging in a "1×2 manner" corresponds to averaging two adjacent pixels (real or virtual) located in the same pixel line but in different pixel columns. Conversely, merging in a "2×1 manner" corresponds to averaging two adjacent pixels (real or virtual) located in the same pixel column but in different pixel lines.

[0051] like Figure 2B As shown, the merging module is configured to merge the pixels of the first line L0 in a 1×2 manner, so that one merged pixel B is generated for every two actual pixels. The merged pixel B can also be called a virtual pixel. The merged pixel B is then inserted between its parent pixels (i.e., the pixels used to generate the merged pixel data). Therefore, in the image line of the image structure, the merged pixel B00 is inserted between its parent pixels P00 and P01. This results in a new line length of 2Npix-1 pixels, where N is the number of columns of the pixel array 200. In this case, the new line lengths of lines L0' and L1' are 7.

[0052] To generate a merged pixel B, the merging module is configured to add the pixel data (e.g., pixel values) of two adjacent pixels P, calculate average data (e.g., average value) based on the sum, and store the average data as the merged pixel data (e.g., merged pixel value) of the merged pixel B. In this case, pixels P00 and P01 are merged (i.e., added and averaged) to derive merged pixel data for merged pixel B00; pixels P01 and P02 are merged (i.e., added and averaged) to derive merged pixel data for merged pixel B01; and pixels P02 and P03 are merged (i.e., added and averaged) to derive merged pixel data for merged pixel B02. The same process is repeated for the pixels in line L01 to produce merged pixels B10, B11, and B12. Therefore, each new line of pixels L0' and L1' includes an arrangement of interlaced actual pixel data and merged pixel data (e.g., interlaced actual pixel values ​​and merged pixel values). As a result, the new lines of pixels L0 ′ and L1 may be referred to as staggered pixel lines, and the pixel array may be referred to as including virtual pixel columns.

[0053] like Figure 2C As shown, the next step is for the merging module to perform the same operation, but on the pixels between the two new lines in a 2×1 manner to form a third line L01, which only includes average pixels (i.e., merged pixels BB). In other words, the averaging operation is performed on adjacent pixels in the same pixel column (including virtual pixel columns inserted between real pixel columns).

[0054] Specifically, the first pixels of the new lines L0' and L1' are averaged together to generate a merged pixel BB (P00; P01); the second pixels of the new lines L0' and L1' are averaged together to generate a merged pixel BB (B00; B10); the third pixels of the new lines L0' and L1' are averaged together to generate a merged pixel BB (P01; P10); the fourth pixels of the new lines L0' and L1' are averaged together to generate a merged pixel BB (B01; B11); the fifth pixels of the new lines L0' and L1' are averaged together to generate a merged pixel BB (P02; P12); etc. As a result, a new line L01 (i.e., a virtual pixel line) with a line length of 2Npix-1 pixels is created.

[0055] Figure 2D The pixel structure of the final image 201 as a result of the pixel interpolation performed by the merging module is shown. In particular, a virtual pixel line L01 is inserted in the image structure or in the memory structure between the interleaved pixel lines L0' and L1'.

[0056] Figure 33 shows a pixel interpolation method according to one or more embodiments. In particular, pixel array 300 represents a simplified version of pixel array 6 and includes three pixel lines L0, L1 and L2 and N pixel columns, where N is equal to 4 in this example. Figure 2B In the process described above, each pixel line is used to generate a virtual line of merged pixels. Figure 2C Instead of inserting merged pixels between their parent pixels as in , each new virtual line of merged pixels is used to replace its corresponding parent pixel line in the image structure 301. Thus, virtual pixel line L0' replaces pixel line L0 in the image structure, virtual pixel line L1' replaces pixel line L1 in the pixel structure, and virtual pixel line L2' replaces pixel line L2 in the image structure to generate the final image 301.

[0057] The newly assembled pixel lines are output by the output circuit as image lines of the image structure. Therefore, the final image structure 301 can be output by the output circuit as image lines arranged according to the sequence indicated by the image structure. As an example, the memory controller can be configured to assign the newly assembled pixel lines to appropriate image lines or memory lines.

[0058] In this case, the pixel noise can be reduced, thereby improving the SNR. To achieve this, the binning module is configured to perform a sliding 2×1 interpolation. In this way, the pixels of each line are binned in a 2×1 manner, resulting in one binned pixel for every two real pixels, resulting in a reduced noise level for each pixel in the image (as a binned pixel). The reduced pixel noise will allow some reduction in the required illumination power, but retain the performance of the image sensor.

[0059] Figure 3 The principles of the pixel interpolation method shown can be extended to include scaling up the resolution. That is, Figures 2A-2D The method shown and Figure 3 The methods shown can be combined. Figure 4 A pixel interpolation method is shown in which noise reduction is combined with resolution improvement according to one or more embodiments.

[0060] Figure 4 A pixel array 300 is shown which represents a simplified version of the pixel array 6 and comprises three pixel lines L0 , L1 and L2 and N pixel columns, wherein N is equal to 4 in this example.

[0061] Similar to the case where the real pixels of each line L0 and L1 are merged (ie, averaged) in a 1×2 manner Figure 2BIn the merging operation shown, each pixel line L0, L1 and L2 of the pixel array 300 is used to create new virtual pixel lines L0', L1' and L'2, respectively. For example, the merging module is configured to merge the pixels of the first line L0 in a 1×2 manner, so that one merged pixel B is generated for every two actual pixels. In this case, pixels P00 and P01 are merged (i.e., added and averaged) to derive merged pixel data of merged pixel B00; pixels P01 and P02 are merged (i.e., added and averaged) to derive merged pixel data of merged pixel B01; pixels P02 and P03 are merged (i.e., added and averaged) to derive merged pixel data of merged pixel B02. The same process is repeated for the pixels in line L01 to generate merged pixels B10, B11 and B12, and the same process is repeated for the pixels in line L02 to generate merged pixels B20, B21 and B22. Each new virtual pixel line L0 ′, L1 ′, and L′2 includes only average pixels (ie, binned pixels).

[0062] In another step, the actual pixel lines L0 and L1 are used to create a new virtual pixel line L01, and the actual pixel lines L2 and L2 are used to create a new virtual pixel line L02. In other words, each pair of adjacent real pixel lines of the pixel array is used to generate a new virtual pixel line. Here, an averaging operation is performed on adjacent pixels in the same pixel column. Therefore, the pixels between the two lines are averaged in a 2×1 manner, thereby forming another pixel line that includes only the averaged pixels.

[0063] The final image structure 302 is assembled by an output circuit configured to assign the newly formed pixel lines to appropriate image lines or memory lines according to the final image structure 302. In this case, the virtual pixel lines are arranged in the following sequence: L0, L01, L1, L02 and L2, so that virtual pixel lines formed by a single actual pixel line in a 1×2 manner are interleaved with those virtual pixel lines formed by two actual pixel lines in a 2×1 manner. In other words, the virtual pixel lines are arranged in the following sequence: L0, L01, L1, L02 and L2, so that virtual pixel lines formed by averaging adjacent pixels from the same actual pixel line are interleaved with virtual pixel lines formed by averaging pixels in adjacent pixel lines. The virtual pixel lines generated in a 2×1 manner from two parent pixel lines are inserted into the image structure between the virtual pixel lines generated in a 1×2 manner from these parent pixel lines.

[0064] As a result of combining noise reduction with improved resolution, the SNR can be improved by reducing pixel noise. This is achieved by a merging module that performs a sliding 2×1 interpolation. The increased possibility of performing the same operation in the vertical direction between pixel lines also improves resolution. As can be seen from the final image structure 302, this requires additional image lines or memory lines to accommodate virtual pixel lines L01, L02, etc. This can be further enhanced by doubling the number of ADCs and reading two lines in parallel. Reduced pixel noise will allow a certain reduction in the required illumination power, but performance is retained, and the increased resolution also brings further benefits.

[0065] Figure 5 A schematic block diagram of a merging pipeline 500 according to one or more embodiments is shown. The merging pipeline 500 is integrated on the sensor chip 3 of the image sensor 1. The merging pipeline includes pixels of the pixel array 6, a multiplexer 15, readout channels 51-0, 51-1, 51-2 and 51-3 selectively coupled to designated pixels of the pixel array 6 through the multiplexer 15, ADCs 53-0, 53-1, 53-2 or 53-3 coupled to corresponding readout channels of the readout channels 51-0, 51-1, 51-2 and 51-3, a merging module 501, and an output circuit 502 coupled to output channels 52-0, 52-1 and 52-2 of the merging module 501.

[0066] The output circuit 502 is configured to receive the actual pixel value from the ADC and the virtual pixel value from the merging module 501 and output the total number of pixel values ​​according to the image structure. The image structure may have more pixels than the pixel array 6.

[0067] As one example of the output circuit 502, the output circuit 502 may include a memory controller 503 coupled to the output channels 52-0, 52-1, and 52-2 of the merging module 501, and an output circuit memory 504 coupled to the memory controller 503. Alternatively, the output circuit 502 may include a first-in-first-out (FIFO) circuit that receives lines of pixel values ​​and outputs the lines of pixel values ​​as image lines of an image structure on a FIFO basis.

[0068] For simplicity, the pixels shown include only a portion of the pixel lines of the pixel array 6. Therefore, the pixels include Pixel[i+0], Pixel[i+1], Pixel[i+2], and Pixel[i+3] located in row "i" and columns 0-3 of the pixel array 6. Each pixel generates an analog electrical signal (i.e., an analog sensor signal) in response to receiving light and is coupled to a corresponding ADC 53-0, 53-1, 53-2, or 53-3. The ADC can be part of the merging module 501, or inserted between the pixel and the merging module 501. Each ADC converts the corresponding analog electrical signal into digital pixel data, which can also be referred to as a digital sensor signal, a digital measurement signal, or actual pixel data.

[0069] Output circuitry 502 may be configured to receive the digital sensor signals as pixel data (ie, pixel values) and output the pixel data as image lines of an image structure.

[0070] The memory controller 503 may be configured to receive the digital sensor signals as pixel data (ie, pixel values) and store the pixel data in appropriate memory lines of the memory 504 corresponding to the image structure.

[0071] The merging module 501 also includes a plurality of arithmetic logic units (ALUs), each ALU being configured to receive a pair of digital sensor signals from the ADC, in which case the pair of digital sensor signals corresponds to two adjacent pixels in the same pixel line (e.g., merging is performed in a "1×2 manner"). In other embodiments, an additional ALU may be provided that is configured to receive digital sensor signals from a pair of adjacent pixels from the same pixel column (e.g., merging is performed in a "2×1 manner"). Thus, typically, each ALU is a digital electronic circuit that includes averaging logic that is configured to receive two digital sensor signals corresponding to a pair of adjacent pixels and generate average (i.e., merged) data as a result of the averaging operation. Pixel data (whether actual or merged) may also be referred to as pixel values.

[0072] In this example, ALU0 is configured to receive digital sensor signals corresponding to Pixel[i+0] and Pixel[i+1], and generate merged pixel data as result R0. Similarly, ALU1 is configured to receive digital sensor signals corresponding to Pixel[i+1] and Pixel[i+2], and generate merged pixel data as result R1. ALU2 is configured to receive digital sensor signals corresponding to Pixel[i+2] and Pixel[i+3], and generate merged pixel data as result R3.

[0073] The merging module 501 may further include data buffers 54-0, 54-1, and 54-2 configured to store the results R0, R1, and R2 as merged data, respectively. In particular, the data buffer 54-0 stores the merged pixel data of the result R0 as WDATA[i+0], the data buffer 54-1 stores the merged pixel data of the result R1 as WDATA[i+1], and the data buffer 54-2 stores the merged pixel data of the result R2 as WDATA[i+2]. Therefore, the data buffers 54-0, 54-1, and 54-2 represent virtual pixels.

[0074] The merging module 501 may also be configured to output the merged data WDATA[i+0], WDATA[i+1], and WDATA[i+2] to the output circuit 502 via the output channels 52-0, 52-1, and 52-2. The output circuit 502 is configured to receive the merged data and arrange the merged data into image lines according to the image structure. In an example, the memory controller 503 may store the merged data in a memory line of the memory 503 corresponding to the image structure.

[0075] The merging module 501 may also include another level ALU coupled to at least one of the data buffers 54-0, 54-1, and 54-2 for receiving the merged pixel data to perform a second level averaging operation. Figure 2C In the operation shown, the second-stage ALU will be coupled to the data buffer storing the merged pixel data B00 and the data buffer storing the merged pixel data B10 to generate the merged pixel data BB (B00; B10) (i.e., the average value of the merged pixel data B00 and B10). In addition, another second-stage ALU will be coupled to the data buffer storing the merged pixel data B01 and the data buffer storing the merged pixel data B11 to generate the merged pixel data BB (B01; B11) (i.e., the average value of the merged pixel data B01 and B11).

[0076] Thus, based on the desired averaging operation, a pair of adjacent pixels to which the ALU is coupled may include two real pixels, two virtual pixels, or a mixture of one real pixel and one virtual pixel, and the ALUs may be arranged in multiple stages to achieve the desired resolution and / or noise reduction.

[0077] Using the final image structure assembled by the output circuit 502, the output circuit 502 can also be configured to read out the image data on an image line by image line basis to the processor for further data processing, including object detection, object classification, etc. Similarly, the memory controller 503 can be configured to read out the image lines from the memory 503 according to the memory lines, and output the image data to the processor for further data processing, including object detection, object classification, etc.

[0078] In view of the above, both aspects (low-resolution merged image and enhanced-resolution image) benefit the system in different ways.

[0079] Merging images can provide a minimum number of 3D points under adverse conditions, which can help the system remain functional even at the expense of reduced performance.

[0080] The enhanced resolution of the image is useful in improving the detection rate of the object detection and classification algorithms used by the system controller 23. Even though the interpolated resolution does not add new data, it can improve the robustness of the existing data and decisions are made at the algorithmic level based on the increased number of pixels, although based on the interpolated data, the increased number of pixels still helps with object detection and classification.

[0081] Although this method can be managed completely in software, the additional computational load may be very high and increase latency to unacceptable levels in production applications. In contrast, by using the merge module 501, although some latency is added, the computational load of the system is not increased (except for processing larger images).

[0082] Interpolation can be a solution to address the gap between the reduction and the video graphics array (VGA) resolution, and object separation can be improved due to the enhanced resolution. The concept is to rework the merging module and memory structure to enable the imager to generate an interpolated image that contains real pixels and interpolated (virtual) pixels. Merging pixels can also have the advantage of better SNR and dynamic range, so it can be expected that the end result will be an increase in the number of effective 3D points.

[0083] In addition, although some aspects have been described in the context of an apparatus, it is apparent that these aspects also represent descriptions of corresponding methods, where a block or device corresponds to a method step or a feature of a method step. Similarly, the various aspects described in the context of a method step also represent descriptions of corresponding blocks or items or features of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device (e.g., a microprocessor, a programmable computer, or an electronic circuit). In some embodiments, one or more method steps may be performed by such an apparatus.

[0084] Depending on certain implementation requirements, the embodiments provided herein may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0085] Instructions may be executed by one or more processors, such as one or more central processing units (CPUs), digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" refers to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules. Similarly, the technology may be implemented entirely in one or more circuits or logic elements.

[0086] The above exemplary embodiments are illustrative only. It should be understood that modifications and variations of the arrangements and details described herein will be clear to other persons skilled in the art. Therefore, the present invention is intended to be limited only by the scope of the upcoming patent claims, and not by the specific details given by the description and explanation of the embodiments herein.

Claims

1. A time-of-flight ToF image sensor system, comprising: an illumination source configured to generate a modulated optical signal; a pixel array comprising a plurality of actual pixel lines and a plurality of actual pixel columns, wherein each pixel of the pixel array is configured to receive the modulated light signal reflected from the object as a reflected modulated light signal and demodulate the reflected modulated light signal to generate an electrical signal; a plurality of analog-to-digital converters ADC, wherein each ADC is coupled to at least one assigned pixel of the pixel array and is configured to convert a corresponding electrical signal generated by the at least one assigned pixel into an actual pixel value and output a plurality of actual pixel values; a merging circuit coupled to the plurality of ADCs for receiving the plurality of actual pixel values ​​therefrom, wherein the merging circuit is configured to generate at least one interpolated pixel and wherein the merging circuit is configured to generate each of the at least one interpolated pixel based on actual pixel values ​​of the plurality of actual pixel values ​​corresponding to different pairs of adjacent pixels of the pixel array, each of the at least one interpolated pixel having a virtual pixel value; as well as an output circuit coupled to the merging circuit for receiving each virtual pixel value therefrom and configured to output the virtual pixel value of each interpolated pixel of the at least one interpolated pixel according to an image structure, The merging circuit comprises: A plurality of arithmetic logic units (ALUs) are coupled to the plurality of ADCs and the output circuit, wherein each ALU of the plurality of ALUs is coupled to a different pair of ADCs to receive the actual pixel values ​​from the ADCs and is configured to combine the actual pixel values ​​to generate a different interpolated pixel of the at least one interpolated pixel. 2 . The ToF image sensor system of claim 1 , wherein each virtual pixel value is an average of the actual pixel values ​​corresponding to the different pairs of adjacent pixels of the pixel array.

3. The ToF image sensor system according to claim 1, wherein: The pixel array includes a first actual pixel line including a first plurality of pixels having first actual pixel values, wherein the first plurality of pixels are arranged into a first plurality of pairs of adjacent pixels, and The merging circuit is configured to generate a first plurality of interpolated pixels having first virtual pixel values ​​based on the first actual pixel values ​​of the first plurality of pairs of adjacent pixels. 4 . The ToF image sensor system of claim 3 , wherein the output circuit is configured to interleave the first actual pixel value with the first virtual pixel value in an image line of the image structure.

5. The ToF image sensor system according to claim 3, wherein: The pixel array includes a second actual pixel line including a second plurality of pixels having second actual pixel values, wherein the second plurality of pixels are arranged into a second plurality of pairs of adjacent pixels, and The merging circuit is configured to generate a second plurality of interpolated pixels having second virtual pixel values ​​based on the second actual pixel values ​​of the second plurality of pairs of adjacent pixels.

6. The ToF image sensor system according to claim 5, wherein the output circuit is configured as: interleaving the first actual pixel value with the first virtual pixel value in a first image line of the image structure, and The second actual pixel value is interleaved with the second virtual pixel value in a second image line of the image structure.

7. The ToF image sensor system according to claim 5, wherein the output circuit is configured as: outputting the first virtual pixel value in a first image line of the image structure, The second virtual pixel value is output in a second image line of the image structure. 8 . The ToF image sensor system of claim 7 , wherein the image structure comprises only virtual pixel values ​​of interpolated pixels.

9. The ToF image sensor system according to claim 5, wherein: The first actual pixel line is adjacent to the second actual pixel line so that the first plurality of pixels and the second plurality of pixels are combined into a third plurality of pairs of adjacent pixels, and The merging circuit is configured to generate a third plurality of interpolated pixels having third virtual pixel values, each of the third virtual pixel values ​​being based on a different first actual pixel value and a different second actual pixel value.

10. The ToF image sensor system according to claim 9, wherein the output circuit is configured as: outputting the first virtual pixel value in a first image line of the image structure, storing the second virtual pixel value in a second image line of the image structure, The third virtual pixel value is stored in a third image line of the image structure, wherein the third image line is inserted between the first image line and the second image line. 11 . The ToF image sensor system of claim 10 , wherein the image structure comprises only virtual pixel values ​​of interpolated pixels.

12. The ToF image sensor system according to claim 9, wherein: The merging circuit is configured to generate a fourth plurality of interpolated pixels having fourth virtual pixel values, each of the fourth virtual pixel values ​​being based on a different first virtual pixel value and a different second virtual pixel value.

13. The ToF image sensor system according to claim 12, wherein the output circuit is configured as: interlacing the first actual pixel value with the first virtual pixel value in a first image line of the image structure, interleaving the second actual pixel value with the second virtual pixel value in a second image line of the image structure, and The third virtual pixel value is interleaved with the fourth virtual pixel value in a third image line of the image structure. 14 . The ToF image sensor system of claim 13 , wherein the third image line is inserted between the first image line and the second image line. 15 . The ToF image sensor system of claim 14 , wherein the image structure comprises a plurality of virtual pixel columns, each of the virtual pixel columns comprising interpolated pixels.

16. The ToF image sensor system of claim 1, wherein a total number of pixel values ​​in the image structure is greater than a total number of pixels in the pixel array.

17. The ToF image sensor system of claim 1, wherein: The merging circuit is configured to receive a first number of actual pixel values ​​from the plurality of ADCs, and The output circuit is configured to output a second number of total pixel values ​​according to the image structure, wherein the second number is greater than the first number.

18. The ToF image sensor system of claim 17, wherein the second number of total pixel values ​​comprises the first number of actual pixel values ​​and a third number of virtual pixel values, the third number of virtual pixel values ​​comprising each virtual pixel value of the at least one interpolated pixel.

19. A pixel interpolation method in a time-of-flight ToF image sensor system, the method comprising: generating a modulated optical signal; generating a plurality of electrical signals based on pixels of a pixel array, the pixel array receiving a modulated light signal reflected from an object as a reflected modulated light signal; converting the plurality of electrical signals from analog to digital into a plurality of actual pixel values ​​by an analog-to-digital converter ADC, wherein each actual pixel value of the plurality of actual pixel values ​​is a digital value corresponding to at least one pixel of the pixel array; generating, by a merging circuit, at least one interpolated pixel after converting the plurality of electrical signals from analog to digital to a plurality of actual pixel values, including generating each of the at least one interpolated pixel based on actual pixel values ​​corresponding to different pairs of adjacent pixels of the pixel array, each of the at least one interpolated pixel having a virtual pixel value; and outputting the virtual pixel value of each interpolated pixel of the at least one interpolated pixel according to an image structure, wherein the total number of pixel values ​​in the image structure is different from the total number of pixels in the pixel array, The merging circuit comprises: A plurality of arithmetic logic units (ALUs), wherein each ALU of the plurality of ALUs is coupled to a different pair of ADCs to receive the actual pixel values ​​from the ADCs and is configured to combine the actual pixel values ​​to generate a different interpolated pixel of the at least one interpolated pixel.

20. The method of claim 19, wherein the pixel array comprises a first actual pixel line, the first actual pixel line comprising a first plurality of pixels having first actual pixel values, wherein the first plurality of pixels are arranged into a first plurality of pairs of adjacent pixels, and the method further comprising: A first plurality of interpolated pixels having first virtual pixel values ​​are generated based on the first actual pixel values ​​of the first plurality of pairs of adjacent pixels.

21. The method of claim 20, wherein the pixel array comprises a second actual pixel line, the second actual pixel line comprising a second plurality of pixels having second actual pixel values, wherein the second plurality of pixels are arranged into a second plurality of pairs of adjacent pixels, and the method further comprising: A second plurality of interpolated pixels having second virtual pixel values ​​are generated based on the second actual pixel values ​​of the second plurality of pairs of adjacent pixels.

22. The method according to claim 21, further comprising: interleaving the first actual pixel value with the first virtual pixel value in a first image line of the image structure; as well as The second actual pixel value is interleaved with the second virtual pixel value in a second image line of the image structure.

23. The method according to claim 21, further comprising: outputting the first virtual pixel value in a first image line of the image structure; as well as The second virtual pixel value is output in a second image line of the image structure.

Images