Image sensor capable of outputting time domain difference image and frame image

Abstract

The invention relates to the technical field of image sensors, and provides an image sensor capable of outputting a time domain difference image and a frame image, which comprises a pixel array, a row control circuit, a coding circuit, a data processing circuit, a bias generation circuit and a slope generation circuit, the bias generation circuit is respectively connected with the row control circuit, the coding circuit and the slope generation circuit; the row control circuit is connected with the pixel array; the pixel array is connected with the coding circuit; the coding circuit is connected with the data processing circuit; the image sensor has a frame difference mode that outputs event information and an image mode that outputs a grayscale image. According to the invention, two continuous frames of time domain difference images or single frame images can be output, and the format of the output data is dynamically selected when the time domain difference images are output, so that the compression of the output data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image sensors, and in particular to an image sensor capable of outputting a time-domain differential image and a frame image. Background Art

[0002] In today's digital age, image sensors, as core components for information acquisition, are widely used in fields such as video surveillance, intelligent transportation, aerospace, and medical imaging. Frame difference algorithms, also known as time-domain difference algorithms, are a classic motion detection technique, widely adopted for their simplicity, efficiency, and minimal computational effort. However, most current implementations of frame difference algorithms rely on software processing, requiring image data to be transferred from the sensor to an off-chip processor for computation. This process not only increases data volume but also leads to high power consumption and processing latency. This model of generating frame data on-chip and performing frame difference calculations off-chip has significant limitations when it comes to applications requiring large-scale data processing, low power consumption, and high real-time performance. With the rapid advancement of hardware design technology, particularly advances in microelectronics, integrating frame difference algorithms directly into image sensors has become a new research direction. This integrated design allows frame difference calculations to be performed within the sensor chip, directly outputting event information, thereby reducing data volume and processing latency.

[0003] Current research includes a 64×64 frame-difference image sensor proposed by Eugenio's team at Yale University and Chen Shoushun et al. at Nanyang Technological University from 2008 to 2010. Operating in rolling shutter exposure mode, it boasts a maximum frame rate of 60 frames per second. In 2023, Xu Jiangtao's team at Tianjin University proposed a low-fixed-pattern-noise, global-shutter frame-difference image sensor with an array size of 640×480 and a frame rate of 120 frames per second.

[0004] However, existing technologies achieve relatively low frame rates, making them unsuitable for capturing high-speed moving objects. Furthermore, frame-difference image sensors are significantly affected by changes in ambient light, requiring frame image output to facilitate external algorithms in identifying moving objects. The high frame rate required by frame-difference image sensors also requires an extremely high data bandwidth. For scenes with sparse events, the data output by high-frame-rate frame-difference image sensors is largely invalid. Therefore, on-chip compression of the data output by frame-difference image sensors is essential. However, current research progress, both domestically and internationally, has yet to address this issue. Summary of the Invention

[0005] The present invention mainly solves the technical problem that the existing frame difference algorithm relies on off-chip software processing, resulting in large data transmission volume, high power consumption, obvious delay, and difficulty in meeting the requirements of large data volume, low power consumption or real-time scenarios. The present invention proposes an image sensor that can output time domain difference images and frame images, which can output two consecutive frames of time domain difference images or a single frame image. When outputting the time domain difference image, the output data format is dynamically selected to achieve compression of the output data.

[0006] The present invention provides an image sensor capable of outputting a time-domain differential image and a frame image, comprising: a pixel array, a row control circuit, an encoding circuit, a data processing circuit, a bias generating circuit, and a ramp generating circuit;

[0007] The bias generating circuit is connected to the row control circuit, the encoding circuit, and the ramp generating circuit respectively;

[0008] The row control circuit is connected to the pixel array; the pixel array is connected to the encoding circuit; the encoding circuit is connected to the data processing circuit;

[0009] The image sensor has a frame difference mode and an image mode. The frame difference mode outputs event information, and the image mode outputs a grayscale image.

[0010] Preferably, it further comprises: a timing generator circuit;

[0011] The timing generator circuit is connected to the bias generating circuit, the ramp generating circuit and the row control circuit.

[0012] Preferably, the pixel array includes: a photodiode PPD, a transmission tube M1, a reset tube M2, a first source follower M3, a second source follower M7, a third source follower M9, a current source load M4, a first selection switch tube M5, a second selection switch tube M6, a third selection switch tube M8, a fourth selection switch tube M10, a first storage capacitor C1, and a second storage capacitor C2;

[0013] The transmission tube M1 connects the cathode of the photodiode PPD to the source end of the reset tube M2 and the input end of the first source follower M3;

[0014] The first source follower M3 is connected to the current source load M4;

[0015] The output end of the source follower M3 is connected to the first storage capacitor C1 through the first selection switch tube M5; the second source follower M7 is connected to the first storage capacitor C1 to provide driving capability for the signal stored in the first storage capacitor C1; the second source follower M7 is connected to the third selection switch tube M8; the third selection switch tube M8 is connected to the column bus;

[0016] The output end of the source follower M3 is also connected to the second storage capacitor C2 through the second selection switch tube M6; the third source follower M9 is connected to the second storage capacitor C2; the third source follower M9 is connected to the fourth selection switch tube M10; the fourth selection switch tube M10 is connected to the column bus.

[0017] Preferably, the row control circuit generates a PC signal, a TG signal, an RST signal, an S1 signal, an S2 signal, a SEL1 signal, and a SEL2 signal;

[0018] The PC signal is a bias current source load control signal of the first source follower M3 in the pixel array;

[0019] The TG signal is a strobe signal for the transmission tube M1 in the pixel array;

[0020] The RST signal is a reset signal of the pixel array;

[0021] The S1 signal is a selection signal for the first selection switch M5 in the pixel array;

[0022] The S2 signal is a selection signal for the second selection switch M6 in the pixel array;

[0023] The SEL1 signal is a selection signal for the third selection switch tube M8 in the pixel array;

[0024] The SEL2 signal is a selection signal for the fourth selection switch tube M10 in the pixel array;

[0025] The ramp generating circuit generates V Ramp signal; the V Ramp The signal is the reference voltage required by the threshold comparison circuit TC in the encoding circuit.

[0026] Preferably, the encoding circuit is a plurality of columns; each column of the encoding circuit includes: an encoding preprocessing unit, an event encoding unit and an image encoding unit;

[0027] The encoding preprocessing unit includes: a programmable gain amplifier circuit, an analog voltage storage circuit, a threshold comparison circuit, and a level conversion circuit connected in sequence;

[0028] The event encoding unit includes: a column address generation circuit and a comparator result sampling circuit; the column address generation circuit outputs column address information to the data processing circuit; the comparator result sampling circuit is connected to the encoding preprocessing unit; the comparator result sampling circuit includes two cascaded event encoding unit D flip-flops;

[0029] The image encoding unit includes: an AND gate, an inverter, a multiplexer MUX and a counter; the output of the multiplexer MUX is connected to one input of the AND gate, the output of the level conversion circuit LS in the encoding preprocessing unit is connected to the other input of the AND gate through the inverter, and the output of the AND gate is connected to the enable terminal of the counter.

[0030] Preferably, the data processing circuit includes: a data pre-processing unit, an event processing unit and an image processing unit;

[0031] The data pre-processing unit includes: a plurality of parallel-to-serial circuits, a clock generation circuit, a clock transfer circuit, and a first D flip-flop DFF1 for sampling and outputting data;

[0032] The clock generation circuit includes: an OR gate and an AND gate; the clock transfer circuit includes: a plurality of buffers connected in series;

[0033] The event processing unit includes: a data compression selection circuit;

[0034] The image processing unit includes: a second D flip-flop DFF2 for sampling and outputting data, a third D flip-flop DFF3, a fourth D flip-flop DFF4 and three AND gates.

[0035] Preferably, the data compression selection circuit is composed of two delay chains, and the two delay chains have the same structure;

[0036] Each delay chain consists of 256 delay units connected in series, where the inputs of the first delay units of the two delay chains are connected together, and the outputs of the last delay units of the two delay chains are connected to the S end of the RS latch and the R end of the RS latch respectively; each delay unit includes a gate and a delay buffer.

[0037] Preferably, when the image sensor operates in image mode, global correlated double sampling is performed after the pixel array is globally exposed, and the storage capacitor in the pixel array stores the reset voltage of the current frame and the light signal voltage of the current frame; the row control circuit controls the pixel array to transmit the reset voltage and light signal voltage of the current frame to the encoding preprocessing unit row by row; the programmable gain amplifier circuit in the encoding preprocessing unit takes the difference between the two voltages transmitted by the pixel and transmits the difference to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit, the analog voltage memory outputs the stored signal of the previous row to the threshold comparison circuit; the threshold comparison circuit in the encoding preprocessing unit, together with the counter in the image encoding unit, grayscale quantizes the frame signal and outputs the grayscale information to the data processing module; the data processing module transmits the data to the outside of the image sensor through the IO column by column;

[0038] When the image sensor operates in frame difference mode, the storage capacitors within the pixel array can store the illumination information of the current frame and the illumination information of the previous frame; the row control circuit controls the pixel array to transmit the illumination information of the current frame and the previous frame to the encoding preprocessing unit row by row; the programmable gain amplifier circuit within the encoding preprocessing unit takes the difference between the two frame signals transmitted by the pixels and transmits the difference to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit, the analog voltage memory outputs the stored frame difference signal of the previous row to the threshold comparison circuit; the threshold comparison circuit compares the frame difference signal with the positive and negative thresholds and outputs an event signal to the data processing module. Each column has a fixed column address signal that is transmitted to the data processing module together with the event signal;

[0039] The data processing module determines the optimal data output format and finally transmits the data to the image sensor column by column.

[0040] The present invention provides an image sensor capable of outputting time-domain difference images and frame images. By dynamically selecting the optimal data compression method based on event density, the image sensor effectively reduces the output of redundant data. Compared with existing technologies, the image sensor has the following advantages:

[0041] 1. It has a data compression function that can dynamically select the optimal data compression method based on event density, effectively reducing redundant data. In terms of circuit design, self-zeroing technology is used to reduce the offset voltage of the event encoding circuit, thereby significantly improving the circuit's accuracy and stability.

[0042] 2. The image sensor of the present invention can operate in global exposure mode and rolling exposure mode, and can output two consecutive frames of time-domain difference images or a single-frame frame image. When outputting the time-domain difference image, the output data format is dynamically selected to achieve compression of the output data.

[0043] 3. The image sensor of the present invention supports both frame difference data output and 8-bit precision image data output. By alternately outputting image and frame difference signals, it can rapidly acquire information about changes in high-speed motion scenes while simultaneously using image data to compensate for the shortcomings of frame difference sensors, which are insensitive to slow-moving objects and overly sensitive to scenes with varying light intensity.

[0044] 4. Experimental results show that when the sensor operates in frame-difference mode, the data compression method performs well in event-sparse scenarios, achieving compression ratios of 94% to 98%. This design offers significant advantages in reducing data transmission bandwidth and storage pressure, making it suitable for frame-difference sensors in intelligent surveillance and computer vision applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1This is a diagram of the overall architecture of an image sensor capable of outputting a time-domain difference image and a frame image provided by the present invention;

[0046] Figure 2 is a schematic diagram of a pixel array provided by the present invention;

[0047] Figure 3 is a timing diagram of a pixel array in image mode provided by the present invention;

[0048] Figure 4 is a timing diagram of a pixel array in a frame difference mode provided by the present invention;

[0049] Figure 5 is a schematic diagram of the encoding circuit provided by the present invention;

[0050] Figure 6 Schematic diagram of extracting the encoding preprocessing unit 301 and the event encoding unit 302 from the encoding circuit (encoding circuit in frame difference mode);

[0051] Figure 7 It is a detailed internal structure diagram of the programmable gain amplifier circuit, analog voltage storage circuit, and threshold comparison circuit;

[0052] Figure 8 It is the working sequence of the encoding pre-processing unit 301 and the event encoding unit 302;

[0053] Figure 9 Schematic diagram of extracting the encoding preprocessing unit 301 and the image encoding unit 303 from the encoding circuit (encoding circuit in image mode);

[0054] Figure 10 is the working sequence of the encoding preprocessing unit 301 and the image encoding unit 303;

[0055] Figure 11 is a schematic diagram of a data processing circuit provided by the present invention;

[0056] Figure 12 Extract the schematic diagram of the data pre-processing unit 501 and the event processing unit 502 from the data processing circuit (data processing circuit in frame difference mode);

[0057] Figure 13 It is the working sequence of the data pre-processing unit 501 and the event processing unit 502;

[0058] Figure 14 It is a structural diagram of the data compression selection circuit provided by the present invention;

[0059] Figure 15 Extract the schematic diagram of the data pre-processing unit 501 and the image processing unit 503 from the data processing circuit (data processing circuit in image mode);

[0060] Figure 16 This is the working sequence of the data pre-processing unit 501 and the image processing unit 503.

[0061] Figure numerals: 1 pixel array; 2, row control circuit; 3, encoding circuit; 4, timing generator circuit; 5, data processing circuit; 6, bias generating circuit; 7, ramp generating circuit; 301, encoding preprocessing unit; 302, event encoding unit; 303, image encoding unit; 501, data preprocessing unit; 502, event processing unit; 503, image processing unit. DETAILED DESCRIPTION

[0062] To make the technical problems solved, the technical solutions adopted, and the technical effects achieved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, rather than all of the contents.

[0063] like Figure 1 As shown, an embodiment of the present invention provides an image sensor capable of outputting a time-domain differential image and a frame image, comprising: a pixel array (Pixel Array) 1, a row control circuit (Row Control & Driver) 2, an encoding circuit (Encoding Circuit) 3, a timing generator circuit (Sequencer) 4, a data processing circuit (Data Process Circuit) 5, a bias generating circuit (Bias) 6, and a ramp generating circuit (Ramp) 7.

[0064] The bias generating circuit 6 is connected to the row control circuit 2, the encoding circuit 3, and the ramp generating circuit 7 respectively;

[0065] The row control circuit 2 is connected to the pixel array 1; the pixel array 1 is connected to the encoding circuit 3; the encoding circuit 3 is connected to the data processing circuit 5;

[0066] The image sensor has a frame difference mode and an image mode, wherein the frame difference mode outputs event information and the image mode outputs a grayscale image. The image sensor is a frame difference type sensor.

[0067] The image sensor capable of outputting time-domain differential images and frame images of the present invention further includes: a timing generator circuit (Sequencer) 4 ; the timing generator circuit 4 is connected to the bias generating circuit 6 , the ramp generating circuit 7 , and the row control circuit 2 .

[0068] The row control circuit 2 generates a PC signal, a TG signal, an RST signal, an S1 signal, an S2 signal, a SEL1 signal, and a SEL2 signal;

[0069] The PC signal is a bias current source load control signal of the first source follower M3 in the pixel array 1;

[0070] The TG signal is a strobe signal for the transmission tube M1 in the pixel array 1;

[0071] The RST signal is a reset signal of the pixel array 1;

[0072] The S1 signal is a selection signal of the first selection switch tube M5 in the pixel array 1;

[0073] The S2 signal is a selection signal of the second selection switch tube M6 in the pixel array 1;

[0074] The SEL1 signal is a selection signal for the third selection switch M8 in the pixel array 1; there is one SEL1 signal per row, N in total, and rows from the 1st row to the Nth row are represented by SEL1<0:N-1>;

[0075] The SEL2 signal is a selection signal for the fourth selection switch tube M10 in the pixel array 1 ; there is one SEL2 signal per row, N in total, and rows from the 1st row to the Nth row are represented by SEL2<0:N−1>.

[0076] The ramp generating circuit 7 generates V Ramp signal, the V Ramp The signal is the reference voltage required by the threshold comparison circuit TC in the encoding circuit 3.

[0077] Pixel array 1 has the same pixel structure. This pixel array 1 structure can realize global exposure and rolling exposure, and there are two storage nodes in the pixel. Each storage node can store the analog voltage signal generated after the pixel is exposed, and has an output structure that can output the stored analog voltage to the column bus.

[0078] like Figure 2 As shown, the pixel array 1 includes: a pinned-photodiode (PPD), a transmission tube M1, a reset tube M2, a first source follower M3, a second source follower M7, a third source follower M9, a current source load M4, a first selection switch tube M5, a second selection switch tube M6, a third selection switch tube M8, a fourth selection switch tube M10, a first storage capacitor C1, and a second storage capacitor C2;

[0079] The transmission tube M1 connects the cathode of the photodiode PPD to the source end of the reset tube M2 and the input end of the first source follower M3;

[0080] The first source follower M3 is connected to the current source load M4;

[0081] The output end of the source follower M3 is connected to the first storage capacitor C1 through the first selection switch tube M5; the second source follower M7 is connected to the first storage capacitor C1 to provide driving capability for the signal stored in the first storage capacitor C1; the second source follower M7 is connected to the third selection switch tube M8; the third selection switch tube M8 is connected to the column bus;

[0082] The output end of the source follower M3 is also connected to the second storage capacitor C2 through the second selection switch tube M6; the third source follower M9 is connected to the second storage capacitor C2 to provide driving capability for the signal stored on the second storage capacitor C2; the third source follower M9 is connected to the fourth selection switch tube M10; the fourth selection switch tube M10 is connected to the column bus.

[0083] The transmission tube M1 connects the photodiode PPD and the floating diffusion node (FD) FD1. The reset tube M2 resets FD1 and PPD before exposure begins, and resets FD1 before exposure ends and readout. The source follower tube M3 transfers the electrons of FD1 from the charge domain to the voltage domain for output. The current source load M4 serves as the current source load of the first source follower M3. The first selection switch tube M5, the second selection switch tube M6, the first storage capacitor C1, and the second storage capacitor C2 form a sample-and-hold circuit. When operating in the frame difference mode, S1 samples the odd frame signal V sig1 The first storage capacitor C1 holds the signal until the output ends; S2 samples the even frame signal V sig2 , the second storage capacitor C2 holds this signal until the output ends. When working in image mode, S1 samples the frame reset signal V reset The first storage capacitor C1 holds the signal until the output ends; S2 samples the frame light intensity signal V sig, the second storage capacitor C2 maintains this signal until the output ends. The second source follower M7 / the third source follower M9 acts as a second-level source follower to provide driving capability for the signal stored on the first storage capacitor C1 / the second storage capacitor C2. The third selection switch tube M8 and the fourth selection switch tube M10 act as row selection switches, which are turned on successively to output the signals stored on the first storage capacitor C1 and the second storage capacitor C2 to the column bus in sequence. Among them, the first source follower M3, the current source load M4, the second source follower M7, and the third source follower M9 operate in the saturation region, and the remaining MOS tubes operate as switch tubes in the linear region to ensure that the signal can be correctly transmitted, sampled and held between FD1, the storage capacitor, and the column bus. Both the first storage capacitor C1 and the second storage capacitor C2 use MOS capacitors. MOS capacitors have a high capacitance per unit area, which helps to use capacitors with higher capacitance to improve the parasitic light sensitivity (PLS) of pixels. However, the capacitance of MOS capacitors is affected by the size of the stored voltage on the capacitor. It is necessary to ensure that the voltage on the MOS capacitor is greater than the threshold voltage of the MOS capacitor to limit the capacitance variation range to a smaller range.

[0084] The timing of pixel array 1 when the image sensor works in image mode is as follows Figure 3 As shown, for a frame of pixel timing, there are two working stages. The first working stage is the global sampling and holding stage: First, the PC signal is pulled high, corresponding to Figure 2 The source follower circuit composed of the first source follower M3 and the current source load M4 starts to work normally, the RST signal is pulled low, the S1 signal is pulled high, and the V of the FD1 point in the sampling pixel is reset Signal, TG is pulled high, S2 is pulled high, and the V of FD1 point is sampled sig The second stage is the row-by-row readout stage, where the SEL1 and SEL2 signals of the selected rows are pulled high in turn, and the V reset signal and V sig The signals are sequentially output to the column bus. During the row-by-row readout phase, the TG signal globally resets the PPD according to the exposure time, starting the exposure of the next frame.

[0085] When the image sensor works in the frame difference mode, the timing of pixel array 1 is as follows: Figure 4 As shown, there are still two working stages. In the global sampling and saving stage, in odd frames, the PC signal is first pulled high, corresponding to Figure 2 The source follower circuit composed of the first source follower M3 and the current source load M4 starts to work normally. The RST signal is pulled low, the TG signal is pulled high, and the S1 signal is pulled high. The V of the sampling point FD1 is sig1The signal is stored in the first storage capacitor C1. In the even frame, the difference from the odd frame is that after TG is pulled high, it is not the S1 signal that is pulled high but the S2 signal that is pulled high. The V sig2 The signal is stored on the second storage capacitor C2. In the row-by-row readout stage, in odd frames, the selected row first pulls up the SEL1 signal to output the V sig1 signal, and then pull up the SEL2 signal to output the V sig2 In the even frame, the opposite happens. The selected row first pulls up the SEL2 signal to output the V sig2 signal, and then pull up the SEL1 signal to output the V sig1 Signal.

[0086] like Figure 5 As shown, the encoding circuit 3 is a plurality of columns; each column of the encoding circuit 3 includes: an encoding preprocessing unit 301 , an event encoding unit 302 and an image encoding unit 303 .

[0087] like Figure 5-7 As shown in FIG, the encoding preprocessing unit 301 includes: a programmable gain amplifier (PGA), an analog voltage memory (AVM), a threshold comparator (TC), and a level shift (LS) circuit connected in sequence. Figure 7 As shown, the pixel output signal is used as the input signal of the PGA. After being amplified by the PGA, the signal is transmitted to the AVM. Then the output of the AVM is connected to the input of the TC circuit, and finally the output signal of the comparator is obtained through the TC circuit.

[0088] The event encoding unit 302 includes a column address generation circuit and a comparator result sampling circuit. The column address generation circuit outputs column address information ADDR Column to the parallel-to-serial circuit P / S in the data preprocessing unit 501. The comparator result sampling circuit is connected to the encoding preprocessing unit 301. The comparator result sampling circuit includes two cascaded event encoding unit D flip-flops (D Flip-Flops, DFFs).

[0089] like Figure 6As shown, the encoding pre-processing unit 301 and the event encoding unit 302 implement the frame difference mode of the image sensor. In the frame difference mode, the programmable gain amplifier circuit PGA samples and makes a difference on the signal on the column bus and outputs it to the analog voltage storage circuit AVM. The analog voltage storage circuit AVM then processes the signal output by the programmable gain amplifier PGA and outputs it to the threshold comparison circuit TC. The threshold comparison circuit TC compares the signal processed by the analog voltage storage circuit AVM with the ramp voltage V Ramp A comparison is performed, and the comparison result is output to two cascaded D flip-flops DFF.

[0090] The frame difference mode working sequence of the encoding preprocessing unit 301 and the event encoding unit 302 is as follows: Figure 7 As shown in Figure 1, the programmable gain amplifier (PGA) completes the auto-zeroing process in phase 1. In phase 2, the PGA samples and amplifies the signal on the column bus, outputting it to the storage capacitor of the analog voltage storage circuit AVM. For the threshold comparator circuit, phase 1 also performs the auto-zeroing process. In phase 2, the input voltage is quantized, while the D-type flip-flop (DFF) samples the output of the threshold comparator (TC) twice.

[0091] like Figure 8 As shown, the line processing time required by the event encoding circuit is 2.4μs. However, depending on the data output format selected by the data processing circuit and the amount of data, the total line time is between 2.4μs and 10.4μs. This also means that the reading time of the frame difference signal of a 256×256 chip is between 0.61ms and 2.66ms, and the frame rate is between 375fps and 1627fps.

[0092] like Figure 5 and Figure 9 As shown, the encoding preprocessing unit 301 and image encoding unit 303 implement the image mode of the image sensor. The event encoding unit 302 is disabled by the enable signal IMAGE_EN. The counter is enabled in conjunction with the Simple_ADC signal. The image encoding unit 303 includes an AND gate, an inverter, a multiplexer MUX, and a counter. The output of the multiplexer MUX is connected to one input of the AND gate. The output of the level shifter circuit LS in the encoding preprocessing unit 301 is connected to the other input of the AND gate via the inverter. The output of the AND gate is connected to the EN (enable terminal) of the counter.

[0093] The output of the programmable gain amplifier PGA is connected to the input of the analog voltage storage circuit AVM, the output of the analog voltage storage circuit AVM is connected to the input of the threshold comparison circuit TC, the output of the threshold comparison circuit TC is connected to the level conversion circuit LS, the enable signal IMAGE_EN selects the Simple_ADC signal through the multiplexer MUX, the Simple_ADC signal and the inverted output signal LS_OUT of the level conversion circuit LS are ANDed and connected to the enable terminal (COUNTER) of the counter.

[0094] The encoding pre-processing unit 301 is a common part of the frame difference mode and the image mode. The difference from the frame difference mode is that the input voltage V Ramp It is no longer a fixed level switch, but a ramp voltage is input. The ramp voltage is initially reset to a value less than V ref Voltage: V ref -V set As a result, the comparator in the threshold comparison circuit TC outputs a low level after the reset because the voltage at the positive input terminal is less than the voltage at the negative input terminal. The output of the comparator passes through the level conversion circuit LS and then the inverter to enable the counter. When the ramp voltage is equal to V ref When the Simple_ADC signal is enabled, the counter starts counting. When the ramp signal is greater than V C_par =V ref +V diff_sig When the comparator output voltage flips, the counter count enable is turned off. This achieves a linear conversion between the input voltage and the count value ( Figure 10 In the timing sequence, the reset voltage V is sampled in phase 2. reset and light intensity signal V sig Different input voltages correspond to different flip times, which means different count values, and the relationship between the input voltage value and the count value is linear, thus realizing the conversion of the input voltage from the analog domain to the digital domain.

[0095] The image mode working sequence of the encoding preprocessing unit 301 and the image encoding unit 300 is as follows: Figure 10 The timing of the programmable gain amplifier circuit PGA and the analog voltage storage circuit AVM is the same as the event encoding mode. The difference is that the signal on the column bus sampled by the programmable gain amplifier circuit PGA in stage 2 becomes the reset voltage V reset and light intensity signal voltage V sig .

[0096] like Figure 11 As shown, the data processing circuit 5 includes: a data pre-processing unit 501, an event processing unit 502 and an image processing unit 503;

[0097] like Figure 11-12 As shown, the data pre-processing unit 501 includes: multiple parallel-to-serial conversion circuits (P / S), a clock generation circuit, a clock transfer circuit, and a first D flip-flop DFF1 for sampling and outputting data. The clock generation circuit includes an OR gate and an AND gate; the clock transfer circuit includes: multiple buffers (BUFFs) connected in series.

[0098] The event processing unit 502 includes: a data compression selection circuit (Compress Select). Figure 14 The figure shows the detailed structure of the data compression selection circuit. The circuit consists of two delay chains with identical structures. Each delay chain includes 256 delay cells connected in series. The inputs of the first delay cells in the two delay chains are connected together, and the outputs of the last delay cells in the two delay chains are connected to the S and R terminals of the RS latch, respectively. Each delay cell includes a selector (MUX) and a delay buffer (Delay BUFF).

[0099] The function of judging event density is controlled by the data compression selection circuit ( Figure 14) is implemented as follows: The delay chain consists of a cascade of delay cells, with the delay path selected by the EVENT_EN signal. For a delay cell, if the SEL signal is high, the high-delay path is selected; if the SEL signal is low, the low-delay path is selected. The delay chain's input, Pulse_Input, is a step signal. The EVENT_EN signal for each column is used to select the delay of the delay cell. This results in a linear correlation between the delay of the delay chain's output signal and the Pulse_input signal and the number of high levels in EVENT_EN<0:255>. The number of high levels in the second delay chain's select signal, Delay_Set<0:255>, is set to the selection threshold of 51 corresponding to 256 columns, or K = 51. An RS latch consisting of two NAND gates is used to determine the delay of the two delay chains. Initially, the outputs of both delay chains are low, and the RS latch output is high. After the Pulse_input signal goes high for a while, if the output of the first delay chain goes high before the second delay chain, it indicates that the number of events in a row exceeds the selection threshold and data compression is not required. At this time, the RS latch inputs R = 1, S = 0, and RS_OUT = 0. The Simple_Compress signal goes high to sample RS_OUT later than the second delay chain output. At this point, both delay chain outputs are high. The RS latch inputs R = 1, S = 1, which keeps RS_OUT = 0. After sampling, COMPRESS_EN = 0. Conversely, if the delay of the second delay chain is shorter than that of the first, COMPRESS_EN = 1 after sampling, regardless of whether the first delay chain output is high or low when the Simple_Compress signal goes high, triggering data compression.

[0100] The data preprocessing unit 501 and event processing unit 502 perform data processing in frame difference mode. There are 256 parallel-to-serial circuits P / S, and the control terminals of multiple parallel-to-serial circuits P / S are connected together. The inputs of the parallel-to-serial circuits P / S are the event information (EVENTON, EVENTOFF) of the previous row output by each column event encoding unit 302 and the address information (ADDR Column) of each column. The data compression selection circuit Compress Select then generates a compression enable signal COMPRESS_EN and returns it to the parallel-to-serial circuit P / S. Another output of the data compression selection circuit Compress Select, Simple_Compress, is ANDed with the output of the OR gate and then passes through a clock transmission circuit composed of a buffer BUFF.

[0101] like Figure 13As shown in the timing diagram of the data pre-processing unit 501 and the event processing unit 502 in the frame difference mode, if the data is transmitted to the outside of the chip without compression, 520 clock cycles, i.e., 10.4 μs, are required, of which 1 clock cycle is used to load the output of the event encoding unit 302, 5 clock cycles are used for the data compression selection circuit to generate the COMPRESS_EN signal, 512 clock cycles are used to output data, and 2 clock cycles are used to output a 2-bit high level for the outside of the chip to determine the end of row data output (the outside of the chip determines every 2 bits because there is no 2-bit high-level signal in the normal event signal). If the data is output after compression, 8+(10×number of events) clock cycles are required, of which 8 clock cycles are used to load data, generate the COMPRESS_EN signal, generate a 2-bit off-chip row data end judgment signal, and generate 10 bits of information for each event (2-bit event information + 8-bit address information). However, because the event encoding unit 302 takes at least 2.4 μs to process a row, the shortest row time is 2.4 μs.

[0102] like Figure 11 and Figure 15 As shown, the image processing unit 503 includes a second D flip-flop DFF2, a third D flip-flop DFF3, and a fourth D flip-flop DFF4 for sampling and outputting data, and three AND gates. These two D flip-flops, together with the first D flip-flop DFF1 in the data pre-processing unit 501, divide the multiple parallel-to-serial circuits into four blocks. This means that the image processing unit 503 can output 4 bits of data per data output clock cycle, which is four times faster than the event processing unit 502 in frame difference mode. In image mode, the control terminals of the 256 parallel-to-serial circuits P / S in the data pre-processing unit 501 are connected together. The input of the parallel-to-serial circuits P / S is the light intensity quantization information for each column in the previous row output by the image encoding unit 303. The clock enable circuit, consisting of an OR gate and an AND gate, generates CLK_DATE_IN, which is connected to the CLK terminal of the first D flip-flop DFF1. The clock transfer circuit, consisting of multiple buffers BUFF, outputs CLK_Out_To_IO.

[0103] The timing of the data pre-processing unit 501 and the image processing unit 503 in the image mode is as follows: Figure 16 As shown, the IMAGE_EN signal is always pulled high, and the parallel-to-serial circuit samples the 8-bit light intensity quantization information (CNTO<0:7>) of each column of the previous row output by the image encoding circuit of each column through the rising edge of CLK_DATA during the period when the DATA_LOAD signal is at a high level. The signal Last_Col_Out_Data output by the previous column is output to the next column through the parallel-to-serial circuit P / S of this column.

[0104] The overall working principle of the image sensor capable of outputting time-domain differential images and frame images of the present invention is as follows:

[0105] When the image sensor operates in image mode, after global exposure of the pixel array 1, global correlated double sampling is performed, and the storage capacitor in the pixel array 1 stores the reset voltage of the current frame and the light signal voltage of the current frame; the row control circuit 2 controls the pixel array 1 to transmit the reset voltage and light signal voltage of the current frame to the encoding preprocessing unit 301 row by row; the programmable gain amplifier circuit in the encoding preprocessing unit 301 takes the difference between the two voltages transmitted by the pixel and transmits it to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit 301, the analog voltage memory outputs the stored signal of the previous row to the threshold comparison circuit; the threshold comparison circuit in the encoding preprocessing unit 301, together with the counter in the image encoding unit 303, quantizes the frame signal into grayscale (e.g., 8 bits), and then outputs the grayscale information to the data processing module 5; the data processing module 5 transmits the data to the outside of the image sensor through IO column by column;

[0106] When the image sensor operates in the frame difference mode, the storage capacitor in the pixel array 1 can store the illumination information of the current frame and the illumination information of the previous frame; the row control circuit 2 controls the pixel array 1 to transmit the illumination information of the current frame and the previous frame to the encoding preprocessing unit 301 row by row; the programmable gain amplifier circuit in the encoding preprocessing unit 301 takes the difference between the two frame signals transmitted by the pixels and transmits the difference to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit 301, the analog voltage memory outputs the stored frame difference signal of the previous row to the threshold comparison circuit; the threshold comparison circuit compares the frame difference signal with the positive and negative thresholds and outputs an event signal to the data processing module 5. Each column has a fixed column address signal that is transmitted to the data processing module 5 together with the event signal;

[0107] The data processing module 5 will determine the optimal data output format, and finally transmit the data column by column to the outside of the image sensor through IO.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications to the technical solutions described in the above embodiments, or equivalent replacement of some or all of the technical features therein, do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image sensor capable of outputting a time-domain difference image and a frame image, characterized in that: include: A pixel array (1), a row control circuit (2), an encoding circuit (3), a data processing circuit (5), a bias generating circuit (6), and a ramp generating circuit (7); The bias generating circuit (6) is respectively connected to the row control circuit (2), the encoding circuit (3), and the ramp generating circuit (7); The row control circuit (2) is connected to the pixel array (1); the pixel array (1) is connected to the encoding circuit (3); the encoding circuit (3) is connected to the data processing circuit (5); The image sensor has a frame difference mode and an image mode. The frame difference mode outputs event information, and the image mode outputs a grayscale image.

2. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 1, wherein: Also includes: Timing generator circuit (4); The timing generator circuit (4) is connected to the bias generating circuit (6), the ramp generating circuit (7), and the row control circuit (2).

3. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 2, wherein: The pixel array (1) comprises: a photodiode PPD, a transmission tube M1, a reset tube M2, a first source follower M3, a second source follower M7, a third source follower M9, a current source load M4, a first selection switch tube M5, a second selection switch tube M6, a third selection switch tube M8, a fourth selection switch tube M10, a first storage capacitor C1, and a second storage capacitor C2; The transmission tube M1 connects the cathode of the photodiode PPD to the source end of the reset tube M2 and the input end of the first source follower M3; The first source follower M3 is connected to the current source load M4; The output end of the source follower M3 is connected to the first storage capacitor C1 through the first selection switch tube M5; the second source follower M7 is connected to the first storage capacitor C1 to provide driving capability for the signal stored in the first storage capacitor C1; the second source follower M7 is connected to the third selection switch tube M8; the third selection switch tube M8 is connected to the column bus; The output end of the source follower M3 is also connected to the second storage capacitor C2 through the second selection switch tube M6; the third source follower M9 is connected to the second storage capacitor C2; the third source follower M9 is connected to the fourth selection switch tube M10; the fourth selection switch tube M10 is connected to the column bus.

4. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 3, wherein: The row control circuit (2) generates a PC signal, a TG signal, an RST signal, an S1 signal, an S2 signal, a SEL1 signal, and a SEL2 signal; The PC signal is a bias current source load control signal of the first source follower M3 in the pixel array (1); The TG signal is a strobe signal of the transmission tube M1 in the pixel array (1); The RST signal is a reset signal of the pixel array (1); The S1 signal is a selection signal of the first selection switch tube M5 in the pixel array (1); The S2 signal is a selection signal of the second selection switch tube M6 in the pixel array (1); The SEL1 signal is a selection signal of the third selection switch tube M8 in the pixel array (1); The SEL2 signal is a selection signal of the fourth selection switch tube M10 in the pixel array (1); The ramp generating circuit (7) generates V Ramp signal; the V Ramp The signal is the reference voltage required by the threshold comparison circuit TC in the encoding circuit (3).

5. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 1, wherein: The encoding circuit (3) is a plurality of columns; each column of the encoding circuit (3) comprises: an encoding preprocessing unit (301), an event encoding unit (302) and an image encoding unit (303); The coding preprocessing unit (301) comprises: a programmable gain amplifier circuit, an analog voltage storage circuit, a threshold comparison circuit, and a level conversion circuit connected in sequence; The event encoding unit (302) includes: a column address generation circuit and a comparator result sampling circuit; the column address generation circuit outputs column address information to the data processing circuit (5); the comparator result sampling circuit is connected to the encoding preprocessing unit (301); the comparator result sampling circuit includes two cascaded event encoding unit D flip-flops; The image encoding unit (303) comprises: an AND gate, an inverter, a multiplexer MUX and a counter; the output of the multiplexer MUX is connected to one input of the AND gate, the output of the level conversion circuit LS in the encoding preprocessing unit (301) is connected to the other input of the AND gate through the inverter, and the output of the AND gate is connected to the enable terminal of the counter.

6. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 5, characterized in that: The data processing circuit (5) comprises: a data pre-processing unit (501), an event processing unit (502) and an image processing unit (503); The data pre-processing unit (501) comprises: a plurality of parallel-to-serial circuits, a clock generation circuit, a clock transfer circuit, and a first D flip-flop DFF1 for sampling and outputting data; The clock generation circuit includes: an OR gate and an AND gate; the clock transfer circuit includes: a plurality of buffers connected in series; The event processing unit (502) includes: a data compression selection circuit; The image processing unit (503) comprises: a second D flip-flop DFF2 for sampling and outputting data, a third D flip-flop DFF3, a fourth D flip-flop DFF4 and three AND gates.

7. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 6, wherein: The data compression selection circuit is composed of two delay chains, and the two delay chains have the same structure; Each delay chain consists of 256 delay units connected in series, where the inputs of the first delay units of the two delay chains are connected together, and the outputs of the last delay units of the two delay chains are connected to the S end of the RS latch and the R end of the RS latch respectively; each delay unit includes a gate and a delay buffer.

8. The image sensor capable of outputting a time-domain difference image and a frame image according to claim 7, wherein: When the image sensor operates in image mode, after the pixel array (1) is globally exposed, global correlated double sampling is performed, and the storage capacitor in the pixel array (1) stores the reset voltage of the current frame and the light signal voltage of the current frame; the row control circuit (2) controls the pixel array (1) to transmit the reset voltage and light signal voltage of the current frame to the encoding preprocessing unit (301) row by row; the programmable gain amplifier circuit in the encoding preprocessing unit (301) transmits the difference between the two voltages transmitted by the pixel to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit (301), the analog voltage memory outputs the stored signal of the previous row to the threshold comparison circuit; The threshold comparison circuit in the encoding preprocessing unit (301) and the counter in the image encoding unit (303) together perform grayscale quantization on the frame signal and output the grayscale information to the data processing module (5); the data processing module (5) transmits the data column by column to the outside of the image sensor through the IO; When the image sensor operates in a frame difference mode, the storage capacitor in the pixel array (1) can store the illumination information of the current frame and the illumination information of the previous frame; the row control circuit (2) controls the pixel array (1) to transmit the illumination information of the current frame and the previous frame to the encoding preprocessing unit (301) row by row; the programmable gain amplifier circuit in the encoding preprocessing unit (301) performs a difference between the two frame signals transmitted by the pixels and transmits the difference to the analog voltage storage circuit; at the beginning of the readout timing of each row of the analog voltage storage circuit in the encoding preprocessing unit (301), the analog voltage memory outputs the stored frame difference signal of the previous row to the threshold comparison circuit; the threshold comparison circuit compares the frame difference signal with the positive and negative thresholds and outputs an event signal to the data processing module (5), and each column has a fixed column address signal and transmits it together with the event signal to the data processing module (5); The data processing module (5) determines the optimal data output format and finally transmits the data column by column to the outside of the image sensor.

Images