Near-infrared convolution image sensor structure design

Abstract

The invention discloses a near-infrared convolutional image sensor structure design, and belongs to the technical field of infrared intelligent vision and convolutional neural network calculation. The circuit comprises a macro pixel unit and a corresponding reading circuit, wherein the macro pixel unit is composed of n * n sub-pixels. And each sub-pixel integrates an infrared photodiode and a static random access memory and is used for converting an optical signal into charges and storing a convolution kernel weight. The readout circuit adopts a convolution type metal wire interconnection structure, controls current of each sub-pixel to be reused according to weight and directionally flow in an analog domain, integrates the current through a double-path integrator, subtracts the current through a subtractor, and directly completes product accumulation operation of near-infrared image information and a convolution kernel on a pixel level, so that the readout circuit is more accurate and reliable. And outputting a voltage signal representing a convolution result. According to the design, perception and calculation integration is achieved, the problems of power consumption and delay caused by data carrying in a traditional architecture are solved, and the method is suitable for real-time edge intelligent visual processing under the near-infrared complex background.

Description

Technical Field

[0001] This invention belongs to the field of infrared intelligent vision and convolutional neural network computing technology, specifically relating to the structural design of a near-infrared convolutional image sensor. Background Technology

[0002] In complex near-infrared environments, efficient feature extraction of targets typically relies on image convolution operations. Most current mainstream intelligent vision systems adopt a "perception-processing separation" model based on the von Neumann architecture: first, the image sensor performs integration and analog-to-digital conversion of the optical signal to form digital image data; then, the data is transferred to a back-end processor (such as a CPU, GPU, or dedicated accelerator) via an on-chip bus or I / O interface to execute algorithms such as convolutional neural networks. This discrete processing architecture has several inherent drawbacks:

[0003] First, the frequent transfer of massive amounts of raw image data between sensors, memory, and computing units generates enormous power consumption and bandwidth pressure during data transmission, posing a severe challenge to the chip's internal interconnects and external interfaces. Second, the long and multi-layered signal processing chain, requiring multiple levels of caching, conversion, and scheduling from photon sensing to final output, leads to significant accumulated system latency, making it difficult to meet the real-time requirements of fast-moving scenarios or highly dynamic environments. More importantly, the physically separated sensing and computing modules result in repeated memory accesses and data copies for each convolution operation, creating a severe "memory wall" bottleneck that limits the overall system energy efficiency.

[0004] With the rapid development of edge computing and intelligent terminal applications, especially in resource-constrained and power-sensitive near-infrared vision systems (such as unmanned platforms, security monitoring, and mobile devices), the high power consumption and high latency issues of traditional architectures are becoming increasingly prominent. Therefore, the industry urgently needs an innovative solution that can move convolutional computation to the sensing end, fundamentally reducing data movement and achieving "sensing and computing integration," thereby significantly improving the system's real-time response capability and energy efficiency while ensuring processing accuracy.

[0005] To address the aforementioned technical bottlenecks, the present invention aims to overcome the limitations of existing discrete architectures and provide a novel circuit design that directly performs analog domain convolution operations within a pixel, thereby completely solving the power consumption and latency problems caused by data transfer. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention provides a near-infrared convolutional image sensor structure design that integrates perception and computation, avoiding the power consumption and latency problems caused by data transfer in traditional architectures. It is suitable for real-time edge intelligent vision processing in complex near-infrared backgrounds.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A near-infrared convolutional image sensor structure design includes a macro-pixel unit array and a corresponding readout circuit, wherein:

[0009] The macro-pixel unit array includes multiple macro-pixel units, each macro-pixel unit is composed of n×n sub-pixels, and each sub-pixel integrates an infrared photodiode and a static random access memory. The static random access memory stores a weight value and controls the direction of the photocurrent output by the photodiode.

[0010] The readout circuit includes a first integrator, a second integrator, and a subtractor. The first integrator and the second integrator receive photocurrents from each sub-pixel within the plurality of macropixel units, which are weighted and distributed, through a metal wire interconnection structure. The first integrator integrates the photocurrents received by the first integrator and the second integrator to generate a first sampling voltage and a second sampling voltage, respectively. The subtractor subtracts the first sampling voltage from the second sampling voltage and directly outputs a voltage signal representing the result of the n×n convolution operation.

[0011] Furthermore, the weight values ​​stored in the static random access memory are logic "1" or "0", corresponding to the +1 and -1 weights in the convolution kernel, respectively.

[0012] Furthermore, the metal wire interconnect structure is a convolutional global interconnect, and its topology layout is determined according to the convolution kernel size, sliding path, and sliding step size.

[0013] Furthermore, the photocurrents of each sub-pixel flowing into the same integration node are automatically summed and accumulated in the analog domain based on Kirchhoff's current law.

[0014] Furthermore, the first integrator and the second integrator have the same structure, both adopting a voltage-clamped correlated dual-sampling capacitor transimpedance amplifier structure.

[0015] Furthermore, both the first and second integrators include two five-transistor amplifiers, two metal-oxide-metal capacitors, and two complementary MOS switches, and subtract the pre-stage fixed voltage and noise voltage through a correlated double sampling method.

[0016] Furthermore, the output of the first five-transistor amplifier is connected to one end of the first metal-oxide-metal capacitor via the first complementary MOS switch, and the other end of the first metal-oxide-metal capacitor is connected to the non-inverting input of the second five-transistor amplifier; the inverting input of the second five-transistor amplifier is shorted to its output to form a unity-gain amplifier; one end of the second metal-oxide-metal capacitor is connected between the output of the first five-transistor amplifier and the first complementary MOS switch, and the other end is grounded via the second complementary MOS switch and simultaneously connected to the non-inverting input of the second five-transistor amplifier.

[0017] Furthermore, the subtractor consists of a two-stage amplifier and four resistors of the same value, enabling a linear operation of subtracting the second sampling voltage from the first sampling voltage.

[0018] Furthermore, the first sampling voltage is connected to the inverting input terminal of the secondary amplifier through a first resistor, and the inverting input terminal of the secondary amplifier is also connected to its output terminal through a second resistor, which outputs the final voltage signal; the second sampling voltage is connected to the non-inverting input terminal of the secondary amplifier through a third resistor, and the non-inverting input terminal of the secondary amplifier is also grounded through a fourth resistor.

[0019] Furthermore, the size n of the macro-pixel unit array is a positive integer greater than 1, which is suitable for convolution kernel operations of different sizes.

[0020] The beneficial effects of this invention are as follows:

[0021] Achieving "integrated sensing and computing" completely eliminates data transfer bottlenecks: By directly embedding the multiplication and addition operations of n×n convolution kernels into the macro-pixel units, real-time calculation of photocurrent signals and convolution weights is performed in the analog domain, directly outputting the convolutional voltage signal. This fundamentally avoids the large-scale, high-power transmission of raw image data between sensors, memory, and processors in traditional architectures, significantly reducing the system's total power consumption and data transmission bandwidth requirements.

[0022] Significantly reduces system latency and enhances real-time response capabilities: Convolution operations and the photocurrent-to-photovoltage conversion process are performed simultaneously, eliminating the need to wait for column-level readout, analog-to-digital conversion, and back-end digital processing. The signal processing chain is greatly shortened, making it particularly suitable for edge intelligence applications such as near-infrared rapid target detection and tracking, which have extremely high real-time requirements.

[0023] It features high energy efficiency and flexibility: the circuit employs full analog domain computation, utilizes Kirchhoff's current law for current accumulation, and leverages programmable SRAM to store weights (±1), resulting in high energy efficiency during computation. This architecture offers a high degree of freedom in scalable design and reconfigurable weights, making it adaptable to convolutional kernels of different sizes and various infrared focal plane detectors (such as shortwave, midwave, and longwave), with broad application potential.

[0024] Maintaining basic imaging functions without additional overhead: While implementing convolution calculations, the circuit's core integration and sampling operations are compatible with traditional imaging units. Simulation results demonstrate that this design performs well in specific convolution operations (such as smoothing filtering and edge extraction). Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall structural design of a near-infrared convolutional image sensor according to the present invention;

[0026] Figure 2 A schematic diagram of a switch for controlling the direction of photocurrent output from an infrared photodiode using SRAM;

[0027] Figure 3 A schematic diagram of the core structure of the macropixel unit readout circuit;

[0028] Figure 4 Detailed internal circuit diagram of a voltage-clamped correlated dual-sampling capacitor transimpedance amplifier;

[0029] Figure 5 This is a timing diagram showing the voltage variation of key nodes over time in a voltage-clamped correlated double sampling circuit.

[0030] Figure 6 This is a diagram of the internal circuit structure of a subtractor used to subtract two voltages.

[0031] Figure 7 A schematic diagram of the path summation of sub-pixel currents and a verification diagram of the output voltage after integration and subtraction;

[0032] Figure 8 The images show a comparison of the simulation results using smooth convolution kernels, where a is the original image, b is the output image of the circuit of this invention, and c is the image calculated by the software. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] This invention aims to overcome the bottlenecks of existing discrete processing architectures by moving convolution operations from the back-end computing unit to the pixel level of the image sensor, achieving "integrated perception and convolution operations." This allows for direct multiplication and addition of near-infrared image information and convolution kernels within the analog domain, enabling the readout circuit to directly output the convolved image voltage signal, fundamentally reducing power consumption and latency issues caused by data transfer. Specifically, this invention provides a near-infrared convolution image sensor, such as... Figure 1 As shown, the macropixel readout circuit includes a macropixel unit array and its corresponding communication connection. Each macropixel unit consists of n×n sub-pixels, and each n×n macropixel unit can perform an n×n convolution kernel operation, forming the smallest operational unit. The direction of photocurrent generated by the sub-pixel units within each macropixel unit is controlled by the static random access memory (SRAM) to which the sub-pixel unit belongs.

[0035] The following example illustrates the concept of a macropixel unit consisting of 3×3 subpixels. Figure 2 As shown, each sub-pixel consists of an infrared photodiode and a static random access memory (SRAM) in the silicon-based circuitry below it. When a bias voltage Vsub is applied to one end of the infrared photodiode, the diode receives the photocurrent generated by the infrared light. , The current flows to the switch controlled by SRAM, which stores a single weight value (logic '1' or '0'). This value directly maps to the weights of the convolution kernel (e.g., +1 or -1) and controls the on / off state of a switch. When the SRAM stores a high level "1", the first path A is turned on. The wire flowing towards the first path A (i.e., the sub-pixel unit current flowing towards the first integrator) Figure 2 (As shown in the blue section); when the SRAM stores a low level "0", the second path B is turned on. The wire flowing to the second path B (i.e., the sub-pixel unit current flows to the second integrator) Figure 2 (As shown in the green section), the first and second integrators are interconnected by a specific metal wire. By writing values ​​into the SRAM from the outside, the state of the photodiode can be stored and the current flow can be adjusted, which is equivalent to completing the calculation of one bit weight and current before the current forms the integrated voltage.

[0036] To enable the operation of the entire n×n convolution kernel, the sub-pixels within each macropixel unit are not independent, but connected at the array level through a global, convolutional metal wire interconnection structure. The topology of this interconnection structure is determined by the kernel size, sliding path, and sliding stride. For example, for a 3×3 convolution kernel, the current path (A or B) of each sub-pixel within the current macropixel unit is connected to the corresponding integration node of the adjacent macropixel unit according to its weight value. All currents flowing into the same node are automatically summed (accumulated) at the metal wire node based on Kirchhoff's Current Law (KCL).

[0037] A macropixel unit is constructed from 3×3 subpixel units. A binary convolution operation is performed on the 3×3 region formed by the first macropixel unit H1 to the ninth macropixel unit H9. The convolution kernels are shown in Table 1.

[0038] Table 1:

[0039]

[0040] like Figure 2As shown, the interconnection method is as follows: the ninth sub-pixel unit S9 in the first macropixel unit H1 is connected to the first path A in the fifth macropixel unit H5; the eighth sub-pixel unit S8 in the second macropixel unit H2 is connected to the first path A in the fifth macropixel unit H5; the seventh sub-pixel unit S7 in the third macropixel unit H3 is connected to the first path A in the fifth macropixel unit H5; the sixth sub-pixel unit S6 in the fourth macropixel unit H4 is connected to the first path A in the fifth macropixel unit H5; the fifth sub-pixel unit S5 in the fifth macropixel unit H5 is connected to the second path B in the fifth macropixel unit H5; the fourth sub-pixel unit S4 in the sixth macropixel unit H6 is connected to the first path A in the fifth macropixel unit H5; the third sub-pixel unit S3 in the seventh macropixel unit H7 is connected to the second path B in the fifth macropixel unit H5; the second sub-pixel unit S2 in the eighth macropixel unit H8 is connected to the second path B in the fifth macropixel unit H5; and the first sub-pixel unit S1 in the ninth macropixel unit H9 is connected to the second path B in the fifth macropixel unit H5. The currents on the first path A and the second path B are automatically summed using Kirchhoff's current law.

[0041] like Figure 3 As shown, the macro-pixel unit readout circuit consists of a first integrator, a second integrator, and a subtractor. The first integrator integrates and samples the summed current in the first path A to form a first sampling voltage. The second integrator integrates and samples the summed current in the second path B to form the second sampling voltage. First sampling voltage Second sampling voltage The output voltage is obtained by subtracting the values ​​using a subtractor. This completes the convolution operation for the 3×3 macro-pixel unit region. The mathematical derivation formula is as follows:

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] Among them, V AV represents the sampling voltage generated by the first integrator. B V represents the sampling voltage generated by the second integrator. O I represents the operational voltage output of the subtractor. sram=1 I represents the photocurrent of a sub-pixel with a weight of 1. sram=0 w represents the photocurrent of a sub-pixel with a weight of -1. i V represents the weight at the i-th position in the convolution kernel. i represents the sub-pixel integral voltage at the i-th position in the convolution kernel, and t represents the sampling time.

[0050] like Figure 4 As shown, the first and second integrators have the same structure and will be described uniformly below. The internal circuit structure of the integrator mainly consists of two five-transistor amplifiers, two capacitors, and two switches. Of the two five-transistor amplifiers, the first is an N-type five-transistor amplifier, and the second is a P-type five-transistor detector, designed for use with PonN infrared detectors. If the front-end infrared photodiode is a NonP-type infrared detector, then the first detector is a P-type input tube five-transistor detector, and the second amplifier is an N-type input tube five-transistor amplifier. Both capacitors are metal-oxide-metal (MIM) capacitors with a capacitance of 2fF / um². Both switches are complementary MOS switches consisting of a P-type MOS transistor and an N-type MOS transistor arranged back-to-back. The integrator uses a voltage-clamped correlated double sampling method. Specifically, taking a PonN infrared photodiode detector as an example, a voltage V1 is applied to the positive input terminal of the N-type input tube five-transistor amplifier to apply a working bias voltage to the infrared photodiode detector. When the CMOS complementary switches INT and RST are closed, the voltage across the MIM capacitor Cin is equal to V1, the voltage across the left plate of the MIM capacitor Ccds (V2) is equal to V1, and the voltage across the right plate (V3) is 0. Subsequently, switches INT and RST are opened, and current A / B is input from the negative input of the N-type input transistor quintuplet amplifier. Integrating across Cin, a voltage is formed. Since the charge on the left and right plates of capacitor Ccds has nowhere to dissipate, utilizing the characteristic that the voltage across a capacitor cannot change abruptly, a right plate voltage V3, integrating from 0V, is formed on the right plate of capacitor Ccds, effectively subtracting the V1 voltage from the previous stage and the noise voltage generated by the input noise. To meet impedance matching requirements with the subsequent stage, the right plate voltage of capacitor Ccds is connected to the positive input of the P-type input transistor quintuplet amplifier. The P-type input transistor quintuplet amplifier, configured as a unity-gain amplifier, transmits the integrated right plate voltage V3 to the subsequent stage. The voltage variations of V2 and V3 over time are shown in the graph below. Figure 5 As shown.

[0051] like Figure 6As shown, the subtractor internally consists of a two-stage amplifier and four resistors. The two-stage amplifier is constructed in a well-known manner and will not be described in detail here. The purpose of using a two-stage amplifier is to utilize its high gain to improve the linearity of the voltage subtraction result. The four resistors are R1, R2, R3, and R4 in the figure; since their resistance values ​​are all the same, the voltage V... A V B V O The relationship is: V O =V A -V B .

[0052] Example 1:

[0053] Different photocurrents were injected into 3×3 macropixel units to simulate various lighting conditions. Using Cadence Virtuoso software, simulations were performed following the aforementioned timing sequence. The summation effect of the convolution circuit within the macropixel unit on different currents and the corresponding changes in the integrated voltage were recorded to verify whether the macropixel unit affected or burdened the original imaging. Figure 7 As shown in the figure above, the simulation results clearly show that the photocurrent of each sub-pixel flows correctly to path A or B under the control of its SRAM, and the current is summed at the integration node according to Kirchhoff's laws. Figure 7 The figure below shows the relationship between the integral voltage and the final output. By comparing the integral output results of a traditional imaging unit with the same structure, the improved intra-pixel integration unit does not affect the imaging function.

[0054] Example 2:

[0055] Using Cadence Virtuoso software simulation and following the working method described in the specific implementation method, an original image is input into a 6×6 macropixel unit array, such as... Figure 8 As shown in Figure a, the detector current source parameters for mapping image pixel grayscale to photocurrent and inputting it to the circuit are configured. A convolution kernel value is set as shown in Table 2 below:

[0056] Table 2:

[0057]

[0058] This refers to the smoothing filtering function. It acquires the output voltage of the macro-pixel unit array, converts this voltage to grayscale values ​​using the ADC program in Matlab 2024b software, and reconstructs the grayscale image, such as... Figure 8 As shown in Figure b, at the same time Figure 8 The kernel 'a' was calculated using Matlab 2024b software. Figure 8 c. (Comparison) Figure 8 b and Figure 8 The imaging differences of c were not found to be significant in terms of grayscale contrast, which proves that the macropixel array can effectively perform convolution calculations.

[0059] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A near-infrared convolutional image sensor structure design, characterized in that, This includes a macro-pixel unit array and the corresponding readout circuit, wherein, The macro-pixel unit array includes multiple macro-pixel units, each macro-pixel unit is composed of n×n sub-pixels, and each sub-pixel integrates an infrared photodiode and a static random access memory. The static random access memory stores a weight value and controls the direction of the photocurrent output by the photodiode. The readout circuit includes a first integrator, a second integrator, and a subtractor. The first integrator and the second integrator receive photocurrents from each sub-pixel within the plurality of macropixel units, which are weighted and distributed, through a metal wire interconnection structure. The first integrator integrates the photocurrents received by the first integrator and the second integrator to generate a first sampling voltage and a second sampling voltage, respectively. The subtractor subtracts the first sampling voltage from the second sampling voltage and directly outputs a voltage signal representing the result of the n×n convolution operation.

2. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The static random access memory stores weight values ​​of logic "1" or "0", which correspond to the +1 and -1 weights in the convolution kernel, respectively.

3. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The metal wire interconnect structure is a convolutional global interconnect, and its topology layout is determined based on the convolution kernel size, sliding path, and sliding step size.

4. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The photocurrents flowing into the same integration node are automatically summed and accumulated in the analog domain based on Kirchhoff's current law.

5. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The first integrator and the second integrator have the same structure, both of which adopt a voltage-clamped correlated dual-sampling capacitor transimpedance amplifier structure.

6. The near-infrared convolutional image sensor structure design according to claim 5, characterized in that, Both the first and second integrators include two five-transistor amplifiers, two metal-oxide-metal capacitors, and two complementary MOS switches, and subtract the pre-stage fixed voltage and noise voltage through a correlated double sampling method.

7. The near-infrared convolutional image sensor structure design according to claim 6, characterized in that, The output of the first five-transistor amplifier is connected to one end of the first metal-oxide-metal capacitor through the first complementary MOS switch, and the other end of the first metal-oxide-metal capacitor is connected to the non-inverting input of the second five-transistor amplifier. The inverting input of the second five-transistor amplifier is shorted to its output, forming a unity-gain amplifier. One end of the second metal-oxide-metal capacitor is connected between the output of the first five-transistor amplifier and the first complementary MOS switch, and the other end is grounded through the second complementary MOS switch and simultaneously connected to the non-inverting input of the second five-transistor amplifier.

8. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The subtractor consists of a two-stage amplifier and four resistors of the same value, and performs a linear operation by subtracting the second sampling voltage from the first sampling voltage.

9. The near-infrared convolutional image sensor structure design according to claim 8, characterized in that, The first sampling voltage is connected to the inverting input of the secondary amplifier through a first resistor. The inverting input of the secondary amplifier is also connected to its output through a second resistor, and the output outputs the final voltage signal. The second sampling voltage is connected to the non-inverting input of the secondary amplifier through a third resistor. The non-inverting input of the secondary amplifier is also grounded through a fourth resistor.

10. The near-infrared convolutional image sensor structure design according to claim 1, characterized in that, The size n of the macro-pixel unit array is a positive integer greater than 1, which is suitable for convolution kernel operations of different sizes.

Images