A sensing and computing integrated multi-channel image sensor and feature extraction method
By combining global shutters and multiple parallel convolution operations in the image sensor, the convolution neural network operation within the sensor is realized, solving the problem of low data transmission and computing efficiency in traditional systems, and improving the speed and performance of image feature extraction .
Patent Information
- Application Number
- CN202311454493.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-01-19
AI Technical Summary
Traditional intelligent processing systems separate data acquisition, image processing and visual analysis, resulting in large storage space consumption, long data transmission time, increased computing power demand, and security risks in the data transmission process of edge equipment.
Design a sensor-computing integrated multi-channel image sensor, combining global shutters and multiple parallel convolution operations, and using the limited hardware resources inside the sensor to complete the first layer of convolutional neural network, reducing redundant data through multiplication and accumulation operations Transmission, realizes fast feature extraction.
It improves data transmission speed, reduces computing needs, avoids redundant data transmission, improves computing efficiency and performance, and provides a new breakthrough in machine vision.
Smart Images

Figure CN118138909B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of in-sensor computing, and in particular relates to a sensing-computing integrated multi-channel image sensor and a feature extraction method. Background Art
[0002] The rapid development and popularization of the Internet of Things has driven the massive growth of various sensing devices. Among them, machine vision systems that combine image sensors with artificial intelligence algorithms are widely used and play an important role in industries such as autonomous driving, environmental monitoring, and remote sensing images. However, traditional intelligent processing systems separate functions such as data acquisition, image processing, and visual analysis. The massive visual data captured by sensors poses a great challenge to communication, storage, and computing, resulting in a series of problems such as large consumption of storage space, extended data transmission time, and increased computing power requirements. In addition, edge devices also carry security risks of data leakage during data transmission.
[0003] In order to break through this bottleneck, scholars have proposed migrating computing units to sensing devices, or even designing them directly into image sensors, in order to simultaneously meet the requirements of data collection, processing, and analysis. For example, using 3D stacking technology, the processing chip is integrated under the sensor chip, directly reducing the distance of data transmission and thus reducing power consumption. For another example, using new materials and device processes to prepare photodiodes with adjustable sensitivity, the pixel array can be used as an artificial neural network. In addition, there are also sensing-storage-computing systems combined with in-memory computing and in-sensor computing architectures for binary networks.
[0004] Although existing research results have achieved certain breakthroughs, there are still problems such as immature technology, chip area occupation, complex circuit design, and low accuracy of binary networks. Summary of the invention
[0005] In order to make full use of the characteristics of CIS and realize fast image feature extraction, the present invention proposes an in-sensor calculation scheme combining global shutter and multiple parallel convolution operations. First, the image data is collected to the storage capacitor in the pixel at one time, and then the multi-channel feature map output is realized through multiple column-parallel data reading and calculation processes. The present invention has the advantages of fast feature extraction of "one exposure, multiple calculations" and also has the traditional camera function.
[0006] From photoelectric conversion, data acquisition to quantized output, the image sensor itself has the conditions to complete simple operations. The present invention uses the limited hardware resources inside the sensor to complete the first layer of convolution operations of the convolutional neural network (CNN). It only relies on simple multiplication and accumulation (MAC) operations to reduce redundant data transmission, bring about performance improvement, and has practical significance and good application prospects.
[0007] To achieve the above object, the technical solution adopted by the present invention is:
[0008] A sensing and computing integrated multi-channel image sensor, comprising a digital control unit, a pixel array circuit, an operation circuit and a column-level analog-to-digital converter;
[0009] The pixel array circuit includes a row-column gate and a plurality of pixel basic units; the pixel basic unit is used to convert the received light signal into a data current signal and a reset current signal; the row-column gate controls the output time of the data current signal and the reset current signal of each pixel basic unit according to the weight; the weight is generated by a digital control unit;
[0010] An operation circuit, used for performing multiplication and accumulation operation on the data current signal and the reset current signal output by the pixel array circuit, wherein the multiplication and accumulation operation is divided into two processes and performed successively to finally obtain positive and negative weighted integral results;
[0011] The column-level analog-to-digital converter is used to calculate the difference between the positive and negative weighted integration results to obtain the final integration result. All final integration results form a feature map and are converted into digital signal quantization output.
[0012] Furthermore, the pixel basic unit includes a reset unit and four sub-pixels with exactly the same structure; the reset unit is used to receive an external global reset signal and control the reset of four sub-pixels; the four sub-pixels are used to receive light signals from different channels respectively, and the signal lines used to output sub-pixel data current signals and sub-pixel reset current signals in the four sub-pixels are respectively connected to jointly output the data current signal and reset current signal of the pixel basic unit.
[0013] Furthermore, the pixel basic units are arranged in a matrix form, with 5 consecutive columns of pixel basic units forming a group, the first column and the last column in each group being shared with adjacent groups, and each column of pixel basic units outputting the sum of data current signals of the column through a data readout line, and outputting the sum of reset current signals of the column through a reset readout line; the data readout lines of the same group are connected together through switches, respectively, and the sum of data current signals of the group is output through a regulator unit, and the reset readout lines of the same group are connected together through switches, respectively, and the sum of reset current signals of the group is output through another regulator unit; the function of the regulator unit is to isolate the pixel basic unit from the operation unit and improve the linearity of the pixel current.
[0014] Furthermore, the row-column selector includes a row selector and a column selector, the column selector is used to control the switch corresponding to each column of pixel basic units, the row selector is a weight control line, any i-th row of pixel basic units is connected to the weight control lines Wi1, Wi2, Wi3, the 5 pixel basic units in the same group in the i-th row are connected to the weight control lines Wi1, Wi2, Wi3, Wi2, Wi1 in sequence, and the row-column selector includes the following operating steps: 1) the column selector closes the switches corresponding to the first three columns in each group of pixel basic units, the row selector selects three rows of pixel basic units, and the three groups of weight control lines of each row of pixel basic units transmit weights in sequence Where k = 1, 2, 3; They represent the weights of the kth row, 1st, 2nd, and 3rd columns in the 3×3 weight matrix respectively;
[0015] 2) Repeat step 1) with a step size of 2 to traverse all row pixel basic units;
[0016] 3) The column selector disconnects the switches corresponding to the first two columns in each group of pixel basic units and closes the switches corresponding to the last three columns. The row selector selects three rows of pixel basic units. The three groups of weight control lines of each row of pixel basic units transmit weights in turn.
[0017] 4) Repeat step 3) with a step size of 2 to traverse all row pixel basic units.
[0018] Furthermore, each group of weight control lines controls the output time of the data current signal and the reset current signal of the pixel basic unit according to the weights it transmits; each group of weight control lines includes weight control lines of four sub-pixels, and the weight control lines of the four sub-pixels control the output time of the data current signal and the reset current signal of the sub-pixels according to the sub-pixel weights.
[0019] Furthermore, the operation circuit includes a positive weight integration unit and a negative weight integration unit; the positive weight integration unit and the negative weight integration unit have the same structure, including capacitor CH, capacitor CL1, capacitor CL2, switch CHG, switch LK and two reset switches; capacitor CH and capacitor CL1 are connected in parallel through switch CHG, capacitor CL1 and capacitor CL2 are connected in parallel through switch LK, the negative electrodes of the three capacitors are grounded in common, the first reset switch is connected in parallel with capacitor CH, and the second reset switch is connected in parallel with capacitor CL1; the data current signal output end of the pixel array circuit is connected to the capacitor CH of the positive weight integration unit and the positive pole of capacitor CL1 through the high switch PH and the low switch PL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive pole of capacitor CL1 through the high switch NH and the low switch NL; the reset current signal output end is connected to the capacitor CH of the positive weight integration unit and the positive pole of capacitor CL1 through another pair of high switches NH and the low switch NL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive pole of capacitor CL1 through another pair of high switches PH and the low switch PL.
[0020] Furthermore, the multiplication and accumulation operation is divided into two processes of positive weight integration and negative weight integration, which are performed successively. The implementation steps of each process are the same, and are divided into high-order weight integration and low-order weight integration. The specific steps of positive weight integration are as follows:
[0021] a) Close all reset switches C_RST and switch LK to reset the voltage values of all capacitors to zero;
[0022] b) The high-order switch PH is closed, and the other switches are opened, so that the current signal of the input operation unit charges the capacitor CH, and the high-order weighted integral result is obtained;
[0023] c) The low-level switch PL and the switch LK are closed, so that the current signal of the input operation unit charges the capacitors CL1 and CL2 connected in parallel, and obtains the low-level weighted integral result;
[0024] d) The switch CHG is closed and the other switches are opened, so that the capacitor CH is connected to the two ends of the capacitor CL1, and the charge is redistributed to complete the positive weight integration process. Finally, the voltage on the capacitor CL1 of the positive weight integration unit is the positive weight integration result;
[0025] In the negative weight integration, the high switch is NH, the low switch is NL, and the other steps are the same as the positive weight integration. Finally, the voltage on the capacitor CL1 of the negative weight integration unit is the negative weight integration result.
[0026] Furthermore, the weights are all signed weights, which are divided into 1 sign bit, (nm-1) high-order weights, and m low-order weights. The capacitance ratio of the capacitors CH, CL1, and CL2 in the operation circuit is 2 m :1:(2m -1).
[0027] Furthermore, the column-level analog-to-digital converter includes a charging unit, a voltage comparator, and a counter. Each column-level analog-to-digital converter processes the positive and negative weighted integration results generated by each 5 columns of pixel basic units, obtains the difference between the positive and negative weighted integration results, and quantizes and outputs them as an N-bit binary number with a sign bit. The specific steps are:
[0028] First, the counter is cleared, and the comparator compares the positive and negative weighted integral results output by the operation unit and outputs the sign bit; the number of bits of the counter is (N-1) bits;
[0029] Next, the one with the lower voltage value in the positive and negative integral results is charged by the charging unit using 2 M The current is times higher than that of the first phase, and the high (NM-1) bit of the counter starts counting upward until the comparator output is reversed to obtain a coarse quantization result; M is a positive integer less than (N-1);
[0030] Finally, the other integration result is charged by the charging unit with 1 times current, and the counter counts downward as a whole until the comparator output is reversed to obtain a fine quantization result, which is the difference between the positive and negative weight integration results.
[0031] The present invention also proposes a feature extraction method based on the above-mentioned sensing and computing integrated multi-channel image sensor, comprising the following steps:
[0032] The pixel basic unit converts the received light signal into a data current signal and a reset current signal;
[0033] The digital control unit generates weights, and the row and column selectors control the output time of the data current signal and the reset current signal of each pixel basic unit according to the weights;
[0034] The operation circuit performs two-process multiplication and accumulation operations on the data current signal and the reset current signal output by the pixel array circuit, respectively obtaining positive and negative weighted integral results, and traverses all pixel basic units to obtain a series of corresponding positive and negative weighted integral results;
[0035] The column-level analog-to-digital converter calculates the difference between the positive and negative weighted integration results to obtain the final integration result. All final integration results form a feature map and are converted into a digital signal quantization output.
[0036] The beneficial effects of the present invention are:
[0037] 1. The present invention utilizes the limited hardware resources inside the image sensor to complete the first layer of convolution operation of the convolutional neural network (CNN), directly inputs the image features into the terminal host, eliminates the transmission of redundant data, and greatly improves the "perception wall" problem caused by the transmission speed of edge data failing to keep up with the computing power of the host, providing a new breakthrough for the development of machine vision.
[0038] 2. The present invention proposes a feature extraction method of "one exposure, multiple calculations", which uses a global shutter method to store the results of a single exposure of the pixel array, and implements column-parallel convolution operations through a unique convolution kernel multiplexing and sliding scheme to output multi-channel image features. This feature extraction method avoids the deficiency of other designs that require re-exposure for each calculation, improves the parallelism of calculations, and greatly improves energy efficiency;
[0039] 3. The present invention uses the time folding method, through weight division, charge redistribution, transformation of calculation logic and other means, to achieve the same accuracy of calculation result output in only a small part of the original calculation process, thereby improving the calculation efficiency. At the same time, this method is universal and can be expanded for various types of operation circuits. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic diagram of the overall circuit architecture of the present invention.
[0041] Figure 2 It is a circuit diagram of the pixel basic unit of the present invention.
[0042] Figure 3 It is a schematic diagram of the convolution operation flow of the present invention.
[0043] Figure 4 It is a circuit diagram of the multiplication and accumulation unit of the present invention.
[0044] Figure 5 1 is a schematic diagram of the circuit structure of the column-level analog-to-digital converter of the present invention.
[0045] Figure 6 It is a comparison diagram of the quantization output process of the present invention. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure.
[0047] like Figure 1The overall circuit architecture of the present invention is shown, including a pixel basic unit for realizing a global shutter, a pixel array circuit composed of a number of pixel basic units and row and column gates, a column-parallel multiplication and accumulation circuit and an analog-to-digital converter, and a digital control unit including a digital-to-time converter (DTC).
[0048] The present invention first collects image data into the storage capacitor in the pixel at one time, then loads the weight pulse to the pixel basic units in multiple convolution areas through the row and column selectors, performs column-parallel multiplication and accumulation operations and quantization output, and repeats this calculation process until the multi-channel feature map output of a frame of image is completed. Therefore, the present invention has the advantage of fast feature extraction of "one exposure, multiple calculations".
[0049] like Figure 2 As shown, the pixel basic unit includes a reset unit and four sub-pixels with exactly the same structure, the four sub-pixels represent four channels of R, G1, G2, and B respectively, usually the image is three channels of R, G, and B, and the G channel is obtained by taking the average value of G1 and G2; the reset unit includes a PMOS transistor RT and a capacitor CRT, the source of the PMOS transistor RT is connected to the reset voltage, the drain is connected to the positive electrode of the capacitor CRT, and the negative electrode of the capacitor CRT is grounded; the sub-pixel is composed of a photodiode PD, a capacitor CPD, and six PMOS transistors P1 to P6, the source and drain of the PMOS transistor P1 are connected to the positive electrode of the capacitor CPD and the negative electrode of the photodiode PD respectively, the PMOS transistor The source of the transistor P2, the positive electrode of the capacitor CRT and the gate of the PMOS transistor P5 are connected together, the drain of the PMOS transistor P2, the positive electrode of the capacitor CPD and the gate of the PMOS transistor P3 are connected together, the sources of the PMOS transistors P3 and P5 are connected to the power supply, the drains of the PMOS transistors P3 and P5 are connected to the sources of the PMOS transistors P4 and P6 respectively, the positive electrode of the photodiode PD and the negative electrode of the capacitor CPD are grounded; the gates of the PMOS transistors P1 and P2 are connected to the exposure control signal tg and the control signal gs of the global shutter; the drains of the PMOS transistors P4 and P6 of each sub-pixel are respectively connected together as the data readout line DATA and the reset readout line RESET.
[0050] The signal output principle of the pixel basic unit is as follows: the PMOS transistor RT controls the pixel basic unit to perform global reset, and the PMOS transistors P1 and P2 control the pixel basic unit to perform exposure. PD With reset voltage V RST The data is stored in capacitors CPD and CRT respectively. The read transistors P3 and P5 work in the linear region and convert the voltage into data current I PD , reset current I RSTOutput, according to the principle of photoelectric conversion and the linear region formula of MOS tube, it is related to the light intensity P in The following relations are satisfied:
[0051] (I PD -I RST )∝(V PD -V RST )∝P in
[0052] The basic principle of the multiplication and accumulation operation is as follows: the selection transistors P4 and P6 have the same control signal, and the four sub-pixels are controlled by the pulse width signals w1 to w4 generated by the weights. The pulse width signals w1 to w4 generated by the weights control the output time of the data current and reset current of the sub-pixels, and the data current and reset current of the multiple pixel basic units in the convolution area are accumulated on the two direct integration capacitors respectively. The subtraction of the integral voltage obtained by the multiplication and accumulation can realize the multiplication and accumulation operation of the weight and the image data.
[0053] Since the weights used in the present invention are both positive and negative, the overall operation process will be carried out in sequence according to the two processes of positive weight integration and negative weight integration. Specifically, for one of the direct integration capacitors, in the positive weight integration process, the data current is integrated using a positive weight, and in the negative weight integration process, the reset current is integrated using a negative weight; for the other direct integration capacitor, in the positive weight integration process, the reset current is integrated using a positive weight, and in the negative weight integration process, the data current is integrated using a negative weight; thereafter, the voltage values on the two direct integration capacitors are subtracted to obtain the final multiplication and accumulation result.
[0054] Taking the four sub-pixels in a basic pixel unit as an example, assuming that w1 and w2 are positive weights and w3 and w4 are negative weights, the final result is:
[0055] CV P =I PD1 w1+I PD2 w2+I RST3 (-w3)+I RST4 (-w4)
[0056] CV N =I RST1 w1+I RST2 w2+I PD (-w3)+I PD (-w4)
[0057] V o =V P -V N =I1w1+I2w2+I3w3+I4w4
[0058] Ik =I PDk -I RSTk (k=1,2,3,4)
[0059] Among them, the capacitance of the two direct integration capacitors is equal, C is the capacitance of the direct integration capacitor, V P 、V N are the current integration results of the two direct integration capacitors, V o is the final multiplication and accumulation result of the pixel basic unit, I PDk (k=1,2,3,4) and I RSTk (k=1, 2, 3, 4) are the data current and reset current output by the four sub-pixels respectively. The reset current is multiplied and accumulated here to eliminate the reset noise generated when the pixel basic unit is reset, so as to realize the correlated double sampling operation.
[0060] The above calculation is performed on the pixel basic units in the 3×3 window to obtain the multiplication and accumulation operation results of 9 pixel basic units, which correspond to the 3×3 weight matrix one by one, and the weighted sum is obtained to obtain the final multiplication and accumulation operation result.
[0061] like Figure 1 The pixel array circuit shown includes a plurality of pixel basic units and row and column selectors. The pixel basic units are arranged in a matrix form, with 5 consecutive columns of pixel basic units forming a group. The first column and the last column in each group are shared with the adjacent group. Each column of pixel basic units outputs the sum of the data current signals of the column through a data readout line and outputs the sum of the reset current signals of the column through a reset readout line. The data readout lines of the same group are connected together through switches S1 to S5 respectively, and the sum of the data current signals of the group is output through a voltage regulator unit. The reset readout lines of the same group are connected together through another group of switches S1 to S5 respectively, and the sum of the reset current signals of the group is output through another voltage regulator unit. The function of the voltage regulator unit is to isolate the pixel basic unit from the operation unit and improve the linearity of the pixel current.
[0062] The row and column selectors include a row selector and a column selector. The column selector is used to control switches S1 to S5 corresponding to each column of pixel basic units. The row selector is a weight control line. Any i-th row of pixel basic units is connected to weight control lines Wi1, Wi2, and Wi3. Five pixel basic units in the same group in the i-th row are connected to weight control lines Wi1, Wi2, Wi3, Wi2, and Wi1 in sequence. The present invention is directed to convolution operations with a convolution kernel size of 3×3 and a step size of 2. Through the control of the row and column selectors, every 3 rows are a group in the row direction, and every 3 columns are a group in the column direction. Convolution operations of multiple groups in the row direction and one group in the column direction can be implemented each time. Figure 3 This is a convolution operation flow diagram, according to the following 3×3 weight matrix:
[0063]
[0064] The row and column selector includes the following operation steps: 1) The column selector closes the switches corresponding to the first three columns in each group of pixel basic units, and the row selector selects three rows of pixel basic units. The three groups of weight control lines of each row of pixel basic units transmit weights in turn. Where k = 1, 2, 3; They represent the weights of the kth row, 1st, 2nd, and 3rd columns in the 3×3 weight matrix respectively;
[0065] 2) Repeat step 1) with a step size of 2 to traverse all row pixel basic units;
[0066] 3) The column selector disconnects the switches corresponding to the first two columns in each group of pixel basic units and closes the switches corresponding to the last two columns. The row selector selects three rows of pixel basic units. The three groups of weight control lines of each row of pixel basic units transmit weights in turn.
[0067] 4) Repeat step 3) with a step size of 2 to traverse all row pixel basic units.
[0068] Each group of weight control lines includes weight control lines of four sub-pixels, and the four sub-pixel weight control lines control the output time of the data current signal and the reset current signal of the sub-pixel according to the sub-pixel weight.
[0069] Traversing all pixel basic units, the matrix composed of the multiplication and accumulation results is the final feature map.
[0070] The circuit structure of multiplication and accumulation operation is as follows Figure 4 As shown, it includes two parts, a positive weight integration unit and a negative weight integration unit. The positive weight integration unit and the negative weight integration unit have the same structure, and both include a capacitor CH, a capacitor CL1, a capacitor CL2, a switch CHG, a switch LK and two reset switches; the capacitor CH is connected in parallel with the capacitor CL1 through the switch CHG, the capacitor CL1 is connected in parallel with the capacitor CL2 through the switch LK, the negative electrodes of the three capacitors are grounded in common, the first reset switch is connected in parallel with the capacitor CH, and the second reset switch is connected in parallel with the capacitor CL1; the data current signal output end of the pixel array circuit is connected to the capacitor CH of the positive weight integration unit and the positive electrode of the capacitor CL1 through the high switch PH and the low switch PL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive electrode of the capacitor CL1 through the high switch NH and the low switch NL; the reset current signal output end is connected to the capacitor CH of the positive weight integration unit and the positive electrode of the capacitor CL1 through another pair of high switches NH and the low switch NL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive electrode of the capacitor CL1 through another pair of high switches PH and the low switch PL.
[0071] The multiplication and addition operations are performed in sequence according to the two processes of positive weight integration and negative weight integration. The positive weight integration unit is used to receive the data current signal output by the row and column gate during the positive weight integration, and receive the reset current signal output by the row and column gate during the negative weight integration, and obtain the positive weight integration result by charging the capacitor for integration; the negative weight integration unit is used to receive the reset current signal output by the row and column gate during the positive weight integration, and receive the data current signal output by the row and column gate during the negative weight integration, and obtain the negative weight integration result by charging the capacitor for integration;
[0072] The positive weight integration result includes the data current integration result during positive weight integration and the reset current integration result during negative weight integration. The negative weight integration result includes the reset current integration result during positive weight integration and the data current integration result during negative weight integration. The role of the reset current integration is to offset the reset noise caused by the reset of the pixel basic unit when the positive and negative weight integration results are subsequently subtracted.
[0073] For n-bit signed weights, each process directly integrates, which costs a total of 2 n clock cycles. In order to improve the computational efficiency, the present invention utilizes the time folding method, and adopts weight division, charge redistribution and other means to divide each process into two steps: high-order weight integration and low-order weight integration, which greatly reduces the integration time. In this embodiment, the n-bit signed weight is divided into the highest sign bit, (nm-1)-bit high-order weight, and m-bit low-order weight; at the same time, a combination of capacitors CH, CL1 and CL2 is used instead of direct integration capacitors for accumulation.
[0074] Taking positive weight integral as an example, the specific steps are as follows:
[0075] a) Close all reset switches C_RST and switch LK to reset the voltage values of all capacitors to zero;
[0076] b) The high-order switch PH is closed, and the other switches are opened, so that the current signal of the input operation unit charges the capacitor CH, and the high-order weighted integral result is obtained;
[0077] c) The low-level switch PL and the switch LK are closed, so that the current signal of the input operation unit charges the capacitors CL1 and CL2 connected in parallel, and obtains the low-level weighted integral result;
[0078] d) Switch CHG is closed and the other switches are opened, so that the capacitor CH is connected to the two ends of the capacitor CL1, and the charge is redistributed to complete the positive weight integration process;
[0079] For negative weight integration, the high switch is NH and the low switch is NL. Finally, the voltage on the capacitor CL1 of the positive weight integration unit is the positive weight integration result, and the voltage on the capacitor CL1 of the negative weight integration unit is the negative weight integration result.
[0080] For n-bit signed weight, it is divided into 1 sign bit, (nm-1) high-bit weight and m low-bit weight. At the same time, the capacitance ratio of capacitors CH, CL1 and CL2 is 2 m :1:(2 m -1), the capacitance of the parallel capacitors CL1 and CL2 is the same as CH. By calculation, if 2m=n-1, the integration time is reduced the most compared with direct integration, which is only (2 n-m +2 m+1 ) clock cycles.
[0081] like Figure 5 The schematic diagram of the circuit structure of the column-level analog-to-digital converter of the present invention is shown, which includes the capacitor CL1 of the positive and negative weight integration units in the multiplication and accumulation circuit, the charging unit, the voltage comparator, the counter and the digital logic unit, and its function is to calculate the difference between the positive and negative weight integration results and convert it into a digital signal for quantization output. The column-level analog-to-digital converter designed by the present invention utilizes the basic principle of the ramp ADC. The specific steps of the traditional ramp ADC to quantize and output an N-bit binary number with a signed bit are as follows:
[0082] First, the (N-1)-bit counter is cleared, and the comparator performs a comparison to obtain the sign bit;
[0083] Secondly, the one with lower voltage value in the positive and negative integration results is charged by the charging unit with 1 times current, and at the same time, the (N-1)-bit counter starts counting up until the comparator output is reversed to obtain the final quantization result.
[0084] In order to improve the quantization efficiency, the present invention adopts the time folding method and divides the quantization process into three steps. The following are the specific steps of quantizing into an N-bit signed binary number:
[0085] First, the (N-1)-bit counter is cleared, and the comparator performs a comparison to obtain the sign bit;
[0086] Next, the one with the lower voltage value in the positive and negative integral results is charged by the charging unit using 2 M The current is times higher for charging, and the high (NM-1) bit of the counter starts counting upwards until the comparator output is reversed, and a coarse quantization result is obtained;
[0087] Finally, the other integration result is charged by the charging unit with 1 times current, and the counter counts downward as a whole until the comparator output is reversed to obtain a fine quantization result.
[0088] The charging unit and the counter are controlled by a digital logic unit, and the digital logic unit sends an IPH or INH signal to control the charging unit to adopt 2 M The IPL or INL signal controls the charging unit to charge with 1 times the current, and the UP and DOWN signals control the counter to count up and down.
[0089] The M is a positive integer less than (N-1), representing the bit width of the fine quantization result.
[0090] like Figure 6 The figure shows a timing comparison diagram of the direct quantization output process and the improved two-step quantization output process of the present invention. P 、V N are the voltage values on the capacitor CL1 of the multiplication and accumulation units respectively. If direct quantization is used, the voltage value is V N The other side will continue to charge with 1 times the current until the comparator completes the quantization output in reverse, which takes a total of 2 N-1 clock cycles. If the two-step quantization of the present invention is adopted, the voltage value is V N The party first adopts 2 M The other side is charged with 1 times the current until the comparator is reversed; the other side is charged with 1 times the current until the comparator is reversed to complete the quantization output. The total time for the two-step quantization is (2 N-M-1 +2 M ) clock cycles, the quantization speed is greatly improved in comparison.
[0091] The digital control unit of the present invention comprises a weight storage unit, a digital time converter (DTC), an operation state machine, etc., and its functions are to realize the generation and loading of weights and the operation timing control, and the operation timing control is the time pulse signal required by the output circuit. Among them, the digital time converter (DTC) converts the weight into a pulse width signal according to the positive and negative, high weight, and low weight of the weight.
[0092] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A sensing and computing integrated multi-channel image sensor, characterized in that: It includes a digital control unit, a pixel array circuit, an operation circuit and a column-level analog-to-digital converter; The pixel array circuit includes a row-column gate and a plurality of pixel basic units; the pixel basic unit is used to convert the received light signal into a data current signal and a reset current signal; the row-column gate controls the output time of the data current signal and the reset current signal of each pixel basic unit according to the weight; the weight is generated by a digital control unit; An operation circuit, used for performing multiplication and accumulation operation on the data current signal and the reset current signal output by the pixel array circuit, wherein the multiplication and accumulation operation is divided into two processes and performed successively to finally obtain positive and negative weighted integral results; The column-level analog-to-digital converter is used to calculate the difference between the positive and negative weighted integral results to obtain the final integral result. All the final integral results form a feature map and are converted into a digital signal for quantization output; The pixel basic units are arranged in a matrix form, with five consecutive columns of pixel basic units forming a group, the first column and the last column in each group being shared with adjacent groups, each column of pixel basic units outputting the sum of data current signals of the column through a data readout line, and outputting the sum of reset current signals of the column through a reset readout line; the data readout lines of the same group are connected together through switches, and the sum of data current signals of the group is output through a voltage regulator unit, and the reset readout lines of the same group are connected together through switches, and the sum of reset current signals of the group is output through another voltage regulator unit; The operation circuit includes a positive weight integration unit and a negative weight integration unit; the positive weight integration unit and the negative weight integration unit have the same structure, including a capacitor CH, a capacitor CL1, a capacitor CL2, a switch CHG, a switch LK and two reset switches; the capacitor CH and the capacitor CL1 are connected in parallel through the switch CHG, the capacitor CL1 and the capacitor CL2 are connected in parallel through the switch LK, the negative electrodes of the three capacitors are grounded in common, the first reset switch is connected in parallel with the capacitor CH, and the second reset switch is connected in parallel with the capacitor CL1; the data current signal output end of the pixel array circuit is connected to the capacitor CH of the positive weight integration unit and the positive electrode of the capacitor CL1 through the high switch PH and the low switch PL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive electrode of the capacitor CL1 through the high switch NH and the low switch NL; the reset current signal output end is connected to the capacitor CH of the positive weight integration unit and the positive electrode of the capacitor CL1 through another pair of high switches NH and the low switch NL, respectively, and is connected to the capacitor CH of the negative weight integration unit and the positive electrode of the capacitor CL1 through another pair of high switches PH and the low switch PL.
2. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The pixel basic unit includes a reset unit and four sub-pixels with exactly the same structure; the reset unit is used to receive an external global reset signal and control the reset of the four sub-pixels; the four sub-pixels are used to receive light signals from different channels respectively, and the signal lines used to output sub-pixel data current signals and sub-pixel reset current signals in the four sub-pixels are respectively connected to jointly output the data current signal and reset current signal of the pixel basic unit.
3. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The row and column selector includes a row selector and a column selector. The column selector is used to control the switches corresponding to each column of pixel basic units. The row selector is a weight control line. Any pixel basic unit in the i-th row is connected to the weight control lines Wi1, Wi2, and Wi3. The five pixel basic units in the same group in the i-th row are connected to the weight control lines Wi1, Wi2, Wi3, Wi2, and Wi1 in sequence. The row and column selector includes the following operating steps: 1) The column selector closes the switches corresponding to the first three columns in each group of pixel basic units, and the row selector selects three rows of pixel basic units. The three groups of weight control lines of each row of pixel basic units transmit weights in sequence. Where k = 1, 2, 3; They represent the weights of the kth row, 1st, 2nd, and 3rd columns in the 3×3 weight matrix respectively; 2) Repeat step 1) with a step size of 2 to traverse all row pixel basic units; 3) The column selector disconnects the switches corresponding to the first two columns in each group of pixel basic units and closes the switches corresponding to the last three columns. The row selector selects three rows of pixel basic units. The three groups of weight control lines of each row of pixel basic units transmit weights in turn. 4) Repeat step 3) with a step size of 2 to traverse all row pixel basic units.
4. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: Each group of weight control lines controls the output time of the data current signal and the reset current signal of the pixel basic unit according to the weight it transmits; each group of weight control lines includes weight control lines of four sub-pixels, and the weight control lines of the four sub-pixels control the output time of the data current signal and the reset current signal of the sub-pixels according to the sub-pixel weights.
5. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The multiplication and accumulation operation is divided into two processes, positive weight integration and negative weight integration, which are carried out in sequence. The implementation steps of each process are the same, and are divided into high-order weight integration and low-order weight integration. The specific steps of positive weight integration are as follows: a) Close all reset switches C_RST and switch LK to reset the voltage values of all capacitors to zero; b) The high-order switch PH is closed, and the other switches are opened, so that the current signal of the input operation unit charges the capacitor CH, and the high-order weighted integral result is obtained; c) The low-level switch PL and the switch LK are closed, so that the current signal of the input operation unit charges the capacitors CL1 and CL2 connected in parallel, and obtains the low-level weighted integral result; d) The switch CHG is closed and the other switches are opened, so that the capacitor CH is connected to the two ends of the capacitor CL1, and the charge is redistributed to complete the positive weight integration process. Finally, the voltage on the capacitor CL1 of the positive weight integration unit is the positive weight integration result; In the negative weight integration, the high switch is NH, the low switch is NL, and the other steps are the same as the positive weight integration. Finally, the voltage on the capacitor CL1 of the negative weight integration unit is the negative weight integration result.
6. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The weights are all signed weights, which are divided into 1 sign bit, (nm-1) high-order weights, and m low-order weights. The capacitance ratio of the capacitors CH, CL1, and CL2 in the operation circuit is 2 m :1:(2 m -1).
7. The sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The column-level analog-to-digital converter includes a charging unit, a voltage comparator, and a counter. Each column-level analog-to-digital converter processes the positive and negative weighted integration results generated by each 5 columns of pixel basic units, obtains the difference between the positive and negative weighted integration results, and quantizes and outputs them as an N-bit binary number with a sign bit. The specific steps are as follows: First, the counter is cleared, and the comparator compares the positive and negative weighted integral results output by the operation unit and outputs the sign bit; the number of bits of the counter is (N-1) bits; Next, the one with the lower voltage value in the positive and negative integral results is charged by the charging unit using 2 M The current is times higher than that of the first phase, and the high (NM-1) bit of the counter starts counting upward until the comparator output is reversed to obtain a coarse quantization result; M is a positive integer less than (N-1); Finally, the other integration result is charged by the charging unit with 1 times current, and the counter counts downward as a whole until the comparator output is reversed to obtain a fine quantization result, which is the difference between the positive and negative weight integration results.
8. A feature extraction method based on the sensing and computing integrated multi-channel image sensor according to claim 1, characterized in that: The following steps are involved: The pixel basic unit converts the received light signal into a data current signal and a reset current signal; The digital control unit generates weights, and the row and column selectors control the output time of the data current signal and the reset current signal of each pixel basic unit according to the weights; The operation circuit performs two-process multiplication and accumulation operations on the data current signal and the reset current signal output by the pixel array circuit, respectively obtaining positive and negative weighted integral results, and traverses all pixel basic units to obtain a series of corresponding positive and negative weighted integral results; The column-level analog-to-digital converter calculates the difference between the positive and negative weighted integration results to obtain the final integration result. All final integration results form a feature map and are converted into a digital signal quantization output.
Citation Information
Patent Citations
Convolution operation architecture in CMOS image sensor
CN111669527A
Image sensing, storing and calculating integrated pixel unit based on floating gate device and pixel array
CN112601037A
Light-sensing-calculation integrated display unit, display array and operation method and device thereof
CN112949848A
Focal plane detector reading circuit for realizing convolution calculation
CN113489925A
Image sensor integrated with convolutional neural network operational circuit
CN116266889A
