Image sensor integrating convolutional neural network operation circuit

Abstract

An image sensor integrating a convolutional neural network operation circuit, comprising: a pixel array including a plurality of pixels divided into a plurality of pixel groups, each pixel converting a light signal into a first pulse width modulation signal; a convolution operation circuit controlling the on-time of a corresponding weight current according to the first pulse width modulation signal of each pixel in each pixel group, and accumulating the weight currents of the plurality of pixels into an integration current; a comparison circuit converting the integration current into a second pulse width modulation signal and comparing it with an adjacent pixel group, and outputting the second pulse width modulation signal with a larger value; and a classification circuit quantizing the second pulse width modulation signal into a quantized value according to the weight of a full connection layer node corresponding to each pixel group, accumulating the quantized values of all pixel groups into a feature value, and comparing the feature value with a feature threshold to obtain a classification result.

Description

Technical Field

[0001] This disclosure relates to an image sensor, and more particularly to an image sensor that integrates convolutional neural network operation circuitry. Background Technology

[0002] With the advanced research and development of convolutional neural networks (CNNs), artificial intelligence (AI) has achieved increasing success in various applications, such as image recognition and image classification. However, complex network training and computation require hardware resources such as field-programmable gate arrays (FPGAs), central processing units (CPUs), and graphics processing units (GPUs) to support the massive computational workload. The large amount of data access and the multiply-accumulate (MAC) operations in the algorithms result in considerable energy consumption and data transmission latency.

[0003] Therefore, power limitations, computing power, and local data storage are key challenges in bringing intelligent networks to Internet of Things (IoT) devices.

[0004] Public content

[0005] This disclosure proposes an image sensor that integrates convolutional neural network operations, enabling image sensing, convolutional neural network operations, and face or feature discrimination to be performed in a single image sensing chip.

[0006] This disclosure provides an image sensor integrating convolutional neural network operations, comprising a pixel array, a convolution operation circuit, a comparison circuit, and a classification circuit. The pixel array includes multiple pixels divided into multiple pixel groups, and each pixel converts the received light signal into a first pulse width modulation (PWM) signal. The convolution operation circuit controls the on-time of the corresponding weight current according to the first PWM signal of each pixel in each pixel group, and accumulates the weight currents of multiple pixels into an integral current, where the value of the weight current corresponds to the weight of the convolutional layer node in the trained convolutional neural network. The comparison circuit converts the integral current into a second PWM signal, compares the second PWM signals of adjacent pixel groups, and outputs the second PWM signal with the larger value. The classification circuit quantizes the second PWM signal of the pixel group into one of multiple quantized values ​​according to the weight of the fully connected layer node corresponding to each pixel group in the trained convolutional neural network, accumulates the quantized values ​​of all pixel groups into feature values, and compares them with a feature threshold to obtain a classification result.

[0007] To make this disclosure more apparent and understandable, specific embodiments are described below, along with detailed descriptions in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating the execution concept according to an embodiment of the present disclosure.

[0009] Figure 2 This is an architectural diagram of an image sensor integrating convolutional neural network operations, illustrated according to an embodiment of the present disclosure.

[0010] Figure 3 This is an operation flowchart of the image sensor 20 in different modes according to an embodiment of the present disclosure.

[0011] Figure 4 This is a circuit diagram of a pixel circuit according to an embodiment of the present disclosure.

[0012] Figure 5 This is a circuit diagram illustrating a convolution operation circuit and a maximum pooling operation circuit according to an embodiment of the present disclosure.

[0013] Figure 6 This is a circuit diagram of a weight generation circuit according to an embodiment of the present disclosure.

[0014] Figure 7 This is a schematic diagram illustrating a max pooling operation according to an embodiment of the present disclosure.

[0015] Figures 8A to 8F This is a circuit diagram and corresponding timing diagram of a maximum pooling operation circuit drawn according to an embodiment of the present disclosure.

[0016] Figure 9A and Figure 9B These are circuit diagrams and corresponding timing diagrams of the comparison circuit and the classification circuit drawn according to an embodiment of the present disclosure.

[0017] Symbol Explanation

[0018] 10: Feature Image Dataset

[0019] 12: Convolutional Neural Networks

[0020] 20: Image sensor

[0021] 21: Pulse Width Modulation Pixel Array

[0022] 22: Sensor internal processing circuit

[0023] 221, 50: Analog convolution operation circuit

[0024] 222: Activation function operation circuit

[0025] 223, 60: Max Pooling Operation Circuit

[0026] 224: Analog-to-Digital Converter Circuit

[0027] 225: Fully Connected Digital Operational Circuit

[0028] 23: Digital control circuit

[0029] 24: Horizontal control circuit

[0030] 25: Column Selection Circuit

[0031] 26: Circuit for temporary storage and processing of convolution operation weights

[0032] 27: Fully Connected Operation Weight Temporary Storage and Processing Circuit

[0033] 40: Pixel Circuit

[0034] 52: Sign Bit Logic

[0035] 54: Switching Current Integrator Unit

[0036] 70: Weight Generation Circuit

[0037] 72: Buffer

[0038] 74: Current-to-Analog Converter

[0039] 92: Quantization counter

[0040] 94: Decoder

[0041] 96: Reversible Counter

[0042] A, B, C: Process

[0043] A1, B1, B2, C1, C2, C3, C4: Steps

[0044] A1: Total Switch

[0045] ADD: Add signal

[0046] AN: AND gate

[0047] B2T FC Quantized value

[0048] CLK Load CLK FC_QZ CLK FC_SUM Frequency signal

[0049] CLK FC_CNT Quantization frequency signal

[0050] CLK FCE Pulse signal

[0051] CM E CMo: Central capacitor

[0052] CMP: Comparator

[0053] CMP EN,MODE : mode signal

[0054] CNT FC Number of pulses

[0055] CSEL FC Selection signal

[0056] COL FC Fully connected selection signal

[0057] C R Integrating capacitor

[0058] DFF, FF1~FF9: Triggers

[0059] FCE: Fully Connected Component Signal

[0060] I B I: Current

[0061] I P Positive weighted correlation current

[0062] I N Negative weighted related current

[0063] INT, MCN, MCP, MRD, MRST, SCI E SCIO SCI RST SUB E SUB O :switch

[0064] MP0~MP20: OR gate

[0065] MUX: Multiplexer

[0066] ph: photocurrent

[0067] PD: Light sensor

[0068] PIX_RST: Reset signal line

[0069] PW, PW MP Pulse width modulation signal

[0070] P: Filter weights

[0071] RAMP1, RAMP2: Ramp Generators

[0072] RAMP E RAMP O SUB E SUB O :Signal

[0073] RSEL, RSELB: Select signal lines

[0074] QZ0~QZ20: AND gate

[0075] S1~S4: Weight switches

[0076] SIGN, SIGN FC : Sign bit signal

[0077] SPA(0,0), SPA(0,1), SPA(10,0), SPA(1,1): Pixel group

[0078] SUB: Decrease Signal

[0079] VB, V PD V PW :Voltage

[0080] VIN, VIP: Input end

[0081] VP E V P Positive integral voltage

[0082] VN E V N Negative integral voltage

[0083] VR, V REFReference voltage

[0084] UDC: Reversible Counter

[0085] W1, W2, W4: Weighted bit signals

[0086] ZERO: Zero position signal Detailed Implementation

[0087] This disclosure proposes a computationally executed, trained, and validated convolutional neural network (CNN) that can perform image sensing, CNN computation, and face or feature discrimination (such as face detection) within a single image sensing chip. It can also provide multi-mode output, including the original image and the image from the CNN computation. Specifically, this disclosure uses pulse-width modulation (PWM) pixels as photosensitive elements, executing the necessary computations of the complete CNN in a vertically parallel circuit to output face or feature discrimination results. Therefore, it solves the problems of frame rate reduction, increased power consumption, and slow discrimination speed caused by data transfer between chips.

[0088] Figure 1 This is a schematic diagram illustrating the execution concept according to an embodiment of the present disclosure. Please refer to... Figure 1 In this embodiment, a convolutional neural network 12 is trained and validated using a dataset 10 of feature images to be distinguished, such as face and non-face images (including 3x3 convolution operations, 2x2 max pooling operations, fully connected operations, etc.). This generates convolution operation weights and fully connected operation weights, thereby creating a customized convolutional neural network 12. The weights and operations of this convolutional neural network 12 can be implemented on the image sensor described below, enabling it to independently perform image capture and face feature discrimination.

[0089] Figure 2 This is an architectural diagram of an image sensor integrating convolutional neural network operations according to an embodiment of this disclosure. Please refer to... Figure 2In this embodiment, the image sensor 20 integrates image sensing, convolutional neural network operations, and feature discrimination onto a single image sensing chip. It mainly includes a pulse-width modulation pixel array (hereinafter referred to as the pixel array) 21, a vertical parallel sensor-internal computation circuit 22, and peripheral control circuits (including a digital control circuit 23, a row control circuit 24, a column selection circuit 25, a convolution operation weight storage and processing circuit 26, and a fully connected operation weight storage and processing circuit 27). The pixel array 21, for example, has 128×128 pixels and is responsible for sensing environmental images and capturing light signals. The sensor-internal computation circuit 22 can implement a complete convolutional neural network, including, for example, a 3x3 analog convolution operation circuit (hereinafter referred to as the convolution operation circuit) 221 with a stride of 3, a ReLU activation function operation circuit 222 performing linear rectification, a 2x2 max pooling operation circuit (i.e., a comparison circuit) 223 with a stride of 2, an analog-to-digital converter circuit 224, and a fully connected digital operation circuit (i.e., a classification circuit) 225.

[0090] Figure 3 This is a flowchart illustrating the operation of the image sensor 20 in different modes according to an embodiment of this disclosure. Please refer to... Figure 3 The pixel array data output from pixel array 21 is processed through process A (including step A1) and then through analog-to-digital converter circuit 224 to output a general image. Process B (including steps B1 and B2) outputs a convolutional image through analog convolution operation circuit 221 and analog-to-digital converter circuit 224. Process C (including steps C1, C2, C3, and C4) completes a full convolutional neural network and outputs the operation results for face or feature discrimination.

[0091] In one embodiment, the image sensor 20 mainly includes a pixel array 21, a convolution operation circuit 221, a max pooling operation circuit 223, and a fully connected operation circuit 224. The pixel array 21 includes multiple pixels divided into multiple pixel groups, and each pixel converts the received light signal into a first pulse width modulation (PWM) signal. The convolution operation circuit 221 controls the on-time of the corresponding weight current according to the first PWM signal of each pixel in each pixel group, and accumulates the weight currents of multiple pixels into an integral current. The value of the weight current corresponds to the weight of the convolutional layer node in the trained convolutional neural network. The max pooling operation circuit 223 converts the integral current into a second PWM signal and compares it with the second PWM signals of adjacent pixel groups, thereby outputting a second PWM signal with a larger value. The fully connected operation circuit 224 quantizes the second PWM signal of the pixel group into one of multiple quantized values ​​according to the weight of the fully connected layer node corresponding to each pixel group in the trained convolutional neural network, accumulates the quantized values ​​of all pixel groups into feature values, and compares them with a feature threshold to obtain a classification result. The number of pixels in each pixel group is determined, for example, by the size of the kernel used by the trained convolutional neural network, such as 3×3, but is not limited thereto. The pixel array 21, convolution operation circuit 221, max pooling operation circuit 223, and fully connected operation circuit 224 in the image sensor 20 will be described in detail below with examples.

[0092] In one embodiment, the circuitry for each pixel in the pixel array 21 includes, for example, a light sensor, an intra-pixel comparator, a pixel reset switch, and an output selection switch. For instance, Figure 4 This is a circuit diagram of a pixel circuit according to an embodiment of the present disclosure. Please refer to... Figure 4 The pixel circuit 40 includes a light sensor PD, an intra-pixel comparator (including switches MCP and MCN), an output selection switch MRD, and a pixel reset switch MRST. The light sensor is, for example, a photodiode. Switch MCP has a control terminal connected to a reference voltage VR and a first terminal connected to the selection signal line RSEL. Switch MCN has a first terminal connected to a second terminal of switch MCP, a second terminal connected to a first terminal of the light sensor PD and a ground terminal, and a control terminal connected to the second terminal of the light sensor PD. During the exposure stage, the value of voltage VPD is determined by the photocurrent ph sensed by the photodiode PD from the initial voltage VPD. <rst>Discharge begins. After an exposure period, the voltage VPD is read out by an intra-pixel comparator based on a ramp reference voltage using pulse-width modulation conversion. The pixel reset switch MRST has a control terminal connected to the reset signal line PIX_RST, a first terminal connected to the second terminal of the photosensitive sensor PD, and a second terminal connected to the second terminal of the switch MCP and the first terminal of the switch MCN, and is configured to selectively turn on the first and second terminals according to the reset signal of the reset signal line PIX_RST. The output selection switch MRD has a control terminal connected to the second terminal of the switch MCP and the first terminal of the switch MCN, a first terminal connected to the selection signal line RSELB, and a second terminal connected to the signal output line, and is configured to output a pulse-width modulated signal PW converted from the optical signal. <m>For the same column (e.g., Col...) <n>In this embodiment, three pixels along the row direction can output three pulse width modulation signals PW at a time. <m-1>、PW <m>PW<m+1> .

[0093] In one embodiment, the convolution operation circuit 221 includes multiple column convolution operation circuits corresponding to multiple columns of pixels in a pixel group. Taking a pixel group of size 3×3 pixels as an example, the convolution operation circuit 221 includes three column convolution operation circuits corresponding to three columns of pixels respectively. Each column convolution operation circuit includes multiple sign bit logic, multiple switch-current integration (SCI) units, and integration circuits.

[0094] The symbol bit logic receives the first pulse width modulation signal of each pixel and gates the first pulse width modulation signal according to the symbol bit signal and zero bit signal in the weight correlation signal corresponding to each pixel.

[0095] In detail, when the image sensor 20 starts operating, it loads the nine weights of a 3×3 core, represented by five-bit signals (W1, W2, W4, SIGN, ZERO), into a buffer over nine frequency cycles. During the switching current integration operation, the image sensor 20 uses three of these signals (W1, W2, W4) to control the array-shared current digital-to-analog converter (DAC) at the correct current level, and uses the remaining two bits (SIGN, ZERO) for the sign bit logic. For example, the sign bit logic outputs a signal that selects the first pulse width modulation signal when the zero bit signal (ZERO) is 1, and outputs a signal that does not select the first pulse width modulation signal when the zero bit signal is 0. Furthermore, the sign bit logic outputs a signal that selects addition (ADD) when the sign bit signal (SIGN) is 1, and outputs a signal that selects subtraction (SUB) when the sign bit signal is 0.

[0096] The switching current integration unit receives the first pulse width modulation signal that is logically selected by the sign bit, and controls the turn-on time of the weight current corresponding to the pixel according to the weight bit signal in the weight correlation signal corresponding to each pixel.

[0097] The integrator circuit includes at least one integrating capacitor to accumulate the weighted current output from all the switching current integration units to generate an integrated voltage.

[0098] In one embodiment, each switching current integration unit further includes a positive switch and a negative switch. When the sign bit signal in the received weighted correlation signal is positive, the switching current integration unit will close the negative switch and open the positive switch to output the weighted current via the positive switch; when the sign bit signal in the received weighted correlation signal is negative, the switching current integration unit will close the positive switch and open the negative switch to output the weighted current via the negative switch. Furthermore, the aforementioned integration circuit includes a first-side circuit and a second-side circuit. The first-side circuit includes a first integrating capacitor for accumulating the weighted current output from all positive switches to generate a positive integrating voltage, while the second-side circuit includes a second integrating capacitor for accumulating the weighted current output from all negative switches to generate a negative integrating voltage.

[0099] For example, Figure 5 This is a circuit diagram illustrating a convolution operation circuit and a max-pooling operation circuit according to an embodiment of this disclosure. Please refer to... Figure 5 This embodiment includes a convolution operation circuit 50 and a maximum pooling operation circuit 60. The left side shows the filter weights P corresponding to each pixel in a 3×3 pixel group. <x>From top left to bottom right, they are P <1> ~P <9> It can be divided into the zero bit signal ZERO<1:9>, the sign bit signal SIGN<1:9>, and the weight bit signal W. 1,2,4 <1:9>, where the zero bit signal ZERO<1:9> and the sign bit signal SIGN<1:9> are respectively input to the three column convolution operation circuits COL of the right-side convolution operation circuit 52. <n-1>~COL<n+1> The sign bit logic; the weighted bit signal W 1,2,4 <1:9> is then converted into a voltage signal VB<1:9> via 9 sets of current-to-analog converters (IDAC) 74, and input to the three column convolution operation circuits COL of the convolution operation circuit 52 on the right. <n-1>~COL<n+1> The switching current integration (SCI) unit in the middle.

[0100] In detail, Figure 6 This is a circuit diagram of a weight generation circuit according to an embodiment of the present disclosure. Please refer to... Figure 6 The weight generation circuit 70 in this embodiment includes nine 5-bit flip-flops FF1 to FF9, a buffer 72, and multiple current digital-to-analog converters (IDACs) 74.

[0101] Flip-flops FF1 through FF9 are connected in series. The input of the first flip-flop FF1 receives weighted correlation signals (including SIGN, ZERO, W1, W2, and W4). The inputs of the other flip-flops FF1 through FF8 are connected to the output of the preceding flip-flop. The frequency inputs of flip-flops FF1 through FF9 receive the frequency signal CLK. Load .

[0102] Buffer 72 temporarily stores the zero bit signal ZERO<1:9>, the sign bit signal SIGN<1:9>, and the weight bit signal W from the weight correlation signals output by flip-flops FF1 to FF9. 1,2,4 <1:9>. Nine current-to-analog converters (IDACs) 74 respectively receive the pixel weight bit signals W temporarily stored in buffer 72. 1,2,4 <1:9>. Each current-to-analog converter includes multiple weighted switches S1 to S4 and a summing switch A1. Weighted switches S1 to S4 have a first terminal connected to each other and a second terminal connected to each other and grounded. The areas of the weighted switches S1 to S4 have a preset ratio (e.g., ...). Figure 6 As shown in the 1:1:2:4 ratio, the weight switches S1 to S4 are controlled at the control terminal according to the weight bit signal W. 1,2,4 When <1:9> is enabled, a current with this preset ratio is conducted (e.g., ...). Figure 6 The I shown B I B 2×I B 4×I B The summing switch A1 has a first terminal connected to the supply voltage, a second terminal connected to the first terminal of the weighting switches S1 to S4, and a control terminal, wherein the voltage VB<1:9> at the control terminal corresponds to the weighting current (I) flowing through the weighting switches S1 to S4. B I B 2*I B , 4*I B The sum of ).

[0103] In this embodiment, the convolution operation circuit performs convolution operations by simultaneously activating three rows ( <m-1> 、 <m>,<m+1> To select a 3x3 pixel subarray, output adjacent columns ( <n-1> 、 <n>,<n+1> The output of these 9 pixels will be multiplied by 9 weighted currents and integrated over the integrating capacitor CM. E This is used to complete the convolution (MAC) operation.

[0104] With the first <n>Taking the column convolution operation circuit as an example, when the sign bit signal SIGN... <m>When = 1 or 0, three signal correlation pulses (PW) from the three selected rows <m>First, the three groups of sign bits of the logic 52 are gated according to positive / negative weights to add (ADD). <m>) / subtract(SUB) <m>) signal. Simultaneously, in the switching current integrator 54, the voltage VB <m>Bias weighting current I P <m> / I N <m>It is turned on, and in the integrating capacitor CM E The cumulative sum of the left and right sides of the column is COL. <n>Integral current I P <n> / I N <n>.

[0105] In the integrating capacitor CM E The accumulated charge on is due to the addition in the time domain (ADD) <m>) / subtract(SUB) <m>The signal and weighted current determine the decision. This is achieved by simultaneously activating two adjacent columns (COL). <n-1>and COL<n+1> The column convolution operation circuit, with positive weight correlation current (I) P <n-1> 、I P <n>I P <n+1> ) and negative weighted related current (I N <n-1> 、I N <n>I N <n+1> The two numbers are added together to form I. P and I N And integrate them respectively on the integrating capacitor CM. E On both sides, thus realizing MAC operations of 3×3 pixel subarrays and 3×3 kernels.

[0106] After the MAC operation using SCI, the integrated voltage V across the integrating capacitor CM is... P and V N These can represent the positive and negative results of the simulated convolution, respectively. In one embodiment, the maximum pooling operation circuit uses a voltage comparator to compare the positive and negative integral voltages to output the convolution result, and uses judging logic (JG) to check the signal polarity (POL) of the convolution result, that is, to check whether the integral voltage V is positive. P Greater than the negative integral voltage V N Specifically, the judgment logic can control the integrator circuit to save the voltage difference between the positive and negative integral voltages when the convolution result is that the positive integral voltage is greater than the negative integral voltage, and can control the integrator circuit to reset the positive and negative integral voltages when the convolution result is that the positive integral voltage is less than the negative integral voltage.

[0107] In addition, the maximum pooling operation circuit also includes a first ramp circuit connected to the first side circuit of the integrator circuit and a second ramp circuit connected to the second side circuit of the integrator circuit.

[0108] For example, when the signal polarity = 1 (i.e., V... P >V N When ), the negative integral voltage V N It will be controlled by the signal SUB and connected to the integration start voltage V. REF The positive integral voltage V P Simultaneously, it will shift downwards by V. N Voltage quantity, completing the subtraction of positive and negative integral voltages (V) P -V N On the other hand, in order to implement the ReLU activation function operation, if the input is negative (i.e., V...), P -V N <0), positive and negative integral voltage V N With V P It will be controlled by the signal SUB and connected to the integration start voltage V. REF Reset the positive and negative integral voltage results.

[0109] Using the architecture described above, the ReLU function can be easily implemented by quantizing only the positive convolution results based on the signal polarity and ignoring the negative convolution results.

[0110] For example, Figure 7 This is a schematic diagram illustrating a max pooling operation according to an embodiment of the present disclosure. Figures 8A to 8F This is a circuit diagram and corresponding timing diagram of a maximum pooling operation circuit according to an embodiment of this disclosure. Please refer to... Figure 7 This embodiment uses 3×3 pixels as pixel groups to illustrate the process of performing convolution and max pooling operations on four adjacent pixel groups SPA(0,0), SPA(0,1), SPA(10,0), and SPA(1,1).

[0111] First, for the pixel group SPA(0,0) in even-numbered rows (row 0), please also refer to... Figure 7 and Figure 8A In the process Figure 5 After the convolution operation circuit 50 performs its operation, the integrated current output by it will accumulate in capacitor C through the switch INT. R And by turning on the SCI switch E In the central capacitor CM E The cumulative positive and negative integral voltages VP on both sides E and VN E Positive and negative integral voltage VP E and VN E Input VIP and VIN to the comparator CMP respectively, and then use the comparator CMP to determine VP. E >VN E Please refer to this. Figure 8B At this point, logic 62 and 64 output the signal SUB based on the result of comparator CMP. E To control the connection to the reference voltage V REF SUB switch E Conduction, while the central capacitor CM E The left side stores the complete convolution operation result.

[0112] Next, for the pixel group SPA(1,0) in the odd-numbered rows (row 1), please also refer to... Figure 7 and Figure 8C In the process Figure 5 After the convolution operation circuit 50 performs its operation, the integrated current output by it will accumulate in capacitor C through the switch INT. R And by turning on the SCI switch O In the central capacitor CM O The cumulative positive and negative integral voltages VP on both sides O and VN O Positive and negative integral voltage VP O and VN O Input VIP and VIN to the comparator CMP respectively, and then use the comparator CMP to determine VP. E <VN E Please refer to this. Figure 8D At this point, logic 62 and 64 output the signal SUB based on the result of comparator CMP. O To control the connection to the reference voltage V REF SUB switches on both sides O Turn on, and the positive and negative integral voltages VP will be turned on. O and VN O Simultaneously reset to the integration starting point, which means the complete convolution operation result can be considered as 0.

[0113] Then, perform the max pooling operation, please refer to the following: Figure 7 and Figure 8E In this mode, switch INT is turned off to stop receiving integrated current, and judgment logics 62 and 64 output signal SUB based on the judgment result of comparator CMP. E and RAMP E To target the central capacitor CM E The control is connected to the reference voltage V. REF The right-side switch SUB E Turn on and control the left switch RAMP connected to the ramp generator RAMP1. E The circuit is turned on, causing the accumulated capacitance CM to be turned on. E The integral voltage VP on the left E Input the comparator CMP to its input terminal VIP. On the other hand, judgment logics 62 and 64 also output the signal SUB based on the judgment result of comparator CMP. O and RAMP O To target the central capacitor CM O The control is connected to the reference voltage V. REF SUB switches on the left and right sides O The circuit is turned on, and the energy accumulated in the central capacitor CM will be transferred. O The integral voltage VP on the left O Reset and connect to the input VIN of comparator CMP.

[0114] Please refer to the following at the same time Figure 7 and Figure 8F The comparator CMP compares the voltages at its input terminals VIP and VIN, and outputs a pulse-width modulated signal PW. MP <j>< / j> Specifically, the result of performing convolution and ReLU operations on the pixel group SPA(0,0) in even rows (row 0) is 0.7, while the result of performing convolution and ReLU operations on the pixel group SPA(1,0) in odd rows (row 1) is 0. Therefore, the pulse width modulation signal PW output by the comparator CMP is... MP <j>< / j> The result is 0.7. On the other hand, the result of performing convolution and ReLU operations on the adjacent pixel group SPA(0,1) in the row direction for even-numbered rows (row 0) is 0, while the result of performing convolution and ReLU operations on the adjacent pixel group SPA(1,1) in the row direction for odd-numbered rows (row 1) is 0.2. Therefore, the pulse width modulation signal PW output by the comparator CMP is... MP<j+1> The value is 0.2. Two pulse width modulation signals PW MP <j>< / j> and PW MP<j+1> The maximum pooling result (0.7) can be obtained by taking a pulse width modulation signal with a longer pulse width through an OR gate.

[0115] In detail, the comparison circuit includes, for example, an OR gate, for taking the second pulse width modulation signal with the longest pulse width among the second pulse width modulation signals of adjacent pixel groups in the row direction, and outputting it to the classification circuit.

[0116] In one embodiment, the classification circuit includes multiple column classification circuits, a reversible counter (up-down counter), and a feature comparator. Each column classification circuit corresponds to multiple columns of pixels in two adjacent pixel groups along the row direction within a pixel group, and includes an AND gate, a quantization counter, and a decoder. The AND gate takes as input the zero-bit signal from the weight correlation signal of the fully connected layer node, the second pulse width modulation signal output by the comparator circuit, and the frequency signal, and outputs a non-zero quantized frequency signal within the pulse width of the second pulse width modulation signal. The quantization counter calculates the number of pulses in the quantized frequency signal. The decoder decodes the number of pulses into quantized values. The reversible counter accumulates the quantized values ​​output by all column classification circuits based on the sign bit signal in the weight correlation signal of the fully connected layer node to obtain a feature value. The feature value comparator compares the calculated feature value with a feature threshold to obtain the classification result.

[0117] For example, Figure 9A and Figure 9B These are circuit diagrams and timing diagrams of the comparison circuit and the classification circuit drawn according to an embodiment of this disclosure. Please refer to... Figure 9A The comparison circuit in this embodiment includes OR gates MP0 to MP20. Taking OR gate MP0 as an example, it can receive the second pulse width modulation signal PW of adjacent pixel groups in the row direction. MP <0> PW MP <1> Please also refer to Figure 9A and Figure 9B During the processing time of filter #1, the OR gate MP0, for example, takes the pulse width modulation signal PW. MP <0> PW MP <1> The pulse width modulation signal with the longest pulse width is output as the pulse width modulation signal PW. FC <0> Similarly, the OR gate MP1, for example, takes the pulse width modulation signal PW. MP <2> PW MP <3> The pulse width modulation signal with the longest pulse width is output as the pulse width modulation signal PW. FC <1> .

[0118] On the other hand, the classification circuit of this embodiment includes multiple column classification circuits, each column classification circuit corresponding to multiple columns of pixels in two adjacent pixel groups in the row direction. Each column classification circuit has a 6-column spacing and includes an AND gate, a quantization counter, and a decoder.

[0119] Taking the first column classification circuit as an example, the AND gate QZ0 will receive the zero-bit signal ZERO from the weight correlation signal of the fully connected layer node. FC <0> The pulse width modulation signal PW output by the OR gate MP0 of the maximum pooling operation circuit. FC <0> and frequency signal CLK FC_QZ The input is a non-zero pulse width modulation signal PW, and the output is a non-zero signal. FC <0> Quantized frequency signal CLK within the pulse width FC_CNT <0> Up to quantization counter 92.

[0120] In one embodiment, the quantization counter 92 includes, for example, a 3-bit counter and a latch, which uses 3 bits to record the number of pulses. In the first column classification circuit, the quantization counter 92 calculates the quantization frequency signal CLK. FC_CNT <0> The number of pulses in CNT FC <0> Please also refer to Figure 9A and Figure 9B In this embodiment, the zero-bit signal ZERO received by AND gate QZ0 FC <0> =0, therefore the output quantization frequency signal CLK is 0. FC_CNT <0> The value is 0, which represents the number of pulses (CNT) calculated by the quantization counter 92. FC <0> It is a 3-bit

[000] . On the other hand, the zero-bit signal ZERO received by AND gate QZ1 FC <0> =1, therefore the output is non-zero and is in the pulse width modulation signal PW FC <1> Quantized frequency signal CLK within the pulse width FC_CNT <1> (Including 3 pulses), the number of pulses calculated by the quantization counter 92 (CNT) FC <0> It is 3 bits

[011] .

[0121] Next, the decoder 94 converts the number of pulses CNT calculated by the quantization counter 92 into a value. FC <0> Decoded into quantized value B2T FC <0> In one embodiment, decoder 94 is, for example, a binary-to-thermometer decoder, used to convert a 3-bit pulse count into a 4-bit quantized value. The 4 bits of this quantized value are then passed through four flip-flops (DFFs) cascaded to the input of a multiplexer (MUX), and through a switch (selected by the CSEL signal). FC The switching of the control is sequentially input into the AND gate AN, so as to pass the frequency signal CLK. FC_SUM Converted to pulse signal CLK FCE Finally, the data is fed into the reversible counter UDC for counting. In this embodiment, the reversible counter UDC is, for example, a 15-bit counter capable of both up and down counting, but is not limited to this.

[0122] The reversible counter UDC receives quantized values ​​sequentially output by the column sorting circuit (converted into pulse signals CLK). FCE In addition, it also receives the sign bit signal SIGN from the weight correlation signal of the fully connected layer nodes. FC According to the sign bit signal SIGN FC The positive and negative values ​​are accumulated into characteristic values ​​by summing the quantized values ​​output by all column classification circuits. Among them, if the sign bit signal SIGN... FC If the sign bit signal is positive, the reversible counter UDC will use the quantized value to multiply the accumulated characteristic value. FC If the value is negative, the reversible counter UDC uses the quantized value to count down the accumulated feature value, and finally obtains the feature value of the quantized value that integrates the output of all column classification circuits.

[0123] Please refer to the following at the same time Figure 9A and Figure 9B During the processing time of filter #2, the column fully connected selection signal COL is used. FC <0:20> Sequentially turn on the output switches of the flip-flop DFF to output the fully connected element signal FCE<0:20>, which is then used by the AND gate AN to output the frequency signal CLK. FC_SUM Converted to pulse signal CLK FCE The signal is then fed into the reversible counter UDC to determine the sign bit signal SIGN. FC Perform up and down counting. This is done using the pulse signal CLK. FCE In the middle, the frequency corresponding to the second column classification circuit includes 3 pulses and the sign bit signal SIGN. FC Since it is positive, the reversible counter UDC will increment the eigenvalue by 3 (i.e., +3); the frequency corresponding to the 3rd column classification circuit includes 2 pulses and the sign bit signal SIGN. FC Since the value is negative, the reversible counter UDC will count down by 2 (i.e., -2) the feature value, and so on, until the reversible counter UDC can finally output the feature value that integrates the quantized values ​​of all column classification circuits.

[0124] In summary, the image sensor integrating convolutional neural network operations in this embodiment uses pulse width modulation pixels as photosensitive elements and performs the operations required for a complete convolutional neural network in a vertical parallel circuit, directly outputting face or feature discrimination results. This solves the problems of frame rate reduction, power consumption increase, and slow discrimination speed caused by data processing requiring inter-chip transmission.

[0125] However, this disclosure has been disclosed above with reference to embodiments, but it is not intended to limit this disclosure. Any person skilled in the art may make some modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims and their equivalents.< / n> < / n-1> < / n> < / n-1> < / m> < / m> < / n> < / n> < / n> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / n> < / n> < / n-1> < / m> < / m-1> < / x> < / m> < / n> < / m> < / rst>

Claims

1. An image sensor integrating convolutional neural network operations, comprising: A pixel array, comprising multiple pixels divided into multiple pixel groups, wherein the optical signal received by each pixel is a first pulse width modulation signal; The convolution operation circuit controls the on-time of a corresponding weight current according to the first pulse width modulation signal of each pixel in each pixel group, and accumulates the weight current of the plurality of pixels as an integral current, wherein the value of the weight current corresponds to the weight of the convolutional layer node in the trained convolutional neural network. The comparator circuit converts the integral current into a second pulse width modulation signal, compares the second pulse width modulation signals of adjacent pixel groups in the row direction, and outputs the second pulse width modulation signal with the longest pulse width. as well as The classification circuit quantizes the second pulse width modulation signal of each pixel group into one of a plurality of quantized values ​​according to the weights of the fully connected layer nodes corresponding to each pixel group in the trained convolutional neural network. It then accumulates the quantized values ​​of all pixel groups as feature values ​​and compares them with a feature threshold to obtain a classification result. The classification circuit includes: Multiple column classification circuits, each column classification circuit corresponding to multiple columns of pixels in two adjacent pixel groups in the row direction within the pixel group, including: The AND gate takes as input the zero-bit signal in the weight correlation signal of the fully connected layer node, the second pulse width modulation signal output by the comparator circuit, and the frequency signal, and outputs a non-zero quantized frequency signal that is within the pulse width of the second pulse width modulation signal. A quantization counter is used to calculate the number of pulses in the quantized frequency signal; and The decoder decodes the number of pulses into a quantized value; A reversible counter, based on the sign bit signal in the weight correlation signal of the fully connected layer nodes, accumulates all the quantized values ​​output by the column classification circuits as the feature values; and A feature value comparator compares the feature values ​​and the feature thresholds to obtain the classification result.

2. The image sensor according to claim 1, wherein each of the pixels comprises: Optical sensor; The intra-pixel comparator includes a first switch and a second switch, wherein... The first switch has a control terminal connected to a reference voltage and a first terminal connected to a selection signal line. The second switch has a first end connected to the second end of the first switch, a second end connected to the first end of the optical sensor and the ground end, and a control end connected to the second end of the optical sensor; A pixel reset switch has a control terminal connected to a reset signal line, a first terminal connected to a second terminal of the light sensor, and a second terminal connected to a second terminal of the first switch and a first terminal of the second switch, configured to selectively turn on the first terminal and the second terminal according to a reset signal from the reset signal line; and An output selection switch has a control terminal connected to a second terminal of the first switch and a first terminal of the second switch, a first terminal connected to a second selection signal line, and a second terminal connected to a signal output line, configured to output the first pulse width modulation signal converted from the optical signal.

3. The image sensor according to claim 1, wherein the convolution operation circuit includes a plurality of column convolution operation circuits corresponding to multiple columns of pixels of the pixel group, each of the column convolution operation circuits comprising: Multiple sign bit logics respectively receive the first pulse width modulation signal of the pixel, and select the first pulse width modulation signal according to the sign bit signal and zero bit signal in the weight correlation signal corresponding to each pixel; Multiple switching current integration units respectively receive the first pulse width modulation signal that is logically selected by the sign bit, and control the on-time of the weight current corresponding to the pixel according to the weight bit signal in the weight correlation signal corresponding to each pixel. as well as An integrating circuit, including at least one integrating capacitor, accumulates the weighted current output from all the switching current integrating units to generate an integrated voltage.

4. The image sensor according to claim 3, wherein the convolution operation circuit further comprises: The weight generation circuit includes: Multiple flip-flops are connected in series in sequence, wherein the input of the first flip-flop receives the weight-related signal, and the inputs of the other flip-flops are connected to the output of the preceding flip-flop in the series. A buffer temporarily stores multiple zero-bit signals, multiple sign-bit signals, and multiple weight-bit signals of the weight-related signal output by the trigger; and Multiple current-to-analog converters (DACs) respectively receive the weight bit signals of the pixels temporarily stored in the buffer, and each current-to-analog converter includes: Multiple weighted switches have first terminals connected to each other and second terminals connected to each other and grounded, wherein the area of ​​each weighted switch has a preset ratio, such that when the weighted switch is turned on at the control terminal according to the weight bit signal, it conducts a current having the preset ratio; and A summing switch has a first terminal connected to a supply voltage, a second terminal connected to the first terminal of the weighting switch, and a control terminal, wherein the voltage at the control terminal corresponds to the sum of the weighting currents flowing through the weighting switch.

5. The image sensor according to claim 3, wherein Each of the aforementioned switching current integration units further includes a positive switch and a negative switch, wherein When the sign bit signal in the received weight-related signal is positive, the negative switch is closed and the positive switch is opened to output the weight current via the positive switch. When the sign bit signal in the received weighted correlation signal is negative, the positive switch is closed and the negative switch is opened to output the weighted current via the negative switch; and The integrating circuit includes a first-side circuit and a second-side circuit, wherein The first-side circuit includes a first integrating capacitor for accumulating the weighted current output from all the positive switches to generate a positive integral voltage. The second-side circuit includes a second integrating capacitor for accumulating the weighted current output by all the negative switches to generate a negative integrated voltage.

6. The image sensor according to claim 5, wherein the comparison circuit comprises: A voltage comparator compares the positive integral voltage and the negative integral voltage to output the convolution result; as well as The judgment logic examines the signal polarity of the convolution result. When the convolution result shows that the positive integral voltage is greater than the negative integral voltage, the integration circuit is controlled to save the voltage difference between the positive integral voltage and the negative integral voltage. When the convolution result shows that the positive integral voltage is less than the negative integral voltage, the integration circuit is controlled to reset the positive integral voltage and the negative integral voltage.

7. The image sensor of claim 6, wherein the comparison circuit further comprises: The first ramp circuit is connected to the first side circuit of the integrator circuit; as well as The second ramp circuit is connected to the second side circuit of the integrating circuit, wherein... When the judgment logic determines that the positive integral voltage is greater than the negative integral voltage in the convolution result, it resets the negative integral voltage using a reference voltage and controls the first ramp circuit to shift the positive integral voltage downward by an amount equal to the negative integral voltage, thereby preserving the voltage difference between the positive and negative integral voltages. When the judgment logic determines that the positive integral voltage is less than the negative integral voltage, it exits the first ramp circuit and the second ramp circuit, and resets the positive integral voltage and the negative integral voltage using a reference voltage.

8. The image sensor according to claim 1, wherein the comparison circuit comprises: OR gate, which takes the second pulse width modulation signal with the longest pulse width among the second pulse width modulation signals of the adjacent pixel groups in the row direction and outputs it.

9. The image sensor of claim 1, wherein the number of pixels in each of the pixel groups is determined based on the size of the kernel of the trained convolutional neural network.

Images