A SPAD array suitable for convolution operations

Abstract

The application discloses a SPAD array suitable for convolution operation and belongs to the technical field of integrated circuits of single photon avalanche devices, which is composed of M*N pixel units arranged in two dimensions, wherein the pixel unit comprises a SPAD, a pulse compression circuit, a negative pulse matrix generating circuit, a positive pulse matrix generating circuit and a multi-input up-down counter; the SPAD detects a photon signal and outputs a digital pulse signal; the pulse compression circuit performs pulse width compression on the digital pulse signal output by the SPAD; the positive and negative pulse matrix generating circuits expand one signal sequence after pulse width compression into a plurality of positive and negative pulse matrices; and the multi-input up-down counter performs accumulation operation, so as to realize addition operation of a signal sequence after multiplication of a pixel matrix and a specific convolution kernel. The application combines the SPAD detector with the convolution neural network technology, realizes one layer of convolution calculation, and has the advantages of relatively simple circuit structure, high detection sensitivity, high operation real-time performance, low power consumption and the like.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology for single-photon avalanche devices, and more specifically to a SPAD array suitable for convolution operations. Background Technology

[0002] In recent years, optoelectronic image sensors have made rapid progress, especially in the field of CMOS image sensors involving neural networks. However, traditional convolutional neural networks perform step convolutions between the image and the convolution kernel, which results in excessive parameters and low efficiency. When the image is unfolded into a vector, spatial information is lost. At the same time, the output signal is an analog signal, which requires analog-to-digital conversion, making the overall circuit structure more complex.

[0003] SPAD (Single Photon Avalanche Diode) is a solid-state semiconductor device that operates at room temperature and ambient pressure. Based on its digital output characteristics and extremely high gain when operating in Geiger mode, SPAD possesses very powerful single-photon detection and photon counting capabilities. Therefore, SPAD-based image sensors are a key technology in many next-generation applications.

[0004] Therefore, providing a SPAD array suitable for convolution operations is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, and to address the shortcomings of traditional convolutional neural networks in CMOS image sensors, namely the inability to perform image convolution and imaging simultaneously and the overly complex circuit structure, this invention provides a SPAD array suitable for convolution operations. This array outputs pure digital pulses, can directly perform digital operations, can achieve high frame rate readout, and has a relatively simple circuit structure.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A SPAD array suitable for convolution operations is composed of M×N two-dimensionally arranged pixel units, wherein the pixel unit includes a SPAD unit, a pulse compression circuit, a negative pulse matrix generation circuit, a positive pulse matrix generation circuit, and a multi-input up and down counter.

[0008] The SPAD unit is used to detect photon signals and output digital pulse signals;

[0009] The pulse compression circuit is used to compress the pulse width of the digital pulse signal output by the SPAD unit and output a narrow pulse width signal.

[0010] The negative pulse matrix generating circuit expands the narrow pulse width signal into multiple negative pulse signals under the control of the negative weight matrix input. The multiple negative pulse signals form a negative pulse matrix, and each negative pulse signal of the negative pulse matrix is ​​input to the multi-input up and down counter.

[0011] The positive pulse matrix generating circuit expands the narrow pulse width signal into multiple positive pulse signals under the control of the positive weight matrix input. The multiple positive pulse signals form a positive pulse matrix, and each positive pulse signal of the positive pulse matrix is ​​input to the multi-input upper and lower counters.

[0012] The multi-input up / down counter is used to receive one negative pulse signal from the negative pulse matrix and one positive pulse signal from the positive pulse matrix of each pixel unit in the pixel matrix formed with the current pixel unit as the center, and to perform an accumulation operation.

[0013] Optionally, the pulse compression circuit includes an inverter with a series PMOS transistor and a NOR gate. The input terminal of the inverter is connected to the output terminal of the SPAD unit. The gate terminal of the PMOS transistor is controlled by the VM voltage to adjust the inverter delay. One input terminal of the NOR gate is connected to the output terminal of the inverter, and the other input terminal is connected to the output terminal of the SPAD unit. The output terminal of the NOR gate is connected to the positive pulse matrix generation circuit and the negative pulse matrix generation circuit, respectively.

[0014] Optionally, the positive pulse matrix generation circuit consists of several AND gates. One input of each AND gate is connected to the output of the NOR gate, and the other input of each AND gate is used to input a weight value. The several AND gates expand the narrow pulse width signal into multiple positive pulse signals. Each AND gate of the positive pulse matrix generation circuit inputs a weight value, that is, the weight value is the duration of the high-level signal.

[0015] Optionally, the negative pulse matrix generation circuit consists of several AND gates, and the number of AND gates is equal to the number of AND gates in the positive pulse matrix generation circuit. One input terminal of the AND gate is connected to the output terminal of the pulse compression circuit, and the other input terminal of the AND gate is used to input a weight value. The several AND gates expand the narrow pulse width signal into multiple negative pulse signals. Each AND gate of the negative pulse matrix generation circuit inputs a weight value, that is, the weight value is the duration of the high-level signal.

[0016] Optionally, the multi-input up / down counter includes an up / down counter and two multi-input OR gates. The multi-input OR gates are respectively connected to the P and N terminals of the up / down counter, and are used to integrate multiple positive pulse signals output by the positive pulse matrix generation circuit into a single pulse signal and transmit it to the P terminal, or to integrate multiple negative pulse signals output by the negative pulse matrix generation circuit into a single pulse signal and transmit it to the N terminal. When the P terminal receives a pulse signal, the count value of the up / down counter increases by "1", and when the N terminal receives a pulse signal, the count value of the up / down counter decreases by "1".

[0017] Optionally, the positive weight matrix and the negative weight matrix are applied to a pixel matrix centered on the current pixel unit through a convolution kernel. The weight matrix of the current pixel unit is a matrix obtained by rotating the convolution kernel by 180 degrees. The positive weight matrix retains all positive weight values ​​and sets the negative weight values ​​to zero. The negative weight matrix retains all negative weight values ​​and sets the positive weight values ​​to zero.

[0018] Optionally, the SPAD unit includes a SPAD device and a quenching circuit. The reverse bias voltage of the SPAD device exceeds the breakdown voltage of the PN junction. When a photon is incident on the active region of the SPAD device, it will trigger avalanche breakdown to generate a large current. Under the action of the quenching circuit, a pulse signal is generated and output to the pulse compression circuit.

[0019] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a SPAD array suitable for convolution operations. Using SPAD technology for convolution operations results in high sensitivity. Image convolution is achieved by controlling the integration time, and negative weights can participate in real-time image convolution. Furthermore, the present invention uses a SPAD detector as the core detection unit, which has an extremely high ability to unwrap photon sequences, enabling purely digital output without the need for an additional analog-to-digital converter, thus simplifying the circuit structure. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the SPAD array structure proposed in this invention, suitable for convolution calculation.

[0022] Figure 2 This is a schematic diagram of the SPAD structure proposed in this invention.

[0023] Figure 3 This is a schematic diagram of the pulse compression circuit proposed in this invention.

[0024] Figure 4 This is a schematic diagram of the positive and negative pulse matrix generation circuit structure proposed in this invention.

[0025] Figure 5 This is a schematic diagram of the positive and negative weight matrix multiplication operation proposed in this invention.

[0026] Figure 6 This is a schematic diagram of the multi-input upper and lower counter structure proposed in this invention.

[0027] Figure 7 This is a schematic diagram of the working process of the multi-input up and down counter proposed in this invention.

[0028] Figure 8 This is a schematic diagram illustrating the relationship between the convolution kernel and the weight matrix proposed in this invention.

[0029] Figure 9 This is a schematic diagram of the pixel unit proposed in this invention from the perspective of SPAD.

[0030] Figure 10 This is a schematic diagram of the operation of the pixel unit proposed in this invention from the perspective of multiple input upper and lower counters. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] This invention addresses the shortcomings of traditional convolutional neural networks in CMOS image sensors, namely the inability to perform image convolution and imaging simultaneously and the overly complex circuit structure. It proposes a SPAD array suitable for convolution calculation. This array outputs pure digital pulses, which can be directly used for digital operations, achieve high frame rate readout, and has a relatively simple circuit structure.

[0033] like Figure 1 As shown, this embodiment of the invention provides a SPAD array suitable for convolution operations, which consists of M×N two-dimensionally arranged pixel units. Each pixel unit includes a SPAD unit, a pulse compression circuit, a negative pulse matrix generation circuit, a positive pulse matrix generation circuit, and a multi-input up and down counter.

[0034] The SPAD unit is used to detect photon signals and output digital pulse signals;

[0035] The pulse compression circuit is used to compress the pulse width of the digital pulse signal output by the SPAD unit, and output a narrow pulse width signal.

[0036] The negative pulse matrix generation circuit expands a narrow pulse width signal into multiple negative pulse signals under the control of the negative weight matrix input. The multiple negative pulse signals form a negative pulse matrix input to a multi-input up-down counter. Under the control of the negative weight matrix input, the negative pulse matrix generation circuit expands a pulse width compressed signal sequence into multiple negative pulse signal outputs. Each output performs a multiplication operation between the current signal sequence and the corresponding negative weight.

[0037] The positive pulse matrix generation circuit expands a narrow pulse width signal into multiple positive pulse signals under the control of the positive weight matrix input. The multiple positive pulse signals form a positive pulse matrix and are input to a multi-input up and down counter. Under the control of the positive weight matrix input, the positive pulse matrix generation circuit expands a pulse width compressed signal sequence into multiple positive pulse signal outputs. Each output performs a multiplication operation between the current signal sequence and the corresponding positive weight.

[0038] The multi-input up / down counter is used to receive one negative pulse signal from each negative pulse matrix and one positive pulse signal from each positive pulse matrix in the pixel matrix formed around the pixel unit, and perform an accumulation operation to realize the addition operation of the signal sequence after multiplying the pixel matrix with a specific convolution kernel. The pixel matrix is ​​generally selected as a 3×3 pixel matrix formed around the pixel unit.

[0039] In another embodiment, a schematic diagram of the SPAD unit structure of the present invention is shown below. Figure 2 As shown, it includes a single SPAD device and a quenching circuit. The reverse bias voltage of the SPAD device exceeds the breakdown voltage of the PN junction. When a photon is incident on the active region of the SPAD device, it will trigger avalanche breakdown and generate a large current. Under the action of the quenching circuit, a pulse signal is generated for the subsequent output.

[0040] In another embodiment, a schematic diagram of the pulse compression circuit of the present invention is shown below. Figure 3 As shown, the pulse compression circuit includes an inverter with a series PMOS transistor and a NOR gate. The IN input of the inverter is connected to the pulse signal output from the previous stage. The source of the PMOS transistor is connected to a voltage, the drain is connected to the source of the inverter, and the gate is controlled by the VM voltage to adjust the inverter delay. One input of the NOR gate is connected to the inverter output, and the other input is connected to the pulse signal output from the previous stage. The pulse compression circuit is used to compress the wide-width pulse signal output from the previous stage into a narrow-width pulse signal, thereby increasing the bus bandwidth.

[0041] In another embodiment, the positive and negative pulse matrix generation circuits of the present invention are each composed of several AND gates, with one input terminal of the AND gate connected to the output of the pulse compression circuit and the other input terminal connected to the weight input value. Figure 4 Taking the 18-channel example, it is divided into 9 positive pulse matrix generation circuits and 9 negative pulse matrix generation circuits. The 9 AND gates of the positive pulse matrix generation circuit are input with 9 positive weight values ​​through the positive weight bus to realize the positive weight matrix input; the 9 AND gates of the negative pulse matrix generation circuit are input with 9 negative weight values ​​through the negative weight bus to realize the negative weight matrix input.

[0042] Figure 5 This is a schematic diagram of the positive and negative weight matrix multiplication operation proposed in this invention. The pulse-width compressed signal sequence output from the pulse compression circuit, under the control of the positive and negative weight matrix inputs, is processed by the positive and negative pulse matrix generation circuit to output multiple positive and negative pulse matrix outputs. Each output performs a multiplication operation between the current signal sequence and its corresponding weight. Specifically, each AND gate input of the positive and negative pulse matrix generation circuit uses the normalized weight integration time, i.e., a high-level signal lasting a specific duration, the length of which represents the weight value. The pulse-width compressed signal sequence is ANDed with the high-level signal to obtain the weighted modulated output pulse signal. The multiple weighted modulated output pulse signals form positive and negative pulse matrix outputs, which are then transmitted to the corresponding multi-input upper and lower counters.

[0043] In another embodiment, the multi-input up / down counter of the present invention is as follows: Figure 6 As shown, it includes upper and lower counters and two multi-input OR gates. Figure 6 Taking a 9-input OR gate as an example, the multi-input OR gate is connected to the P and N terminals of the upper and lower counters respectively. It is used to integrate multiple positive pulse signals output by the positive pulse matrix generation circuit into one pulse signal and transmit it to the P terminal, or to integrate multiple negative pulse signals output by the negative pulse matrix generation circuit into one pulse signal and transmit it to the N terminal. When the P terminal receives a pulse signal, the count value of the upper and lower counters increases by "1", and when the N terminal receives a pulse signal, the count value of the upper and lower counters decreases by "1".

[0044] Figure 7This is a schematic diagram illustrating the working process of the multi-input up / down counter proposed in this invention. DIN <2> DIN <1> DIN <0> The lower 3 bits of the multi-input up / down counter are used. The P terminal receives the pulse signal from input P, ​​and the N terminal receives the pulse signal from input N. The initial value of the multi-input up / down counter is 000. When the rising edge of the first positive pulse arrives at the P terminal, the count value is incremented by "1", becoming 001. Similarly, when the rising edge of the first negative pulse arrives at the N terminal, the count value is decremented by "1" from 011 to 010. Table 1 shows the process of the multi-input up / down counter's count value changing with the arrival of positive and negative pulses. DIN <2> DIN <1> DIN <0> The lower 3 bits are the count values. The numerical superscripts in the pulse sequence are used to distinguish between positive and negative pulses. For example, 2+ represents the arrival of the second positive pulse, and 1- represents the arrival of the first negative pulse.

[0045] Table 1 shows the process of how the count value of the multi-input up and down counter changes with the arrival of positive and negative pulses.

[0046]

[0047] In another embodiment, the positive weight matrix and the negative weight matrix are applied to the pixel matrix formed with the current pixel unit as the center through the convolution kernel. The weight matrix of the current pixel unit is the matrix obtained by rotating the convolution kernel by 180 degrees. The weight matrix is ​​decomposed into a positive weight matrix and a negative weight matrix according to the positive and negative weight values. The positive weight matrix retains all positive weight values ​​and sets the negative weight values ​​to zero. The negative weight matrix retains all negative weight values ​​and sets the positive weight values ​​to zero. Figure 8 This is a schematic diagram illustrating the relationship between the convolution kernel and the weight matrix proposed in this invention. Taking a 3×3 convolution kernel as an example, the convolution kernel is... The convolution kernel operates on a 3×3 pixel matrix centered on the current pixel unit. For the current SPAD pixel unit, its weight matrix is ​​the matrix obtained by rotating the convolution kernel by 180 degrees. The weight matrix is ​​decomposed into two weight matrices, positive and negative, based on the sign of the weight values. The positive weight matrix retains all positive weight values ​​and sets the negative weight values ​​to zero; the negative weight matrix retains all negative weight values ​​and sets the positive weight values ​​to zero. For example, it can be decomposed into a negative weight matrix. With positive weight matrix The generation process of the weight matrix is ​​the same for each SPAD pixel unit, so the weight matrix... The weight matrix for each SPAD.

[0048] Figure 9 This is a schematic diagram illustrating the operation of the pixel unit proposed in this invention from the perspective of SPAD. For the current pixel unit, after detecting the photon signal, it generates a pulse signal sequence A, which is then processed by the negative weight matrix. With positive weight matrix Under its influence, a 3×3 negative pulse matrix and a 3×3 positive pulse matrix are generated through the negative pulse matrix generation circuit and the positive pulse matrix generation circuit, respectively. The two pulse matrices are output to the corresponding input terminals of the nine multi-input up and down counters of the 3×3 pixel matrix formed with the current pixel unit as the center, respectively. For each pixel unit, it outputs a weighted modulated signal to itself and the surrounding eight pixel units (i.e., the 3×3 pixel matrix formed with the current pixel unit as the center) through a weight matrix; for example, the current pixel unit outputs a signal modulated by the weight value w33 to the upper left corner and a signal modulated by the weight value w11 to the lower right corner.

[0049] Figure 10 This is a schematic diagram illustrating the operation of the pixel unit proposed in this invention from the perspective of a multi-input up-down counter. The multi-input up-down counter receives weighted signals from the current pixel and its eight surrounding pixels (i.e., a 3×3 pixel matrix centered on the current pixel unit), and accumulates these signals. Specifically, the pixel unit at the top left of the current pixel unit outputs a signal modulated by weight value w11, the pixel unit at the bottom right outputs a signal modulated by weight value w33, and so on. The signals received by the multi-input up-down counter for the current pixel unit are precisely the signals multiplied by the corresponding convolution kernel within the 3×3 pixel matrix centered on the current pixel unit. The multi-input up-down counter accumulates these signals to achieve a complete convolution operation.

[0050] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0051] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A SPAD array suitable for convolution operations, characterized in that, It consists of M×N two-dimensionally arranged pixel units, each pixel unit including a SPAD unit, a pulse compression circuit, a negative pulse matrix generation circuit, a positive pulse matrix generation circuit, and a multi-input up and down counter; The SPAD unit is used to detect photon signals and output digital pulse signals; The pulse compression circuit is used to compress the pulse width of the digital pulse signal output by the SPAD unit and output a narrow pulse width signal. The negative pulse matrix generating circuit expands the narrow pulse width signal into multiple negative pulse signals under the control of the negative weight matrix input. The multiple negative pulse signals form a negative pulse matrix, and each negative pulse signal of the negative pulse matrix is ​​input to the multi-input up and down counter. The positive pulse matrix generating circuit expands the narrow pulse width signal into multiple positive pulse signals under the control of the positive weight matrix input. The multiple positive pulse signals form a positive pulse matrix, and each positive pulse signal of the positive pulse matrix is ​​input to the multi-input upper and lower counters. The multi-input up / down counter is used to receive one negative pulse signal from the negative pulse matrix and one positive pulse signal from the positive pulse matrix of each pixel unit in the pixel matrix formed with the current pixel unit as the center, and to perform an accumulation operation.

2. A SPAD array suitable for convolution operations according to claim 1, characterized in that, The pulse compression circuit includes an inverter with a series PMOS transistor and a NOR gate. The input terminal of the inverter is connected to the output terminal of the SPAD unit. The gate terminal of the PMOS transistor is controlled by the VM voltage to adjust the inverter delay. One input terminal of the NOR gate is connected to the output terminal of the inverter, and the other input terminal is connected to the output terminal of the SPAD unit. The output terminal of the NOR gate is connected to the positive pulse matrix generation circuit and the negative pulse matrix generation circuit, respectively.

3. A SPAD array suitable for convolution operations according to claim 2, characterized in that, The positive pulse matrix generation circuit consists of several AND gates. One input of each AND gate is connected to the output of the NOR gate, and the other input of each AND gate is used to input a weight value. The several AND gates expand the narrow pulse width signal into multiple positive pulse signals. Each AND gate of the positive pulse matrix generation circuit inputs a weight value, that is, the weight value is the duration of the high-level signal.

4. A SPAD array suitable for convolution operations according to claim 3, characterized in that, The negative pulse matrix generation circuit consists of several AND gates, and the number of AND gates is equal to the number of AND gates in the positive pulse matrix generation circuit. One input terminal of each AND gate is connected to the output terminal of the pulse compression circuit, and the other input terminal of each AND gate is used to input a weight value. The several AND gates expand the narrow pulse width signal into multiple negative pulse signals. Each AND gate of the negative pulse matrix generation circuit inputs a weight value, that is, the weight value is the duration of the high-level signal.

5. A SPAD array suitable for convolution operations according to claim 1, characterized in that, The multi-input up / down counter includes an up / down counter and two multi-input OR gates. The multi-input OR gates are respectively connected to the P and N terminals of the up / down counter, and are used to integrate multiple positive pulse signals output by the positive pulse matrix generation circuit into a single pulse signal and transmit it to the P terminal, or to integrate multiple negative pulse signals output by the negative pulse matrix generation circuit into a single pulse signal and transmit it to the N terminal. When the P terminal receives a pulse signal, the count value of the up / down counter increases by "1", and when the N terminal receives a pulse signal, the count value of the up / down counter decreases by "1".

6. A SPAD array suitable for convolution operations according to claim 1, characterized in that, The positive weight matrix and the negative weight matrix are applied to the pixel matrix formed with the current pixel unit as the center through a convolution kernel. The weight matrix of the current pixel unit is the matrix obtained by rotating the convolution kernel by 180 degrees. The positive weight matrix retains all positive weight values ​​and sets the negative weight values ​​to zero. The negative weight matrix retains all negative weight values ​​and sets the positive weight values ​​to zero.

7. A SPAD array suitable for convolution operations according to claim 1, characterized in that, The SPAD unit includes a SPAD device and a quenching circuit. The reverse bias voltage of the SPAD device exceeds the breakdown voltage of the PN junction. When a photon is incident on the active region of the SPAD device, it will trigger avalanche breakdown and generate a large current. Under the action of the quenching circuit, a pulse signal is generated and output to the pulse compression circuit.

Images