Pulse convolutional neural network algorithm, integrated circuit, computing device and storage medium

Abstract

A pulse convolutional neural network algorithm and related integrated circuits, computing devices and storage media are used to perform pulse convolutional neural network operations in artificial intelligence in an efficient and low-energy manner. The algorithm is characterized in that it is based on a storage-computation integrated unit, the storage-computation integrated unit includes at least one storage input terminal, at least one computing input terminal and an output terminal, and is characterized in that the weights of the first layer of the pulse convolutional neural network are copied to a plurality of copies, the number of copies is at least the number of bits of the binary number converted from the quantity used to characterize the attribute of the analyte and the minimum value of the storage input terminal of the storage-computation integrated unit, and the weights of the copied copies are processed so that the values ​​of each copied weight are reduced by two times in turn, and the obtained values ​​are respectively input into the storage input terminals of a plurality of the storage-computation integrated units, and the number of the storage-computation integrated units is the same as the number of copies.

Description

Technical Field

[0001] The present invention relates to a pulse convolutional neural network, a pulse convolutional neural network algorithm, an integrated circuit, a computing device and a storage medium, which can convert a traditional convolutional neural network into a pulse neural network, and specifically relates to the field of image classification. Background Art

[0002] Most traditional computers adopt the von Neumann architecture. However, the separation of storage units and computing units in the von Neumann architecture results in a huge energy consumption in data transmission and affects the computing speed. At present, convolutional neural networks have very good results in the field of image classification, with a large number of mature training methods and tools, as well as a large number of verified classic convolutional neural network models, such as LeNet-5, AlexNet, VGG-16, etc. If it is run on hardware using the von Neumann architecture, such as CPU, GPU, FPGA, a large amount of data transmission is required. For large matrices, the computing speed is much faster than the data transmission speed. Optimizing the computing speed cannot speed up the entire system.

[0003] The storage-computing integrated unit can save data directly on the computing unit without on-chip cache, thus solving the above problems. However, there is still a large amount of intermediate data that needs to be cached in the convolutional neural network operation, and when implemented on the storage-computing integrated unit, a large number of analog-to-digital converters are needed to convert the current into digital signals, which occupies most of the area and power consumption of the entire system. In addition, due to the limited frequency of the analog-to-digital converter, the computing speed of the entire system is also limited by it and cannot be improved. Summary of the invention

[0004] The pulse neural network attempts to simulate the computing method of the human brain as much as possible. The obvious feature is that data flows in the network in the form of pulse signals. When implemented in hardware, the power consumption is much lower than that of the convolutional neural network.

[0005] The pulse convolutional neural network combines the characteristics of the convolutional neural network and the pulse neural network. After some modifications to the convolutional neural network model, the weights can be obtained by training the convolutional neural network, and the classification accuracy is only slightly reduced compared to the convolutional neural network used. Since the data in the network are all in the form of pulses, the hardware resource consumption is small. At present, research in this field is still only at the algorithm level, and there is no relevant hardware implementation solution.

[0006] In the existing pulse convolutional neural network algorithm, one input method is that the input pulse is generated by random number generation. According to the law of large numbers, it takes a long time to generate a large number of pulses before it can converge to the original value and get close to the classification result of the convolutional neural network. This requires a lot of computing time and has low computing efficiency. Another input method is that the input is not a pulse, but an analog value. When the circuit is implemented, on the one hand, the input accuracy cannot be guaranteed. On the other hand, considering the actual application, the input source is likely to be an image sensor, and the output is all digital signals, so compatibility needs to be considered.

[0007] The Batch Normalization (BN) layer is a commonly used layer in convolutional neural networks to optimize the network. It can improve the accuracy of training and reduce the dependence of training results on the initialization method. Through mathematical deduction, if a BN layer is to be added to the pulse convolutional neural network algorithm, after the convolutional layer / fully connected layer is merged with the BN layer, a bias will inevitably be generated in the convolutional layer / fully connected layer. However, the existing pulse convolutional neural network algorithms avoid the use of bias, so it is impossible to add a BN layer, which brings trouble to the training of large-scale convolutional neural networks.

[0008] In addition, the existing pulse convolutional neural network algorithms have not considered the problem of termination. However, in actual simulations and circuits, calculation time is also a very important consideration, and this is also worth improving.

[0009] In view of the above, according to one aspect of the present invention, a pulse convolutional neural network algorithm is proposed. By changing the input method, merging the average pooling layer into the next convolutional layer or fully connected layer, supporting the calculation of convolutional layers and fully connected layers with biases, supporting the addition of BN layers in the network, setting calculation end judgment conditions, adding auxiliary judgment for special cases and other optimization and improvement methods, the calculation time of the existing pulse convolutional neural network algorithm can be greatly saved, the accuracy of image classification can be improved, the functional support of the pulse convolutional neural network algorithm for bias and BN layers can be increased, and the input method can be adjusted to increase compatibility.

[0010] According to another aspect of the present invention, a pulse convolutional neural network computing device is proposed. When implementing the pulse convolutional neural network on the storage and computing integrated unit, the multi-bit digital signal representing the true value in the convolutional neural network is converted into a time series pulse signal, and the analog-to-digital converter is replaced by a current integral comparison circuit, thereby greatly reducing the area and power consumption. Moreover, the mapping method of the convolutional layer and the fully connected layer is fully expanded, that is, all the output results of each layer are calculated at the same time, and connected to the next layer as the input of the next layer, and the weight coefficients of each convolutional layer / fully connected layer are stored in the storage and computing integrated unit, so that no data needs to be cached during the operation process, and the calculation speed of the entire system is significantly accelerated.

[0011] However, for large-scale pulse convolutional neural networks, the required storage and computing units are proportional to the square of the input image size and the number of convolutional layer channels, which requires a large area. In addition, the calculation speed of this solution is very fast. When the image is very large, it far exceeds the transmission speed of the input image data. In other words, the calculation speed will be limited because the data transmission speed cannot keep up.

[0012] In view of the above, according to another aspect of the present invention, a pulse convolutional neural network computing device with a memory is proposed. By packaging the pulse signal according to a fixed time length and adding it to the on-chip or off-chip memory to save the intermediate data, the number of storage and computing units required is greatly reduced, thereby reducing the area and power consumption. Although such a method will make the theoretical calculation speed much lower than that of the solution without saving intermediate data, in fact, due to the bottleneck limit of data transmission speed, the final speed is within an acceptable range.

[0013] According to one aspect of the present invention, there is provided a pulse convolutional neural network algorithm, based on a storage-computation integrated unit, the storage-computation integrated unit comprising at least one storage input terminal, at least one calculation input terminal and an output terminal, characterized in that: 1) the weights of the first layer of the pulse convolutional neural network are copied to a plurality of copies, the number of copies being at least the number of bits of the binary number converted from the quantity used to characterize the property of the analyte and the minimum value of the storage input terminal of the storage-computation integrated unit, and the weights of the copied number of copies are processed so that the values ​​of the copied weights are successively reduced by two times, and the obtained values ​​are respectively input to the storage input terminals of a plurality of the storage-computation integrated units, the number of the storage-computation integrated units being the same as the number of copies; 2) the selected quantity concentrated for characterizing the property of the analyte is converted into a binary number, and the value of each bit of the binary number to be input, or the value after truncating according to the system bit width, is used as the input pulse , input into the storage-computation integrated computing unit of the pulse convolutional neural network; and, for each input set characterizing the properties of the analyte, within the time period corresponding to the input set, keep the input pulse unchanged and continuously input into the corresponding computing unit in the pulse convolutional neural network until the processing of the properties of the analyte is completed; 3) For each basic binary number corresponding to the one group used to characterize the properties of the analyte, the input quantity of each storage input terminal corresponds to the input quantity of a computing input terminal, and the input quantity of the storage input terminal with a larger absolute value corresponds to the input quantity of the higher-order computing input terminal one by one; 4) In each of the storage-computation integrated units, the quantity of the storage input terminal is calculated with the quantity of the computing input terminal, and the current value obtained at the output terminal represents the result of multiplication of the value of the storage input terminal of the storage-computation integrated unit and the value of the computing input terminal.

[0014] In addition, according to an embodiment of the present invention, the pulse convolutional neural network algorithm is further characterized in that: 1) it includes the operations of the first layer and the operations of other layers, and in any layer therein, in addition to the operations at the storage input end and the calculation input end, an operation accumulation term is added, and the operation accumulation term is a corrected bias value, and the corrected bias value is proportional to its original value divided by the cumulative multiplication of the positive thresholds of all layers before the layer, and the proportional ratio is related to the weight scaling ratio of the layer where the bias is located and the previous layer; 2) the pulse convolutional neural network algorithm continuously accumulates the output of the storage and calculation integrated unit, and when the accumulated sum exceeds a set positive threshold, the accumulated sum is cleared, and an output pulse is released to the calculation input end of the corresponding position of the next layer; and when the accumulated sum is less than a set negative threshold, the accumulated sum is kept at the negative threshold.

[0015] In addition, according to one embodiment of the present invention, the pulse convolutional neural network includes a batch normalization layer, and a linear transformation is performed on the weights and biases in a convolutional layer or a fully connected layer before the batch normalization layer, wherein the parameters in the linear transformation are obtained in the previous training process.

[0016] In addition, according to one embodiment of the present invention, multiple counters are used to count the number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time when the earliest pulse occurs, and the number of counters is the number of neurons or twice thereof.

[0017] In addition, according to an embodiment of the present invention, if the counting results of at least two counters among the multiple counters are the same maximum value, the category value corresponding to the counter that receives the pulse earliest is selected as the final result.

[0018] In addition, according to an embodiment of the present invention, if there are significantly more counters, the output terminates the operation and the final classification result is output as the category value corresponding to the maximum value of the counting results of the multiple counters.

[0019] In addition, according to one embodiment of the present invention, after the operation of the first layer, at least one of average pooling, maximum pooling, convolutional layer and fully connected layer operations is also performed.

[0020] In addition, according to one embodiment of the present invention, the pulse convolutional neural network algorithm is also characterized by: 1) setting the duration of several clock signals as an analysis period; 2) dividing the target object to be analyzed into several partitions; 3) taking the analysis period as the time unit, analyzing the time series signal of a partition one by one, and sending the operation result representing the partition to a memory; 4) analyzing the signal of the next partition, and sending the operation result representing the partition to the memory, until the signals of the completed multiple partitions jointly meet the analysis conditions of the next layer; 5) sending the signals of each of the partitions stored in the memory to the next layer for calculation.

[0021] In addition, according to an embodiment of the present invention, the memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage, or a combination thereof.

[0022] According to another aspect of the present invention, there is provided an integrated circuit based on a pulse convolutional neural network, characterized in that the integrated circuit executes the pulse convolutional neural network algorithm as described above.

[0023] According to another aspect of the present invention, there is provided a computer-readable recording medium storing computer-readable instructions, which, when executed by a computer, causes the computer to execute a pulse convolutional neural network algorithm, wherein the pulse convolutional neural network algorithm is characterized in that: 1) the weights of the first layer of the pulse convolutional neural network are copied to a plurality of copies, the number of copies being at least the number of bits of the binary number converted from the quantity used to characterize the property of the analyte and the minimum value of the storage input terminal of the storage-computation integrated unit, and the weights of the copied number of copies are processed so that the values ​​of the copied weights are successively reduced by two times, and the obtained values ​​are respectively input into the storage input terminals of a plurality of the storage-computation integrated units, the number of the storage-computation integrated units being the same as the number of copies; 2) the selected quantity concentrated for characterizing the property of the analyte is converted into a binary number, and each bit value of the binary number to be input, or the value of each bit after truncation according to the system bit width, is converted into a binary number. The numerical value is used as an input pulse and input into the storage and calculation integrated calculation unit of the pulse convolutional neural network; and, for each input set characterizing the properties of the analyte, within the time period corresponding to the input set, the input pulse is kept unchanged and continuously input into the corresponding calculation unit in the pulse convolutional neural network until the processing of the properties of the analyte is completed; 3) For each basic binary number corresponding to the one group used to characterize the properties of the analyte, the input quantity of each storage input terminal corresponds to the input quantity of a calculation input terminal, respectively, and the input quantity of the storage input terminal with a larger absolute value corresponds to the input quantity of the higher-order calculation input terminal one by one; 4) In each of the storage and calculation integrated units, the quantity of the storage input terminal is calculated with the quantity of the calculation input terminal, and the current value obtained at the output terminal represents the result of the multiplication of the value of the storage input terminal of the storage and calculation integrated unit and the value of the calculation input terminal.

[0024] In addition, according to an embodiment of the present invention, the computer-readable recording medium is also characterized in that: 1) the pulse convolutional neural network algorithm includes the operations of the first layer and the operations of other layers, and in any layer therein, in addition to the operations of the storage input terminal and the calculation input terminal, an operation accumulation item is added, and the operation accumulation item is a corrected bias value, and the corrected bias value is proportional to its original value divided by the cumulative multiplication of the positive thresholds of all layers before this layer, and the proportional ratio is related to the weight scaling ratio of the layer where the bias is located and the previous layer; 2) the pulse convolutional neural network algorithm continuously accumulates the output of the storage and calculation integrated unit, and when the accumulated sum exceeds a set positive threshold, the accumulated sum is cleared, and an output pulse is released to the calculation input terminal of the corresponding position of the next layer; and when the accumulated sum is less than a set negative threshold, the accumulated sum is kept at the negative threshold.

[0025] In addition, according to one embodiment of the present invention, the pulse convolutional neural network includes a batch normalization layer, and a linear transformation is performed on the weights and biases in a convolutional layer or a fully connected layer before the batch normalization layer, wherein the parameters in the linear transformation are obtained in the previous training process.

[0026] In addition, according to one embodiment of the present invention, multiple counters are used to count the number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time when the earliest pulse occurs, and the number of counters is the number of neurons or twice thereof.

[0027] In addition, according to an embodiment of the present invention, if the counting results of at least two counters among the multiple counters are the same maximum value, the category value corresponding to the counter that receives the pulse earliest is selected as the final result.

[0028] In addition, according to an embodiment of the present invention, during the counting process of the multiple counters, if the number of pulses collected by one counter is significantly greater than that of other counters, the output terminates the operation and the final classification result is output as the category value corresponding to the maximum value of the counting results of the multiple counters.

[0029] In addition, according to one embodiment of the present invention, after the operation of the first layer, at least one of average pooling, maximum pooling, convolutional layer and fully connected layer operations is also performed.

[0030] In addition, according to one embodiment of the present invention, the pulse convolutional neural network algorithm includes the following: 1) setting the duration of several clock signals as an analysis cycle; 2) dividing the target object to be analyzed into several partitions; 3) taking the analysis cycle as the time unit, analyzing the time series signal of one partition one by one, and sending the operation result representing the partition to a memory, and the analyzed signal can be covered by the subsequent signal; 4) analyzing the signal of the next partition, and sending the operation result representing the partition to the memory, until the completed signals of multiple partitions jointly meet the analysis conditions of the next layer; 5) sending the signals of each of the partitions stored in the memory to the next layer for calculation.

[0031] In addition, according to an embodiment of the present invention, the memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage, or a combination thereof.

[0032] According to another aspect of the present invention, there is provided an integrated circuit based on a pulse convolutional neural network, wherein the pulse convolutional neural network comprises multiple layers of neurons, each layer of neurons comprises multiple neuron components, multiple neurons in each layer of neurons are not connected to each other, but are connected to neurons in the next layer; at least one of the neuron components carries at most one digital logic circuit, and the digital logic circuit is used for operations, and the operations include data distribution and may also include maximum pooling, clock synchronization, and data caching; and each neuron component of the last layer carries a counter group to count the number of pulses with a high level in the output pulses of the neuron component; wherein each neuron comprises at least one storage-computation-in-one unit and at least one integral comparison circuit, and the current output terminals of the multiple storage-computation-in-one units are connected to each other and collectively connected to the integral comparison circuit. In the circuit; each of the integral comparison circuits includes at least one integrator and at least one comparator, the integrator is used to accumulate the output of the current output terminal, the comparator is used to compare the accumulated output in the integrator with a previously set threshold value, and to reset and pulse output the comparator, the reset operation enables the integrator to perform the next accumulation operation; and each of the storage and calculation integrated units includes at least one storage input terminal and at least one calculation input terminal and at least one current output terminal, the storage input terminal is configured to receive carriers representing the weights sent by the upper computer, and the calculation input terminal is configured to receive carriers representing the external environment or the set upper input pulse; the current output terminal is configured to output the carriers after the carriers used as weights and the carriers used as input pulses act together in the form of current.

[0033] In addition, according to an embodiment of the present invention, the integrated storage and computing unit is one of a semiconductor-based photoelectric computing unit, a memristor, and a flash memory.

[0034] In addition, according to one embodiment of the present invention, the digital logic circuit is configured to find the first high-level pulse signal from a plurality of output signals output from the previous layer of neuron components of the current pooling layer, the number of which is the square of the size of the pooling layer; and the digital logic circuit is also configured to include a functional device of a multiplexer, so that after the high-level pulse signal passes through the multiplexer, the path corresponding to the high-level pulse signal is kept open, and the path is connected to the next convolutional layer or fully connected layer; at the same time, the signals of other paths parallel to the path corresponding to the high-level pulse signal are ignored, or the other paths are closed.

[0035] In addition, according to one embodiment of the present invention, the average pooling operation is merged into the next convolutional layer or fully connected layer, including: 1) a convolutional layer or a fully connected layer, the number of storage-computational units in each neuron component of the convolutional layer or the fully connected layer is several times the original size of the corresponding algorithm of the layer, and the multiple is the square of the pooling layer size, and each weight in the corresponding algorithm appears several times in the neuron component, and the number is the square of the pooling layer size; 2) wherein the output pulse signal output from the neuron component of the previous layer and to be transmitted to the next pooling layer, the number of which is the square of the pooling layer size, is directly used as the calculation input of the storage-computational unit in the convolutional layer or the fully connected layer, and the storage-computational units correspond to the same weights respectively.

[0036] Furthermore, according to an embodiment of the present invention, each of the neuron components includes a neuron and has a register, and the register is used to achieve time synchronization of the data operations involved.

[0037] According to another aspect of the present invention, there is provided a pulse convolutional neural network operation device for performing pulse convolutional neural network operation, comprising a host computer and the above-mentioned integrated circuit; wherein the host computer is configured to process and generate weights of the first layer, and the process of generating the weights of the first layer comprises: generating a group of weights according to a training initial weight through several linear transformations, the group of weights comprising multiple weight numbers, wherein the latter weight value is 1 / 2 of the former weight value; and the host computer sends the group of weights to the storage input end of the storage-computation integrated unit in each neuron component of the first layer of the pulse convolutional neural network; and the host computer sends the initial weights to the storage input end of the storage-computation integrated unit of other layers after the first layer after several linear transformations, and for the weights of the convolutional layer or the fully connected layer immediately after the average pooling layer, the weights are also copied several times according to the pooling size, and the number of copies is the square of the pooling layer size.

[0038] In addition, according to one embodiment of the present invention, the device is used to analyze the target object by partitions, and then synthesize the target object signals of each partition to form complete target object information, and the pulse convolutional neural network operation device also includes a memory, which is used to store the signals of at least one partition of the target object that have been processed in steps and represent the target object, and after all the partition signals are processed, all the partition signals are synthesized, or all the partition signals are sent to another processor for synthesis; the memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage.

[0039] According to another aspect of the present invention, a method for manufacturing the above-mentioned integrated circuit is provided, and the method includes the following steps: 1) forming a dielectric layer and a gate of transistors in a digital logic circuit, an integral comparison circuit and a storage-computing integrated unit by thermal oxidation and deposition; the transistors include at least a common logic transistor, a high-voltage transistor and a floating gate transistor; 2) forming a capacitor in an integral comparison circuit by depositing a MIM dielectric layer and depositing a metal layer, or by thermal oxidation and deposition processes; 3) forming a source and a drain of transistors in a digital logic circuit, an integral comparison circuit and a storage-computing integrated unit, and a P-level and N-level of a PN junction by ion implantation; 4) forming metal wiring and active area-metal layer and metal layer-metal layer through-hole processes of the overall circuit; 5) generating a storage-computing integrated unit of a CMOS process by a process applied to a memristor or a flash memory.

[0040] The purpose of the present invention is at least to convert the data in the convolutional neural network into a time pulse sequence and replace the analog-to-digital converter with large power consumption and area by a current integration comparison circuit, thereby greatly reducing the area and power consumption of the entire system.

[0041] Another purpose of the present invention is to directly connect the output results of each convolutional layer / fully connected layer to the next convolutional layer / fully connected layer, and the weight data can be directly stored in the storage and computing unit. No on-chip cache is required in the entire system, which saves a lot of data transfer processes, thereby speeding up the calculation. For large-scale networks, the present invention proposes a pulse convolutional neural network computing device with a memory. The output results of each convolutional layer / fully connected layer are directly connected to the next convolutional layer / fully connected layer. The storage and computing units required are too many and the area is too large. Therefore, part of the data is stored in an on-chip or off-chip memory, and the method of exchanging time for space will greatly reduce the required hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a block diagram of a multi-function area of ​​a computing unit according to an embodiment.

[0043] Figure 2 is a schematic diagram of the structure of an optoelectronic computing array according to an embodiment.

[0044] Figure 3 (a) and (b) are a cross-sectional view and a stereoscopic view of the calculation unit structure of Example 1-1.

[0045] Figure 4 (a) and (b) are a cross-sectional view and a stereoscopic view of the calculation unit structure of Example 1-2.

[0046] Figure 5Schematic diagram of the structure of the computing unit (a) and the multi-functional area (b) of Examples 1-3.

[0047] Figure 6 1 is a schematic diagram of the structure of an RRAM device according to an embodiment and an overview of its three terminals.

[0048] Figure 7 is a basic cell unit structure diagram of a flash memory according to an embodiment.

[0049] Figure 8 It is a schematic diagram of the structure of Spiking-Lenet-5 of Example 4-1 (average pooling).

[0050] Fig. 9 It is a schematic diagram of the structure of Spiking-Lenet-5 of Example 4-1 (maximum pooling).

[0051] Fig.10 This is a schematic diagram of a neuron composed of a storage and computing unit in Example 4-1.

[0052] Fig.11 It is a block diagram of the entire system of Example 4-1 (average pooling).

[0053] Fig.12 It is a block diagram of the entire system of Example 4-1 (maximum pooling).

[0054] Fig.13 This is a calculation flow chart of the entire system of Example 4-1 (average pooling).

[0055] Fig.14 This is the calculation flow chart of the entire system of Example 4-1 (maximum pooling).

[0056] Fig.15 This is a schematic diagram of a neuron composed of a storage and computing unit in Example 4-2 (minus the register).

[0057] Fig.16 This is a block diagram of the entire system of Example 4-2 (average pooling, register removal).

[0058] Fig.17 This is a block diagram of the entire system of Example 4-2 (maximum pooling, register removal).

[0059] Fig.18 It is a schematic diagram of the structure of Spiking-Alexnet of Example 4-3.

[0060] Fig.19 This is a schematic diagram of a neuron composed of a storage and computing unit in Example 4-3.

[0061] Fig. 20 It is a block diagram of the entire system of Example 4-3.

[0062] Fig.21 It is a calculation flow chart of the entire system of Example 4-3.

[0063] Fig. 22 It is a block diagram of the entire system of Example 4-4.

[0064] Fig.23 It is a block diagram of the entire system of Example 4-5.

[0065] Fig.24 It is a block diagram of the entire system of Examples 4-6.

[0066] Fig.25 This is the Alexnet network structure diagram of Example 5.

[0067] Fig.26 This is the Spiking-Alexnet network structure diagram of Example 5 (average pooling).

[0068] Fig. 27 This is the Spiking-Alexnet network structure diagram (maximum pooling) of Example 5.

[0069] Fig.28 This is a diagram of the structure of the neuron of Example 5. DETAILED DESCRIPTION

[0070] The storage-computation integrated unit described in the present invention is not specific to a particular device. As long as the storage-computation integrated unit can store data, the vector dot multiplication operation can be completed by combining multiple storage-computation integrated units. For each storage-computation integrated unit, there is a storage input terminal, a calculation input terminal and an output terminal. The data at the storage input terminal can be stored for a long time, and the value of the output terminal is proportional to the product of the calculation input terminal and the storage input terminal, and the output terminals of multiple storage-computation integrated units can be summed.

[0071] Next, the storage and computing integrated unit is described using the optoelectronic computing unit, memristor, and flash memory as examples.

[0072] Example 1

[0073] The computing unit in the optoelectronic computing unit is a multifunctional area structure including three major functional areas, such as Figure 1 As shown, the three functional areas are: carrier control area, coupling area, photogenerated carrier collection area and readout area, and their specific functions are as follows:

[0074] Carrier control area: responsible for controlling and modulating the carriers in the optoelectronic computing unit, and serving as the electrical input port of the computing unit, inputting one of the operands as the electrical input; or only controlling and modulating the carriers in the computing unit, and inputting the electrical input through other areas.

[0075] Coupling region: responsible for connecting the photogenerated carrier collection region and the readout region, so that the photogenerated carriers generated by the incident photons act on the carriers in the photoelectric computing unit to form an operational relationship.

[0076] Photogenerated carrier collection region and readout region: the collection region is responsible for absorbing incident photons and collecting generated photogenerated carriers, and serves as the optical input port of the computing unit, inputting one of the operands as the optical input; the readout region can serve as the electrical input port of the computing unit, inputting one of the operands as the electrical input, and serving as the output port of the computing unit, outputting the carriers after being acted upon by the optical input and the electrical input as the unit output; or the electrical input is input through other regions, and the readout region only serves as the output port of the computing unit, outputting the carriers after being acted upon by the optical input and the electrical input as the unit output.

[0077] In the above example, because the light input is actually the photogenerated carriers stored in the semiconductor device, these carriers can be stored in the optoelectronic computing unit for a long time relative to the computing speed (usually seconds, and even longer up to several years). Therefore, the light input is the storage input in the storage-computing integrated unit, and the collection area in the photogenerated carrier collection and readout area is the storage input end of the storage-computing integrated unit; the electrical input does not have the function of being stored in the unit for a long time, so the point input is the calculation input in the storage-computing integrated unit, and the readout area or the carrier control area in the photogenerated carrier collection and readout area is the calculation input end of the storage-computing integrated unit, depending on the specific working mode; the final calculation result of the optoelectronic computing unit is output in the form of current in the readout area in the photogenerated carrier collection and readout area, so the readout area in the photogenerated carrier collection and readout area is the output end of the storage-computing integrated unit.

[0078] The light emitted by the light-emitting unit is used as photons incident on the photo-generated carrier collection and readout area of ​​the computing unit and participates in the calculation.

[0079] Figure 2 Figure 1 is a schematic diagram of the structure of the optoelectronic computing array, where 1 is the light-emitting array and 2 is the computing array. Figure 2 As shown, the optoelectronic computing array includes a light emitting array 1 and a computing array 2. The light emitting array 1 is composed of a plurality of light emitting units arranged periodically, and the computing array 2 is composed of a plurality of computing units arranged periodically.

[0080] The optoelectronic computing unit described in this embodiment has the following three specific device implementation forms:

[0081] Example 1-1

[0082] Figure 3 1-1 is a cross-sectional view (a) and a stereoscopic view (b) of the calculation unit structure of Example 1-1. Figure 3 As shown, the computing unit of this embodiment includes: a control gate as a carrier control region, a charge coupling layer as a coupling region, and a P-type substrate as a photogenerated carrier collection region and a readout region. The P-type substrate is divided into a left collection region and a right readout region, wherein the right readout region includes a shallow trench isolation, an N-type source terminal and an N-type drain terminal formed by ion implantation. The shallow trench isolation is located in the middle of the semiconductor substrate, between the collection region and the readout region, and the shallow trench isolation is formed by etching and filling with silicon dioxide to isolate the electrical signals of the collection region and the readout region. The N-type source terminal is located on one side of the readout region close to the underlying dielectric layer and is formed by ion implantation. The N-type drain terminal is located on the other side of the semiconductor substrate close to the underlying dielectric layer opposite to the N-type source terminal and is also formed by ion implantation. It should be understood that the left side, right side, top and bottom mentioned in this article only represent that the relative positions under the viewing angle shown in the figure change with the viewing angle, and are not to be understood as limitations on specific structures.

[0083] A pulse with a negative voltage range is applied to the substrate of the collection area, or a pulse with a positive voltage range is applied to the control gate, so that a depletion layer for collecting photoelectrons is generated in the substrate of the collection area, and the number of collected photoelectrons is read out through the right readout area as the input quantity of the light input end. During readout, a positive voltage is applied to the control gate to form a conductive channel between the N-type source end and the N-type drain end of the collection area, and then a bias pulse voltage is applied between the N-type source end and the N-type drain end to accelerate the electrons in the conductive channel to form a current between the source and the drain. The carriers that form the current in the channel between the source and the drain are jointly affected by the control gate voltage, the voltage between the source and the drain, and the number of photoelectrons collected in the collection area, and are output in the form of current as electrons that are jointly affected by the light input and the electrical input, where the control gate voltage and the voltage between the source and the drain can be used as the electrical input of the device, and the number of photoelectrons is the light input of the device.

[0084] The charge coupling layer in the coupling region is used to connect the collection region and the readout region, so that after the depletion region in the collection region substrate starts to collect photoelectrons, the surface potential of the collection region substrate will be affected by the number of collected photoelectrons; through the connection of the charge coupling layer, the surface potential of the semiconductor substrate in the readout region is affected by the surface potential of the semiconductor substrate in the collection region, thereby affecting the magnitude of the source-drain current in the readout region, so that the number of photoelectrons collected in the collection region can be read out by judging the source-drain current in the readout region;

[0085] The control gate of the carrier control region is used to apply a pulse voltage thereon, so that a depletion region for exciting photoelectrons is generated in the readout region of the P-type semiconductor substrate. It can also be used as an electrical input terminal to input one of the calculation quantities.

[0086] In addition, there is a bottom dielectric layer for isolation between the P-type semiconductor substrate and the charge coupling layer; there is also a top dielectric layer for isolation between the charge coupling layer and the control gate.

[0087] Example 1-2

[0088] Figure 4 The cross-sectional view (a) and the stereoscopic view (b) of the computing unit structure of Example 1-2 are shown. Figure 4 As shown, the computing unit of this embodiment includes: a control gate as a carrier control region, a charge coupling layer as a coupling region, and a P-type semiconductor substrate as a photogenerated carrier collection region and a readout region, wherein the P-type substrate includes an N-type source terminal and a drain terminal formed by ion implantation. The P-type semiconductor substrate can simultaneously undertake the work of photosensitivity and readout. The N-type source terminal is located on one side of the readout region close to the underlying dielectric layer, and is formed by ion implantation. The N-type drain terminal is located on the other side of the semiconductor substrate close to the underlying dielectric layer opposite to the N-type source terminal, and is also formed by ion implantation.

[0089] During photosensitivity, a pulse with a negative voltage range is applied to the P-type semiconductor substrate, and a pulse with a positive voltage range is applied to the control gate as the carrier control area, so that a depletion layer for collecting photoelectrons is generated in the P-type substrate. The electrons generated in the depletion area are accelerated by the electric field between the control gate and the two ends of the P-type substrate, and when they reach a sufficiently high energy, they pass through the underlying dielectric layer barrier between the P-type substrate and the charge coupling layer, enter the charge coupling layer and are stored there. The amount of charge in the charge coupling layer will affect the threshold when the device is turned on, and then affect the source-drain current during reading; during reading, a pulse voltage is applied to the control gate to form a conductive channel between the N-type source terminal and the N-type drain terminal, and then a pulse voltage is applied between the N-type source terminal and the N-type drain terminal to accelerate the electrons in the conductive channel to form a current between the source and the drain. The current between the source and the drain is affected by the control gate pulse voltage, the voltage between the source and the drain, and the number of electrons stored in the charge coupling layer. The electrons, which are affected by the light input and the electrical input, are output in the form of current. The control gate voltage and the voltage between the source and the drain can be used as the electrical input of the device, and the number of photoelectrons stored in the charge coupling layer is the light input of the device.

[0090] The charge coupling layer in the coupling region is used to store the photoelectrons entering it and change the device threshold during readout, thereby affecting the source-drain current in the readout region. The number of photoelectrons generated during photosensitivity and entering the charge coupling layer can be read out by judging the source-drain current in the readout region.

[0091] The control gate of the carrier control region is used to apply a pulse voltage thereon, so that a depletion region for exciting photoelectrons is generated in the readout region of the P-type semiconductor substrate. It can also be used as an electrical input terminal to input one of the calculation quantities.

[0092] In addition, there is a bottom dielectric layer for isolation between the P-type semiconductor substrate and the charge coupling layer; there is also a top dielectric layer for isolation between the charge coupling layer and the control gate.

[0093] Examples 1-3

[0094] Figure 5 Schematic diagram of the structure of the computing unit (a) and the multi-functional area (b) of Embodiment 1-3. Figure 5 As shown, the calculation unit of this embodiment includes: a photodiode and a readout tube as a photogenerated carrier collection and readout region, wherein the photodiode is formed by ion doping and is responsible for light sensing. The N region of the photodiode is connected to the control gate of the readout tube and the source end of the reset tube through the photoelectron coupling lead as the coupling region, and a positive voltage pulse is applied to the drain end of the readout tube as the driving voltage of the readout current; before exposure, the reset tube is turned on, and the voltage at the drain end of the reset tube is applied to the photodiode, so that the photodiode as the collection region is in a reverse bias state, and a depletion layer is generated; during exposure, the reset tube is turned off, and the photodiode is electrically isolated. After photons enter the depletion region of the photodiode, photoelectrons are generated and accumulated in the diode, and the N region of the diode and the control gate of the readout tube electrically connected to the N region through the photoelectron coupling lead as the coupling region begin to decrease in potential, thereby affecting the electron concentration in the channel of the readout tube. The readout tube is responsible for reading, and a positive pulse voltage is applied to its drain end. The source end is connected to the drain end of the site selection tube. When reading, the site selection tube is turned on, and current is generated in the readout tube. The magnitude of the current is affected by the drain voltage of the reset tube, the drain voltage of the readout tube and the number of incident photons. The electrons in the readout tube channel, as electrons affected by the light input and the electrical input, are output in the form of current. The drain voltage of the reset tube and the drain voltage of the readout tube can be used as the electrical input of the device, and the number of incident photons is the light input of the device.

[0095] The optoelectronic coupling lead in the coupling region is used to connect the photodiode as the collection area in the photo-generated carrier collection and readout region and the readout tube as the readout region, and apply the potential of the photodiode N region to the readout tube control gate.

[0096] The reset tube, which serves as the carrier control area, inputs a positive voltage through its drain terminal to act on the photodiode. When the reset tube is turned on, the positive voltage will act on the photodiode, causing the photodiode to produce a depletion region and be sensitive to light. It can also be used as an electrical input terminal to input one of the calculation quantities.

[0097] In addition, the site selection tube is used to control the output of the output current of the entire computing device as the output quantity, and can be used for row and column site selection when the optoelectronic computing units form an array.

[0098] Example 2

[0099] The full name of RRAM is memory resistor. The device can be summarized as a special non-volatile (NVM) memory device that can switch between "high resistance state" and "low resistance state" and store the resistance value for a long time.

[0100] Figure 6 This is a schematic diagram of the RRAM device structure and an overview of its three terminals. Figure 6 As shown, the device usually consists of two layers of metal electrodes with a special through-hole layer group that can form a conductive through-hole. The through-hole layer is mostly composed of metal oxides, such as WO x , TaO x When the RRAM device is in the initial mode, the device is in a high-resistance state. When a large bias is applied to both ends of the device, the device enters the programming state, a conductive channel is formed in the special through-hole layer, and the conductive channel continues to exist and store the current resistance value after the voltage is reduced, until a large negative bias is applied and the device enters the erase state, the conductive channel is broken, and the device returns to the initial high-resistance state.

[0101] RRAM devices are used as storage and computing devices because they have the function of storing resistance values ​​for a long time. Therefore, their storage input ends are the two ends of the device when it is in the programming state. After the resistance input is completed, the device is in a low-resistance state and can be used as a linear resistor within a certain voltage range. The calculations required by the storage and computing unit can be completed by utilizing this linear resistance range. Therefore, its computing input ends are the two ends of the device within the linear resistance range. When a bias voltage within the linear resistance range is applied to both ends of the device, current flows from one end of the RRAM to the other end. Therefore, the end of the device from which the current flows out is the output end of the storage and computing device.

[0102] Because RRAM is usually a two-terminal device, its storage input terminal, computing input terminal, and output terminal are usually the same area in different operating modes.

[0103] Example 3

[0104] Flash memory (FLASH) is the most common non-volatile (NVM) memory device, and its basic storage unit is a floating gate device, such as a structure similar to the photoelectric computing unit described in Embodiments 1-2, or as Figure 7 The structure shown.

[0105] Figure 7 This is a basic cell structure diagram of a flash memory. Figure 7 As shown, EG and WL for erasing and selecting are added. The basic principle is to add a charge storage layer surrounded by an oxide isolation layer between the channel and the control gate of a normal MOSFET transistor, use this isolation storage layer to store charge to store data, and read out the stored charge by judging the threshold of the transistor. The isolation layer can be a floating gate made of polysilicon, such as Figure 7 The FG (floating gate) in the transistor can also be a nitride layer, etc. The charge storage in the isolation layer is mostly achieved through the channel hot electron injection (CHE) mechanism.

[0106] When using flash devices as integrated storage and computing devices, the charge stored in the isolated charge storage layer can be stored in the device for a long time. Therefore, the stored charge is the storage input in the integrated storage and computing device, and the storage input end is the hot electron injection end. This mechanism usually occurs just below the charge storage layer of the surface channel in the P-type substrate of the flash device, such as Figure 7 When the flash device is read out, the channel current of the MOSFET transistor is affected by the source-drain voltage Vds, the control gate voltage Vgs and the charge stored in the charge storage layer. Therefore, the input terminal can be the control gate of the flash device, such as Figure 7 The CG (coupling gate) or WL (wordline) in the memory may be the source and drain. Because the data after the power input and the storage input are combined flows through the flash source and drain in the form of current, the output end of the flash device as a storage and computing device is the source and drain.

[0107] Example 4

[0108] This embodiment uses any one of the above embodiments as a storage and calculation integrated unit to perform the calculation of the pulse convolutional neural network, and has the following specific implementation methods:

[0109] Example 4-1

[0110] Taking MNIST as an example, the dataset size is 10000*28*28, with a total of 10,000 sets of test data, the image size is 28*28, the number of channels is 1, the data is a floating point number between 0-1, and the number of categories is 10.

[0111] Taking Lenet-5 as an example, the pooling layer can be either maximum pooling or average pooling. For the specific network structure, see Figure 8 and Fig. 9 .in Figure 8 This is the structural diagram of Spiking-Lenet-5 (average pooling). Fig. 9 This is a schematic diagram of the structure of Spiking-Lenet-5 (maximum pooling).

[0112] Specifically, Figure 8 and Fig. 9 The input image size in is 28*28, and each pixel value needs to be converted into a binary number with a bit width of width. The first layer is the convolution layer, the convolution kernel size is 5*5, the number is 6, and each weight needs to be copied into a geometric progression with a ratio of 1 / 2, a total of width, and multiplied with the 0 / 1 of different bits of the same pixel value; the size of each convolution window is 5*5, because the first layer has only one channel. If it is a multi-channel input, each convolution kernel should also have multiple channels, and the pixel value in each channel is multiplied with the convolution kernel weight; for the 5*5 convolution window, 5*5 pixel values ​​are selected on the input image, and the pixel values ​​at the same position are multiplied with the convolution kernel weight; the result of all the products and accumulations in the same convolution window corresponds to the increment of the integral value in a neuron current integral comparison circuit; sliding the convolution window on the input image in a fixed order corresponds to different neurons; then changing different convolution kernels corresponds to a different group of neurons.

[0113] like Figure 8 and Fig. 9 As shown in the figure, the 28*28*6 above the convolution layer 1 is the total number of neurons in the convolution layer 1, 28*28 is the size of the output image (when performing convolution operation, the insufficient part of the image edge in the convolution window is filled with 0), and 6 corresponds to the number of convolution kernels, indicating the number of channels of the output image.

[0114] exist Figure 8In the example, the 28*28*6 output image is directly used as the input of convolution layer 2 to start a new calculation. Because the average pooling method is used here, the original 28*28 image should be averaged 2*2 to generate a 14*14 image. Here, the 4 pixels corresponding to any pixel in the 14*14 image in the 28*28 image are directly integrated into the same convolution window. Accordingly, the original 5*5 convolution window becomes 10*10, and the weights corresponding to adjacent 2*2 pixels are the same. The convolution calculation process is similar to that of convolution layer 1.

[0115] And in Fig. 9 In the example, because the maximum pooling is used, it is necessary to add the maximum pooling layer 1 before the convolution layer 2, and the function is 4-choose-1, so that the input image size of the convolution layer 2 is 14*14.

[0116] For the fully connected layer, the matrix-vector multiplication operation is performed directly. Figure 8 The 1600*120 in the figure is based on the 400*120 because of the average pooling replication weights.

[0117] The last 10 counters count the number of high levels in the output pulse signals of the 10 neurons in the fully connected layer 3. According to the specific implementation plan of the system, 10 counters can be added to record the earliest time when each neuron generates a high level.

[0118] In the host computer, the convolutional neural network must be trained first. The trained convolutional neural network is calculated according to the following formula:

[0119]

[0120]

[0121] Among them, I is the input of a layer of the convolutional neural network, W is the weight, B is the bias, O is the output, channel is the number of input channels, kernelsize is the convolution kernel size, ii is the row of the output image, jj is the column of the output image, and nn is the channel of the output image.

[0122] Then the obtained weights W and bias B of each layer and the input data from the data set, that is, I of the first layer, are processed as follows:

[0123] First consider the input data from the data set. Between the quantization bit width of the storage input and the input bit width of the data set, select the smaller value as the system bit width. Quantize the grayscale value in the data set according to the false data bit width to obtain a binary number of width bits. The insufficient bits are filled with zeros in the high bit. The original input data is expanded into a binary number of width times, that is, a pulse signal.

[0124] Consider weights and biases. If there is a BN layer (batch normalization) in the convolutional neural network, then during training, you need to derive bn.weight(γ), bn.bias(β), bn.running_mean(mean), bn.running_var(var), and bn.eps (eps, a small amount added to the denominator, the default is 1e-5), where bn.weight(γ) represents: the scaling factor learned during training; bn.bias(β) represents: the offset factor learned during training; bn.running_mean(mean) represents: the statistical mean of the data obtained during training; bn.running_var(var) represents: the statistical variance of the data obtained during training. And modify the weights W and bias B of the convolutional layer or fully connected layer before the BN layer according to the following formula:

[0125]

[0126]

[0127] In this way, the convolutional layer or fully connected layer is merged with the BN layer. When performing inference tasks, only the convolutional layer or fully connected layer calculations of the modified W' and B' need to be retained, without the need for redundant BN layer operations.

[0128] There are some special cases. For the first layer, assuming that the grayscale value of the input image should be multiplied by a weight W' in a convolution kernel and accumulated according to the calculation formula of the convolutional neural network, then the weight of the first layer is copied into width copies, and then kept unchanged, divided by 2, divided by 4, and other powers of 2, and the corrected weight is recorded as W". The bias of this layer should be multiplied by 2 based on the above correction B', recorded as B". Among them, the binary numbers obtained by quantizing the grayscale values ​​are sorted in order from high to low, corresponding to W', W' / 2, W' / 4, W' / 8...

[0129] If the average pooling layer is used in the convolutional neural network, then each weight W' in the next convolutional layer or fully connected layer of the average pooling layer will be copied into several copies, the number of which is the square of the size of the pooling layer. For example, if the pooling layer is 2*2, then each weight is copied into 4 copies, and the corrected weight is recorded as W". If the layer has a bias, the bias value is amplified by 4 times based on B' and recorded as B".

[0130] At this point, at the convolutional neural network level, the processing of input, weights and biases has been completed. Consider the pulse convolutional neural network level.

[0131] First, the user scales the weights of each layer according to actual needs (for example, scaling and adjusting according to the actual quantization bit width and the maximum absolute value of the weight of the layer to achieve the highest possible accuracy), and sets the new weight to W''.

[0132] Then, based on the principles of pulse convolutional neural networks, the bias of each layer is corrected.

[0133] For the first layer, the calculation formula is O'=I'*W'+B'. The complex summation expression in the convolution calculation formula above is omitted here. Although the form here is slightly different, the only difference is that I is changed into several inputs I' of binary expansion, and W' is also processed into several times W'. Since the relationship between W' and B' should be able to calculate O' corresponding to O in the convolutional neural network at any time, that is, if I'*W'=A1*(I'*W"), then B'=A1*B, O'=A1*O, A1 is a scaling ratio.

[0134] Consider the second layer. For the second layer, its input I'' is the O'' of the first layer accumulated over time. After that, it generates 1 once, otherwise it generates 0. Assuming that this time is T1, that is, For I', there is only one 1 in every T1 time, and the rest are all zero. Assume that I'*W'+B' of the second layer is accumulated according to time, and every time it exceeds the threshold After that, it generates 1 once, otherwise it generates 0. Assuming that this time is T2, that is, Substituting the formula of the first layer into the equation: Since in the convolutional neural network, the output of the second layer = O"*W"+B", where W"' = A2*W", and O"' = A1*O", then The same deduction can be made for the nth layer: The hyperparameters vth in the denominator + These are the values ​​set by the user on the host computer.

[0135] All weights and biases should be binary quantized according to the bit width of width on the basis of correction to obtain the final value written into the storage input end of the storage and computing integrated unit, denoted as W"" and B"".

[0136] All the above work is completed in the host computer. After completion, the weights and biases are sorted according to the calculation formula of the convolutional neural network and written into the storage and computing unit.

[0137] After all the inputs at the storage input end are completed, the host computer sends an input pulse to the calculation input end of the first-layer storage and calculation integrated unit, and this device starts to perform the calculation task.

[0138] Fig.10 This is a schematic diagram of a neuron composed of storage and calculation units in Example 4-1. In the pulse convolutional neural network algorithm, except that the correspondence between input and weight is consistent with the convolutional neural network algorithm, all the circulating data are pulse signals, that is, 0 or 1, and the basic calculation unit is the storage and calculation unit, which is responsible for multiplication. On this basis, Fig.10 As shown, a neuron includes multiple storage-computation units. The inputs of the storage input terminals of these storage-computation units correspond to the synapses of neurons in the human brain, that is, W", and the inputs of the computation input terminals correspond to the synaptic connection strength, that is, I". In addition, the neuron must also have a cell body, which is responsible for accumulating the output results ∑I""·W""+1*B"" of these storage-computation units in each clock cycle, and accumulating them with the potential v(t-1) of the neuron cell body at this time.

[0139] The formula is:

[0140] v(t)=v(t-1)+∑I″″·W″″+1*B″″

[0141] The output pulse generation formula is:

[0142]

[0143] After the output pulse is generated, the neuron potential undergoes the following changes, all of which are completed before the next clock cycle arrives:

[0144]

[0145] where vth - and vth + These are hyperparameters that can be set by each layer. + is the positive threshold, vth - is the negative threshold. For unbiased neural networks, the negative threshold can also be set to 0.

[0146] This function is implemented by the current integration comparison circuit, and the output result is saved in the register, aligned with the rising edge of the clock, and transmitted to the neurons in the next layer.

[0147] For a convolutional layer or a fully connected layer that needs to be biased, its corrected bias value has been stored in a storage-computation unit in each neuron of the layer. It is only necessary to always set the computation input of the storage-computation unit to 1.

[0148] In some networks, zero padding may be required in many convolutional layers. Since the positions of zero padding are fixed, just set the corresponding input to 0.

[0149] For the average pooling layer, it is necessary to directly connect all inputs that need average pooling to the corresponding neurons in the next convolutional layer or fully connected layer, and copy the weights multiple times (already completed in the host computer) to achieve multiplication and accumulation in the same proportion as the original ones.

[0150] For the maximum pooling layer, it is not necessary to operate in the next convolutional layer or fully connected layer, but it is necessary to add an additional judgment condition between the two adjacent convolutional layers or fully connected layers, that is, from the beginning of the calculation, the input signal corresponding to each pooling window is selected with the earliest 1, and connected to the next convolutional layer or fully connected layer, and the rest of the input signals can be ignored. The specific implementation here is some digital logic plus a multiplexer.

[0151] Behind the fully connected layer 3 (the last fully connected layer), there are 10 counters that have been counting the number of pulses (high level) received in these ten categories and sending them to the host computer through the control system.

[0152] Solution 1: The values ​​of 10 counters are transmitted to the host computer in real time.

[0153] In the host computer, the following end condition judgment is required: when at a certain moment, the number of pulses in one type of counter is a more than that in other types, a is a set constant, the calculation is considered to be finished, and the type number with the largest number of pulses is output. It is recommended to set it to 4.

[0154] If the end judgment condition is not met after the set maximum time, it will be forced to end and find the category with the largest number of pulses among the 10 categories.

[0155] If there are at least two categories with the same number of pulses, then compare which one receives the pulse first and output that category.

[0156] After the calculation of the image is completed, the host computer sends the corresponding control signal to the control system to clear and reset some parts of the system that need to be cleared, and then sends the input pulse signal of the next image to start the next round of calculation.

[0157] Solution 2: There is no way to transmit the values ​​of the 10 counters to the host computer in real time.

[0158] In the hardware part, the values ​​of these 10 counters are transmitted to a special end condition judgment module, which needs to implement the following function: when at a certain moment, the number of pulses in one type of counter is a more than that in other types, and a is a set constant (it is recommended to be set to 4), it is considered that the calculation can be ended. Or after the set maximum time, if the end judgment condition is not met, it is forced to end. The end here means pulling up an output end signal, transmitting it to the control system and the host computer, resetting the corresponding position of the hardware part, and the host computer transmitting new image data to the control system. The control system saves the image data sent by the host computer and distributes the image data that needs to be calculated next to the storage and computing unit. (Here, depending on the usage of the memory capacity in the actual system, there can be several different data transmission schemes, which are not limited)

[0159] In addition to the end signal, when the end signal is pulled high, the values ​​of these 10 counters need to be transmitted to the host computer through the control system. In addition, in addition to these 10 counters, 10 counters need to be set after the fully connected layer 3 to record the earliest time when these 10 neurons generate high-level outputs. These 10 counters will also be transmitted to the host computer.

[0160] In the host computer, it is necessary to first find the maximum value among the 10 counters that count the number of high levels. If there are the same ones, the one that generates the high level earliest is selected as the final classification result.

[0161] The block diagram of the whole system is shown in Fig.11 and Fig.12 . Fig.11 is the block diagram of the entire system (average pooling). Fig.12 This is the block diagram of the entire system (maximum pooling). CONV represents the convolutional layer and FC represents the fully connected layer. Figure 8 , 9 The Spiking-Lenet-5 pulse convolutional neural network structure diagram described, Fig.11 and Fig.12 Each layer is implemented with hardware, and data flows in different modules. In addition, the hardware part also has a control system, which is used to receive input data and control signals from the host computer, and then distribute them to the Conv1 module, and receive statistical results from the counter module and send them to the host computer.

[0162] The calculation flow chart of the whole system is shown in Fig.13 and Fig.14 . Fig.13 It is the calculation flow chart of the whole system (average pooling). Fig.14 This is the computational flow chart of the entire system (maximum pooling). CONV represents the convolutional layer and FC represents the fully connected layer.

[0163] Before calculating the image, the trained weights and biases need to be corrected and written into the storage input of the storage-computing integrated unit. After that, all modules of the entire hardware accelerator are reset except for the data written into the storage input. Then the host computer starts to transmit input data to the hardware accelerator, and the control system receives these data. After all the input data of the first image is transmitted, it starts to distribute data to the storage-computing integrated unit at the same time. Since the input data of the same image remains unchanged until the image is calculated, according to the specific design of the system, the input data of the next image can be transmitted after one image is calculated, or the input data of the next or several images can be saved in the hardware accelerator before the first image is calculated, so as to realize ping-pong operation.

[0164] For each convolutional layer or fully connected layer module, the storage-computation integrated unit receives the input signal from the computing input terminal, and the calculation results of all storage-computation integrated units are added in series to input the current into the current integration comparison circuit, which is integrated and compared with the threshold, and then generates an output pulse, which is aligned with the rising edge of the clock in the next register to obtain the output of this layer. These modules are performing independent operations at the same time and non-stop.

[0165] exist Fig.14 In the figure, there is also a maximum pooling module. The outputs of conv1 and conv2 are first connected to maximum pooling 1 and 2. In the maximum pooling module, the one with the earliest high level is selected and transmitted to the next layer.

[0166] Regarding the counter group, Solution 1: The counter group counts the number of high-level pulses in the output pulses of each neuron in the last fully connected layer, and the result is transmitted back to the host computer by the control system. The host computer determines whether the calculation of this picture is completed according to the conditions set by the user. If not, it continues to maintain the status quo. If completed, it changes the control signal, resets the current integral comparison circuit in the hardware accelerator and other registers and counters in the system, and starts transmitting a new picture.

[0167] Solution 2: The counter group counts the number of high-level pulses in the output pulses of each neuron in the last fully connected layer, as well as the earliest time when each neuron generates a high-level pulse. The result is transmitted back to the host computer by the control system only after the current round of calculation is completed. The judgment of the end of this round of calculation is completed by digital logic based on the statistical results of the counter group. If it is not finished, the status quo will continue. If it is finished, a high end signal is sent to the control system to reset the current integral comparison circuit in the hardware accelerator and other registers and counters in the system. The storage and calculation integrated unit waits for the control system to distribute the data of the next image. After receiving the end signal, the host computer transmits a new picture to the control system and processes the data of the counter group sent back to obtain the final classification result.

[0168] According to the above embodiment, by converting the data in the convolutional neural network into a time pulse sequence, the current integral comparison circuit is used to replace the analog-to-digital converter with large power consumption and area, thereby greatly reducing the area and power consumption of the entire system. In addition, the output results of each convolutional layer / fully connected layer are directly connected to the next convolutional layer / fully connected layer, and the weight data can be directly stored in the storage and computing integrated unit. No on-chip cache is required in the entire system, which saves a lot of data transfer process, thereby speeding up the calculation speed.

[0169] Example 4-2

[0170] Based on Example 4-1, this embodiment adds a function of synchronizing the output result of the integral comparison circuit with the clock signal in the integral comparison circuit. Each register connected to the integral comparison circuit in Example 4-1 is removed, and the output of the integral comparison circuit is directly connected to the next layer of neurons, the maximum pooling module or the counter. See the schematic diagram of the neuron without the register. Fig.15 The block diagram of the entire system (excluding registers and average pooling) is shown in Fig.16 , the block diagram of the entire system (excluding registers and maximum pooling) can be found in Fig.17 .

[0171] Example 4-3

[0172] Taking Cifar-10 as an example, the dataset size is 10000*32*32*3, with a total of 10,000 sets of test data, the image size is 32*32, the number of channels is 3, the data is an integer from 0 to 255, and the number of categories is 10.

[0173] Taking Alexnet as an example, the model used here has been changed. The BN layer is added immediately after the first and second convolutional layers, the pooling layer is changed to average pooling, and the convolution kernel size of all convolutional layers is 3*3. For the specific network structure, see Fig.18 .

[0174] Specifically, Fig.18 The input image size is 32*32, the number of channels is 3, and each pixel value needs to be converted into a binary number with a bit width of width. The first layer is the convolution layer, the convolution kernel size is 3*3, the number of channels is 3, the number is 96, and each weight needs to be copied into a geometric progression with a ratio of 1 / 2, a total of width, and multiplied with the 0 / 1 of different bits of the same pixel value; the convolution kernels of the three channels correspond to the input image; the convolution window size is 3*3, 3*3 pixel values ​​are selected on the input image, and the pixel values ​​at the same position are multiplied with the convolution kernel weights; the result of all the products and accumulations in the same convolution window corresponds to the increment of the integral value in a neuron current integral comparison circuit; sliding the convolution window on the input image in a fixed order corresponds to different neurons; then changing different convolution kernels corresponds to a different group of neurons.

[0175] like Fig.18 As shown in the figure, the 32*32*96 above the convolution layer 1 is the total number of neurons in the convolution layer 1, 32*32 is the size of the output image (when performing convolution operations, the insufficient part of the image edge in the convolution window is filled with 0), and 96 corresponds to the number of convolution kernels, indicating the number of channels of the output image. The 3 output images are directly used as the input of the convolution layer 2 to start a new calculation. Because the average pooling method is used here, the original 32*32 image should be averaged 2*2 to generate a 16*16 image. Here, the 4 pixels corresponding to any pixel in the 16*16 image in the 32*32 image are directly integrated into the same convolution window. Accordingly, the original 3*3 convolution window becomes 6*6, and the weights corresponding to adjacent 2*2 pixels are the same. The convolution calculation process is similar to that of the convolution layer 1. The same is true for other convolution layers.

[0176] For the fully connected layer, the matrix-vector multiplication operation is performed directly. Fig.18 The 16384*1024 in the figure is based on 4096*1024 because of the average pooling replication weights.

[0177] The last 10 counters count the number of high levels in the output pulse signals of the 10 neurons in the fully connected layer 3. According to the specific implementation plan of the system, 10 counters can be added to record the earliest time when each neuron generates a high level.

[0178] In the host computer, the convolutional neural network must be trained first. The trained convolutional neural network is calculated according to the following formula:

[0179]

[0180]

[0181] Among them, I is the input of a layer of the convolutional neural network, W is the weight, B is the bias, O is the output, channel is the number of input channels, kernelsize is the convolution kernel size, ii is the row of the output image, jj is the column of the output image, and nn is the channel of the output image.

[0182] Then the obtained weights W and bias B of each layer and the input data from the data set, that is, I of the first layer, are processed as follows:

[0183] First consider the input data from the data set. Between the quantization bit width of the storage input and the input bit width of the data set, select the smaller value as the system bit width. Quantize the RGB values ​​in the data set according to the false data bit width to obtain a binary number of width bits. The insufficient bits are filled with zeros in the high bit. The original input data is expanded into a binary number times width, that is, a pulse signal.

[0184] Consider weights and biases. If there is a BN layer (batch normalization) in the convolutional neural network, then during training, you need to derive bn.weight(γ), bn.bias(β), bn.running_mean(mean), bn.running_var(var), and bn.eps (eps, a small amount added to the denominator, the default is 1e-5), where bn.weight(γ) represents: the scaling factor learned during training; bn.bias(β) represents: the offset factor learned during training; bn.running_mean(mean) represents: the statistical mean of the data obtained during training; bn.running_var(var) represents: the statistical variance of the data obtained during training. And modify the weights W and bias B of the convolutional layer or fully connected layer before the BN layer according to the following formula:

[0185]

[0186]

[0187] In this way, the convolutional layer or fully connected layer is merged with the BN layer. When performing inference tasks, only the convolutional layer or fully connected layer calculations of the modified W' and B' need to be retained, without the need for redundant BN layer operations.

[0188] There are some special cases. For the first layer, assuming that the RGB value of the input image should be multiplied by a weight W' in a convolution kernel and accumulated according to the calculation formula of the convolutional neural network, then the weight of the first layer is copied into width copies, and then kept unchanged, divided by 2, divided by 4, and other powers of 2, and the corrected weight is recorded as W". The bias of this layer should be multiplied by 2 based on the above correction B', recorded as B". Among them, the binary numbers obtained by quantizing the RGB values ​​are sorted in order from high to low, corresponding to W', W' / 2, W' / 4, W' / 8...

[0189] If the average pooling layer is used in the convolutional neural network, then each weight W' in the next convolutional layer or fully connected layer of the average pooling layer will be copied into several copies, the number of which is the square of the size of the pooling layer. For example, if the pooling layer is 2*2, then each weight is copied into 4 copies, and the corrected weight is recorded as W". If the layer has a bias, the bias value is amplified by 4 times based on B' and recorded as B".

[0190] At this point, at the convolutional neural network level, the processing of input, weights and biases has been completed. Consider the pulse convolutional neural network level.

[0191] First, the user scales the weights of each layer according to actual needs (for example, scaling and adjusting according to the actual quantization bit width and the maximum absolute value of the weight of the layer to achieve the highest possible accuracy), and sets the new weight to W''.

[0192] Then, based on the principles of pulse convolutional neural networks, the bias of each layer is corrected.

[0193] For the first layer, the calculation formula is O'=I'*W'+B'. The complex summation expression in the convolution calculation formula above is omitted here. Although the form here is slightly different, the only difference is that I is changed into several inputs I' of binary expansion, and W' is also processed into several times W'. Since the relationship between W' and B' should be able to calculate O' corresponding to O in the convolutional neural network at any time, that is, if I'*W'=A1*(I'*W"), then B'=A1*B, O'=A1*O, A1 is a scaling ratio.

[0194] Consider the second layer. For the second layer, its input I'' is the O'' of the first layer accumulated over time. After that, it generates 1 once, otherwise it generates 0. Assuming that this time is T1, that is, For I', there is only one 1 in every T1 time, and the rest are all zero. Assume that I'*W'+B' of the second layer is accumulated according to time, and every time it exceeds the threshold After that, it generates 1 once, otherwise it generates 0. Assuming that this time is T2, that is, Substituting the formula of the first layer into the equation: Since in the convolutional neural network, the output of the second layer = O"*W"+B", where W"' = A2*W", and O"' = A1*O", then The same deduction can be made for the nth layer: The hyperparameters vth in the denominator + These are the values ​​set by the user on the host computer.

[0195] All weights and biases should be binary quantized according to the bit width of width on the basis of correction to obtain the final value written into the storage input end of the storage and computing integrated unit, denoted as W"" and B"".

[0196] All the above work is completed in the host computer. After completion, the weights and biases are sorted according to the calculation formula of the convolutional neural network and written into the storage and computing unit.

[0197] After all the inputs at the storage input end are completed, the host computer sends an input pulse to the calculation input end of the first-layer storage and calculation integrated unit, and this device starts to perform the calculation task.

[0198] Fig.19 This is a schematic diagram of a neuron composed of storage and calculation units in Example 4-2. In the pulse convolutional neural network algorithm, in addition to the correspondence between input and weights, which is consistent with the convolutional neural network algorithm, all circulating data are pulse signals, that is, 0 or 1, and the basic calculation unit is the storage and calculation unit, which is responsible for multiplication. On this basis, Fig.19 As shown, a neuron includes multiple storage-computation units. The inputs of the storage input terminals of these storage-computation units correspond to the synapses of neurons in the human brain, that is, W", and the inputs of the computation input terminals correspond to the synaptic connection strength, that is, I". In addition, the neuron must also have a cell body, which is responsible for accumulating the output results ∑I""·W""+1*B"" of these storage-computation units in each clock cycle, and accumulating them with the potential v(t-1) of the neuron cell body at this time.

[0199] The formula is:

[0200] v(t)=v(t-1)+∑I″″·W″″+1*B″″

[0201] The output pulse generation formula is:

[0202]

[0203] After the output pulse is generated, the neuron potential undergoes the following changes, all of which are completed before the next clock cycle arrives:

[0204]

[0205] where vth - and vth + These are hyperparameters that can be set by each layer. + is the positive threshold, vth - is the negative threshold. For unbiased neural networks, the negative threshold can also be set to 0.

[0206] This function is implemented by the current integration comparison circuit, and the output result is saved in the on-chip cache. For each neuron, it is necessary to collect the output pulse signal of a fixed duration as a data packet. The fixed duration is consistent with the transmission duration of the input pulse. When all the data packets required for the calculation of the next layer of the output result are cached, they will be transmitted to the next layer of neurons in the form of data packets, and the accumulated value in the current integration comparison circuit will be cleared. The capacity of the on-chip cache and the number of neurons required need to be considered based on the actual situation, taking into account the balance of area, power consumption, speed and calculation speed of each layer.

[0207] For a convolutional layer or a fully connected layer that needs to be biased, its corrected bias value has been stored in a storage-computation unit in each neuron of the layer. It is only necessary to always set the computation input of the storage-computation unit to 1.

[0208] In some networks, many convolutional layers may need to be padded with zeros. Since the positions of the zeros are fixed, it is sufficient to set the corresponding input to 0 throughout its transmission duration.

[0209] For the average pooling layer, it is necessary to directly connect all inputs that need average pooling to the corresponding neurons in the next convolutional layer or fully connected layer, and copy the weights multiple times (already completed in the host computer) to achieve multiplication and accumulation in the same proportion as the original ones.

[0210] After the fully connected layer 3, there are 10 counters that keep counting the number of pulses received in these ten categories and send them to the host computer through the control system. Additional logic circuits are needed here to determine when there is a pulse, because the running time of the last layer is very short compared to the previous layers and it is not always running. According to the specific implementation plan of the system, 10 counters can also be added to record the earliest time when each neuron generates a high level.

[0211] In the host computer, it is necessary to compare the number of 10 pulses received within a fixed time length and select the maximum value. If 10 counters are added to record the earliest time when each neuron generates a high level, an auxiliary comparison can be performed: if the number of pulses in at least 2 categories is the same, then compare who receives the pulse first and output that category.

[0212] After the calculation of the image is completed, the host computer sends the corresponding control signal to the control system to clear and reset some parts of the system that need to be cleared, and then sends the input pulse signal of the next image to start the next round of calculation.

[0213] The block diagram of the whole system is shown in Fig. 20 If combined with Fig.18 The Spiking-Alexnet pulse convolutional neural network structure diagram described, Fig. 20 Each layer is implemented by hardware, and data circulates in different modules. In addition, the hardware part also has a control system, which is used to receive input data and control signals from the host computer, write the input data into the on-chip cache, and receive the statistical results from the counter module, and then send them to the host computer. In particular, the entire system also has an on-chip cache and a corresponding logic circuit. The logic circuit receives the control signal of the control system, generates the control signal and storage address of the on-chip cache according to the order of data distribution calculation, and saves the output pulses received from Conv1~Conv5 to the on-chip cache; according to the actual cache capacity, the data that will no longer be used in the cache is overwritten by new data; according to the order of data distribution calculation, the control signal and reading address of the on-chip cache are generated, and the input data required for Conv1~Conv5 and FC1 calculation are read out.

[0214] The calculation flow chart of the whole system is shown in Fig.21 .like Fig.21 As shown in the figure, before calculating the image, the trained weights and biases need to be corrected and written into the storage input of the storage-computation integrated unit. After that, all modules of the entire hardware accelerator are reset except for the data written into the storage input. Then the host computer starts to transmit input data to the hardware accelerator, the control system receives these data and writes them into the on-chip cache. After all the input data required for the Conv1 module to perform a calculation are transmitted, the on-chip cache starts to distribute data to the storage-computation integrated unit of the Conv1 module. After all the data are distributed, the Conv1 module starts to calculate. At this time, the transmission speed of the host computer should be considered in combination with the calculation speed of each module of the entire system and the on-chip cache capacity. However, it should be ensured that after a fixed time T, the data required for the next calculation of the Conv1 module has been stored in the on-chip cache within T time.

[0215] For each convolutional layer or fully connected layer module, the storage-computation integrated unit receives the input signal from the computing input terminal. The calculation results of all storage-computation integrated units are connected in series to add the current and input into the current integration comparison circuit. After integration and comparison with the threshold, the output pulse is generated in the circuit, and the operation of aligning with the rising edge of the clock is completed in the next register to obtain the output of the layer. These modules are performing independent operations at the same time. The continuous working time of each module is in units of a fixed duration T. Within T clock cycles, the storage-computation integrated unit and the current integration comparison circuit are continuously and uninterruptedly calculating. After the end of the T clock cycles, the current integration comparison circuit receives a control signal from the control system, performs a reset operation, and waits for the start of the next calculation.

[0216] For each convolution or fully connected 1 module, the input signal of the input terminal comes from the on-chip cache, that is, for each calculation of these modules with a duration of T clock cycles, the prerequisite for starting is that all the inputs required for this calculation have been read out from the on-chip cache. For the fully connected 2 and fully connected 3 modules, their input signals come from the output signals of the previous fully connected layer. For each convolution module, the output signal of each neuron will be packaged in units of T and stored in the on-chip cache.

[0217] The counter group counts the number of high levels in the output pulses of each neuron in the last fully connected layer. When the fixed calculation time is over, the result is transmitted back to the host computer by the control system. The host computer changes the control signal to reset the current integration comparison circuit and its registers and counters in the hardware accelerator and start transmitting a new picture. According to the specific implementation plan of the system, 10 counters can also be added to record the earliest time when each neuron generates a high level, which is used for auxiliary judgment of the classification result.

[0218] According to the above embodiment, for a large-scale network, by using on-chip cache to store part of the data, the required hardware resources can be greatly reduced by exchanging time for space.

[0219] Example 4-4

[0220] This embodiment is based on the embodiment 4-3, and the on-chip cache is changed to a register. The logic control circuit should also be modified accordingly because the positioning method of the register is different from that of the on-chip cache. Fig. 22 .

[0221] Embodiment 4-5

[0222] Based on Example 4-3, this embodiment changes the on-chip cache to an off-chip memory. For the hardware accelerator part, it only includes the storage and calculation integrated unit and the counter part of each layer. The on-chip cache and its logic control circuit in Example 4-3 are moved off-chip, and their functions are replaced by FPGA development board (field programmable gate array) and DDR (double data rate synchronous dynamic random access memory). See the system block diagram Fig.23 .

[0223] Embodiment 4-6

[0224] Based on Example 4-3, this embodiment changes the on-chip cache to off-chip cloud storage. For the hardware accelerator part, it only includes the storage and calculation integrated unit and counter part of each layer. The on-chip cache and its logic control circuit in Example 4-3 are moved off-chip, and their functions are replaced by the host computer and cloud storage. See the system block diagram Fig.24 .

[0225] Example 5

[0226] This embodiment uses any one of the above embodiments as a storage and calculation integrated unit to perform the calculation of the pulse convolutional neural network, and has the following specific implementation methods:

[0227] Fig.25 Alexnet network structure diagram of Example 5. Fig.25 As shown, the data set takes cifar-10 as an example. The data set size is 10000*32*32*3, of which there are 10000 groups of test data, the input image size is 32*32, the number of channels is 3, the data is an integer between 0-255, and the number of categories is 10.

[0228] Taking Alexnet as an example of a convolutional neural network, the model used here has been changed. A BN layer is added immediately after the first and second convolutional layers. The pooling layer can be either maximum pooling or average pooling, and the convolution kernel size of all convolutional layers is 3*3.

[0229] The output of each convolutional layer is obtained according to the following formula:

[0230]

[0231] Where I is the input of the layer, W is the weight, B is the bias, O is the output, channel is the number of input channels, and kernelsize is the convolution kernel size, all of which are 3 here.

[0232] The output of each fully connected layer is obtained according to the following formula:

[0233]

[0234] Where I is the input of the layer, W is the weight, B is the bias, O is the output, and channel is the number of input channels.

[0235] Now, based on the above convolutional neural network, a pulse convolutional neural network is generated. The most basic computing unit of the pulse convolutional neural network is a storage-computation integrated unit, which is responsible for completing multiplication. Fig.28 is a structural diagram of a neuron in Example 5. Fig.28 As shown, a neuron includes multiple storage-computation integrated units, and the storage-computation integrated units are similar to those described in the above embodiment 4 and will not be repeated here.

[0236] For a layer Layer M that needs to be added to the BN layer, during training, you need to export bn.weight(γ), bn.bias(β), bn.running_mean(mean), bn.running_var(var) and bn.eps (eps, a small amount added to the denominator, the default is 1e-5), and modify the weights and biases of Layer M according to the following formula:

[0237]

[0238]

[0239] In this way, the Layer M (convolutional layer or fully connected layer) and the BN layer are merged. When performing inference tasks, only the Layer M with modified W and B needs to be retained, without the need for additional BN layer operations.

[0240] Consider the input data of the first layer of the pulse convolutional neural network. Assume that the pixel value is 64, which is converted to binary as 01000000. 0-255 can be represented by an 8-bit binary number, and the insufficient bits are padded with zeros in the high bit. Assume that the pixel value should be multiplied by a certain weight W in a convolution kernel. Then the original 64*W needs to be converted to 128*(0*W+1*W / 2+0*W / 4+0*W / 8+0*W / 16+0*W / 32+0*W / 64+0*W / 128), that is, the input is expanded to 8 times the original, and the weight is copied 7 times, and then divided by different exponential powers of 2, and then accumulated. And the input remains unchanged until the end of the whole round of calculation. The whole round of calculation corresponds to a set of test data in the data set, which is a 32*32*3 image in Cifar-10. In addition, if the layer has a bias, if the weight of the layer maintains the original ratio, then its bias value should be multiplied by 2 based on the above correction formula; if the weight of the layer is scaled as a whole, then the bias value should also be scaled by the same ratio.

[0241] Compared with the existing pulse convolutional neural network algorithm, the input pulse is generated by a random number, that is, a decimal of 0-1 is randomly generated and compared with the pixel value / 255. If the random number is smaller than this number, a pulse is generated, otherwise no pulse is generated. However, because this method has a large randomness, it can only be as close to the original pixel value as possible after calculating a large number of pulses. In the algorithm of the present invention, the input pulse is completely equivalent to the original pixel value, and a large number of pulses are not required.

[0242] In addition, according to actual needs, there will be cases where the weights are quantized. That is, assuming that the weights can only be represented by WW bits of binary numbers at most, for the case where WW is not less than 8 (the 8-bit binary number required for 0-255, that is, not 8 for input data with a larger range), the input data and weight replication are as described above. However, if WW is less than 8, then the weights divided by the larger exponential power of 2 may be directly equal to 0, and the input has no effect on it, so it can be directly omitted. That is, the input takes the WW bits starting from the high bit, and the copied weights also take the WW bits with larger absolute values.

[0243] For the bias correction of each layer, in addition to the correction caused by the threshold, the weight scaling of the previous layers will be accumulated in this layer. For example, after the weight of the first layer is adjusted, the final result accumulated in the integral comparison circuit is twice the theoretical value in the convolutional neural network model. In addition to the bias of the first layer needing to be doubled, in the second layer, this 2 times will still be reflected in the frequency of the input pulse, that is, the I in the I*W+B formula of the second layer is already twice the original, so B should also be doubled accordingly. The same is true for other layers. In short, the correction basis is that the scaling multiples of I*W and B are consistent.

[0244] In addition to the convolutional layer, the fully connected layer and the BN layer, there are also pooling layers in the convolutional neural network. The most commonly used ones are maximum pooling and average pooling. Assuming that the size of the pooling window is 2*2, the role of pooling is to convert the original 4 inputs into 1 output, reducing the size of the image and reducing the amount of calculation. Maximum pooling is to select the maximum value output among the 4 inputs, and average pooling is to calculate the average value output of the 4 inputs.

[0245] For the average pooling layer, it is necessary to directly connect all inputs that need average pooling to the corresponding neurons in the next convolutional layer or fully connected layer, and copy the weights multiple times to achieve multiplication and accumulation in the same proportion as the original ones. The processing is similar to that described in the above embodiment 4-2 and will not be repeated here.

[0246] In the spiking neural network algorithm, proportionally scaling all weights and biases of a layer has no effect on the final output result. Therefore, in the next convolutional layer or fully connected layer, ∑2*2 O·W is sufficient. If the layer has a bias, the bias value will be amplified by 4 times based on all the corrections mentioned above and written into the storage input end of the storage and computing integrated unit.

[0247] Compared with adding an average pooling layer composed of neurons, the pulse signal generated by the threshold method will have a certain error from the theoretical result. After merging the average pooling layer into the next layer, it can be guaranteed that there is no loss of accuracy in the average pooling calculation. The resulting Spiking-Alexnet network structure (average pooling) is as follows Fig.26 shown.

[0248] For the maximum pooling layer, it is not necessary to operate in the next convolutional layer or fully connected layer. Instead, it is necessary to add additional judgment conditions between the two adjacent convolutional layers or fully connected layers. That is, from the beginning of the calculation, the earliest input signal corresponding to each pooling window is selected to connect with the next convolutional layer or fully connected layer. The rest of the input signals can be ignored. The resulting Spiking-Alexnet network structure (maximum pooling) is as follows: Fig. 27 shown.

[0249] As shown in the figure, behind the fully connected layer 3, there are 10 or 20 counters that are constantly counting the number of pulses (high level) received and the time when the earliest pulse (high level) was received. Each counter corresponds to a neuron and also corresponds to a type of image classification result.

[0250] At a certain moment, if the number of pulses in one category of counters is a more than that in other categories, a is the set hyperparameter, that is, the calculation is considered to be finished, and the category number of the category with the largest number of pulses is output. It is recommended to set it to 4.

[0251] If the end judgment condition is not met after the set maximum time, it will be forced to end and find the category with the largest number of pulses among the 10 categories.

[0252] If there are at least two categories with the same number of pulses, then compare which one receives the pulse first and output that category.

[0253] According to the pulse convolutional neural network algorithm of the above embodiment, by changing the input mode, merging the average pooling layer into the next convolutional layer or fully connected layer, supporting the calculation of convolutional layers and fully connected layers with biases, supporting the addition of BN layers in the network, setting the conditions for determining the end of calculation, adding auxiliary judgments for special cases and other optimization and improvement methods, the calculation time of the existing pulse convolutional neural network algorithm can be greatly saved and the accuracy of image classification can be improved. In addition, considering the end of the pulse convolutional neural network algorithm, improvements have also been made to the calculation time.

[0254] In addition, the storage-computing-in-one unit described in the above embodiment of the present invention can be implemented in an integrated circuit. Next, a method for manufacturing such an integrated circuit will be described, which includes the following steps:

[0255] 1) Forming dielectric layers and gates of transistors in digital logic circuits, integral comparison circuits and storage-computation integrated units by thermal oxidation and deposition; the transistors include ordinary logic transistors, high-voltage transistors and floating-gate transistors, etc.,

[0256] 2) forming a capacitor in the integral comparison circuit by depositing a MIM dielectric layer and a metal layer, or by thermal oxidation and deposition processes, wherein the capacitor may be a MIM capacitor or a MOS capacitor;

[0257] 3) Forming the source and drain of transistors in digital logic circuits, integral comparison circuits and storage-computation integrated units, as well as the P-level and N-level of PN junctions by ion implantation;

[0258] 4) Forming metal wiring and active area-metal layer and metal layer-metal layer through-holes of the overall circuit through metal layer process, metal layer dielectric process and through-hole process;

[0259] 5) A CMOS-based storage and computing unit is generated using a process corresponding to a memristor or flash memory.

[0260] The production process of integrated circuits based on pulse convolutional neural networks, wherein the digital logic circuits and the integral comparison circuits in neurons can be produced using standard CMOS processes. If the storage and calculation integrated unit in neurons uses an optoelectronic computing unit or flash memory, it can also be produced using standard CMOS processes. The production process of devices based on this process such as transistors, diodes or capacitors and other semiconductor devices using the standard CMOS process is not described in detail here. The optoelectronic computing unit can obtain better device performance by using the CIS image sensor process. If a memristor is used as the storage and calculation integrated unit in neurons, a special process compatible with such a memristor is required. The integration of the storage and calculation integrated device using a special process and the digital logic circuit and integral comparison circuit using the standard CMOS process can be achieved by directly using a special process to make a special device on a silicon-based substrate, or by wafer-level integration or off-chip integration. For example, there are various memristor manufacturing process methods such as the method for generating a high-durability memristor on a silicon-based substrate mentioned in Chinese patent CN110098324A.

[0261] Those of ordinary skill in the art will appreciate that the devices and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0262] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0263] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0264] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0265] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A pulse convolutional neural network algorithm, based on a storage-computation integrated unit, the storage-computation integrated unit includes at least one storage input terminal, at least one calculation input terminal and an output terminal, characterized in that: 1) copying the weights of the first layer of the pulse convolutional neural network into a plurality of copies, the number of copies being at least the number of bits of the binary number converted from the quantity used to characterize the attribute of the analyte and the minimum value of the storage input terminal of the storage-computation integrated unit, and processing the weights of the copied number of copies so that the values ​​of the copied weights are successively reduced by two times, and the obtained values ​​are respectively input into the storage input terminals of a plurality of the storage-computation integrated units, the number of which is the same as the number of copies; 2) converting the selected quantity used to characterize the property of the analyte into a binary number, and inputting each bit value of the binary number to be input, or the value after truncating according to the system bit width as an input pulse into the storage and calculation integrated computing unit of the pulse convolutional neural network; and, for each input set characterizing the property of the analyte, keeping the input pulse unchanged and continuously inputting it into the corresponding computing unit in the pulse convolutional neural network within a time period corresponding to the input set until the processing of the property of the analyte is completed; 3) For each bit value in the binary number corresponding to the property of the analyte, the input amount of each storage input terminal corresponds to the input amount of a calculation input terminal, and the input amount of the storage input terminal with a larger absolute value corresponds to the input amount of the higher-order calculation input terminal one by one; 4) In each of the storage-computation integrated units, the quantity at the storage input terminal is operated with the quantity at the calculation input terminal, and the current value obtained at the output terminal represents the result of the multiplication operation of the value at the storage input terminal and the value at the calculation input terminal of the storage-computation integrated unit.

2. The spiking convolutional neural network algorithm of claim 1, wherein: 1) including the operation of the first layer and the operation of other layers, and in any layer thereof, in addition to the operation of the storage input terminal and the calculation input terminal, an operation accumulation item is added, the operation accumulation item is a corrected bias value, the corrected bias value is proportional to its original value divided by the cumulative multiplication of the positive thresholds of all layers before the layer, and the proportional ratio is related to the weight scaling ratio of the layer where the bias is located and the previous layer; 2) The pulse convolutional neural network algorithm continuously accumulates the output of the storage and computing unit. When the accumulated sum exceeds a set positive threshold, the accumulated sum is cleared and an output pulse is released to the computing input end of the corresponding position of the next layer; and when the accumulated sum is less than a set negative threshold, the accumulated sum is maintained at the negative threshold.

3. The spiking convolutional neural network algorithm of claim 2, wherein: The pulse convolutional neural network includes a batch normalization layer, and a linear transformation is performed on the weights and biases in a convolutional layer or a fully connected layer before the batch normalization layer, wherein the parameters in the linear transformation are obtained in the previous training process.

4. The spiking convolutional neural network algorithm according to any one of claims 1 to 3, characterized in that: The number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time when the earliest pulse occurs are counted using multiple counters, and the number of the counters is the number of the neurons or twice thereof.

5. The spiking convolutional neural network algorithm of claim 4, wherein: If the counting results of at least two counters among the multiple counters are the same maximum value, the category value corresponding to the counter that receives the pulse earliest is selected as the final result.

6. The spiking convolutional neural network algorithm of claim 1, wherein: After the operation of the first layer, at least one of average pooling, maximum pooling, convolutional layer and fully connected layer operation is also performed.

7. The spiking convolutional neural network algorithm according to any one of claims 1 to 3, characterized in that: 1) Set the duration of several clock signals as one analysis cycle; 2) Divide the subject matter to be analyzed into several partitions; 3) Taking the analysis period as the time unit, analyzing the time series signal of a partition one by one, and sending the calculation result representing the partition to a memory; 4) analyzing the signal of the next partition, and sending the operation result representing the partition to the memory, until the signals of the completed multiple partitions jointly meet the analysis conditions of the next layer; 5) Sending the signals of each partition stored in the memory to the next layer for calculation.

8. The spiking convolutional neural network algorithm of claim 7, wherein: The memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage, or a combination thereof.

9. An integrated circuit based on a pulse convolutional neural network, characterized in that: The integrated circuit executes the pulse convolutional neural network algorithm as described in any one of claims 1-3 and 7.

10. An integrated circuit based on a pulse convolutional neural network, characterized in that: The integrated circuit executes the spiking convolutional neural network algorithm of claim 8.

11. A computer-readable recording medium storing computer-readable instructions, which, when executed by a computer, cause the computer to execute the spiking convolutional neural network algorithm of claim 1, wherein the spiking convolutional neural network algorithm is characterized in that: 1) copying the weights of the first layer of the pulse convolutional neural network into a plurality of copies, the number of copies being at least the number of bits of the binary number converted from the quantity used to characterize the attribute of the analyte and the minimum value of the storage input terminal of the storage-computation integrated unit, and processing the weights of the copied number of copies so that the values ​​of the copied weights are successively reduced by two times, and the obtained values ​​are respectively input into the storage input terminals of a plurality of the storage-computation integrated units, the number of which is the same as the number of copies; 2) converting the selected quantity used to characterize the property of the analyte into a binary number, and inputting each bit value of the binary number to be input, or the value after truncating according to the system bit width as an input pulse into the storage and calculation integrated computing unit of the pulse convolutional neural network; and, for each input set characterizing the property of the analyte, keeping the input pulse unchanged and continuously inputting it into the corresponding computing unit in the pulse convolutional neural network within a time period corresponding to the input set until the processing of the property of the analyte is completed; 3) For each bit value in the binary number corresponding to the property of the analyte, the input amount of each storage input terminal corresponds to the input amount of a calculation input terminal, and the input amount of the storage input terminal with a larger absolute value corresponds to the input amount of the higher-order calculation input terminal one by one; 4) In each of the storage-computation integrated units, the quantity at the storage input terminal is operated with the quantity at the calculation input terminal, and the current value obtained at the output terminal represents the result of the multiplication operation of the value at the storage input terminal and the value at the calculation input terminal of the storage-computation integrated unit.

12. The computer-readable recording medium according to claim 11, wherein: 1) The pulse convolutional neural network algorithm includes the operation of the first layer and the operation of other layers, and in any layer thereof, in addition to the operation of the storage input end and the calculation input end, an operation accumulation term is added, and the operation accumulation term is a corrected bias value, and the corrected bias value is proportional to its original value divided by the cumulative multiplication of the positive thresholds of all layers before the layer, and the proportional ratio is related to the weight scaling ratio of the layer where the bias is located and the previous layer; 2) The pulse convolutional neural network algorithm continuously accumulates the output of the storage and computing unit. When the accumulated sum exceeds a set positive threshold, the accumulated sum is cleared and an output pulse is released to the computing input end of the corresponding position of the next layer; and when the accumulated sum is less than a set negative threshold, the accumulated sum is maintained at the negative threshold.

13. The computer-readable recording medium according to claim 11, wherein The pulse convolutional neural network includes a batch normalization layer, and a linear transformation is performed on the weights and biases in a convolutional layer or a fully connected layer before the batch normalization layer, wherein the parameters in the linear transformation are obtained in the previous training process.

14. The computer-readable recording medium according to any one of claims 11 to 13, wherein: The number of pulses of each neuron in the last fully connected layer of the pulse convolutional neural network and the time when the earliest pulse occurs are counted using multiple counters, and the number of the counters is the number of the neurons or twice thereof.

15. The computer-readable recording medium according to claim 14, wherein If the counting results of at least two counters among the multiple counters are the same maximum value, the category value corresponding to the counter that receives the pulse earliest is selected as the final result.

16. The computer-readable recording medium according to claim 11, wherein After the operation of the first layer, at least one of average pooling, maximum pooling, convolutional layer and fully connected layer operation is also performed.

17. The computer-readable recording medium according to any one of claims 11 to 13, and 16, wherein: The spiking convolutional neural network algorithm includes the following: 1) Set the duration of several clock signals as one analysis cycle; 2) Divide the subject matter to be analyzed into several partitions; 3) Taking the analysis period as the time unit, analyzing the time series signal of a partition one by one, sending the operation result representing the partition to a memory, and the analyzed signal can be covered by the subsequent signal; 4) analyzing the signal of the next partition, and sending the operation result representing the partition to the memory, until the signals of the completed multiple partitions jointly meet the analysis conditions of the next layer; 5) Sending the signals of each partition stored in the memory to the next layer for calculation.

18. The computer-readable recording medium according to claim 17, wherein: The memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage, or a combination thereof.

19. An integrated circuit based on a spiking convolutional neural network, the spiking convolutional neural network comprising multiple layers of neurons, each layer of neurons comprising multiple neuron components, multiple neurons in each layer of neurons are not connected to each other but are connected to neurons in a subsequent layer; At least one of the neuron components has at most one digital logic circuit, and the digital logic circuit is used for operation, and the operation includes data distribution; and each neuron component of the last layer has a counter group, which counts the number of pulses with high level in the output pulses of the neuron component; wherein each neuron includes at least one storage-calculation unit and at least one integral comparison circuit, and the current output terminals of the plurality of storage-calculation units are connected to each other and collectively connected to the integral comparison circuit; Each of the integration comparison circuits comprises at least one integrator and at least one comparator, wherein the integrator is used to accumulate the output of the current output terminal, and the comparator is used to compare the accumulated output of the integrator with a previously set threshold value, and to reset and pulse output the comparator, wherein the reset operation enables the integrator to perform the next accumulation operation; Furthermore, each of the storage-computation integrated units includes at least one storage input terminal, at least one calculation input terminal, and at least one current output terminal, wherein the storage input terminal is configured to receive carriers representing weights sent by an upper computer, and the calculation input terminal is configured to receive carriers representing external or set upper layer input pulses; The current output terminal is configured to output, in the form of current, carriers that are the result of the interaction between the carriers used as weights and the carriers used as input pulses.

20. The integrated circuit of claim 19, wherein: The storage and computing integrated unit is one of a semiconductor-based photoelectric computing unit, a memristor, and a flash memory.

21. An integrated circuit as claimed in claim 19 or 20, characterized in that The operations of the digital logic circuit also include maximum pooling, clock synchronization, and data caching.

22. The integrated circuit of claim 19, wherein: The digital logic circuit is configured to find the first high-level pulse signal from a plurality of output signals output from the neuron components in the previous layer of the current pooling layer, the number of which is the square of the size of the pooling layer; and, The digital logic circuit is also configured to include a functional device of a multiplexer, so that after the high-level pulse signal passes through the multiplexer, the path corresponding to the high-level pulse signal is kept open, and the path is connected to the next convolutional layer or fully connected layer; at the same time, the signals of other paths parallel to the path corresponding to the high-level pulse signal are ignored, or the other paths are closed.

23. The integrated circuit of claim 19, wherein: Incorporate the average pooling operation into the next convolutional layer or fully connected layer, including: 1) A convolutional layer or a fully connected layer, wherein the number of storage-computation-in-one units in each neuron component of the convolutional layer or the fully connected layer is several times the original size of the corresponding algorithm of the layer, the multiple is the square of the size of the pooling layer, and each weight in the corresponding algorithm appears in the neuron component several times, the number of times is the square of the size of the pooling layer, 2) The output pulse signal output from the previous layer of neuron components to be transmitted to the next pooling layer, the number of which is the square of the size of the pooling layer, is directly used as the calculation input of the storage-computation integrated unit in the convolutional layer or the fully connected layer, and the storage-computation integrated units correspond to the same weights respectively.

24. The integrated circuit of claim 19, wherein: Each of the neuron components includes a neuron and has a register, and the register is used to achieve time synchronization of the data operations involved.

25. A pulse convolutional neural network operation device for performing pulse convolutional neural network operation, comprising a host computer and the integrated circuit as claimed in claim 19; wherein, The host computer is configured to process and generate weights of the first layer, and the process of generating the weights of the first layer includes: generating a group of weights according to an initial weight obtained through training through several linear transformations, the group of weights including multiple weight numbers, wherein the value of the latter weight is 1 / 2 of the value of the previous weight; and the host computer sends the group of weights to the storage input end of the storage-computation-integrated unit in each neuron component of the first layer of the pulse convolutional neural network; and the host computer sends the initial weights to the storage input end of the storage-computation-integrated unit of other layers after the first layer after several linear transformations, and for the weights of the convolutional layer or the fully connected layer immediately after the average pooling layer, the weights are also copied several times according to the pooling size, and the number of copies is the square of the size of the pooling layer.

26. The pulse convolutional neural network computing device according to claim 25, characterized in that: The device is used to analyze the target object by partition, and then synthesize the target object signals of each partition to form complete target object information, and The pulse convolutional neural network operation device also includes a memory, which is used to store the signals of at least one partition representing the subject matter that have been processed step by step, and synthesize all the partition signals after all the partition signals are processed, or send all the partition signals to another processor for synthesis; The memory is at least one of a register, an on-chip cache, an off-chip storage or a cloud storage.

27. A method for manufacturing an integrated circuit, characterized in that: The integrated circuit is the integrated circuit of claim 21, and the method comprises the following steps: 1) Forming dielectric layers and gates of transistors in digital logic circuits, integral comparison circuits and storage-computation integrated units by thermal oxidation and deposition; the transistors at least include ordinary logic transistors, high-voltage transistors and floating-gate transistors; 2) Forming the capacitor in the integral comparison circuit by depositing a MIM dielectric layer and a metal layer, or by thermal oxidation and deposition processes; 3) Forming the source and drain of transistors in digital logic circuits, integral comparison circuits and storage-computation integrated units, as well as the P-level and N-level of PN junctions by ion implantation; 4) Forming metal wiring and active area-metal layer and metal layer-metal layer through-holes of the overall circuit through metal layer process, metal layer dielectric process and through-hole process; 5) A CMOS-based memory and computing unit is generated by applying the process to memristors or flash memories.

Images