A storage and computing integrated device and method capable of deploying large-scale neural networks

Abstract

The present invention discloses a storage and computing integrated device and method capable of deploying large-scale neural networks. The device includes a preprocessing module, an input drive circuit module, a storage and computing integrated module (including an SRAM array and a Flash array), a readout circuit module, an ADC module, and a SIMD module. The preprocessing module is responsible for preprocessing the input data. The preprocessed data is input into the input drive circuit, and then selected to be input into the SRAM array or the Flash array for storage and calculation. The current signal output by the array calculation is input into the readout circuit and converted into a voltage signal, and then input into the ADC module and converted into a digital signal, and finally input into the SIMD module for data processing and output of a digital signal. The device of the present invention combines the advantages of Flash and SRAM, and has both large capacity and flexibility. The present invention adopts the method of transfer learning to deploy large-scale neural networks on the storage and computing integrated device, which can realize the deployment of different tasks.

Description

Technical Field

[0001] The present invention relates to a storage and computing integrated device and method capable of deploying a large-scale neural network, belonging to the field of ultra-large-scale integrated circuits. Background Art

[0002] In recent years, the Computing-In-Memory (CIM) chip architecture has garnered significant attention from researchers to address the memory bottleneck of the von Neumann architecture. CIM chip architectures utilize memory for computation, reducing data transfer between memory and the processor, thereby improving performance and reducing power consumption.

[0003] Flash devices are non-volatile memory devices that store data by adjusting the voltage on their floating gates. Today, the manufacturing process for Flash devices is relatively mature, and the finished products are stable, enabling large-scale mass production. Furthermore, Flash cells offer the advantages of high efficiency and density, making them a common medium for integrated storage and computing devices. However, current Flash-based integrated storage and computing devices suffer from slow weight writes and difficulty in modifying them.

[0004] Static random access memory (SRAM) is a volatile memory with the advantages of high speed, low power consumption, and high robustness. Due to its design flexibility and mature manufacturing process, it is a common medium for storage and computing devices. However, due to the disadvantage of SRAM's low density, SRAM-based storage and computing devices have the disadvantages of large area, high manufacturing cost, and difficulty in deploying large-scale neural networks. Summary of the Invention

[0005] In order to overcome the shortcomings of low flexibility of existing Flash storage and computing devices and small capacity of SRAM storage and computing devices, the present invention provides a storage and computing device and method that can deploy large-scale neural networks.

[0006] The technical solution adopted by the device of the present invention is as follows:

[0007] A storage-computing integrated device capable of deploying large-scale neural networks comprises a preprocessing module, an input drive circuit, a storage-computing integrated module, a readout circuit, an analog-to-digital conversion module, and a single-instruction multiple data (SIMD) stream module, wherein the input of the preprocessing module is connected to an input digital signal; the output of the preprocessing module is connected to the input of the input drive circuit; the output of the input drive circuit is connected to the input of the storage-computing integrated array; the storage-computing integrated array comprises an SRAM array and a Flash array, the SRAM array is used to deploy trainable weights, and the Flash array is used to store fixed weights; the output of the storage-computing integrated array is connected to the input of the readout circuit; the output of the readout circuit is connected to the input of the analog-to-digital conversion module; the output of the analog-to-digital conversion module is connected to the input of the SIMD stream module, and the SIMD stream module outputs a digital signal after calculation.

[0008] Furthermore, the input driving circuit includes a word line driving circuit and a bit line driving circuit; the word line driving circuit controls the word line input of the storage and computing integrated module; the bit line driving circuit controls the bit line input of the storage and computing integrated module.

[0009] Furthermore, the SRAM array includes multiple sub-arrays, each sub-array consists of a column of SRAM cells and a local computing unit; the SRAM cells store variable weights in the pre-trained model and perform multiplication operations on input data within the cells; the local computing unit redistributes charge on the output voltage of each sub-array, that is, performs voltage summing and averaging operations.

[0010] Furthermore, the Flash array is composed of multiple Flash units; the Flash array adopts an analog multiplication circuit based on NORFlash. In a multiplication unit, two Flash tubes share a gate and drain, and the Flash source voltage is fixed so that the Flash operates in the linear region. By subtracting the current flowing through the two Flash tubes, the multiplication operation of the drain-source voltage and the threshold voltage difference is realized.

[0011] The present invention also provides a method for deploying a storage and computing integrated device for large-scale neural networks, the method comprising the following specific steps:

[0012] (1) The pre-trained model is deployed on the storage and computing module, the fixed weights are deployed on the Flash array, and the trainable weights are deployed on the SRAM array; wherein, the weights of the first n convolutional layers in the convolutional neural network are fixed, and the weights of the last m convolutional layers are trainable. <n;

[0013] (2) The preprocessing module receives the input digital signal for preprocessing, and the preprocessed signal is input to the word line driving circuit and the bit line driving circuit of the input driving circuit respectively. The input driving circuit drives the corresponding word line and bit line of the storage and calculation module according to the preprocessed signal to perform storage and calculation;

[0014] (3) The storage and computing module performs inference and training based on the input data. During the transfer learning process, each training session will calculate the loss function based on the deployed weights, and then update the size of the variable weights for a new round of training. The Flash array performs inference, i.e., multiplication and accumulation calculations, and the SRAM array infers the value of the loss function and outputs it to the preprocessing module. The preprocessing module controls the SRAM array to deploy new weights for training based on the learning rate and the loss function.

[0015] (4) The storage and calculation module outputs a current signal to the data readout circuit after multiplication and accumulation calculation. The readout circuit converts the current signal into a voltage signal and outputs it to the analog-to-digital conversion module. The analog-to-digital conversion module converts the voltage signal into a digital signal and outputs it to the single instruction multiple data stream module.

[0016] (5) After the SIMD module activates and pools the digital signal, it outputs it to the preprocessing module for the next layer of operation; the loss function is calculated for the fully connected result of the last layer, and it is output to the next round of training for weight update.

[0017] This invention combines an SRAM array with a Flash array to create a storage-and-computing device capable of deploying large-scale neural networks. When deploying large neural networks, this device saves approximately 10 times the area of ​​a pure SRAM storage-and-computing device. It also overcomes the difficulty of modifying Flash device weights. By combining transfer learning with neural network model training to handle different tasks, the modified weights are directly updated in the SRAM array, achieving greater flexibility compared to pure Flash storage-and-computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a diagram of the overall architecture of the device of the present invention;

[0019] Figure 2 This is a structural diagram of the storage and computing module;

[0020] Figure 3 This is the structure diagram of the Flash array computing unit;

[0021] Figure 4 This is the composition structure of the SRAM array. DETAILED DESCRIPTION

[0022] The overall architecture of the storage and computing integrated device used in the present invention is as follows: Figure 1As shown in the figure, it consists of a preprocessing module, an input driving circuit, a computing-in-memory module, a readout circuit, an analog-to-digital conversion (ADC) module, and a single instruction multiple data (SIMD) module. Data is input to the preprocessing module for preprocessing, and then input to the input driving circuit to drive the corresponding word lines and bit lines of the computing-in-memory module for multiply-accumulate (MAC) calculation. The current output after the calculation is output to the readout circuit to be converted into a voltage signal, converted into a digital signal by the ADC module, and finally input to the SIMD module for calculation and then data is output.

[0023] The composition structure of the computing-in-memory module is as Figure 2 shown, and it consists of a SRAM array and a Flash array. For the SRAM array and the Flash array, corresponding word line and bit line signals are used for array partitioning. For the deployment of large-scale neural networks, a transfer learning method is used for deployment: the SRAM array deploys variable weights, and the Flash array deploys fixed weights. Among them, the underlying features of the convolutional layer should remain fixed, that is, the weights of the first n layers of the convolution remain unchanged; the weights changed by transfer learning are the weights of the last m layers, and the value of m can be selected according to the scenario and task. Usually, m = 1 to 3 and m < n. Therefore, the Flash array deploys the weights of the 1st to nth layers of the large-scale neural network, and the SRAM array deploys the weights of the (n + 1)th to (n + m)th layers of the large-scale neural network.

[0024] The Flash array consists of multiple Flash cells and adopts an analog multiplication circuit based on NOR Flash. The circuit diagram is as Figure 3 shown, where V G represents the gate voltage of the Flash tube, V TH1 , V TH2 respectively represent the threshold voltages of two Flash tubes, V DS represents the drain-source voltage of the Flash tube, I D1 and I D2 respectively represent the drain currents of two Flash tubes, I D is the current difference between two Flash tubes. In a multiplication unit, two Flash tubes share the gate and drain, and the source voltage of the Flash is fixed to make the Flash operate in the linear region. By subtracting the currents flowing through the two Flash tubes, the multiplication operation of the difference between the drain-source voltage and the threshold voltage is realized.

[0025] The calculation principle of this analog multiplication current is based on the I / V characteristic of the Flash in the linear region, and the calculation formula is as follows:

[0026]

[0027]

[0028] ID =I D1 -I D2 (3)

[0029] (1)-(2), and let ΔV TH =V TH1 -V TH2 ,have to:

[0030]

[0031] When deploying the neural network weights, the input data x is mapped to the drain-source voltage V of the Flash tube. DS , the weight w is mapped to the threshold voltage difference ΔV TH , the calculation result is mapped to I D , It can be expressed as process parameter K, where μ n is the electron mobility, C ox is the capacitance coefficient, W is the channel width, and L is the channel length. Then (4) can be transformed into:

[0032] y=Kwx(5)

[0033] Here, x represents input data and y represents output data.

[0034] The Flash array can then multiply the input data with the pre-deployed weights.

[0035] The SRAM array uses array local computing to achieve storage and computing integration. The internal architecture of the SRAM array is as follows: Figure 4 As shown in the figure. In the figure, MA and Memory Array represent memory arrays, Memory Cell represents memory cells, LCC and Local Computing Cell represent local computing cells, Local Computing represents local computing arrays, and GBL represents global bit lines. The SRAM array contains m sub-columns, each of which consists of one local computing cell (LCC) and n SRAM cells. A single SRAM cell stores variable weights in the pre-trained model and performs multiplication operations on input data within the cell. The LCC cell of a single sub-column redistributes charge on the output voltage of each sub-column, that is, it performs voltage summing and averaging operations.

[0036] SRAM has the characteristics of fast read and write operation speed, so the weights stored in the SRAM array can be updated after each round of transfer learning training until the training is completed.

[0037] The following takes a small data set transfer learning as an example to describe a method for deploying a pre-trained VGG-16 model on the storage and computing integrated device capable of deploying large-scale neural networks.

[0038] For small data training, we deploy the model by fixing all convolutional layer weights and training the fully connected layer weights. During the deployment phase, the pre-trained model weights are deployed on the integrated storage and computing array: the Flash array deploys all convolutional layer weights, and the SRAM array deploys the fully connected layer weights. The number of classes in the fully connected layer is determined by the current task.

[0039] After the deployment is completed, the network is inferred and trained. Figure 1 As shown in the process, the initial image data is input to the pre-processing module. After being processed by the pre-processing module, the data is mapped to the corresponding word lines and bit lines of the storage and computing integrated array weight deployment, and output to the input drive circuit. The storage and computing integrated module is driven by the word line and bit line drive circuit. The corresponding position in the Flash array multiplies and accumulates the input data with the corresponding deployed weights. The output current signal is converted into a voltage signal through the output circuit, and the voltage signal is output to the ADC module and converted into a digital signal output. This storage and computing integrated device completes a convolution operation. The result of the first convolution operation is input to the SIMD module for activation operation, and the activated data is output to the pre-processing module again for a second convolution operation. After the result of the second convolution calculation is output from the ADC module, it is input to the SIMD module for activation and pooling operations. Subsequent convolution layers perform convolution operations, activation, and pooling operations according to the above operations. After all 13 layers of convolution are completed, the pooled data is output to the preprocessing module. After preprocessing, it is then output to the input driver circuit. The wordline and bitline driver circuits drive the integrated memory and computation module. The corresponding position in the SRAM array performs multiplication and accumulation operations on the input data and the corresponding deployed weights. The output current signal is converted to a voltage signal, which is then sent to the ADC module for digital output. This completes a fully connected operation. The result is input to the SIMD module for activation. The activated data is then output to the preprocessing module for a second fully connected and pooled operation. After three fully connected and pooled operations, the output is subjected to the loss function calculation and fed back to the preprocessing module. The preprocessing module updates the weights for the next round of training based on the preset learning rate, controls the weights deployed in the SRAM, and then proceeds to the next round of training. After multiple rounds of training, a model suitable for the current task is obtained.

[0040] For trained models, the present invention can deploy a large-scale neural network storage and computing device that can deploy weights in a storage and computing array to perform inference operations on the same task scenario. Compared with a pure SRAM storage and computing device, this device saves about 10 times the area, greatly improving the capacity of devices with the same area. Compared with a pure Flash storage and computing device, this device can update a part of the weights in real time during the training process to achieve the migration of different tasks without erasing and rewriting all the weights, greatly improving the efficiency of updating the weights.

Claims

1. A storage and computing integrated device capable of deploying large-scale neural networks, comprising a preprocessing module, an input drive circuit, a storage and computing integrated module, a readout circuit, an analog-to-digital conversion module, and a single instruction multiple data (SIMD) module, characterized in that: The input end of the preprocessing module is connected to the input digital signal; the output end of the preprocessing module is connected to the input end of the input drive circuit; the output end of the input drive circuit is connected to the input end of the storage and computing array; the storage and computing array includes an SRAM array and a Flash array, the SRAM array is used to store trainable weights, and the Flash array is used to store fixed weights, and the output end of the storage and computing array is connected to the input end of the readout circuit; the output end of the readout circuit is connected to the input end of the analog-to-digital conversion module; the output end of the analog-to-digital conversion module is connected to the input end of the single instruction multiple data stream module, and the single instruction multiple data stream module outputs a digital signal after calculation.

2. The storage and computing integrated device capable of deploying large-scale neural networks according to claim 1, characterized in that: The input drive circuit includes a word line drive circuit and a bit line drive circuit; the word line drive circuit controls the word line input of the storage and computing integrated module; the bit line drive circuit controls the bit line input of the storage and computing integrated module.

3. The storage and computing integrated device capable of deploying large-scale neural networks according to claim 1, characterized in that: The SRAM array includes multiple sub-arrays, each sub-array consists of a column of SRAM cells and a local computing unit; the SRAM cells store variable weights in the pre-trained model and perform multiplication operations on input data within the cells; the local computing unit redistributes charge on the output voltage of each sub-array, that is, performs voltage summing and averaging operations.

4. The storage and computing integrated device capable of deploying large-scale neural networks according to claim 1, characterized in that: The Flash array is composed of multiple Flash units; the Flash array adopts an analog multiplication circuit based on NOR Flash. In a multiplication unit, two Flash tubes share a gate and drain, and the Flash source voltage is fixed so that the Flash operates in the linear region. By subtracting the current flowing through the two Flash tubes, the multiplication operation of the drain-source voltage and the threshold voltage difference is achieved.

5. A method for utilizing the storage and computing integrated device capable of deploying a large-scale neural network as claimed in claim 1, characterized in that: The specific steps of the method include: (1) The pre-trained model is deployed on the storage and computing module, the fixed weights are deployed on the Flash array, and the trainable weights are deployed on the SRAM array; wherein, the weights of the first n convolutional layers in the convolutional neural network are fixed, and the weights of the last m convolutional layers are trainable. <n; (2) The preprocessing module receives the input digital signal for preprocessing, and the preprocessed signal is input to the word line driving circuit and the bit line driving circuit of the input driving circuit respectively. The input driving circuit drives the corresponding word line and bit line of the storage and calculation module according to the preprocessed signal to perform storage and calculation; (3) The storage and computing module performs inference and training based on the input data. During the transfer learning process, each training session will calculate the loss function based on the deployed weights, and then update the size of the variable weights for a new round of training. The Flash array performs inference, i.e., multiplication and accumulation calculations, and the SRAM array infers the value of the loss function and outputs it to the preprocessing module. The preprocessing module controls the SRAM array to deploy new weights for training based on the learning rate and the loss function. (4) The storage and calculation module outputs a current signal to the data readout circuit after multiplication and accumulation calculation. The readout circuit converts the current signal into a voltage signal and outputs it to the analog-to-digital conversion module. The analog-to-digital conversion module converts the voltage signal into a digital signal and outputs it to the single instruction multiple data stream module. (5) After the SIMD module activates and pools the digital signal, it outputs it to the preprocessing module for the next layer of operation; the loss function is calculated for the fully connected result of the last layer, and it is output to the next round of training for weight update.

Images