A binary neural network acceleration system based on 1T1R memristor and in-situ computing

Abstract

The present invention discloses a binary neural network acceleration system based on a 1T1R memristor and in-situ computing, and relates to the fields of semiconductor integrated circuits and neural networks. The present invention utilizes the storage characteristics of the memristors in the 1T1R memristor array, and designs an in-situ computing counter circuit and a corresponding input data encoding method to accelerate the binary neural network through an in-memory computing structure. Because the present invention does not include an ADC structure, it also has high reliability while accelerating the network. Compared with traditional memristor in-situ computing accelerators, the present invention not only only requires the use of two states of the memristor, with high hardware feasibility, but also has no ADC structure, with higher integration and reliability. The present invention can be applied to high-real-time target recognition, image classification, point cloud classification, semantic segmentation, natural language processing, and human posture estimation.

Description

Technical Field

[0001] The present invention relates to the fields of semiconductor integrated circuits and neural networks, and specifically relates to a binary neural network acceleration system based on 1T1R memristors and in-situ calculations. Background Art

[0002] Binary neural networks quantize both the weights and activation function outputs of a neural network into binary values. With the rapid development of big data and the Internet of Things, society's demand for neural networks is increasing. However, large-scale neural networks require enormous computational resources and parameter storage, which consumes a significant amount of resources. Compared to traditional neural networks, binary neural networks significantly reduce memory requirements while maintaining high accuracy, saving significant memory and computing resources. This is particularly important for edge deployment of neural networks.

[0003] A memristor is a non-volatile memory whose resistance changes depending on the amount of charge flowing through it. Currently, the technology for memristors with both high and low resistance states is relatively mature. This dual-state memristor in-memory computing architecture holds great practical significance in large-scale neural network computing.

[0004] Existing circuit systems that implement binary neural networks through memristor arrays require high-precision analog-to-digital converters (ADCs) or large-area digital counters to achieve statistical calculation results. This makes it difficult to ensure high computational reliability while also maintaining a small area. Summary of the Invention

[0005] To address the technical challenges of achieving high computational reliability and a compact footprint for a binary neural network accelerator based on a memristor array, the present invention provides a high-reliability, in-memory binary neural network acceleration system without an ADC, based on a 1T1R (1 transistor, 1 memristor) memristor array and in-situ computation. This system can be used for high-real-time object recognition, image classification, point cloud classification, semantic segmentation, natural language processing, and human pose estimation. Because it lacks an ADC, it accelerates binary neural networks with high reliability. The technical solution is as follows:

[0006] A binary neural network acceleration system based on 1T1R memristor and in-situ computing, specifically including the following modules: a memristor array module, a memristor read / write circuit module, an in-situ computing counter module, a data input module, a shifter and adder module, a communication module, and a controller module;

[0007] Among them, the data input module, shifter and adder module, communication module, and controller module are located in the FPGA; the weight of the binary neural network is sent to the communication module through the host computer, and the communication module then sends the weight data of the binary neural network to the memristor read-write circuit module, and then the memristor read-write circuit module loads the weight into the memristor array module; the host computer sends the input data of the binary neural network operation to the data input module, and the data input module converts the input data of the binary neural network operation into a WL signal and inputs it into the memristor array module, thereby controlling the opening and closing of the MOS tube connected to the memristor in the memristor array. Close; the memristor array module is used to store the weights of the binary neural network and participate in the operation of the binary neural network. Multiplication is performed through the memristor array module, and the result of the multiplication is output to the in-situ calculation counter module through the BL signal. The in-situ calculation counter module is responsible for completing the addition operation in the matrix operation, performing addition or subtraction operations, and then sending the operation result to the communication module. The communication module sends the operation result to the shifter and adder module. The shifter and adder module shifts and accumulates the operation result of each bit in the input, and then sends the result to the communication module, and then sends it to the host computer through the communication module;

[0008] The controller module is responsible for controlling the startup of the data input module, shifter and adder module, and communication module in the FPGA.

[0009] Among them, after changing the parameters of the binary neural network, it is also necessary to modify the weights in the memristor array module through the memristor read-write circuit module.

[0010] The beneficial effects of the present invention are as follows: the present invention designs a binary neural network acceleration system based on 1T1R memristor and in-situ calculation with smaller area and higher computational reliability. When performing high-real-time target recognition, image classification, point cloud classification, semantic segmentation, natural language processing, human posture estimation and other tasks that need to be deployed at the edge, it can provide more accurate accelerated operations and consume less chip resources. While improving computing efficiency, it can also reduce chip costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a binary neural network acceleration system based on 1T1R memristor and in-situ computing.

[0012] Figure 2 This is a schematic diagram of the circuit implementation of the binary convolution operation;

[0013] Figure 3 Schematic diagram of memristor array operation scenario 1;

[0014] Figure 4 Schematic diagram of memristor array operation scenario 2;

[0015] Figure 5 Schematic diagram of memristor array operation scenario three;

[0016] Figure 6 Schematic diagram of memristor array operation scenario 4;

[0017] Figure 7 Schematic diagram of the in-situ counting counter; DETAILED DESCRIPTION

[0018] First, let’s introduce the weight loading of this system. The host computer first sends the weights of the binary neural network required for calculation to the communication module, and then the communication module drives the memristor read-write circuit to load the weights into the memristor array. The 1T1R memristor array consists of multiple 1T1R memristors and WL signals (word lines) and BL signals (bit lines). The 1T1R memristor consists of 1 memristor and 1 transistor. In the 1T1R memristor array, one end of all memristors is connected to the same common terminal, and the other end is connected to a different transistor. The end of each transistor that is not connected to the memristor is connected to the bit line, and the gate of each transistor is connected to the word line. Each memristor in the memristor array has two states: high resistance and low resistance, corresponding to the two weights +1 and -1 in the binary neural network. After loading the weights, it is necessary to continue to read out the weights of the memristors through the memristor read-write circuit to ensure that all weights are correctly loaded into the memristor array. As Figure 2 As shown in the figure, taking a convolutional binary neural network with a convolution kernel size of 3x3 as an example, each convolution kernel corresponds to a BL signal, and the 9 memristors connected to the same BL signal respectively store the 9 weights corresponding to the convolution kernel.

[0019] Next, we'll describe the computational process of the memristor array. The input for the binary neural network operation is fed into the memristor array in the form of WL signals by the data input module in the FPGA. Each input data point corresponds to a corresponding WL signal. The data input module in the FPGA primarily consists of multiple D-type flip-flops and several logic gates. The input is a clock signal, representing the input data to be sent to the memristor array, and the output is a WL signal. The number of D-type flip-flops is the same as the number of WL signals. The combination of the D-type flip-flops and several logic gates generates the WL signals, ensuring that each clock cycle causes the WL signal corresponding to an input data point of 1 to be pulled high for one cycle, while the WL signal corresponding to an input data point of 0 remains unchanged. When the input of the leftmost D-type flip-flop in the data input module is pulled high for one cycle, all WL signals with inputs of 1 will also be pulled high for one cycle in a time-sharing manner. If the weight of the memristor connected to the WL signal is +1, then when the WL signal is pulled high, the BL signal corresponding to the memristor will also be pulled high. If the weight of the memristor connected to the WL signal is -1, then when the WL signal is pulled high, the BL signal corresponding to the memristor will be pulled low. Suppose there are n WL signals in total, and k Signal and BL m The weight stored in the memristor connected to the signal is W k m , and WL k The input corresponding to the signal is X k , where m is the column label and k is the row label, then BL m The calculation result OUT corresponding to the signal m , as shown in formula (1):

[0020]

[0021] Next, we will use four cases as examples to illustrate the operation process of the memristor array. Figure 3 As shown in the figure, there are 9 WL signals (WL signal 0-WL signal 8) and all 9 inputs are 1, and the weights of the 9 memristors are all 1 as an example. Among them, XIN is the input terminal of the leftmost D flip-flop in the data input module. When XIN is pulled high for one cycle, because the inputs corresponding to all WL signals are 1, each WL signal is pulled high for one cycle in turn, and because the weights stored in all memristors are also 1, when each WL signal is pulled high, the BL signal is also pulled high at the same time. Figure 4As shown, there are 9 WL signals (WL signal 0-WL signal 8), and the inputs corresponding to WL signal 0, WL signal 2, WL signal 4, WL signal 6, and WL signal 7 are -1, and the inputs corresponding to WL signal 1, WL signal 3, WL signal 5, and WL signal 8 are +1, and the weights stored in the 9 memristors are all 1. For example, when XIN is pulled high for one cycle, because the inputs corresponding to WL signal 0, WL signal 2, WL signal 4, WL signal 6, and WL signal 7 are -1, they are always low. Because the inputs corresponding to WL signal 1, WL signal 3, WL signal 5, and WL signal 8 are 1, they are pulled high for one cycle in turn, and because the weights stored in the memristors corresponding to them are all 1, when they are pulled high, the BL signal is also pulled high, and the memristor array completes this operation within four cycles. Figure 5 As shown in the figure, there are 9 WL signals (WL signal 0-WL signal 8) and all inputs are 1, the weight of the memristor corresponding to WL signal 1, WL signal 3, WL signal 6, and WL signal 7 is -1, and the weight of the memristor used by the remaining WL signals is 1. For example, when XIN is pulled high for one cycle, because the inputs corresponding to all WL signals are 1, each WL signal is pulled high for one cycle in turn. Because the weight of the memristor corresponding to WL signal 1, WL signal 3, WL signal 6, and WL signal 7 is -1, when they are pulled high, the BL signal is low, and when the remaining WL signals corresponding to the memristor weight of 1 are pulled high, the BL signal is also pulled high at the same time. Figure 6 As shown, taking the example of 9 WL signals (WL signal 0-WL signal 8) and the input and memristor weight corresponding to WL signal 0, WL signal 3, WL signal 7, and WL signal 8 are 1, the input corresponding to WL signal 1 and WL signal 4 is 0, and the input corresponding to WL signal 2, WL signal 5, and WL signal 6 is 1 and the corresponding memristor weight is -1, because the input corresponding to WL signal 1 and WL signal 4 is 0, they remain low, and the remaining WL signals are pulled high in turn. Because the memristor weight corresponding to WL signal 2, WL signal 5, and WL signal 6 is -1, when they are pulled high, the BL signal is pulled low. When WL signal 0, WL signal 7, and WL signal 8 are pulled high, the BL signal is pulled high. A total of 7 cycles are required to complete the operation.

[0022] Next, we will introduce the operation process of the in-situ calculation counter. Figure 7As shown, the inputs to the per-bit in-situ counter circuit are the system clock signal CLK, CLK_N, BL, and BL_N. CLK_N is the inverted version of CLK, and BL_N is the inverted version of BL. The W_en, WDATA, and WDATA_N signals from the data input module are used to modify the counter data. WDATA_N is the inverted version of WDATA. The outputs of the per-bit in-situ counter circuit are data OUT and CLK_out, which serves as the next-stage in-situ counter. The per-bit in-situ counter circuit includes three inverters, 11 NMOS transistor switches, and one capacitor. The leading end of inverter 1 is connected to the trailing end of inverter 2. The leading end of inverter 2 is node 1, and the trailing end of inverter 2 is node 2. The trailing end of inverter 1 is connected to the drain of NMOS transistor 1. The source of NMOS transistor 1 is connected to node 1, and the gate of NMOS transistor 1 is connected to the CLK_N signal. When CLK is low, the switch opens, and the two inverters latch data OUT. Node 2 stores output data OUT. One end of the capacitor is grounded, and the other end is node 3. Node 3 is connected to the source of NMOS transistor 2, node 2 is connected to the drain of NMOS transistor 2, and the CLK_N signal is connected to the gate of NMOS transistor 2. Node 3 is connected to the drain of NMOS transistor 3, node 1 is connected to the source of NMOS transistor 3, and the gate of NMOS transistor 3 is connected to the CLK signal. Node 1 is connected to the source of NMOS transistor 4, the drain of NMOS transistor 4 is connected to the WDATA_N signal, and the gate of NMOS transistor 4 is connected to the WEN signal. Node 2 is connected to the source of NMOS transistor 5, the drain of NMOS transistor 5 is connected to the WDATA signal, and the gate of NMOS transistor 5 is connected to the WEN signal. Node 2 is connected to the drain of NMOS transistor 6, the source of NMOS transistor 6 is node 4, and the gate of NMOS transistor 6 is connected to the CLK signal. The drain of NMOS transistor 7 is connected to node 4, its source is grounded, and its gate is connected to the CLK_N signal. Node 2 is connected to the head end of inverter 3, the tail end of inverter 3 is connected to the drain of NMOS transistor 8, its source is node 5, and its gate is connected to the CLK signal. Node 5 is connected to the drain of NMOS transistor 9, its source is grounded, and its gate is connected to the CLK_N signal. Node 4 is connected to the drain of NMOS transistor 10, its source is node 6, and its gate is connected to the BL_N signal. Node 5 is connected to the drain of NMOS transistor 11, its source is node 6, and its gate is connected to the BL signal. Node 6 is used as the CLK_out of the next-stage in-situ counter.

[0023] When the system clock CLK signal is low, the initial value of the counter can be written by pulling the W_en signal high and configuring the WDATA signal. At this time, the two inverters form a latch and store the data OUT through the capacitor.

[0024] When the CLK signal is high, data OUT is flipped compared to the previous cycle because the voltage level stored in the capacitor in the previous cycle reaches node 2 after passing through an inverter. If data OUT flips from 0 to 1 when the CLK signal rises, and the BL signal is high, then an addition operation is performed, and there is no borrow or carry. The data of the next-level in-situ calculation counter does not need to be flipped, and CLK_out continues to be pulled low. If data OUT flips from 0 to 1 when the CLK signal rises, and the BL signal is low, then a subtraction operation is performed, and a borrow is required. The data of the next-level in-situ calculation counter needs to be flipped, and CLK_out is pulled high. If data OUT flips from 1 to 0 when the CLK signal rises, and the BL signal is high, then an addition operation is performed, and a carry is required. The data of the next-level in-situ calculation counter needs to be flipped, and CLK_out is pulled high. If the data OUT flips from 1 to 0 when the CLK signal rises, and the BL signal is low, then a subtraction operation is performed without borrow or carry. The data of the next level of the in-situ calculation counter does not need to be flipped, and CLK_out continues to be pulled low. The number of levels of the in-situ calculation counter can be set according to the number of WL signals.

[0025] Finally, we will introduce the relevant functions of the FPGA. The FPGA includes a controller module, a data input module, a shifter and adder module, and a communication module. The data input module is responsible for converting the input parameters of the binary neural network operation into WL signals and consists of multiple D flip-flops and several logic gates. If the binary neural network input is multi-bit, the shifter and adder modules are required to shift and accumulate the operation results of each bit in the input. The adder also needs to add the result of the matrix multiplication operation in the binary neural network with the bias. The communication module is responsible for communication between the FPGA and the host computer, the memristor read / write circuit, and the in-situ calculation counter. Communication between the FPGA and the host computer includes the host computer sending binary neural network data to the FPGA and the FPGA sending calculation results to the host computer. Communication between the FPGA and the memristor includes the FPGA sending binary neural network weight data to the memristor read / write circuit. Communication between the FPGA and the in-situ calculation counter includes the in-situ calculation counter sending calculation results to the FPGA. The controller module is responsible for controlling the data input module, the shifter and adder module, and the communication module.

[0026] The above design scheme, based on a 1T1R memristor array, an in-situ computation counter, and an FPGA, enables the realization of a highly reliable, in-memory binary neural network acceleration system without an ADC. This system accelerates binary neural networks with high reliability, making it suitable for tasks such as object recognition, image classification, point cloud classification, semantic segmentation, natural language processing, and human pose estimation, which are suitable for binary neural networks. Taking image classification as an example, the weights of the binary neural network used for image classification are first transmitted from the host computer to the FPGA communication module. The FPGA communication module then transmits the weights to the memristor read / write circuit module, which then writes the weights to the memristor array. The host computer then sends the image data that needs to be classified to the data input module. The data input module converts the input data into a WL signal and inputs it into the memristor array. The memristor array sends the multiplication result to the in-situ calculation counter module via the BL signal. The in-situ calculation counter module performs addition and subtraction counting and sends the calculation result to the FPGA's shifter and adder module via the communication module for shifting and accumulation. After obtaining the final classification result, it is sent to the host computer through the communication module to complete the image classification task.

Claims

1. A binary neural network acceleration system based on 1T1R memristor and in-situ computing, characterized in that: It includes the following modules: memristor array module, memristor read / write circuit module, in-situ calculation counter module, data input module, shifter and adder module, communication module, and controller module; Among them, the data input module, shifter and adder module, communication module, and controller module are located in the FPGA; the weight of the binary neural network is sent to the communication module through the host computer, and the communication module then sends the weight data of the binary neural network to the memristor read-write circuit module, and then the memristor read-write circuit module loads the weight into the memristor array module; the host computer sends the input data of the binary neural network operation to the data input module, and the data input module converts the input data of the binary neural network operation into a WL signal and inputs it into the memristor array module, thereby controlling the opening and closing of the MOS tube connected to the memristor in the memristor array. Close; the memristor array module is used to store the weights of the binary neural network and participate in the operation of the binary neural network. Multiplication is performed through the memristor array module, and the result of the multiplication is output to the in-situ calculation counter module through the BL signal. The in-situ calculation counter module is responsible for completing the addition operation in the matrix operation, performing addition or subtraction operations, and then sending the operation result to the communication module. The communication module sends the operation result to the shifter and adder module. The shifter and adder module shifts and accumulates the operation result of each bit in the input, and then sends the result to the communication module, and then sends it to the host computer through the communication module; The controller module is responsible for controlling the startup of the data input module, shifter and adder module, and communication module in the FPGA; The data input module is composed of multiple D flip-flops and several logic gates. The input is a clock signal, and the input data to be sent to the memristor array is output as a WL signal. The number of D flip-flops is the same as the number of WL signals. The combination circuit composed of D flip-flops and several logic gates is used to generate WL signals, so that the WL signal corresponding to an input data of 1 is pulled high for one cycle in each clock cycle, and the WL signal corresponding to the input data of 0 is not pulled high; when the input end of the D flip-flop at the left end of the data input module is pulled high for one cycle, all WL signals with input of 1 will also be pulled high for one cycle in turn; if the weight of the memristor connected to the WL signal is +1, then when the WL signal is pulled high, the BL signal corresponding to the memristor will also be pulled high; if the weight of the memristor connected to the WL signal is -1, then when the WL signal is pulled high, the BL signal corresponding to the memristor will be pulled low; suppose there are n WL signals in total, and they are connected to the WL k Signal and BL m The weight stored in the memristor connected to the signal is W k m , and WL k The input corresponding to the signal is X k , where m is the column label and k is the row label, then BL m The calculation result OUT corresponding to the signal m , as shown in formula (1):

2. A binary neural network acceleration system based on 1T1R memristor and in-situ computing according to claim 1, characterized in that: After changing the parameters of the binary neural network, the weights in the memristor array module also need to be modified through the memristor read-write circuit module.

3. The binary neural network acceleration system based on 1T1R memristor and in-situ computing according to claim 1, characterized in that: The number of stages of the in-situ calculation counter can be set according to the number of WL signals; the input of each bit in-situ calculation counter circuit is the system clock CLK signal, CLK_N signal, BL signal and BL_N signal, wherein the CLK_N signal is the inverted signal of the CLK signal, the BL_N signal is the inverted signal of the BL signal, and the W_en signal, WDATA signal and WDATA_N signal from the data input module are used to change the counter data, wherein the WDATA_N signal is the inverted signal of the WDATA signal; the output of each bit in-situ calculation counter circuit is data OUT and CLK_out used as the next stage in-situ calculation counter; each bit in-situ calculation counter circuit The counter circuit includes three inverters, 11 NMOS switches and one capacitor; the first end of inverter 1 is connected to the tail end of inverter 2, the first end of inverter 2 is node 1, and the tail end of inverter 2 is node 2; the tail end of inverter 1 is connected to the drain of NMOS tube 1, the source of NMOS tube 1 is connected to node 1, and the gate of NMOS tube 1 is connected to the CLK_N signal; so that when the CLK signal is low, the switch is turned on and the two inverters will latch the data OUT; node 2 stores the output data OUT; one end of the capacitor is grounded and the other end is node 3, node 3 is connected to the source of NMOS tube 2, node 2 is connected to the drain of NMOS tube 2, and the CLK_N signal is connected to the gate of NMOS tube 2; node 3 is connected to the drain of NMOS tube 3, node 1 is connected to the source of NMOS tube 3, and the gate of NMOS tube 3 is connected to CLK; node 1 is connected to the source of NMOS tube 4, the drain of NMOS tube 4 is connected to the WDATA_N signal, and the gate of NMOS tube 4 is connected to the W_en signal; node 2 is connected to the source of NMOS tube 5, the drain of NMOS tube 5 is connected to the WDATA signal, and the gate of NMOS tube 5 is connected to the W_en signal; node 2 is connected to the drain of NMOS tube 6, the source of NMOS tube 6 is node 4, and the gate of NMOS tube 6 is connected to the CLK signal; the drain of NMOS tube 7 is connected to node 4, the source of NMOS tube 7 is grounded, and the gate of NMOS tube 7 is connected to CLK_ N signal; node 2 is connected to the head end of inverter 3, the tail end of inverter 3 is connected to the drain of NMOS transistor 8, the source of NMOS transistor 8 is node 5, and the gate of NMOS transistor 8 is connected to the CLK signal; node 5 is connected to the drain of NMOS transistor 9, the source of NMOS transistor 9 is grounded, and the gate of NMOS transistor 9 is connected to the CLK_N signal; node 4 is connected to the drain of NMOS transistor 10, the source of NMOS transistor 10 is node 6, and the gate of NMOS transistor 10 is connected to the BL_N signal; node 5 is connected to the drain of NMOS transistor 11, the source of NMOS transistor 11 is connected to node 6, and the gate of NMOS transistor 11 is connected to the BL signal. Node 6 is used as CLK_out of the next-level in-situ calculation counter.

Images