A hardware convolutional neural network system based on memristors

Abstract

The application discloses a kind of convolutional neural network systems constructed using memristor, it is related to the field of semiconductor integrated circuit and neural network.The core of the proposed convolutional neural network system is the convolution layer and fully connected layer composed of memristor, and the data encoding method matched with it.The application utilizes the characteristic that the memristor can realize multiple resistance states, maps the weight in the convolutional neural network to the conductance of the memristor, maps the input in the convolutional neural network to the voltage across the memristor, and reads the current flowing through the memristor to obtain the result after the convolution layer and fully connected layer operation.Compared with the traditional convolutional neural network, the network proposed in the application can be implemented based on hardware, fully utilizes the multiple resistance state characteristics of the memristor, and has the advantages of low power consumption, high efficiency, high integration, good compatibility with CMOS process, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semiconductor integrated circuits and neural networks, and in particular to an application of a convolutional neural network based on a memristor in image recognition. BACKGROUND

[0002] In a convolutional neural network, there are a large number of convolution (or multiplication and addition) operations, which consume most of the resources in network operation and are the main reason affecting the speed of network operation. At present, the convolution (or multiplication and addition) operation in a neural network is mostly completed in software.

[0003] The conductance value of a memristor, or RRAM (Resistive Random Access Memory), changes with the voltage applied to the upper and lower poles, and remains the previous conductance value when no voltage is applied. The memristor is a two-terminal passive device compatible with the CMOS process, and the relatively mature working mode is to switch between two high-discrimination conductance states HRS and LRS to store data as a non-volatile memory, or to construct a binary neural network using the two conductance states. However, the conductance of the memristor does not jump between the two states, but there is a continuous and monotonic increase / decrease process with the voltage. Considering its high integration, passivity, non-volatility, and compatibility with the CMOS process, the conductance value of the memristor can be used to represent the weight of the trained neural network, so as to perform high-speed and low-power convolution (or multiplication and addition) operation at the hardware level using Kirchhoff's law. SUMMARY

[0004] In order to realize the optimization of convolutional neural network calculation at the hardware level using a memristor, the present application aims to provide a hardware convolutional neural network system based on a memristor, which can be used for (but not limited to) image recognition of MNIST dataset, including the following modules.

[0005] Memristor array module. This module is the core calculation unit of the entire model, which selects the target memristor by controlling the voltage on WL, BL and WL, and then programs the conductance of the target memristor by applying a pulse voltage on BL or reads the current on SL to complete the convolution operation on the input data using Kirchhoff's law.

[0006] Peripheral circuit module matched with the memristor array. This module serves the memristor array module and contains a voltage generation module and a digital-to-analog conversion module for generating specific voltage values, etc., for generating specific voltages to drive the memristor array and realizing the communication between the memristor array and the FPGA and the computer.

[0007] FPGA module. The main function of this module is to control the operation of the memristor array module and the peripheral circuit module, and to realize the communication with the computer.

[0008] Computer module. The main function of this module is data processing, including weight encoding, inter-layer data quantization, and pooling, padding, classification and other operations required during neural network operation. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The method for representing weights by RRAM differential pairs.

[0010] Figure 2 The circuit implementation diagram of convolution calculation.

[0011] Figure 3 The method for representing gray value by voltage coding. DETAILED DESCRIPTION

[0012] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in conjunction with the drawings and examples. For the convenience of understanding, the following description takes the convolutional neural network for MNIST dataset as an example, and it should be understood that the implementation examples described herein are only used to illustrate and explain the present application, but the present application is not limited to the implementation of the convolutional neural network based on MNIST dataset.

[0013] Mapping of weights and RRAM conductance, see Figure 1 After software training, the neural network weights are floating-point numbers between -1 and +1, which need to be mapped with the conductance of RRAM, so as to represent the weights by the conductance of RRAM. Taking RRAM capable of realizing 32 relatively independent and high-resolution conductance states (2 μs-20 μs, Δ=0.58 μs) as an example, 7 conductance states (2 μs-20 μs, Δ=2.5 μs) are selected, and a pair of RRAMs is used to form a differential RRAM pair. By applying positive and negative voltages respectively and combining different positive and negative voltages, a differential RRAM pair can realize 15 conductance states between -17.5 μs and +17.5 μs, Δ=2.5 μs, that is, a differential RRAM pair can represent the positive or negative weight of the connection between two neurons in the neural network. The weights after software training are mapped to the conductance values of the hardware RRAM according to the interval, such as The conductance of the RRAM differential pair corresponding to the weight between -17.5 μs and +17.5 μs is 0 μs, The conductance of the RRAM differential pair corresponding to the weight between -17.5 μs and +17.5 μs is 0 μs,

[0014] Programming method for RRAM conductance. The RRAM conductance value can be continuously changed by applying large pulse voltages. For SET operation, the voltages on WL, BL, and SL are 1.8V, 2.0V (50ns pulse), and 0V, respectively; for RESET operation, the voltages on WL, BL, and SL are 4.7V, 0V, and 1.8V (50ns pulse), respectively.

[0015] For the mapping between convolution calculations and hardware circuits, see [link to relevant documentation]. Figure 2 For the convolution window in the figure, x 2-2 Let w be the grayscale value of the original input image. 2-2 The weights are the values ​​of the convolution kernel. After programming the conductance values ​​of the RRAM to the weights of the corresponding convolution kernels as described above, voltages representing the grayscale values ​​of different pixels in the image are input into each BL. Then, SL... + With SL - The differential current is the result of convolving the nine points covered by the kernel. Sampling the current value allows it to be used for subsequent pooling and calculations of other network layers. The same applies to fully connected layers.

[0016] For the method of transferring input data to convolutional / fully connected layers via voltage, see [link to relevant documentation]. Figure 3 Taking a convolutional layer as an example, for a given convolutional window, the nine input values ​​correspond to the voltages on nine black bars (BLs). For a given input, its value is programmed into a corresponding 8-bit binary value, represented by eight pulse voltage cycles (0.2V). A "1" or "0" in the binary representation indicates whether a pulse exists in that cycle. For example, if... Figure 2 Chinese x 2-2 If the input value at a certain point is 109, its corresponding binary value is 01101101. During calculation, the pulse voltage waveform input to the RRAM array is as follows: Figure 3 As shown. Let the current sampled on SL in the k-th period be I. k The current on SL is sampled for eight cycles, and the current is obtained by shifting and adding the currents. SL This is the result of the convolution corresponding to the convolution window, as shown in formula (1).

[0017]

[0018] In practical applications, subsequent layers of the neural network will have a larger input range. For this larger input range, it will be quantized and compressed into an 8-bit fixed-point number and sent to the next layer. The quantization method is described in section

[0019] . For example, if the output of a certain layer is 0-2550, it will be converted back into a fixed-point number of 0-255 with a step size of 10 as the input of the next layer.

[0019] The method of inter-layer data quantization. Since the input of each layer of the hardware neural network model in the present application is represented by voltage, it is impossible to achieve the high-precision input of software neural network, therefore, a quantization method is adopted, i.e., the output of each layer is quantized to 8 bits after being sampled to current by the peripheral circuit mentioned in paragraph

[0006] , and then the current value is input to the computer for quantization, and then the quantized data is input to the next layer by the method mentioned in paragraph

[0016] . The quantization can greatly simplify the operation while ensuring the accuracy. Assuming that the data of the output of a certain layer is a high-precision floating point number r (r1, r2, r3, …), and the fixed point number obtained after quantization is q (q1, q2, q3, …), the quantization relationship can be established by the following formula (1).

[0020]

[0021] wherein S is a scaling factor, which is the proportional relationship between the value range of the floating point number before quantization and the fixed point number after quantization; and Z is a fixed point zero point, which represents the integer corresponding to 0 in the real number after quantization. S and Z are calculated by the following formula (2) and formula (3), respectively.

[0022]

[0023]

[0024] The role of the peripheral circuit module matched with the RRAM array. Under the driving of the FPGA, the peripheral circuit generates corresponding programming voltage to program the RRAM conductance, inputs corresponding pulse sequence to make the array perform convolution operation during the calculation operation, and reads the calculation result, and communicates with the FPGA through the ADC and DAC modules.

[0025] The role of the FPGA development board. The FPGA development board in the model assumes the role of interacting with the computer for the hardware circuit part composed of the RRAM array and the peripheral circuit, and simultaneously generates control signals to drive the orderly operation of each module of the hardware circuit part.

[0026] The role of the computer. The computer in the model assumes the role of other calculations except convolution: the mapping of the weight and the RRAM conductance described in paragraph

[0013] needs the computer to rewrite the trained weight and program it as the conductance of the corresponding RRAM differential pair; the quantization described in paragraph

[0019] needs to be completed in the computer; the pooling and padding after each layer calculation of the neural network need to be completed in the computer; and the calculation of the activation function in the fully connected layer and the classification function of the last fully connected layer also need to be completed in the computer.

[0027] After the above parts are fully configured, a complete convolutional neural network in which the convolution calculation is completed in the hardware circuit composed of RRAM can be realized.

Claims

1. A convolutional neural network system based on a memristor, comprising a memristor array module as the core of the whole system, a peripheral circuit module for generating a specific voltage to drive the memristor array and realizing communication between the memristor array and an FPGA and a computer, an FPGA module for controlling the operation of the memristor array module and the peripheral circuit module and realizing communication between the FPGA and the computer, and a computer module for data processing, including encoding of weights, quantization of interlayer data, and pooling, padding, and classification operations required during neural network operation. The structure of the memristor array is a crossbar structure, and the connection mode is 1T1R. For a single weight corresponding to a pair of memristor differential pairs, one end is connected through a shared BL, and the other end is connected to two MOS tubes used as switches. The gate of the MOS tube is connected to two different word lines WL, and the other end of the MOS tube not connected to the memristor is connected to two different selection lines SL. For multiple weights corresponding to multiple memristor differential pairs in a convolution window, one end of the memristor to which a positive voltage is applied is connected to different bit lines BL, and the other end is connected to different MOS tubes. The gate of the connected MOS tube is connected to the same WL, and the other end of the MOS tube not connected to the memristor is connected to the same SL. For multiple weights corresponding to multiple memristor differential pairs in a convolution window, one end of the memristor to which a negative voltage is applied is connected to different BL, and the other end is connected to different MOS tubes. The gate of the connected MOS tube is connected to the same WL, and the other end of the MOS tube not connected to the memristor is connected to the same SL. 2.The convolutional neural network system of claim 1, wherein: The variable conductance of the memristor is used to represent the weight in the neural network; a set of differential memristor pairs working at positive voltage and negative voltage are used to realize the positive weight or negative weight in the neural network.

3. The convolutional neural network system of claim 1, wherein, Through the method of quantization, each layer of data is compressed; through the form of pulse voltage, the quantized data is sent into the memristor array in the form of digital pulse voltage for calculation, and then the obtained current is weighted and summed by bit to obtain the result of convolution calculation.

4. The convolutional neural network system of claim 1, wherein, By applying voltage to the memristor and reading the corresponding current, the convolution calculation in the convolution neural network is realized on the hardware by using Kirchhoff's law.

Images