Neural network algorithm design method and neural network operation method

Abstract

The application discloses a neural network algorithm design method and a neural network operation method based on resistive memory, and comprises the following steps: determining the nonlinear response characteristics of the resistive memory, and determining the adjustment mode of the resistive memory according to the response characteristics; determining the parameterization method of the neural network back propagation, wherein the number, connection mode and signal conversion mode between neural network layers of the resistive memory are determined according to the task of the neural network; realizing the forward propagation process of the neural network operation according to the adjustment mode of the resistive memory, the number, connection mode and signal conversion mode between neural network layers of the resistive memory; realizing the back propagation process of the neural network according to the parameterization method of the neural network back propagation; and training the target task to obtain a trained neural network model. The hardware architecture in the application does not need to occupy a large area and power consumption analog-to-digital converter, and can greatly reduce the power consumption and design area of the storage and calculation hardware.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural networks, and particularly relates to a neural network algorithm design method and a neural network operation method. BACKGROUND

[0002] In recent years, the powerful parallel computing performance of a graphics processing unit (GPU) has significantly reduced the computing bottleneck in the training process of a deep learning algorithm, greatly promoting the development and prosperity of the field of artificial intelligence. However, the current mainstream computing architecture, such as a central processing unit (CPU) and a GPU of a von Neumann structure, not only usually consumes power up to kilowatts during computing, but also has its computing efficiency limited by the frequent transmission of data between the computing and storage units.

[0003] In order to solve the above problems, a series of non-von Neumann structure neuromorphic computing devices and technologies have been proposed, including traditional CMOS devices and emerging phase change memory (PCM), resistive random access memory (RRAM) and conductive bridging random access memory (CBRAM). Among them, the memory device based on resistance characteristics can represent the weight of the neural network by storing the conductance value, and the current output on the bit line realizes the sum of the product of the conductance of multiple storage units and the corresponding input voltage, which makes them one of the important representative devices for matrix multiplication operation acceleration.

[0004] Traditional storage and computing hardware cannot be separated from the support of analog-to-digital converter (ADC), digital-to-analog converter (DAC) and other peripheral circuits. Among them, ADC consumes most of the design area and power consumption of the storage and computing system hardware. When the network has multiple layers, in order to realize the interlayer nonlinear activation process, the signal needs to be frequently converted between analog and digital between the storage and computing arrays, which weakens the power consumption advantage of the storage and computing integration.

[0005] Therefore, it is particularly necessary to realize nonlinear activation by physical means using the natural electrical characteristics of the device to save hardware area and power consumption, and to realize the performance optimization of the neural network circuit. SUMMARY

[0006] In view of the defects of the prior art and the improvement needs, the application provides a neural network algorithm design method based on resistive memory and a neural network operation method, which aims to save hardware area and power consumption and realize performance optimization of a neural network circuit.

[0007] To achieve the above-mentioned purpose, according to one aspect of the present application, a neural network algorithm design method based on resistive memory is provided, which comprises:

[0008] determining the nonlinear response characteristics of the resistive memory, and determining the adjustment mode of the resistive memory according to the response characteristics;

[0009] determining the parameterization method of the neural network back propagation according to the adjustment mode of the resistive memory, wherein the number, connection mode and signal conversion mode between neural network layers of the resistive memory are determined according to the task of the neural network;

[0010] implementing the forward propagation process of the neural network operation according to the adjustment mode of the resistive memory, the number, connection mode and signal conversion mode between neural network layers of the resistive memory;

[0011] implementing the back propagation process of the neural network according to the parameterization method of the neural network back propagation;

[0012] training the target task by combining the forward propagation process and the back propagation process to obtain a trained neural network model. Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0013] The natural electrical characteristics of the resistive memory are used to realize the activation function operation and the matrix multiplication operation, replacing the operation processes such as Sigmoid and ReLU in the traditional neural network. Based on this, the hardware architecture in the neural network algorithm design disclosed by the present application no longer needs to occupy a large area and power consumption of the analog-to-digital converter and digital-to-analog converter part, which can greatly reduce the power consumption and design area of the storage and calculation hardware. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A nonlinear response schematic diagram of the resistive memory in the neural network circuit according to the embodiment of the present application is provided.

[0015] Figure 2 A schematic diagram of the resistive memory in the neural network circuit according to the embodiment of the present application is provided.

[0016] Figure 3 A structure schematic diagram of the neural network circuit based on the resistive memory according to the embodiment of the present application is provided.

[0017] Figure 4 is a structural schematic diagram of a traditional neural network circuit.

[0018] Figure 5 is a structural schematic diagram of a resistive memory-based neural network circuit according to an embodiment of the present application.

[0019] Figure 6 is a circuit structural diagram of a nonlinear memory-computing array according to an embodiment of the present application Figure 5 DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0021] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0022] The present application proposes a neural network algorithm design method and a neural network operation method based on the nonlinear response characteristics of resistive memory. The neural network algorithm design method is based on a resistive memory array. The resistive memory array is connected in a Crossbar structure, as shown in Figure 6 The core operation process of the neural network algorithm design is determined by the physical characteristics of the memory device itself. In other words, the natural nonlinear response characteristics of resistive memory are used to realize two core steps in artificial neural networks: product accumulation operation and activation function step. The problems solved include: reducing the activation function step and the product-add step into one step, i.e. using the nonlinear characteristics of resistive memory to complete.

[0023] Embodiment 1:

[0024] The present application discloses a neural network algorithm design method based on resistive memory, comprising:

[0025] determining the nonlinear response characteristics of the resistive memory, and determining the adjustment mode of the resistive memory according to the response characteristics, the adjustment mode of the resistive memory comprising: determining the nonlinear response function through external input parameters or determining the nonlinear response function by internal parameters of the resistive memory;

[0026] ​The parameterization method of neural network back propagation is determined according to the adjustment mode of the resistive memory, wherein the number, connection mode and signal conversion mode between neural network layers of the resistive memory are determined according to the task of the neural network; the number and connection mode of the resistive memory are determined by the target processing task, and are associated with processing data and the like; the data to be processed is input into the resistive memory, and the resistive memory outputs a calculation result; the input is a voltage signal, and the output is a current signal.

[0027] The forward propagation process of neural network operation is realized according to the adjustment mode of the resistive memory, the number, connection mode and signal conversion mode between neural network layers of the resistive memory.

[0028] The back propagation process of the neural network is realized according to the parameterization method of neural network back propagation; specifically, in the continuous adjustment mode, the parameterization method of back propagation realizes the back propagation calculation of the neural network by deriving the universal function; in the discrete adjustment mode, the parameterization method of back propagation realizes the calculation of back propagation by making the derivative of the discrete adjustable state continuous based on Gumbel-softmax.

[0029] The target task is trained to obtain a trained neural network model by combining the forward propagation process and the back propagation process.

[0030] In order to better understand the present application, the design idea of the neural network algorithm based on the nonlinear response characteristics of the resistive memory will be described in conjunction with the following figure. Figure 1 The nonlinear response of the resistive memory in the neural network circuit according to the embodiments of the present application is shown in the figure. Figure 1 The voltage-current curve of the resistive memory provided by the present application is shown in the figure. The resistive memory is a resistive memory based on a phase change material, and the voltage-current curve has nonlinear characteristics. The response characteristics and material characteristics of the device determine that the adjustment mode can be realized by changing the crystallization or amorphization ratio of the phase change material inside by applying an electric pulse. In other words, different voltage-current curves are obtained by applying different electric pulses to the resistive memory.

[0031] Specifically, Figure 2 The schematic diagram of the nonlinear output adjustment of the resistive memory in the neural network circuit based on the external input parameter A according to the embodiments of the present application is shown in the figure.

[0032] By applying different electric pulses to the resistive memory, the resistive memory is designed to have different resistance states, thereby designing different nonlinear response characteristics. Different electric pulse application processes correspond to different response characteristic curves, and in the embodiments of the present application, the adjustment process of the phase change material is determined by the external input electric pulse.

[0033] Based on the above resistive memory, the present application is based on the design method of resistive memory neural network algorithm, which realizes the nonlinear activation operation and matrix multiplication operation in neural network operation. Specifically, the response characteristic curve of the resistive memory is designed based on the electric pulse, which includes the following processes: determining the parameterized method of back propagation, and designing the specific device number, connection mode and conversion form of interlayer signal according to the target neural network task. Since the response characteristic curve relies on different pulse application processes, the response characteristic curve creation process includes discrete or continuous.

[0034] In an embodiment of the present application, different response characteristics can still achieve intermediate nonlinear response states by continuing to adjust the pulse application process, that is, any intermediate state can change the response characteristic curve by continuing to adjust the pulse, and any intermediate state can be described by the total functional, which is a continuous perspective; If the existing different nonlinear response characteristics are directly operated, the intermediate state is no longer realized by continuing to adjust the pulse application process, or the intermediate state is difficult to describe by the functional, and the calculation process can only be achieved by a specific number of intermediate states, the total number of different nonlinear response characteristics is certain, which is a discrete perspective. Under the pulse response curve of the continuous perspective, the nonlinear response characteristic FA(X) of different parameter adjustment amounts A is fitted to form a derivable functional of A and F(X), that is, the entire neural network construction is completed by back propagation. For the discrete perspective, the Gumbel-Softmax parameterization method can be used to make the derivative of the discrete adjustable state continuous to realize the calculation of back propagation for the nonlinear response characteristic FB(X) corresponding to different parameters B, wherein the determination process of the parameters A and B is obtained in the selection process of the target neural network task based on the adjustable state of the resistive memory, and the external input parameter A and the internal parameter B of each resistive memory in the trained neural network model are determined.

[0035] It should be noted that the neural network algorithm design method based on the resistive memory disclosed in the present embodiment will be described below with the embodiment of designing a 2-layer network with input features of 10 and output features of 3 to illustrate the neural network algorithm design process, as shown in Figure 3 The neural network architecture is shown, and those skilled in the art should understand that the above-mentioned neural network architecture is only an example and is not a limitation of the present application, and the number of network layers, the number of input features and the number of output features are not limited to the above-mentioned embodiments.

[0036] Further, in order to better understand the idea of the present application, as Figure 3Fig. 1 shows a structural diagram of a neural network algorithm architecture according to one embodiment of the present application, which includes 10 inputs and 3 outputs. Those skilled in the art should understand that Figure 3 In one embodiment, the number of inputs and outputs of the neural network algorithm architecture can be designed according to the data to be processed. Specifically, the inputs are used to receive data to be processed, and the inputs are located in the first layer of the neural network algorithm architecture. For example, when the neural network algorithm architecture uses Figure 1 resistive memory, the voltage amplitude is used as the information carrier, and the voltage and current are in a nonlinear function relationship. The second layer of the neural network algorithm architecture includes 5 nodes, which correspond to the intermediate layer or hidden layer in a traditional neural network. Each node of the second layer is connected to the 10 inputs, and the results of the 10 nonlinear operations are summed, which is equivalent to completing the matrix nonlinear transformation and summation operation. The second layer converts the current to a voltage value, and the second layer continues to serve as the input voltage of the next layer. The third layer represents the categories of the target task, such as the categories of the classification task or the categories of the probability prediction. The 10-dimensional data is processed by the neural network operation to obtain the categories of the target task or the classification. The number of categories obtained by the target task is not limited to 3. The structure of the neural network and the target task are not limited to the above architecture. The size of the resistive memory is 10x5 for the first layer and 5x3 for the second layer. Since the resistive memory is still a traditional voltage input and current output of the computing-in-memory, the connection mode can still use the Crossbar structure to achieve current summation by Kirchhoff's law. Since the output of the resistive memory is current, the conversion form of the interlayer signal is to convert the output current of the previous layer to the input voltage of the next layer, i.e., the current-to-voltage form. As described above, all information of the neural network operation, such as the nonlinear operation function, the interlayer signal conversion form, and the neural network parameter size, can be learned by the backpropagation method. For example, when the data to be processed has 10 inputs and the output has 3 categories, the data processing is achieved by the combination of the nonlinear response states of 65 resistive memories. The nonlinear response of each resistive memory is trained or learned according to the specific task.

[0037] According to the corresponding relationship between the input parameter A and different nonlinear response states, the state adjustment process corresponding to each element can be obtained.

[0038] It should be noted that the resistive memory includes a phase change memory based on the change of resistance caused by the transformation between the crystalline state and the amorphous state to store information, or a memristor based on the change of resistance caused by the generation and rupture of conductive wires in the material to store information.

[0039] In summary, the neural network algorithm design based on resistive memory disclosed in the application is realized based on the nonlinear characteristics of resistive memory, and is flexibly designed according to the task required to be processed by the neural network, thereby greatly saving hardware resources.

[0040] Figure 4 For the traditional part of the hardware architecture according to the device as voltage input and current output, Figure 4 The nonlinear activation device in the traditional hardware architecture is realized by a digital circuit, which includes a computing array for receiving an analog input voltage and calculating a current sum; the input voltage is converted into a current sum after being input into the computing array, the current is input into an I / V converter to convert the calculated current into a voltage, and the voltage is input into an analog-to-digital converter to convert the voltage into a digital voltage signal; after the digital voltage signal is activated by a digital nonlinear activation unit, the signal is output to the next layer of computing array through a digital-to-analog converter. The calculation process of the traditional hardware architecture can be described as a combination of matrix multiplication operation and nonlinear activation operation, which corresponds to two core steps in the neural network.

[0041] Figure 5 For the hardware architecture diagram adopted by the neural network algorithm design based on resistive memory according to the embodiment of the application, the nonlinear activation function unit is omitted based on the nonlinear characteristics of resistive memory, and the convolution operation and the nonlinear activation operation of data in the target task are realized through the analog characteristics of resistive memory, since the value of the current sum itself has nonlinearity, it can be transmitted to the next layer as long as it can complete I / V conversion. As shown in Figure 5 The processed data is input into the first nonlinear computing array in the form of a voltage signal, and the output result is input into the second nonlinear computing array as the input of the next layer after I / V conversion, that is, the next layer of operation, here only one embodiment of the operation between adjacent layers in the neural network operation process is listed, different arrays are designed based on different operation requirements, in one implementation, the nonlinear computing array is composed of resistive memory connected by a Crossbar structure, as shown in Figure 6 .

[0042] Embodiment 2:

[0043] The application discloses a method for realizing neural network operation by the neural network algorithm design method based on resistive memory in embodiment 1:

[0044] The controller adjusts the volt-ampere characteristic curve of the resistive memory to obtain a target nonlinear characteristic curve, wherein the adjustment process includes applying an electric pulse to the resistive memory to change the resistance state of the resistive memory;

[0045] According to the nonlinear response curve of the resistive memory, the adjustment mode of the resistive memory is determined, wherein the adjustment mode of the resistive memory comprises: determining the nonlinear response function by an external input parameter or determining the nonlinear response function by an internal parameter of the resistive memory;

[0046] According to the adjustment mode of the resistive memory, the parameter method of the neural network circuit back propagation is determined, and according to the task target of the neural network, the number, connection mode and interlayer signal conversion form of the resistive memory are determined;

[0047] According to the adjustment mode, number, connection mode and interlayer signal conversion form of the resistive memory, the forward propagation of the neural network operation is completed;

[0048] According to the parameterization method of the neural network back propagation, the back propagation process of the neural network circuit is completed;

[0049] Based on the forward propagation and the back propagation, the neural network circuit is trained, the data to be processed is input to the trained neural network circuit, and the operation result is output.

[0050] The adjustment mode, number, etc. of the resistive memory are described in Embodiment 1, and are not described in detail to avoid repetition.

[0051] As described above, the electrical characteristics of the resistive memory are used to realize the activation function operation and the matrix multiplication operation, and replace the operation processes such as Sigmoid and ReLU in the traditional neural network. Based on this, the hardware architecture in the neural network algorithm design disclosed in the present application does not need to occupy a large area and power consumption of the analog-to-digital converter and digital-to-analog converter part, which can greatly reduce the power consumption and design area of the storage and calculation hardware.

[0052] It is easy for those skilled in the art to understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for implementing neural network operation based on a neural network algorithm design method of resistive memory: a controller adjusts a volt-ampere characteristic curve of the resistive memory to obtain a target nonlinear characteristic curve, wherein the adjustment process includes applying an electrical pulse to the resistive memory to change the resistance state of the resistive memory; determines the adjustment mode of the resistive memory according to the nonlinear response curve of the resistive memory, wherein the adjustment mode of the resistive memory includes: determining a nonlinear response function through an external input parameter or determining a nonlinear response function from an internal parameter of the resistive memory; determining a parameter method for back propagation of a neural network circuit according to the adjustment mode of the resistive memory, and determining the number, connection mode and interlayer signal conversion form of the resistive memory according to the task target of the neural network; completing the forward propagation of the neural network operation according to the adjustment mode, number, connection mode and interlayer signal conversion form of the resistive memory; completing the back propagation process of the neural network circuit according to the parameterization method of the neural network back propagation; training the neural network circuit based on the forward propagation and the back propagation, inputting the data to be processed into the trained neural network circuit, and outputting the operation result; wherein the neural network algorithm design method includes: determining the nonlinear response characteristics of the resistive memory, and determining the adjustment mode of the resistive memory according to the response characteristics; determining the parameterization method for back propagation of the neural network according to the adjustment mode of the resistive memory, wherein the number, connection mode and interlayer signal conversion mode of the neural network are determined according to the task of the neural network; completing the forward propagation process of the neural network operation according to the adjustment mode of the resistive memory, the number of resistive memories, the connection mode and the interlayer signal conversion mode of the neural network; completing the back propagation process of the neural network according to the parameterization method of the neural network back propagation; training the target task by combining the forward propagation process and the back propagation process to obtain a trained neural network model; wherein the nonlinear function F(X) of the nonlinear response characteristics of the resistive memory is determined according to an external input parameter A or by adjusting the internal parameter B of the resistive memory, and the determination process of the parameters A and B is obtained by selecting the target neural network task based on the adjustable state of the resistive memory, and the external input parameter A and the internal parameter B of each resistive memory in the trained neural network model are determined. 2.The method for implementing neural network operation based on the method for designing neural network algorithm of resistive memory according to claim 1, wherein, The neural network algorithm design method further includes that the input of the resistive memory is a voltage signal and the output is a current signal.

3. The method of claim 2, wherein the method of designing a neural network algorithm based on a resistive memory device performs a neural network operation, and wherein the method comprises: determining a plurality of weights of the neural network algorithm; and determining a plurality of weight values of the neural network algorithm based on the plurality of weights. The resistive memory includes a phase change memory based on the change of resistance caused by the transformation of material crystal state and amorphous state to store information, or a memristor based on the change of resistance caused by the generation and fracture of conductive wires to store information.

4. The method of claim 1, wherein the method of designing a neural network algorithm based on a resistive memory performs a neural network operation, and wherein the method comprises: determining a plurality of weights of the neural network algorithm; and determining a plurality of weight values of the neural network algorithm based on the plurality of weights. The parameterization method for back propagation includes: under the continuous adjustment mode, the parameterization method for back propagation is realized by deriving the functional function to realize the back propagation calculation of the neural network. In the discrete adjustment mode, the parameterized method of the back propagation realizes the calculation of the back propagation based on the Gumbel-softmax to make the derivative of the discrete adjustable state continuous.

5. The method of claim 1, wherein the method of designing a neural network algorithm based on a resistive memory performs a neural network operation, and wherein the method comprises: determining a plurality of weights of the neural network algorithm; and determining a plurality of weight values of the neural network algorithm based on the plurality of weights. The signal conversion mode between the neural network layers comprises an output result of a nonlinear transformation of a previous layer as an input signal of a next layer.

6. The method of claim 1, wherein the method of designing a neural network algorithm based on a resistive memory device performs a neural network operation, and wherein the method comprises: The operation process of the neural network is based on a plurality of resistive memories, different adjustable nonlinear states of each resistive memory, a connection mode of the resistive memories and signal conversion between neural network layers, wherein the connection mode of the resistive memories comprises a Crossbar structure connection. ​

Images