Method for implementing neuron random inactivation in neural network

Abstract

The application provides a realization method of neuron random inactivation in a neural network, and belongs to the technical field of semiconductors, artificial intelligence and neuromorphic computing. The realization method utilizes the threshold voltage adjustable characteristic of a laminated device to complete the dropout function of the neural network. Compared with the traditional CMOS realization dropout function, the realization method has small area consumption, and has important significance for the research of a future new type of neural network acceleration chip.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of semiconductor, artificial intelligence and neuromorphic computing, and particularly relates to a method for implementing random inactivation of neurons in a neural network. BACKGROUND

[0002] With the revolution of information technology, artificial intelligence is leading a new round of revolution and promoting the progress of society in various aspects. Although deep learning represented by neural networks has excellent performance in image recognition, speech processing and autonomous driving, etc., the frequent matrix-vector multiplication operation is required in neural network computing. The existing von Neumann computing architecture separates the calculation unit and the storage unit, which causes the problem of "memory wall" in data exchange, limits the running efficiency of the computing system, and thus reduces the performance of information transmission and storage. On the other hand, in the intelligent era, the depth and scale of the neuron network are also increasing due to the massive data, which causes the explosive growth of network parameters and the problem of overfitting, resulting in poor generalization ability of the network model. To solve the problem of overfitting, researchers propose the dropout method, but this puts forward new requirements for the performance of neurons, which requires the neurons to be inactivated with a certain probability during the training process, and an additional circuit module is needed to achieve this, which poses a great challenge to the implementation of high-density computing platforms.

[0003] The neuromorphic computing platform that realizes the integration of storage and calculation based on new neuromorphic devices is the future development direction. At present, artificial synapses are mainly realized by multi-value memory devices with good gradual change characteristics, such as phase change memory and other new non-volatile devices. Artificial neurons are mainly realized by threshold transition devices and ion conductive devices, but it is difficult to realize selective inactivation of artificial neurons, which makes it difficult to implement the dropout algorithm of neural networks based on these new devices, and inhibits the performance improvement of neural networks. The stacked device based on the volatile layer and the non-volatile layer can realize the regulation of the threshold voltage of the neuron. When the threshold voltage of the neuron is high, the neuron is in an inactivated state, and when the threshold voltage of the neuron is low, the neuron is in an active state. This stacked neuron can switch between active and inactivated states with good power consumption and performance, providing a new solution for implementing the dropout of neural networks based on new devices. SUMMARY

[0004] In order to overcome the problems of separation of storage and calculation in the existing von Neumann computing architecture and large area consumption of implementing the dropout function of neural networks based on traditional CMOS devices, the application provides a method for implementing random inactivation of neurons in a neural network based on a stacked device and a peripheral circuit.

[0005] The technical solutions of the present application are as follows:

[0006] A method for implementing random inactivation of neurons in a neural network, characterized in that,

[0007] 1) Construct a neural network storage and calculation array, each column of the neural network storage and calculation array is connected with an operational amplifier and a neuron module at the end, the positive input end of the operational amplifier is connected with the array, and the negative input end is clamped to zero level, the operational amplifier is connected with the neuron module through a feedback resistor G, the neuron module includes a laminated device, a resistor and a capacitor, wherein the laminated device and the capacitor are connected in parallel and then connected in series with the resistor, and the laminated device is composed of a top electrode, a volatile layer, a non-volatile layer and a bottom electrode.

[0008] 2) Convert the signal into a matrix pulse vector V i Input the neural network storage and calculation array, and obtain the input voltage V in of the neuron module through the operational amplifier. in Charge the parasitic capacitor of the laminated device, and the voltage on the capacitor increases continuously, when the voltage V on the volatile layer exceeds the threshold opening voltage V th of the neuron, that is, the threshold voltage of the neuron, the volatile layer becomes low resistance, and the laminated device is in a low resistance state, and the neuron module emits a pulse signal "1", when the voltage on the volatile layer is lower than the holding voltage V hold of the neuron, which is equivalent to the neuron returning to the resting potential, the volatile layer becomes high resistance again, and the laminated device is in a high resistance state, and the laminated device is recharged and starts the accumulation process, and then the emission process is accompanied, and the cycle is repeated to realize periodic oscillation.

[0009] 3) Adjust the threshold voltage of the neuron by applying voltages of different polarities to the positive and negative electrodes of the laminated device, when a negative voltage is applied, the conductance of the non-volatile layer decreases, the threshold voltage of the neuron increases, and the neuron inactivation Reset operation is realized; when a positive voltage is applied, the conductance of the non-volatile layer increases, the threshold voltage of the neuron decreases, and the neuron activation Set operation is realized, so that the random inactivation of neurons in the neural network is realized.

[0010] The neural network storage and calculation array can be built by using devices such as memristor, phase change memory, ferroelectric memory, static random access memory and dynamic random access memory.

[0011] The material of the volatile layer of the laminated device is VO x or NbO x , and the material of the non-volatile layer is TaO x or HfO xThe electrode material can be selected from Pt or TiN material compatible with the CMOS process. Generally, the amplitude of the operating voltage increases with the thickness of the stacked device. In order to better combine the use of the circuit, the operating voltage of the device should not be too large, so the device cannot be too thick. Generally, the thickness of the non-volatile layer should be less than 10nm, and the thickness of the volatile layer should be less than 30nm, so as to ensure that the operating voltage does not exceed 3 volts.

[0012] The beneficial effects of the present application are as follows:

[0013] The present application proposes a neural network dropout method based on a stacked device. The stacked neuron can switch between the active and inactive states, and complete the neural network dropout function. Compared with the traditional CMOS dropout function, the area consumption is small, and it has important significance for the research of future new type neural network acceleration chip. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A 1T1R cross array storage and calculation architecture structure diagram is constructed in the embodiment of the present application;

[0015] Figure 2 The neuron module diagram in the embodiment of the present application, (a) is a neuron module structure, and (b) is a structure diagram of the stacked device shown in (a);

[0016] Figure 3 The oscillation waveform diagram of the stacked device in the embodiment of the present application;

[0017] Figure 4 The diagram of the pulse firing frequency with the size of the excitation signal in the embodiment of the present application;

[0018] Figure 5 The diagram of the current-voltage curve change in the process of implementing the neural network Set and Reset in the embodiment of the present application;

[0019] Figure 6 The DC voltage scanning result diagram of the stacked device in the embodiment of the present application, (a) shows the response pulse waveform diagram of the stacked artificial neuron when the amplitude of the pulse excitation is 5V and the pulse width is 200us, and the stacked neuron oscillates (HfO2 is in a low resistance state); (b) is the I-V curve when HfO2 is in a low resistance state; (c) shows the response pulse waveform diagram of the stacked artificial neuron when the amplitude of the pulse excitation is 5V and the pulse width is 200us, and the stacked neuron cannot oscillate (HfO2 is in a high resistance state); (d) is the I-V curve when HfO2 is in a high resistance state. DETAILED DESCRIPTION

[0020] The present application will be further clarified by specific embodiments with reference to the accompanying drawings.

[0021] In specific embodiments of the present application, a memristor is selected to construct a neural network storage and calculation array. The memristor is widely concerned due to its small area, low power consumption, adjustable resistance simulation and other advantages. The continuous electrical change characteristics and non-volatile characteristics of the memristor can simulate the synaptic weight storage and continuous change of the mutual connection of biological neurons, and have great potential in the application of brain-like neural morphological chips. A 1T1R memristor neural network storage and calculation array is constructed as shown in the figure. Figure 1 The memristor device is connected in series at the drain end of the transistor to form a storage and calculation unit, which avoids the lifting of the threshold voltage of the transistor due to the voltage division of the memristor. Each row in the array is called a bit line (BL), and the word line (WL) is connected to the gate end of each column of transistors, and the source line (SL) is connected to the source end of each column of transistors. At the same time, in order to represent positive and negative weights Two rows of memristors are used to represent a weight w ij The total weight The bit line of the array is connected to the input signal, the word line is connected to the control signal, and the output signal of the multiplication and accumulation is output by the source end for subsequent processing. The end of each column of the 1T1R memristor array is connected to an operational amplifier and a neuron module. The positive input end of the operational amplifier is connected to the source end of the array, and the negative input end is clamped to zero level. The operational amplifier is connected to the neuron module through a feedback resistor G. The neuron module is shown in the figure. Figure 2 The neuron module includes a laminated device, a resistor and a capacitor, wherein the laminated device and the capacitor are connected in parallel and then connected in series with the resistor. The laminated device is composed of a top electrode, a volatile layer, a non-volatile layer and a bottom electrode. For example, the laminated device is an HfO2 / NbO2 laminated device.

[0022] The multiplication and accumulation function is realized: first, the signal is converted into a matrix pulse vector V i The total current on the SL of each column is obtained from the BL according to Kirchhoff's law:

[0023]

[0024] This step corresponds to the output result of matrix multiplication in the neural network. In order to realize the nonlinear transformation function, the multiplication needs to be activated through the neuron. The sum current passes through the operational amplifier module to obtain the input voltage Vin j of the neuron (each column has a neuron module).

[0025]

[0026] Where G is the output resistance of the operational amplifier.

[0027] Activation function implementation: input voltage Vin jThe neuron will be excited to fire a spike, the spike firing process is as follows, the input voltage V in charges the parasitic capacitance of the stacked device, the voltage on the capacitance increases continuously, when the voltage on the volatile layer VO2 is divided over its threshold opening voltage V th , that is, the threshold voltage of the neuron, the volatile layer VO2 becomes low resistance, and the stacked device is in a low resistance state, the neuron fires a spike signal "1", and the charging time is set as t r . Due to the decrease of the resistance of the volatile layer in the stacked device, the total voltage of the stacked device decreases, when the voltage on the volatile layer VO2 is divided hold below its holding voltage V f , which is equivalent to the neuron returning to the resting potential, the volatile layer VO2 becomes high resistance again, and the stacked device is in a high resistance state, the stacked device is recharged, and the accumulation process begins, and then the firing process is repeated, and periodic oscillation is realized. The oscillation waveform is shown in Figure 3 , and the discharge time is set as t r and t f , as shown in formulas (3) and (4), so the oscillation frequency f can be represented by formula 5. The pulse firing frequency changes with the size of the excitation signal, as shown in the attached Figure 4 , which is approximately a Relu activation function. The subsequent pulse is converted into a digital voltage by a counter, which can become the input voltage of the next layer network.

[0028]

[0029]

[0030]

[0031] where C m is the parasitic capacitance, V is the voltage on the stacked device, R is the series resistance, is the resistance of the volatile layer VO2, R l is the resistance of the non-volatile layer HfO2, R off and R on are the high resistance and low resistance of the stacked device, is the voltage divided on the volatile layer VO2, V l is the voltage divided on the non-volatile layer HfO2, V th and V hold are the threshold voltage and resting potential of the neuron, respectively.

[0032] Neural network Dropout function implementation: In order to avoid overfitting in the process of neural network training, dropout is introduced, which refers to the inactivation of neurons in the neural network, that is, no pulse firing. Initially, all the neuron modules in the stack device are in the same blocking state, and the threshold voltage of all the neuron modules is the same. The number and position of randomly inactivated neurons in each layer of the neural network are determined by the network algorithm. If the dropout ratio is 50%, the probability of inactivation of each neuron in this layer is 0.5, and the number and position of inactivated neurons in each layer of the network are determined by the computer to generate random numbers. In order to realize the dropout function, the threshold voltage of the selected neurons is raised after each training, so that they cannot oscillate under the same input, thus realizing the dropout function. The regulation of neuron threshold voltage is realized by changing the conductance of non-volatile layer HfO2, and the change of conductance of non-volatile layer HfO2 can be realized by applying different polarity voltages to the positive and negative electrodes. When a negative voltage is applied, the conductance of the non-volatile layer decreases, and the threshold voltage of the neuron increases, realizing the Reset operation. When a positive voltage is applied, the conductance of the non-volatile layer increases, and the threshold voltage of the neuron decreases, realizing the Set operation. Figure 5 The current-voltage curve in the Set and Reset processes is shown. When HfO2 is in LRS (Low Resistance State, LRS), the threshold voltage V th is about 2.1V. Thereafter, as the negative voltage increases, the resistance gradually increases, and the device undergoes the Reset process. During the negative voltage back scan process, the slope of the device decreases, indicating that the resistance increases at this time, and HfO2 is placed in the high resistance state HRS (How Resistance State, HRS). Subsequently, forward voltage scanning is performed, and when the forward turn-on voltage is reached, the current suddenly increases. Thereafter, the Set process is performed, the resistance of HfO2 decreases, and the current flowing through the device reaches the instrument limit current and is clamped. When the forward voltage back scan decreases to about 1.9V, the current of the device suddenly decreases, and the resistance value suddenly increases, indicating that the threshold transition occurs at this time. As can be seen from the I-V curve, the resistance value at this time is smaller than the initial high resistance value, indicating that HfO2 is placed in the low resistance state, and the device undergoes non-volatile transition. Figure 5 It is shown that the device can switch between Non-volatile-HRS (NVHRS) and Non-volatile-LRS (NVLRS), and has threshold switching characteristics. When HfO2 is in the low resistance state LRS, an excitation with an amplitude of 5V is applied to the device, and the stack neuron fires a pulse. When direct current scanning is performed, the device reaches the clamping current of 100μA at about 2V, as shown in Figure 6 (a)-(b). Then negative direct current scanning is performed to reset the operation, and HfO2 is placed in the high resistance state, as shown in Figure 5The left part curve of Fig. 2 shows that when the device is applied with an excitation of 5V in amplitude and 1ms in width, the laminated neuron has no impulse response. When direct current is scanned, the current of the device is only 5.9μA at 2V. Figure 6 (c)-(d) show that the neuron is inactivated at this time, and the dropout function is realized.

[0033] The above examples are only used to illustrate the technical solutions of the present application but not to limit it, and the ordinary skilled in the art can modify or equivalently replace the technical solutions of the present application without departing from the spirit and scope of the present application, and the protection scope of the present application should be subject to the description of the claims.

Claims

1. A method for implementing neuron random deactivation in a neural network, the method comprising: Comprising the following steps: 1) Constructing a neural network memory and calculation array, connecting an operational amplifier and a neuron module at the end of each column of the neural network memory and calculation array, the positive input end of the operational amplifier is connected with the array, and the negative input end is clamped to zero level, and the operational amplifier is connected with the neuron module through a feedback resistor The neuron module comprises a laminated device, a resistor and a capacitor, wherein the laminated device and the capacitor are connected in parallel and then connected in series with the resistor, and the laminated device is composed of a top electrode, a volatile layer, a non-volatile layer and a bottom electrode 2) converting the signal into a matrix pulse vector The input voltage of the neuron module is obtained by inputting the neural network storage and calculation array through an operational amplifier The input voltage charges the parasitic capacitance of the laminated device, and the voltage on the capacitor continuously increases When the voltage on the volatile layer exceeds its threshold opening voltage , that is, the threshold voltage of the neuron, the volatile layer becomes low resistance, and the laminated device is in a low resistance state, and the neuron module emits a pulse signal "1"; when the voltage on the volatile layer is lower than its holding voltage , which is equivalent to the neuron returning to the resting potential, the volatile layer becomes high resistance again, and the laminated device is in a high resistance state, and the laminated device is recharged and starts to accumulate, and then accompanied by the emission process, the cycle is repeated, realizing periodic oscillation; 3) The threshold voltage of the neuron is regulated by applying different polarity voltage on the positive and negative electrodes of the stacked device, when a negative voltage is applied, the conductance of the non-volatile layer decreases, the threshold voltage of the neuron increases, the neuron inactivation operation is realized; when a positive voltage is applied, the conductance of the non-volatile layer increases, the threshold voltage of the neuron decreases, the neuron activation operation is realized, thereby realizing the random inactivation of neurons in the neural network.

2. The method of claim 1, wherein the method further comprises: In step 1), a 1T1R resistive memory neural network storage and calculation array is constructed, each storage and calculation unit in the array is composed of a resistive memory device connected in series at the drain end of a transistor, each row in the array is called a bit line, and the word line connected with the gate end of each column transistor is the word line, and the source line connected with the source end of each column transistor is the source line, and two rows of resistive memory devices are used to represent a weight The input signal is connected to the bit line in the array, the control signal is connected to the word line, and the output signal obtained by multiplication and accumulation is output by the source end.

3. The method of claim 2, wherein the step of randomly deactivating neurons in the neural network is performed by the steps of: randomly selecting a neuron in the neural network; and deactivating the randomly selected neuron. In step 2 The total current on each column of source lines: then the input voltage ; wherein Rout is the output resistance of the operational amplifier.

4. The method of claim 3, wherein the method further comprises: In step 2 Let the charging time be , let the discharging time be , and the oscillation frequency be calculated as follows: wherein Cp is the parasitic capacitance, V is the voltage across the stack, Rs is the series resistance, Rv is the volatile layer resistance, Rnv is the non-volatile layer resistance, and Rhi and Rlo are the high and low resistances of the stack, Vv is the voltage across the volatile layer, Vnv is the voltage across the non-volatile layer, and Vth and Vr are the neuron threshold voltage and resting potential, respectively.

5. The method of claim 1, wherein the method further comprises: The material of the non-volatile layer of the stacked device employs TaO x or HfO x .

6. The method of claim 1, wherein the method further comprises: The material of the volatile layer of the stacked device employs VO x or NbO x .

7. The method of claim 1, wherein the method further comprises: determining a number of neurons to be randomly inactivated; and randomly inactivating the determined number of neurons. The thickness of the non-volatile layer of the stacked device is less than 10 nm.

8. The method of claim 1, wherein the method further comprises: determining a number of neurons to be randomly inactivated; and randomly inactivating the determined number of neurons. The thickness of the non-volatile layer of the stacked device is less than 30 nm.

Images