ReLU artificial neuron based on threshold switching device and improved Leaky ReLU artificial neuron thereof

Abstract

The ReLU artificial neurons play a key nonlinear activation function role in the deep artificial neural network. The ReLU artificial neuron is realized on a hardware circuit and is based on a special Pt / Ag / Al2O3 / HfO2 / Ag-NIs (5nm) / Pt threshold switching device embedded with a nano island, and the device has extremely low threshold voltage, low dispersity and high tolerance. The ReLU artificial neuron circuit is mainly composed of a threshold switching device and a parallel capacitor Cm, and a diode D1 and an adjusting resistor Rs are connected in series in front of the threshold switching device and the parallel capacitor Cm. An RC filtering unit composed of a capacitor Cf and a resistor Rf and a resistor Ro used for output detection are added to the output end of the circuit. Furthermore, a resistor Rd is added to be connected with the diode D1 in parallel, so that the improved Leaky ReLU artificial neuron circuit is formed. The artificial neuron provided by the invention has a great application prospect in constructing a brain-like artificial neural network.

Description

Technical Field

[0001] The invention relates to a method for realizing artificial neurons for brain-like artificial neural networks, in particular to a method for realizing artificial neurons based on threshold switching devices, and belongs to the field of information materials and applications. Background Art

[0002] In recent years, the development of deep neural networks has brought about the development of large language models based on Transformer, which has produced remarkable achievements and is expected to play a disruptive role in science and technology, social life, national security and other aspects. However, the application and development of emerging large-scale pre-trained models has shown a huge increase in the demand for energy and devices in computing and storage, so hardware acceleration is urgently needed to solve the computing power needs. With the advent of the post-Moore era, the miniaturization of device sizes is approaching physical limits, and it is difficult for computing chips to obtain more computing power and energy efficiency improvements from advanced CMOS processes. Brain-like artificial neural networks are expected to break through the bottlenecks of the von Neumann architecture storage wall and power consumption wall to increase computing power and reduce energy consumption. At the hardware level, brain-like artificial neural networks need to develop two key parts: artificial synapses and artificial neurons. Previously, researchers designed a method based on VO based on the leaky integration and release (LIF) model. 2 Artificial neurons of threshold switching devices of materials, but it is prone to the problem of gradient vanishing, resulting in poor network performance. To address this challenge, the best solution is to introduce the rectified linear unit (ReLU) activation function, which is one of the most popular activation functions in neural networks. Compared with the typical S-type saturation activation function (Tanh, Sigmoid and other functions), the ReLU activation function can effectively solve the gradient vanishing problem and improve efficiency. Although the ReLU activation function completely discards the negative information part, which may cause the "dead ReLU problem" in deep neural networks (that is, some neurons will not be able to update information), the improved LeakyReLU of the ReLU activation function solves this problem well. It can basically retain the advantages of ReLU while effectively using negative information to alleviate the possible "dead ReLU" problem. The function images of the ReLU function and its improved LeakyReLU are shown in the figure. Figure 1 and Figure 2As shown in the figure. As the most commonly used activation function in neural networks, ReLU has the following uses in deep learning: (1) Increasing the nonlinear ability of the network: The ReLU function maps negative inputs to 0, while keeping positive inputs unchanged. This makes the ReLU function a nonlinear function, which enables the neural network to better fit nonlinear relationships. (2) Avoiding the gradient vanishing problem: During the backpropagation process, if traditional activation functions such as Sigmoid or Tanh are used, the gradient may become very small, which will lead to the gradient vanishing problem. The derivative of the ReLU function is always equal to 1 when the input is positive, which can effectively avoid the gradient vanishing problem. (3) Fast calculation speed: The calculation of the ReLU function is very simple, and it only needs to determine whether the input is a negative number. Therefore, compared with other activation functions, the ReLU function is faster in calculation speed. In general, the ReLU function plays a very important role in deep learning. It can increase the nonlinear ability of the network, avoid the gradient vanishing problem, and has a fast calculation speed. In past studies, the realization of activation functions in deep neural networks often requires the use of general-purpose processors or large and complex neuron peripheral circuits, while artificial neurons implemented by CMOS circuits are usually composed of capacitors and more than 10 transistors, which limits their scalability and seriously reduces the energy and area efficiency of the system. In contemporary neural network computing, data transmission and computation are huge. For example, for the inference of the MNIST dataset, each ReLU device should generate 10,000 outputs in each epoch (or the entire test set). This puts high requirements on the energy consumption and occupied area of ​​the hardware-implemented ReLU artificial neurons. Although emerging nanoelectronic devices such as phase change memory, magnetic tunnel junction, leaky ferroelectric field effect transistor and single latch transistor have been studied and successfully simulated the leaky integration and discharge (LIF) model of artificial neurons, the nonlinear activation function function has not been introduced. Therefore, there is still a need to develop more advanced nanoelectronic devices and highly integrated and low-power ReLU artificial neurons to meet the needs of realizing highly integrated and low-power brain-like artificial neural networks on hardware. Summary of the invention

[0003] The purpose of the present invention is to realize the artificial neuron from the hardware, mainly designing an artificial neuron circuit based on a threshold switch device with a low threshold voltage, and realizing the functions of a highly integrated and low-power ReLU artificial neuron and its improved LeakyReLU artificial neuron.

[0004] The specific steps are:

[0005] (1) The artificial neuron is based on Pt / Ag / Al embedded nanoislands 2 O 3 / HfO 2 / Ag-NIs(5nm) / Pt threshold switching device with extremely low threshold voltage, low scatter and high tolerance.

[0006] (2) Pt / Ag / Al 2 O 3 / HfO 2 Preparation process of the threshold switch device of / Ag-NIs(5nm) / Pt structure: A Ti adhesion layer with a thickness of 30 nm and a Pt bottom electrode with a thickness of 100 nm were deposited on an oxidized silicon wafer substrate by magnetron sputtering, with sputtering powers of 150 W and 60 W, respectively, and sputtering times of 380 seconds and 1100 seconds. Then, an ultrathin anodized aluminum oxide (AAO) template (thickness 200 nm, pore diameter 60 nm, pore center spacing 100 nm) with a polymethyl methacrylate (PMMA) support layer was transferred to the SiO 2 / Pt substrate, and then acetone was used to remove PMMA. Electron beam evaporation was used to prepare Ag nano-islands with a thickness of about 5 nm. Then, atomic layer deposition was used to prepare 8 nm HfO2 and 5 nm Al 2 O 3 layer, wherein the precursor and oxygen source of the hafnium oxide layer are tetrakis(dimethylamino)hafnium(Hf(NC 2 H 6 ) 4 ) and H 2 O, the growth temperature was 250 °C, and the precursor and oxygen source of the aluminum oxide layer were trimethylaluminum (TMA) and H 2 O, and the deposition temperature was 250°C. A 5 nm Ag layer was deposited using magnetron sputtering at 50 W power, 3 mT sputtering pressure, and 25 seconds of sputtering time. Finally, a 100 nm thick Pt top electrode was deposited using magnetron sputtering at 60 W power, 3 mT sputtering magnetic field, and 1100 seconds of sputtering time. Embedding silver nanoislands (Ag NIs) to avoid the electroforming process greatly reduces the threshold voltage of the device and improves the stability of the threshold voltage of the threshold switching device by promoting the formation and limiting the location of the conductive channel; inserting Al 2 O 3 Layer to limit excessive diffusion of metal into the oxide layer, thereby improving the on / off ratio and durability of threshold switching devices. Pt / Ag / Al 2 O 3 / HfO 2 The / Ag NIs(5nm) / Pt device does not require electroforming and has bidirectional threshold switching performance. The positive threshold voltage of the device is 0.036 V, the negative threshold voltage is -0.037 V, the threshold voltage fluctuation is less than 8%, and the high-resistance resistance of the device is about 1.6×10 5 Ω, low resistance is about 1.5×10 4Ω, the switching ratio is about 10, and the endurance is more than 100,000 times.

[0007] (3) ReLU artificial neuron circuit ( Figure 1 ) is mainly composed of a threshold switch device and a parallel capacitor C m , and then connect a diode D in series 1 , and adjust the resistor R s A capacitor C is added to the output of the circuit. f and resistor R f The RC filter unit and a resistor R for output detection o Furthermore, by adding a resistor R d With diode D 1 In parallel, an improved LeakyReLU artificial neuron circuit is formed ( Figure 2 ).

[0008] (4) When the input of the ReLU artificial neuron circuit is a negative voltage signal, the output of the circuit is in a non-responsive state, showing the negative function characteristics of the ReLU activation function: When the input of the ReLU artificial neuron circuit is a positive voltage signal, the output of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal, showing the positive function characteristics of the ReLU activation function. When the input of the LeakyReLU artificial neuron circuit is a negative voltage signal, the output of the circuit is no longer in a non-responsive state, but when the amplitude of the input negative voltage signal reaches a certain value (so that the capacitor C m The negative voltage at both ends is greater than the negative threshold voltage of the threshold switch device), and a small-amplitude negative spike voltage signal is output, and its amplitude increases with the increase of the amplitude of the input signal, showing the negative function characteristics of the LeakyReLU activation function. When the input of the LeakyReLU artificial neuron circuit is a positive voltage signal, the output end of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal, showing the positive function characteristics of the LeakyReLU activation function.

[0009] The technical solution of the present invention has the following beneficial effects:

[0010] The present invention proposes a Pt / Ag / Al structure with embedded nano-islands 2 O 3 / HfO 2 / Ag-NIs(5nm) / Pt threshold switching device with extremely low threshold voltage, based on which a highly integrated and low-power ReLU artificial neuron is designed, the relationship between the amplitude of its output spike and the input voltage amplitude realizes the function of the ReLU activation function. Compared with the previous ReLU artificial neuron based on the Mott device, it no longer relies on the relationship between the output spike frequency and the input voltage to realize ReLU, and can be driven with a lower input voltage, which effectively improves the operation speed of the ReLU artificial neuron and reduces power consumption. In addition, we use the bipolar threshold switching characteristics of the threshold switching device, and further propose its improved LeakyReLU artificial neuron circuit based on the ReLU artificial neuron circuit. The artificial neuron proposed in the present invention has great application prospects in the future construction of large-scale brain-like artificial neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic diagram of the ReLU artificial neuron circuit.

[0012] Figure 2 Schematic diagram of the LeakyReLU artificial neuron circuit.

[0013] Figure 3 Typical input-output characteristic curve of ReLU artificial neuron.

[0014] Figure 4 Typical input-output characteristic curve of the LeakyReLU artificial neuron. DETAILED DESCRIPTION

[0015] In order to make the present invention more clear, the present invention is further described in detail below in conjunction with the embodiments. The specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0016] Embodiment 1:

[0017] (1) Pt / Ag / Al for building ReLU artificial neurons 2 O 3 / HfO 2 Preparation process of / Ag-NIs(5nm) / Pt threshold switch device: A Ti adhesion layer with a thickness of 30 nm and a Pt bottom electrode with a thickness of 100 nm were deposited on an oxidized silicon wafer substrate by magnetron sputtering, with sputtering powers of 150 W and 60 W, respectively, and sputtering times of 380 seconds and 1100 seconds. Then, an ultrathin anodized aluminum oxide (AAO) template (thickness 200 nm, pore diameter 60 nm, pore center spacing 100 nm) with a polymethyl methacrylate (PMMA) support layer was transferred to the SiO 2 / Pt substrate, and then PMMA was removed with acetone. Electron beam evaporation was used to prepare Ag nano-islands with a thickness of about 5 nm. Then, atomic layer deposition was used to prepare 8 nm HfO2 and 5 nm Al 2 O 3 layer, wherein the precursor and oxygen source of the hafnium oxide layer are tetrakis(dimethylamino)hafnium(Hf(NC 2 H 6 ) 4 ) and H 2 O, the growth temperature was 250 °C, and the precursor and oxygen source of the aluminum oxide layer were trimethylaluminum (TMA) and H 2 O, and the deposition temperature was 250 °C. A 5 nm Ag layer was deposited using magnetron sputtering at 50 W power, 3 mT sputtering pressure, and 25 s sputtering time. Finally, a 100 nm thick Pt top electrode was deposited using magnetron sputtering at 60 W power, 3 mT sputtering magnetic field, and 1100 s sputtering time.

[0018] (2) Connect the threshold switch device and a capacitor C m In parallel, then the threshold switching device and the parallel capacitor C m A diode D is connected in series in front of 1 and resistor R s A capacitor C is added to the output of the circuit. f and resistor R f The RC filter unit and an output resistor R o When diode D1 is in the conducting state, the internal resistance is about 10 Ω, and the series resistor R s The resistance is 100 KΩ, the filter capacitor C f And filter resistor R f The output resistor R is 47 nF and 10 KΩ. o is 2 KΩ.

[0019] (3) A DC or AC signal can be input to the ReLU artificial neuron circuit. For example, when a pulse voltage (-0.1V to 2V, 100Hz) is input to the ReLU artificial neuron, when the input is a negative voltage signal, the output of the circuit is in a non-responsive state, showing the negative function characteristics of the ReLU activation function: when the input of the ReLU artificial neuron circuit is a positive voltage signal, the output of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal ( Figure 3 ), showing the positive function characteristics of the ReLU activation function.

[0020] Embodiment 2:

[0021] (1) Based on the threshold switch device prepared in Example 1, a threshold switch device and a capacitor Cm In parallel, then the threshold switching device and the parallel capacitor C m A diode D is connected in series in front of 1 and resistor R s A capacitor C is added to the output end of the circuit. f and resistor R f Furthermore, by adding a resistor R d With diode D 1 In parallel, a LeakyReLU artificial neuron circuit is formed. When the diode D1 is in the conducting state, the internal resistance is about 10 Ω, and the series resistor R s The resistance is 100 KΩ, the filter capacitor C f And filter resistor R f The output resistor R is 47 nF and 10 KΩ. o is 2 KΩ, the resistor R d is 1 MΩ.

[0022] (2) A DC or AC signal can be input to the LeakyReLU artificial neuron circuit. For example, when a pulse voltage (-0.1V to 2V, 100Hz) is input to the ReLU artificial neuron, when the input of the LeakyReLU artificial neuron circuit is a negative voltage signal, the output of the circuit is no longer in an unresponsive state, but when the amplitude of the input negative voltage signal reaches a certain value (so that the capacitor C m The negative voltage at both ends is greater than the negative threshold voltage of the threshold switch device), outputting a small-amplitude negative spike voltage signal, and its amplitude increases with the increase of the amplitude of the input signal ( Figure 4 ), showing the negative function characteristics of the LeakyReLU activation function. When the input of the LeakyReLU artificial neuron circuit is a positive voltage signal, the output of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal, showing the positive function characteristics of the LeakyReLU activation function.

Claims

1. A hardware implementation method of ReLU artificial neurons for deep brain-like artificial neural networks, characterized in that: The specific structure consists of: (1) The ReLU artificial neuron is based on a Pt / Ag / Al2O3 / HfO2 / Ag-NIs (5nm) / Pt device with embedded nanoislands, which has extremely low threshold voltage, low dispersion and high tolerance; (2) The device according to claim (1), characterized in that: the preparation process of the threshold switching memristor based on the Pt / Ag / Al2O3 / HfO2 / Ag-NIs(5nm) / Pt structure is as follows: a Ti adhesion layer with a thickness of 30 nm and a Pt bottom electrode with a thickness of 100 nm are deposited on an oxidized silicon wafer substrate by magnetron sputtering, with sputtering powers of 150 W and 60 W, respectively, and sputtering times of 380 seconds and 1100 seconds; then, an ultrathin anodized aluminum template (thickness 200 nm, pore diameter 60 nm, pore center spacing 100 nm) with a polymethyl methacrylate (PMMA) support layer is transferred to the SiO2 / Pt substrate, and the PMMA is removed by acetone; Ag nano-islands with a thickness of about 5 nm are prepared by electron beam evaporation; then, 8 nm HfO2 and 5 nm Ag nano-islands are prepared by atomic layer deposition. A 5 nm thick Al2O3 layer was deposited using magnetron sputtering at a power of 50 W, a sputtering pressure of 3 mT, and a sputtering time of 25 s. Finally, a 100 nm thick Pt top electrode was deposited using magnetron sputtering at a power of 60 W, a sputtering magnetic field of 3 mT, and a sputtering time of 1100 s. (3) The device according to claim (1), characterized in that: silver nano-islands (Ag NIs) are embedded to avoid the electroforming process, greatly reduce the threshold voltage, and improve the stability of the threshold voltage of the threshold switch device by promoting the formation and limiting the location of the conductive channel; an Al2O3 layer is inserted to limit the excessive diffusion of metal into the oxide layer, thereby improving the switching ratio and durability of the threshold switch device; the Pt / Ag / Al2O3 / HfO2 / Ag NIs (5nm) / Pt device does not require electroforming and has bidirectional threshold switching performance. The device has a positive threshold voltage of 0.036 V, a negative threshold voltage of -0.037 V, a threshold voltage fluctuation of less than 8%, and a high-resistance resistance of about 1.6×10 5 Ω, low resistance is about 1.5×10 4 Ω, switch ratio is about 10, and the tolerance is more than 100,000 times; (4) The ReLU artificial neuron circuit is mainly composed of a threshold switch device and a parallel capacitor C m , a diode D1 is connected in series in front, and an adjusting resistor R s ; A capacitor C is added to the output end of the circuit f and resistor R f The RC filter unit and a resistor R for output detection o ; Further, by adding a resistor R d Connected in parallel with diode D1, it forms an improved LeakyReLU artificial neuron circuit; (5) The device according to claim (4), characterized in that the capacitor C m is 100 nF. When diode D1 is in the on state, the internal resistance is about 10 ~ 30 Ω. The series resistor R s The resistance is 100 KΩ, the filter capacitor C f And filter resistor R f is 47 nF and 10 KΩ, or 68 nF and 5 KΩ, the output resistance R o 2 KΩ resistor R d is 1 MΩ.

2. The method according to claim 1, characterized in that When the input of the ReLU artificial neuron circuit is a negative voltage signal, the output of the circuit is in a non-responsive state, showing the negative function characteristics of the ReLU activation function: when the input of the ReLU artificial neuron circuit is a positive voltage signal, the output of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal, showing the positive function characteristics of the ReLU activation function; when the input of the LeakyReLU artificial neuron circuit is a negative voltage signal, the output of the circuit is no longer in a non-responsive state, but when the amplitude of the input negative voltage signal reaches a certain value, the capacitor C m The negative voltage at both ends is greater than the negative threshold voltage of the threshold switch device, and a small-amplitude negative spike voltage signal is output, and its amplitude increases with the increase of the amplitude of the input signal, showing the negative function characteristics of the LeakyReLU activation function; when the input of the LeakyReLU artificial neuron circuit is a positive voltage signal, the output end of the circuit continuously outputs positive spike signals, and the amplitude of the spike signal increases with the increase of the amplitude of the input signal, showing the positive function characteristics of the LeakyReLU activation function.

Images