A neural network activation function simulation circuit and its design method

Abstract

The present invention discloses a neural network activation function simulation circuit and a design method thereof. The neural network activation function simulation circuit is composed of a cascade of three amplifier circuits for implementing a Sigmoid function. The first amplifier circuit is composed of a biased in-phase operational amplifier, and the second and third amplifier circuits are composed of inverting operational amplifiers. The Sigmoid function is simplified to y=0.2x+0.5, and the analog circuit input voltage signal is divided into three segments. The first amplifier circuit maps the three segments of the signal in parallel to 0-5V. The second and third amplifier circuits sequentially perform linear transformations, ultimately mapping 0-5V to 0-1V. The circuit structure of the present invention is simple, power consumption is low, and the Sigmoid function can be approximately realized. This reduces the resource consumption of the hardware system used to implement the neural network and improves the operation speed of the neural network activation function hardware circuit. The circuit has broad application prospects in the field of artificial neural network hardware implementation circuits.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic information and artificial neural network, and in particular to a neural network activation function simulation circuit and a design method thereof. Background Art

[0002] Artificial neural networks, or NNs for short, are machine learning methods that evolved from the idea of ​​simulating the human brain. They are human-created neural networks. In recent years, with their rapid development, NNs have penetrated into every aspect of human life, such as image recognition and processing, and speech recognition. Furthermore, NNs, with their unique nonlinear adaptive information processing capabilities, strong robustness and fault tolerance, and superior information integration capabilities, have become a research hotspot in the 21st century and will propel humanity into the era of deep artificial intelligence.

[0003] Activation functions are the fundamental reason neural networks can process data. Without activation functions, regardless of the number of layers in a neural network, the input values ​​of each node in each layer are linear functions of the nodes in the previous layer. This eliminates the ability to process input data and is equivalent to having no hidden layers. In this case, the neural network is similar to a perceptron, with a weak ability to approximate functions and unable to perform complex tasks. However, if activation functions are added to a neural network, they introduce nonlinearity, allowing it to approximate any nonlinear function. This allows neural networks to be applied to virtually any nonlinear model and handle nearly all complex tasks, greatly enhancing their capabilities.

[0004] Currently, the mainstream methods for implementing neural network activation functions are through software algorithms or digital circuits. The few methods that use analog hardware circuits to implement activation functions also use ADCs to transmit data to MCUs for processing. There are no truly dedicated analog circuits for implementing neural network activation functions. Therefore, the invention of a dedicated analog circuit for implementing neural network activation functions has become an urgent need. Summary of the Invention

[0005] The purpose of the present invention is to propose a neural network activation function simulation circuit and a design method thereof, which can easily approximate the neural network activation function - the Sigmoid function, reduce the hardware system resource consumption used to implement the neural network, improve the hardware circuit operation speed, and solve a very critical link in the process of building a neural network using an analog circuit. It can be used for the expansion and popularization of artificial neural networks.

[0006] The purpose of the present invention is achieved through the following technical means:

[0007] According to the first aspect of the present specification, a neural network activation function simulation circuit is provided, wherein the neural network activation function is a Sigmoid function, and the simulation circuit is composed of a cascade of three-stage amplification circuits, wherein the first-stage amplification circuit is composed of a biased in-phase operational amplifier, and the second-stage and third-stage amplification circuits are composed of inverting operational amplifiers; the Sigmoid function is simplified to y=0.2x+0.5, where x and y are the independent variable and dependent variable, respectively, and the analog circuit input voltage signal is divided into the following three segments: (-∞,-2.5V), [-2.5V,2.5V], and (2.5V,+∞). The first-stage amplification circuit maps the three-segment signal in parallel to 0-5V; the second-stage and third-stage amplification circuits perform linear transformation in sequence, and finally map 0-5V to 0-1V, thereby realizing the Sigmoid function function.

[0008] Furthermore, the first-stage amplifier circuit is composed of an integrated operational amplifier with a 5V single power supply and rail-to-rail output, a bias voltage of 2.5V, and an amplification factor of 1.

[0009] Furthermore, the second-stage amplifying circuit is composed of an integrated operational amplifier powered by ±5V dual power supplies, and the amplification factor is 0.2 times.

[0010] Furthermore, the third-stage amplification circuit is composed of an integrated operational amplifier powered by ±5V dual power supplies, and the amplification factor is 1.

[0011] Furthermore, the second-stage and third-stage amplification circuits are implemented by two single-channel integrated operational amplifiers, or by one multi-channel integrated operational amplifier.

[0012] According to a second aspect of this specification, a method for designing a neural network activation function simulation circuit is provided, wherein the neural network activation function is a Sigmoid function, and the method comprises the following steps:

[0013] S1, simplify the Sigmoid function to y = 0.2x + 0.5, where x and y are the independent variable and dependent variable respectively, and divide the analog circuit input voltage signal into the following three segments: (-∞, -2.5V), [-2.5V, 2.5V], (2.5V, +∞);

[0014] S2. Design a first-stage amplifier circuit. The first-stage amplifier circuit is composed of a biased, non-inverting operational amplifier for mapping an input voltage signal ranging from negative infinity to positive infinity to an output voltage signal ranging from 0 to 5V.

[0015] S3, designing a second-stage amplifier circuit, wherein the second-stage amplifier circuit is composed of an inverting operational amplifier, and is used to linearly convert the output voltage signal of the first stage from a range of 0 to 5V to an output voltage signal from a range of -1V to 0V;

[0016] S4. Design a third-stage amplifier circuit. The third-stage amplifier circuit is composed of an inverting operational amplifier, which is used to reverse the voltage polarity of the output voltage signal of the second stage so that the output voltage signal range of the third stage is between 0 and 1V, realizing the Sigmoid function.

[0017] Furthermore, in step S2, if the input voltage signal is less than -2.5V, the output voltage signal is 0V; if the input voltage signal is greater than or equal to -2.5V and less than or equal to 2.5V, the input voltage signal is linearly converted to an output voltage signal of 0-5V; if the input voltage signal is greater than 2.5V, the output voltage signal is 5V.

[0018] The beneficial effects of the present invention are as follows: compared with the traditional neural network activation function simulation circuit, the present invention simplifies the formula of the neural network activation function Sigmoid function, converts it into a piecewise linear form that is easy to implement, and utilizes advanced integrated operational amplifiers and is implemented using a full analog circuit, which has high integration, low power consumption, and high speed, thereby further realizing the use of a full analog circuit to implement a neural network, and has important application value and significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the function image of the neural network activation function of the embodiment of the present invention - Sigmoid function;

[0020] Figure 2 1 is a circuit schematic diagram of a neural network activation function simulation circuit according to an embodiment of the present invention;

[0021] Figure 3 It is an output waveform diagram of the neural network activation function simulation circuit of the example of the present invention. DETAILED DESCRIPTION

[0022] The following examples are used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0023] The present invention provides a neural network activation function simulation circuit, wherein the neural network activation function is a Sigmoid function. The simulation circuit is composed of three cascaded amplifier circuits, wherein the first amplifier circuit is composed of a biased in-phase operational amplifier, and the second and third amplifier circuits are composed of inverting operational amplifiers. The present invention simplifies the Sigmoid function to y=0.2x+0.5, where x and y are the independent variable and the dependent variable, respectively. The analog circuit input voltage signal is divided into the following three segments: (-∞, -2.5V), [-2.5V, 2.5V], and (2.5V, +∞). The first amplifier circuit maps the three signal segments in parallel to 0-5V. The second and third amplifier circuits sequentially perform linear transformations, ultimately mapping the 0-5V signal to 0-1V, thereby realizing the Sigmoid function.

[0024] In one embodiment, the first-stage amplifier circuit is composed of an integrated operational amplifier with a 5V single power supply and rail-to-rail output, a bias voltage of 2.5V, and an amplification factor of 1; it can be implemented using TI's OPA4350 series integrated operational amplifier, which has the characteristics of single power supply, rail-to-rail output, high speed, and low noise.

[0025] The second-stage amplifier circuit is composed of an integrated operational amplifier powered by a ±5V dual power supply, with an amplification factor of 0.2 times; it can be implemented using TI's OPA2227 series integrated operational amplifier, which has the characteristics of dual power supply, wide power supply voltage range, high precision and low noise.

[0026] The third-stage amplifier circuit is composed of an integrated operational amplifier powered by a ±5V dual power supply, with an amplification factor of 1. It can be implemented using TI's OPA2227 series integrated operational amplifier, which has the characteristics of dual power supply, wide power supply voltage range, high precision and low noise.

[0027] In one embodiment, a method for designing a neural network activation function circuit is provided, comprising the following steps:

[0028] Step 1: The formula of the Sigmoid function to be implemented by the neural network activation function simulation circuit of the present invention is: Simplified to y = 0.2x + 0.5, where x and y are the independent variable and dependent variable respectively, the analog circuit input voltage signal is divided into the following three segments: (-∞, -2.5V), [-2.5V, 2.5V], (2.5V, +∞), and a three-stage amplifier circuit is used to implement an approximate Sigmoid function. The waveform of the Sigmoid function is as follows Figure 1 shown.

[0029] Step 2: Use TI's integrated operational amplifier OPA4350 to build a non-inverting amplifier circuit. Use a positive power supply of 5V to power the operational amplifier. The non-inverting amplifier circuit has a DC bias at the input, and the DC bias voltage value is 2.5V. The feedback loop resistors of the non-inverting amplifier circuit are selected as R1 = 10KΩ and R2 = 100Ω. Using the gain formula of the non-inverting amplifier circuit The gain of the amplifier is approximately 1. This amplifier circuit is used to map the input voltage signal from the range of negative infinity to positive infinity to the output voltage signal from 0 to 5V, that is, if the input voltage signal is less than -2.5V, the output voltage signal is 0V; if the input voltage signal is greater than or equal to -2.5V and less than or equal to 2.5V, the input voltage signal is linearly converted to an output voltage signal of 0-5V; if the input voltage signal is greater than 2.5V, the output voltage signal is 5V.

[0030] Step 3: Use TI's integrated operational amplifier OPA2227 to build an inverting configuration amplifier circuit. Use positive and negative power supplies of 5V and -5V to power the operational amplifier. The feedback loop resistors of the inverting configuration amplifier circuit are selected as R3 = 5KΩ and R4 = 1KΩ. Use the inverting configuration amplifier circuit gain formula The amplifier circuit of this stage is used to linearly convert the output voltage signal of the first stage from the range of 0 to 5V to the output voltage signal from the range of -1V to 0V.

[0031] Step 4: Use TI's integrated operational amplifier OPA2227 to build an inverting configuration amplifier circuit. Use positive and negative power supplies of 5V and -5V to power the operational amplifier. The feedback loop resistors of the inverting configuration amplifier circuit are selected as R5 = 1KΩ and R6 = 1KΩ. Use the inverting configuration amplifier circuit gain formula The gain of the amplifier is 1. This amplifier circuit is used to reverse the polarity of the output voltage signal of the second stage, so that the output voltage signal of the third stage is between 0 and 1V.

[0032] Step 5: cascade the three-stage amplifier circuit to form a complete neural network activation function simulation circuit so that it can work normally. The schematic diagram of the neural network activation function simulation circuit is as follows: Figure 2 The simulation waveform is shown in Figure 3 As shown in the figure, the similarity with the theoretical waveform is high, and the function of approximating the Sigmoid function is well realized.

[0033] It can be seen from the above embodiments that the examples of the present invention greatly reduce the complexity of the neural network activation function by simplifying the form of the Sigmoid function and converting it into a piecewise linear form, so that the originally very complex neural network activation function can be simply implemented using a full analog circuit.

[0034] The examples described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be noted that a person skilled in the art can make several improvements and substitutions without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A neural network activation function simulation circuit, characterized in that: The neural network activation function is a Sigmoid function, and the analog circuit is composed of a cascade of three-stage amplifier circuits, wherein the first-stage amplifier circuit is composed of a biased in-phase operational amplifier, and the second-stage and third-stage amplifier circuits are composed of inverting operational amplifiers; the Sigmoid function is simplified to y=0.2x+0.5, where x and y are the independent variable and the dependent variable, respectively; the analog circuit input voltage signal is divided into the following three segments: (-∞,-2.5V), [-2.5V,2.5V], and (2.5V,+∞); the first-stage amplifier circuit maps the three-segment signal to 0-5V in parallel; the second-stage and third-stage amplifier circuits perform linear transformation in sequence, and finally map 0-5V to 0-1V, thereby realizing the Sigmoid function function.

2. The neural network activation function simulation circuit according to claim 1, wherein The first-stage amplifier circuit is composed of an integrated operational amplifier with a 5V single power supply and rail-to-rail output, a bias voltage of 2.5V, and an amplification factor of 1.

3. The neural network activation function simulation circuit according to claim 1, wherein The second-stage amplifier circuit is composed of an integrated operational amplifier powered by a ±5V dual power supply, and has an amplification factor of 0.

2.

4. The neural network activation function simulation circuit according to claim 1, wherein The third-stage amplifier circuit is composed of an integrated operational amplifier powered by a ±5V dual power supply, and the amplification factor is 1.

5. The neural network activation function simulation circuit according to claim 1, wherein The second-stage and third-stage amplification circuits are implemented by two single-channel integrated operational amplifiers, or by one multi-channel integrated operational amplifier.

6. A method for designing a neural network activation function simulation circuit, characterized in that: The neural network activation function is a Sigmoid function, and the method comprises the following steps: S1, simplify the Sigmoid function to y = 0.2x + 0.5, where x and y are the independent variable and dependent variable respectively, and divide the analog circuit input voltage signal into the following three segments: (-∞, -2.5V), [-2.5V, 2.5V], (2.5V, +∞); S2. Design a first-stage amplifier circuit. The first-stage amplifier circuit is composed of a biased, non-inverting operational amplifier for mapping an input voltage signal ranging from negative infinity to positive infinity to an output voltage signal ranging from 0 to 5V. S3, designing a second-stage amplifier circuit, wherein the second-stage amplifier circuit is composed of an inverting operational amplifier, and is used to linearly convert the output voltage signal of the first stage from a range of 0 to 5V to an output voltage signal from a range of -1V to 0V; S4. Design a third-stage amplifier circuit. The third-stage amplifier circuit is composed of an inverting operational amplifier, which is used to reverse the voltage polarity of the output voltage signal of the second stage so that the output voltage signal range of the third stage is between 0 and 1V, realizing the Sigmoid function.

7. The method according to claim 6, characterized in that In step S2, if the input voltage signal is less than -2.5V, the output voltage signal is 0V; if the input voltage signal is greater than or equal to -2.5V and less than or equal to 2.5V, the input voltage signal is linearly converted to an output voltage signal of 0-5V; if the input voltage signal is greater than 2.5V, the output voltage signal is 5V.

Images