Neural network hardware acceleration method and system based on adaptive activation function circuit

Abstract

The invention discloses a neural network hardware acceleration method and system based on an adaptive activation function circuit, and belongs to the field of neural network hardware acceleration. The method comprises the following steps: constructing an adaptive activation function circuit by using a curve fitting method, and storing adaptive parameters through a memristor; constructing a memristor cross array, and based on the Ohm's law and the Kirchhoff's current law, converting input voltage into current output through the memristor cross array; inputting the current output into a self-adaptive activation function circuit, and performing nonlinear processing through self-adaptive parameters to obtain network layer output after self-adaptive activation function processing; the output of the network layer calculates a loss function through an off-chip calculation mode, and calculates the gradient of an adaptive parameter; the adaptive parameters in the memristor are updated according to the gradient of the adaptive parameters, and the form of the adaptive activation function is dynamically optimized. The adaptive activation function circuit accelerates model convergence more quickly by updating parameters in real time.

Description

Technical Field

[0001] This invention belongs to the field of neural network hardware acceleration, specifically relating to a method and system for accelerating neural networks based on adaptive activation function circuits. Background Technology

[0002] Activation functions are an indispensable part of neural network training. Their main function is to introduce non-linear capabilities into the network, enabling it to learn and represent complex patterns. If only linear mapping between layers is performed... Regardless of the number of layers stacked in the network, it remains equivalent to a single matrix multiplication, and the model space is strictly limited to the set of affine transformations. This makes it unable to express nonlinear decision boundaries and thus unsuitable for complex tasks such as image generation, language modeling, and policy control. Activation functions, through element-wise nonlinear processing, can map the output of each layer to a high-dimensional nonlinear manifold. If this satisfies the universal approximation theorem mathematically, then with a sufficient number of neurons and nonlinear activations, the network can approximate any continuous function with arbitrary precision.

[0003] Activation functions, as a key component introducing nonlinearity in neural networks, directly affect the model's learning ability and generalization performance. Traditional activation functions such as ReLU, Sigmoid, and Tanh, while simple in structure and computationally efficient, often suffer from performance bottlenecks when handling complex tasks or specific data distributions. They struggle to meet the needs of different layers or tasks and fail to demonstrate universal advantages across all datasets and network architectures. Compared to traditional activation functions, adaptive activation functions offer greater flexibility and expressiveness, helping to improve the model's nonlinear modeling ability, accelerate convergence, and mitigate gradient propagation issues in deep networks.

[0004] However, most adaptive activation functions still rely on software-level computational implementation. With the continuous expansion of the scale of deep neural network models and the diversification of application scenarios, especially in edge computing fields such as the Internet of Things, smart terminals, and autonomous driving, traditional software computing models based on general-purpose processors face huge challenges in terms of speed, energy efficiency, and real-time performance. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a hardware acceleration method and system for neural networks based on adaptive activation function circuits, thereby solving the problems in the prior art.

[0006] The objective of this invention can be achieved through the following technical solutions: The hardware acceleration method for neural networks based on adaptive activation function circuits includes the following steps: An adaptive activation function circuit is constructed using curve fitting, and adaptive parameters are stored using a memristor. A memristor cross array is constructed, and based on Ohm's law and Kirchhoff's current law, the input voltage is converted into current output through the memristor cross array; the current output is then input into an adaptive activation function circuit, and nonlinear processing is performed through adaptive parameters to obtain the network layer output after adaptive activation function processing; The neural network is trained using an adaptive activation function, and the loss function and gradient of the adaptive parameters are calculated by off-chip computation of the network layer outputs. The adaptive parameters in the memristor are updated based on the gradient of the adaptive parameters, and the shape of the adaptive activation function is dynamically optimized.

[0007] Furthermore, the types of adaptive activation function circuits include: Sigmoid function circuit, Tanh function circuit, and ReLU function circuit; the Sigmoid function circuit is implemented by curve fitting through the nonlinear output characteristics of the CMOS inverter; the Tanh function circuit is implemented by applying an inverting amplifier to the Sigmoid circuit.

[0008] Furthermore, the activation function expression in the adaptive activation function is: in, These are adaptive parameters, stored via memristors. , , These are the adaptive activation functions for the Sigmoid function circuit, the Tanh function circuit, and the ReLU function circuit, respectively. x This is the input voltage for the activation function circuit.

[0009] Furthermore, the process by which the input voltage is converted into a current output z through a memristor cross array is as follows: Where w is the weight matrix stored in the memristor cross array, x is the input of the memristor cross array, and c is the array bias term.

[0010] An adaptive activation function circuit, including any one of a Sigmoid function circuit, a Tanh function circuit, and a ReLU function circuit; The Sigmoid function circuit includes: two memristors for storing adaptive parameters, three inverting amplifiers, and a CMOS inverter. The output curve of the CMOS inverter is fitted with a Tanh curve, and the output is then passed through an inverting amplifier to obtain the Sigmoid function curve. When the input signal...x When entering the function circuit, the parameters are first achieved through an inverting amplifier. With input x The product of / 2 is then output as Tanh ( ) via a CMOS inverter. ax / 2), and the transformation of the Tanh function to the Sigmoid function is achieved through an inverting proportional amplifier, and finally the parameter is achieved through an inverting proportional amplifier. b With Sigmoid ( ax The product of ) The Tanh function circuit includes: two memristors for storing adaptive parameters, two inverting amplifiers, and a CMOS inverter; the Tanh curve is fitted through the output curve of the CMOS inverter, and when the input signal... x When entering the function circuit, the parameters are first achieved through an inverting amplifier. a With input x The product of these two terms is then output as Tanh( ) via a CMOS inverter. ax Finally, the parameters are achieved through an inverting amplifier. b With Tanh ( ax The product of ) The ReLU function circuit includes: two memristors for storing adaptive parameters, two diode switches for controlling positive and negative conduction, two inverting operational amplifiers for performing multiplication, and an inverting operational amplifier for converting current to voltage; when the input signal... x Upon entering the circuit, the voltage on the right side of the diode is clamped to 0. When the input is positive, the lower branch diode conducts, and the input signal... x Parameters are achieved using an inverting operational amplifier. b With input x The product of the two signals yields a positive output current; conversely, when the input is negative, the upper branch diode conducts, and the input signal... x Parameters are achieved using an inverting operational amplifier. a With input x The product of the two currents yields the negative output current. The positive and negative currents flow into the same node and are converted into a voltage output by an inverting operational amplifier. function.

[0011] The above-mentioned adaptive activation function circuit is applied in CNN image recognition.

[0012] The neural network hardware acceleration system based on adaptive activation function circuits executes the aforementioned neural network hardware acceleration method based on adaptive activation function circuits, including: Activation function operation module: Constructs an adaptive activation function circuit using curve fitting method and stores the adaptive parameters through memristors; Matrix operation module: Constructs a memristor cross array, and based on Ohm's law and Kirchhoff's current law, converts the input voltage into a current output through the memristor cross array; and inputs the current output into an adaptive activation function circuit, performs nonlinear processing through adaptive parameters, and obtains the network layer output after adaptive activation function processing; Off-chip computation module: Trains the neural network using an adaptive activation function, calculates the loss function and the gradient of the adaptive parameters by off-chip computation of the network layer outputs; Dynamic optimization module: Updates the adaptive parameters in the memristor based on the gradient of the adaptive parameters, and dynamically optimizes the shape of the adaptive activation function.

[0013] A computer storage medium storing a readable program that, when executed, instructs a computing device to perform the aforementioned hardware acceleration method for neural networks based on an adaptive activation function circuit.

[0014] An electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the neural network hardware acceleration method based on the adaptive activation function circuit described above.

[0015] A computer program product includes computer instructions that instruct a computing device to perform operations corresponding to the aforementioned neural network hardware acceleration method based on adaptive activation function circuits.

[0016] The beneficial effects of this invention are: 1. This invention achieves dynamic switching of multiple activation functions and continuous adjustment of parameters through adaptive parameter adjustment and configurable operation, which can adapt to the operation requirements of different neural network models and different layers, solving the problem of the single application scenario of traditional circuits; the activation function characteristics are dynamically adjusted based on the input data features, which effectively improves the operation accuracy in complex data scenarios; at the same time, combined with the calculation of the loss function, the parameter update can be controlled by applying voltage pulses, without the need to design additional configurable operation units.

[0017] 2. The adaptive activation function circuit of the present invention can accelerate model convergence faster by updating parameters in real time while ensuring the highly nonlinear characteristics of the output. In image recognition tasks, the accuracy of this circuit structure is improved to a certain extent compared with the fixed traditional function circuit model, which can effectively meet the needs of high-precision task scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of a neural network hardware acceleration framework based on memristor adaptive activation function; Figure 2 This is a schematic diagram of the hardware implementation of the network training process of this invention; Figure 3 This is a schematic diagram of the output characteristic curves of circuits with different activation functions in this invention; Figure 4 This is a schematic diagram illustrating the training loss and test accuracy of the present invention; Figure 5 This is a schematic diagram of the noise immunity of the present invention at different intensities; Figure 6 This is a schematic diagram illustrating the parameter changes at different layers during neural network training according to the present invention; Figure 7 This is a schematic diagram of the adaptive activation function circuit of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 like Figure 1 As shown, the hardware acceleration method for neural networks based on adaptive activation function circuits includes the following steps: S1, an adaptive activation function circuit is constructed using curve fitting, and adaptive parameters are stored through a memristor; like Figure 2 As shown, the adaptive activation function circuit includes: a Sigmoid function circuit, a Tanh function circuit, and a ReLU function circuit; The Sigmoid function circuit is implemented by curve fitting using the highly nonlinear output characteristics of a CMOS inverter. Since the CMOS inverter is highly nonlinear and its output characteristic curve is close to an S-shape, additional parameters can be introduced to adjust the steepness and amplitude of the output curve, resulting in the A-Sigmoid function. In the formula, a, b For adaptive parameters, they are stored in a memristor. The non-volatility of the memristor ensures that the stored parameters are not easily leaked. At the same time, the parameter update process can also be controlled by the continuously variable conductance of the memristor (depending on the effect of the pulse voltage on the memristor). x This is the input voltage for the activation function circuit.

[0022] The Tanh function circuit can be implemented using the Sigmoid function circuit, mathematically speaking: This can be achieved simply by adding an inverting amplifier to the sigmoid circuit. Similar to A-Sigmoid, A-Tanh is added... Adaptive parameters: Due to its piecewise linearity, the ReLU function circuit can control the corresponding input and output through switches. Using a voltage comparator, when the input voltage is higher than 0V, the voltage comparator outputs a high level, the NMOS transistor is turned on, and the output is amplified by an inverting amplifier to obtain the positive range output; conversely, when the input voltage is lower than 0V, the voltage comparator outputs a low level, the PMOS transistor is turned on, and the output is amplified by an inverting amplifier to obtain the negative range output. A -ReLU function: in, These are three types of adaptive activation functions, which can be used for different types of neural networks.

[0023] The output curves of the Sigmoid function circuit, Tanh function circuit, and ReLU function circuit are as follows: Figure 3 As shown, for different a, b The parameter values ​​and the shape of the circuit output curve change: For A-Sigmoid and A-Tanh function circuits, the adjustable steepness and amplitude of the function output curve not only give the model better flexibility and expressive power, but also alleviate the gradient saturation problem of fixed Sigmoid and Tanh functions; for the A-ReLU function, the introduction of negative parameters avoids the "neuron death" problem, while the introduction of positive parameters enables the model to better adapt to the data distribution and learn more complex features.

[0024] The structures of the Sigmoid function circuit, Tanh function circuit, and ReLU function circuit are respectively as follows: Figure 7 As shown in (a), (b), and (c) in the figure; like Figure 7As shown in (a), the Sigmoid function circuit includes: two memristors for storing adaptive parameters, three inverting amplifiers, and a CMOS inverter. Since the Tanh function and the Sigmoid function are related: Similarly, by fitting the highly nonlinear Tanh curve to the output curve of a CMOS inverter, the output is passed through an inverting amplifier (with a bias voltage at the non-inverting input) to obtain the Sigmoid function curve. When the input signal... x When entering the function circuit, the parameters are first implemented through an inverting amplifier. a With input x / The product of 2 is then output through a CMOS inverter. The transformation from the Tanh function to the Sigmoid function is achieved through an amplifier, and finally, the parameter is achieved through an inverting amplifier. b and The product of.

[0025] like Figure 7 As shown in (b), the Tanh function circuit includes: two memristors for storing adaptive parameters, two inverting amplifiers, and a CMOS inverter. The highly nonlinear Tanh curve is fitted through the output curve of the CMOS inverter. When the input signal... x When entering the function circuit, the parameters are first implemented through an inverting amplifier. b With input x The product is then output through a CMOS inverter. Finally, the parameters are achieved through an inverting amplifier. b and The product of.

[0026] like Figure 7 As shown in (c), the ReLU function circuit includes: two memristors for storing adaptive parameters, two diode switches for controlling positive and negative conduction, two inverting operational amplifiers for implementing multiplication, and an inverting operational amplifier that converts current into voltage. When the input signal... x Upon entering the circuit, because the voltage on the right side of the diode is clamped to 0, when the input is positive, the lower branch diode conducts, and the input signal... x Parameters are achieved through an inverting amplifier. With input x The product of these two factors yields the positive output current of the function; conversely, when the input is negative, the upper branch diode conducts, and the input signal... x Parameters are achieved through an inverting amplifier. With input x The product of the two is used to obtain the negative output current of the function. The positive and negative currents flow into the same node and are converted into a voltage output by an inverting amplifier. function.

[0027] S2, construct a memristor cross array, based on Ohm's law and Kirchhoff's current law, to convert the input voltage into current output through the memristor cross array; and input the current output into the adaptive activation function circuit, and perform nonlinear processing through the adaptive activation function to obtain the network layer output after the adaptive activation function processing; Input voltage x The process of converting the memristor cross array into a current output z is as follows: in, w The weight matrix stored in the memristor cross array. c This is the array bias term.

[0028] The memristor cross array includes a 1T1R large-scale integrated structure where storage and computation are co-located. It consists of two sets of mutually orthogonal nanowires (word lines WL and bit lines BL) and a source line SL. The memristor cross array is used to simulate matrix operations in a neural network. Each intersection of the array grid places a transistor and a memristor device. When the input enters the array through the bit line, a voltage signal is applied at the word line WL to control the switching of the transistor. When the transistor is turned on, a current will be generated and flow through the memristor. The current is superimposed and output through the source line SL to realize the weighted summation process of the neural network.

[0029] The process of nonlinear processing of the current output y through the adaptive activation function includes: taking A-Tanh as an example, the output current of the memristor array is converted into voltage through TIA and input to the function circuit; the memristor stores adaptive parameters; and the current is processed by the inverting amplifier formula. By combining the parameters with the input voltage, and considering the S-shaped transfer characteristic curve of the CMOS inverter, an approximately nonlinear output can be obtained. Upon completion of a training iteration, continuous pulse voltages are applied across the memristor based on calculations, causing a change in the memristor's resistance and resulting in an adaptive change in the shape of the circuit function.

[0030] S3 uses an adaptive activation function to train the neural network, calculates the loss function L by off-chip computation of the network layer outputs, and calculates the gradient of the adaptive parameters. To avoid the impact of noise on the accuracy of gradient calculation during backpropagation, the loss function, gradient calculation, and optimizer update are performed on the software by an off-chip CPU or GPU.

[0031] In order for the network to learn more complex features, the output of the network layers is obtained. x Nonlinear processing is then required, which applies to conventional activation functions. After calculating the loss function L, the weights are updated: get: in, These are the updated parameters. For the learning rate, and after introducing a learnable parameter (adaptive parameter): get: After introducing a , b After two adaptive parameters, the actual learning rate becomes... ,when ab> At a learning rate of 1, a larger learning rate results in faster model convergence. ab When the learning rate is less than 1, the learning rate is lower, which helps improve the stability of the model. It allows the model to adaptively adjust the learning rate and ensures smooth gradient updates.

[0032] As for parameters a , b have: Taking A-Sigmoid as an example, due to a , b >0, , Therefore, the parameters a , b Update is affected by current input z As a result, the network will adaptively adjust the shape of the function during training.

[0033] This step uses the adaptive activation function instead of the traditional activation function to train the relevant neural network. By adjusting the function's shape, the function's characteristics are dynamically adjusted to optimize computational accuracy.

[0034] S4 updates the adaptive parameters in the memristor based on the gradient of the adaptive parameters, dynamically optimizes the shape of the adaptive activation function, and accelerates the neural network.

[0035] The gradient is calculated using the loss function to guide the magnitude and direction of the adaptive parameter update. The LTP (Long-Term Enhancement) and LTD (Long-Term Suppression) characteristics of the memristor are utilized to update the memristor in real time by outputting coded pulse voltage.

[0036] Specifically: The adaptive parameter change curve can be realized through the continuous change characteristic of the conductance of the memristor. When the image input is output through the memristor cross array and the adaptive function circuit, the output calculates the loss function, gradient and optimizer update through the PC. Then, the parameter update instruction is issued through the ARM. After the instruction is encoded into a continuous voltage pulse, the parameters (conductance) stored in the memristor will change continuously. When the loss function decreases, the pulse voltage is continuously applied. When the loss function increases, the polarity of the pulse voltage is reversed. If the loss function no longer changes, it means that the network has converged and the pulse voltage is stopped.

[0037] Example 2 In this embodiment, an adaptive activation function is used to train the convolutional neural network, and the performance of the traditional activation function is tested and compared in an image recognition task.

[0038] The activation function of the convolutional neural network is replaced with an adaptive activation function. A two-layer shallow CNN structure is used, with a pooling layer and an adaptive activation function following each convolutional layer. To ensure input stability, regularization is applied to the output of each layer. The corresponding... a , b The adaptive parameters are randomly initialized, and the optimizer uses the Adam algorithm with a learning rate of 10. -3 The training cycles were tested on the MNIST and CIFAR10 datasets for 100 epochs respectively.

[0039] like Figure 4 The figure shows the training loss and test accuracy of the proposed method for various activation functions on the MNSIT dataset. This figure demonstrates that different types of adaptive activation functions, compared to traditional fixed activation functions, exhibit lower training loss and faster convergence. This indicates that adaptive activation functions can change their shape by adjusting parameters, better adapting to the network, while fixed activation functions can only adapt to the input by continuously adjusting weights. Adaptive activation functions continuously provide higher model expressiveness and optimization freedom throughout the training process, reducing approximation errors and improving the final convergence test accuracy.

[0040] The model's noise resistance was then tested by applying Gaussian noise of varying intensities to the input. Figure 5As shown, the adaptive activation function has good noise resistance. In environments with noise intensity not exceeding 0.1, networks using the A-Tanh or A-ReLU functions can still maintain an accuracy of over 98%. Networks using the A-Sigmoid function have slightly worse performance, but can still achieve high accuracy in environments with noise intensity not exceeding 0.05, indicating that networks using adaptive functions are not easily affected by environmental interference.

[0041] To investigate whether the parameters can be updated stably, this invention tested the parameter variation curves of each layer with the iteration period, such as... Figure 6 As shown in the figure, the adaptive parameter update curves exhibit significant differences depending on the input to each layer, indicating that the adaptive function has better expressive power and can learn more complex nonlinear features.

[0042] To explore the versatility of function circuits on more complex datasets and network structures, this invention was tested on the CIFAR10 dataset at different layers, such as... Figure 6 As shown, the accuracy of image recognition increases with the number of network layers. However, after a certain number of layers, the improvement in network recognition accuracy becomes less significant. In fact, deeper networks increase model complexity, indirectly affecting computational efficiency and energy efficiency. Therefore, a 5-layer CNN was used for testing. Similar to the results of the MNIST dataset test, the convergence speed and accuracy of the adaptive activation function were improved to a certain extent, while maintaining a high accuracy. This indicates that the adaptive activation function can still show better performance on some more complex networks and datasets.

[0043] Example 3 In this embodiment, a neural network hardware acceleration system based on an adaptive activation function circuit is proposed, specifically including: Activation function operation module: Constructs an adaptive activation function circuit using curve fitting method and stores the adaptive parameters through memristors; Matrix operation module: Constructs a memristor cross array, and based on Ohm's law and Kirchhoff's current law, converts the input voltage into a current output through the memristor cross array; and inputs the current output into an adaptive activation function circuit, performs nonlinear processing through adaptive parameters, and obtains the network layer output after adaptive activation function processing; Off-chip computation module: Trains the neural network using an adaptive activation function, calculates the loss function and the gradient of the adaptive parameters by off-chip computation of the network layer outputs; Dynamic optimization module: Updates the adaptive parameters in the memristor based on the gradient of the adaptive parameters, dynamically optimizes the shape of the adaptive activation function, and realizes hardware acceleration of the neural network.

[0044] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A hardware acceleration method for neural networks based on adaptive activation function circuits, characterized in that, Includes the following steps: An adaptive activation function circuit is constructed using curve fitting, and adaptive parameters are stored using a memristor. A memristor cross array is constructed, and based on Ohm's law and Kirchhoff's current law, the input voltage is converted into current output through the memristor cross array; the current output is then input into an adaptive activation function circuit, and nonlinear processing is performed through adaptive parameters to obtain the network layer output after adaptive activation function processing; The neural network is trained using an adaptive activation function, and the loss function and gradient of the adaptive parameters are calculated by off-chip computation of the network layer outputs. The adaptive parameters in the memristor are updated based on the gradient of the adaptive parameters, and the shape of the adaptive activation function is dynamically optimized.

2. The neural network hardware acceleration method based on adaptive activation function circuit according to claim 1, characterized in that, The types of adaptive activation function circuits include: Sigmoid function circuit, Tanh function circuit, and ReLU function circuit; the Sigmoid function circuit is implemented by curve fitting through the nonlinear output characteristics of the CMOS inverter; the Tanh function circuit is implemented by adding an inverting amplifier to the Sigmoid circuit.

3. The neural network hardware acceleration method based on adaptive activation function circuit according to claim 2, characterized in that, The activation function expression in the adaptive activation function is: in, These are adaptive parameters, stored via memristors. , , These are the adaptive activation functions for the Sigmoid function circuit, the Tanh function circuit, and the ReLU function circuit, respectively. x This is the input voltage for the activation function circuit.

4. The neural network hardware acceleration method based on adaptive activation function circuit according to claim 1, characterized in that, The process by which the input voltage is converted into a current output z through a memristor cross array is as follows: Where w is the weight matrix stored in the memristor cross array, x is the input of the memristor cross array, and c is the array bias term.

5. An adaptive activation function circuit, characterized in that, This includes any one of the following: Sigmoid function circuit, Tanh function circuit, and ReLU function circuit; The Sigmoid function circuit includes: two memristors for storing adaptive parameters, three inverting amplifiers, and a CMOS inverter. The output curve of the CMOS inverter is fitted with a Tanh curve, and the output is then passed through an inverting amplifier to obtain the Sigmoid function curve. When the input signal... x When entering the function circuit, the parameters are first achieved through an inverting amplifier. With input x The product of / 2 is then output as Tanh ( ) via a CMOS inverter. ax / 2), and the transformation of the Tanh function to the Sigmoid function is achieved through an inverting proportional amplifier, and finally the parameter is achieved through an inverting proportional amplifier. b With Sigmoid ( ax The product of ) The Tanh function circuit includes: two memristors for storing adaptive parameters, two inverting amplifiers, and a CMOS inverter; the Tanh curve is fitted through the output curve of the CMOS inverter, and when the input signal... x When entering the function circuit, the parameters are first achieved through an inverting amplifier. a With input x The product of these two terms is then output as Tanh( ) via a CMOS inverter. ax Finally, the parameters are achieved through an inverting amplifier. b With Tanh ( ax The product of ) The ReLU function circuit includes: two memristors for storing adaptive parameters, two diode switches for controlling positive and negative conduction, two inverting operational amplifiers for performing multiplication, and an inverting operational amplifier for converting current to voltage; when the input signal... x Upon entering the circuit, the voltage on the right side of the diode is clamped to 0. When the input is positive, the lower branch diode conducts, and the input signal... x Parameters are achieved using an inverting operational amplifier. b With input x The product of the two signals yields a positive output current; conversely, when the input is negative, the upper branch diode conducts, and the input signal... x Parameters are achieved using an inverting operational amplifier. a With input x The product of the two currents yields the negative output current. The positive and negative currents flow into the same node and are converted into a voltage output by an inverting operational amplifier. function.

6. The application of the adaptive activation function circuit according to claim 5 in CNN image recognition.

7. A neural network hardware acceleration system based on an adaptive activation function circuit, comprising executing the neural network hardware acceleration method based on an adaptive activation function circuit as described in any one of claims 1-4, characterized in that, include: Activation function operation module: Constructs an adaptive activation function circuit using curve fitting method and stores the adaptive parameters through memristors; Matrix operation module: Constructs a memristor cross array, and based on Ohm's law and Kirchhoff's current law, converts the input voltage into a current output through the memristor cross array; and inputs the current output into an adaptive activation function circuit, performs nonlinear processing through adaptive parameters, and obtains the network layer output after adaptive activation function processing; Off-chip computation module: Trains the neural network hardware using an adaptive activation function, calculates the loss function and gradients of the adaptive parameters by off-chip computation of the network layer outputs; Dynamic optimization module: Updates the adaptive parameters in the memristor based on the gradient of the adaptive parameters, and dynamically optimizes the shape of the adaptive activation function.

8. A computer storage medium storing a readable program, characterized in that, When the program runs, it can instruct the computing device to execute the neural network hardware acceleration method based on the adaptive activation function circuit as described in any one of claims 1-4.

9. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the neural network hardware acceleration method based on the adaptive activation function circuit as described in any one of claims 1-4.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions instruct the computing device to perform the operations corresponding to the neural network hardware acceleration method based on adaptive activation function circuits as described in any one of claims 1-4.

Images