Universal circuit capable of realizing two-section or three-section piecewise linear activation function

Abstract

The invention provides a piecewise linear activation function general circuit capable of realizing two sections or three sections, and relates to the technical field of circuit design, the piecewise linear activation function general circuit comprises a piecewise module and an activation module, and the piecewise module is connected with the activation module; the segmentation module is used for carrying out voltage segmentation processing on the signal to be subjected to segmentation processing to obtain segmented voltage; and the activation module is used for performing activation control processing on the sectional voltage to obtain a final output voltage and outputting the final output voltage. The method is high in configurability, supports the realization of various piecewise linear activation functions, is good in universality, is simple in circuit structure, is few in types of used elements, is suitable for being applied to a pure hardware neural network circuit, and facilitates the improvement of the integration level and the operation efficiency of a whole system.

Description

Technical Field

[0001] The present invention relates to the technical field of circuit design, and in particular to a general circuit capable of realizing a two-segment or three-segment piecewise linear activation function. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep neural networks (DNNs) have been widely used in fields such as image recognition and natural language processing. However, traditional computing platforms face a "memory wall" bottleneck, encountering significant energy consumption challenges when processing large-scale neural networks. To address this issue, computing-in-memory (CIM) has emerged. It integrates computing units directly into non-volatile memory, enabling in-situ computing without moving data, significantly reducing the energy consumption associated with data transmission. Among various emerging memory technologies, memristors have become the device of choice for accelerating DNN computing due to their low power consumption, high-speed response, long lifespan, and high compatibility with CMOS technology.

[0003] In traditional memristive neural network architectures, the multiply-accumulate (MAC) operation is typically performed within a memristive crossbar array, while the activation function relies on CPU computation. As a key component of neural networks, the activation function introduces nonlinearity, enabling the network to fit complex functional relationships and data distributions, significantly improving the model's expressiveness and generalization capabilities. During each computation cycle, the input vector is converted into an analog voltage via a digital-to-analog converter (DAC) and applied to the memristive crossbar array. The MAC operation is calculated based on Ohm's law and Kirchhoff's laws. The result is converted back to a digital signal via an analog-to-digital converter (ADC), where a nonlinear activation function is applied in the digital domain. However, this analog-to-digital conversion process wastes resources and increases power consumption. By contrast, directly implementing the activation function in analog circuitry not only avoids the ADC conversion overhead but also allows integration into the peripheral circuitry of the memristive crossbar array, resulting in lower power consumption and higher computation speed. Therefore, developing efficient hardware circuits for the activation function is crucial.

[0004] The piecewise linear activation function is a nonlinear activation function widely used in neural networks. Its main feature is that it divides the input value into multiple linear intervals, using a different linear expression for each interval. This not only enhances the network's expressiveness, enabling it to better fit complex data distributions, but also reduces computational complexity, thereby accelerating the neural network's computational process. Typical piecewise linear activation functions include ReLU and its variants, such as Leaky ReLU and ReLU6. In addition, some piecewise linear activation functions are implemented by approximating exponential functions. For example, Hard Sigmoid and Hard Tanh are simplified forms of the Sigmoid and Tanh functions, respectively.

[0005] However, existing circuit implementations of piecewise linear activation functions remain relatively limited. Most designs are only applicable to specific function structures and lack versatility. When adapting to different activation functions, the circuit architecture often needs to be redesigned, resulting in low resource utilization and poor adaptability.

[0006] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0007] In view of this, the present invention provides a general circuit that can implement a two-segment or three-segment piecewise linear activation function to solve the above-mentioned problems.

[0008] In order to solve the above problems, the specific technical solutions adopted by the present invention are as follows: A general circuit for realizing a two-segment or three-segment piecewise linear activation function, comprising: a segmentation module and an activation module, wherein the segmentation module is connected to the activation module; The segmentation module is used to perform voltage segmentation processing on the signal to be segmented to obtain segmented voltage; The activation module is used to perform activation control processing on the segmented voltage to obtain and output the final output voltage.

[0009] Furthermore, in order to generate multiple reference voltages at key nodes of the circuit through four independent voltage division branches and in combination with corresponding power supplies, the segmentation module includes: Input terminal V in , a first resistor R1, a second resistor R2, a third resistor R3, a fourth resistor R4, a fifth resistor R5, a sixth resistor R6, a seventh resistor R7, an eighth resistor R8, a first voltage source V1, a second voltage source V2, a third voltage source V3 and a fourth voltage source V4; the input terminal V in One end of is connected to one end of the first resistor R1, one end of the third resistor R3, one end of the fifth resistor R5, and one end of the seventh resistor R7 in sequence, the other end of the first resistor R1 is connected to one end of the second resistor R2, the other end of the second resistor R2 is connected to one end of the first voltage source V1, the other end of the third resistor R3 is connected to one end of the fourth resistor R4, the other end of the fourth resistor R4 is connected to one end of the second voltage source V2, the other end of the fifth resistor R5 is connected to one end of the sixth resistor R6, the other end of the sixth resistor R6 is connected to one end of the third voltage source V3, and the other end of the seventh resistor R7 is connected to one end of the fourth voltage source V4; the other end of the first voltage source V1, the other end of the second voltage source V2, the other end of the third voltage source V3, and the other end of the fourth voltage source V4 are all grounded.

[0010] Furthermore, in order to implement activation control processing for the segmented voltage, the activation module includes: Output V out , the ninth resistor R9, the tenth resistor R 10 , a first transistor T1, a second transistor T2 and a fifth voltage source V5; the output terminal V out One end of the tenth resistor R 10 One end of the tenth resistor R is connected to the drain of the second transistor T2. 10 The other end of the ninth resistor R9 is connected to one end of the ninth resistor R9 and the drain of the first transistor T1 in sequence, the gate of the second transistor T2 is connected to the other end of the third resistor R3 and one end of the fourth resistor R4 in sequence, the source of the second transistor T2 is connected to the other end of the first resistor R1 and one end of the second resistor R2 in sequence, the other end of the ninth resistor R9 is connected to the other end of the fifth resistor R5 and one end of the sixth resistor R6 in sequence, the gate of the first transistor T1 is connected to the fifth voltage source V5, the source of the first transistor T1 is connected to the other end of the seventh resistor R7 and one end of the eighth resistor R8 in sequence; the other end of the fifth voltage source V5 is grounded, the first transistor T1 is a P-type field effect transistor, the threshold of the first transistor T1 is -1V, the second transistor T2 is an N-type field effect transistor, the threshold of the second transistor T2 is 0.5V, the resistance of the ninth resistor R9 is 1kΩ, the tenth resistor R 10 The resistance value is 1kΩ.

[0011] Furthermore, in order to realize the two-stage and three-stage segmentation, the voltage segmentation processing of the signal to be segmented includes: two-stage segmentation and three-stage segmentation; The two-stage segmentation satisfies: R4 / R3=R2 / R1; V2R3 / (R3+R4)-V1R1 / (R1+R2)>0.5; The three-stage segmentation satisfies: R4 / R3>R2 / R1.

[0012] The beneficial effects of the present invention are: the present invention is highly configurable, supports the implementation of multiple piecewise linear activation functions, including but not limited to ReLU, ReLU6, Leaky ReLU, HardSigmoid and HardTanh activation functions, has good versatility, and has a simple circuit structure and uses a small number of components. It is suitable for application in pure hardware neural network circuits, which helps to improve the integration and operating efficiency of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 1 is a schematic diagram of a general circuit for implementing a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention; Figure 2 This is a simulation diagram of a ReLU function in a general circuit that can implement a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention; Figure 3 This is a simulation diagram of a ReLU6 function in a general circuit that can implement a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention; Figure 4 This is a simulation diagram of a Leaky ReLU function in a general circuit that can implement a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention; Figure 5 This is a simulation diagram of a HardSigmoid function in a general circuit that can implement a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention; Figure 6 This is a simulation diagram of the HardTanh function in a general circuit that can implement a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention.

[0014] In the picture: 1. Segmentation module; 2. Activation module. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0016] According to an embodiment of the present invention, a general circuit capable of implementing a two-segment or three-segment piecewise linear activation function is provided.

[0017] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1As shown, a general circuit for realizing a two-segment or three-segment piecewise linear activation function according to an embodiment of the present invention includes: a segmentation module 1 and an activation module 2, wherein the segmentation module 1 is connected to the activation module 2; The segmentation module 1 is used to perform voltage segmentation processing on the signal to be segmented to obtain segmented voltage; The activation module 2 is used to perform activation control processing on the segmented voltage to obtain and output the final output voltage.

[0018] By means of the above-mentioned technical solution of the present invention, the present invention is highly configurable and supports the implementation of multiple piecewise linear activation functions, including but not limited to ReLU, ReLU6, Leaky ReLU, Hard Sigmoid, and Hard Tanh activation functions. It has good versatility, a simple circuit structure, and uses a small number of components. It is suitable for application in pure hardware neural network circuits, helping to improve the integration and operating efficiency of the overall system. The core of the present invention is to achieve a piecewise linear relationship between the output voltage and the input voltage through the flexible configuration of resistance parameters and bias voltage.

[0019] In an optional embodiment, for the above segmentation module 1, the segmentation module 1 includes: Input terminal V in , a first resistor R1, a second resistor R2, a third resistor R3, a fourth resistor R4, a fifth resistor R5, a sixth resistor R6, a seventh resistor R7, an eighth resistor R8, a first voltage source V1, a second voltage source V2, a third voltage source V3 and a fourth voltage source V4; the input terminal V in One end of is connected to one end of the first resistor R1, one end of the third resistor R3, one end of the fifth resistor R5, and one end of the seventh resistor R7 in sequence; the other end of the first resistor R1 is connected to one end of the second resistor R2, the other end of the second resistor R2 is connected to one end of the first voltage source V1, the other end of the third resistor R3 is connected to one end of the fourth resistor R4, the other end of the fourth resistor R4 is connected to one end of the second voltage source V2, the other end of the fifth resistor R5 is connected to one end of the sixth resistor R6, the other end of the sixth resistor R6 is connected to one end of the third voltage source V3, and the other end of the seventh resistor R7 is connected to one end of the fourth voltage source V4; the other ends of the first voltage source V1, the second voltage source V2, the third voltage source V3, and the fourth voltage source V4 are all grounded, so that multiple reference voltages can be generated at key nodes of the circuit through four independent voltage division branches and in combination with corresponding power supplies.

[0020] In an optional embodiment, for the above-mentioned activation module 2, the activation module 2 includes: Output V out , the ninth resistor R9, the tenth resistor R 10 , a first transistor T1, a second transistor T2 and a fifth voltage source V5; the output terminal V out One end of the tenth resistor R 10 One end of the tenth resistor R is connected to the drain of the second transistor T2. 10 The other end of the ninth resistor R9 is connected to one end of the ninth resistor R9 and the drain of the first transistor T1 in sequence, the gate of the second transistor T2 is connected to the other end of the third resistor R3 and one end of the fourth resistor R4 in sequence, the source of the second transistor T2 is connected to the other end of the first resistor R1 and one end of the second resistor R2 in sequence, the other end of the ninth resistor R9 is connected to the other end of the fifth resistor R5 and one end of the sixth resistor R6 in sequence, the gate of the first transistor T1 is connected to the fifth voltage source V5, the source of the first transistor T1 is connected to the other end of the seventh resistor R7 and one end of the eighth resistor R8 in sequence; the other end of the fifth voltage source V5 is grounded, the first transistor T1 is a P-type field effect transistor, and the threshold of the first transistor T1 is -1V; the second transistor T2 is an N-type field effect transistor, and the threshold of the second transistor T2 is 0.5V; the resistance value of the ninth resistor R9 is 1kΩ; the tenth resistor R 10 The resistance value is 1kΩ, so that activation control processing of the segmented voltage can be achieved.

[0021] In an optional embodiment, the voltage segmentation processing of the signal to be segmented includes: two-segment segmentation and three-segment segmentation, wherein the two-segment segmentation implements a two-segment piecewise linear activation function, and the three-segment segmentation implements a three-segment piecewise linear activation function.

[0022] In addition, in order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are further described below. As shown in Table 1, circuit parameter configurations corresponding to different activation functions are shown. Table 1 Parameter configuration of various activation functions in general circuits Specifically, the segmented implementation logic of the present invention includes: 1) To realize the two-stage piecewise linear activation function, the resistance ratio relationship R4 / R3=R2 / R1 must be satisfied, and the inequality: V2R3 / (R3+R4)-V1R1 / (R1+R2)>0.5 must be satisfied. Under the above conditions, the second transistor T2 is in the cut-off state, and the circuit operates in the two-stage linear region. When the input source V in When it is less than [(V5+1)(R7+R8)-V4R7] / R8, the output terminal Vout =V in R6 / (R5+R6)+V3R5 / (R5+R6); when the input source V in When the output voltage V out =V in R8 / (R7+R8)+V4R7 / (R7+R8).

[0023] 2) To realize the three-stage piecewise linear activation function, the resistance relationship must be satisfied: R4 / R3>R2 / R1. For the input source, when the input source V in When it is less than [(V5+1)(R7+R8)-V4R7] / R8, the output V out =V in R6 / (R5+R6)+V3R5 / (R5+R6); when the input source V in Greater than [(V5+1)(R7+R8)-V4R7] / R8 and the input source V in When the output voltage is less than [1 / 2+V1R1 / (R1+R2)-V2R3 / (R3+R4)] / [R4 / (R3+R4)-R2 / (R1+R2), the output voltage V out =V in R8 / (R7+R8)+V4R7 / (R7+R8); In addition, when the input source V in When it is greater than [1 / 2+V1R1 / (R1+R2)-V2R3 / (R3+R4)] / [R4 / (R3+R4)-R2 / (R1+R2), the output V out =V in R2 / (R1+R2)+V1R1 / (R1+R2).

[0024] In order to verify the effectiveness of the universal circuit of the piecewise linear activation function, the present invention uses LTspice simulation software to perform modeling and simulation. The first transistor T1 uses a P-type MOS tube (P-type field effect transistor) with a model number of SP40P04TQ, and the second transistor T2 uses an N-type MOS tube (N-type field effect transistor) with a model number of SVDZ24NT. The circuit is then stimulated by V in It is a sine wave with an amplitude of 10V and a frequency of 100Hz. Figures 2 to 6 The simulation input and output waveforms of different functions are shown respectively ( Figure 2-Figure 6 Figure a in the figure is a comparison of the simulation and ideal curves of different functions. Figure 2-Figure 6 (Figure b in the figure is the error curve diagram of different functions) It can be seen from the simulation results that the output voltage waveform is highly consistent with the expected activation function characteristics, which verifies the effectiveness and feasibility of the universal piecewise linear activation function circuit proposed in the present invention in functional implementation.

[0025] To sum up, with the help of the above technical solutions of the present invention, the present invention has strong configurability and supports the implementation of multiple piecewise linear activation functions, including but not limited to ReLU, ReLU6, Leaky ReLU, HardSigmoid and HardTanh activation functions. It has good versatility, a simple circuit structure, and uses a small number of components. It is suitable for application in pure hardware neural network circuits, which helps to improve the integration and operation efficiency of the overall system.

[0026] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0027] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A general circuit that can implement a two-segment or three-segment piecewise linear activation function, characterized in that: include: A segmentation module and an activation module, wherein the segmentation module is connected to the activation module; The segmentation module is used to perform voltage segmentation processing on the signal to be segmented to obtain segmented voltage; The activation module is used to perform activation control processing on the segmented voltage to obtain and output the final output voltage.

2. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 1, characterized in that: The segmentation module includes: Input terminal V in , a first resistor R1, a second resistor R2, a third resistor R3, a fourth resistor R4, a fifth resistor R5, a sixth resistor R6, a seventh resistor R7, an eighth resistor R8, a first voltage source V1, a second voltage source V2, a third voltage source V3 and a fourth voltage source V4; The input terminal V in one end of the resistor R1 is connected to one end of the first resistor R1, one end of the third resistor R3, one end of the fifth resistor R5, and one end of the seventh resistor R7 in sequence, the other end of the first resistor R1 is connected to one end of the second resistor R2, the other end of the second resistor R2 is connected to one end of the first voltage source V1, the other end of the third resistor R3 is connected to one end of the fourth resistor R4, the other end of the fourth resistor R4 is connected to one end of the second voltage source V2, the other end of the fifth resistor R5 is connected to one end of the sixth resistor R6, the other end of the sixth resistor R6 is connected to one end of the third voltage source V3, and the other end of the seventh resistor R7 is connected to one end of the fourth voltage source V4; The other end of the first voltage source V1 , the other end of the second voltage source V2 , the other end of the third voltage source V3 , and the other end of the fourth voltage source V4 are all grounded.

3. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 2, characterized in that: The activation module includes: Output V out , the ninth resistor R9, the tenth resistor R 10 , a first transistor T1, a second transistor T2 and a fifth voltage source V5; The output terminal V out One end of the tenth resistor R 10 One end of the tenth resistor R is connected to the drain of the second transistor T2. 10 the other end of the ninth resistor R9 is connected in sequence to one end of the ninth resistor R9 and the drain of the first transistor T1, the gate of the second transistor T2 is connected in sequence to the other end of the third resistor R3 and one end of the fourth resistor R4, the source of the second transistor T2 is connected in sequence to the other end of the first resistor R1 and one end of the second resistor R2, the other end of the ninth resistor R9 is connected in sequence to the other end of the fifth resistor R5 and one end of the sixth resistor R6, the gate of the first transistor T1 is connected to the fifth voltage source V5, and the source of the first transistor T1 is connected in sequence to the other end of the seventh resistor R7 and one end of the eighth resistor R8; The other end of the fifth voltage source V5 is grounded.

4. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 3, characterized in that: The first transistor T1 is a P-type field effect transistor, and the threshold of the first transistor T1 is -1V.

5. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 3, characterized in that: The second transistor T2 is an N-type field effect transistor, and the threshold of the second transistor T2 is 0.5V.

6. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 3, characterized in that: The resistance value of the ninth resistor R9 is 1 kΩ.

7. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 4, characterized in that: The tenth resistor R 10 The resistance value is 1kΩ.

8. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 4, characterized in that: The voltage segmentation processing of the signal to be segmented includes two-segment segmentation and three-segment segmentation.

9. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 8, characterized in that: The two-stage segmentation satisfies: R4 / R3=R2 / R1; V2R3 / (R3+R4)-V1R1 / (R1+R2)>0.

5.

10. A general circuit capable of implementing a two-segment or three-segment piecewise linear activation function according to claim 8, characterized in that: The three-stage segmentation satisfies: R4 / R3>R2 / R1.

Images