Method for realizing simulation domain ELU activation function based on programmable field effect transistor

Abstract

The invention provides a method for realizing an analog domain ELU activation function based on a programmable field effect transistor, and belongs to the field of neural network accelerators. According to the method, ELU activation of an input voltage is realized by using a response charge-voltage relationship obtained by a programmable field effect transistor through pre-programmed capacitance-voltage characteristic integration, and a charge integrator linearly converts a response charge into an output voltage representing an activation value. According to the method, the important ELU activation function in the neural network is realized in the simulation domain in a high-energy-efficiency manner, the expense of analog-to-digital conversion of a computing system in the simulation domain is reduced, and the realization of a high-energy-efficiency neural network accelerator is facilitated. In addition, the invention also provides support for in-situ representation of signed weights in the in-memory computing array.

Description

Technical Field

[0001] This invention relates to the field of neural network accelerators, and more specifically to a method for implementing an analog domain ELU activation function based on a programmable field-effect transistor. Background Technology

[0002] With the rapid development of artificial intelligence technology, neural networks have become core algorithms in many fields. Among them, nonlinear activation functions are crucial for introducing the nonlinear expressive power of models, enabling them to capture complex patterns and relationships in input data. Among numerous activation functions, the Exponential Linear Unit (ELU) has received widespread attention due to its superior performance. Compared to the traditional Rectified Linear Unit (ReLU), ELU can effectively alleviate the "neuron death" problem caused by ReLU's zero gradient when the input is negative, thus generally helping to improve the convergence speed and accuracy of the model.

[0003] In the field of neural network accelerators, analog domain in-memory computing (ACIM) technology has emerged to overcome the energy efficiency and latency bottlenecks of traditional von Neumann architectures in data movement. By performing analog multiplication and accumulation operations directly within memory cells, this technology greatly reduces the frequent movement of data between the processor and memory, thus demonstrating superior energy efficiency and throughput advantages when implementing computations in linear layers of neural networks (such as fully connected layers and convolutional layers).

[0004] However, existing ACIM technologies can typically only efficiently perform linear computations. For nonlinear activation functions in networks, current mainstream solutions still require analog-to-digital converters (ADCs) to convert analog computation results into digital signals, followed by nonlinear computations in the digital domain. This frequent inter-domain conversion not only introduces significant overhead but also largely obscures the energy efficiency advantages inherent in ACIM itself.

[0005] To fundamentally address this issue, some cutting-edge research has begun exploring schemes to directly implement nonlinear activation functions in the analog domain, aiming to eliminate the significant overhead of ADCs and construct fully analog CIM systems. While some work has implemented simple functions such as ReLU in the analog domain, there is currently a lack of effective schemes for directly implementing the more powerful and mathematically complex ELU function in analog circuits. This technological gap severely restricts the energy-efficient implementation of the ELU activation function in fully analog CIM systems, becoming a key obstacle to the practical application of corresponding neural network accelerators. Summary of the Invention

[0006] To address the problems in the prior art described above, this invention proposes a method for implementing the analog domain ELU activation function based on a programmable field-effect transistor, which can reduce the A / D conversion overhead of ACIM when accelerating neural networks, realize fully analog CIM, and improve computational energy efficiency.

[0007] This invention provides the following technical solution: A method for implementing an analog domain ELU activation function based on a programmable field-effect transistor, characterized in that, 1) A charge generation module is constructed using n-type programmable field-effect transistors; wherein the gate terminal of the programmable field-effect transistor receives the input voltage, and its drain terminal is shorted to the source terminal and connected to virtual ground; 2) Pre-program the programmable field-effect transistor (FET) so that its CV function matches the first derivative of the ELU function; change the gate input voltage of the FET from 0 to V. in During the process, the response charge Q of the programmable field-effect transistor is proportional to the ELU; 3) A charge-to-voltage conversion module is constructed using a charge integrator to convert the response charge Q generated by the programmable field-effect transistor into a proportional voltage output V. out This enables the output of ELU activation results.

[0008] Furthermore, the charge integrator consists of a two-stage operational amplifier (OPAMP) and three reference capacitors. The positive input terminal of the first-stage operational amplifier is grounded, and its negative input terminal is connected to the drain and source terminals of the programmable field-effect transistor and the first reference capacitor, with a potential of -V. out The output terminal is connected to the first reference capacitor and the second reference capacitor; the positive input terminal of the second-stage operational amplifier is grounded, and the negative input terminal is connected to the second reference capacitor and the third reference capacitor. The output voltage V connected to the third reference capacitor is... out .

[0009] Furthermore, the charge integrator output voltage V out At the same time, it also obtained -V out Voltage; in the ACIM array, the output voltage V out V, as input to the next layer of the network x At that time, the -V out Voltage input -V in the weighted sign bit x This enables the in-situ representation of the weight sign bits in the ACIM array.

[0010] Furthermore, the programmable field-effect transistor (FET) employs a floating-gate transistor, a charge-trapping transistor, or a ferroelectric transistor. The ferroelectric transistor utilizes a metal-ferroelectric-semiconductor (MFS) structure, a metal-ferroelectric-channel-side dielectric-semiconductor (MFIS) structure, a metal-ferroelectric-metal-channel-side dielectric-semiconductor (MFMIS) structure, or a metal-gate-side dielectric-ferroelectric-channel-side dielectric-semiconductor (MIFIS) structure. The ferroelectric element employs perovskite ferroelectric materials, two-dimensional ferroelectric materials, organic ferroelectric materials, group III-V nitride ferroelectric materials, or hafnium oxide-doped ferroelectric materials.

[0011] The technical effects of this invention are as follows: This invention utilizes the pre-programmed response charge-voltage relationship of a programmable field-effect transistor to achieve ELU activation of the input voltage, and uses a charge integrator to linearly convert the response charge into an output voltage representing the activation value. This realizes the important ELU activation function in the analog domain, reduces the analog-to-digital conversion overhead of the ACIM system, and is beneficial for realizing a high-efficiency neural network accelerator.

[0012] This invention obtains representative ELU(V) in Output voltage V out At the same time, it also obtained -V out Voltage; in V out V, as input to the next layer of the network x At that time, the -V out Voltage can support inputting -V in the weighted sign bit. x This allows the sign bit of the weight to be represented in situ within the ACIM array, simplifying the representation of signed weights. Attached Figure Description

[0013] Figure 1 This is a circuit diagram illustrating the implementation of the analog domain ELU activation function based on a programmable field-effect transistor according to the present invention.

[0014] Figure 2 This is a schematic diagram illustrating the principle of implementing the analog domain ELU activation function based on a programmable field-effect transistor according to the present invention.

[0015] Figure 3 This is a test result diagram of the implementation of the analog domain ELU activation function using a MIFIS structure ferroelectric transistor based on doped hafnium oxide ferroelectric material according to an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram illustrating the principle of how the analog domain ELU activation function implemented based on a programmable field-effect transistor in this invention helps to represent signed weights in an ACIM array. Detailed Implementation

[0017] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0018] In this embodiment, an n-type MIFIS structure ferroelectric transistor (FeFET) based on doped hafnium oxide ferroelectric material is used as the programmable field-effect transistor. The spontaneous polarization of the ferroelectric material will induce additional charge on the gate oxide layer of the transistor, causing a change in the threshold voltage of the transistor, and its CV relationship will shift left and right accordingly.

[0019] The circuit structure for implementing the analog domain ELU activation function based on FeFET in this embodiment is as follows: Figure 1 As shown: The FeFET constitutes a charge generation module, and its gate terminal receives the input voltage V. in The drain and source terminals are shorted to convert the input voltage V. in Converted to ELU(V) in The response charge Q is proportional to the voltage output V. A charge integrator forms a charge-to-voltage conversion module, consisting of two operational amplifier stages (OPAMP) and three reference capacitors, used to convert the FeFET's response charge Q into a proportional voltage output V. out The positive input terminal of the first-stage OPAMP is grounded, and the negative input terminal is connected to the drain and source terminals of the nFET (which are shorted), and the first reference capacitor, with a potential of -V. out The output terminal is connected to the first reference capacitor and the second reference capacitor; the positive input terminal of the second-stage OPAMP is grounded, and the negative input terminal is connected to the second reference capacitor and the third reference capacitor. The output voltage V connected to the third reference capacitor is... out Represents V in The result of ELU activation.

[0020] The principle of implementing the analog domain ELU activation function based on FeFET in this embodiment is as follows: Figure 2 As shown: Under the aforementioned port configuration, when the gate voltage of an n-type FeFET increases, its MOS capacitance tends to invert, as electrons in the inversion layer of the channel are provided by the n-type source and drain, resulting in a high capacitance value. When the gate voltage decreases, its MOS capacitance tends to accumulate, but the n-type source and drain cannot provide the required holes, resulting in a very low capacitance value. When the gate voltage is at an intermediate value, the MOS capacitance value exhibits a smooth transition. Furthermore, considering the programmable CV characteristic of the FeFET, it can be pre-programmed to ensure that the first derivative of the C(V) function matches the first derivative of the ELU function before calculation. When the FeFET gate voltage changes from 0 to V... in During the process, the resulting response charge Q, after being integrated with the voltage across the capacitor, will be related to ELU(V). in It is directly proportional to.

[0021] This embodiment is a specific implementation of the analog domain ELU activation function based on an n-type MIFIS structure ferroelectric transistor (FeFET) doped with hafnium oxide ferroelectric material: By shorting the source and drain, the FeFET can be regarded as a capacitor at both ends; a first pre-programmed pulse of (+15 V, 1 μs) is applied to the gate, followed by a second pre-programmed pulse of (-11 V, 1 μs), and AC small-signal capacitance tests are performed at these two ends. The CV relationship of the FeFET is programmed as follows: Figure 3 As shown; by integrating the capacitance with respect to voltage according to this relationship, we can obtain... Figure 3 The QV relationship is very close to that of the ELU function, for input voltage V in The response charge amount that satisfies the QV relationship can be obtained; it is then linearly converted into the output voltage V via the charge integrator module. out Output voltage V out Represents V in The result of ELU activation is then used to output the ELU activation result.

[0022] This invention obtains representative ELU(V) in Output voltage V out At the same time, you can also get -V out Voltage; in V out V, as input to the next layer of the network x At that time, the -V out Voltage can support inputting -V in the weighted sign bit. x This allows the sign bit of the weights to be represented in situ within the ACIM array, simplifying the representation of signed weights. The specific principle is as follows: Figure 4 As shown: The weights are stored in the array in two parts: the sign bit and the absolute value. The absolute value part receives the input V. x The sign bit portion receives -V x ±V x Both voltages are derived from the aforementioned output voltage V. out and -V out Voltage control; the absolute value part is based on the input V. x The multiplication operation is completed normally; when the sign bit is 0, the voltage of the multiplication result for that bit is 0, and the voltage of the multiplication result for that weight depends only on the absolute value part, indicating that the weight is positive; when the sign bit is 1, the voltage of the multiplication result for that bit is -V. x The voltage resulting from the multiplication of this weight will be shifted negatively from the absolute value of the result, indicating that the weight is negative.

[0023] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for implementing an analog domain ELU activation function based on a programmable field-effect transistor, comprising the following steps: 1) A charge generation module is constructed using n-type programmable field-effect transistors; The gate terminal of the programmable field-effect transistor receives the input voltage V. in Its drain end is shorted to the source end and connected to virtual ground; 2) Pre-program the programmable field-effect transistor (FET) so that its CV function matches the first derivative of the ELU function; change the gate input voltage of the FET from 0 to V. in During the process, the response charge Q of the programmable field-effect transistor is proportional to the ELU; 3) A charge-to-voltage conversion module is constructed using a charge integrator to convert the response charge Q generated by the programmable field-effect transistor into a proportional voltage output V. out This enables the output of ELU activation results.

2. The method for implementing an analog domain ELU activation function based on a programmable field-effect transistor as described in claim 1, characterized in that, The charge integrator consists of two OPAMP stages and three reference capacitors. The positive input of the first OPAMP stage is grounded, and its negative input is connected to the drain and source of the programmable field-effect transistor and the first reference capacitor, with a potential of -V. out The output terminal is connected to the first reference capacitor and the second reference capacitor; the positive input terminal of the second-stage OPAMP is grounded, and the negative input terminal is connected to the second reference capacitor and the third reference capacitor. The output voltage V connected to the third reference capacitor is... out Represents V in The result of ELU activation.

3. The method for implementing an analog domain ELU activation function based on a programmable field-effect transistor as described in claim 2, characterized in that, In the ACIM array, the output voltage V out V, as input to the next layer of the network x At that time, the -V out Voltage input -V in the weighted sign bit x This enables the in-situ representation of the weight sign bits in the ACIM array.

4. The method for implementing an analog domain ELU activation function based on a programmable field-effect transistor as described in claim 1, characterized in that, The programmable field-effect transistor is a floating-gate transistor, a charge-trapping transistor, or a ferroelectric transistor.

5. The method for implementing the analog domain ELU activation function based on a programmable field-effect transistor as described in claim 4, characterized in that, The ferroelectric transistor adopts a metal-ferroelectric-semiconductor structure (MFS), a metal-ferroelectric-channel-side dielectric-semiconductor structure (MFIS), a metal-ferroelectric-metal-channel-side dielectric-semiconductor structure (MFMIS), or a metal-gate-side dielectric-ferroelectric-channel-side dielectric-semiconductor structure (MIFIS).

6. The method for implementing an analog domain ELU activation function based on a programmable field-effect transistor as described in claim 5, characterized in that, The ferroelectric material used is perovskite ferroelectric material, two-dimensional ferroelectric material, organic ferroelectric material, group III-V nitride ferroelectric material or doped hafnium oxide-based ferroelectric material.

Images