Circuits, neural network processing devices, chips, and electronic devices that implement the Softmax activation function in the analog domain.

Abstract

This invention discloses a circuit, neural network processing device, chip, and electronic device for implementing the Softmax activation function in the analog domain, relating to the field of neural network technology. It solves the technical problem of the Softmax activation function consuming large amounts of power and area. The circuit for the Softmax activation function includes: a natural exponent analog processing circuit, which calculates the natural exponent of the input voltage value based on the analog circuit and replicates the result of the natural exponent calculation; a second current mirror circuit, which accumulates the converged output current based on the current mirror, with the output current being the result of the natural exponent summation; and an analog divider circuit formed by several switching transistors, used to perform division operations. This invention implements the Softmax activation function calculation in the analog domain, which, compared to the digital domain implementation, significantly reduces the number of transistors, lowers chip area and power consumption, and provides better energy efficiency and area efficiency.

Description

Technical Field

[0001] This utility model relates to the field of neural network technology, and in particular to a circuit, neural network processing device, chip and electronic device for implementing the Softmax activation function in an analog domain. Background Technology

[0002] As Moore's Law approaches its limits, the "memory wall" and "power wall" problems of AI chips based on the von Neumann architecture are becoming increasingly prominent, slowing down the growth rate of chip computing power. Against this backdrop, in-memory computing, an emerging chip architecture, is gradually rising to provide next-generation AI chips with greater computing power and better energy efficiency. In-memory computing can be understood as embedding computing power in memory, performing two-dimensional and three-dimensional matrix multiplication / addition operations with a new computing architecture. This fundamentally eliminates unnecessary data movement latency and power consumption, improving AI computing efficiency by hundreds or thousands of times, reducing costs, and breaking down the "memory wall" and "power wall."

[0003] Deep Neural Networks (DNNs) are a machine learning method based on artificial neural network architectures. Artificial Neural Networks (ANNs) use layers of interconnected nodes (called neurons) to process and learn input data. Large Language Models (LLMs) are natural language processing models based on deep learning techniques. They are generative AI models built using Transformer networks. Transformers use self-attention mechanisms for encoding and decoding, enabling them to handle long sequences of data, giving them an advantage in processing natural language data. Furthermore, Transformers are frequently used to build various large language models, such as the GPT series of models.

[0004] The Self-Attention mechanism in the Transformer model uses the Softmax activation function as its normalization factor to calculate the attention weights for each input unit. These weights are summed to 1, and the final result is a weighted sum of the attention weights over the value. The Softmax activation function can be expressed in the model as follows:

[0005]

[0006] in Output the i-th data for the Softmax matrix. Input the i-th bit of the Softmax matrix. The difficulty in implementing the Softmax function in digital hardware mainly stems from the exponential and division operations it requires. Compared to other layers that only require simple addition and multiplication, such as convolutional layers, pooling layers, and fully connected layers, the exponential and division operations required by the Softmax layer are more complex. This presents corresponding difficulties for the digital hardware design of the Softmax function. Generally, implementing Softmax in digital circuits requires using multiple polynomial approximations, as shown in the following formula:

[0007]

[0008]

[0009] For general Softmax operations, all the aforementioned calculations require 32-bit or 16-bit floating-point computation. The multipliers and adders for these calculations require a large number of transistors, consuming significant power and area during operation. Furthermore, a single input requires multiple polynomials for computation; with multiple inputs processed simultaneously, the required power and area increase dramatically. Therefore, high parallelism is generally not feasible when deploying Softmax operations. Consequently, the digital hardware implementation of the Softmax activation function layer introduces substantial latency and power consumption, becoming a bottleneck limiting the speed of AI chips in building various large language models. Utility Model Content

[0010] The purpose of this invention is to provide a circuit, neural network processing device, chip, and electronic device for implementing the Softmax activation function in the analog domain, so as to at least solve the aforementioned technical problems. The various technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] This utility model provides a circuit for implementing the Softmax activation function in an analog domain. The circuit includes: a natural exponent simulation processing circuit, used to calculate and replicate the natural exponent from the input voltage value based on the analog circuit; and a second current mirror circuit, used to converge the N output currents based on the current mirror. The output current of the second current mirror circuit is accumulated. The result of summing the N natural exponents; an analog divider circuit formed by several switching transistors is used to perform the division operation, and the first input terminal of the analog divider circuit is connected to the output current of the natural exponent analog processing circuit. Its second input terminal is connected to the output current of the second current mirror circuit. The current output at the output terminal of the analog divider circuit The output current of the natural index simulation processing circuit With the output current of the second current mirror circuit The result of the division; the analog divider circuit includes a ninth MOS transistor M9, a tenth MOS transistor M10, an eleventh MOS transistor M11, and a twelfth MOS transistor M12; all of the ninth MOS transistors M9, M10, M11, and M12 are PMOS transistors; the source of the ninth MOS transistor M9 and the gate of the eleventh MOS transistor M11 are connected to the second current mirror circuit; the drains of the ninth MOS transistor M9 and the tenth MOS transistor M10 are grounded, their gates are connected to each other and connected to the drain of the eleventh MOS transistor M11; the sources of the eleventh MOS transistor M11 and the twelfth MOS transistor M12 are connected to the power supply VDD; the drain of the twelfth MOS transistor M12 outputs the current corresponding to the division calculation result of the analog divider. The gate of the twelfth MOS transistor M12 is connected to the source of the tenth MOS transistor M10, and is connected to the replicated output current. The drain output comparison current of the eleventh MOS transistor M11 .

[0013] Preferably, the natural index simulation processing circuit includes: N first MOS transistors M1, and N first current mirror circuits respectively connected to the N first MOS transistors M1, wherein the gate of each first MOS transistor is connected to an input voltage. VIN(i) Each first MOSFET M1 operates in the subthreshold operating region, so that the output current of each first MOSFET is... With the input voltage VIN(i) The relationship is a natural exponential function, 1≤i≤N, and each first current mirror circuit is used to replicate the output current of the first MOSFET electrically connected to it. And make each output current Convergence; the first MOS transistor M1 is a PMOS transistor, the gate of the first MOS transistor M1 is connected to the input voltage, the source of the first MOS transistor M1 is connected to the power supply terminal VDD, and the drain is connected to the first current mirror circuit.

[0014] Preferably, the first current mirror circuit of the natural index simulation processing circuit includes a second MOSFET M2, a third MOSFET M3, a fourth MOSFET M4, and a fifth MOSFET M5; the second MOSFET M2 and the third MOSFET M3 are NMOS transistors, and the fourth MOSFET M4 and the fifth MOSFET M5 are PMOS transistors; the gates of the second MOSFET M2 and the third MOSFET M3 are interconnected and connected to the drain of the first MOSFET M1; the sources of the second MOSFET M2 and the third MOSFET M3 are both grounded, and the drain of the second MOSFET M2 is connected to the drain of the first MOSFET; the sources of the fourth MOSFET M4 and the fifth MOSFET M5 are both connected to the power supply terminal VDD; the gates of the fourth MOSFET M4 and the fifth MOSFET M5 are interconnected and connected to the drain of the third MOSFET M3; the drain of the fourth MOSFET M4 is connected to the drain of the third MOSFET M3; and the drain of the fifth MOSFET M5 outputs the replicated output current. .

[0015] Preferably, the second current mirror circuit includes a sixth MOS transistor M6, a seventh MOS transistor M7, an eighth MOS transistor M8, and a current mirror MOS transistor group; the sixth MOS transistor M6 and the seventh MOS transistor M7 are both NMOS transistors, and the eighth MOS transistor M8 is a PMOS transistor; the drain of the sixth MOS transistor M6 is connected to the drains of N fifth MOS transistors M5, and the drain of the fifth MOS transistors M5 is also connected to the gates of the sixth MOS transistor M6 and the seventh MOS transistor M7; the sources of the sixth MOS transistor M6 and the seventh MOS transistor M7 are grounded and their gates are connected to each other; the drain of the seventh MOS transistor M7 is connected to the drain of the eighth MOS transistor M8; the gate of the eighth MOS transistor M8 is connected to the power supply terminal VDD; the current mirror MOS transistor group includes N parallel-connected accumulator PMOS transistors, the source of each accumulator PMOS transistor is connected to the power supply terminal VDD, and its gate is connected to the gate of the eighth MOS transistor M8 and the drain of the seventh MOS transistor M7; the drain of each accumulator PMOS transistor outputs current. .

[0016] Preferably, it further includes N input voltage driving and biasing circuits, which are respectively connected to the gates of the first MOS transistors M1 of the N natural exponent analog processing circuits, for generating the N input voltages. VIN(i) This is to enable the first MOS transistor M1 to operate in the subthreshold operating region, where 1≤i≤N, and i and N are both positive integers.

[0017] A neural network processing device includes a circuit that implements the Softmax activation function in an analog domain as described in any of the preceding claims.

[0018] Preferably, the neural network processing device further includes: a CIM (Central In-Memory) computing core for multiply-accumulate calculation and outputting simulated multiply-accumulate calculation results, a voltage readout amplifier, and an analog-to-digital converter (ADC). The CIM core, voltage readout amplifier, circuit implementing the Softmax activation function in the analog domain, and ADC are connected sequentially. The voltage readout amplifier amplifies and converts the voltage corresponding to the calculation result of the CIM core and sends it to the simulated Softmax activation function circuit. The simulated Softmax activation function circuit performs simulated Softmax activation function calculations, and the ADC converts the simulated calculation result of the Softmax activation function into a digital calculation result.

[0019] Preferably, the CIM in-memory computing core includes several in-memory computing units, each having an 8T2C structure, used for single-bit data storage and single-bit data computation.

[0020] A chip comprising a circuit that implements the Softmax activation function in the analog domain as described in any of the above claims.

[0021] An electronic device comprising the chip described above.

[0022] Implementing one of the above-described technical solutions of this utility model has the following advantages or beneficial effects:

[0023] In this application, the natural exponent of the Softmax activation function is calculated based on the subthreshold characteristics of the MOS transistor through a natural exponent analog processing circuit. The summation calculation in the Softmax activation function is performed through a second current mirror circuit, and the summation calculation of the Softmax activation function is performed through an analog divider circuit with several switching transistors. Thus, this embodiment realizes the Softmax activation function calculation in the analog domain. Compared with the implementation in the digital domain, it can significantly reduce the number of transistors required, thereby reducing the chip area and power consumption. At the same time, it can significantly improve the parallelism and computing speed of the Softmax activation function calculation. Under the same computing power, it can provide better energy efficiency and area efficiency, which is convenient for artificial intelligence chips to build various large language models. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this utility model, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this utility model. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0025] Figure 1This is a circuit block diagram of the simulated Softmax activation function circuit in the circuit of the simulated domain implementing the Softmax activation function in Embodiment 1 of this utility model;

[0026] Figure 2 This is a circuit diagram of the simulated Softmax activation function circuit in the simulated domain implementation circuit of Embodiment 1 of this utility model;

[0027] Figure 3 This is a circuit diagram of the second current mirror circuit and the analog divider circuit in the analog Softmax activation function circuit of the circuit implementing the Softmax activation function in the analog domain of Embodiment 1 of this utility model.

[0028] Figure 4 This is a block diagram of the in-memory computing architecture of the neural network processing device according to Embodiment 2 of this utility model;

[0029] Figure 5 This is a schematic diagram of a single in-memory computing core circuit in the in-memory computing architecture of the neural network processing device according to Embodiment 2 of this utility model;

[0030] In the diagram: 001, CIM in-memory computing core; 002, voltage readout amplifier; 003, analog Softmax activation function circuit; 004, analog-to-digital converter; 005, input voltage drive and bias circuit; 006, natural exponent analog processing circuit; 007, second current mirror circuit; 008, analog divider circuit. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this utility model clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be adopted to implement this utility model. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of this utility model disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of this utility model.

[0032] In the description of this utility model, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing this utility model and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this utility model according to the specific circumstances.

[0033] To illustrate the technical solution described in this utility model, specific embodiments are described below, showing only the parts related to the embodiments of this utility model.

[0034] Example 1:

[0035] This embodiment provides a circuit for implementing the Softmax activation function in an analog domain, such as... Figures 1-3 As shown, the circuit for implementing the Softmax activation function includes a natural exponent simulation processing circuit 006, a second current mirror circuit 007, and an analog divider circuit 008. The natural exponent simulation processing circuit 006 is used to calculate and replicate the natural exponent from the input voltage value using an analog circuit. The natural exponent simulation processing circuit includes N first MOS transistors M1 and N first current mirror circuits connected to the N first MOS transistors M1 respectively. The gate of each first MOS transistor is connected to an input voltage VIN(i). Each first MOS transistor M1 operates in the subthreshold operating region, such that the output current of each first MOS transistor... The current mirror circuit exhibits a natural exponential relationship with the input voltage VIN(i), where 1 ≤ i ≤ N. Each first current mirror circuit is used to replicate the output current of the first MOS transistor electrically connected to it. And make each output current Convergence. Each first MOSFET M1 operates in the subthreshold region, generating N subthreshold currents. Based on the IV characteristics of the subthreshold region, the natural exponent of the Softmax activation function is calculated, i.e., simulating... The IV characteristic formula for the subthreshold working region is as follows: , W / L represents the output current of the MOSFET, where W / L is the ratio of the MOSFET's channel width to its length. This represents the characteristic current of the MOSFET. This indicates the gate voltage of the MOSFET. This represents the threshold voltage of the MOSFET, where n is the subthreshold correction factor. The thermal voltage is approximately 26 mV at room temperature. The i subthreshold currents correspond to the molecule of the Softmax activation function, yielding the calculated i natural exponents. After simplifying the above formula, the subthreshold current formula for the first MOSFET M1 is obtained. Can be converted into formula , Proportional to the ratio of the channel width to the length of the MOSFET, W / L Parameters introduced for computational convenience. The subthreshold operating region is a state in which the gate voltage Vgs of a MOSFET is below the threshold voltage and no conductive channel appears. In this state, a small current flows through the MOSFET; this current is called the subthreshold current. Although the subthreshold current is small, it is well controlled by the gate voltage. Therefore, MOSFETs in the subthreshold state are advantageous in low-voltage, low-power applications. Simulating the Softmax activation function in this way can significantly reduce computational power consumption. Each first current mirror circuit is used to replicate the output current of the first MOSFET electrically connected to it. And make each output current Converging, various output currents After aggregation, it is easier to control the output current of all the first MOSFETs. Summation is performed for the summation calculation in the Softmax activation function. The circuit implementing the Softmax activation function in the analog domain also includes a second current mirror circuit 007, which combines the N output currents from the first current mirror circuit. The output current of the second current mirror circuit is accumulated. The result of summing N natural indices, specifically This corresponds to the denominator of the Softmax activation function. The circuit implementing the Softmax activation function in the analog domain also includes an analog divider circuit 008 formed by several switching transistors. This divider performs the division operation based on a current-domain divider. The first input terminal of the analog divider circuit is connected to the output current of the natural exponent analog processing circuit. Its second input terminal is connected to the output current of the second current mirror circuit. The current output at the output terminal of the analog divider circuit The output current of the natural index simulation processing circuit With the output current of the second current mirror circuit The result of the division corresponds to the calculation result of the Softmax activation function. This embodiment calculates the natural exponent of the Softmax activation function based on the subthreshold characteristics of MOS transistors using a natural exponent analog processing circuit. It performs the summation calculation in the Softmax activation function using a second current mirror circuit and an analog divider circuit with several switching transistors. Therefore, this embodiment implements the Softmax activation function calculation in the analog domain. Compared to the digital domain implementation, this significantly reduces the number of transistors required, thereby reducing chip area and power consumption. Simultaneously, it greatly improves the parallelism and processing speed of the Softmax activation function calculation. With the same computing power, it provides better energy efficiency and area efficiency, facilitating the construction of various large language models in artificial intelligence chips.

[0036] As an optional implementation method, such as Figure 2 As shown, the first MOSFET M1 is a PMOS transistor. The gate of the first MOSFET M1 is connected to the input voltage, thus controlling the input voltage to keep the first MOSFET M1 in a subthreshold operating state. The source of the first MOSFET M1 is connected to the power supply terminal VDD, and the drain is connected to a first current mirror circuit, which is also connected to a second current mirror circuit. The second current mirror circuit is connected to the natural exponent analog processing circuit and the analog divider, and the analog divider circuit is also connected to the natural exponent analog processing circuit. This enables signal transmission between the natural exponent analog processing circuit, the second current mirror circuit, and the analog divider.

[0037] As an optional implementation method, such as Figure 2 As shown, the first current mirror circuit of the natural exponent simulation processing circuit includes a second MOSFET M2, a third MOSFET M3, a fourth MOSFET M4, and a fifth MOSFET M5; the second MOSFET M2 and the third MOSFET M3 are NMOS transistors, and the fourth MOSFET M4 and the fifth MOSFET M5 are PMOS transistors. Through the first current mirror circuit, the subthreshold currents of the N first MOSFETs M1 are copied to the fifth MOSFET M5, thereby facilitating the acquisition and aggregation of the subthreshold currents corresponding to each first MOSFET M1, and enabling the calculation of the natural exponent of the Softmax activation function. The gates of the second MOSFET M2 and the third MOSFET M3 are interconnected and connected to the drain of the first MOSFET M1. The sources of both the second MOSFET M2 and the third MOSFET M3 are grounded, and the drain of the second MOSFET M2 is connected to the drain of the first MOSFET. The sources of the fourth MOSFET M4 and the fifth MOSFET M5 are both connected to the power supply terminal VDD. The gates of the fourth MOSFET M4 and the fifth MOSFET M5 are interconnected and connected to the drain of the third MOSFET M3. The drain of the fourth MOSFET M4 is connected to the drain of the third MOSFET M3. The drain of the fifth MOSFET M5 outputs the replicated current. .

[0038] As an optional implementation method, such as Figure 3 As shown, the second current mirror circuit includes a sixth MOSFET M6, a seventh MOSFET M7, an eighth MOSFET M8, and a current mirror MOSFET group; the sixth MOSFET M6 and the seventh MOSFET M7 are both NMOS transistors, and the eighth MOSFET M8 is a PMOS transistor. The drain of the sixth MOSFET M6 is connected to the drains of N fifth MOSFETs M5, and the drain of the fifth MOSFETs M5 is also connected to the gates of the sixth MOSFET M6 and the seventh MOSFET M7. The sources of the sixth MOSFET M6 and the seventh MOSFET M7 are grounded, and their gates are connected to each other. The drain of the seventh MOSFET M7 is connected to the drain of the eighth MOSFET M8, and the gate of the eighth MOSFET M8 is connected to the power supply terminal VDD. The second current mirror circuit facilitates the measurement of the output current corresponding to each first MOSFET M1. This allows for the aggregation of the output currents corresponding to all first MOSFETs M1. Perform summation. This circuit has a simple structure, low power consumption, and high accuracy in current accumulation.

[0039] As an optional implementation method, such as Figure 4 As shown, the current mirror MOS transistor group includes N accumulator PMOS transistors connected in parallel ( Figure 3 In the PMOS transistors M(1)~M(N)), the source of each accumulator PMOS transistor is connected to the power supply terminal VDD, and the gate is connected to the gate of the eighth MOS transistor M8 and the drain of the seventh MOS transistor M7. The drain of each accumulator PMOS transistor outputs the same current as the first MOS transistor. Specifically, Adding the i currents together gives the result. ,thereby By using a current mirror MOSFET group, it is easy to ensure that the output current of each first MOSFET is equal. In turn, an electric current is obtained. This simplifies the circuit.

[0040] As an optional implementation method, such as Figures 2-3As shown, the analog divider circuit includes a ninth MOSFET M9, a tenth MOSFET M10, an eleventh MOSFET M11, and a twelfth MOSFET M12; all four are PMOS transistors. The analog divider circuit in this embodiment has a simple implementation principle, requires fewer MOSFETs, and facilitates rapid division calculations using the Softmax activation function. The source of the ninth MOSFET M9 and the gate of the eleventh MOSFET M11 are connected to the second current mirror circuit. The drains of the ninth MOSFET M9 and the tenth MOSFET M10 are grounded, their gates are interconnected and connected to the drain of the eleventh MOSFET M11. The sources of the eleventh MOSFET M11 and the twelfth MOSFET M12 are connected to the power supply VDD, and the drain of the twelfth MOSFET M12 outputs the division current corresponding to the division calculation result of the analog divider. The gate of the twelfth MOSFET M12 is connected to the source of the tenth MOSFET M10, and is connected to the replicated output current. The drain output comparator current of the eleventh MOSFET M11 Specifically, By measuring and comparing current The value of this value yields the simulation result of the Softmax activation function, at which point the comparison current I in the simulated divider circuit is obtained. SCALE It can be set to 100nA. In this embodiment, the power consumption of the simulated Softmax activation function circuit is extremely low, requiring only... nW arrive uW The power consumption is at a similar level, while the second current mirror circuit and the analog divider circuit require only a small number of transistors to implement. Compared to the circuit implementation of the digital Softmax function, this analog Softmax activation function circuit achieves at least a two-exponential reduction in power consumption and chip area.

[0041] As an optional implementation method, such as Figures 1-2 As shown, the circuit implementing the Softmax activation function also includes N input voltage driving and biasing circuits 005. These N input voltage driving and biasing circuits are respectively connected to the gates of the first MOS transistors M1 of the N natural exponent analog processing circuits, and are used to generate N input voltages. VIN(i) That is, the first MOSFET M1 V GS Voltage( V GS = VIN(i)-VDDThe input voltage drives and biases the first MOSFET M1 to operate in the subthreshold region, where 1 ≤ i ≤ N, and i and N are both positive integers. The input voltage drive and bias circuit 005 controls the gate voltage of the first MOSFET M1 according to the computational needs of the Softmax activation function, allowing the first MOSFET M1 to operate in the subthreshold region or be cut off, thereby achieving rapid calculation of the Softmax activation function.

[0042] The embodiment is merely a special case and does not indicate that this utility model is implemented in such a way.

[0043] Example 2:

[0044] This second embodiment provides a neural network processing device, including the circuit for implementing the Softmax activation function in the analog domain as described in the first embodiment. By integrating the circuit for implementing the Softmax activation function in the analog domain as described in the first embodiment, Softmax function calculation can be performed directly, resulting in faster calculation speed and lower computational resource consumption. The power consumption and area of ​​the circuit required for calculation are significantly reduced, avoiding the transistor area and power consumption of logic circuits such as floating-point multipliers and floating-point adders. It realizes the implementation of Softmax with a less transistor-intensive and lower-power analog method instead of the traditional pure digital approximation algorithm, achieving a circuit design with high area efficiency and high energy efficiency. This can improve the parallelism of Softmax operations, enhance the data processing capabilities of Transformer models and large language models, and improve the user experience. The circuit for implementing the Softmax function calculation provided by this utility model can be used in an NPU (Neural Processing Unit) to achieve brain-like functions, better supporting various fields requiring artificial intelligence. As those skilled in the art know, power consumption is a crucial limiting factor for artificial intelligence. Therefore, this embodiment can achieve low-power and low-area support for AI-related products, improving the user experience of AI products.

[0045] As an optional implementation method, such as Figure 4As shown, the system also includes a CIM (Computing in Memory) in-memory computing core 001, a voltage sense amplifier 002, and an analog-to-digital converter 004 for multiply-accumulate calculations and outputting simulated multiply-accumulate results, forming an in-memory computing architecture. The CIM in-memory computing core 001, the voltage sense amplifier 002, the circuit 003 implementing the Softmax activation function in the analog domain, and the analog-to-digital converter 004 are connected sequentially. This embodiment uses a hybrid analog-digital in-memory computing architecture (CIMA) to calculate the Softmax activation function. Existing digital in-memory computing architectures convert the output of floating-point numbers to 16-bit floating-point data format through two data format conversion (floating-point numbers of different bit widths need to be converted to each other for corresponding calculations) and quantization algorithm modules, and then send it to the digital domain Softmax activation function module for calculation to generate the output signal of the neural network layer. This consumes a significant amount of chip area and power consumption. The hybrid analog-digital in-memory computing architecture of this embodiment can replace the digital domain implementation method, significantly reducing the power consumption and chip area required for the in-memory computing architecture while improving the parallel computing speed of Softmax. The CIM in-memory computing core is used for feature value and weight value input (both are input through word lines WLBUS and bit lines BL BUS; the word line is responsible for activating a single in-memory computing unit and is used to select which row of data to read; the bit line is the data line, and the data is read from the bit line BL BUS) and analog domain multiply-accumulate (MAC, including matrix multiplication, convolution, and other tensor operations commonly used in deep learning, used to implement the specific operation methods required for neural networks). In the CIM in-memory computing core, Vp represents positive voltage and Vn represents negative voltage. After being detected by the voltage sense amplifier VSA (Voltage Sense Amplifier), the voltage sense amplifier is used to amplify and convert the voltage corresponding to the calculation result (analog signal) of the CIM in-memory computing core and send it to the analog softmax activation function circuit for analog softmax activation function operation. The analog softmax activation function circuit performs analog softmax activation function operation based on the subthreshold operating region of the MOS transistor. An analog-to-digital converter (ADC) converts the analog calculation result of the Softmax activation function into a digital calculation result. Preferably, the ADC has a 12-14 bit accuracy, which is comparable to the accuracy of the 16-bit floating-point Softmax activation function in the digital domain. This digital calculation result is then input into subsequent neural network layers to achieve the corresponding function of the Transformer model. Compared to the digital domain implementation, this embodiment significantly reduces the number of transistors required, thereby reducing chip area and power consumption, while greatly improving its parallelism and processing speed.Analog domain multiplication and accumulation calculations are implemented based on a charge distribution mechanism, which does not generate additional power consumption requirements. This architecture consists of only a small number of transistors and computational unit capacitors, whose computational capacitors can be stacked together with the transistors of the SRAM memory without generating additional chip area consumption. Compared to digital architectures, the hybrid analog-digital architecture of this embodiment provides better energy efficiency and areal efficiency while offering the same computing power.

[0046] As an optional implementation, the CIM in-memory computing core includes several in-memory computing units, which are the smallest working units of the CIM in-memory computing core. Preferably, multiple in-memory computing unit arrays are formed, such as 128*128, 256*256, or M*N arrays. Figure 5 As shown, the in-memory computing unit is an 8T2C (8 Transistor 2 Capacitance) structure, meaning that the in-memory computing unit in this embodiment includes 8 transistors (M1'~M8') and 2 capacitors (C1', C2'). The in-memory computing unit is used for single-bit data storage and single-bit data computation. Transistors M5' and M6' are connected via word line WL. Transistor M5' is also connected to bit line BL for data read / write, and transistor M6' is also connected to bit line BLB for data read / write. Transistors M5' and M6' are also connected to transistors M7' and M8' respectively. Transistor M7' is connected to transistor M8' via computation word line CWL. Transistor M7' is also connected to the first plate of capacitor C2' via computation bit line CBL (the second plate of capacitor C2' is connected to a positive voltage Vp). Transistor M8' is also connected to the first plate of capacitor C1' via computation bit line CBLB (the second plate of capacitor C1' is connected to a negative voltage Vp). N Of course, in this embodiment, the number of transistors and capacitors can be adjusted to other combinations, such as 2T2C or 4T2C, to achieve faster computing speed or lower computing power consumption, depending on the actual computing scenario.

[0047] Example 3:

[0048] This embodiment provides a chip, including the circuit for implementing the Softmax activation function in the analog domain as described in embodiment one. The chip provided by this invention can also be an integrated circuit including the neural network processing device described in embodiment two. By integrating the circuit for implementing the Softmax activation function in the analog domain as described in embodiment one, the traditional pure digital approximation algorithm is replaced with an analog method using fewer transistors and lower power consumption to implement the Softmax activation function. This achieves a circuit design with high area efficiency and high energy efficiency, reducing chip area, cost, and power consumption. Furthermore, more diverse and complex functions can be integrated on the chip, making it more versatile and suitable for use in more complex working scenarios. The chip provided by this invention allows each module to be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computing device, or stored in software in the memory of the computing device, so that the processor can call and execute the operations corresponding to each module, effectively reducing chip area, cost, and power consumption.

[0049] Example 4:

[0050] This fourth embodiment provides an electronic device, including the chip described in embodiment three. By employing the chip from embodiment three, a simulated method with fewer transistors and lower power consumption is used to replace the traditional pure digital approximation algorithm for Softmax activation function calculation, achieving high surface efficiency and high energy efficiency in the circuit design, thus facilitating the reduction of device power consumption and cost. It provides a range of devices with high requirements for low power consumption and high energy efficiency, applicable to computing centers, computing devices, edge computing, autonomous driving, AR, VR, LiDAR, as well as smartphones, tablets, wearable electronic devices, smart home electronic products, industrial or medical or battery-powered devices.

[0051] The above description is merely a preferred embodiment of the present utility model. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present utility model. Furthermore, under the teachings of the present utility model, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present utility model. Therefore, the present utility model is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present utility model.

Claims

1. A circuit for implementing the Softmax activation function in an analog domain, characterized in that, The circuit used to implement the Softmax activation function includes: Natural exponent analog processing circuit, used to calculate the natural exponent based on the value of the input voltage from the analog circuit and to replicate the result of the natural exponent calculation; a second current mirror circuit to sum the N output currents based on a current mirror to accumulate the output current of the second current mirror circuit the result of the summation of the N natural exponents Analog divider circuit formed by several switching transistors for implementing a division operation, the first input of the analog divider circuit being connected to the output current of the natural exponential analog processing circuit , the second input of the analog divider circuit being connected to the output current of the second current mirror circuit , the output of the analog divider circuit outputting a current which is the result of dividing the output current of the natural exponential analog processing circuit by the output current of the second current mirror circuit . The analog divider circuit includes a ninth MOS transistor M9, a tenth MOS transistor M10, an eleventh MOS transistor M11, and a twelfth MOS transistor M12; all of the ninth MOS transistor M9, the tenth MOS transistor M10, the eleventh MOS transistor M11, and the twelfth MOS transistor M12 are PMOS transistors. The source of the ninth MOSFET M9 and the gate of the eleventh MOSFET M11 are both connected to the second current mirror circuit. The drains of the ninth MOSFET M9 and the tenth MOSFET M10 are grounded, and their gates are connected to each other and to the drain of the eleventh MOSFET M11. The sources of the eleventh MOSFET M11 and the twelfth MOSFET M12 are connected to the power supply VDD. The drain of the twelfth MOSFET M12 outputs the current corresponding to the division calculation result of the analog divider. The gate of the twelfth MOS transistor M12 is connected to the source of the tenth MOS transistor M10, and is connected to the replicated output current. The drain output comparison current of the eleventh MOS transistor M11 .

2. The circuit for implementing the Softmax activation function in the analog domain according to claim 1, characterized in that, The natural index simulation processing circuit includes: N first MOS transistors M1, and N first current mirror circuits respectively connected to the N first MOS transistors M1, wherein the gate of each first MOS transistor is connected to an input voltage. VIN(i) Each first MOSFET M1 operates in the subthreshold operating region, so that the output current of each first MOSFET is... With the input voltage VIN(i) The relationship is a natural exponential function, 1≤i≤N, and each first current mirror circuit is used to replicate the output current of the first MOSFET electrically connected to it. And make each output current Convergence; the first MOS transistor M1 is a PMOS transistor, the gate of the first MOS transistor M1 is connected to the input voltage, the source of the first MOS transistor M1 is connected to the power supply terminal VDD, and the drain is connected to the first current mirror circuit.

3. The circuit for implementing the Softmax activation function in the analog domain according to claim 2, characterized in that, The first current mirror circuit of the natural index simulation processing circuit includes a second MOSFET M2, a third MOSFET M3, a fourth MOSFET M4, and a fifth MOSFET M5; The second MOS transistor M2 and the third MOS transistor M3 are NMOS transistors, and the fourth MOS transistor M4 and the fifth MOS transistor M5 are PMOS transistors; The gates of the second MOSFET M2 and the third MOSFET M3 are interconnected and connected to the drain of the first MOSFET M1. The sources of the second MOSFET M2 and the third MOSFET M3 are both grounded, and the drain of the second MOSFET M2 is connected to the drain of the first MOSFET. The sources of the fourth MOSFET M4 and the fifth MOSFET M5 are both connected to the power supply terminal VDD. The gates of the fourth MOSFET M4 and the fifth MOSFET M5 are interconnected and connected to the drain of the third MOSFET M3. The drain of the fourth MOSFET M4 is connected to the drain of the third MOSFET M3. The drain of the fifth MOSFET M5 outputs the replicated output current. .

4. The circuit for implementing the Softmax activation function in the analog domain according to claim 3, characterized in that, The second current mirror circuit includes a sixth MOSFET M6, a seventh MOSFET M7, an eighth MOSFET M8, and a current mirror MOSFET group; the sixth MOSFET M6 and the seventh MOSFET M7 are both NMOS transistors, and the eighth MOSFET M8 is a PMOS transistor; The drain of the sixth MOS transistor M6 is connected to the drains of N fifth MOS transistors M5, and the drain of the fifth MOS transistor M5 is also connected to the gate of the sixth MOS transistor M6 and the seventh MOS transistor M7; the sources of the sixth MOS transistor M6 and the seventh MOS transistor M7 are grounded and their gates are connected to each other; the drain of the seventh MOS transistor M7 is connected to the drain of the eighth MOS transistor M8; and the gate of the eighth MOS transistor M8 is connected to the power supply terminal VDD. The current mirror MOS transistor group includes N parallel-connected accumulator PMOS transistors. The source of each accumulator PMOS transistor is connected to the power supply terminal VDD, and the gate is connected to the gate of the eighth MOS transistor M8 and the drain of the seventh MOS transistor M7. The drain of each accumulator PMOS transistor outputs current. .

5. The circuit for implementing the Softmax activation function in the analog domain according to any one of claims 1-4, characterized in that, It also includes N input voltage driving and biasing circuits, which are respectively connected to the gates of the first MOS transistors M1 of the N natural exponent analog processing circuits to generate the N input voltages. VIN(i) This is to enable the first MOS transistor M1 to operate in the subthreshold operating region, where 1≤i≤N, and i and N are both positive integers.

6. A neural network processing device, characterized in that, The circuit includes any one of the analog domains of claims 1-5 that implements the Softmax activation function.

7. A neural network processing device according to claim 6, characterized in that, Also includes: A CIM (Central In-Memory) computing core, a voltage readout amplifier, and an analog-to-digital converter are used for multiply-accumulate calculations and outputting simulated multiply-accumulate calculation results. The CIM core, voltage readout amplifier, circuit implementing the Softmax activation function in the analog domain, and analog-to-digital converter are connected in sequence. The voltage readout amplifier amplifies and converts the voltage corresponding to the calculation result of the CIM core and sends it to the simulated Softmax activation function circuit. The simulated Softmax activation function circuit performs simulated Softmax activation function operations. The analog-to-digital converter converts the simulated calculation result of the Softmax activation function into a digital calculation result.

8. A neural network processing device according to claim 7, characterized in that, The CIM in-memory computing core includes several in-memory computing units, each with an 8T2C structure, used for single-bit data storage and single-bit data computation.

9. A chip, characterized in that, The circuit includes any one of the analog domains of claims 1-5 that implements the Softmax activation function.

10. An electronic device, characterized in that, Includes the chip described in claim 9.

Images