Tranformer model acceleration in-memory calculation circuit and method based on ferroelectric transistor

Abstract

The invention relates to a ferroelectric transistor-based Transform model acceleration in-memory calculation circuit and method, belongs to the technical field of in-memory calculation, and solves the problems of large power consumption of a core area, long delay and low storage density of a storage calculation array of a traditional Transform in-memory calculation accelerator SIMD (Single Instruction Multiple Data). The circuit comprises a storage array used for executing score matrix calculation and scaling operation in parallel, and each storage unit comprises a ferroelectric transistor; one end of each word line is connected with the word line decoding circuit and the multiplexer, and the other end of each word line is connected with the grid electrode of the corresponding row ferroelectric transistor; one end of each bit line and one end of each source line are connected to the bit line and source line decoding circuit and the multiplexer, and the other end of each bit line and one end of each source line are connected to drain electrodes and source electrodes of the corresponding columns of ferroelectric transistors; the word line driving circuit and the read-write driving circuit are used for enhancing the signal driving capability, and the scaling voltage generating circuit is used for generating scaling voltage and applying the scaling voltage to a plurality of source lines. According to the invention, the ferroelectric transistor is utilized to realize the # imgabs0 # operator, the area power consumption overhead and delay of the SIMD core are saved, and the storage density of the storage array is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of in-memory computing, and in particular to a Transformer model acceleration in-memory computing circuit and method based on ferroelectric transistors. Background Art

[0002] The Transformer model is a deep learning model based on the self-attention mechanism and has milestone significance in the field of natural language processing. Different from traditional RNN and CNN models, the Transformer can process input sequences in parallel, has a global receptive field, is very suitable for extracting feature information of long text sequences, and has been widely used in natural language processing fields such as machine translation and text generation, leading the innovation of deep learning. In September 2023, the large language model DeepSeek based on the Transformer architecture emerged, promoting the technological empowerment and intelligent transformation of multiple industries such as the Internet, finance, healthcare, education, translation, and customer service, bringing efficiency improvement and emerging application scenarios to all walks of life, and further promoting China's global competitiveness in the AI industry.

[0003] The huge number of parameters of the Transformer model poses higher requirements for the computing power and energy efficiency of hardware. At present, various in-memory computing (CIM) accelerators for the Transformer have been developed at home and abroad, including digital in-memory computing accelerators based on SRAM, digital-analog hybrid in-memory computing accelerators based on SRAM and RRAM, and accelerators based on PE arrays. In current Transformer in-memory computing accelerators, there are problems such as large hardware area power consumption overhead of peripheral computing circuits, long computing latency, and low storage density and computing power density of the memory-computation array.

[0004] Reference Figure 1 , in the traditional SRAM-based Transformer in-memory computing array, the memory-computation unit consists of a 6T SRAM and a 4T NOR gate, and the inverted signal of the query vector Q is applied to the BLB end of the bit line as the input signal, and after being gated by the MUX the signal is sent to the input end of the NOR gate, and the key-value transposed vector K T is stored in the SRAM cell and inverted to obtain the signal as the weight, and performs a NOR operation with the input signal to complete the multiplication of the input signal and the weight: thus realizing the Q·K T operator calculation. The scale of the SRAM-based Transformer in-memory computing array is M×N, and the score matrix is output at the end of the array after parallel calculation. Figure 1The shown SRAM-based memory cell cannot complete the score matrix scaling operation within the array. It is necessary to use the SIMD core to perform division calculations element by element to complete the scaling operation, resulting in a large area overhead and calculation latency. In addition, the memory and computing unit of this structure has 10 transistors, with a large area overhead, which limits the improvement of the storage density and computing power density of the memory and computing array.

[0005] Reference Figure 2 , in terms of hardware area and power consumption, the traditional Transformer in-memory computing accelerator needs to use a high-width and high-parallel SIMD core outside the array to perform the score matrix Q·K T Execute the division by to complete the scaling operation. The processing unit for performing high-parallel division calculations in the SIMD core has a large hardware area and power consumption overhead. The SIMD (Single Instruction, Multiple Data) core is a computing unit based on a parallel computing architecture. Its core idea is that one instruction acts on multiple groups of data elements simultaneously, performs multiple groups of repetitive calculations in parallel, and outputs multiple groups of results. As shown in the A module of Figure 2 , a SIMD core performs the same division operation on N elements of vectors V1 and V2 in parallel, that is, V3 = V1÷V2, and finally outputs a V3 vector with a length of N elements as the output result. The overall architecture of a typical SIMD core is as shown in the B module of Figure 2 . The workflow of the SIMD core for performing parallel computing tasks is as follows: First, under the control of the programmable input processor, operands with the same size as the total processing unit array are serially input to the SIPO, and then parallelly output to the internal storage unit of the processing unit array. A SIMD controller broadcasts the same instruction to multiple processing units, and all processing units perform the same calculation operation on different operands under the same instruction. Inside the processing unit, the operands are stored in the storage unit of the processing unit. The multiplexer selects the operands and sends them to the arithmetic unit to perform the calculation operation. After the intermediate calculation results are sent to the register bank for temporary storage, they can either be sent to the arithmetic unit through the multiplexer to perform the next calculation or be stored in the storage unit as the final result. Finally, under the control of the programmable output processor, the calculation results of all processing units are parallelly output to the PISO, and then serially output outside the SIMD core. Although the SIMD core supports parallel computing, it has many computing and storage hardware modules with complex structures, large circuit area and power consumption overheads, a long data path, and large data transfer latency and power consumption overheads.

[0006] In terms of computing latency and throughput, on the one hand, when the SIMD core performs the score matrix scaling operation, the data transfer path includes the external path from the memory - computing array to the SIMD core and the internal path of data communication between various circuit modules in the SIMD core. An overly long data transfer path will lead to a long computing latency and reduce the throughput. On the other hand, when the size and quantity of the score matrix are large, the total size of the score matrix will exceed the size of the total processing unit array of the SIMD core. Performing the scaling operation on the score matrix will exceed the computing parallelism of the SIMD core. Therefore, the score matrix needs to be cut and divided and queued to be sent to the SIMD core for computing, resulting in data congestion and conflicts, and there are problems of stalls or blocks caused by computing resource competition, further increasing the latency of the SIMD core to perform the score matrix scaling calculation. The SIMD core will become a bottleneck for improving the performance of the in - memory computing accelerator of the Transformer. Therefore, when the traditional in - memory computing circuit of the Transformer performs the score matrix calculation of the attention layer, there is a lack of a scheme to implement score matrix scaling inside the array, with a long data transfer path, high computing latency, high area overhead, and high power consumption overhead.

[0007] In terms of computing accuracy, some in - memory computing accelerators do not support score matrix scaling calculation. The scaled score matrix output by the memory - computing array can be sent to the softmax activation function module for further calculation. The score matrix with a large amplitude is likely to enter the non - linear region of the softmax activation function, reducing the correct rate of the calculation results of the attention layer during the inference process of the Transformer model. Summary of the Invention

[0008] In view of the above analysis, embodiments of the present invention aim to provide an in - memory computing circuit and method for accelerating the Transformer model based on ferroelectric transistors to solve the problems of long data transfer path, high computing latency, high circuit area overhead, and high power consumption overhead caused by performing the score matrix scaling operation in the SIMD core outside the array in the existing computing methods.

[0009] On the one hand, an embodiment of the present invention provides an in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors, including: a memory computing array, including multiple rows and columns of memory computing units, for parallelly executing score matrix calculation and scaling operations, wherein each memory computing unit includes a ferroelectric transistor; a word line decoding circuit and a multiplexer, for connecting to one end of multiple word lines, and the other ends of the multiple word lines are connected to the gates of the corresponding row of ferroelectric transistors; a bit line source line decoding circuit and a multiplexer, for connecting to one end of multiple bit lines and multiple source lines, the other ends of the multiple bit lines are connected to the drains of the corresponding column of ferroelectric transistors, and the other ends of the multiple source lines are connected to the sources of the corresponding column of ferroelectric transistors; and a scaling voltage generation circuit, for generating a scaling voltage, and applying the scaling voltage to the multiple source lines via the read / write driving circuit and the bit line source line decoding circuit and the multiplexer.

[0010] The beneficial effects of the above technical solutions are as follows: The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors in the embodiment of the present invention utilizes the characteristics of the four-terminal device of the ferroelectric transistor to complete the calculation inside the memory array, saving the area and power consumption overhead of the peripheral SIMD core. The voltage output by the additionally introduced scaling voltage generation circuit can be multiplexed by each bit line of the memory computing array, which is much smaller than the area and power consumption overhead of the internal division circuit of the high parallel processing unit adopted by the traditional SIMD core scaling method.

[0011] Based on the further improvement of the above circuit, the ferroelectric transistor (FeFET) includes a field effect transistor MOSFET and a ferroelectric capacitor located above the MOSFET, wherein the internal gate of the MOSFET is connected to the lower plate of the ferroelectric capacitor via a metal contact.

[0012] Based on the further improvement of the above circuit, the MOSFET includes: source / drain, located in the semiconductor substrate, and its top surface is flush with the top surface of the semiconductor substrate; an intermediate dielectric layer, located between the source and the drain above the semiconductor substrate; an internal gate, located above the intermediate dielectric layer; and the ferroelectric capacitor includes: a lower plate, located above the MOSFET and connected to the internal gate of the MOSFET through a metal contact; a ferroelectric layer, located above the lower plate, and its material is a ferroelectric oxide; and an upper plate, serving as the external gate of the ferroelectric transistor, located above the ferroelectric layer.

[0013] Based on the further improvement of the above circuit, the multiple rows and columns of memory computing units of the memory computing array are used to store the transposed matrix K of the key-value matrix T , wherein each memory computing unit is used to store the transposed matrix K of the key-value matrixT The corresponding elements in it; and the multiple word lines, through the word line decoding circuit and the multiplexer, transmit the elements in the query matrix Q to the external gate of the ferroelectric transistor in each memory and computing unit in the form of voltage signals in a periodic order through the word line WL.

[0014] Based on the further improvement of the above circuit, the in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors further includes a word line driving circuit and a read / write driving circuit. Among them, the word line driving circuit is connected to the word line decoding circuit and the multiplexer, and is used to amplify the word line signal and increase the driving ability of the word line signal; the word line decoding circuit and the multiplexer are used to select a word line with a specific address from the multiple word lines to perform a computing and storage operation on the memory and computing unit at the selected specific address; the read / write driving circuit is connected to the scaled voltage generation circuit and is used to amplify the read signal and the write signal to increase the driving ability of the read / write signal. Among them, the read signal is the scaled score matrix The bit line source line decoding circuit and the multiplexer are used to select a bit line and a source line with a specific address from the multiple bit lines and the multiple source lines to perform a computing and storage operation on the memory and computing unit at the selected specific address.

[0015] Based on the further improvement of the above circuit, the in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors further includes an analog-to-digital converter readout circuit, a digital shift accumulator, and an address timing control circuit. Among them, the analog-to-digital converter readout circuit is used to convert the analog read signal into a digital read signal; the digital shift accumulator is used to shift and weight the digital read signal to output a multi-bit signal; the address timing control circuit is connected to other peripheral circuits and controls the peripheral circuits to perform storage and computing operations on the memory and computing units at specific positions of the memory and computing array at a specified timing.

[0016] The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors further includes a softmax activation function, which is used to convert each element Xi in the scaled score matrix into the corresponding probability Yi, where 0 < Yi < 1. In the input-output relationship of the softmax activation function, the magnitude of each output result Yi is related to the magnitude of the corresponding input element Xi itself, and the sum of all output results Yi is equal to 1. Among them, when the scaled score matrix is substituted as the input element into the softmax activation function, all input elements are within the linear region range, and the output result of the softmax activation function changes linearly with the input element to avoid the output result of the softmax activation function entering the non-linear region.

[0017] On the other hand, an embodiment of the present invention provides a method for accelerating in-memory computing based on a Transformer model of a ferroelectric transistor, comprising a writing phase and a computing phase, wherein in the writing phase, the transposed matrix K of the key-value matrix is ​​converted into T All elements in are stored in a storage array and stored in the storage array in the form of threshold voltages, wherein the storage array includes M rows×N columns of storage units; in the calculation stage: the scaling voltage generating circuit generates a scaling voltage Then, the scaled voltage is applied in parallel to multiple source lines SL through the read / write drive circuit, the bit line source line decoding circuit and the multiplexer in sequence; the row vectors in the query matrix Q are applied to multiple word lines WL in each cycle of X cycles through the word line decoding circuit and the multiplexer to control the conduction state of the ferroelectric transistor; and the row vectors in the query matrix Q and the transposed matrix K are output through the bit line BL. T The scaled result of the column vector product of , and after X cycles, the scaled score matrix is ​​obtained

[0018] Based on the further improvement of the above method, the scaled score matrix is ​​transformed into Each element Xi in is converted into the corresponding probability Yi, where 0 <Yi<1,在softmax激活函数的输入输出关系中,每个输出结果Yi的大小与对应输入元素Xi本身的大小相关,且所有的输出结果Yi相加等于1,其中,当缩放后的得分矩阵作为输入元素代入softmax激活函数时,所有输入元素均处于线性区范围内,softmax激活函数的输出结果随所述输入元素线性变化,以避免softmax激活函数的输出结果进入非线性区。

[0019] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0020] 1. The Transformer model based on ferroelectric transistors in the embodiment of the present invention accelerates the in-memory computing circuit, which uses the characteristics of the four-terminal ferroelectric transistor device to complete the Calculation saves the area and power consumption of the peripheral SIMD core. The Vs voltage output by the additional scaling voltage generation circuit can be reused by each bit line of the storage array, which is much smaller than the area and power consumption of the internal division circuit of the high parallel SIMD core processing unit;

[0021] 2. The circuit according to the embodiment of the present invention completes the scaling calculation of the score matrix inside the array, reducing the latency of calculation and data transfer. It avoids the calculation stalls and data blockages caused by the large number and scale of matrix division calculations executed by the SIMD cores in the traditional solution;

[0022] 3. The circuit according to the embodiment of the present invention uses a single ferroelectric transistor as the storage unit of the Transformer in-memory computing circuit, and uses one ferroelectric transistor to implement operators, completing the score matrix calculation and scaling operations inside the memory computing array, which is beneficial to improving the storage density and computing power density, and is beneficial to supporting the parallel calculation of larger-scale Transformer models.

[0023] 4. The circuit according to the embodiment of the present invention supports the scaling calculation of the score matrix, avoiding some elements in the score matrix having too large amplitudes from entering the non-linear region of the softmax activation function, and improving the accuracy of the results of the attention layer.

[0024] 5. By effectively reducing the power consumption and latency caused by frequent memory access, the in-memory computing technology based on the new non-volatile memory is expected to greatly improve the computing energy efficiency and computing power, thereby providing hardware support for data-centric computing tasks represented by artificial intelligence. Ferroelectric transistors (FeFETs) have shown great potential in terms of operating power consumption, switching ratio, storage density, and process compatibility. The new non-volatile memory represented by ferroelectric transistors features low power consumption, small latency, high density, and high process compatibility. The non-volatile in-memory computing technology based on it can effectively reduce the data transfer between the processor and the memory as well as between the memory hierarchies (from non-volatile storage to memory) by enabling the memory to support both computing and storage functions, thereby significantly reducing the power consumption and latency brought about by this, and thus breaking through the bottleneck brought about by the memory wall.

[0025] In the present invention, the above technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can be made obvious from the description, or understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained from the content specifically pointed out in the description and the drawings. Description of the Drawings

[0026] The drawings are only for the purpose of showing specific embodiments, and are not considered as limitations of the present invention. Throughout the drawings, the same reference signs denote the same components.

[0027] Figure 1 It is a diagram of the existing SRAM-based Transformer in-memory computing array structure and memory computing unit;

[0028] Figure 2 It is the overall architecture diagram of an existing typical SIMD;

[0029] Figure 3 It is a diagram of an in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to an embodiment of the present invention;

[0030] Figure 4A It is the internal structure diagram of a post-channel ferroelectric transistor according to an embodiment of the present invention;

[0031] Figure 4B It is the measured transfer characteristic curve diagram of an N-type post-channel ferroelectric transistor according to an embodiment of the present invention;

[0032] Figure 5 It is a diagram of the operation process of a circuit for performing score matrix calculation and scaling according to an embodiment of the present invention;

[0033] Figure 6A It is the formula and working schematic diagram of the softmax activation function according to an embodiment of the present invention;

[0034] Figure 6B It is the curve diagram of the softmax activation function image according to an embodiment of the present invention; and

[0035] Figure 7 It is the flowchart of a method for accelerating in-memory computing of a Transformer model based on ferroelectric transistors according to an embodiment of the present invention. Detailed implementation manners

[0036] The following specifically describes the preferred embodiments of the present invention in conjunction with the accompanying drawings. Among them, the accompanying drawings form a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, rather than to limit the scope of the present invention.

[0037] Referring to Figure 3 , a specific embodiment of the present invention discloses an in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors, which includes a memory computing array and the peripheral circuit of the memory computing array. The peripheral circuit includes a word line decoding circuit and a multiplexer, a bit line source line decoding circuit and a multiplexer, a scaling voltage generation circuit, a word line driving circuit, a read-write driving circuit, an analog-to-digital converter reading circuit, a digital shift accumulator, an address timing control circuit, and a softmax activation function module. In the following, referring to Figures 3 to 5 the above components will be described in detail respectively.

[0038] The memory computing array includes multiple rows and multiple columns of memory computing units (i.e., M rows × N columns of memory computing units), which are used to perform score matrix calculation and scaling operations in parallel. Among them, the multiple rows and multiple columns of memory computing units in the memory computing array are used to store the transposed matrix K of the key value matrix TEach memory - computing unit includes a ferroelectric transistor, and each memory - computing unit is used to store the corresponding element in the transposed matrix K of the key - value matrix. T in

[0039] Reference Figure 4A and Figure 4B , the N - type ferroelectric transistor includes a metal - oxide - semiconductor field - effect transistor (MOSFET) and a ferroelectric capacitor located above the MOSFET. The internal gate of the MOSFET is connected to the lower plate of the ferroelectric capacitor via a metal contact. Specifically, the MOSFET includes a source S and a drain D located in a P - type semiconductor substrate, and its top surface is flush with the top surface of the semiconductor substrate; an intermediate dielectric layer is located between the source and the drain above the semiconductor substrate; the internal gate Gint is located above the intermediate dielectric layer (Interlayer). The metal contact is located above the central part of the internal gate, and the internal gate is connected to the lower plate of the ferroelectric capacitor through the metal contact; and the ferroelectric capacitor includes: a lower plate, located above the internal gate and connected to the internal gate; a ferroelectric layer (Fe), located above the lower plate. The ferroelectric capacitor uses a ferroelectric high - k oxide (such as zirconium oxide HZO) as the ferroelectric layer. Applying an electric field to the external gate of the ferroelectric transistor can change the polarization direction of the ferroelectric layer, thereby changing the threshold voltage of the ferroelectric transistor and realizing data storage in multiple threshold - voltage states; and an upper plate, serving as the external gate G of the ferroelectric transistor, is located above the ferroelectric layer. Specifically, multiple word lines WL are connected to the word - line decoder circuit and the multiplexer, and the elements in the query matrix Q are transmitted to the external gate G of the ferroelectric transistor in each memory - computing unit through the word line WL in the form of voltage signals in a periodic order. One end of the bit line BL and the source line SL is connected to the bit - line and source - line decoder circuit and the multiplexer, the other end of the bit line BL is connected to the drain D of the ferroelectric transistor, and the other end of the source line SL is connected to the source S of the ferroelectric transistor.

[0040] The word - line driving circuit is connected to the word - line decoder circuit and the multiplexer, and is used to amplify the word - line signal and increase the driving ability of the word - line signal. The word - line decoder circuit and the multiplexer are used to connect to one end of multiple word lines, and the other ends of multiple word lines are connected to the gates of the corresponding - row ferroelectric transistors. The word - line decoder circuit and the multiplexer are used to select a word line with a specific address from multiple word lines to perform a computing - storage operation on the memory - computing unit corresponding to the selected specific address.

[0041] The read - write driving circuit is connected to the scaled - voltage generating circuit, and is used to amplify the read - out signal and the write signal to increase the driving ability of the read - write signal. Among them, the read - out signal is the scaled score matrix The bit-line source-line decoding circuit and the multiplexer are used to connect to one ends of multiple bit-lines BL and multiple source-lines SL. The other end of the bit-line BL is connected to the drain of the corresponding column ferroelectric transistor, and the other end of the source-line SL is connected to the source of the corresponding column ferroelectric transistor. The bit-line source-line decoding circuit and the multiplexer are used to select a bit-line and a source-line of a specific address from the multiple bit-lines BL and the multiple source-lines SL to perform a compute-storage operation on the compute-storage unit of the selected specific address.

[0042] The scaling voltage generation circuit is used to generate a scaling voltage and transmit it to the bit-line source-line decoding circuit and the multiplexer through the read-write drive circuit to apply the scaling voltage on the multiple source-lines. Specifically, the scaling voltage is sequentially applied to the multiple source-lines in parallel via the read-write drive circuit and the bit-line source-line decoding circuit and the multiplexer. For example, the scaling voltage generation circuit can receive an instruction sent by an external host system and output a configurable scaling voltage with variable amplitude, so as to flexibly control the scaling ratio of the score matrix, increase the versatility of data processing and computing tasks supported by the compute-storage array, and improve the adaptability of the compute-storage array in various application scenarios.

[0043] The analog-to-digital converter readout circuit is used to convert the amplified readout signal into a digital readout signal. The digital shift accumulator is used to shift-weight the digital readout signal to output a multi-bit signal. The address timing control circuit is connected to other peripheral circuits and controls the peripheral circuits to perform storage and computing operations on the compute-storage unit at a specific position of the compute-storage array at a specified timing.

[0044] Reference Figure 6A and Figure 6B , the compute-storage array proposed in this application outputs a scaled score matrix, and this score matrix can be further calculated through a softmax activation function module, that is, the scaled score matrix Each element Xi in it is transformed into a corresponding probability Yi, where 0 < Yi < 1. In the input-output relationship of the softmax activation function, the magnitude of each output result Yi is strongly correlated with the magnitude of the corresponding input element Xi itself, and the sum of all output results Yi is equal to 1. When the input element received by the softmax activation function is in the linear region, the output result of the softmax activation function changes linearly with the input element; while when the input element received by the softmax activation function is in the non-linear region, the output result of the softmax activation function cannot change linearly with the input element, and the output result of the softmax activation function enters the non-linear region. The output result not being able to change linearly with the input element will lead to a decrease in the accuracy of the output result of the softmax activation function, thereby reducing the accuracy of the output result of the attention layer of the Transformer model. Specifically, the amplitude of the scaled score matrix decays significantly, and the significantly decayed score matrices are all within the range of the linear region of the input element of the softmax activation function. When the input element received by the softmax activation function is in the linear region, the output result of the softmax activation function changes linearly with the input element to avoid the output result of the softmax activation function entering the non-linear region and causing loss of calculation accuracy.

[0045] Compared with the prior art, the Transformer model acceleration in-memory computing circuit based on ferroelectric transistors according to the embodiments of the present invention utilizes the characteristics of the FeFET four-terminal device to complete computation inside the memory array, saving the area and power consumption overhead of the peripheral SIMD core. The Vs voltage output by the additionally introduced scaling voltage generation circuit can be multiplexed by each bit line of the memory computing array, which is much smaller than the area and power consumption overhead of the division circuit inside the high-parallel processing unit of the SIMD core.

[0046] Refer to Figure 7 , a specific embodiment of the present invention discloses a method for accelerating in-memory computing of a Transformer model based on ferroelectric transistors, including a writing stage and a computing stage.

[0047] In the writing stage, in step S701, the transposed matrix K of the key-value matrix T All elements in are stored in the memory computing array and stored in the memory computing array in the form of threshold voltages, where the memory computing array includes M rows × N columns of memory computing units;

[0048] In the computing stage: in step S702, a scaling voltage is generated by the scaling voltage generation circuit Then, the scaled voltage is applied in parallel to multiple source lines SL through the read / write drive circuit and the bit line-source line decoding circuit and the multiplexer in sequence; in step S703, the corresponding row vectors in the query matrix Q are applied to multiple word lines WL through the word line decoding circuit and the multiplexer in each of the X cycles to control the on-state of the ferroelectric transistors; in step S704, the scaled results of the product of the row vectors in the query matrix Q and the column vectors of the transposed matrix K are sequentially output via the bit line BL, and the scaled score matrix is obtained after X cycles. T After that, each element Xi in the scaled score matrix is transformed into the corresponding probability Yi through the softmax activation function, where 0 < Yi < 1. In the input-output relationship of the softmax activation function, the magnitude of each output result Yi is related to the magnitude of the corresponding input element Xi itself, and the sum of all the output results Yi is equal to 1. When the scaled score matrix is substituted as the input element into the softmax activation function, all the input elements are within the linear region range, and the output result of the softmax activation function changes linearly with the input elements to avoid the output result of the softmax activation function entering the non-linear region. Specifically, the amplitude of the scaled score matrix decays significantly, and all the score matrices with significant decay can be within the linear region range of the input elements of the softmax activation function. When the input elements received by the softmax activation function are within the linear region, the output result of the softmax activation function changes linearly with the input elements to avoid the output result of the softmax activation function entering the non-linear region and causing loss of calculation accuracy.

[0049] In the following text, with reference to FIG. 6 to FIG. 6, a specific example is used to elaborate in detail on the in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to the embodiments of the present invention.

[0050] Reference Figure 3 to FIG. 6, the ferroelectric transistor memory and computing array: for parallel execution of score matrix calculation and scaling operations, and each ferroelectric transistor can be used as a memory and computing unit to implement

[0051] Reference Figure 3 , the ferroelectric transistor memory and computing array: for parallel execution of score matrix calculation and scaling operations, and each ferroelectric transistor can be used as a memory and computing unit to achieve Operator; word line voltage drive circuit: applies a voltage signal to the word line; word line decoding circuit and multiplexer: select the WL of a specific address, and select the corresponding memory cell to perform a computing and storage operation; bit line and source line decoding circuit and multiplexer: select the BL and SL of a specific address, and select the corresponding memory cell to perform a computing and storage operation; read / write drive circuit: amplifies read and write signals and increases the drive capability of read / write signals. Analog-to-digital converter readout circuit: converts the analog signal output by the array into a digital signal; digital shift accumulator: shifts and weights data to output a multi-bit signal; scaling voltage generation circuit: generates a scaling voltage This voltage will be applied to each column SL in the memory and computing array; Address timing control circuit: This circuit is connected to other peripheral circuits and controls the peripheral circuits to perform storage and computing operations on the memory and computing units at specific positions in the memory and computing array in a normal timing. Refer to Figure 3 , WL[M] is the Mth word line; BL[N] is the Nth bit line; SL[N] is the Nth source line; Q 1,1 is the element in the first row and first column of the Q matrix (query matrix), and is transmitted to the gate of the ferroelectric transistor in the form of a voltage signal through the word line WL[1]; K T 1,1 is the transpose matrix K of the K matrix (key value matrix) T The element in the first row and first column of, is represented in the form of the threshold voltage of the FeFET; VS is the scaling voltage generated by the scaling voltage generation circuit

[0052] First, the word line drive circuit transmits the Q element voltage signal. The word line decoding circuit and multiplexer select to apply the Q element voltage signal to the WL. The scaling voltage generation circuit generates a scaling voltage and the BL bias voltage are amplified by the read / write drive circuit to enhance the signal, and the V is applied through the bit line and source line decoding circuit and multiplexer S voltage signal to the SL, and the BL bias voltage is applied to the BL. After the selected ferroelectric transistor receives the WL, SL, and BL signals, the calculation result is output at the BL end. The calculation results of a column of ferroelectric transistors converge on the BL and are output to the end of the array, representing the scaled result of an element of the score matrix, denoted as the A element, which is represented as an analog signal in current form at this time. The A element passes through the bit line and source line decoding circuit and multiplexer, is enhanced by the read / write drive circuit signal, is transmitted to the analog-to-digital converter output circuit, is converted into a digital domain signal, and finally the data is shifted and weighted by the digital shift accumulator to output a multi-bit signal.

[0053] The biggest feature of this circuit is that it supports completing inside the memory array Operator calculation realizes the scaling operation of the score matrix inside the array. The ferroelectric transistor (FeFET) has the characteristics of a four-terminal device and can apply more input signals compared to traditional two-terminal memories such as RRAM. By storing K T elements inside the FeFET, applying a voltage of Q elements to the WL terminal, and applying a VS voltage with an amplitude of to the SL terminal, the current result can be output at the BL terminal, completing the analog-domain ternary matrix-vector multiplication calculation in one step. This application uses a single FeFET unit to implement the operator and completes the score matrix calculation and the scaling operation of dividing by inside the FeFET memory and computing array.

[0054] Figure 4A Figure shows the internal structure of the post-channel ferroelectric transistor. This memory and computing unit uses the front-channel process to manufacture MOSFETs and the post-channel process to manufacture ferroelectric capacitors. The upper and lower layers of the ferroelectric capacitor are metals, and a ferroelectric high-k oxide (such as zirconium oxide HZO) is added in the middle as the ferroelectric layer. The internal ferroelectric domains will change the polarization direction under gate voltages of different polarities. Programmability under multiple threshold voltage states can be achieved by applying an electric field to the external gate of the post-channel ferroelectric transistor through the word line WL. It has the advantages of strong CMOS process compatibility and high erase / write endurance. In the invention embodiment, the memory and computing units of the Transformer model accelerated in-memory computing circuit can be implemented using ferroelectric transistors such as traditional ferroelectric transistors and post-channel ferroelectric transistors, as well as other four-terminal transistor-based storage devices with storage capabilities.

[0055] Figure 4B Figure shows the transfer characteristic curve of the N-type post-channel ferroelectric transistor in actual tests. The abscissa is the gate voltage V G , and the logarithmic ordinate is the drain current log(I D ). During the process of scanning and increasing the gate voltage from -4.2V to 4.2V, the programming operation is performed when the gate voltage is positive. When the gate voltage applied is 4.2V at the rightmost side of the transfer characteristic curve, the programming is completed. The polarization direction of the ferroelectric domains is downward, and electrons are induced at the channel of the FeFET unit to form an inversion layer. At this time, the FeFET changes from the high-threshold voltage state to the low-threshold voltage state, and the FeFET drain current I G is at the highest value. During the process of scanning and decreasing the gate voltage from 4.2V to -4.2V, the erase operation is performed when the gate voltage is negative. When VWL = -4.2V, the erase is completed. The polarization direction of the ferroelectric domains is upward, and holes are induced at the channel of the memory and computing unit to form an accumulation layer. At this time, the FeFET changes from the low-threshold voltage state to the high-threshold voltage state, and the FeFET drain current I D D ​​It is at the lowest value. When programming and erasing the FeFET, there is no direct current and direct current power consumption, realizing a low-power write operation. At the same time, the large storage window (1.89V) of the FeFET is conducive to realizing multi-value weight storage, providing strong support for high-precision calculation of the Transformer model.

[0056] Transferring the calculation result to the high-hardware-overhead SIMD core outside the array for scaling. In contrast, the solution proposed in this application can bring the following improvements:

[0057] In terms of calculation latency and throughput, this application completes the score matrix calculation and scaling operation in one step in the memory and computing array, avoiding the long-distance transfer of data between the memory and computing array and the peripheral circuit and then performing the division scaling operation during this operation process, improving the overall calculation speed. In addition, the score matrix scaling calculation is supported to be completed in parallel inside the memory and computing array. Since there are multiple rows and columns inside the memory and computing array, it naturally has the property of parallel computing. The score matrix scaling calculation less than or equal to the parallelism of the array can be executed inside the array. The bottleneck of parallel computing lies in the scale of the array. When using the SIMD core, the bottleneck of parallel computing is the scale of the memory and computing array and the parallelism of the SIMD core. Therefore, the solution proposed in this application avoids the situation where the parallelism of the output result of the memory and computing array is greater than the input parallelism of the SIMD core, and avoids the matrix cutting, partitioning, and queuing calculations that need to be performed before the score matrix is sent to the SIMD core for calculation, thus causing data conflict problems caused by input matrix blocking and computing resource competition, effectively improving the throughput of the computing system.

[0058] In terms of hardware area and power consumption, the traditional SIMD core internally includes a programmable input processor, a programmable output processor, a serial input parallel output queue module, a parallel input parallel output queue module, a processing unit array, and a SIMD controller. There are many internal circuit modules, large area and power consumption overheads, complex computing processes, and long data transfer paths. Replacing the SIMD core with a complex circuit structure, large area, and high power consumption with a scaling voltage generation circuit with low area and power consumption overheads to complete the scaling calculation of the score matrix inside the ferroelectric transistor array can effectively reduce the area and power consumption overheads of the peripheral circuit required for the scaling calculation of the score matrix.

[0059] Reference Figure 5 , taking the 3*3 score matrix calculation and scaling operation as an example to introduce the operation process. This process is divided into a write stage and a calculation stage. In the write stage, all elements of the K T matrix are written into the memory array and stored in the memory and computing unit in the form of a threshold voltage. In the calculation stage, the scaling voltage generated by the scaling voltage generation circuit It is applied to all SLs through the read / write drive circuit, the bit line and source line decoding circuit, and the multiplexer. Three voltage pulse-encoded elements of the Q vector are input in parallel for each of the 3 WLs in each cycle to control the on-state of the FeFET. The current results output by a column of FeFET cells converge on each BL to obtain the multiplication and scaling result of a row of the Q vector and a column of the K T vector, as shown in Equation 1 below. Three scaled score vectors are output from the 3 SLs. After three cycles, a 3×3 scaled score matrix is obtained.

[0060]

[0061] In terms of calculating accuracy, some traditional in-memory computing accelerators do not support the calculation of score matrix scaling. The scaled score matrix output by the memory and computing array can be sent to the softmax activation function module in a circuit outside this application for calculation. A score matrix with a large amplitude is likely to enter the non-linear region of the softmax activation function, reducing the accuracy of the result of the attention layer. In the embodiments of this application, the amplitude of the score matrix is scaled so that most of the values of the scaled score matrix are distributed in the linear region of the softmax activation function, alleviating the problem of accuracy loss caused by a score matrix with a large amplitude entering the non-linear region of the softmax activation function, improving the output accuracy of the softmax activation function module, and thus enhancing the calculation accuracy of the attention layer of the Transformer model.

[0062] Reference Figure 6A , softmax is an activation function for multi-classification problems. In multi-class classification problems, more than two class labels require class membership. For any real vector of length k, softmax can compress it into a real vector of length k, with values in the range (0, 1) and the sum of the elements in the vector being 1. The softmax activation function can transform each element of the input vector into a probability Yi (0 < Yi < 1), which is the output result of the softmax activation function. The size of the probability is strongly correlated with the size of the element itself. The sum of the output probabilities Yi corresponding to all input elements Xi is equal to 1, thus intuitively showing the relative size of each element to other elements. The softmax activation function is as follows:

[0063]

[0064] The image of the softmax activation function is as Figure 6BAs shown, when the input element is in the linear region of the softmax function, the output result of the softmax activation function can vary linearly with the input. When the input element is within this range, the output result of the softmax activation function can accurately represent the relative magnitudes between each input element. When the input element is in the non-linear region of the softmax function, the softmax activation function saturates, and the output result cannot vary linearly with the input. When the input element is within this range, the output result of the softmax activation function cannot accurately represent the relative magnitudes between each input element, resulting in output errors and accuracy losses.

[0065] In terms of storage density, the circuit designed in the present invention uses a single ferroelectric transistor as the memory and computing unit of the Transformer in-memory computing circuit. Utilizing the characteristics of the four-terminal device of the ferroelectric transistor, a single ferroelectric transistor is used to complete the analog-domain ternary matrix-vector multiplication calculation in one step, realizing the operator, and completing the calculation and scaling operations of the score matrix inside the memory and computing array. The memory and computing unit structure of the traditional SRAM+NOR structure has 10 transistors. The large area overhead of the memory and computing unit leads to low storage density and computing power density of the memory and computing array. At the same time, limited by the limited number of input ports, the SRAM+NOR memory and computing unit can only perform the score matrix calculation operation. In contrast, the memory and computing unit based on ferroelectric transistors proposed in this application is conducive to improving the storage density and computing power density, thereby supporting the parallel calculation of larger-scale Transformer models.

[0066] Those skilled in the art can understand that to implement all or part of the processes of the above-described embodiment methods, program instructions can be sent through a computing system to control the circuit hardware to complete operations other than the score matrix calculation and scaling, and complete other calculations in the Transformer model. The program instructions can be stored in a readable storage medium of the computing system. Among them, the readable storage medium of the computing system is a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0067] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A Transformer model accelerated in-memory computing circuit based on ferroelectric transistors, characterized in that Comprising: A memory - in - compute array, including multiple rows and columns of memory - in - compute units, for parallel execution of score matrix calculation and scaling operations. Each memory - in - compute unit includes a ferroelectric transistor. A word - line decoding circuit and a multiplexer, which are used to connect to one end of multiple word - lines, and the other ends of the multiple word - lines are connected to the gates of the ferroelectric transistors in the corresponding rows. A bit - line and source - line decoding circuit and a multiplexer, which are used to connect to one end of multiple bit - lines and multiple source - lines. The other ends of the multiple bit - lines are connected to the drains of the ferroelectric transistors in the corresponding columns, and the other ends of the multiple source - lines are connected to the sources of the ferroelectric transistors in the corresponding columns; and A scaling voltage generation circuit, which is used to generate a scaling voltage and apply the scaling voltage to the multiple source - lines via the bit - line and source - line decoding circuit and the multiplexer.

2. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 1, wherein The ferroelectric transistor FeFET includes a field - effect transistor MOSFET and a ferroelectric capacitor located above the MOSFET. The internal gate of the MOSFET is connected to the lower plate of the ferroelectric capacitor via a metal contact.

3. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 2, wherein The MOSFET includes: a source, a drain, which are located in the semiconductor substrate and whose top surfaces are flush with the top surface of the semiconductor substrate; an intermediate dielectric layer, which is located between the source and the drain above the semiconductor substrate; an internal gate, which is located above the intermediate dielectric layer. The ferroelectric capacitor includes: a lower plate, which is located above the MOSFET and is connected to the internal gate of the MOSFET through a metal contact; a ferroelectric layer, which is located above the lower plate and is made of ferroelectric oxide; and an upper plate, which serves as the external gate of the ferroelectric transistor and is located above the ferroelectric layer.

4. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 2, wherein The multi-row and multi-column memory and computing units of the memory and computing array are used to store the transposed matrix K of the key-value matrix T , where each memory and computing unit is used to store the corresponding element in the transposed matrix K of the key-value matrix T ; and The multiple word - lines transmit the elements in the query matrix Q to the external gates of the ferroelectric transistors in each memory - in - compute unit in a periodic order in the form of voltage signals through the word - line WL via the word - line decoding circuit and the multiplexer.

5. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 4, wherein It further includes a word - line driving circuit, where The word - line driving circuit is connected to the word - line decoding circuit and the multiplexer, and is used to amplify the word - line signal and increase the driving ability of the word - line signal. The word - line decoding circuit and the multiplexer are used to select a word - line with a specific address from the multiple word - lines to perform a compute - storage operation on the memory - in - compute unit corresponding to the selected specific address.

6. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 5, wherein It further includes a read - write driving circuit, where The read / write driving circuit is connected to the scaling voltage generating circuit and is configured to amplify a read signal and a write signal to increase the driving capability of the read / write signals, wherein the read signal is a scaled score matrix The bit - line and source - line decoding circuit and the multiplexer are used to select a bit - line and a source - line with a specific address from the multiple bit - lines and the multiple source - lines to perform a compute - storage operation on the memory - in - compute unit corresponding to the selected specific address.

7. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 6, wherein It further includes an analog - to - digital converter read - out circuit, a digital shift - accumulator, and an address timing control circuit. Among them, The analog - to - digital converter read - out circuit is used to convert the amplified read - out signal from analog to digital read - out signal. The digital shift - accumulator is used to shift - weight the digital read - out signal to output a multi - bit signal. The address timing control circuit is connected to other peripheral circuits and controls the peripheral circuits to perform storage and compute operations on the memory - in - compute units at specific positions in the memory - in - compute array at a specified timing.

8. The in-memory computing circuit for accelerating the Transformer model based on ferroelectric transistors according to claim 5, wherein It further includes a softmax activation function module for converting each element Xi in the scaled score matrix into a corresponding probability Yi, where 0 < Yi < 1. In the input-output relationship of the softmax activation function, the magnitude of each output result Yi is related to the magnitude of the corresponding input element Xi itself, and the sum of all output results Yi is equal to 1. When the scaled score matrix is substituted as the input element into the softmax activation function, all input elements are within the linear region range, and the output result of the softmax activation function changes linearly with the input element to prevent the output result of the softmax activation function from entering the non-linear region.

9. A method for accelerating in-memory computing of a Transformer model based on ferroelectric transistors, characterized in that, Including a write phase and a compute phase. Among them, During the writing phase, all elements in the transposed matrix K of the key-value matrix are stored in the memory-computation array in the form of threshold voltages, where the memory-computation array includes M rows × N columns of memory-computation units; T ​ In the computing stage: a scaled voltage is generated by the scaled voltage generation circuit Then, the scaled voltage is applied in parallel to multiple source lines SL through the read / write drive circuit, the bit line and source line decoding circuit, and the multiplexer in sequence; a row vector in the query matrix Q is applied to multiple word lines WL in each of the X cycles through the word line decoding circuit and the multiplexer to control the on-state of the ferroelectric transistors; and the scaled result of the product of the row vector in the query matrix Q and the column vector of the transposed matrix K is output sequentially via the bit line BL, and a scaled score matrix is obtained after X cycles T and after X cycles, a scaled score matrix is obtained 10. The method for accelerating in-memory computing of a Transformer model based on a ferroelectric transistor according to claim 9, wherein It further includes: Convert each element Xi in the scaled score matrix through the softmax activation function into the corresponding probability Yi, where 0 < Yi < 1. In the input-output relationship of the softmax activation function, the magnitude of each output result Yi is related to the magnitude of the corresponding input element Xi itself, and the sum of all output results Yi is equal to 1. Among them, when the scaled score matrix is substituted as the input element into the softmax activation function, all input elements are within the linear region range, and the output result of the softmax activation function changes linearly with the input element to prevent the output result of the softmax activation function from entering the non-linear region.

Images