Scalable 2x2 processing unit matrix multiplier and transform acceleration method

Abstract

The application discloses an extensible 2*2 processing unit matrix multiplier and a Transformer acceleration method, when a control module executes an OS data flow mode, each processing unit respectively performs multiplication and addition processing according to received activation values and weight values, and obtains a first multiplication and addition result; each processing unit respectively performs multiplication operation of activation values and weight values according to processing units interconnected with the processing unit, and obtains a first operation result, and the first operation result is accumulated with the first multiplication and addition result to obtain a two-matrix product result; when the control module executes a WS data flow mode, each processing unit respectively performs multiplication and addition processing according to received activation values and weight values, and obtains a second multiplication and addition result; each processing unit locks the weight values unchanged, and respectively performs multiplication operation of the activation values according to the processing units interconnected with the processing unit, and obtains a second operation result, and the second operation result is accumulated with the second multiplication and addition result to obtain a matrix multiplication result based on fixed weight, and the efficiency ratio and flexibility are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of integrated circuits, and particularly relates to a scalable 2x2 processing unit matrix multiplier and a Transformer (a deep learning model based on an attention mechanism) acceleration method. BACKGROUND

[0002] Matrix multiplication of activation matrices and weight matrices is a core computing task in the training and reasoning of a Transformer model. With the expansion of the model size, the demand for low-power and diverse matrix multiplication is growing. In a Transformer model, large-scale matrix operation tasks such as full connection feedforward network linear transformation are required, and there are also a large number of small and medium-sized matrix operation tasks such as attention head blocking and dynamic sparse activation, which require the matrix multiplier to reasonably distinguish between large and small scale matrix multiplication. There are different data flow modes in the Transformer model. In the WS (Weight Stationary, weight fixed) data flow mode, the weight data in the matrix remains unchanged, and the elements of the input matrix enter the PE (Processing Element, processing unit) for calculation in sequence, which is suitable for the weight update stage and can effectively reduce the data transmission and computing overhead. In the OS (Output Stationary, output fixed) data flow mode, the input data remains unchanged, and the calculation results are accumulated gradually in each cycle, which is suitable for the forward propagation stage in matrix multiplication. The difference between the data flow modes will affect the efficiency of matrix multiplication. Matrix multiplication plays a key role in the training and reasoning of a Transformer model based on a neural network.

[0003] The existing matrix multiplication task in the Transformer model can be implemented through the following schemes:

[0004] The first patent, "Matrix Multiplier for Transformer-like Model Training," employs a systolic array structure. This array contains multiple processing units (PEs). Horizontally, activation data flows from left to right, with a one-cycle interval between rows. Vertically, weight data flows from top to bottom, also with a one-cycle interval. When all data has flowed through the systolic array, the matrix multiplication result is stored within each PE. However, during certain computational phases, not all PEs are performing calculations; some may only be transferring data or waiting for the next task, leading to decreased computational efficiency. Simultaneously, idle PEs cannot be enabled, resulting in them not only not participating in computation but also consuming significant power. Therefore, when performing matrix multiplication, the systolic array cannot maximize computational efficiency due to partially idle PEs. Furthermore, the lack of dynamic enable control causes the system to be in a high-power state at all times.

[0005] The second patent, "Method and Apparatus for Performing Convolution Operations by Traversing Tensors in a Patch," discloses a hybrid firmware-hardware convolution acceleration method that optimizes the efficiency of convolution operations by traversing plots on the input tensor. In this scheme, the input tensor is divided into multiple smaller plots, each requiring a minimum matrix size of 16×16. If this requirement is not met, zero-value padding to 16×16 is applied. The input plots undergo convolution calculations via a hardware accelerator, reducing the computational burden when processing large-scale data. Furthermore, it proposes different types of plot traversal structures, such as batch traversal, input reuse traversal, and weight reuse traversal. These traversal methods can be flexibly configured based on the tensor's dimensions and hardware capabilities. By allocating convolution operation tasks to the hardware accelerator and firmware in collaboration, this scheme not only improves processing speed but also reduces the burden on the main processor. Overall, the technology in this patent helps improve data processing efficiency during neural network training and inference, especially in scenarios requiring the processing of large-scale tensors. However, the tensor core approach trades area for significant speed gains. In large-scale matrix multiplication, each multiplication-addition of matrix elements requires a separate multiplier-adder. While the extremely high parallelism brings a significant speed advantage, it also results in very high power consumption. For small matrix multiplication, the input matrix must satisfy a 16×16 minimum matrix partitioning scheme. If this standard is not met, zero-value padding is required. With an input matrix of size 4×4, to meet the 1×16 minimum input matrix partitioning requirement, 240 zero-value elements must be padded, resulting in a resource utilization rate of only 6.25%. Furthermore, the existing architecture exhibits poor scalability as the computational task scales, making it unable to flexibly provide efficient computational solutions for tasks of varying sizes, especially when dealing with irregular matrices.

[0006] Therefore, how to provide a 2×2 processing unit matrix multiplier with high energy efficiency and high flexibility has become an important issue. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, this invention provides a scalable 2×2 processing unit matrix multiplier and a Transformer acceleration method.

[0008] The technical problem to be solved by this invention is achieved through the following technical solution:

[0009] In a first aspect, the present invention provides a scalable 2×2 processing unit matrix multiplier, the 2×2 processing unit matrix multiplier including a control module and four processing units;

[0010] When the control module executes the OS data stream mode, the four processing units perform the following operations:

[0011] During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the first multiplication and addition result; the activation value and weight value are determined based on the input activation matrix and weight matrix; the size of the activation matrix and the weight matrix are both 2×2;

[0012] During the second clock cycle, each processing unit performs a multiplication operation on the activation value and weight value according to the processing unit interconnected with itself to obtain the first operation result, and then accumulates the first operation result with the first multiplication-addition result to obtain the product result of the two matrices;

[0013] When the control module executes the WS data stream mode, the four processing units perform the following operations:

[0014] During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the second multiplication and addition result;

[0015] During the second clock cycle, each processing unit locks its weight value unchanged and performs a multiplication operation with the activation value according to the processing unit interconnected with itself to obtain a second operation result. The second operation result is then accumulated with the second multiplication-addition result to obtain a matrix multiplication result based on fixed weights.

[0016] Optionally, the four processing units include a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit;

[0017] The first processing unit and the second processing unit are interconnected by activation values, the third processing unit and the fourth processing unit are interconnected by activation values, the first processing unit and the third processing unit are interconnected by weight values, and the second processing unit and the fourth processing unit are interconnected by weight values.

[0018] Optionally, the first processing unit includes a first activation value register, a first weight value register, a first intermediate calculation result register, a first data selector, a second data selector, a first multiplier, and a first adder;

[0019] In this system, the second input terminal of the first data selector receives the weight value; the output terminal of the first data selector is connected to the input terminal of the first weight value register; the first output terminal of the first weight value register is connected to the first input terminal of the first data selector; the second output terminal of the first weight value register is connected to the first input terminal of the second data selector and the fourth input terminal of the sixth data selector in the third processing unit; the input terminal of the first activation value register receives the activation value; the output terminal of the first activation value register is connected to the second input terminal of the second data selector and the third input terminal of the fourth data selector in the second processing unit; and the third input terminal of the second data selector is connected to the... The output of the second activation value register is connected to the output of the third weight value register in the third processing unit. The fourth input of the second data selector is connected to the second output of the third weight value register in the third processing unit. The first output of the second data selector is connected to the first input of the first multiplier. The second output of the second data selector is connected to the second input of the first multiplier. The output of the first multiplier is connected to the first input of the first adder. The second input of the first adder is connected to the second output of the first intermediate calculation result register. The output of the first adder is connected to the input of the first intermediate calculation result register. The first output of the first intermediate calculation result register outputs the calculation result in the first processing unit.

[0020] Optionally, the second processing unit includes a second activation value register, a second weight value register, a second intermediate calculation result register, a third data selector, a fourth data selector, a second multiplier, and a second adder;

[0021] In this system, the second input terminal of the third data selector receives a weight value, and the output terminal of the third data selector is connected to the input terminal of the second weight value register. The first output terminal of the second weight value register is connected to the first input terminal of the third data selector, and the second output terminal of the second weight value register is connected to the first input terminal of the fourth data selector and the fourth input terminal of the eighth data selector in the fourth processing unit. The input terminal of the second activation value register receives an activation value, and the output terminal of the second activation value register is connected to the second input terminal of the fourth data selector and the third input terminal of the second data selector in the first processing unit. The third input terminal of the fourth data selector is connected to the first processing unit. The output of the first activation value register is connected to the second output of the fourth weight value register in the fourth processing unit. The first output of the fourth data selector is connected to the first input of the second multiplier. The second output of the fourth data selector is connected to the second input of the second multiplier. The output of the second multiplier is connected to the first input of the second adder. The second input of the second adder is connected to the second output of the second intermediate calculation result register. The output of the second adder is connected to the input of the second intermediate calculation result register. The first output of the second intermediate calculation result register outputs the calculation result in the second processing unit.

[0022] Optionally, the third processing unit includes a third activation value register, a third weight value register, a third intermediate calculation result register, a fifth data selector, a sixth data selector, a third multiplier, and a third adder;

[0023] In this system, the second input terminal of the fifth data selector receives a weight value, and the output terminal of the fifth data selector is connected to the input terminal of the third weight value register. The first output terminal of the third weight value register is connected to the first input terminal of the fifth data selector, and the second output terminal of the third weight value register is connected to the first input terminal of the sixth data selector and the fourth input terminal of the second data selector in the first processing unit. The input terminal of the third activation value register receives an activation value, and the output terminal of the third activation value register is connected to the second input terminal of the sixth data selector and the third input terminal of the eighth data selector in the fourth processing unit. The third input terminal of the sixth data selector is connected to the... The output of the fourth activation value register is connected to the output of the sixth data selector. The fourth input of the sixth data selector is connected to the second output of the first weight value register in the first processing unit. The first output of the sixth data selector is connected to the first input of the third multiplier. The second output of the sixth data selector is connected to the second input of the third multiplier. The output of the third multiplier is connected to the first input of the third adder. The second input of the third adder is connected to the second output of the third intermediate calculation result register. The output of the third adder is connected to the input of the third intermediate calculation result register. The first output of the third intermediate calculation result register outputs the calculation result in the third processing unit.

[0024] Optionally, the fourth processing unit includes a fourth activation value register, a fourth weight value register, a fourth intermediate calculation result register, a seventh data selector, an eighth data selector, a fourth multiplier, and a fourth adder.

[0025] In this system, the second input terminal of the seventh data selector receives the weight value, and the output terminal of the seventh data selector is connected to the input terminal of the fourth weight value register. The first output terminal of the fourth weight value register is connected to the first input terminal of the seventh data selector, and the second output terminal of the fourth weight value register is connected to the first input terminal of the eighth data selector and the fourth input terminal of the fourth data selector in the second processing unit. The input terminal of the fourth activation value register receives the activation value, and the output terminal of the fourth activation value register is connected to the second input terminal of the eighth data selector and the third input terminal of the sixth data selector in the third processing unit. The third input terminal of the eighth data selector is connected to the... The output of the third activation value register is connected to the output of the second weight value register in the second processing unit. The output of the eighth data selector is connected to the first input of the fourth multiplier. The output of the eighth data selector is connected to the second input of the fourth multiplier. The output of the fourth multiplier is connected to the first input of the fourth adder. The second input of the fourth adder is connected to the second output of the fourth intermediate calculation result register. The output of the fourth adder is connected to the input of the fourth intermediate calculation result register. The first output of the fourth intermediate calculation result register outputs the calculation result in the fourth processing unit.

[0026] Optionally, when the weight matrix and the activation matrix are both 4×4, the weight matrix and the activation matrix are processed using four 2×2 processing unit matrix multipliers.

[0027] Optionally, when the weight matrix and the activation matrix are both 8×8, the weight matrix and the activation matrix are processed using eight 2×2 processing unit matrix multipliers.

[0028] Secondly, the present invention provides a Transformer acceleration method, which is implemented based on any of the above-mentioned 2×2 processing unit matrix multipliers.

[0029] The present invention provides a scalable 2×2 processing unit matrix multiplier in which the activation and weight values ​​in the activation and weight matrices are sent to each processing unit. The parallelism is improved by the simultaneous calculation of each processing unit, thereby improving the energy efficiency ratio. It can easily switch between OS and WS data streams in zero cycles and adapt to different matrix calculation scenarios in Transformer.

[0030] Furthermore, the 2×2 processing unit matrix multiplier provided by this invention can be used as a basic unit and can be flexibly expanded into a larger matrix multiplier according to actual needs, in order to deal with activation matrices and weight matrices of different sizes, thereby improving the flexibility of the matrix multiplier.

[0031] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of an scalable 2×2 processing unit matrix multiplier provided in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the internal data flow of a scalable 2×2 processing unit matrix multiplier provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of the 4×4 processing unit matrix multiplier provided in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the workflow of the extended computation array based on a 2×2 processing unit matrix multiplier provided in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0037] To address the issues of low energy efficiency and poor flexibility in existing matrix multipliers, this invention provides a scalable 2×2 processing unit matrix multiplier. (See also...) Figure 1 , Figure 1 This is a schematic diagram of a scalable 2×2 processing unit matrix multiplier provided in an embodiment of the present invention. The scalable 2×2 processing unit matrix multiplier includes a control module and four processing units, namely a first processing unit PE0, a second processing unit PE1, a third processing unit PE2, and a fourth processing unit PE3.

[0038] In this embodiment of the invention, the control module can be specifically used to control the 2×2 PE calculation cycle, WS / OS data stream switching, and dynamic power consumption control. The four processing units have the same structure. Each processing unit includes an activation value register, a weight value register, an intermediate calculation result register, two data selectors, a multiplier, and an adder.

[0039] In this embodiment of the invention, the interconnection of the four processing units is divided into two types. One type is for exchanging activation values ​​in the Transformer, that is, the activation values ​​are interconnected between the first processing unit and the second processing unit, and between the third processing unit and the fourth processing unit. The other type is for exchanging weight values ​​in the Transformer, that is, the weight values ​​are interconnected between the first processing unit and the third processing unit, and between the second processing unit and the fourth processing unit.

[0040] The structures of the first processing unit, the second processing unit, the third processing unit, and the fourth processing unit are described below. (See attached text.) Figure 1 , Figure 1 This is a schematic diagram of the structure of an scalable 2×2 processing unit matrix multiplier provided in an embodiment of the present invention.

[0041] In this embodiment of the invention, the first processing unit PE0 includes a first activation value register a11_reg, a first weight value register b11_reg, a first intermediate calculation result register partialsum_reg1, a first data selector MUX1, a second data selector MUX2, a first multiplier, and a first adder.

[0042] The first data selector has a weight value input at its second input terminal, an output terminal connected to the input terminal of a first weight value register, a first output terminal connected to the first input terminal of the first data selector, a second output terminal connected to the first input terminal of the second data selector and the fourth input terminal of the sixth data selector in the third processing unit, an activation value input at its input terminal, an output terminal connected to the second input terminal of the second data selector and the third input terminal of the fourth data selector in the second processing unit, a third input terminal of the second data selector connected to the output terminal of the second activation value register in the second processing unit, a fourth input terminal of the second data selector connected to the second output terminal of the third weight value register in the third processing unit, a first output terminal of the second data selector connected to the first input terminal of the first multiplier, a second output terminal of the second data selector connected to the second input terminal of the first multiplier, an output terminal of the first multiplier connected to the first input terminal of the first adder, a second input terminal of the first adder connected to the second output terminal of the first intermediate calculation result register, an output terminal of the first adder connected to the input terminal of the first intermediate calculation result register, and a first output terminal of the first intermediate calculation result register outputting the calculation result from the first processing unit.

[0043] In this embodiment of the invention, the second processing unit PE1 includes a second activation value register a12_reg, a second weight value register b22_reg, a second intermediate calculation result register partialsum_reg2, a third data selector MUX3, a fourth data selector MUX4, a second multiplier, and a second adder.

[0044] The third data selector has its second input terminal receiving the weight value, its output terminal connected to the input terminal of the second weight value register, its first output terminal connected to the first input terminal of the third data selector, its second output terminal connected to the first input terminal of the fourth data selector and the fourth input terminal of the eighth data selector in the fourth processing unit, its second activation value register receiving the activation value, its output terminal connected to the second input terminal of the fourth data selector and the third input terminal of the second data selector in the first processing unit, its third input terminal connected to the output terminal of the first activation value register in the first processing unit, its fourth input terminal connected to the second output terminal of the fourth weight value register in the fourth processing unit, its first output terminal connected to the first input terminal of the second multiplier, its second output terminal connected to the second input terminal of the second multiplier, its output terminal connected to the first input terminal of the second adder, its second input terminal connected to the second output terminal of the second intermediate calculation result register, its output terminal connected to the input terminal of the second intermediate calculation result register, and its first output terminal outputting the calculation result from the second processing unit.

[0045] In this embodiment of the invention, the third processing unit PE2 includes a third activation value register a22_reg, a third weight value register b21_reg, a third intermediate calculation result register partialsum_reg3, a fifth data selector MUX5, a sixth data selector MUX6, a third multiplier, and a third adder.

[0046] The fifth data selector has its second input terminal receiving the weight value, its output terminal connected to the input terminal of the third weight value register, its first output terminal connected to the first input terminal of the fifth data selector, its second output terminal connected to the first input terminal of the sixth data selector and the fourth input terminal of the second data selector in the first processing unit, its input terminal receiving the activation value, its output terminal connected to the second input terminal of the sixth data selector and the third input terminal of the eighth data selector in the fourth processing unit, its third input terminal connected to the output terminal of the fourth activation value register in the fourth processing unit, its fourth input terminal connected to the second output terminal of the first weight value register in the first processing unit, its first output terminal connected to the first input terminal of the third multiplier, its second output terminal connected to the second input terminal of the third multiplier, its output terminal connected to the first input terminal of the third adder, its second input terminal connected to the second output terminal of the third intermediate calculation result register, its output terminal connected to the input terminal of the third intermediate calculation result register, and its first output terminal outputting the calculation result from the third processing unit.

[0047] In this embodiment of the invention, the fourth processing unit PE3 includes a fourth activation value register a21_reg, a fourth weight value register b12_reg, a fourth intermediate calculation result register partialsum_reg4, a seventh data selector MUX7, an eighth data selector MUX8, a fourth multiplier, and a fourth adder.

[0048] The seventh data selector has its second input terminal receiving the weight value, its output terminal connected to the input terminal of the fourth weight value register, its first output terminal connected to the first input terminal of the seventh data selector, its second output terminal connected to the first input terminal of the eighth data selector and the fourth input terminal of the fourth data selector in the second processing unit, its input terminal receiving the activation value, its output terminal connected to the second input terminal of the eighth data selector and the third input terminal of the sixth data selector in the third processing unit, its third input terminal connected to the output terminal of the third activation value register in the third processing unit, its fourth input terminal connected to the second output terminal of the second weight value register in the second processing unit, its first output terminal connected to the first input terminal of the fourth multiplier, its second output terminal connected to the second input terminal of the fourth multiplier, its output terminal connected to the first input terminal of the fourth adder, its second input terminal connected to the second output terminal of the fourth intermediate calculation result register, its output terminal connected to the input terminal of the fourth intermediate calculation result register, and its first output terminal outputting the calculation result from the fourth processing unit.

[0049] See Figure 2 , Figure 2 This is a schematic diagram of the internal data flow of a scalable 2×2 processing unit matrix multiplier provided in an embodiment of the present invention. The timing diagram for multiplying two 2×2 PE matrices is as follows:

[0050] The control module in the 2×2 processing unit matrix multiplier controls the 2×2 PE to complete the matrix calculation in two cycles, while configuring different data streams to enable the entire 2×2 PE.

[0051] In this embodiment of the invention, the control module can be used to switch WS / OS data streams.

[0052] When the control module executes the OS data stream mode, the four processing units perform the following operations:

[0053] During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the first multiplication and addition result; the activation value and weight value are determined based on the input activation matrix and weight matrix; the size of the activation matrix and weight matrix is ​​2×2.

[0054] During the second clock cycle, each processing unit performs a multiplication operation on the activation value and weight value of the processing unit interconnected with itself to obtain the first operation result, and then accumulates the first operation result with the first multiplication-addition result to obtain the product result of the two matrices.

[0055] In OS mode, during the first clock cycle, the eight data elements of the 2×2 activation matrix A and weight matrix B are passed into the 2×2 PE matrix in the following specific manner: the activation value and weight value are respectively sent to the activation value register (the register starting with the lowercase letter 'a') and the weight value register (the register starting with the lowercase letter 'b').

[0056] In this embodiment of the invention, activation value a11 and weight value b11 are respectively sent to the first activation value register a11_reg and the first weight value register b11_reg of the first processing unit PE0; activation value a12 and weight value b22 are respectively sent to the second activation value register a12_reg and the second weight value register b22_reg of the second processing unit PE1; activation value a22 and weight value b21 are respectively sent to the third activation value register a22_reg and the third weight value register b21_reg of the third processing unit PE2; and activation value a21 and weight value b12 are respectively sent to the fourth activation value register a21_reg and the fourth weight value register b12_reg of the fourth processing unit PE3. During the first clock cycle, the activation value and weight value sent to the four PEs are simultaneously multiplied and added to obtain the first multiply-added result. At this time, the value in the partial sum register, i.e., the intermediate calculation result register, is 0, and the first multiply-added result is stored in the intermediate calculation result register.

[0057] Specifically, in the first clock cycle, the data stored in the first processing unit PE0 is a11×b11+0, the data stored in the second processing unit PE1 is a12×b22+0, the data stored in the third processing unit PE2 is a22×b21+0, and the data stored in the fourth processing unit PE3 is a21×b12+0.

[0058] In the second clock cycle, each PE (Processing Entity) performs a multiplication operation on the activation and weight values ​​according to a specific order, based on the activation and weight values ​​in the processing units interconnected with it, to obtain the first calculation result. Specifically, the first processing unit PE0 selects the second activation value register a12_reg from the second processing unit PE1 and the third weight value register b21_reg from the third processing unit PE2 for the calculation; the second processing unit PE1 selects the first activation value register a11_reg from the first processing unit PE0 and the fourth weight value register b12_reg from the fourth processing unit PE3 for the calculation; and the third processing unit PE2 selects the fourth weight value register b12_reg from the fourth processing unit PE3. The fourth activation value register a21_reg in PE3 and the first weight value register b11_reg in the first processing unit PE0 participate in the operation. The fourth processing unit PE3 selects the third activation value register a22_reg in the third processing unit PE2 and the second weight value register b22_reg in the second processing unit PE1 to participate in the operation. In the second clock cycle, the eight elements sent to the four PEs are multiplied simultaneously under the selection of the data selector to obtain the first operation result. Then, the first operation result is accumulated with the partial sum register that stores the first multiplication and addition result of the first cycle. At this time, the four partial sum registers store the product result of the two matrices.

[0059] Specifically, during the second clock cycle, the stored result in the first processing unit is a11×b11+a12×b21, the stored result in the second processing unit PE1 is a12×b22+a11×b12, the stored result in the third processing unit PE2 is a22×b21+a21×b11, and the stored result in the fourth processing unit PE3 is a21×b12+a22×b22.

[0060] By integrating the stored results from the four processing units through the control module, the product of the two matrices can be obtained.

[0061] Compared to the OS data stream mode, in the WS data stream mode, the values ​​of the weight registers in each processing unit are locked, always using the weight values ​​input in the first clock cycle, and do not participate in the weight value exchange in the second clock cycle; the transmission, exchange, and calculation process of activation values ​​is consistent with the OS mode.

[0062] When the control module executes the WS data stream mode, the four processing units perform the following operations:

[0063] During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the second multiplication and addition result;

[0064] During the second clock cycle, each processing unit locks its weight value unchanged and performs a multiplication operation with the activation value according to the processing unit interconnected with itself to obtain a second operation result. The second operation result is then accumulated with the second multiplication-addition result to obtain a matrix multiplication result based on fixed weights.

[0065] In this embodiment of the invention, during the first clock cycle, similar to the OS mode, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the second multiplication and addition result.

[0066] During the second clock cycle, activation values ​​are exchanged according to the rules of the OS data stream mode, but weight values ​​are not exchanged; the weight values ​​received in the first clock cycle are used directly.

[0067] The first processing unit PE0 selects the second activation value register a12_reg from the second processing unit PE1 and its own first weight value register b11_reg for operation. The second processing unit PE1 selects the first activation value register a11_reg from the first processing unit PE0 and its own second weight value register b22_reg for operation. The third processing unit PE2 selects the fourth activation value register a21_reg from the fourth processing unit PE3 and its own fourth weight value register b21_reg for operation. The fourth processing unit PE3 selects the third activation value register a22_reg from the third processing unit PE2 and its own fourth weight value register b12_reg for operation.

[0068] Specifically, during the second clock cycle, the stored result in the first processing unit PE0 is a11×b11+a12×b11, the stored result in the second processing unit PE1 is a12×b22+a11×b22, the stored result in the third processing unit PE2 is a22×b21+a21×b21, and the stored result in the fourth processing unit PE3 is a21×b12+a22×b12.

[0069] By integrating the stored results from the four processing units through the control module, a matrix multiplication result based on fixed weights can be obtained.

[0070] In WS mode, the weight matrix does not need to be updated. The data selector related to the weights continuously selects the value stored in the first clock cycle as the weight input. The rest of the behavior is the same as in OS mode, effectively reducing data transmission and computational overhead. The switch between the two modes can be achieved in zero cycles, without waiting.

[0071] Furthermore, the 2×2 processing unit matrix multiplier provided in this embodiment of the invention can serve as the basic unit for calculating matrix expansion. A schematic diagram illustrating the structure of a 4×4 processing unit matrix multiplier from the 2×2 processing unit matrix multiplier is shown below.Figure 3 As shown, its interconnection structure is consistent with that of a 2×2 processing unit matrix multiplier. When performing matrix calculations, the 4×4 processing unit matrix multiplier first divides the 4×4 activation matrix and weight matrix into four 2×2 subarrays, and then obtains the matrix calculation result over four cycles. Following the same expansion method, the 4×4 PE can be expanded into an 8×8 PE, etc. If the PEs after expansion into new computing arrays are not enabled, they will all go into sleep mode to save power.

[0072] See Figure 4 , Figure 4 This is a schematic diagram of the workflow of the extended computation array based on a 2×2 processing unit matrix multiplier provided in this embodiment of the invention. First, the input matrix of odd-numbered multiplication is expanded into an even-numbered multiplication matrix by zero-value filling. Then, two different data flow modes, WS / OS, are configured. Subsequently, two control modes, large matrix multiplication and small matrix multiplication, are configured. In the large matrix multiplication mode, the extended array participates in the matrix operation as a whole. If the calculation is completed, the process ends; if not, the matrix multiplication mode is checked again. In the small matrix multiplication mode, the process first checks a portion of the 2×2 processing unit matrix multiplier. 2. Whether PE is enabled, and if enabled, 2. 2. PEs participate in the computation in parallel. If the computation is not completed, the small matrix computation is repeated until it is completed.

[0073] In this embodiment of the invention, 2 2. PE serves as the basic unit, expanding the matrix size according to different needs. It supports both large and small matrix multiplication modes, and when dealing with odd matrices, it only needs to be padded to an even matrix. It uses a 16-dimensional tensor core. Compared to the 16 fixed blocks that require extensive zero-value filling, this invention improves computational efficiency and resource utilization. It enables seamless switching between OS and WS data streams in zero cycles, adapting to different matrix computation scenarios in Transformer. When dealing with small or sparse matrices, this invention can disable some processing units as needed, saving power and improving resource utilization, whereas the pulsating array of the TPU (Tensor Processing Unit) cannot perform PE enable control.

[0074] The following will explain the differentiated processing mechanism of the matrix multiplier provided by this invention for large-scale and small-scale matrix multiplication under the Transformer model training and inference co-optimization architecture:

[0075] When dealing with large matrices: treat the expanded matrix multiplier as a whole to handle the matrix multiplication after the large matrix is ​​divided into blocks, 2 2 PE processing unit matrix multiplier and 2 2. PE processes the exchange of activation and weight values ​​between unit matrix multipliers to improve data reusability and efficiency, such as performing two 4x4 multipliers. When multiplying 4 matrices, 4 The 4 PE processing unit matrix multiplier will be configured in large matrix calculation mode, in which case the calculation result will be obtained quickly after 4 cycles;

[0076] When dealing with small matrices: traditional tensor cores require filling blocks, leading to resource waste, while traditional systolic arrays have a large number of idle physical elements (PEs). In this case, expansion units will be configured to... 2 PE processing unit matrix multipliers are used as the smallest granularity, 2 2 PE processing unit matrix multiplier and 2 In 2PE processing units, the matrix multipliers do not exchange activation and weight values; the matrix to be processed is multiplied by 2... 2. Divide the data into blocks, and then directly feed the blocks into different 2. The PE processing unit matrix multiplier calculates and adds the block matrices to obtain the result. For example, when performing two 4... When multiplying 4 matrices, 8 The 8 PE processing unit matrix multiplier will be configured in small matrix calculation mode, in which case each input matrix will be decomposed into four 2x2 matrices. A 2x2 block matrix, these 8 block matrices will be fed into 8 2x2 blocks simultaneously. 2 PE processing unit matrix multipliers, with 8 remaining 2 PEs. The 2PE processing unit matrix multiplier will not be enabled to save power; after 2 cycles, the 8 2... The product results of the matrix multipliers in the PE processing unit are added in a specific order to obtain two 4s. 4. Matrix product results.

[0077] In this embodiment of the invention, the activation values ​​and weight values ​​in the activation matrix and weight matrix are sent to each processing unit. The parallelism is improved by the simultaneous calculation by each processing unit, thereby improving the energy efficiency ratio. It can easily switch between OS and WS data streams in zero cycles and adapt to different matrix calculation scenarios in Transformer.

[0078] Furthermore, the 2×2 processing unit matrix multiplier provided in this embodiment of the invention can be used as a basic unit and can be flexibly expanded into a larger matrix multiplier according to actual needs, in order to deal with activation matrices and weight matrices of different sizes, thereby improving the flexibility of the matrix multiplier.

[0079] This invention provides a scalable 2×2 processing unit matrix multiplier. Targeting the numerous idle PE units in a systolic array, it utilizes parallel computation across a large number of computing modules, fully leveraging the idle PE units and improving computational efficiency. For multiplying two 4×4 matrices, the systolic array solution requires 16 multiplies and 10 computation cycles; the tensor core solution requires 64 multiplies and 1 computation cycle, while this invention requires only 16 multiplies and 4 computation cycles. This invention significantly improves energy efficiency when computing small matrices. This invention supports modular expansion into a large-scale computing array, which can then be configured for either large or small matrix multiplication modes to handle different matrix operation tasks. The expansion module uses 2×2 PE units as small-granularity processing units and employs partial padding techniques to handle small-scale irregular matrices, reducing padding waste and improving resource utilization. This invention achieves zero-cycle switching of OS / WS data streams through a hardware-level dual-mode register group to handle different computational tasks during Transformer model training. This invention can partially enable the 2×2 processing unit based on the resource requirements of different computational tasks. PE reduces power consumption and improves energy efficiency.

[0080] Based on the same inventive concept, this embodiment of the invention also provides a Transformer acceleration method, which is implemented based on the aforementioned 2×2 processing unit matrix multiplier.

[0081] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0083] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0084] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0085] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A scalable 2×2 processing unit matrix multiplier, characterized in that, The 2×2 processing unit matrix multiplier includes a control module and four processing units; When the control module executes the OS data stream mode, the four processing units perform the following operations: During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the first multiplication and addition result; the activation value and weight value are determined based on the input activation matrix and weight matrix; the size of the activation matrix and the weight matrix are both 2×2; During the second clock cycle, each processing unit performs a multiplication operation on the activation value and weight value according to the processing unit interconnected with itself to obtain the first operation result, and then accumulates the first operation result with the first multiplication-addition result to obtain the product result of the two matrices; When the control module executes the WS data stream mode, the four processing units perform the following operations: During the first clock cycle, each processing unit performs multiplication and addition processing based on the received activation value and weight value to obtain the second multiplication and addition result; During the second clock cycle, each processing unit locks its weight value unchanged and performs a multiplication operation with the activation value according to the processing unit interconnected with itself to obtain a second operation result. The second operation result is then accumulated with the second multiplication-addition result to obtain a matrix multiplication result based on fixed weights. The four processing units include a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit; The first processing unit and the second processing unit are interconnected by activation values, the third processing unit and the fourth processing unit are interconnected by activation values, the first processing unit and the third processing unit are interconnected by weight values, and the second processing unit and the fourth processing unit are interconnected by weight values. The first processing unit includes a first activation value register, a first weight value register, a first intermediate calculation result register, a first data selector, a second data selector, a first multiplier, and a first adder; In this system, the second input terminal of the first data selector receives the weight value; the output terminal of the first data selector is connected to the input terminal of the first weight value register; the first output terminal of the first weight value register is connected to the first input terminal of the first data selector; the second output terminal of the first weight value register is connected to the first input terminal of the second data selector and the fourth input terminal of the sixth data selector in the third processing unit; the input terminal of the first activation value register receives the activation value; the output terminal of the first activation value register is connected to the second input terminal of the second data selector and the third input terminal of the fourth data selector in the second processing unit; and the third input terminal of the second data selector is connected to the... The output of the second activation value register is connected to the output of the third weight value register in the third processing unit. The fourth input of the second data selector is connected to the second output of the third weight value register in the third processing unit. The first output of the second data selector is connected to the first input of the first multiplier. The second output of the second data selector is connected to the second input of the first multiplier. The output of the first multiplier is connected to the first input of the first adder. The second input of the first adder is connected to the second output of the first intermediate calculation result register. The output of the first adder is connected to the input of the first intermediate calculation result register. The first output of the first intermediate calculation result register outputs the calculation result in the first processing unit. The second processing unit includes a second activation value register, a second weight value register, a second intermediate calculation result register, a third data selector, a fourth data selector, a second multiplier, and a second adder; In this system, the second input terminal of the third data selector receives a weight value, and the output terminal of the third data selector is connected to the input terminal of the second weight value register. The first output terminal of the second weight value register is connected to the first input terminal of the third data selector, and the second output terminal of the second weight value register is connected to the first input terminal of the fourth data selector and the fourth input terminal of the eighth data selector in the fourth processing unit. The input terminal of the second activation value register receives an activation value, and the output terminal of the second activation value register is connected to the second input terminal of the fourth data selector and the third input terminal of the second data selector in the first processing unit. The third input terminal of the fourth data selector is connected to the first processing unit. The output of the first activation value register is connected to the output of the fourth data selector. The fourth input of the fourth data selector is connected to the second output of the fourth weight value register in the fourth processing unit. The first output of the fourth data selector is connected to the first input of the second multiplier. The second output of the fourth data selector is connected to the second input of the second multiplier. The output of the second multiplier is connected to the first input of the second adder. The second input of the second adder is connected to the second output of the second intermediate calculation result register. The output of the second adder is connected to the input of the second intermediate calculation result register. The first output of the second intermediate calculation result register outputs the calculation result in the second processing unit. The third processing unit includes a third activation value register, a third weight value register, a third intermediate calculation result register, a fifth data selector, a sixth data selector, a third multiplier, and a third adder; In this system, the second input terminal of the fifth data selector receives a weight value, and the output terminal of the fifth data selector is connected to the input terminal of the third weight value register. The first output terminal of the third weight value register is connected to the first input terminal of the fifth data selector, and the second output terminal of the third weight value register is connected to the first input terminal of the sixth data selector and the fourth input terminal of the second data selector in the first processing unit. The input terminal of the third activation value register receives an activation value, and the output terminal of the third activation value register is connected to the second input terminal of the sixth data selector and the third input terminal of the eighth data selector in the fourth processing unit. The third input terminal of the sixth data selector is connected to the... The output of the fourth activation value register is connected to the output of the sixth data selector. The fourth input of the sixth data selector is connected to the second output of the first weight value register in the first processing unit. The first output of the sixth data selector is connected to the first input of the third multiplier. The second output of the sixth data selector is connected to the second input of the third multiplier. The output of the third multiplier is connected to the first input of the third adder. The second input of the third adder is connected to the second output of the third intermediate calculation result register. The output of the third adder is connected to the input of the third intermediate calculation result register. The first output of the third intermediate calculation result register outputs the calculation result in the third processing unit. The fourth processing unit includes a fourth activation value register, a fourth weight value register, a fourth intermediate calculation result register, a seventh data selector, an eighth data selector, a fourth multiplier, and a fourth adder. In this system, the second input terminal of the seventh data selector receives the weight value, and the output terminal of the seventh data selector is connected to the input terminal of the fourth weight value register. The first output terminal of the fourth weight value register is connected to the first input terminal of the seventh data selector, and the second output terminal of the fourth weight value register is connected to the first input terminal of the eighth data selector and the fourth input terminal of the fourth data selector in the second processing unit. The input terminal of the fourth activation value register receives the activation value, and the output terminal of the fourth activation value register is connected to the second input terminal of the eighth data selector and the third input terminal of the sixth data selector in the third processing unit. The third input terminal of the eighth data selector is connected to the... The output of the third activation value register is connected to the output of the second weight value register in the second processing unit. The output of the eighth data selector is connected to the first input of the fourth multiplier. The output of the eighth data selector is connected to the second input of the fourth multiplier. The output of the fourth multiplier is connected to the first input of the fourth adder. The second input of the fourth adder is connected to the second output of the fourth intermediate calculation result register. The output of the fourth adder is connected to the input of the fourth intermediate calculation result register. The first output of the fourth intermediate calculation result register outputs the calculation result in the fourth processing unit.

2. The scalable 2×2 processing unit matrix multiplier according to claim 1, characterized in that, When the weight matrix and the activation matrix are both 4×4, the weight matrix and the activation matrix are processed using four 2×2 processing unit matrix multipliers.

3. The scalable 2×2 processing unit matrix multiplier according to claim 1, characterized in that, When the weight matrix and the activation matrix are both 8×8, the weight matrix and the activation matrix are processed using eight 2×2 processing unit matrix multipliers.

4. A Transformer acceleration method, characterized in that, The Transformer acceleration method is implemented based on the scalable 2×2 processing unit matrix multiplier as described in any one of claims 1-3.

Images