Matrix multiplier for transformer-based model training

Abstract

This invention provides a matrix multiplier for training Transformer-type models, comprising an M-row, N-column systolic array. The systolic array is two-dimensional and consists of R-row, C-column interconnected processing units (PEs). Each PE includes one multiplier, one adder, two internal registers, one left-side multiplexer, and two right-side multiplexers. The left-side multiplexer can select whether the input to the multiplier comes from outside the PE or retains the input from the previous cycle. When retaining the input from the previous cycle, the PE maintains the WS data stream with weights. This invention designs a reconfigurable processing unit (PE) that can flexibly support multiple data streams at different stages and cycles of training and select the data source according to requirements.

Description

Technical Field

[0001] This invention relates to a matrix multiplier for training Transformer-type models. Background Technology

[0002] The Transformer is a classic model for NLP (Natural Language Processing) proposed by Google in 2017. The Transformer model uses a self-attention mechanism, allowing for parallel training and the acquisition of global information about the samples. Popular models such as BERT and GPT are also based on the Transformer architecture.

[0003] In recent years, Transformer-based deep neural networks (DNNs) have achieved remarkable results in fields such as natural language processing (NLP), computer vision (CV), and speech processing. Transformer-based models are typically pre-trained on large-scale datasets and then fine-tuned for downstream tasks. With the expanding application scenarios of Transformer-type models, training (fine-tuning) these models on edge platforms has become crucial, considering data privacy and real-time processing requirements. However, due to the massive number of parameters and high computational complexity of Transformer-type models, deploying the fine-tuning training process on resource-constrained edge platforms presents numerous challenges. The majority of computation in Transformer-type models consists of matrix multiplications. Due to the varying sample lengths in NLP tasks and the matrix transpose characteristics at different training stages, matrix multiplication calculations during model training exhibit irregularities. Deploying these computations to traditional matrix computation architectures leads to low resource utilization and computational throughput. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a matrix multiplier for training Transformer-type models, comprising an M-row N-column systolic array, wherein the systolic array is two-dimensional and is composed of R-row C-column interconnected processing elements (PEs), where M, N, R, and C are all positive integers, and M is usually set to an integer power of 2.

[0005] Each processing unit (PE) includes one multiplier, one adder, two internal registers, one left-side multiplexer, and two right-side multiplexers.

[0006] The left-side multiplexer allows selection of whether the multiplier's input b comes from outside the processing unit PE or retains the input from the previous cycle. When the input from the previous cycle is retained, the processing unit PE maintains the WS data stream with weights.

[0007] The right-hand multiplexer allows selection of whether the adder's input c comes from another processing unit PE or the output of the adder in the previous cycle of this processing unit PE. When the output of the adder in the previous cycle of this processing unit PE is selected, the processing unit PE maintains the OS data stream with its output.

[0008] The right-side multiplexer within the processing unit PE is used to transfer the calculation results between processing units PE for output to the outside of the pulsating array.

[0009] The a_in input of each processing unit PE in the same row of the pulse array comes from the broadcast input on the left side of the pulse array. That is, the input port a_in of each processing unit PE is connected to the data input on the left side of the pulse array, and the data input by the input port a_in between columns is not transmitted. The input data of different rows comes from different input ports on the left side of the pulse array. That is, the processing units PE in the first row share one input port a_in, the processing units PE in the second row share one input port a_in, and the processing units PE in the third row share one input port a_in.

[0010] Each processing unit PE located in the same column has its input b_in coming from the broadcast input above the pulsating array, or from the b_out output of the processing unit PE above it, controlled by a multiplexer between the processing units PE;

[0011] When the broadcast input above the systolic array is selected, the systolic array operates to maintain the OS data stream at the output. When the b_out output from the processing unit PE above is selected, the systolic array is used to preload input data for subsequent calculations of the weight-maintaining WS data stream.

[0012] The input c_in of each processing unit PE located in the same column, except for the processing unit PE in the first row, comes from the c_out output of the processing unit PE above it. The input c_in of the first row comes from outside the systolic array. The transmission of data c between rows can be used for partial sum transmission under weight-preserving WS data stream and result transmission under output-preserving OS data stream.

[0013] The c_in input of the first processing unit PE comes from above the pulsating array.

[0014] When the systolic array operates with weighted WS data streams, it includes the weighted data preloading and weight holding phases.

[0015] Weight data preloading phase: During the weight data preloading phase, the internal structure and connection of the systolic array are organized as follows: weight data from above the systolic array is input to the b_in input port of the first row processing unit PE; the b_in input of the remaining rows of processing units PE comes from the b_in input of the processing unit above in the previous cycle; and the b_out output is for the current cycle. It takes N clock cycles to load the weights into all processing units PE. Setting the input matrix. This represents a 4x3 real matrix, and the weight matrix is... Output matrix During the weight loading phase, the elements of the weight matrix are loaded row by row into the corresponding processing unit PE in the systolic array. It takes 3 clock cycles for the weights to be loaded into all processing units PE.

[0016] Weight Preservation Phase: After the elements of the weight matrix are loaded into the corresponding processing unit PE in the systolic array, the weight preservation phase begins. At this time, the internal structure and connections of the systolic array are organized as follows: the b-input port of the multiplier within the processing unit PE remains unchanged, holding the corresponding weight data; the input matrix is ​​broadcast along each row of the systolic array, with a one-cycle time difference between rows; partial sums propagate along the c_in and c_out ports between rows, and the calculation result is output along the processing unit PE below the systolic array; the input matrix is ​​then set. weight matrix After summing the partial sums from each row, the final result y is obtained. 11 =x 11 w 11 +x 12 w 21 +x 13 w 31 Output from the c_out port of PE3 in the lower left corner. 31 This indicates the processing unit PE in the 3rd row and 1st column; x 13 It is the element in the 1st row and 3rd column of the input matrix x, w 31 It is the element in the 3rd row and 1st column of the weight matrix W.

[0017] When the systolic array operates to maintain the OS data stream at the output, this includes the multiply-accumulate and result output phases within the processing unit PE.

[0018] Multiplication and Accumulation Stage within Processing Units (PEs): During the multiplication and accumulation stage within processing units (PEs) that maintain the OS data stream at output, the internal structure and connections of the systolic array are organized as follows: Weight data from above the systolic array is simultaneously broadcast to the b_in input of all processing units (PEs) in one column; input data from the left side of the systolic array is simultaneously broadcast to the a_in input of all processing units (PEs) in the same row; the c input of the adder inside the processing unit (PE) is connected to the output of the adder from the previous cycle, forming an accumulator; weight data and input data are updated every cycle, and the product of weight data and input data is accumulated within the processing unit (PE) until the final result is obtained; during the multiplication and accumulation stage within processing units (PEs) that maintain the OS data stream at output, there is no time difference between the input and weight data in each row and column; the input matrix is ​​set. weight matrix Output matrix During each cycle of the multiply-accumulate phase, the effective input data and weight data of each processing unit (PE) within the pulsating array are organized as follows: Each cycle, the input and weight data are updated and simultaneously broadcast to the input of the multipliers of all processing units (PE) in their respective rows or columns, and the calculated result y... 11 =x 11 w 11 +x 12 w 21 +x 13 w 31 +x 14 w 41 PE in the upper left corner of the pulsating array 11 PE is obtained by internal accumulation. 11 This indicates the processing unit PE in the first row and first column;

[0019] Result Output Stage: After all input data and weight data are transmitted into the systolic array, the element-wise products are accumulated within the processing unit PE to obtain the final result. At this point, the result output stage begins. The calculation result needs to be output from the bottom of the systolic array along the c-ports of each column of processing unit PE. The internal structure and connection of the result output systolic array are organized as follows: the c_out output of the right half of each processing unit PE is no longer connected to the output of the adder, but is instead connected to the c_out output of the previous cycle of the processing unit PE above it, thus completing the shifted output of the calculation result of each processing unit PE within the systolic array. Taking the calculation process described above as an example, the result is output along the columns from the c_out port of the processing unit PE in the bottom row of the systolic array.

[0020] An M-row, N-column systolic array is interconnected via an on-chip network to form a matrix multiplication module; the matrix multiplication module is connected to the input memory block and the weight memory block respectively.

[0021] The input memory block is used to store input data transferred from off-chip DRAM to the on-chip, including M on-chip buffers for storing input data transferred from off-chip DRAM to the on-chip. The data in the input memory block is transmitted to each row of systolic arrays through an input allocation network and connected to the a_in input port on the left side of the systolic array. Each row of systolic arrays shares the same input allocation network. The input allocation network is a module containing M input ports and M output ports, which can realize the connection from any input port to any output port and is responsible for transmitting input data from different memory blocks to the corresponding systolic arrays.

[0022] The weighted memory block is used to store weighted data transferred from off-chip DRAM to the on-chip, and includes N×C on-chip buffers. It is transmitted to each row of systolic arrays through a weight allocation network and connected to the b input port above the systolic array. Systolic arrays in the same column share a weight allocation network, while systolic arrays in different columns use different weight allocation networks. The weight allocation network is a module containing C input ports and C output ports, with a total of N modules. It can realize the connection from any input port to any output port and is responsible for transmitting weighted data from different memory blocks to the corresponding systolic arrays.

[0023] The c_out output port below the systolic array is connected to a result generation network, which is used to transmit the calculation results of the systolic array to the corresponding output memory block.

[0024] The output memory block is connected to the matrix multiplication module and is used to store the result data after the calculation is completed, including N×C on-chip caches.

[0025] The input allocation network and weight allocation network are responsible for transmitting data from different memory blocks to the corresponding systolic array;

[0026] The number of rows of the systolic array in the matrix multiplication module is set to M, and the number of corresponding input memory blocks is also M. An M M-input M M-output M M-input distribution network is used to complete the routing of input data. A configurable multiplexer with M M inputs is used.

[0027] The number of columns in the systolic array in the matrix multiplication module is set to N, the number of columns in the processing unit PE within the systolic array is C, and the number of corresponding weight memory blocks is N×C. There are a total of N weight allocation networks. Each weight allocation network is a CC network containing C input ports and C output ports. The weight allocation network is used to complete the routing of weight data and contains a configurable multiplexer with C C inputs.

[0028] When M=4, a 4-4 input distribution network contains 4 configurable multiplexers. Data sent to different rows comes from any memory block input and is determined by the control signals of the multiplexers.

[0029] The result generation network can post-process the calculation results of the systolic array. When the systolic array in the matrix multiplication module is in the output holding OS data stream, the systolic array can obtain the final result of the matrix multiplication calculation and directly transmit it to the output memory using the result generation network. When the systolic array is in the weight holding WS data stream, the result generation network can realize the addition of any combination of input data and output the result through a configurable addition tree.

[0030] The basic unit of the result generation network includes an adder with bypass, which uses a basic adder and a multiplexer, with two input ports and two output ports. When the adder operates in the first state, it can pass the two input data to the two output ports unchanged. When the adder operates in the second state, the upper port of the adder outputs the sum of the two input values, and the lower port outputs one of the input values. By vertically flipping the adder in the second state, the third state of operation of the adder is obtained, where the lower port of the adder outputs the sum of the two input values, and the upper port outputs one of the input values.

[0031] Based on the adder with bypass, a configurable result generation network is constructed to add any combination of input values;

[0032] When the input to the result generation network is M, there are M input ports on the left side of the result generation network, which are respectively connected to the M / 2 adders with bypasses in the first column. The adders and multiplexers in subsequent stages are conventional structures. The input ports of each stage are connected to the output ports of the previous stage. Registers are inserted between the modules to form a pipeline structure to improve the frequency and computational throughput. The output of the network can come from the output port of any internal module and is determined by the control signals during runtime.

[0033] This invention addresses the irregular matrix multiplication computations during Transformer-type model training by proposing a flexible and reconfigurable hardware architecture. Based on specially designed processing units and flexible data allocation and synthesis networks, it can flexibly adjust the data flow and mapping scheme according to the characteristics of the computational load, thereby better utilizing computing resources and achieving higher performance and energy efficiency at each stage of training. The solution proposed in this invention is the first of its kind and is effective.

[0034] The present invention has the following beneficial effects:

[0035] (1) A reconfigurable processing unit (PE) was designed, which can flexibly support multiple data streams at different stages and cycles of training and select data sources according to needs.

[0036] (2) Based on the reconfigurable processing unit, a systolic array that can flexibly support multiple data streams is proposed, and a broadcast structure for input data is adopted to reduce the transmission time of input data and the resource overhead of internal registers.

[0037] (3) Multiple pulsating arrays are integrated into a matrix multiplication module by using an outward expansion method, which improves computing power while maintaining a high level of hardware resource utilization.

[0038] (4) An input / weight allocation network and a result generation network were designed between the matrix multiplication module and the data storage module. The data allocation network can flexibly map the data to the systolic array according to the computational load characteristics. The result generation network can post-process the computation results and pass the results to the output buffer. Attached Figure Description

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0040] Figure 1 This is a schematic diagram of the internal structure of the reconfigurable processing unit.

[0041] Figure 2 This is a schematic diagram of a 3×3 pulsating array structure.

[0042] Figure 3 This is a schematic diagram of WS data flow and weight preloading.

[0043] Figure 4 This is a schematic diagram of the input matrix, weight matrix, and output matrix.

[0044] Figure 5 This is a diagram illustrating weight preloading.

[0045] Figure 6 This is a schematic diagram of WS data flow and weight preservation.

[0046] Figure 7 This is a schematic diagram of the data flow where the weights are maintained for the first two cycles.

[0047] Figure 8 This is a schematic diagram of OS data flow - PE multiplication and accumulation.

[0048] Figure 9 This is a schematic diagram of the input matrix, weight matrix, and output matrix.

[0049] Figure 10This is a schematic diagram of the data flow in each cycle of the PE multiplication and accumulation stage.

[0050] Figure 11 This is a diagram illustrating the OS data flow and result output.

[0051] Figure 12 This is a schematic diagram of the process of transmitting the result.

[0052] Figure 13 This is a schematic diagram of the matrix multiplication module.

[0053] Figure 14 This is a schematic diagram of a 4-4 input distribution network.

[0054] Figure 15 This is a schematic diagram of an adder with a bypass.

[0055] Figure 16 This is a schematic diagram of the network that generates the results.

[0056] Figure 17 This is a schematic diagram illustrating an example of a network application for generating results.

[0057] Figure 18 This is a schematic diagram of a 4×4 traditional pulse array and WS data flow. Detailed Implementation

[0058] This invention provides a matrix multiplier for training Transformer-type models, comprising:

[0059] 1. A flexible, reconfigurable processing unit

[0060] A systolic array consists of many interconnected processing elements (PEs). In arithmetic operations related to convolution or matrix multiplication, these processing elements are typically multiply-accumulator (MAC) units. MAC units can perform multiplication and addition operations simultaneously within a single cycle, improving computational efficiency in calculations such as matrix multiplication. A MAC unit usually consists of a multiplier and an adder; the multiplier calculates the product of two numbers, and the adder accumulates the multipliers. Typical systolic array architectures support only one type of data flow (e.g., TPUs (Tensor Processing Units) use a WS data flow) due to their simple processing element structure and interconnection. However, in tasks related to Transformer training, different network layers and different training stages require different types of data flows for matrix multiplication calculations. Experiments demonstrate that during the training of a Transformer linear layer, matrix multiplication in the forward propagation (FP) and backward propagation (BP) stages is suitable for OS data flow, while matrix multiplication in the weight gradient computation (WG) stage is suitable for weight stationary (WS) data flow. Therefore, systolic arrays supporting flexible data flow can effectively improve the computational efficiency of matrix multiplication and the utilization of hardware resources.

[0061] This invention proposes a flexible and reconfigurable processing unit (PE) design, which can reconfigure its internal structure according to needs at different stages of training and different computation cycles to operate under different data flow modes. Based on this PE design, a systolic array supporting multiple data flows can be constructed. The structure of the PE is as follows: Figure 1 As shown, the PE contains one multiplier, one adder, two internal registers (D), one multiplexer (①) on the left, and two multiplexers (② and ③) on the right. The input 'a' (a_in) from outside the PE is connected to port 'a' of the multiplier. The input 'b' (b_in) from outside the PE is connected to port 'b' of the multiplier via the left multiplexer (①) and the register. The output of the multiplier is connected to one of the input ports of the adder. The input 'c' (c_in) from outside the PE is connected to port 'c' of the adder via the right multiplexer (②). The output of the adder is connected to port 'c' (c_out) of the PE via the right multiplexer (③) and the register. All internal modules can use different data formats, such as integers or floating-point numbers, depending on the computational needs. The PE adjusts its internal structure using multiplexers (MUX) to support different data streams.

[0062] The left-hand MUX(①) can select whether the multiplier input b comes from outside the PE or retains the input from the previous cycle. When the latter is selected, the PE operates with the WS data stream.

[0063] The MUX (②) on the right can select whether the input c of the adder comes from other PEs or the output of the adder in the previous cycle of this PE. When the latter is selected, the PE maintains the OS (Output Stationary, OS) data stream with the output. The MUX (③) on the right can transfer the calculation result between PEs to output to the outside of the array.

[0064] 2. Pulsating array supporting multiple data streams

[0065] Based on the reconfigurable processing unit, this invention proposes a systolic array that can flexibly support multiple data streams and adopts a broadcast structure for input data to reduce the transmission time of input data and the resource overhead of internal registers.

[0066] Taking a 3×3 pulsating array as an example, PE ij This represents the PE in the i-th row and j-th column. Its structure and the interconnection between PEs are as follows: Figure 2 As shown:

[0067] 1) The input 'a' of each PE in the same row comes from the broadcast input on the left side of the array, meaning that the input port 'a' of each PE is connected to the data input on the left side of the array, and the 'a' data is not passed between columns. The input data of different rows comes from different input ports on the left side of the array, meaning that the PEs in the first row share one 'a' input, the PEs in the second row share one 'a' input, and the PEs in the third row share one 'a' input.

[0068] 2) For each PE located in the same column, its input b can come from the broadcast input above the array or from the b output of the PE above it, controlled by a multiplexer between the PEs. When the former is selected, the array operates with the OS data stream; when the latter is selected, the array is used for input data preloading for subsequent calculations of the weight-preserving WS data stream.

[0069] 3) For each PE located in the same column, except for the PE in the first row, its input c comes from the output c of the PE above it. The transfer of data c between rows can be used for partial sum transfer under WS data stream and result transfer under OS data stream. The input c of the PE in the first row comes from above the array, which can be determined according to the calculation requirements.

[0070] The specific size of the systolic array can be adjusted according to the matrix multiplication calculation load, defined as R×C, where R represents the number of rows and C represents the number of columns. The following will still use a 3×3 systolic array as an example to introduce its calculation process under WS and OS data streams.

[0071] 2.1 Pulsating Array of WS Data Stream

[0072] When a systolic array operates with a WS data stream, its computation is mainly divided into two stages: weight data preloading and weight retention.

[0073] (1) Weighted data preloading

[0074] When the calculation is in the weighted data preloading phase, the internal structure and connection method of the array are as follows: Figure 3 As shown: Solid lines represent valid data paths selected by the MUX, and dashed lines represent invalid paths. In this case, weight data from above the array is input to the b_in input port of the first row of PEs. The b_in inputs of the PEs in the remaining rows come from the b_in input of the PE above them in the previous cycle, and the b_out output of the current cycle. It takes 3 clock cycles to load the weights onto all PEs. Assume the input matrix... weight matrix Output matrix like Figure 4 As shown. During the weight loading phase, the elements of the weight matrix are loaded row by row into the corresponding PEs in the array, as follows. Figure 5 As shown.

[0075] (2) Weight preservation

[0076] After the elements of the weight matrix are loaded into the corresponding PEs in the array, the weight preservation phase begins. At this time, the internal structure and connection method of the array are as follows: Figure 6 As shown: Solid lines represent valid data paths selected by the MUX, and dashed lines represent invalid paths. The b_in input port of the multiplier within the PE remains constant with the corresponding weight data. The input matrix is ​​broadcast along each row of the array, with a one-cycle time difference between the input data of each row. Partial sums propagate along the c_in and c_out ports between rows, and the computation result is output along the PE below the array. The data flow for the first two cycles of the weight holding phase is as follows: Figure 7 As shown, after accumulating the partial sums between each row, the final result y is... 11 =x 11 w 11 +x 12 w 21 +x 13 w 31 The output is from the c_out port of PE31 in the lower left corner, and the calculation process of other elements in the result matrix is ​​similar.

[0077] 2.1 Pulsating Array of OS Data Stream

[0078] When the systolic array operates with OS data stream, its calculation process is mainly divided into two stages: PE multiplication and accumulation and result output.

[0079] (1) PE multiplication and accumulation

[0080] During the multiplication and accumulation phase within the PE of the OS data stream, the internal structure and connection method of the array are as follows: Figure 8 As shown: Weight data from the top of the array is simultaneously broadcast to the b_in input of all PEs in a column, and input data from the left side of the array is simultaneously broadcast to the a_in input of all PEs in the same row. The c input of the adder inside the PE is connected to the output of the adder in the previous cycle, forming an accumulator. The weight data and input data are updated every cycle, and their product is accumulated inside the PE until the final result is calculated. During this stage, there is no time difference between the input and weight data in each row and column. Assume the input matrix... weight matrix Output matrix like Figure 9 As shown. During each cycle of the multiply-accumulate phase, the effective input data and weight data of each PE within the array are as follows: Figure 10 As shown, each cycle's input and weight data is updated and simultaneously broadcast to the inputs of all PE cell multipliers in its row or column. The calculation result y 11 =x 11 w 11 +x 12 w 21 +x 13 w 31 +x 14 w 41 The result is obtained by accumulating the values ​​in PE11 at the top left corner of the array; the rest of the results are similar.

[0081] (2) Results are transmitted

[0082] After all input data and weight data are transmitted into the array, the element-wise products are accumulated within the PE to obtain the final result. At this point, the result output stage begins, where the calculation result is output from the bottom of the array along the c_in and c_out ports of each column of the PE. The internal structure and connection method of the array in this stage are as follows: Figure 11 As shown: the c_out output of the right half of each PE is no longer connected to the output of the adder, but is instead connected to the c_out output of the previous cycle of the PE above it. This structure completes the shifting and output of the calculation results of each PE within the array. Taking the calculation process described above as an example, the result output process is as follows... Figure 12 As shown.

[0083] 3. Matrix Multiplication Module

[0084] A systolic array composed of reconfigurable physical components (PEs) can flexibly adapt to the computational needs of different training stages. However, the irregular matrix computation characteristics during Transformer-type model training still affect computational efficiency. Simply increasing the size of the systolic array to improve computational efficiency will lead to a mismatch between matrix mapping and array size, especially when the sample sequence length is small. Using a very large systolic array will significantly reduce computational efficiency and hardware resource utilization. This invention uses a scale-out approach to organize the systolic array, interconnecting multiple systolic array modules through an on-chip network, thereby improving computational power while maintaining high resource utilization.

[0085] The matrix multiplication module and its matching peripheral memory module structure are as follows: Figure 13 As shown:

[0086] The input memory bank consists of multiple on-chip buffers used to store input data transferred from off-chip DRAM. Data in the input memory bank is transmitted to each row of systolic arrays via an input allocation network and connected to the 'a' input port on the left side of the array. Each row of systolic arrays shares the same input allocation network.

[0087] The weight memory block stores weight data, which is transmitted to each row of the systolic array via the weight allocation network and connected to the b-input port above the array. Systolic arrays in the same column share a single weight allocation network, while systolic arrays in different columns use different weight allocation networks.

[0088] The result generation network is connected to the C output port below the array and is used to transfer the computation results of the systolic array to the corresponding output memory block. When the systolic array operates using WS data streams, the computation results of different rows of the systolic array may need to be further added to obtain the final result. Moreover, it is uncertain which rows' results need to be added. Therefore, the result generation network is a flexible and reconfigurable addition tree that can add any combination of input data and output the result.

[0089] 3.1 Input and Weight Assignment Network

[0090] The input and weight allocation network is responsible for transferring data from different memory blocks to the corresponding systolic array. Unlike the traditional method of one-to-one correspondence between memory blocks and array rows / columns, this invention uses a special input and weight allocation network to flexibly transfer data to any row / column of the matrix multiplication module in order to improve computational efficiency.

[0091] Assuming the systolic array in the matrix multiplication module has 4 rows, and the corresponding number of input memory banks is also 4, this invention uses a 4-input, 4-output 4-to-4 input distribution network to route the input data. The internal logic structure of the 4-to-4 input distribution network is as follows: Figure 14 As shown: It contains four configurable multiplexers. Data sent to different rows can come from any memory block input, determined by the control signals of the multiplexers. The structure of the weight allocation network is the same as that of the input allocation network. Data from multiple weight memory blocks can be transmitted to the systolic array of any row in the same column of the matrix multiplication module.

[0092] 3.2 Results Generation Network

[0093] The result generation network is responsible for transmitting the computation results of the systolic array to the corresponding output memory block and can perform post-processing on the computation results of the systolic array. When the systolic array in the matrix multiplication module is in the OS data stream, the systolic array can obtain the final result of the matrix multiplication calculation and directly transmit it to the output memory using the result generation network. However, when the systolic array is in the WS data stream, its computation result may only be a part of the final valid result and needs to be added to the computation results of other rows of the systolic array. Furthermore, which rows' results need to be further added is uncertain and needs to be adjusted according to the distribution of computational load. Therefore, the result generation network includes a flexible and configurable addition tree, which can realize the addition of arbitrary combinations of input data and output the result.

[0094] The result generation network in this invention uses an adder with a bypass, the structure of which is as follows: Figure 15 The leftmost diagram shows a basic adder and a multiplexer with two input ports and two output ports. When the adder... Figure 15 The structure shown in the second image on the left, when in operation, can pass the two input data points intact to the two output ports; when the adder... Figure 15 When the structure shown in the third image from the left is working, its upper port outputs the sum of two input values, and its lower port outputs one of the input values; when... Figure 15 By vertically flipping the structure shown in the third image from the left, we can obtain... Figure 15 The structure of the output is shown below the fourth image from the left.

[0095] Based on the aforementioned adder with bypass, this invention proposes a flexible and configurable result generation network that can add any combination of input values. Taking a 16-input result generation network as an example, its overall structure is as follows: Figure 16As shown, there are 16 input ports on the left, connected to the first column of 8 adders with bypasses. Subsequent adders and multiplexers are of conventional structure. Registers are inserted between modules to form a pipelined structure to improve frequency and computational throughput. The network output can come from the output ports of any internal module, determined by runtime control signals.

[0096] Figure 17 yes Figure 16 An example application of the input result generation network can be used to calculate x1+x2+x3, x4+x5+x6+x7, x8+x9, x 10 +x 11 +x 12 +x 13 and x 14 +x 15 +x 16 As a result, the first-stage adder with bypass is configured, depending on the requirements, to have either addition functionality or no addition functionality, and... Figure 15 Correspondingly, thick black and thick gray lines represent valid computation paths controlled by the multiplexer. The input to the resulting network is consistent with the number of rows M of the systolic array in the matrix multiplication module, typically set to an integer power of 2 to form a regular structure.

[0097] Example

[0098] Transformer model training involves many irregular matrix calculations. This embodiment uses an irregular matrix multiplication calculation as an example to illustrate the significant improvement in computational efficiency brought about by the matrix multiplication architecture provided by this invention. Under the premise of having the same computing resources, the computational efficiency of a traditional systolic array and the matrix multiplication architecture provided by this invention is compared. The traditional systolic array is set to a size of 16×16 (16 rows, 16 columns) and uses a WS data stream; the matrix multiplication architecture includes M rows and N columns of systolic arrays, each consisting of R rows and C columns of interconnected processing units (PEs), using a WS data stream. M = N = R = C = 4. Both the systolic array and the matrix multiplication architecture contain 256 processing units (PEs). The operation of a 4×4 WS data stream traditional systolic array is as follows... Figure 18 As shown, the elements of the weight matrix are preloaded and stored in each PE before computation begins. Then, the elements of the input matrix flow in from the left edge of the array. Each PE generates a partial sum per cycle, and the resulting intermediate results propagate along the columns.

[0099] Define the input matrix for matrix multiplication. This represents a 4x4 real matrix, the weight matrix. Represent a 4x64 real matrix, and output the matrix. xij It is the element in the i-th row and j-th column of the input matrix X; w ij This refers to the element in the i-th row and j-th column of the weight matrix W, where the index indicates the row and column of the element. When this matrix multiplication is mapped to a traditional 16×16 systolic array, during the weight preloading phase, the elements in rows 1-4 and columns 1-16 of the weight matrix are loaded into the processing units PE of rows 1-4 and columns 1-16 of the traditional systolic array. The remaining 48 columns of the weight matrix cannot be loaded into the remaining processing units PE of the traditional systolic array because the calculation of the remaining 48 columns is unrelated to the calculation of the preceding 16 columns, and the calculation results of processing units PE in the same column of the traditional systolic array need to be added together. The calculation of the subsequent 48 columns needs to wait until the calculation of the preceding 16 columns is completed before proceeding sequentially. The calculation of 16 columns constitutes one round, and the traditional systolic array requires a total of 4 rounds.

[0100] In the matrix multiplication computation architecture proposed in this invention, elements in rows 1-4 and columns 1-16 of the weight matrix can be loaded into four 4×4 systolic arrays in column 1 of the matrix multiplication module; elements in rows 1-4 and columns 17-32 of the weight matrix can be loaded into four 4×4 systolic arrays in column 2 of the matrix multiplication module; elements in rows 1-4 and columns 33-48 of the weight matrix can be loaded into four 4×4 systolic arrays in column 3 of the matrix multiplication module; and elements in rows 1-4 and columns 49-64 of the weight matrix can be loaded into four 4×4 systolic arrays in column 4 of the matrix multiplication module. Since the computation of each row of systolic arrays is independent, the weight data can be loaded into systolic arrays of different rows for separate computation and output. Using the same computational resources, the matrix multiplication computation module proposed in this invention can complete the entire matrix multiplication computation in one round, significantly improving computational efficiency. There are many similar matrix multiplication calculations in the training process of Transformer-type models. The matrix multiplication calculation architecture proposed in this invention can effectively improve the efficiency of model training.

[0101] This invention provides a matrix multiplier for training Transformer-type models. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A matrix multiplier for Transformer-based model training, characterized in that, The systolic array comprises M rows and N columns of processing elements (PEs), which are two-dimensional and interconnected in R rows and C columns, where M, N, R and C are positive integers; Each PE comprises one multiplier, one adder, two internal registers, one left multiplexer and two right multiplexers; The left multiplexer can select the input of the multiplier from outside the PE or from the input of the previous cycle, and when the input of the previous cycle is selected, the PE works in a weight-keeping (WS) data stream; Each PE can select the input of the adder from the PE above or from the output of the adder of the previous cycle by the two internal right multiplexers, and when the output of the adder of the previous cycle is selected, the PE works in an output-keeping (OS) data stream.

2. The matrix multiplier for training of a Transformer-based model according to claim 1, wherein, The right multiplexers in the PE are used to pass the calculation results between the PEs and output to the outside of the systolic array.

3. The matrix multiplier for training of a Transformer-based model according to claim 2, characterized in that, The a_in input of each PE in the same row of the systolic array comes from the broadcast input on the left side of the systolic array, that is, the input port a_in of each PE is connected to the data input on the left side of the systolic array, and the data inputted by the input port a_in between columns is not passed; the input data of different rows comes from different input ports on the left side of the systolic array, that is, the PEs in the first row share one input port a_in, the PEs in the second row share one input port a_in, and the PEs in the third row share one input port a_in; The b_in input of each PE in the same column comes from the broadcast input above the systolic array or from the b_out output of the PE above, which is controlled by the multiplexer between the PEs; When the broadcast input above the systolic array is selected, the systolic array works in an OS data stream, and when the b_out output of the PE above is selected, the systolic array is used for input data preloading for subsequent calculation in a WS data stream; The c_in input of each PE in the same column comes from the c_out output of the PE above except for the PEs in the first row, and the c_in input of the PEs in the first row comes from the outside of the systolic array, and the transmission of the data c between rows can be used for partial sum passing in a WS data stream and result passing in an OS data stream; The c_in input of the PEs in the first row comes from above the systolic array.

4. The matrix multiplier for training of a Transformer-based model according to claim 3, characterized in that, When the systolic array works in a WS data stream, it comprises a weight data preloading stage and a weight keeping stage; Weight data preloading stage: when the calculation is in the weight data preloading stage, the structure and connection mode inside the systolic array are organized in the following way: the weight data from above the systolic array is input to the b_in input port of the first row of processing units PE, the b_in input of the rest rows of processing units PE comes from the b_in input of the processing unit PE above in the last cycle, and the b_out output of the current cycle; It takes N clock cycles to load the weights into all processing elements PE; set the input matrix denotes a 4x3 real matrix, the weight matrix output matrix In the weight loading phase, the elements of the weight matrix are loaded row by row into the corresponding processing elements PE in the systolic array, then it takes 3 clock cycles to load the weights into all processing elements PE; Weight holding stage: when the elements of weight matrix are loaded into the corresponding processing elements (PEs) in systolic array, the weight holding stage is entered, at this time, the structure and connection mode of the internal systolic array is organized in the following way: the b input port of the multiplier in processing element (PE) always holds the corresponding weight data; the input matrix is broadcasted along each row of the systolic array, there is a period of time difference between the input data of each row; the partial sum is propagated between the rows along the c_in and c_out ports, the calculation result is output along the processing elements (PEs) below the systolic array; the input matrix is set as weight matrix the final result y is obtained through the accumulation of the partial sum between each row 11 = x 11 w 11 + x 12 w 21 + x 13 w 31 from the c_out port of the lower left corner PE 31 , the PE 31 represents the processing element (PE) in the 3rd row and 1st column; x 13 is the element in the 1st row and 3rd column of the input matrix X, w 31 is the element in the 3rd row and 1st column of the weight matrix W.

5. The matrix multiplier for training of a Transformer-based model according to claim 4, characterized in that, When the systolic array works in the output holding OS data flow, it includes the processing unit PE multiplication and accumulation inside and the result transmission stage; PE: In the PE inner multiply-accumulate stage of the output hold OS data stream, the structure and connection of the systolic array is organized in the following way: the weight data from above the systolic array is broadcasted to the b_in input of all PEs in a column at the same time, the input data from the left of the systolic array is broadcasted to the a_in input of all PEs in the same row at the same time; the c input of the PE inner adder is connected to the output of the last cycle adder, forming an accumulator; the weight data and input data are updated every cycle, the product of the weight data and input data is accumulated in the PE until the final result is calculated; in the PE inner multiply-accumulate stage of the output hold OS data stream, there is no time difference between the input and weight data of each row and each column; the input matrix weight matrix output matrix In each cycle of the multiply-accumulate stage, the effective input data and weight data of each PE in the systolic array is organized in the following way: the input and weight data are updated and broadcasted to the input of all PE multipliers in the row or column at the same time every cycle, and the calculation result y 11 = x 11 w 11 + x 12 w 21 + x 13 w 31 + x 14 w 41 is accumulated in the PE 11 in the top left corner of the systolic array; PE 11 represents the PE in the first row and the first column. Result transmission stage: when all the input data and weight data are transmitted into the systolic array, the element products are accumulated inside the processing unit PE to obtain the final result, at this time, the result transmission stage is entered, and the calculation result needs to be output from below the systolic array along the c_in and c_out ports of each column of processing units PE, the structure and connection mode inside the result transmission systolic array are organized in the following way: the c_out output of the right half of each processing unit PE is no longer connected to the output of the adder, but to the c_out output end of the processing unit PE above in the last cycle, so as to complete the displacement output of the calculation result of each processing unit PE inside the systolic array.

6. The matrix multiplier for training of a Transformer-based model according to claim 5, wherein, The M-row N-column systolic array is connected to each other through the mode of the network on chip to form a matrix multiplication module; the matrix multiplication module is connected with an input memory block and a weight memory block respectively; The input memory block is used for storing the input data transmitted from the off-chip DRAM to the chip, and includes M on-chip cache areas used for storing the input data transmitted from the off-chip DRAM to the chip; the data in the input memory block is transmitted to each row of systolic arrays through an input distribution network and connected with the a_in input port on the left side of the systolic array; each row of systolic arrays shares the same input distribution network; the input distribution network is a module including M input ports and M output ports, can realize the connection of any input port to any output port, and is responsible for transmitting the input data from different memory blocks to the corresponding systolic array; The weight memory block is used for storing the weight data transmitted from the off-chip DRAM to the chip, and includes NxC on-chip cache areas, which are transmitted to each row of systolic arrays through a weight distribution network and connected with the b input port above the systolic array; the systolic arrays in the same column share a weight distribution network, and the systolic arrays in different columns use different weight distribution networks; the weight distribution network is a module including C input ports and C output ports, and there are N in total, which can realize the connection of any input port to any output port, and is responsible for transmitting the weight data from different memory blocks to the corresponding systolic array; The c_out output port below the systolic array is connected with a result generation network, and the result generation network is used for transmitting the calculation result of the systolic array to the corresponding output memory block; The output memory block is connected with the matrix multiplication module, and is used for storing the result data after the calculation, and includes NxC on-chip cache areas.

7. The matrix multiplier for training of a Transformer-based model according to claim 6, wherein, The input distribution network and the weight distribution network are responsible for transmitting the data from different memory blocks to the corresponding systolic array; The number of rows of the systolic array in the matrix multiplication module is set as M, and the number of blocks of the input memory is also M. An M-M input distribution network with M inputs and M outputs is used to complete the routing of the input data, and the configurable multiplexer with M M inputs is included; The number of columns of the systolic array in the matrix multiplication module is set as N, the number of columns of the processing element (PE) in the systolic array is set as C, and the number of blocks of the weight memory is N×C. There are N weight distribution networks in total, each of which is a C-C network with C input ports and C output ports. The weight distribution network is used to complete the routing of the weight data, and the configurable multiplexer with C C inputs is included.

8. The matrix multiplier for training of a Transformer-based model according to claim 7, characterized in that, When M = 4, the 4-4 input distribution network includes four configurable multiplexers, and the data sent to different rows comes from any memory block input, which is determined by the control signal of the multiplexer.

9. The matrix multiplier for Transformer model training according to claim 7, characterized in that, The result generation network can post-process the calculation results of the systolic array. When the systolic array in the matrix multiplication module is in the output sustain (OS) data flow, the systolic array can obtain the final result of the matrix multiplication calculation, and directly transmit the result to the output memory using the result generation network. When the systolic array is in the weight sustain (WS) data flow, the result generation network realizes the addition of any combination of input data and outputs the result through a configurable addition tree.

10. The matrix multiplier for training of a Transformer-based model according to claim 9, characterized in that, The basic unit of the result generation network includes an adder with a bypass. The adder uses a basic adder and a multiplexer, has two input ports and two output ports. When the adder works in the first state, it can pass the two input data to the two output ports without change. When the adder works in the second state, the upper port of the adder outputs the sum of the two input values, and the lower port outputs one of the input values. The adder in the second state is vertically flipped to obtain the third state of the adder. The lower port of the adder outputs the sum of the two input values, and the upper port outputs one of the input values.

Images