A column reconfigurable systolic array for transformer model

Abstract

The application belongs to the field of information technology and provides a column reconfigurable systolic array for a Transformer model. The main idea is to realize that each column computing unit of the array can work together for a single operator or can be split to work individually for multiple operators. The main scheme includes a reconfigurable computing unit of a two-dimensional network, which is composed of row and column distributed computing units, data is transferred from the previous row to the next row and from the previous column to the next column; a register unit of a two-dimensional network, which is composed of row and column distributed register units, data is transferred from the previous row to the next row and from the previous column to the next column, and the data transfer direction is opposite to that of the reconfigurable computing unit of the two-dimensional network; and the register and the computing unit of the same row and column are connected through a data path. A mixed parallel line is supported to improve the hardware efficiency of the Transformer-based model.

Description

A column-reconfigurable systolic array for Transformer models Technical Field

[0001] This invention belongs to the field of information technology and provides a column-reconfigurable pulsating array for the Transformer model. Background Technology

[0002] Transformer-based models have achieved success in a wide range of deep learning tasks in natural language processing, such as machine translation

[21] , text classification [4], and text generation

[18]

[19] . Such large-scale success is attributed to task-independent model architectures, an increase in the number of encoder and decoder layers, and an increase in vocabulary, all of which can significantly improve the quality of various tasks. At the same time, the arbitrary length of text that language models need to process leads to a large amount of computation and parameters. The complete GPT-3 [2] contains 175 billion parameters and requires 3.14 x 10^23 floating-point operations (FLOPS) to train. If trained on a Tesla V100 cloud instance, a single training run would cost more than $4.6 million. The increasing computational demands place higher demands on the efficiency of using these models on devices.

[0003] Previous work has categorized Transformer acceleration into two paradigms. One approach leverages intra-operator data parallelism on an operator-by-operator basis (Fig. 1a).

[16] optimized operator partitioning for Transformer inference on TPUv4 [9]. GPU performance was improved by using tensor processors, while [3]

[23] focused on memory optimization on GPUs and proposed a fused kernel for attention layers. Popular deep learning frameworks [1]

[14] can only handle rectangular shapes, so variable-length input sequences in batches are padded to the same length before computation. However, the padded zeros add a lot of unnecessary overhead in computation and memory. [6]

[24] reduced padding redundancy by reordering the input sequences during preprocessing, while

[25] eliminated padding on GPUs for linear and attention layers by offsetting variable-length input sequences in memory. These works are inefficient spatially due to the mismatch between fixed-shape computational units and variable-shape workloads, or temporally due to computation time not matching data access time, especially in data-parallel attention layers.

[0004] Another approach resorts to pipelined parallelism between operators (Fig. 1b), where consecutive operators are assigned to different computational units. [5][8]

[12]

[13] Micro-batch processing layer pipelines are used to accelerate the training of deep learning models.

[15] A sub-layer pipeline approximating attention and feedforward networks is constructed. Since the computation of attention layers and linear layers is at least linear in complexity for sequences of variable input lengths, pipelined parallelism at the sub-layer level inevitably leads to severe pipeline stalls, especially when the lengths of consecutive input sequences vary greatly.

[0005] GPUs remain the most widely used hardware for accelerating Transformer models. We analyzed GPU performance with intra-operator data parallelism and made two key findings. First, even with optimized fusion kernels, the attention layer remains inefficient in both time and space. This suggests that the attention layer may benefit more from inter-operator pipeline parallelism than intra-operator data parallelism. Second, highly optimized linear layers achieve only 70% time efficiency and 25% space efficiency. Besides shape mismatch, data parallelism leads to data duplication, which means reduced data reuse, thus limiting it further by shared memory and register capacity.

[0006] To address the aforementioned issues, we propose a hybrid parallelism approach (Figure 1c). In this approach, data parallelism is applied to the linear layer, while pipeline parallelism is applied to the attention layer. In the latter, we use attention head granularities that are smaller than those at the sub-layer level to reduce pipeline stalls. Our proposed hybrid parallelism imposes new requirements on the hardware architecture. On one hand, pipeline parallelism between operators requires the array of computational units to be decomposed to run each pipeline stage. Simultaneously, shape mismatch can be mitigated through decomposition and workload breakdown. On the other hand, data parallelism within operators requires the computational unit array and registers to be jointly used to maximize data reuse. Summary of the Invention

[0007] The purpose of this invention is to propose a column-reconfigurable systolic array (RSA). Each column of computational units in the array can either work together as a single operator or be separated into multiple operators that work independently. Specifically, RSA can be split based on different input tokens in a linear operator, or based on different pipeline stages in an attention layer.

[0008] To achieve the above objectives, the present invention employs the following technical means:

[0009] This invention provides a column-reconfigurable systolic array for Transformer models, comprising:

[0010] The reconfigurable computational unit structure of a two-dimensional network consists of computational units distributed in rows and columns, with data passed from one row to the next, and from one column to the next.

[0011] The register unit of a two-dimensional network consists of register units distributed in rows and columns. Data is passed from the previous row to the next row and from the previous column to the next column, and the direction of data transfer is opposite to that of the reconfigurable computing unit of the two-dimensional network.

[0012] Registers in the same row and column are configured with data path connections to the computing unit.

[0013] In the above technical solution, the reconfigurable computing unit of the two-dimensional network has a two-row, four-column structure. For each row of computing units, data is passed from the first row of computing units in the top-to-bottom direction to the second row of computing units with the same columns as the first row. For each column of computing units, data is passed from the computing unit of the previous column in the left-to-right direction to the next column of computing units with the same rows as the previous column.

[0014] The register cells of the two-dimensional network have a two-row, four-column structure. For each row of register cells, data is passed from the register cells in the first row to the register cells in the second row with the same columns as the first row, from bottom to top. For each column of register cells, data is passed from the register cells in the previous column to the register cells in the next column with the same rows as the previous column, from right to left.

[0015] In the above technical solution, the computing unit includes:

[0016] Includes row register Rfr: used to receive row output data from the previous calculation unit;

[0017] Column register Rfc: Used to receive column output data from the previous calculation unit;

[0018] First selector: The two input terminals are connected to the output terminal of the row register Rfr and the output terminal of the row register group of the register unit, respectively. The control signal use_reg_row controls whether to output the data in the row register Rfr or the data in the row register group to the multiplier.

[0019] Second selector: The two input terminals are connected to the output terminal of the first selector and the row register output terminal respectively, and the output terminal is used as the row output data;

[0020] The third selector has two input terminals connected to the output of the column register Rfc and the output of the column register group of the register unit, respectively. The control signal use_reg_col controls whether to output the data in the column register Rfc or the data in the column register group to the multiplier.

[0021] Fourth selector: Its two inputs are connected to the output of the column register Rfc and the output of the third selector, respectively, and the output is used as the column output data;

[0022] Fifth selector: The two input terminals are the output terminal of the psum register of other computing units and the psum register of this computing unit, respectively. The output is selected as the psum register of other computing units or the psum register of this computing unit by the control signal use_forward_psum.

[0023] Multiplier: Multiplies the output of the third selector with the output of the second selector and outputs the result to the adder;

[0024] Adder: Adds the output of the multiplier to the output of the fifth selector and outputs the result to the psum register;

[0025] Register unit: Contains four registers, specifically a row register group with 2 rows and a column register group with 2 columns.

[0026] In the above technical solution, the method for configuring the data stream of the reconfigurable pulsating array is as follows:

[0027] A. Configure data streams with fixed weights:

[0028] When use_forward_psum is configured to 1, the adder of this computation unit uses the psum register output of other computation units as input, thereby enabling weighted fixed data streams, while the row input of the multiplier is configured by use_reg_row;

[0029] When use_reg_row is configured to 0, the first selector selects the value of the row register Rfr for output, and the second selector selects the output of the row register Rfr as the output, thereby passing the value of register Rfr to the right.

[0030] When use_reg_row is configured to 1, the first selector selects the output of the row register group as the output, and the second selector selects the output of the first selector as the output, thereby realizing the passing of the value in the row register to the right.

[0031] B. Output a fixed data stream

[0032] When use_forward_psum is configured to 0, the fifth selector selects the value of the psum register of the local computing unit and outputs it to the adder, which accumulates the result of the local computing unit to achieve cumulative summation and outputs a fixed data stream.

[0033] When use_reg_col is configured to 0, the third selector selects the output group of column register Rfc as the output, and the fourth selector selects the output of column register Rfc as the output, thereby passing the value of register Rfc down;

[0034] When use_reg_col is configured to 1, the third selector selects the value of the column register group as the output, and the fourth selector first selects the output of the third selector as the output, thereby realizing the downward transmission of the value in the column register connected by the light blue data path.

[0035] In the above technical solution, the configuration method of the register unit is as follows:

[0036] Each register group operates in ping-pong mode.

[0037] When the pingpang_buffer_switch signal is configured to 1, one row register passes data to the next row register to the right, while another row register in the row register group provides input to the multiplier;

[0038] When the pingpang_buffer_switch signal is configured to 0, the functions of the two registers in the row register group are swapped. The timing of the swap is achieved by freely configuring the pingpang_buffer_switch signal.

[0039] In the above technical solution, the reconfigurable pulsating array is configured to achieve decomposability:

[0040] Decomposability refers to the ability to split a reconfigurable systolic array into multiple subarrays. The specific implementation includes the following steps:

[0041] When use_reg_row is configured to 1, the row input of the multiplier is the value in the row register connected to the data path. For the computation unit to the right of this computation unit, the computation unit to the right will not use the value passed from the adjacent computation unit to the left, thus splitting into two subarrays.

[0042] Because the present invention adopts the above-described technical solution, it has the following beneficial effects:

[0043] It supports hybrid parallel lines (data parallelism at the block layer and pipeline parallelism at the attention layer) to improve the hardware efficiency of Transformer-based models.

[0044] A packing optimization based on pulsating array computation unit columns is proposed. Specifically, matrix multiplication computations of different shapes can be packed on the array according to the column dimension to improve the space and time efficiency of computing variable-length input sequences on hardware.

[0045] Dynamic programming was used to explore container optimization strategies to maximize overall throughput.

[0046] To verify the effectiveness of RSA, we applied RSA to a small GPT-3 model, as well as the MRPC, RTE, and SQuADv2 datasets. Our design on the U200 FPGA achieved a 1.16x improvement in normalized (based on the number of compute units) throughput and a 1.94x improvement in compute unit efficiency compared to the GPU.

[0047] Figure 1 shows examples of execution time for different parallel methods, where (a) intra-operator data parallelism, (b) sub-layer pipeline parallelism, and (c) hybrid parallelism. The time length of each block is for illustrative purposes only.

[0048] Figure 2 shows the circuit diagram of the reconfigurable pulse array;

[0049] Figure 3 shows the circuit diagram of the reconfigurable computing unit and register group;

[0050] Figure 4 is a schematic diagram of a single 4-column pulsating array with fixed weights and data flow.

[0051] Figure 5 is a schematic diagram of two 2-column pulsating arrays with fixed weights and data flow.

[0052] Figure 6 shows a schematic diagram of two 2-column pulsating arrays with a fixed weighted data stream (left) and a fixed output data stream (right).

[0053] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0054] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0055] I. Circuit of Reconfigurable Pulsating Array

[0056] The table below lists the innovations of the reconfigurable systolic array proposed in this invention compared to ordinary systolic arrays. We compared them from three perspectives: data flow, registers, and decomposability, highlighting that our reconfigurable systolic array can achieve higher hardware efficiency for variable-length sequences in Transformer models.

[0057] Our proposed reconfigurable systolic array dataflow only supports either fixed weights or fixed outputs. It also supports both fixed weights and fixed outputs. The reconfigurable register can be switched before running a specific load and can be freely switched during runtime. It can be split into multiple subarrays for independent computation. (Note: The original text contains some inconsistencies and unclear phrasing, making a precise translation difficult. A more accurate translation would require the full context.) surface

[0058] The reconfigurable systolic array shown in Figure 2 consists of reconfigurable computational units (PEs) of a two-dimensional network, as shown in Figure 1. Each computational unit can pass data to the right and down (dark black arrows, i.e., → and ↓ arrows). Simultaneously, we constructed a register group (REG) of the same two-dimensional network, with data transfer directions (light black arrows, i.e., ← and ↑ arrows) opposite to the computational units, allowing data transfer to the up and left each clock cycle. As shown in the dashed box, a computational unit and a corresponding register unit are connected via a configurable data path.

[0059] The circuit diagram within the dashed box in Figure 2 is shown in Figure 3. Each computation unit (PE) contains a multiplier (x sign) and an adder (+ sign). Each register unit (REG) contains four registers, specifically a row register group and a column register group. Its circuit structure specifically implements configurable data flow, registers, and descrambling.

[0060] Configurable data stream

[0061] Fixed-weight data stream

[0062] When `use_forward_psum` is configured to 1, the adders of the computation unit use the partial sum (psum) passed from the computation unit above as input, thus enabling weighted fixed-flow data. In this case, each computation unit passes the partial sum to the computation unit below and the row input of the multiplier to the computation unit to the right. The row input of the multiplier is configured by `use_reg_row`. When `use_reg_row` is configured to 0, the value of Rrf is passed to the right; otherwise, the value of the row register group is passed to the right.

[0063] Output fixed data stream

[0064] When `use_forward_psum` is configured to 0, the adder of the computation unit uses the value of `psum`, and the accumulation function adds the result of the local computation unit to achieve a fixed data stream output. In this case, each computation unit passes the column input of the multiplier to the computation unit below and the row input of the multiplier to the computation unit to the right. When `use_reg_col` is configured to 0, the value of `Rfc` is passed down; otherwise, the value in the column register connected by the light blue data path is passed down.

[0065] 1. Configurable registers

[0066] Each register bank operates in a ping-pong mode. Taking row registers as an example, when the `pingpang_buffer_switch` signal is configured to 1, one row register passes data to the next row register to its left via the orange data path, while the other row register in the row register bank provides input to the multiplier via the light blue data path. When the `pingpang_buffer_switch` signal is configured to 0, the two registers in the row register bank swap functions; the timing of the swap can be achieved by freely configuring the `pingpang_buffer_switch` signal.

[0067] 2. Configurable Splittability

[0068] Taking column-level splitting as an example, this illustrates how a reconfigurable systolic array can be divided into multiple subarrays column-wise. When `use_reg_row` is configured to 0, the row input of the multiplier is the value of `Rfr`. When `use_reg_row` is configured to 1, the row input of the multiplier is the value in the row register connected by the light blue data path. For the latter, the computation unit does not use the value passed from the computation unit to its left, thus creating two subarrays: one containing the computation unit to its left and the computation unit to its left, and the other containing the current computation unit and the computation unit to its right.

[0069] The following are examples of different configurations of reconfigurable systolic arrays. First, we show several examples of configurable data flows and configurable descalability. Figure 4 shows a single 4-column systolic array with a fixed-weight data flow; Figure 5 shows two 2-column systolic arrays with fixed-weight data flows; and Figure 6 shows two 2-column systolic arrays with fixed-weight data flows (left) and a fixed-output data flow (right).

[0070] To facilitate a better understanding of the technical concept of this invention by those skilled in the art, the following references are provided:

[0071] [1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M.Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: a system forlarge- scale machine learning.” in Osdi, vol. 16, no. 2016. Savannah, GA,USA, 2016, pp. 265–283.[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D.Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al.,“Language mod- els are few-shot learners,” Advances in neural informationprocessing systems, vol. 33, pp. 1877–1901, 2020.

[0072] [3] T.Dao,D.Y.Fu,S.Ermon,A.Rudra,andC.Re ́,“Flashattention:Fast andmemory-efficient exact attention with io-awareness,” arXiv preprint arXiv:2205.14135, 2022.

[0073] [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018.

[0074] [5] S. Fan, Y. Rong, C. Meng, Z. Cao, S. Wang, Z. Zheng, C. Wu, G.Long, J. Yang, L. Xia et al., “Dapple: A pipelined data parallel approach fortraining large models,” in Proceedings of the 26th ACM SIGPLAN Symposium onPrinciples and Practice of Parallel Programming, 2021, pp. 431–445.

[0075] [6] J. Fang, Y. Yu, C. Zhao, and J. Zhou, “TurboTransformers: AnEfficient GPU Serving System for Transformer Models,” ser. PPoPP ’21. NewYork, NY, USA: Association for Computing Machinery, 2021, p. 389–402.[Online].

[0076] [7] S. Feng, B. Hou, H. Jin, W. Lin, J. Shao, R. Lai, Z. Ye, L.Zheng, C. H. Yu, Y. Yu et al., “Tensorir: An abstraction for automatictensorized program optimization,” in Proceedings of the 28th ACMInternational Conference on Architectural Support for Programming Languagesand Operating Systems, Volume 2, 2023, pp. 804–817.

[0077] [8] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee,J. Ngiam, Q. V. Le, Y. Wu et al., “Gpipe: Efficient training of giant neu-ral networks using pipeline parallelism,” Advances in neural informationprocessing systems, vol. 32, 2019.

[0078] [9] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R.Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers et al., “In-datacenterperformance analysis of a tensor processing unit,” in Proceedings of the 44thannual international symposium on computer architecture, 2017, pp. 1–12.

[0079]

[10] A.Kerr,D.Merrill,J.Demouth,andJ.Tran,“Cutlass:Fastlinearalgebrain cuda c++,” NVIDIA Developer Blog, 2017.

[0080]

[11] Z. Liu, G. Li, and J. Cheng, “Hardware acceleration of fullyquantized bert for efficient natural language processing,” in 2021 Design,Automa- tion & Test in Europe Conference & Exhibition (DATE). IEEE, 2021, pp.513–516.

[0081]

[12] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R.Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: Generalizedpipeline parallelism for dnn training,” in Proceedings of the 27th ACMSymposium on Operating Systems Principles, 2019, pp. 1–15.

[0082]

[13] D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia,“Memory-efficient pipeline-parallel dnn training,” in International Con-ference on Machine Learning. PMLR, 2021, pp. 7937–7947.

[0083]

[14] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan,T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperativestyle, high-performance deep learning library,” Advances in neuralinformation processing systems, vol. 32, 2019.

[0084]

[15] H. Peng, S. Huang, S. Chen, B. Li, T. Geng, A. Li, W. Jiang, W.Wen, J. Bi, H. Liu, and C. Ding, “A Length Adaptive Algorithm-Hardware Co-Design of Transformer on FPGA through Sparse Attention and DynamicPipelining,” in Proceedings of the 59th ACM / IEEE Design AutomationConference, ser. DAC ’22. New York, NY, USA: Association for ComputingMachinery, 2022, p. 1135–1140.

[0085]

[16] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A.Lev- skaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scalingtransformer inference,” arXiv preprint arXiv:2211.05102, 2022.

[0086]

[17] E. Qin, A. Samajdar, H. Kwon, V. Nadella, S. Srinivasan, D.Das, B. Kaul, and T. Krishna, “SIGMA: A Sparse and Irregular GEMM Ac-celerator with Flexible Interconnects for DNN Training,” in 2020 IEEEInternational Symposium on High Performance Computer Architecture (HPCA),2020, pp. 58–70.

[0087]

[18] A.Radford,K.Narasimhan,T.Salimans,I.Sutskeveretal.,“Improvinglanguage understanding by generative pre-training,” 2018.

[0088]

[19] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever etal., “Language models are unsupervised multitask learners,” OpenAI blog, vol.1, no. 8, p. 9, 2019.

[0089]

[20] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250, 2016.

[0090]

[21] A.Vaswani,N.Shazeer,N.Parmar,J.Uszkoreit,L.Jones,A.N.Gomez, Ł.Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neuralinformation processing systems, vol. 30, 2017.

[0091]

[22] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R.Bowman, “GLUE: A multi-task benchmark and analysis platform for naturallanguage understanding,” arXiv preprint arXiv:1804.07461, 2018.

[0092]

[23] X. Wang, Y. Xiong, X. Qian, Y. Wei, L. Li, and M. Wang,“Lightseq2: Accelerated training for transformer-based models on gpus,” arXivpreprint arXiv:2110.05722, 2021.

[0093]

[24] J. Zeng, M. Li, Z. Wu, J. Liu, Y. Liu, D. Yu, and Y. Ma,“Boosting Distributed Training Performance of the Unpadded BERT Model,” arXivpreprint arXiv:2208.08124, 2022.

[0094]

[25] Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu,and Y. Zhu, “ByteTransformer: A High-Performance Transformer Boosted forVariable-Length Inputs,” 2023。

Claims

1. A column-reconfigurable systolic array for the Transformer model, characterized in that, The system includes: a reconfigurable computational unit structure for a two-dimensional network, consisting of rows and columns of computational units, with data passed from one row to the next and from one column to the next; register units for a two-dimensional network, also consisting of rows and columns of register units, with data passed from one row to the next and from one column to the next, and the data transfer direction being opposite to that of the reconfigurable computational unit structure; registers and computational units in the same row and column are connected via data paths; the systolic array supports hybrid parallel processing, including data parallelism in linear layers and pipelined parallelism in attention layers; the systolic array supports mounting matrix multiplication calculations of different shapes on the array along the column dimension to improve the space and time efficiency of computing variable-length input sequences on hardware.

2. The column-reconfigurable systolic array for Transformer models according to claim 1, characterized in that, The reconfigurable computing units of the two-dimensional network have a two-row, four-column structure. For each row of computing units, data is passed from the first row of computing units in the top-to-bottom direction to the second row of computing units with the same columns as the first row. For each column of computing units, data is passed from the previous column of computing units in the left-to-right direction to the next column of computing units with the same rows as the previous column. The register units of the two-dimensional network also have a two-row, four-column structure. For each row of register units, data is passed from the first row of register units in the bottom-to-top direction to the second row of register units with the same columns as the first row. For each column of register units, data is passed from the previous column of register units in the right-to-left direction to the next column of register units with the same rows as the previous column.

3. A column-reconfigurable systolic array for Transformer models according to claim 1, characterized in that, The computation unit includes: a row register Rfr (for receiving row output data from the previous computation unit); a column register Rfc (for receiving column output data from the previous computation unit); a first selector (with two inputs connected to the output of the row register Rfr and the output of the row register group of the register unit, respectively, and controlled by the use_reg_row signal to output either the data in the row register Rfr or the data in the row register group to the multiplier); a second selector (with two inputs connected to the output of the first selector and the output of the row register, and the output is used as the row output data); and a third selector (with two inputs connected to the output of the column register Rfc and the output of the column register group of the register unit, and controlled by the use_reg_col signal to output either the data in the column register Rfc or the data in the row register group to the multiplier). The data in the column register group is output to the multiplier; the fourth selector: its two inputs are connected to the output of the column register Rfc and the output of the third selector, respectively, and the output is used as the column output data; the fifth selector: its two inputs are the output of the psum register of another computing unit and the psum register of this computing unit, respectively, and the output is selected as the psum register of another computing unit or the psum register of this computing unit by the control signal use_forward_psum; the multiplier: multiplies the output of the third selector with the output of the second selector and outputs the result to the adder; the adder: adds the output of the multiplier with the output of the fifth selector and outputs the result to the psum register; the register unit: contains four registers, specifically a row register group with 2 rows and a column register group with 2 columns.

4. A column-reconfigurable systolic array for Transformer models according to claim 1, characterized in that: The configuration data stream method for the reconfigurable systolic array is as follows: A. Configure fixed-weight data stream: When use_forward_psum is configured to 1, the adder of this computation unit uses the output of the psum register of other computation units as input, thereby enabling the fixed-weight data stream, while the row input of the multiplier is configured by use_reg_row; when use_reg_row is configured to 0, the first selector selects the value of the row register Rfr as output, and the second selector selects the output of the row register Rfr as output, thereby realizing the rightward transfer of the value of the register Rfr; when use_reg_row is configured to 1, the first selector selects the output of the row register group as output, and the second selector selects the output of the first selector as output, thereby realizing the rightward transfer of the value in the row register; B. Output fixed data stream: When use_forward_psum is configured to 0, the fifth selector selects the value of the psum register of this computation unit as output to the adder, that is, to accumulate the result of the local computation unit, realize the cumulative sum, and output a fixed data stream; When use_reg_col is configured to 0, the third selector selects the output group of column register Rfc as the output, and the fourth selector selects the output of column register Rfc as the output, thereby passing the value of register Rfc down; when use_reg_col is configured to 1, the third selector selects the value of column register group as the output, and the fourth selector selects the output of the third selector as the output, thereby passing the value in column register down.

5. A column-reconfigurable systolic array for Transformer models according to claim 1, characterized in that: The register unit is configured as follows: Each register group operates in ping-pong mode. When the pingpang_buffer_switch signal is configured to 1, one row register group passes data to the next row register group on the left, while the other row register in the row register group provides input to the multiplier. When the pingpang_buffer_switch signal is configured to 0, the two registers in the row register group switch functions. The timing of the switch is achieved by freely configuring the pingpang_buffer_switch signal.

6. A column-reconfigurable systolic array for Transformer models according to claim 1, characterized in that: Configure reconfigurable systolic arrays to achieve descalability: Descalability refers to splitting a reconfigurable systolic array into multiple subarrays. The specific implementation includes the following steps: When use_reg_row is configured to 1, the row input of the multiplier is the value in the row register connected to the data path. For the computation unit to the right of this computation unit, the computation unit to the right will not use the value passed from the adjacent computation unit to the left, thus splitting into two subarrays.

Images