Chiplet-based matrix chain multiplication accelerator and acceleration method

Abstract

The invention discloses a Chiplet-based matrix chain multiplication accelerator and an acceleration method. The matrix chain multiplication accelerator comprises a first matrix multiplication module, a first external memory interface and a nonlinear processing module, and a second matrix multiplication module and a second external memory interface which are connected with each other through Chiplet. The method comprises the following steps: acquiring operator data of a Transform network and preprocessing the operator data to obtain a matrix chain data stream; based on the matrix chain data stream, performing initialization processing on a matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator; and inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator for matrix multiplication operation to obtain attention matrix data. According to the embodiment of the invention, the utilization rate of the processing element can be improved, so that the memory access times and the data transmission delay are reduced, and the calculation efficiency is improved. The method can be widely applied to the technical field of computer hardware acceleration.

Description

Technical Field

[0001] The present application relates to the field of computer hardware acceleration technology, and in particular to a chiplet-based matrix chain multiplication accelerator and an acceleration method. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning models have shown excellent performance in many fields, especially in natural language processing (NLP) and computer vision (CV) tasks. Among them, the Transformer model has become one of the core technologies in these fields due to its self-attention mechanism. The Transformer model captures the global dependencies of the input data by allowing the model to calculate attention at each position in the sequence, significantly improving processing efficiency and accuracy. However, the high performance of the Transformer model is accompanied by huge computational and memory requirements. The self-attention mechanism in the model involves a large number of matrix multiplication operations, which run slowly on traditional CPU or GPU platforms, mainly because they require a large amount of data access and movement, resulting in significant hardware overhead. In addition, intermediate nonlinear units, such as Relu and other operations, also increase the difficulty of acceleration. Most of the existing works at this stage mainly explore the sparsity of the model to find the weak correlation between Q and K. In order to meet the accuracy requirements, they need to be retrained and fine-tuned, resulting in additional overhead. They use the process of multiplying the q vector with the K matrix one by one, but the matrix-level parallelism and data flow characteristics are not fully utilized. Although existing studies have considered the data flow characteristics, the current accelerator design usually requires data reconfiguration and transmission between the weight matrix and the output matrix, which leads to additional storage access and delay.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a chiplet-based matrix chain multiplication accelerator and acceleration method, which can improve the utilization of processing elements, thereby reducing the number of memory accesses and data transmission delays, and improving computing efficiency.

[0005] To achieve the above purpose, an embodiment of the present application proposes a chiplet-based matrix chain multiplication accelerator, wherein the matrix chain multiplication accelerator includes a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module, and a second external memory interface, wherein an input end of the first matrix multiplication module is connected to the first external memory interface, an output end of the first matrix multiplication module is connected to an input end of the nonlinear processing module through a chiplet interconnection, an output end of the nonlinear processing module is connected to an input end of the second matrix multiplication module through a chiplet interconnection, and an output end of the second matrix multiplication module is connected to the second external memory interface, wherein:

[0006] The first external memory interface is used to obtain Q matrix data;

[0007] The first matrix multiplication module is used to perform parallel internal product calculation on the Q matrix data and the K matrix data, and output score matrix data;

[0008] The nonlinear processing module is used to perform mask operation processing on the score matrix data to obtain the score matrix data after nonlinear processing;

[0009] The second matrix multiplication module is used to perform parallel internal product calculation on the score matrix data after the nonlinear processing and the V matrix data, and output the attention matrix data;

[0010] The second external memory interface is used to initialize the attention matrix data to achieve Tranformer network reasoning acceleration.

[0011] In some embodiments, the first matrix multiplication module is used to perform matrix chain multiplication optimization on the attention mechanism layer of the Tranformer network, and the second matrix multiplication module is used to perform matrix chain multiplication optimization on the feedforward layer of the Tranformer network.

[0012] In some embodiments, the first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first computing units, and the first on-chip memories are divided into a plurality of Bank sub-modules, wherein:

[0013] A plurality of the first on-chip memories are used to store the K matrix data, wherein each of the Bank submodules is used to store a column block of the K matrix data;

[0014] The first computing units are used to perform a multiplication operation of the Q matrix data and the K matrix data.

[0015] In some embodiments, the nonlinear processing module is provided with a plurality of nonlinear processing units, and the nonlinear processing units are used to configure different nonlinear processing function functions to support corresponding different nonlinear operations.

[0016] In some embodiments, the second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second computing units, wherein:

[0017] A plurality of the second on-chip memories are used to store the V matrix data;

[0018] The plurality of second calculation units are used to perform a multiplication operation between the V matrix data and the score matrix data after the nonlinear processing.

[0019] To achieve the above purpose, another aspect of an embodiment of the present application provides an acceleration method of a chiplet-based matrix chain multiplication accelerator, the method comprising the following steps:

[0020] Acquire operator data of the Tranformer network and perform preprocessing to obtain a matrix chain data stream, wherein the matrix chain data stream includes Q matrix data, V matrix data, and K matrix data;

[0021] Based on the matrix chain data stream, initializing the matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator;

[0022] The matrix chain data stream is input into the initialized matrix chain multiplication accelerator to perform matrix multiplication operation to obtain attention matrix data.

[0023] In some embodiments, the step of obtaining operator data of the Tranformer network and preprocessing the data to obtain a matrix chain data flow includes:

[0024] Acquiring operator data of the Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data flow;

[0025] The preliminary matrix chain data stream is divided and processed according to the modular design of the chiplet technology to obtain the matrix chain data stream.

[0026] In some embodiments, the initialization processing of the matrix chain multiplication accelerator based on the matrix chain data stream to obtain the initialized matrix chain multiplication accelerator includes:

[0027] According to the space size of the on-chip memory of the matrix chain multiplication accelerator, the computing unit of the matrix chain multiplication accelerator is divided to obtain a divided matrix chain multiplication accelerator;

[0028] The V matrix data in the matrix chain data stream and the K matrix data in the matrix chain data stream are pre-stored in the on-chip memory of the divided matrix chain multiplication accelerator, and the computing unit of the divided matrix chain multiplication accelerator is set to a fused multiplication-addition working mode to obtain the initialized matrix chain multiplication accelerator.

[0029] In some embodiments, the step of inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform a matrix multiplication operation to obtain attention matrix data includes:

[0030] Inputting the Q matrix data in the matrix chain data stream into the initialized matrix chain multiplication accelerator;

[0031] Perform parallel inner product calculation on the Q matrix data and the K matrix data, and output score matrix data;

[0032] Performing mask operation processing on the score matrix data to obtain score matrix data after nonlinear processing;

[0033] The score matrix data after the nonlinear processing is parallelized with the V matrix data to output the attention matrix data.

[0034] In some embodiments, the matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and the computing unit of the matrix chain multiplication accelerator configured and expanded according to the parallelism of the transmission channel of the matrix chain data flow processes the data in parallel.

[0035] The embodiments of the present application include at least the following beneficial effects: the present application provides a chiplet-based matrix chain multiplication accelerator and an acceleration method. The scheme performs parallel internal product calculations on Q matrix data and K matrix data through a first matrix multiplication module, and outputs score matrix data; then performs mask operation processing on the score matrix data to obtain score matrix data after nonlinear processing; finally, a second matrix multiplication module is used to perform parallel internal product calculations on the score matrix data after nonlinear processing and V matrix data, and outputs attention matrix data. The matrix data is on a chiplet interconnection channel. Through the chiplet interconnection technology, different dies can share data and resources to achieve more efficient calculation. In the process of continuously multiplying multiple matrices, the result of the previous matrix multiplication is managed so that it is directly multiplied with the subsequent matrix to improve the utilization rate of the processing element (PE), thereby reducing the number of memory accesses and the delay of data transmission, and improving the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 11 is a schematic diagram of the architecture of a chiplet-based matrix chain multiplication accelerator provided in an embodiment of the present application;

[0037] Figure 2 It is a schematic diagram of the steps of an acceleration method of a chiplet-based matrix chain multiplication accelerator provided in an embodiment of the present application;

[0038] Figure 3 It is a schematic diagram of an on-chip data flow diagram for constructing a Transformer network attention operator through MCM provided in an embodiment of the present application;

[0039] Figure 4 is a structural diagram of a first matrix multiplication module Die0 provided in an embodiment of the present application;

[0040] Figure 5 It is a structural diagram of the second matrix multiplication module Die2 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0042] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0043] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0045] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0046] 1) Chiplet interconnection: Chiplet interconnection technology is an advanced integrated circuit design technology that reduces the complexity and cost of design by splitting large chips into smaller modules, that is, a unit on a single silicon wafer during semiconductor processing that can contain a complete circuit system, also known as a die. These dies are connected through high-bandwidth interconnect technology to achieve efficient data transmission and resource sharing. In the chiplet architecture, multiple dies can be combined to form a complex processor or memory array.

[0047] 2) Matrix Chain Multiplication (MCM) is a classic dynamic programming problem that is of great significance in computer science and algorithm analysis. Its goal is to determine the optimal multiplication order of matrices to minimize the number of multiplication operations required for the calculation.

[0048] There are some deficiencies in the related art, such as most existing works mainly explore the sparsity of the model to find weak correlations between Q and K. Sparsity in machine learning varies from neural network to neural network and is highly dependent on specific applications. Therefore, in order to meet accuracy requirements, retraining and fine-tuning are required, resulting in additional overhead. They use the process of multiplying the q vector by the K matrix one by one, which can perform vector-level parallelism in parallel. However, the inherent parallelism of the attention mechanism, such as matrix-level parallelism and data flow characteristics, has not been fully utilized. Many accelerators for attention mechanisms assume that the Q, K, and V matrices have been calculated and stored in DRAM; however, the generation of the Q, K, and V matrices is very resource-intensive in hardware. In addition, they need to be regenerated in different layers, and these costs have not been considered. Although existing studies have considered data flow characteristics, traditional accelerator designs do not fully utilize the potential characteristics of data flow in Transformer models. For example, existing designs usually require reconfiguration and transmission of data between weight matrices and output matrices, which results in additional storage access and latency.

[0049] In view of this, a matrix chain multiplication accelerator based on chiplet is provided in an embodiment of the present application, which aims to utilize the high bandwidth of chiplet interconnection and the modular scalability of MCM data flow as much as possible to reduce the additional power consumption and area overhead required to support different Transformer operators while ensuring the PE utilization efficiency of the accelerator. The present invention proposes a hardware accelerator based on chiplet design, with MCM data flow and efficient die-level computing architecture. The die-level architecture can be expanded in terms of computing units and storage units, and multi-die integration can be performed through chiplet interconnection, ultimately maintaining a PE efficiency of 97.8%.

[0050] Reference Figure 1 , Figure 1 A schematic diagram of the architecture of a chiplet-based matrix chain multiplication accelerator provided in an embodiment of the present invention, referring to Figure 1 The matrix chain multiplication accelerator includes a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module and a second external memory interface, the input end of the first matrix multiplication module is connected to the first external memory interface, the output end of the first matrix multiplication module is connected to the input end of the nonlinear processing module through a chiplet interconnection, the output end of the nonlinear processing module is connected to the input end of the second matrix multiplication module through a chiplet interconnection, and the output end of the second matrix multiplication module is connected to the second external memory interface, wherein:

[0051] The first external memory interface is used to obtain Q matrix data;

[0052] The first matrix multiplication module is used to perform parallel internal product calculations on the Q matrix data and the K matrix data, and output the score matrix data;

[0053] Specifically, the first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first computing units, the first on-chip memories are divided into a plurality of Bank sub-modules, wherein the plurality of first on-chip memories are used to store K matrix data, wherein each Bank sub-module is used to store a column block of K matrix data; and the plurality of first computing units are used to perform multiplication operations of Q matrix data and K matrix data.

[0054] The nonlinear processing module is used to perform mask operation on the score matrix data to obtain the score matrix data after nonlinear processing;

[0055] Specifically, the nonlinear processing module is provided with a plurality of nonlinear processing units, and the nonlinear processing units are used to configure different nonlinear processing function functions to support corresponding different nonlinear operations.

[0056] The second matrix multiplication module is used to perform parallel internal product calculations on the score matrix data after nonlinear processing and the V matrix data, and output the attention matrix data;

[0057] Specifically, the second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second computing units, wherein the plurality of second on-chip memories are used to store V matrix data; and the plurality of second computing units are used to perform multiplication operations of the V matrix data and the score matrix data after nonlinear processing.

[0058] The second external memory interface is used to initialize the attention matrix data to accelerate Tranformer network reasoning.

[0059] In summary, the embodiment of the present invention proposes a configurable and scalable die-level computing architecture, which adopts a single program multiple data (SPMD) model to achieve efficient parallel processing. Each die can perform computing tasks independently. At the same time, through the chiplet interconnection technology, different dies can share data and resources to achieve more efficient computing.

[0060] First matrix multiplication module (Die0): Die 0 is responsible for the multiplication operation of the Q matrix and the K matrix. In order to store the column data of the K matrix, Die0 is equipped with a large amount of on-chip memory (SRAM). These SRAMs are organized into multiple banks, each of which stores a column block of the K matrix. By accessing these SRAM banks in parallel, Die0 can perform multiple multiplication operations simultaneously, thereby improving the computational throughput.

[0061] Nonlinear processing module (Die1): Die1 is used to insert nonlinear operations between matrix chains, such as ReLU activation function. In order to achieve functional flexibility, the nonlinear processing unit of Die1 can be reconfigured to support different nonlinear operations. This design allows the accelerator to adapt to different Transformer operators and improves the utilization of hardware.

[0062] Second matrix multiplication module (Die2): Die 2 is responsible for performing the multiplication operation of the score matrix and the V matrix to generate the final attention matrix. In order to improve computing efficiency, Die2 uses high-bandwidth chiplet interconnection technology to directly receive the processed score (~) data, that is, the score matrix data after nonlinear processing, from Die1 without going through external memory. This design reduces the delay of data transmission and improves the overall computing performance.

[0063] In addition, it should be noted that the first matrix multiplication module is used to perform matrix chain multiplication optimization on the attention mechanism layer of the Tranformer network, and the second matrix multiplication module is used to perform matrix chain multiplication optimization on the feedforward layer of the Tranformer network.

[0064] Specifically, MCM optimization is implemented in two key operations of the Transformer network, namely the calculation of the attention mechanism and the calculation of the feedforward layer. Performance is improved by reducing memory access. Specifically, the embodiment of the present invention implements MCM optimization in the following ways. First, MCM optimization of the attention mechanism. When calculating the attention score, the present invention uses the MCM method to directly use the result of multiplying the transpose of the Q matrix and the K matrix for the multiplication operation with the V matrix, thereby reducing the number of accesses to the external memory. Secondly, MCM optimization of the feedforward layer: In the calculation of the feedforward layer, the present invention also uses the MCM method to connect two consecutive matrix multiplication operations, thereby reducing the number of data transmissions between different computing units.

[0065] In addition, it should be noted that in order to further improve the computing efficiency, the embodiment of the present invention optimizes the data flow scheduling. By caching key data through the on-chip memory (SRAM), the number of accesses to the external memory is reduced. At the same time, the data transmission path on the die interconnection channel is optimized, the transmission delay is reduced, and the data transmission efficiency is improved. The computing unit in the embodiment of the present invention includes at least one digital signal processor (DSP) for performing multiplication and addition (MAC) operations. With the support of SRAM, these DSPs can perform matrix multiplication operations in parallel, thereby improving the computing throughput.

[0066] See also Figure 2 The embodiment of the present application further provides an acceleration method of a chiplet-based matrix chain multiplication accelerator, which can implement the above chiplet-based matrix chain multiplication accelerator. The method includes the following steps:

[0067] S100, acquiring operator data of the Tranformer network and performing preprocessing to obtain a matrix chain data stream, wherein the matrix chain data stream includes Q matrix data, V matrix data and K matrix data;

[0068] It should be noted that, in some embodiments, step S100 may include: S110, obtaining operator data of the Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data flow; S120, dividing the preliminary matrix chain data flow according to the modular design of the chiplet technology to obtain a matrix chain data flow.

[0069] In some specific embodiments, the embodiments of the present invention construct on-chip data flow through matrix chain multiplication (MCM), convert the main Transformer operators into matrix chain data flow, and divide the matrix chain data flow into die-to-die data flow through the modular design of chiplet technology. In the Transformer network, the self-attention mechanism involves a large number of matrix multiplication operations. The embodiments of the present invention connect these matrix multiplication operations through the MCM method to form a continuous calculation process, such as Figure 3 In this process, the result of the previous matrix multiplication is directly used for the next multiplication operation without writing it back to the memory, thus reducing the number of memory accesses and improving computational efficiency.

[0070] S200, performing initialization processing on the matrix chain multiplication accelerator based on the matrix chain data stream to obtain an initialized matrix chain multiplication accelerator;

[0071] It should be noted that, in some embodiments, step S200 may include: S210, dividing the computing units of the matrix chain multiplication accelerator according to the space size of the on-chip memory of the matrix chain multiplication accelerator to obtain a divided matrix chain multiplication accelerator; S220, pre-storing the V matrix data in the matrix chain data stream and the K matrix data in the matrix chain data stream to the on-chip memory of the divided matrix chain multiplication accelerator, setting the computing units of the divided matrix chain multiplication accelerator to a fused multiplication-addition working mode, and obtaining an initialized matrix chain multiplication accelerator.

[0072] In some specific embodiments, matrix multiplication is divided into blocks according to the size of the on-chip storage space; the block data of two matrices (matrix K and matrix V) are pre-configured to the on-chip storage space; the computing units of die0 and die2 are configured to a fused multiplication-addition working mode, and one of the data interfaces of the multiplication-addition unit is connected to the on-chip memory.

[0073] S300, inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform matrix multiplication operation to obtain attention matrix data.

[0074] It should be noted that, in some embodiments, step S300 may include: S310, inputting the Q matrix data in the matrix chain data stream into the initialized matrix chain multiplication accelerator; S320, performing parallel inner product calculations on the Q matrix data and the K matrix data, and outputting the score matrix data; S330, performing mask operation processing on the score matrix data to obtain the score matrix data after nonlinear processing; S340, performing parallel inner product calculations on the score matrix data after nonlinear processing and the V matrix data, and outputting the attention matrix data.

[0075] In some specific embodiments, taking the attention operator calculation of Tranformer as an example, Figure 1 As shown, the design uses three dies to form a chip, where die0 is responsible for the multiplication of matrices Q and K, die2 processes the multiplication of matrix score and matrix V, and die1 is used to insert nonlinear operations between matrix chains. Specifically, the on-chip memory of die0 is used to store K matrix blocks containing complete column data, and then die0 loads the Q matrix from the off-chip memory interface row by row. After that, the inner product is calculated in parallel on the chip, and the score blocks are output sequentially. The score blocks are then transferred row by row to die1 through the chiplet's inter-die interconnect for masking operations. After nonlinear processing, the score (~) block is transferred to die2 again through the chiplet interconnect. Due to the limited on-chip storage capacity of die0, the K matrix needs to be partitioned, so it is not guaranteed that die2 receives the complete score (~) row. Therefore, the attention matrix needs to load the existing partial and data from the off-chip memory to perform initialization, and then continue the calculation, similar to the process of the outer product. .

[0076] Among them, Chiplet plays a key role in MCM calculation:

[0077] 1) Ensure that the on-chip storage of die0 can accommodate at least one complete row of K blocks. In addition, the larger the K blocks stored, the less additional scheduling the MCM accelerator requires, thereby improving computational efficiency.

[0078] 2) Allows flexible replacement of nonlinear processing dies to adapt to different operators. For example, by simply replacing the processing on die1 with a ReLU function, the framework can adapt the calculation of the feedforward layer.

[0079] 3) Decoupling the resource dependency of the two matrix multiplications. The on-chip storage required for score and V matrix multiplication increases with the size of the K block. Assuming that both matrix multiplication accelerators are placed on a single die, the manufacturing area limitation of a single die becomes a critical issue. Using chiplet design helps alleviate this problem.

[0080] Figure 4 and Figure 5The internal architecture of two computing dies for MCM hardware acceleration is shown, namely the first matrix multiplication module and the second matrix multiplication module. In general, they both adopt SPMD architecture, and programming includes the initialization of registers and SRAMs, as well as the configuration of processing units. For die0, the focus is on performing a complete inner product operation between the rows of Q and the columns of K. Therefore, its area resources are mainly allocated to ensure the storage of K blocks, that is, the capacity of SRAM, and then expand the number of SRAMs. The number of parallel computing paths in die 0 is determined by the number of SRAMs. Each computing path includes a DSP running in fused multiply-add (FMA) mode, which can quickly process data streams. The most significant difference between die2 and die0 is the source of their computing data streams: die2 relies on high-bandwidth chiplet interconnects instead of memory, so more resources can be directly allocated to parallel processing. Each SRAM in die2 only needs to store part of the columns of the V matrix, because the data therein only needs to perform inner product operations with a set of rows of the score matrix to obtain the partial sum of the attention matrix. Under this computing architecture, die2 can further expand the column parallelism of the score matrix. By increasing the number of computation paths, multiple columns of the score matrix can perform inner product operations with the same row of the V matrix simultaneously, which can greatly improve on-chip data reuse of the V matrix.

[0081] It should be noted that the matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and processes the data in parallel according to the parallelism configuration of the transmission channel of the matrix chain data flow and the computing unit of the expanded matrix chain multiplication accelerator.

[0082] In summary, the data processing of the embodiment of the present invention is summarized as follows:

[0083] 1) According to the size of the on-chip storage space, the matrix multiplication operation is divided into blocks.

[0084] 2) Pre-allocate block data of two matrices (matrix K and matrix V) to on-chip storage space.

[0085] 3) Configure the computing units of die0 and die2 to the fused multiplication-addition working mode, and connect one of the data interfaces of the multiplication-addition unit to the on-chip memory.

[0086] 4) Start data stream push, load each row of matrix Q from external memory in sequence, and pass the data stream through two calculation dies in sequence. After completing a matrix block calculation, write it back to the main memory.

[0087] 5) Repeat steps 2) to 4) until all matrix block calculations are completed.

[0088] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0089] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A chiplet-based matrix chain multiplication accelerator, characterized in that: The matrix chain multiplication accelerator includes a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module and a second external memory interface, wherein an input end of the first matrix multiplication module is connected to the first external memory interface, an output end of the first matrix multiplication module is connected to an input end of the nonlinear processing module through a chiplet interconnection, an output end of the nonlinear processing module is connected to an input end of the second matrix multiplication module through a chiplet interconnection, and an output end of the second matrix multiplication module is connected to the second external memory interface, wherein: The first external memory interface is used to obtain Q matrix data; The first matrix multiplication module is used to perform parallel internal product calculation on the Q matrix data and the K matrix data, and output score matrix data; The nonlinear processing module is used to perform mask operation processing on the score matrix data to obtain the score matrix data after nonlinear processing; The second matrix multiplication module is used to perform parallel internal product calculation on the score matrix data after the nonlinear processing and the V matrix data, and output the attention matrix data; The second external memory interface is used to initialize the attention matrix data to achieve Tranformer network reasoning acceleration.

2. The matrix chain multiplication accelerator according to claim 1, characterized in that: The first matrix multiplication module is used to perform matrix chain multiplication optimization on the attention mechanism layer of the Tranformer network, and the second matrix multiplication module is used to perform matrix chain multiplication optimization on the feedforward layer of the Tranformer network.

3. The matrix chain multiplication accelerator according to claim 1, characterized in that: The first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first computing units, wherein the first on-chip memories are divided into a plurality of Bank sub-modules, wherein: A plurality of the first on-chip memories are used to store the K matrix data, wherein each of the Bank submodules is used to store a column block of the K matrix data; The first computing units are used to perform a multiplication operation of the Q matrix data and the K matrix data.

4. The matrix chain multiplication accelerator according to claim 1, characterized in that: The nonlinear processing module is provided with a plurality of nonlinear processing units, and the nonlinear processing units are used to configure different nonlinear processing function functions to support corresponding different nonlinear operations.

5. The matrix chain multiplication accelerator according to claim 1, characterized in that: The second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second computing units, wherein: A plurality of the second on-chip memories are used to store the V matrix data; The plurality of second calculation units are used to perform a multiplication operation between the V matrix data and the score matrix data after the nonlinear processing.

6. A chiplet-based matrix chain multiplication accelerator acceleration method, characterized in that: The method comprises the following steps: Acquire operator data of the Tranformer network and perform preprocessing to obtain a matrix chain data stream, wherein the matrix chain data stream includes Q matrix data, V matrix data, and K matrix data; Based on the matrix chain data stream, initializing the matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator; The matrix chain data stream is input into the initialized matrix chain multiplication accelerator to perform matrix multiplication operation to obtain attention matrix data.

7. The method according to claim 6, characterized in that The operator data of the Tranformer network is obtained and preprocessed to obtain a matrix chain data flow, including: Acquiring operator data of the Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data flow; The preliminary matrix chain data stream is divided and processed according to the modular design of the chiplet technology to obtain the matrix chain data stream.

8. The method according to claim 6, characterized in that The method of initializing the matrix chain multiplication accelerator based on the matrix chain data stream to obtain the initialized matrix chain multiplication accelerator comprises: According to the space size of the on-chip memory of the matrix chain multiplication accelerator, the computing unit of the matrix chain multiplication accelerator is divided to obtain a divided matrix chain multiplication accelerator; The V matrix data in the matrix chain data stream and the K matrix data in the matrix chain data stream are pre-stored in the on-chip memory of the divided matrix chain multiplication accelerator, and the computing unit of the divided matrix chain multiplication accelerator is set to a fused multiplication-addition working mode to obtain the initialized matrix chain multiplication accelerator.

9. The method according to claim 6, characterized in that The step of inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform a matrix multiplication operation to obtain attention matrix data includes: Inputting the Q matrix data in the matrix chain data stream into the initialized matrix chain multiplication accelerator; Perform parallel inner product calculation on the Q matrix data and the K matrix data, and output score matrix data; Performing mask operation processing on the score matrix data to obtain score matrix data after nonlinear processing; The score matrix data after the nonlinear processing is parallelized with the V matrix data to output the attention matrix data.

10. The method according to claim 6, characterized in that The matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and the computing unit of the matrix chain multiplication accelerator configured and expanded according to the parallelism of the transmission channel of the matrix chain data flow processes the data in parallel.

Images