Chiplet-based matrix chain multiplication accelerator and acceleration method

Abstract

The application discloses a Chiplet-based matrix chain multiplication accelerator and a matrix chain multiplication acceleration method. The matrix chain multiplication accelerator comprises a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module and a second external memory interface, and the modules are connected through Chiplet interconnection. The method comprises the following steps: obtaining operator data of a Tranformer network and preprocessing the operator data to obtain a matrix chain data stream; based on the matrix chain data stream, performing initialization processing on the matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator; inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform matrix multiplication operation, and obtaining attention matrix data. The application can improve the utilization rate of processing elements, thereby reducing the memory access frequency and the delay of data transmission, and improving the calculation efficiency. The application can be widely applied to the field of computer hardware acceleration technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer hardware acceleration, and particularly relates to a Chiplet-based matrix chain multiplication accelerator and acceleration method. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, deep learning models have shown excellent performance in many fields, especially in natural language processing (NLP) and computer vision (CV) tasks. Among them, the Transformer model has become one of the core technologies in these fields due to its self-attention mechanism. The Transformer model allows the model to compute attention at each position in the sequence, thereby capturing the global dependencies of the input data, significantly improving processing efficiency and accuracy. However, the high performance of the Transformer model is accompanied by huge computational load and memory requirements. The self-attention mechanism in the model involves a large number of matrix multiplication operations, which run slowly on traditional CPU or GPU platforms, mainly because they need to perform a large number of data access and movement, resulting in significant hardware overhead. In addition, intermediate nonlinear units such as Relu operations also increase the difficulty of acceleration. And most of the existing works at this stage mainly explore the sparsity of the model to find the weak correlation between Q and K, but in order to meet the accuracy requirements, they need to be retrained and fine-tuned, resulting in additional overhead, and they use the process of multiplying each q vector with the K matrix, however, the matrix-level parallelism and data flow characteristics are not fully utilized, although there have been studies considering the data flow characteristics, but the accelerator design at this stage usually needs to perform data reconfiguration and transmission between the weight matrix and the output matrix, which results in additional memory access and delay.

[0003] In summary, the technical problems existing in the related art need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a Chiplet-based matrix chain multiplication accelerator and acceleration method, which can improve the utilization rate of processing elements, thereby reducing the number of memory accesses and the delay of data transmission, and improving the computing efficiency.

[0005] To achieve the above object, an aspect of an embodiment of the present application provides a Chiplet-based matrix chain multiplication accelerator, which comprises a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module, and a second external memory interface. The input end of the first matrix multiplication module is connected with the first external memory interface. The output end of the first matrix multiplication module is connected with the input end of the nonlinear processing module through Chiplet interconnection. The output end of the nonlinear processing module is connected with the input end of the second matrix multiplication module through Chiplet interconnection. The output end of the second matrix multiplication module is connected with the second external memory interface. Wherein:

[0006] The first external memory interface is configured to obtain Q matrix data.

[0007] The first matrix multiplication module is configured to perform parallel internal product calculation on the Q matrix data and K matrix data, and output score matrix data.

[0008] The nonlinear processing module is configured to perform mask operation processing on the score matrix data, and obtain nonlinearly processed score matrix data.

[0009] The second matrix multiplication module is configured to perform parallel internal product calculation on the nonlinearly processed score matrix data and V matrix data, and output attention matrix data.

[0010] The second external memory interface is configured to initialize the attention matrix data to realize Transformer network inference acceleration.

[0011] In some embodiments, the first matrix multiplication module is configured to perform matrix chain multiplication optimization on an attention mechanism layer of the Transformer network, and the second matrix multiplication module is configured to perform matrix chain multiplication optimization on a feedforward layer of the Transformer network.

[0012] In some embodiments, the first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first calculation units. The first on-chip memories are divided into a plurality of Bank sub-modules. Wherein:

[0013] The plurality of first on-chip memories are configured to store the K matrix data. Each Bank sub-module is configured to store one column block of the K matrix data.

[0014] The plurality of first calculation units are configured to perform multiplication operation on the Q matrix data and the K matrix data.

[0015] In some embodiments, the non-linear processing module is provided with a plurality of non-linear processing units configured to implement different non-linear processing functions to support different non-linear operations.

[0016] In some embodiments, the second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second calculation units, wherein:

[0017] The plurality of second on-chip memories are configured to store the V matrix data.

[0018] The plurality of second calculation units are configured to perform multiplication operations of the V matrix data and the non-linearly processed score matrix data.

[0019] To achieve the above object, another aspect of the embodiments of the present application proposes an acceleration method of a Chiplet-based matrix chain multiplication accelerator, which comprises the following steps:

[0020] Obtaining operator data of a Tranformer network and performing preprocessing to obtain a matrix chain data stream, wherein the matrix chain data stream comprises Q matrix data, V matrix data and K matrix data.

[0021] Based on the matrix chain data stream, performing initialization processing on a matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator.

[0022] Inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform matrix multiplication operations to obtain attention matrix data.

[0023] In some embodiments, the step of obtaining operator data of a Tranformer network and performing preprocessing to obtain a matrix chain data stream comprises:

[0024] Obtaining operator data of the Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data stream.

[0025] According to the modular design of Chiplet technology, performing division processing on the preliminary matrix chain data stream to obtain the matrix chain data stream.

[0026] In some embodiments, the step of based on the matrix chain data stream, performing initialization processing on a matrix chain multiplication accelerator to obtain an initialized matrix chain multiplication accelerator comprises:

[0027] According to the space size of the on-chip memory of the matrix chain multiplication accelerator, performing division processing on the calculation units of the matrix chain multiplication accelerator to obtain a divided matrix chain multiplication accelerator.

[0028] Pre-storing V matrix data in the matrix chain data stream and K matrix data in the matrix chain data stream to on-chip memories of the divided matrix chain multiplication accelerator, setting a calculation unit of the divided matrix chain multiplication accelerator to a fused multiply-add working mode, and obtaining the initialized matrix chain multiplication accelerator.

[0029] In some embodiments, the inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator for matrix multiplication operation to obtain attention matrix data comprises:

[0030] Inputting Q matrix data in the matrix chain data stream into the initialized matrix chain multiplication accelerator;

[0031] Performing parallel inner product calculation on the Q matrix data and the K matrix data to output score matrix data;

[0032] Performing mask operation processing on the score matrix data to obtain nonlinearly processed score matrix data;

[0033] Performing parallel inner product calculation on the nonlinearly processed score matrix data and the V matrix data to output attention matrix data.

[0034] In some embodiments, the matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and the calculation unit of the matrix chain multiplication accelerator is configured according to the parallel degree of the transmission channel of the matrix chain data stream to perform parallel processing on data.

[0035] The embodiments of the present application at least have the following beneficial effects: the present application provides a matrix chain multiplication accelerator and an acceleration method based on Chiplet, which performs parallel inner product calculation on Q matrix data and K matrix data through a first matrix multiplication module to output score matrix data, then performs mask operation processing on the score matrix data to obtain nonlinearly processed score matrix data, and finally performs parallel inner product calculation on the nonlinearly processed score matrix data and V matrix data through a second matrix multiplication module to output attention matrix data. The matrix data is on a Chiplet interconnection channel, and different dies can share data and resources through Chiplet interconnection technology, thereby realizing more efficient calculation. In the process of multiplying a plurality of matrices in succession, the result of a previous matrix multiplication is managed to be directly multiplied with a subsequent matrix, so as to improve the utilization rate of processing elements (PEs), thereby reducing the number of memory access and the delay of data transmission, and improving the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1is a schematic diagram of an architecture of a Chiplet-based matrix chain multiplication accelerator provided by an embodiment of the present application.

[0037] Figure 2 is a schematic diagram of a step flow of an acceleration method of a Chiplet-based matrix chain multiplication accelerator provided by an embodiment of the present application.

[0038] Figure 3 is a schematic diagram of an on-chip data flow graph of a Transformer network attention operator constructed by MCM provided by an embodiment of the present application.

[0039] Figure 4 is a schematic diagram of a structure of a first matrix multiplication module Die0 provided by an embodiment of the present application.

[0040] Figure 5 is a schematic diagram of a structure of a second matrix multiplication module Die2 provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0042] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0043] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0045] Before the embodiments of the present application are explained in detail, the description of the embodiments of the present application will be described in brief, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0046] 1) Chiplet interconnect: Chiplet interconnect technology is an advanced integrated circuit design technology that reduces design complexity and cost by splitting large chips into smaller modules, i.e., a single unit on a silicon wafer during semiconductor processing, which can contain complete circuit systems, also known as dies. These dies are connected through high-bandwidth interconnect technology to achieve efficient data transmission and resource sharing. In the Chiplet architecture, multiple dies can be combined to form a complex processor or memory array.

[0047] 2) Matrix Chain Multiplication (MCM) is a classic dynamic programming problem that is of great significance in computer science and algorithm analysis. Its goal is to determine the optimal multiplication order of matrices to minimize the number of multiplication operations required for computation.

[0048] In the related art, there are some deficiencies, such as most existing works mainly explore the sparsity of the model to find the weak correlation between Q and K. The sparsity in machine learning varies with neural networks and is highly dependent on specific applications. Therefore, in order to meet the accuracy requirements, retraining and fine-tuning are required, resulting in additional overhead. They use the process of multiplying the q vector with the K matrix one by one, which can be performed in parallel with vector-level parallelism. However, the inherent parallelism of attention mechanisms, such as matrix-level parallelism and data flow characteristics, has not been fully utilized. Many accelerators for attention mechanisms assume that Q, K, and V matrices have been calculated and stored in DRAM; however, the generation of Q, K, and V matrices is very resource-intensive on hardware. In addition, they need to be regenerated in different layers, and these costs have not been considered. Although there have been studies considering data flow characteristics, traditional accelerator designs have not fully utilized the potential characteristics of data flow in the Transformer model. For example, existing designs usually require data reconfiguration and transmission between weight matrices and output matrices, which results in additional memory access and delay.

[0049] Therefore, in the embodiments of the present application, a Chiplet-based matrix chain multiplication accelerator is provided, which aims to utilize the high bandwidth of Chiplet interconnection and the modular scalability of MCM data flow as much as possible under the premise of ensuring the utilization efficiency of the accelerator PE, and reduce the additional power consumption area overhead required to support different Transformer operators. The present application proposes a hardware accelerator based on Chiplet design, which has MCM data flow and efficient die-level computing architecture. The die-level architecture can be expanded in terms of computing units and storage units, and integrated through Chiplet interconnection, finally maintaining 97.8% PE efficiency.

[0050] Referring to Figure 1 , Figure 1 The architecture schematic diagram of the Chiplet-based matrix chain multiplication accelerator provided by the embodiments of the present application is shown in FIG. 1. Figure 1 The matrix chain multiplication accelerator includes a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module, and a second external memory interface. The input end of the first matrix multiplication module is connected with the first external memory interface. The output end of the first matrix multiplication module is connected with the input end of the nonlinear processing module through Chiplet interconnection. The output end of the nonlinear processing module is connected with the input end of the second matrix multiplication module through Chiplet interconnection. The output end of the second matrix multiplication module is connected with the second external memory interface.

[0051] The first external memory interface is configured to obtain Q matrix data.

[0052] The first matrix multiplication module is configured to perform parallel internal product calculation on the Q matrix data and K matrix data, and output score matrix data.

[0053] Specifically, the first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first computing units. The first on-chip memories are divided into a plurality of Bank sub-modules. The plurality of first on-chip memories are configured to store K matrix data. Each Bank sub-module is configured to store one column block of K matrix data. The plurality of first computing units are configured to perform multiplication operation on the Q matrix data and the K matrix data.

[0054] The nonlinear processing module is configured to perform mask operation processing on the score matrix data, and obtain nonlinearly processed score matrix data.

[0055] Specifically, the nonlinear processing module is provided with a plurality of nonlinear processing units. The nonlinear processing units are configured to implement different nonlinear processing functions to support different nonlinear operations.

[0056] The second matrix multiplication module is configured to perform parallel inner product calculation on the nonlinearly processed score matrix data and the V matrix data, and output attention matrix data.

[0057] Specifically, the second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second calculation units, wherein the plurality of second on-chip memories are configured to store the V matrix data; and the plurality of second calculation units are configured to perform multiplication operation of the V matrix data and the nonlinearly processed score matrix data.

[0058] The second external memory interface is configured to initialize the attention matrix data to realize Transformer network inference acceleration.

[0059] In summary, the embodiment of the present application proposes a configurable and extensible die-level computing architecture, which adopts a single program multiple data (SPMD) model to realize efficient parallel processing. Each die can independently perform computing tasks, and meanwhile, through Chiplet interconnection technology, different dies can share data and resources, realizing more efficient computing.

[0060] The first matrix multiplication module (Die0): Die 0 is responsible for the multiplication operation of the Q matrix and the K matrix. In order to store the column data of the K matrix, Die0 is equipped with a large number of on-chip memories (SRAM). These SRAMs are organized into multiple Banks, and each Bank stores a column block of the K matrix. By accessing these SRAM Banks in parallel, Die0 can simultaneously perform multiple multiplication operations, thereby improving the computing throughput.

[0061] The nonlinear processing module (Die1): Die1 is configured to insert nonlinear operations, such as ReLU activation function, between the matrix chains. In order to realize the flexibility of the function, the nonlinear processing unit of Die1 can be reconfigured to support different nonlinear operations. This design makes the accelerator adapt to different Transformer operators, improving the utilization rate of hardware.

[0062] The second matrix multiplication module (Die2): Die 2 is responsible for performing multiplication operation of the score matrix and the V matrix to generate the final attention matrix. In order to improve the computing efficiency, Die2 adopts high-bandwidth Chiplet interconnection technology to directly receive the processed score(~) data, i.e., the nonlinearly processed score matrix data, from Die1 without passing through external memory. This design reduces the delay of data transmission and improves the overall computing performance.

[0063] In addition, it should be noted that the first matrix multiplication module is used for matrix chain multiplication optimization of the attention mechanism layer of the Transformer network, and the second matrix multiplication module is used for matrix chain multiplication optimization of the feedforward layer of the Transformer network.

[0064] Specifically, the MCM optimization is implemented in two key operations of the Transformer network, namely the calculation of the attention mechanism and the calculation of the feedforward layer. The performance is improved by reducing memory access. Specifically, the MCM optimization is implemented by the following ways, firstly, the MCM optimization of the attention mechanism, when calculating the attention score, the MCM method is used to directly use the result of multiplying the Q matrix and the transpose of the K matrix for the multiplication operation with the V matrix, thereby reducing the number of access to the external memory. Secondly, the MCM optimization of the feedforward layer: in the calculation of the feedforward layer, the MCM method is also used to connect two consecutive matrix multiplication operations, thereby reducing the number of data transmission between different calculation units.

[0065] In addition, it should be noted that, in order to further improve the calculation efficiency, the data flow scheduling is optimized in the embodiments of the present application. By caching key data on-chip memory (SRAM), the number of access to the external memory is reduced. At the same time, the transmission path of data on the die interconnection channel is optimized, the transmission delay is reduced, and the data transmission efficiency is improved. The calculation unit in the embodiments of the present application includes at least one digital signal processor (DSP) for performing multiply-accumulate (MAC) operations. With the support of SRAM, these DSPs can perform matrix multiplication operations in parallel, thereby improving the calculation throughput.

[0066] Please refer to Figure 2 The embodiments of the present application also provide an acceleration method of a Chiplet-based matrix chain multiplication accelerator, which can implement the above-mentioned Chiplet-based matrix chain multiplication accelerator. The method comprises the following steps:

[0067] S100, obtaining operator data of a Tranformer network and performing preprocessing to obtain a matrix chain data stream, the matrix chain data stream comprising Q matrix data, V matrix data and K matrix data;

[0068] It should be noted that in some embodiments, step S100 can comprise: S110, obtaining operator data of a Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data stream; S120, dividing the preliminary matrix chain data stream according to the modular design of the Chiplet technology to obtain the matrix chain data stream.

[0069] In some embodiments, the embodiments of the present application construct on-chip data flow through matrix chain multiplication (MCM), convert the main operator of the Transformer into a matrix chain data flow, and divide the matrix chain data flow into die-to-die data flow through the modular design of the Chiplet technology. In the Transformer network, the self-attention mechanism involves a large number of matrix multiplication operations. The embodiments of the present application connect these matrix multiplication operations through the MCM method to form a continuous calculation flow as shown in Figure 3 In this flow, the result of the previous matrix multiplication is directly used for the next multiplication operation without writing back to the memory, thereby reducing the number of memory accesses and improving the calculation efficiency.

[0070] S200, initializing the matrix chain multiplication accelerator based on the matrix chain data flow to obtain an initialized matrix chain multiplication accelerator;

[0071] It should be noted that in some embodiments, step S200 can include: S210, dividing the calculation unit of the matrix chain multiplication accelerator according to the space size of the on-chip memory of the matrix chain multiplication accelerator to obtain a divided matrix chain multiplication accelerator; S220, pre-storing the V matrix data in the matrix chain data flow and the K matrix data in the matrix chain data flow to the on-chip memory of the divided matrix chain multiplication accelerator, setting the calculation unit of the divided matrix chain multiplication accelerator to a fused multiply-add working mode, and obtaining the initialized matrix chain multiplication accelerator.

[0072] In some embodiments, the matrix multiplication is divided and operated according to the size of the on-chip storage space; the block data of two matrices (matrix K and matrix V) are pre-configured to the on-chip storage space; the calculation units of die0 and die2 are configured to a fused multiply-add working mode, and one of the data interfaces of the multiply-add unit is connected to the on-chip memory.

[0073] S300, inputting the matrix chain data flow to the initialized matrix chain multiplication accelerator for matrix multiplication operation to obtain attention matrix data.

[0074] It should be noted that in some embodiments, step S300 can include: S310, inputting the Q matrix data in the matrix chain data flow to the initialized matrix chain multiplication accelerator; S320, performing parallel inner product calculation on the Q matrix data and the K matrix data to output score matrix data; S330, performing mask operation processing on the score matrix data to obtain nonlinearly processed score matrix data; S340, performing parallel inner product calculation on the nonlinearly processed score matrix data and the V matrix data to output attention matrix data.

[0075] In some embodiments, taking the computation of the attention operator of the Transformer as an example, as shown in FIG. 1 1, Figure 1 The design utilizes three dies to form a chip, where die0 is responsible for the multiplication of matrices Q and K, die2 handles the multiplication of matrix score with matrix V, and die1 is used to insert nonlinear operations between the matrix chains. Specifically, the on-die memory of die0 is used to store the K matrix block containing the complete column data, and then die0 loads the Q matrix from the off-chip memory interface row by row. After that, the inner product is calculated on-chip in parallel, and the score block is output sequentially. The score block is then transmitted to die1 through the inter-die interconnect of the chiplet row by row for masking operation. After the nonlinear processing, the score( ) block is transmitted to die2 again through the chiplet interconnect. Due to the limited on-die storage capacity of die0, the K matrix needs to be partitioned, so it cannot be guaranteed that die2 receives a complete score( ) row. Therefore, the attention matrix needs to load the existing partial and data from the off-chip memory to perform initialization, and then continue to perform calculation, similar to the process of outer product.

[0076] Among them, the key role of Chiplet in MCM computing is:

[0077] 1) Ensure that the on-die storage of die0 can accommodate at least one complete row of K block. In addition, the larger the stored K block, the less additional scheduling the MCM accelerator needs, thereby improving the computing efficiency.

[0078] 2) Allow flexible replacement of the nonlinear processing die to adapt to different operators. For example, by simply replacing the processing on die1 with a ReLU function, the framework can adapt to the computation of the feedforward layer.

[0079] 3) Decouples the resource dependency of the two matrix multiplications. The on-die storage required for the multiplication of score and V matrices increases with the size of the K block. Assuming that both matrix multiplication accelerators are placed on a single die, the manufacturing area limit of a single die becomes a critical issue. Using a chiplet design helps to alleviate this problem.

[0080] Figure 4 and Figure 5The internal architecture of two computing dies for MCM hardware acceleration is shown, which are the first matrix multiplication module and the second matrix multiplication module respectively. In general, they both adopt SPMD architecture, and the programming includes the initialization of registers and SRAM, and the configuration of processing units. For die0, the focus is on performing complete inner product operations between the rows of Q and the columns of K. Therefore, its area resources are mainly allocated to ensure the storage of K blocks, that is, the capacity of SRAM, and then the number of SRAM is expanded. The number of parallel computing paths in die 0 is determined by the number of SRAM. Each computing path includes a DSP running in fused multiply-add (FMA) mode, which can quickly process data streams. The most significant difference between die2 and die0 is their computing data stream sources: die2 relies on high-bandwidth chiplet interconnects rather than memory, so more resources can be directly allocated to parallel processing. Each SRAM in die2 only needs to store part of the columns of the V matrix, because the data in it only needs to perform inner product operations with a set of rows of the fractional matrix to obtain the partial sum of the attention matrix. Under this computing architecture, die2 can further expand the column parallelism of the fractional matrix. By increasing the number of computing paths, multiple columns of the fractional matrix can simultaneously perform inner product operations with the same row of the V matrix, which can greatly improve the on-chip data reuse of the V matrix.

[0081] It should be noted that the matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and the computing units of the matrix chain multiplication accelerator are configured and expanded according to the parallelism of the transmission channel of the matrix chain data stream to perform parallel processing on the data.

[0082] In summary, the data processing of the embodiment of the application is summarized as follows:

[0083] 1) According to the size of the on-chip storage space, the matrix multiplication is divided into block division operations.

[0084] 2) The block data of two matrices (matrix K and matrix V) are pre-configured to the on-chip storage space.

[0085] 3) The computing units of die0 and die2 are configured to work in fused multiply-add mode, and one of the data interfaces of the multiply-add unit is connected to the on-chip memory.

[0086] 4) Start data stream pushing, load each row of matrix Q from the external memory in turn to form a data stream, which passes through two computing dies in turn to complete a matrix block calculation and write back to the main memory.

[0087] 5) Repeat steps 2) to 4) until all matrix block calculations are completed.

[0088] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0089] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A chiplet-based matrix chain multiplication accelerator, characterized by: The matrix chain multiplication accelerator includes a first matrix multiplication module, a first external memory interface, a nonlinear processing module, a second matrix multiplication module, and a second external memory interface. The input end of the first matrix multiplication module is connected to the first external memory interface, the output end of the first matrix multiplication module is connected to the input end of the nonlinear processing module via a chiplet interconnection, the output end of the nonlinear processing module is connected to the input end of the second matrix multiplication module via a chiplet interconnection, and the output end of the second matrix multiplication module is connected to the second external memory interface. The first external memory interface is used to obtain Q matrix data; The first matrix multiplication module is used to perform parallel inner product calculation on the Q matrix data and the K matrix data, and output score matrix data; The nonlinear processing module is used to perform mask operation processing on the score matrix data to obtain score matrix data after nonlinear processing; The second matrix multiplication module is used to perform parallel inner product calculation on the score matrix data after the nonlinear processing and the V matrix data, and output the attention matrix data; The second external memory interface is used to initialize the attention matrix data to achieve Tranformer network inference acceleration.

2. The matrix chain multiplication accelerator according to claim 1, characterized in that The first matrix multiplication module is used to perform matrix chain multiplication optimization on the attention mechanism layer of the Tranformer network, and the second matrix multiplication module is used to perform matrix chain multiplication optimization on the feedforward layer of the Tranformer network.

3. The matrix chain multiplication accelerator according to claim 1, characterized in that The first matrix multiplication module is provided with a plurality of first on-chip memories and a plurality of first computing units, wherein the first on-chip memories are divided into a plurality of Bank sub-modules, wherein: A plurality of the first on-chip memories are used to store the K matrix data, wherein each of the Bank submodules is used to store a column block of the K matrix data; The first computing units are configured to perform a multiplication operation between the Q matrix data and the K matrix data.

4. The matrix chain multiplication accelerator according to claim 1, characterized in that The nonlinear processing module is provided with a plurality of nonlinear processing units, and the nonlinear processing units are used to configure different nonlinear processing functions to support corresponding different nonlinear operations.

5. The matrix chain multiplication accelerator according to claim 1, characterized in that: The second matrix multiplication module is provided with a plurality of second on-chip memories and a plurality of second computing units, wherein: A plurality of the second on-chip memories are used to store the V matrix data; The plurality of second calculation units are used to perform a multiplication operation between the V matrix data and the score matrix data after the nonlinear processing.

6. An acceleration method using the chiplet-based matrix chain multiplication accelerator according to claim 1, characterized in that: The method comprises the following steps: Acquire operator data of the Tranformer network and perform preprocessing to obtain a matrix chain data stream, wherein the matrix chain data stream includes Q matrix data, V matrix data, and K matrix data; Initializing the matrix chain multiplication accelerator based on the matrix chain data stream to obtain an initialized matrix chain multiplication accelerator; The matrix chain data stream is input into the initialized matrix chain multiplication accelerator to perform matrix multiplication operation to obtain attention matrix data.

7. The method according to claim 6, characterized in that The operator data of the Tranformer network is obtained and preprocessed to obtain a matrix chain data flow, including: Acquiring the operator data of the Tranformer network and performing data conversion processing to obtain a preliminary matrix chain data flow; The preliminary matrix chain data flow is divided and processed according to the modular design of the chiplet technology to obtain the matrix chain data flow.

8. The method according to claim 6, characterized in that The method of performing initialization processing on the matrix chain multiplication accelerator based on the matrix chain data stream to obtain an initialized matrix chain multiplication accelerator includes: dividing the computing units of the matrix chain multiplication accelerator according to the space size of the on-chip memory of the matrix chain multiplication accelerator to obtain a divided matrix chain multiplication accelerator; The V matrix data in the matrix chain data stream and the K matrix data in the matrix chain data stream are pre-stored in the on-chip memory of the divided matrix chain multiplication accelerator, and the computing unit of the divided matrix chain multiplication accelerator is set to a fused multiplication-addition working mode to obtain the initialized matrix chain multiplication accelerator.

9. The method according to claim 6, characterized in that Inputting the matrix chain data stream into the initialized matrix chain multiplication accelerator to perform a matrix multiplication operation to obtain attention matrix data includes: Inputting the Q matrix data in the matrix chain data stream into the initialized matrix chain multiplication accelerator; Perform parallel inner product calculation on the Q matrix data and the K matrix data, and output score matrix data; Performing mask operation processing on the score matrix data to obtain score matrix data after nonlinear processing; The score matrix data after the nonlinear processing is parallelized with the V matrix data to output the attention matrix data.

10. The method according to claim 6, characterized in that The matrix multiplication operation of the initialized matrix chain multiplication accelerator adopts a single program multiple data mechanism, and the computing unit of the matrix chain multiplication accelerator configured and expanded according to the parallelism of the transmission channel of the matrix chain data flow processes the data in parallel.

Images