An architecture and acceleration method of a compatible N:M sparse Transformer accelerator

Abstract

The application relates to the technical field of artificial intelligence algorithm hardware acceleration, and provides a compatible N:M sparse Transformer accelerator architecture and an acceleration method.The Transformer accelerator architecture comprises a control unit, a storage unit and a calculation unit; the calculation unit comprises a multi-mode compatible matrix multiplication calculation module, a vector calculation module, a rearrangement module and a Softmax calculation module.In actual application, the multi-mode compatible matrix multiplication calculation module is responsible for the calculation of the matrix multiplication with high operation intensity in the Transformer model, performs the matrix multiplication calculation of two dense matrices or the matrix multiplication calculation of a single N:M structured sparse matrix and a single dense matrix according to dynamic configuration of a control signal; the Softmax calculation module is responsible for the calculation of a Softmax operator in the Transformer model; the vector calculation module is used for accumulating the operation results of the multi-mode compatible matrix multiplication calculation module and sending the accumulated operation results to the rearrangement module; and the rearrangement module is used for rearranging the accumulated operation results and storing the rearranged operation results in an intermediate data storage or an activation storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence algorithm hardware acceleration, and particularly relates to an architecture of a Transformer accelerator compatible with N:M sparsity and an acceleration method. BACKGROUND

[0002] At present, a deep neural network model based on a Transformer (hereinafter referred to as a Transformer model) has been widely applied in technical fields such as machine translation, speech recognition, price prediction, natural language processing or image processing. Since the Transformer model has strong computing capacity, it can be applied to application scenarios with large computing amount and complex operation.

[0003] However, the Transformer model produces a large amount of calculation and parameters, so the Transformer model needs to occupy a large amount of memory and consume super-high computing power during operation, thereby causing the Transformer model to be unable to be directly deployed on an embedded device, which greatly limits the application scenarios of the Transformer model.

[0004] In order to realize efficient deployment of the Transformer model, an existing technology proposes a computing architecture for accelerating the calculation of a multi-head attention module and a feedforward network module in the Transformer model, which improves the inference efficiency of the Transformer. However, the computing architecture does not support sparse Transformer model inference, thereby hindering further performance improvement. Alternatively, an existing technology proposes a compression scheme for a Transformer model, and a high model compression rate is achieved. However, the architectures of the above two schemes do not solve the problem of load imbalance caused by sparsity, thereby causing the actual computing utilization rate of the computing architecture to be relatively low. SUMMARY

[0005] In order to realize efficient deployment of the Transformer model and solve the problem of load imbalance caused by sparsity, thereby causing the actual computing utilization rate of the computing architecture to be relatively low, an embodiment of the present application provides an architecture of a Transformer accelerator compatible with N:M sparsity and an acceleration method.

[0006] An embodiment of the present application provides a first aspect of the architecture of the Transformer accelerator compatible with N:M sparsity.

[0007] The architecture of the Transformer accelerator includes a control unit, a storage unit and a computing unit.

[0008] The control unit is configured to acquire configuration information of an external software end and an operation task of the external software end, generate a control signal dynamic configuration according to the operation task, configure registers of an architecture of the accelerator according to the control signal dynamic configuration, and control running of the computing unit and the storage unit.

[0009] The computing unit includes a multi-mode compatible matrix multiplication computing module, a vector computing module, a rearrangement module, and a Softmax computing module.

[0010] The storage unit includes a weight storage, an activation function storage, and an intermediate data storage.

[0011] The multi-mode compatible matrix multiplication computing module is configured to perform matrix multiplication computation of two dense matrices or matrix multiplication computation of a single N:M structured sparse matrix and a single dense matrix according to the control signal dynamic configuration.

[0012] The Softmax computing module is configured to compute a Softmax operator in a Transformer model.

[0013] The vector computing module is configured to accumulate operation results of the multi-mode compatible matrix multiplication computing module and send the accumulated operation results to the rearrangement module.

[0014] The rearrangement module is configured to rearrange the accumulated operation results and store the rearranged operation results in the intermediate data storage or the activation storage.

[0015] The weight storage is configured to acquire weights stored on an external storage and input the weights to the multi-mode compatible matrix multiplication computing module.

[0016] The activation function storage is configured to acquire activation functions stored on an external storage, store computation results output by the computing unit, and transmit the computation results to the external storage.

[0017] The intermediate data storage is configured to store intermediate data generated by the computing unit.

[0018] In an implementation manner, the control unit includes a general controller, a Softmax controller, an address generator, and a matrix multiplication controller.

[0019] The general controller is configured to acquire configuration information of an external software end and an operation task of the external software end, and distribute the operation task to the Softmax controller, the address generator, and the matrix multiplication controller.

[0020] The matrix multiplication controller is configured to control the multi-mode compatible matrix multiplication computing module to perform a matrix operation task.

[0021] A Softmax controller is configured to control the Softmax module to perform a Softmax operator operation task.

[0022] An address generator is configured to generate a storage address of the read data or the write data.

[0023] In an implementation, the multi-mode compatible matrix multiplication computing module includes eight parallel two-dimensional computing arrays, each of which includes an accumulator and a non-zero element selector, and a plurality of multiplexers and registers arranged at an output end;

[0024] The non-zero element selector is configured to select an activation element corresponding to a non-zero weight.

[0025] The accumulator is configured to perform multiplication operation on the input 16-bit weight and 16-bit activation function, and accumulate the obtained result with a local result stored in the computing unit.

[0026] In an implementation, the Softmax computing module includes an exponential function operator, a data buffer, an accumulation tree module, and a division unit module;

[0027] The data buffer and the accumulation tree module are arranged between the exponential function operator and the division unit module;

[0028] The exponential function operator is configured to split the input data.

[0029] The data buffer is configured to temporarily store the split result of the exponential function operator;

[0030] The accumulation tree module is configured to sum the split result of the exponential function operator;

[0031] The division unit module is configured to perform division operation on the split result of the exponential function temporarily stored in the data buffer and the sum result of the accumulation tree module.

[0032] In an implementation, the division unit module includes a subtractor, a comparator, and a shifter; and a plurality of division unit modules are cascaded through a pipeline.

[0033] ​The second aspect of the application also provides an acceleration method compatible with an N:M sparse Transformer accelerator, which adopts the architecture of the compatible N:M sparse Transformer accelerator provided by the first aspect of the application. For details not disclosed in the second aspect of the application, please refer to the technical solution of the first aspect of the application.

[0034] An acceleration method compatible with an N:M sparse Transformer accelerator, comprising:

[0035] S1, the control unit acquires configuration information and operation tasks loaded by an external software end, generates a control signal for dynamic configuration according to the operation tasks, configures registers of the architecture of the accelerator according to the control signal for dynamic configuration, and controls the operation of the computing unit and the storage unit;

[0036] S2, if the operation task is a multi-head attention module operation, the registers related to the multi-head attention module are configured, and then steps S21 to S25 are executed.

[0037] S21, the control unit activates the address generator, and the weight storage obtains weight data from the external memory storage The total controller obtains the input data X of the completed layer normalization calculation;

[0038] S22, the weight storage obtains the weight data required for the next stage calculation from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication ;

[0039] S23, the weight storage obtains the weight data required for the next stage from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication to obtain matrix , and performs dense matrix multiplication ;

[0040] S24, the weight storage obtains the weight data required for the next stage from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication to obtain matrix , and sequentially sends matrix to the vector calculation module, the rearrangement module, and then writes the rearranged result back to the intermediate data storage area. The Softmax calculation module performs Softmax calculation on matrix to obtain matrix ;

[0041] S25, the multi-mode compatible matrix multiplication calculation module performs dense matrix multiplication obtaining a matrix , and performing sparse matrix multiplication ;

[0042] S3, if the operation task is a feedforward network module operation, configuring the registers related to the feedforward network module, then performing steps S31 to S33;

[0043] S31, the total controller activates the address generator, and the weight storage obtains weight data from the external memory storage The total controller obtains the input data of the layer normalization calculation;

[0044] S32, the weight storage obtains the weight data required for the next stage calculation from the external memory storage The external activation data is written to the corresponding address of the activation function storage, and the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and the vector calculation module performs The ReLU activation function calculation is completed;

[0045] S33, the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and the vector calculation module performs The ReLU activation function calculation is completed.

[0046] From the above technical solutions, the application provides a kind of compatible N:M sparse Transformer accelerator architecture and acceleration method, the architecture of the Transformer accelerator includes control unit, storage unit and computing unit;Control unit is used to obtain the configuration information of external software end, and the operation task of external software end, and according to the operation task, generate control signal dynamic configuration, and, according to control signal dynamic configuration, the register of the architecture of accelerator is configured, and the running of computing unit and storage unit is controlled;Computing unit includes multi-mode compatible matrix multiplication calculation module, vector calculation module, rearrangement module, Softmax calculation module;Storage unit includes weight storage, activation function storage and intermediate data storage.

[0047] In actual application process, the multi-mode compatible matrix multiplication calculation module is responsible for the calculation of the operation amount intensive matrix multiplication in the Transformer model, and can perform the matrix multiplication calculation of two dense matrices or the matrix multiplication calculation of a single N:M structured sparse matrix and a single dense matrix according to the dynamic configuration of the control signal; the flexible Softmax calculation module is responsible for the calculation of the Softmax operator in the Transformer model, and the vector calculation module is responsible for the operation of the accumulation, ReLU and other activation functions, used for accumulating the operation results of the multi-mode compatible matrix multiplication calculation module and sending the accumulated operation results to the rearrangement module; the rearrangement module is used for rearranging the accumulated operation results and storing to the intermediate data storage or the activation storage. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 The overall structure schematic diagram of the architecture of the compatible N:M sparse Transformer accelerator provided by the embodiment of the present application;

[0050] Figure 2 The structure schematic diagram of the multi-mode compatible matrix multiplication calculation module provided by the embodiment of the present application;

[0051] Figure 3 The structure schematic diagram of the single two-dimensional calculation array provided by the embodiment of the present application;

[0052] Figure 4 The structure schematic diagram of the Softmax calculation module provided by the embodiment of the present application;

[0053] Figure 5 The process schematic diagram of the multi-mode compatible matrix multiplication calculation module calculating the dense matrix provided by the embodiment of the present application;

[0054] Figure 6 The process schematic diagram of the multi-mode compatible matrix multiplication calculation module calculating the sparse matrix provided by the embodiment of the present application;

[0055] Figure 7 The timing diagram of the multi-head attention module calculation and the feedforward network module calculation provided by the embodiment of the present application;

[0056] Figure 8 The flow schematic diagram of the acceleration method of the compatible N:M sparse Transformer accelerator provided by the embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments will be described in detail below with reference to examples thereof as illustrated in the accompanying drawings. In the following description, same numbers refer to same or similar elements throughout the drawings. The embodiments described in the following examples do not represent all the implementations consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the appended claims.

[0058] In order to realize efficient deployment of the Transformer model, and solve the problem of load imbalance caused by sparsity, and the problem of relatively low actual computing utilization of the computing architecture, an embodiment of the present application provides a Transformer accelerator architecture compatible with N:M sparsity and an acceleration method.

[0059] Referring to Figure 1 , the first aspect of the embodiment of the present application provides a Transformer accelerator architecture compatible with N:M sparsity, which includes a control unit, a storage unit and a computing unit.

[0060] The control unit includes a total controller, a Softmax controller, an address generator and a matrix multiplication controller.

[0061] The total controller is configured to obtain configuration information of an external software end and an operation task of the external software end, and generate a control signal dynamic configuration according to the operation task, configure registers of the accelerator architecture according to the control signal dynamic configuration, and distribute the operation task to the Softmax controller, the address generator and the matrix multiplication controller. The matrix multiplication controller is configured to control a multi-mode compatible matrix multiplication calculation module of the computing unit to perform a matrix operation task. The Softmax controller is configured to control a Softmax module of the computing unit to perform a Softmax operator operation task. The address generator is configured to generate a storage address of read or write data, thereby controlling the storage unit.

[0062] The computing unit includes a multi-mode compatible matrix multiplication calculation module, a vector calculation module, a rearrangement module and a Softmax calculation module. The storage unit includes a weight storage, an activation function storage and an intermediate data storage.

[0063] The multi-mode compatible matrix multiplication calculation module is responsible for the calculation of the matrix multiplication in the Transformer model, and dynamically configures according to a control signal, so as to perform the matrix multiplication calculation of two dense matrices, or perform the matrix multiplication calculation of a single N:M structured sparse matrix and a single dense matrix; the flexible Softmax calculation module is responsible for the calculation of the Softmax operator in the Transformer model, and the vector calculation module is responsible for the operation of accumulation, ReLU and other activation functions, used for accumulating the operation results of the multi-mode compatible matrix multiplication calculation module, and sending the accumulated operation results to the rearrangement module; the rearrangement module is used for rearranging the accumulated operation results, and storing to an intermediate data storage or an activation storage.

[0064] The storage unit is divided into three sub-modules, respectively responsible for the storage of weights, activation data and intermediate data, specifically, the weight storage is used for obtaining the weights stored on an external storage, and inputting the weights to the multi-mode compatible matrix multiplication calculation module; the activation function storage is used for obtaining the activation function stored on the external storage; and storing the calculation results output by the calculation unit, and transmitting the calculation results to the external storage; and the intermediate data storage stores the intermediate data generated by the calculation unit.

[0065] Referring to Figure 2 The multi-mode compatible matrix multiplication calculation module provided by the embodiment of the application is shown in the structural schematic diagram, and the multi-mode compatible matrix multiplication calculation module includes eight parallel two-dimensional calculation arrays, and a single two-dimensional calculation array can perform the operation of dense matrix multiplication or the operation of sparse matrix multiplication through the configuration of a control unit. The multi-mode operation compatible bottom calculation unit is shown in Figure 3 The single two-dimensional calculation array includes an accumulator, a non-zero element selector, and multiple multiplexers and registers. The accumulator accepts two 16-bit inputs, first performs multiplication, then obtains a 32-bit multiplication result, and then accumulates the obtained multiplication result with a local result stored in the calculation unit to obtain an accumulated output result. The non-zero element selector is only activated in the calculation mode of sparse matrix multiplication, and is used for selecting the activation elements corresponding to the non-zero weights for calculation.

[0066] Referring to Figure 5 The process diagram of the multi-mode compatible matrix multiplication calculation module for calculating a dense matrix is shown in Figure 6 The process diagram of the multi-mode compatible matrix multiplication calculation module for calculating a sparse matrix is shown in

[0067] In dense matrix computation mode, activation data is fed in from the row direction of the multi-mode compatible matrix multiplication computation module, and weight data is fed in from the column direction of the multi-mode compatible matrix multiplication computation module. Each cycle, the data is fed in... In sparse matrix computation mode, the positions of activation data and weight data remain unchanged. However, in each cycle, multiple times the amount of computation data needs to be fed in, and the corresponding bitmask of sparse weight data also needs to be read to participate in the computation. When the same weight data is read in, the computation cycle of sparse matrix is ​​significantly shorter than that of dense matrix, which demonstrates the high computational efficiency of sparse matrix computation mode.

[0068] See Figure 4 This is a schematic diagram of the structure of the Softmax calculation module provided in an embodiment of this application. The Softmax calculation module includes... The system comprises an exponential function arithmetic unit, a data buffer, an accumulation tree module, and a division unit module; the data buffer and the accumulation tree module are both located in the... Between the exponential function arithmetic unit and the division unit module.

[0069] in, The exponential function arithmetic unit is used to segment the input data; specifically, The exponential function arithmetic unit employs a hybrid approach of lookup tables and Taylor expansion. Its input is divided into high-bit and low-bit segments. For the high-bit segment, which has a greater impact on the result's accuracy, a lookup table is used to directly retrieve the result, while the low-bit segment is approximated using a first-order Taylor expansion. A data buffer is used for temporary storage. The segmentation result of the exponential function arithmetic unit is used by the subsequent division unit module. The accumulation tree module is used for... The exponential function arithmetic unit sums the partitioned results, and the configurable accumulation length counter can adapt to inputs of different vector lengths, improving hardware flexibility. The division unit module enhances parallelism by incorporating a multi-stage pipeline. A simple division unit module consists of a subtractor, a comparator, and a shifter. Multiple simple division unit modules can be cascaded through a pipeline to obtain division results that meet the required precision.

[0070] To more clearly illustrate the implementation process of the architecture of an N:M sparse Transformer accelerator provided in this application, please refer to... Figure 7 This is a timing diagram of the multi-head attention module computation and the feedforward network module computation provided in the embodiments of this application.

[0071] The architecture of the Transformer accelerator and the data stored in the external memory are shown in each computing stage, and the operations performed by the computing units inside the architecture of the Transformer accelerator. The layer normalization operation (LN) is completed outside the architecture of the Transformer accelerator, while other key operators of the Transformer are completed inside the architecture of the Transformer accelerator.

[0072] The calculation of the multi-head attention module is divided into five stages.

[0073] The first stage: the total controller activates the address generator, and the weight memory obtains the weight data from the external memory storage The total controller obtains the input data X of the layer normalization calculation.

[0074] The second stage: the weight memory obtains the weight data required for the next stage calculation from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and the calculated matrix is sent to the vector calculation module and the rearrangement module in turn, and then the rearranged matrix is temporarily stored in the intermediate data storage.

[0075] The third stage: the weight memory obtains the weight data required for the next stage from the external memory storage At the same time, the multi-mode compatible matrix multiplication calculation module first performs sparse matrix multiplication to obtain matrix , and stores the obtained matrix in the intermediate data storage, and then the multi-mode compatible matrix multiplication calculation module takes out matrix and matrix from the intermediate data storage area respectively, performs dense matrix multiplication , and then sends the obtained matrix to the vector calculation module and the rearrangement module in turn, and then writes the rearranged matrix back to the intermediate data storage.

[0076] The fourth stage: the weight memory obtains the weight data required for the next stage from the external memory storage In the computing unit, the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication to obtain matrix , and sends matrix to the vector calculation module and the rearrangement module in turn, and then writes the rearranged matrix back to the intermediate data storage, and the Softmax calculation module calculates matrix Softmax calculation is performed to obtain a matrix , and the obtained matrix is written into the intermediate data storage.

[0077] The fifth stage: the multi-mode compatible matrix multiplication calculation module first performs dense matrix multiplication to obtain a matrix , and the matrix is sequentially sent to the vector calculation module and the rearrangement module, and then the rearranged matrix is written back to the intermediate data storage area; then sparse matrix multiplication is performed, and the operation result obtained from the matrix multiplication module is directly sent to the vector calculation module to complete residual calculation, and then the result is written to the activation storage after passing through the rearrangement module.

[0078] The calculation of the feedforward network module is divided into three stages:

[0079] The first stage: the total controller activates the address generator, and the weight storage obtains weight data from the external storage memory. The total controller obtains the input data of the layer normalization calculation.

[0080] The second stage: the weight storage obtains the weight data required for the next stage calculation from the external storage memory, the external activation data is written to the corresponding address of the activation function storage, the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , the operation result obtained from the matrix multiplication module is directly sent to the vector calculation module to complete the activation function ReLU calculation, and finally the result is sequentially sent to the vector calculation module and the rearrangement module, and then written to the intermediate data storage.

[0081] The third stage: the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and directly sends the result to the vector calculation module to complete residual calculation, and finally the result is sequentially sent to the vector calculation module and the rearrangement module, and then written to the activation storage.

[0082] The second aspect of the embodiment of the application further provides an acceleration method of a compatible N:M sparse Transformer accelerator, which adopts the architecture of the compatible N:M sparse Transformer accelerator provided in the first aspect of the embodiment of the application. For details not disclosed in the second aspect of the embodiment of the application, please refer to the technical solution of the first aspect of the embodiment of the application.

[0083] See Figure 8 ​The method for accelerating the accelerator compatible with N:M sparse Transformers comprises the following steps:

[0084] S1, the control unit acquires configuration information and operation tasks loaded by external software, generates a control signal according to the operation tasks, dynamically configures the registers of the architecture of the accelerator according to the control signal, and controls the operation of the computing unit and the storage unit.

[0085] S2, if the operation task is a multi-head attention module operation, the multi-head attention module related registers are configured, and steps S21 to S25 are executed.

[0086] S21, the control unit activates the address generator, and the weight storage obtains weight data from the external memory storage The total controller obtains the input data X of the completed layer normalization calculation.

[0087] S22, the weight storage obtains the weight data required for the next stage calculation from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication .

[0088] S23, the weight storage obtains the weight data required for the next stage from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication to obtain matrix , and performs dense matrix multiplication .

[0089] S24, the weight storage obtains the weight data required for the next stage from the external memory storage The multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication to obtain matrix , and sequentially sends matrix to the vector calculation module, the rearrangement module, and then writes the rearranged result back to the intermediate data storage area. The Softmax calculation module performs Softmax calculation on matrix to obtain matrix .

[0090] S25, the multi-mode compatible matrix multiplication calculation module performs dense matrix multiplication to obtain matrix , and performs sparse matrix multiplication .

[0091] S3, if the operation task is a feedforward network module operation, the feedforward network module related registers are configured, and steps S31 to S33 are executed.

[0092] S31, the total controller activates the address generator, and the weight memory acquires the weight data from the external memory storage The total controller acquires the input data of the layer normalization calculation.

[0093] S32, the weight memory acquires the weight data required for the next stage calculation from the external memory storage The external activation data is written to the corresponding address of the activation function memory, the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and the vector calculation module performs The activation function ReLU calculation is completed.

[0094] S33, the multi-mode compatible matrix multiplication calculation module performs sparse matrix multiplication , and the vector calculation module performs The activation function ReLU calculation is completed.

[0095] From the above technical solutions, the embodiment of the application provides a kind of compatible N:M sparse architecture and acceleration method of Transformer accelerator, the architecture of the Transformer accelerator includes control unit, storage unit and computing unit;Control unit is used to obtain the configuration information of external software end, and the operation task of external software end, and according to the operation task, generate control signal dynamic configuration, and, according to control signal dynamic configuration, the register of the architecture of accelerator is configured, and the running of computing unit and storage unit is controlled;Computing unit includes multi-mode compatible matrix multiplication calculation module, vector calculation module, rearrangement module, Softmax calculation module;Storage unit includes weight memory, activation function memory and intermediate data storage.

[0096] In actual application process, the multi-mode compatible matrix multiplication calculation module is responsible for the calculation of the matrix multiplication of the operation amount intensive in Transformer model, according to control signal dynamic configuration, both dense matrix matrix multiplication calculation can be executed, also single N:M structured sparse matrix and single dense matrix matrix multiplication calculation can be executed;Flexible Softmax calculation module is responsible for calculating the Softmax operator in Transformer model, the vector calculation module is responsible for the operation of accumulation, ReLU and other activation functions, for the operation result of the multi-mode compatible matrix multiplication calculation module is accumulated, and the accumulated operation result is sent to rearrangement module;The rearrangement module is used to rearrange the accumulated operation result, and store to intermediate data storage or activation memory.

[0097] The similar parts among the embodiments provided in the application can be referred to each other, the specific embodiments provided above are only several examples under the general concept of the application, and do not constitute the limitation of the protection scope of the application. Any other embodiments extended according to the application scheme without creative labor for those skilled in the art shall fall within the protection scope of the application.

Claims

1. An architecture for an N:M sparse Transformer accelerator, characterized in that, The architecture of the Transformer accelerator includes a control unit, a storage unit, and a computing unit; The control unit is used to acquire configuration information and computing tasks from the external software, generate dynamic configuration control signals based on the computing tasks, configure the registers of the accelerator architecture based on the dynamic configuration control signals, and control the operation of the computing unit and the storage unit. The computing unit includes a multi-mode compatible matrix multiplication computing module, a vector computing module, a rearrangement module, and a Softmax computing module; The storage unit includes a weight memory, an activation function memory, and an intermediate data memory; The multi-mode compatible matrix multiplication calculation module is configured dynamically according to control signals to perform matrix multiplication calculations of two dense matrices, or to perform matrix multiplication calculations of a single N:M structured sparse matrix and a single dense matrix. The Softmax calculation module is used to calculate the Softmax operator in the Transformer model. The vector calculation module is used to accumulate the calculation results of the multi-mode compatible matrix multiplication calculation module and send the accumulated calculation results to the rearrangement module; The rearrangement module is used to rearrange the accumulated calculation results and store them in an intermediate data memory or an activation memory. The weight memory is used to retrieve the weights stored in the external memory and input the weights into the multi-mode compatible matrix multiplication calculation module; Activation function memory, used to retrieve activation functions stored on external memory; And the calculation results output by the storage computing unit, and the calculation results are transferred to an external memory; Intermediate data storage unit stores intermediate data generated by the computing unit. The multi-mode compatible matrix multiplication calculation module includes eight parallel... A two-dimensional computing array, a single two-dimensional computing array includes an accumulator and a non-zero element selector, as well as multiple multiplexers and registers set at the output; The non-zero element selector is used to select the activation element corresponding to a non-zero weight; The accumulator is used to perform a multiplication operation on the input 16-bit weights and 16-bit activation function, and then accumulate the result of the operation with the local result stored inside the computing unit. The Softmax calculation module includes Exponential function arithmetic unit, data buffer, accumulation tree module, and division unit module; The data buffer and the accumulator tree module are both located in the... Between the exponential function arithmetic unit and the division unit module; An exponential function arithmetic unit is used to segment input data; Data buffer, used for temporary storage The segmentation result of the exponential function arithmetic unit; The accumulation tree module is used for... The results of the exponential function operator's segmentation are summed. The division unit module is used for temporarily storing data in the data buffer. The splitting result of the exponential function is divided by the summation result of the accumulator tree module; The division unit module consists of a subtractor, a comparator, and a shifter; multiple division unit modules are cascaded in a pipeline.

2. The architecture of an N:M sparse Transformer accelerator according to claim 1, characterized in that, The control unit includes a main controller, a Softmax controller, an address generator, and a matrix multiplication controller; The main controller is used to obtain configuration information and computing tasks from the external software, and to distribute the computing tasks to the Softmax controller, address generator and matrix multiplication controller. A matrix multiplication controller is used to control the multi-mode compatible matrix multiplication calculation module to perform matrix operation tasks; The Softmax controller is used to control the Softmax module to perform Softmax operator operations. An address generator is used to generate storage addresses for accessing or writing data.

3. An acceleration method compatible with N:M sparse Transformer accelerators, characterized in that, The architecture of the N:M sparse Transformer accelerator as described in any one of claims 1-2 includes: S1, the control unit obtains the configuration information and computing tasks loaded by the external software, generates control signals for dynamic configuration based on the computing tasks, configures the registers of the accelerator architecture based on the dynamic configuration of the control signals, and controls the operation of the computing unit and the storage unit. S2, If the computation task is a multi-head attention module operation, then configure the relevant registers of the multi-head attention module and execute steps S21 to S25; S21, The control unit activates the address generator, and the weight memory retrieves data from external memory. Weight data: The central controller obtains the input data X to complete the layer normalization calculation; S22, the weight memory retrieves the data needed for the next stage of computation from external memory. Weighted data, multi-modal compatible matrix multiplication calculation module performs sparse matrix multiplication ; S23, the weight memory retrieves the weights needed for the next stage from external memory. Weighted data, multi-modal compatible matrix multiplication calculation module performs sparse matrix multiplication Obtain the matrix And, perform dense matrix multiplication. ; S24, the weight memory retrieves the weights needed for the next stage from external memory. Weighted data, multi-modal compatible matrix multiplication calculation module performs sparse matrix multiplication Obtain the matrix , matrix The data is sequentially fed into the vector calculation module, the rearrangement module, and then the rearranged result is written back to the intermediate data storage area. The Softmax calculation module processes the matrix... Softmax calculation yields the matrix ; S25, the multi-modal compatible matrix multiplication calculation module performs dense matrix multiplication. Obtain the matrix And, perform sparse matrix multiplication. ; S3, if the computation task is a feedforward network module operation, then configure the relevant registers of the feedforward network module, and then execute steps S31 to S33. S31, the main controller activates the address generator, and the weight memory is retrieved from external memory. Weight data: The central controller obtains the input data to complete the layer normalization calculation. S32, the weight memory retrieves the data needed for the next stage of computation from external memory. Weight data and external activation data are written to the corresponding addresses in the activation function memory. The multi-modal compatible matrix multiplication calculation module performs sparse matrix multiplication. And, the vector calculation module for Complete the ReLU activation function calculation; S33, the multi-modal compatible matrix multiplication calculation module performs sparse matrix multiplication. And, the vector calculation module for Complete the ReLU activation function calculation.

4. The acceleration method for an N:M sparse Transformer accelerator according to claim 3, characterized in that, The control unit includes a main controller, a Softmax controller, an address generator, and a matrix multiplication controller; The main controller is used to obtain configuration information and computing tasks from the external software, and to distribute the computing tasks to the Softmax controller, address generator and matrix multiplication controller. A matrix multiplication controller is used to control the multi-modal compatible matrix multiplication calculation module to perform matrix operation tasks, including dense matrix multiplication and sparse matrix multiplication. The Softmax controller is used to control the Softmax calculation module to perform Softmax operator operations. An address generator is used to generate storage addresses for accessing or writing data.

Images