An acceleration system and method based on Transformer

Abstract

This application provides a Transformer-based acceleration system and method. The system includes a hardware accelerator and a neural network module. The neural network module runs a neural network model. The hardware accelerator includes a memory, a first acceleration module, and a second acceleration module. The first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units. The first acceleration module accumulates and calculates attention data by using the reconfigurable units to perform calculations on key-value pairs. The second acceleration module calculates multilayer perceptron data based on weight data and feature map data by using the processing units. The neural network module outputs classification results based on the attention data and the multilayer perceptron data. This application accelerates the computation process in the neural network model through the acceleration module, achieving overall acceleration of the neural network model and improving the efficiency of the neural network model in processing data.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to an acceleration system and method based on Transformer. Background Technology

[0002] The Transformer is a deep learning model used to process sequential data, such as text sequences in natural language. The Transformer model uses a self-attention mechanism, enabling it to understand and process dependencies within the input sequence. Furthermore, the Transformer model employs techniques such as multilayer perceptron architecture and positional encoding, achieving both efficiency and accuracy when processing complex sequential data.

[0003] Because the Transformer model uses self-attention and a multi-layered perceptron architecture, it has a large number of parameters and high computational complexity. During training, it requires longer training times and more computational resources. During inference, it also results in slower model execution speed, making it unable to process large-scale sequence data in real time. Summary of the Invention

[0004] To accelerate the computation speed of neural network models when processing data, in a first aspect, this application provides a Transformer-based acceleration system, including a hardware accelerator and a neural network module, wherein the hardware accelerator and the neural network module are two independent hardware modules; wherein, the neural network module runs a neural network model; the hardware accelerator includes a memory, a first acceleration module and a second acceleration module, the first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units;

[0005] The memory is used to store the data to be processed acquired by the neural network module, including weight data, feature map data, and key-value pair data.

[0006] The first acceleration module is configured to perform an accumulation calculation on the key-value pair data through the reconfigurable unit to obtain attention data;

[0007] The second acceleration module is configured to calculate multilayer perceptron data based on the weight data and the feature map data through the processing unit;

[0008] The neural network module is configured to output a classification result based on the attention data and the multilayer perceptron data.

[0009] In some embodiments, the hardware accelerator further includes a third acceleration module, the third acceleration module including a multiply-accelerate array, the multiply-accelerate array including a plurality of adders and a plurality of multipliers;

[0010] The third acceleration module is configured as follows:

[0011] Detect the normalized layer of the neural network module;

[0012] If the normalization layer exists, the mean and variance are calculated using the adder and the multiplier based on the weight data and the feature map data.

[0013] The normalized layer result is calculated based on the mean and the variance.

[0014] In some embodiments, the memory includes an off-chip storage module, an on-chip first buffer, an on-chip second buffer, and an on-chip third buffer;

[0015] The on-chip first buffer is used to store the weight data input by the off-chip storage module;

[0016] The on-chip second buffer is used to store the feature map input data input by the off-chip storage module and the feature map output data output by the first acceleration module.

[0017] The on-chip third buffer is used to store key-value pair data input by the off-chip storage module.

[0018] In some embodiments, the key-value pair data includes a query matrix, a key matrix, and a value matrix. The first acceleration module performs an accumulation calculation on the key-value pair data through the reconfigurable unit and is further configured to:

[0019] The query matrix, the key matrix, and the value matrix are input into the reconfigurable unit;

[0020] Attention data is calculated using the following formula:

[0021]

[0022] Where A is the output attention data, Q is the query matrix, K is the key matrix, V is the value matrix, T represents the transpose, and d is the feature map dimension of the query matrix, key matrix, or value matrix.

[0023] In some embodiments, the reconfigurable unit includes a first register, and the reconfigurable unit alternately calculates the attention data using a first accumulation mode and a second accumulation mode, wherein the first accumulation mode is a self-accumulation mode and the second accumulation mode is a directional accumulation mode; when the reconfigurable unit is in the first accumulation mode, the first acceleration module performs the operation of inputting the query matrix, the key matrix, and the value matrix into the reconfigurable unit, configured as follows:

[0024] The query matrix and the key matrix are input into the reconfigurable unit to calculate the first product of the query matrix and the key matrix;

[0025] The first register caches the first register value of the reconstructible cell from the previous cycle.

[0026] The preset sum of the first product is added to the first register value of the reconfigurable unit in the previous cycle to obtain the first sum, and the first sum is cached in the first register. The preset sum of the first product is the sum of a preset number of first products.

[0027] In some embodiments, the reconfigurable unit further includes a second register. When the reconfigurable unit is in a second accumulation mode, the first acceleration module is configured to input the query matrix, the key matrix, and the value matrix into the reconfigurable unit.

[0028] The first sum and the value matrix are input into the reconfigurable unit to calculate the second product of the first sum and the value matrix;

[0029] The second product is transmitted to an adjacent reconfigurable unit via the second register, so that the second product is iteratively summed by the adjacent reconfigurable unit to obtain attention data. The adjacent reconfigurable unit is adjacent to the reconfigurable unit in the transmission direction.

[0030] In some embodiments, the first acceleration module is configured to input the query matrix and the key matrix into the reconfigurable unit, as follows:

[0031] The query matrix is ​​input row by row into the cell matrix of the reconfigurable cell, and the key matrix is ​​input column by column into the cell matrix of the reconfigurable cell;

[0032] Perform a multiplication and accumulation calculation on the unit matrix to obtain a first accumulated value, which includes the first product.

[0033] In some embodiments, the processing unit includes an adder, a multiplier, and a third register, and the second acceleration module performs calculation of multilayer perceptron data based on the weight data and the feature map data, and is configured to:

[0034] The weight data and the feature map data are input into the multiplier to calculate the third product of the weight data and the feature map data;

[0035] The second register value of the previous cycle is processed by the third register cache processing unit;

[0036] The preset sum of the third product is added to the second register value of the previous cycle to obtain the second sum, and the third register is updated according to the second sum. The preset sum of the third product is the sum of a preset number of third products.

[0037] The multilayer perceptron data is output through the updated third register.

[0038] In some embodiments, the second acceleration module is configured to input the weight data and the feature map data into the multiplier as follows:

[0039] The feature map data is input into the multiplier matrix row by row, and the weight data is input into the multiplier matrix column by column;

[0040] Perform a multiply-accumulate operation on the multiplier matrix to obtain a third accumulated value, which includes the third product.

[0041] Secondly, some embodiments of this application provide a Transformer-based acceleration method applied to a Transformer-based acceleration system. The system includes a hardware accelerator and a neural network module, which are two independent hardware modules. The neural network module runs a neural network model. The hardware accelerator includes a memory, a first acceleration module, and a second acceleration module. The first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units. The method includes:

[0042] Acquire the data to be processed, which includes weight data, feature map data, and key-value pair data;

[0043] Attention data is obtained by performing an accumulation calculation on the key-value pair data through the reconfigurable unit.

[0044] The processing unit calculates multilayer perceptron data based on the weight data and the feature map data.

[0045] The neural network module outputs classification results based on the attention data and the multilayer perceptron data.

[0046] As can be seen from the above technical solutions, this application provides a Transformer-based acceleration system and method. The system includes a hardware accelerator and a neural network module. The neural network module runs a neural network model. The hardware accelerator includes a memory, a first acceleration module, and a second acceleration module. The first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units. The first acceleration module accumulates and calculates attention data by using the reconfigurable units to accumulate key-value pair data. The second acceleration module calculates multilayer perceptron data based on weight data and feature map data by using the processing units. The neural network module outputs classification results based on the attention data and the multilayer perceptron data. This application accelerates each computational process in the neural network model through the acceleration module, achieving overall acceleration of the neural network model and improving the efficiency of the neural network model in processing data. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the structure of a neural network model;

[0049] Figure 2 An architecture diagram of a Transformer-based acceleration system provided for embodiments of this application;

[0050] Figure 3 The memory architecture diagram provided for embodiments of this application;

[0051] Figure 4 This is a schematic diagram of the reconfigurable unit in the embodiments of this application;

[0052] Figure 5 This is a flowchart illustrating the workflow of the reconfigurable unit in the first accumulation mode in an embodiment of this application.

[0053] Figure 6 This is a flowchart illustrating the workflow of the reconfigurable unit in the second accumulation mode in an embodiment of this application.

[0054] Figure 7 This is an architecture diagram of the processing unit in the embodiments of this application;

[0055] Figure 8 This is a flowchart illustrating the process of calculating the first product in an embodiment of this application. Detailed Implementation

[0056] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0057] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0058] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0059] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0060] The Transformer is a neural network model for processing sequential data, applicable to the field of natural language processing. Transformer models can perform natural language processing tasks such as machine translation, text classification, and sentiment analysis. They can efficiently process sequential data in parallel, and during this process, a self-attention mechanism can capture long-distance dependencies in the input sequence. For example, a Transformer model can be used to process text sequences in natural language, thereby obtaining the dependencies between text sequences through the attention mechanism.

[0061] Attention modules, multilayer perceptron architectures, and positional encoding, among other components, offer high efficiency and accuracy when processing complex sequential data. The Transformer model can be composed of attention modules and multilayer perceptron modules connected sequentially. When processing data, the Transformer model needs to perform computations on the data through the attention module and the multilayer perceptron in sequence. To improve computational efficiency, for example… Figure 1 As shown, the Transformer model can parallelize the attention module and the multilayer perceptron module, enabling them to process data in parallel and improving the processing speed of the Transformer model.

[0062] However, due to the large number of parameters and computational complexity of self-attention mechanisms and multilayer perceptron architectures, training and application require more time and computational resources. Simply changing the order of the neural network architecture cannot significantly improve processing speed, resulting in slow model operation and an inability to process large-scale sequence data in real time.

[0063] To improve the running speed of Transformer models, some embodiments use hardware accelerators to facilitate their deployment at the hardware level. However, hardware accelerators only accelerate parts of the Transformer model's structure, such as matrix multiplication, linear layers, or self-attention layers, and do not fully accelerate the entire network structure. When there is a need to accelerate the entire Transformer model, hardware accelerators cannot achieve the expected acceleration effect.

[0064] To accelerate the computation speed of neural network models when processing data, some embodiments of this application provide a Transformer-based acceleration system. Figure 2 This is an architecture diagram of a Transformer-based acceleration system provided for embodiments of this application. See also... Figure 2 The system includes a hardware accelerator and a neural network module. The neural network module runs a neural network model and can be integrated into the system-on-a-chip of an electronic device, enabling the electronic device to run the neural network model through the neural network module.

[0065] In this embodiment, the Transformer model is used as an example of a neural network model for illustration. In practical applications, other neural network models can also be used as the models for running the neural network system. Hardware accelerators are used to accelerate the process of the neural network model calculating the data to be processed, thereby improving the running speed of the neural network model.

[0066] The neural network model includes an attention module and a multilayer perceptron, which can be configured in parallel to reduce data processing time. The hardware accelerator may include a memory, a first acceleration module, and a second acceleration module. The memory stores the data to be processed acquired by the neural network module. The data to be processed can be determined based on the task type of the Transformer model. For example, the Transformer model can perform natural language processing tasks or computer vision tasks. When performing a natural language processing task, the data to be processed is the text data acquired by the electronic device; when performing a computer vision task, the data to be processed is the visual image data captured by the electronic device.

[0067] It should be noted that electronic devices can collect data to be processed in various ways and transmit the collected data to the neural network model. For example, when the Transformer model performs a natural language processing task, the data to be processed is text data. In this case, the electronic device can generate a collection command. After receiving the collection command, the electronic device can run the text collection interface or start the text data transmission channel to obtain the data to be processed.

[0068] Since the available storage capacity of the system-on-a-chip is limited, in order to save storage space on the system-on-a-chip when reading data to be processed, the memory can include an off-chip storage module. An off-chip storage module is a type of storage hardware, such as a solid-state drive, a hard disk drive, an optical disc, or a USB flash drive. After receiving the data to be processed uploaded by the neural network model, the memory can store the data to be processed in the off-chip storage module to save storage space on the system-on-a-chip.

[0069] like Figure 3 As shown, the memory can be configured with a global buffer (GB). In neural network processing, the global buffer can be used to store shared data or other important intermediate results. Multiple processors or processing units can access and use this data simultaneously, thereby enabling parallel processing and accelerating the inference process.

[0070] The global buffer can include an on-chip first buffer, an on-chip second buffer, and an on-chip third buffer. The on-chip first buffer can store weight data input from the off-chip storage module, the on-chip second buffer can store feature map input data input from the off-chip storage module, and the on-chip third buffer is used to store key-value pair data input from the off-chip storage module. This allows the on-chip system to directly read the required data from the global buffer when running the neural network model, reducing the time latency of data access and thus improving the processing speed.

[0071] After preprocessing, the data to be processed can include weight data, feature map data, and key-value pair data. The weight data are parameters automatically learned during the training of the neural network model. During training, the weight data is adjusted based on the training data to enable the network to better learn and predict the features of the data to be processed. The initial values ​​of the weight data can be randomly set and continuously updated during training. The weight update rule can be based on the gradient descent algorithm, that is, adjusting the weight data according to the gradient of the loss function with respect to the weights to minimize the loss function and improve prediction accuracy.

[0072] In some embodiments, the feature map data includes feature maps obtained by sequentially processing the data to be processed through an attention module and a multilayer perceptron. For example, after the electronic device inputs the acquired image data into the neural network model, the attention module and the multilayer perceptron in the neural network model will extract the image features of the image data and generate feature map data through operations such as activation functions and pooling. These feature map data are intermediate representations of the neural network model, used to extract features of the input data to be processed, and processed and classified in subsequent network structures.

[0073] Key-value pair data includes a query matrix (Q), a key matrix (K), and a value matrix (V). The query matrix, key matrix, and value matrix are all obtained by linear transformation of the data to be processed. At each sequence position of the data to be processed, there exists a query vector, a key vector, and a value vector. The query vectors at each position are summarized into a query matrix, the key vectors at each position are summarized into a key matrix, and the value vectors at each position are summarized into a value matrix. By calculating the similarity between the query vector and the key vector, the weight of each sequence position can be obtained. The calculated weights are then applied to the value vectors at the corresponding sequence positions to obtain a weighted output sequence, thereby better capturing the dependencies of the data to be processed.

[0074] The first acceleration module can accelerate the computation process of the attention module of the neural network model. The first acceleration module includes multiple reconfigurable processing elements (RPEs). When the neural network model performs the computation of the attention module, it reads key-value pair data in the on-chip third buffer. The first acceleration module can perform cumulative computation on the key-value pair data through the reconfigurable processing elements, thereby outputting attention data. The attention data represents the dependency relationship between the data sequences to be processed.

[0075] In some embodiments, the first acceleration module may input the query matrix, key matrix, and value matrix into the reconfigurable unit and calculate the attention data using the following formula to accelerate the attention module through the reconfigurable unit.

[0076]

[0077] Where A is the output attention data, Q is the query matrix, K is the key matrix, V is the value matrix, T represents the transpose, and d is the feature map dimension of the query matrix, key matrix, or value matrix.

[0078] To perform the calculation of the above formula, the reconfigurable unit includes an adder and a multiplier to implement the multiplicative accumulation function. The reconfigurable unit can alternately calculate the attention data in a first accumulation mode and a second accumulation mode, wherein the first accumulation mode is a self-accumulation mode and the second accumulation mode is a directional accumulation mode.

[0079] Figure 4 This is a schematic diagram of the reconfigurable unit in an embodiment of this application. See also... Figure 4 The reconfigurable unit also includes three multiplexers: a first multiplexer C1, a second multiplexer C2, and a third multiplexer C3, as well as a first register and a second register. The multiplexers enable the reconfigurable functionality of the reconfigurable unit. The reconfigurable unit contains an exponential unit (EXP) constructed from a lookup table (LUT). The exponential unit converts exponentiation operations into lookup table lookup operations, achieving fast and accurate exponentiation and thus reducing hardware resource consumption.

[0080] like Figure 5 As shown, when the reconfigurable unit calculates QK T When in the first accumulation mode, the query matrix is ​​input to the reconfigurable unit through the first multiplexer C1, and the key matrix is ​​directly input to the reconfigurable unit. The multiplier calculates the first product of the query matrix and the key matrix, and transmits the preset sum value of the first product to the adder. The preset sum value of the first product is the sum value of the first product of a preset number. In this embodiment, the first product of a preset period can be selected for addition. For example, if the first product of ten periods needs to be accumulated, then the sum value is the result of the first product accumulation calculated for ten periods.

[0081] After calculating the first product, the first register can cache the first register value of the reconfigurable unit in the previous cycle. The first register is used to cache the attention score of the reconfigurable unit during the accumulation process. The second multiplexer C2 can select the preset sum and value of the first register value in the previous cycle from the data cached in the first register, and add it to the preset ratio and value of the first product through the adder to obtain the first sum. Finally, the first sum is cached and updated in the first register through the third multiplexer C3 to perform self-accumulation.

[0082] After completing QK T Afterwards, as Figure 6 As shown, the reconfigurable unit is in the second accumulation mode. The first multiplexer C1 selects the first sum in the first register and multiplies it with the value matrix through a multiplier to obtain the second product. After selection by the third multiplexer C3, the product is input to the adjacent reconfigurable unit through the second register. The adjacent reconfigurable unit is adjacent to the reconfigurable unit in the transmission direction. The adjacent reconfigurable unit performs iterative summation on the second product to obtain the attention data.

[0083] In some embodiments, the reconfigurable unit further includes an additional adder to implement the Softmax function in the attention module, as follows:

[0084]

[0085] Where X is the input feature map data.

[0086] An additional adder can perform a transfer-direction accumulation of the first register value in the first register and the first register value in the first register of the adjacent reconfiguration unit when the reconfigurable unit processes the second accumulation mode. This process continues until the transfer-direction accumulation is completed for all reconfiguration units, thereby obtaining attention data and completing the accelerated calculation of attention.

[0087] In some embodiments, after accelerating the attention calculation, the first acceleration module can also output feature map output data based on the attention data and cache the feature map output data in the on-chip third cache area so that the memory or neural network model can read it.

[0088] To facilitate the calculation of the first and second products, such as Figure 7 As shown, the reconfigurable unit can also include a unit matrix. When the first acceleration module calculates the first product of the query matrix and the key matrix, it can input the query matrix row-wise into the unit matrix of the reconfigurable unit, and input the transpose of the key matrix column-wise into the unit matrix of the reconfigurable unit. This allows the multiplier to perform a multiplication and accumulation calculation on the unit matrix, completing the periodic multiplication and accumulation of different rows of the query matrix and different columns of the transpose of the key matrix within the reconfigurable unit, thus obtaining the first accumulated value. The multiplication and accumulation operation is a method of operation that adds the first product to the value of the accumulator and then stores the result in the accumulator. Therefore, the first accumulated value includes the first product.

[0089] The second acceleration module includes multiple processing elements (PEs). These processing elements accelerate the computation process of the multilayer perceptron based on weight data and feature map data, and generate the multilayer perceptron data. The multilayer perceptron consists of two fully connected layers, each calculated using the following matrix:

[0090] Y = X * W;

[0091] Where Y is the output feature map data, X is the input feature map data, and W is the weight data.

[0092] To speed up the multiplication of the above matrices, such as Figure 8As shown, the processing unit includes adders and multipliers to implement the multiplication and accumulation function of the processing unit. The processing unit also includes a third register to cache preset values ​​in self-accumulation mode. When calculating the matrix of the fully connected layer, the processing unit is in self-accumulation mode and inputs the weight data and feature map data into the multiplier, which calculates the third product of the weight data and feature map data.

[0093] After the feature map data and weight data are multiplied, the second acceleration module caches the second register value from the previous cycle of the processing unit in the third buffer. The processing unit transmits the preset sum value of the third product and the second register value to the adder to complete the addition calculation and obtain the second sum value. The preset sum value of the third product is the sum value of a preset number of third products. After the second sum value calculation is completed, the second sum value can be transmitted to the third register to update the third register, thereby outputting the multilayer perceptron data through the updated third register.

[0094] In some embodiments, to facilitate the calculation of the third product, a multiplier matrix can be set in the multiplier. During the calculation of the third product, the second acceleration module can input the feature map data row by row into the multiplier matrix and the weight data column by column into the multiplier matrix to provide a computational environment for performing the multiplication calculation. After the data input is completed, the second acceleration module performs a multiplication and accumulation calculation on the multiplier matrix through the multiplier to obtain the third product of the feature map data and the weight data. Then, the third product is added to the value of the accumulator, and the resulting third accumulated value is input into the accumulator.

[0095] After the first acceleration module calculates the attention data and the second acceleration module calculates the multilayer perceptron data, the hardware acceleration module can directly transfer the attention data and multilayer perceptron data back to the neural network model via memory. The neural network model can then directly output the classification result of the data to be processed based on the attention data and multilayer perceptron data. This application replaces the calculation of the attention module and multilayer perceptron in the neural network model with a hardware accelerator. The first acceleration module accelerates the calculation process of the attention module, and the second acceleration module accelerates the calculation process of the multilayer perceptron. By accelerating the calculation of each step, the overall neural network model is accelerated, thereby improving the processing speed of the neural network model for the data to be processed.

[0096] In some embodiments, for neural network models without simplified normalization layers (LN), the computation process of the normalization layers can be accelerated to improve the processing speed of the neural network model. To this end, the hardware accelerator may further include a third acceleration module. To reduce the power consumption of the hardware accelerator, the third acceleration module can perform normalization layer detection on the neural network model, starting operation when a normalization layer is detected, and remaining in standby mode when no normalization layer is detected, thereby reducing power consumption.

[0097] The third acceleration module includes a multiplication and summation (MAS) array and several adders and multipliers. The MAS array itself contains multiple adders and multipliers. The third acceleration module can input weight data and feature map data into the MAS array, and then use the MAS array to calculate the average value μ based on the weight data and feature map data according to the following formula. X :

[0098]

[0099] Where N is the number of channels in the input feature map.

[0100] And, calculate the variance according to the following formula. To accumulate X simultaneously 2 Calculate with X and μ X 2 , thereby eliminating μ X Pre-calculate latency overhead and improve throughput:

[0101]

[0102] The average value μ was calculated. X and variance Then, the normalized layer results can be calculated according to the following formula:

[0103]

[0104] in, γ and β are two different learnable parameters in the normalization layer, and ε is a very small constant, the purpose of which is to avoid the denominator of the formula being 0.

[0105] Some embodiments of this application also provide a Transformer-based acceleration method, which is applied to the Transformer-based acceleration system described above. The system includes a hardware accelerator and a neural network module, which are two independent hardware modules. The neural network module runs a neural network model. The hardware accelerator includes a memory, a first acceleration module, and a second acceleration module. The first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units. The method includes:

[0106] Acquire the data to be processed, which includes weight data, feature map data, and key-value pair data;

[0107] Attention data is obtained by performing an accumulation calculation on the key-value pair data through the reconfigurable unit.

[0108] The processing unit calculates multilayer perceptron data based on the weight data and the feature map data.

[0109] The neural network module outputs classification results based on the attention data and the multilayer perceptron data.

[0110] As can be seen from the above technical solutions, this application provides a Transformer-based acceleration system and method. The system includes a hardware accelerator and a neural network module. The neural network module runs a neural network model. The hardware accelerator includes a memory, a first acceleration module, and a second acceleration module. The first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units. The first acceleration module accumulates and calculates attention data by using the reconfigurable units to accumulate key-value pair data. The second acceleration module calculates multilayer perceptron data based on weight data and feature map data by using the processing units. The neural network module outputs classification results based on the attention data and the multilayer perceptron data. This application accelerates each computational process in the neural network model through the acceleration module, achieving overall acceleration of the neural network model and improving the efficiency of the neural network model in processing data.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0112] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A Transformer-based acceleration system, characterized in that, It includes a hardware accelerator and a neural network module, which are two independent hardware modules; wherein, the neural network module runs a neural network model; the hardware accelerator includes a memory, a first acceleration module and a second acceleration module, the first acceleration module includes multiple reconfigurable units, and the second acceleration module includes multiple processing units; The memory is used to store the data to be processed acquired by the neural network module, including weight data, feature map data, and key-value pair data. The first acceleration module is configured to perform an accumulation calculation on the key-value pair data through the reconfigurable unit to obtain attention data; The second acceleration module is configured to calculate multilayer perceptron data based on the weight data and the feature map data through the processing unit; The neural network module is configured to output a classification result based on the attention data and the multilayer perceptron data; The hardware accelerator further includes a third acceleration module, which includes a multiply-accelerate array, comprising multiple adders and multiple multipliers. The third acceleration module is configured as follows: Detect the normalized layer of the neural network module; If the normalization layer exists, the mean and variance are calculated using the adder and the multiplier based on the weight data and the feature map data. The normalization layer result is calculated based on the mean and the variance. The reconfigurable unit includes a first register. The reconfigurable unit alternately calculates the attention data using a first accumulation mode and a second accumulation mode, wherein the first accumulation mode is a self-accumulation mode and the second accumulation mode is a directional accumulation mode. When the reconfigurable unit is in the first accumulation mode, the first acceleration module executes the input of the query matrix, the key matrix, and the value matrix to the reconfigurable unit, configured as follows: The query matrix and the key matrix are input into the reconfigurable unit to calculate the first product of the query matrix and the key matrix; The first register caches the first register value of the reconstructible cell from the previous cycle. The preset sum of the first product is added to the first register value of the reconfigurable unit in the previous cycle to obtain the first sum value, and the first sum value is cached in the first register. The preset sum value of the first product is the sum value of a preset number of first products. The processing unit includes an adder, a multiplier, and a third register. The second acceleration module performs the calculation of multilayer perceptron data based on the weight data and the feature map data, and is configured as follows: The weight data and the feature map data are input into the multiplier to calculate the third product of the weight data and the feature map data; The second register value of the previous cycle is processed by the third register cache processing unit; The preset sum of the third product is added to the second register value of the previous cycle to obtain the second sum, and the third register is updated according to the second sum. The preset sum of the third product is the sum of a preset number of third products. The multilayer perceptron data is output through the updated third register.

2. The Transformer-based acceleration system according to claim 1, characterized in that, The memory includes an off-chip memory module, an on-chip first buffer, an on-chip second buffer, and an on-chip third buffer; The on-chip first buffer is used to store the weight data input by the off-chip storage module; The on-chip second buffer is used to store the feature map input data input by the off-chip storage module and the feature map output data output by the first acceleration module. The on-chip third buffer is used to store key-value pair data input by the off-chip storage module.

3. The Transformer-based acceleration system according to claim 1, characterized in that, The key-value pair data includes a query matrix, a key matrix, and a value matrix. The first acceleration module performs an accumulation calculation on the key-value pair data through the reconfigurable unit and is further configured to: The query matrix, the key matrix, and the value matrix are input into the reconfigurable unit; Attention data is calculated using the following formula: ； Where A is the output attention data, Q is the query matrix, K is the key matrix, V is the value matrix, T represents the transpose, and d is the feature map dimension of the query matrix, key matrix, or value matrix.

4. The Transformer-based acceleration system according to claim 3, characterized in that, The reconfigurable unit further includes a second register. When the reconfigurable unit is in a second accumulation mode, the first acceleration module executes the input of the query matrix, the key matrix, and the value matrix to the reconfigurable unit, configured as follows: The first sum and the value matrix are input into the reconfigurable unit to calculate the second product of the first sum and the value matrix; The second product is transmitted to an adjacent reconfigurable unit via the second register, so that the second product is iteratively summed by the adjacent reconfigurable unit to obtain attention data. The adjacent reconfigurable unit is adjacent to the reconfigurable unit in the transmission direction.

5. The Transformer-based acceleration system according to claim 4, characterized in that, The first acceleration module is configured to input the query matrix and the key matrix into the reconfigurable unit, and is configured as follows: The query matrix is ​​input row by row into the cell matrix of the reconfigurable cell, and the key matrix is ​​input column by column into the cell matrix of the reconfigurable cell; Perform a multiplication and accumulation calculation on the unit matrix to obtain a first accumulated value, which includes the first product.

6. The Transformer-based acceleration system according to claim 1, characterized in that, The second acceleration module is configured to input the weight data and the feature map data into the multiplier. The feature map data is input into the multiplier matrix row by row, and the weight data is input into the multiplier matrix column by column; Perform a multiply-accumulate operation on the multiplier matrix to obtain a third accumulated value, which includes the third product.

7. A Transformer-based acceleration method, characterized in that, Applied to the Transformer-based acceleration system as described in any one of claims 1-6, the method comprises: Acquire the data to be processed, which includes weight data, feature map data, and key-value pair data; Attention data is obtained by performing an accumulation calculation on the key-value pair data through the reconfigurable unit. The processing unit calculates multilayer perceptron data based on the weight data and the feature map data. The neural network module outputs classification results based on the attention data and the multilayer perceptron data.

Images