A universal and configurable Transformer hardware accelerator and its implementation method

Abstract

The invention discloses a universal configurable Transformer hardware accelerator and an implementation method thereof. All operation requirements of the entire network layer can be met through a computing engine array, a Softmax operation module and a LayerNorm operation module. The computing engine array improves the computing parallelism in the accelerator, and improves the utilization rate of data loaded into the internal cache of the accelerator while accelerating the computing speed. The storage areas of the on-chip cache module can store feature data, bias matrix data, weight data and parameter data separately, thereby improving the efficiency of subsequent data reading, reducing the delay of the memory access process and improving the operation efficiency. The parallelism of the computing engines, the number of computing engines and the operation parallelism of the nonlinear layer can all be configured through a control module, and the deployment and implementation of the model can be completed according to different computing resources. The invention can be widely applied to the technical field of neural network models.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network models, and in particular to a universal and configurable Transformer hardware accelerator and an implementation method thereof. Background Art

[0002] The successive introduction of various neural network models has undoubtedly given the internal driving force for the continuous development and progress of many application scenarios such as human-computer interaction, autonomous driving, biomedicine, and security monitoring. In recent years, the Transformer model has received widespread attention. It is a prominent self-attention-based deep learning model that can achieve fast parallelism using the self-attention mechanism, and the unification of the model itself shows great potential in multi-modal input and multi-task scenarios. As the computing tasks that neural network models need to process become more and more dense, deploying them on hardware requires a lot of memory and computing costs. Compared with the inherent defects of cloud processing such as large transmission delay and data insecurity, deploying models on edge and embedded terminals highlights the importance and necessity. Therefore, in order to ensure real-time computing, hardware accelerators are urgently needed to provide hardware-level support.

[0003] Different from the traditional neural network model architecture based on convolution operations, the Transformer model is essentially an encoder-decoder structure, where the encoder is responsible for mapping the input to the hidden layer, and the decoder maps the hidden layer to the output. The Transformer model is based on the self-attention mechanism, adopts an encoder-decoder architecture, and has a special network structure. Many existing general neural network accelerators cannot effectively accelerate the calculation of the Transformer model due to hardware architecture mismatch, low hardware resource utilization, and low scheduling efficiency. The Transformer model contains many matrix multiplications of different sizes, residual structures, nonlinear layer operations of the self-attention layer, normalization layers, and fully connected layers. During the calculation process, a large number of computing units are required to achieve parallelization and a large amount of memory access is required. The memory access delay will also occupy the overall running time of the network. The existing Transformer-related hardware accelerators have not fully proposed the optimized calculation circuits of these calculation layers, but only realized matrix multiplication and linear layers or made acceleration designs for the self-attention layer, and did not fully realize the acceleration of the entire network structure. For the application requirements of accelerating the entire Transformer network, the acceleration effect is poor. In addition, most of the designs of FPGA-based Transformer accelerators are implemented using HLS. Scalability is not considered in the design, and the storage structure and data access mode are not flexible, and cannot handle situations where the model parameters are large. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is: the present invention proposes a universal and configurable Transformer hardware accelerator and an implementation method thereof to improve the scalability of the Transformer hardware accelerator and the computational efficiency of the Transformer model.

[0005] The first technical solution adopted by the present invention is:

[0006] A general-purpose and configurable Transformer hardware accelerator, including:

[0007] A computing engine array, the computing engine array comprising a plurality of configurable computing engines, the computing engines comprising a configurable number of adder trees, accumulators, ReLU units, and a plurality of parallel multipliers, the computing engine array being used to perform linear layer operations, self-attention layer operations, fully connected layer operations, and residual connection operations in parallel;

[0008] An on-chip cache module, the on-chip cache module comprising a weight cache area, a bias cache area, a parameter cache area, and a plurality of feature cache areas, the computing engine array is connected to an off-chip storage via the on-chip cache module, and the on-chip cache module is used to implement data interaction between the computing engine array and the off-chip storage;

[0009] A nonlinear layer acceleration module, the nonlinear layer acceleration module comprising a Softmax operation module and a LayerNorm operation module, the input end of the Softmax operation module and the input end of the LayerNorm operation module are both connected to the output end of the computing engine array, the Softmax operation module and the LayerNorm operation module are both connected to the on-chip cache module, and the Softmax operation module and the LayerNorm operation module are used to perform nonlinear layer operations;

[0010] A control module, the computing engine array, the Softmax operation module and the LayerNorm operation module are all connected to the control module, and the control module is used to perform operation configuration and memory access logic scheduling control on the computing engine array, the Softmax operation module and the LayerNorm operation module.

[0011] Furthermore, the parallelism of the computing engine array is controlled by configuring the number of the computing engines and the parallel multipliers.

[0012] Furthermore, the Transformer hardware accelerator also includes a data rearrangement module, through which the computing engine array is connected to the on-chip cache module, and the data rearrangement module is used to rearrange the output vectors of the computing engine array and transmit them to each cache area of ​​the on-chip cache module.

[0013] Furthermore, the feature cache area includes a first feature cache area and a second feature cache area, the storage bit width of the first feature cache area is consistent with the number of bits of the feature map vector input to the computing engine array, and the storage bit width of the second feature cache area is consistent with the number of bits of the output vector of the computing engine array.

[0014] Furthermore, the Softmax operation module includes a maximum value search unit, a maximum value cache unit and a Softmax calculation unit. The maximum value search unit is used to find the maximum value of each row element of the output vector of the computing engine array, the maximum value cache module is used to cache the maximum value, and the Softmax calculation unit is used to obtain feature cache data from the feature cache area and perform normalized exponential operation based on the feature cache data and the maximum value.

[0015] Furthermore, the LayerNorm operation module includes a mean calculation unit, a mean cache unit, a variance calculation unit, a variance cache unit and a LayerNorm calculation unit. The mean calculation unit is used to calculate the mean of each row element of the output vector of the computing engine array, the mean cache unit is used to cache the mean, the variance calculation unit is used to calculate the variance of each row element of the output vector of the computing engine array, the variance cache module is used to cache the variance, and the LayerNorm calculation unit is used to obtain the bias matrix cache data and the parameter cache data from the bias cache area and the parameter cache area, and perform a layer normalization function operation according to the bias matrix cache data, the parameter cache data, the mean and the variance.

[0016] Furthermore, the control module is also used to receive communication information from the host, and perform operation configuration and memory access logic scheduling control on the computing engine array, the Softmax operation module and the LayerNorm operation module according to the communication information.

[0017] The second technical solution adopted by the present invention is:

[0018] A method for implementing a universal configurable Transformer hardware accelerator, which is implemented by the universal configurable Transformer hardware accelerator, comprises the following steps:

[0019] The control module is used to configure the operation and control the memory access logic scheduling of the computing engine array, the Softmax operation module, and the LayerNorm operation module;

[0020] Inputting the feature map vector and the weight vector into the computing engine array to obtain an output vector;

[0021] Rearranging the data of the output vectors and storing them in sequence in each buffer area of ​​the on-chip buffer module;

[0022] Input the output vector into the Softmax operation module and the LayerNorm operation module;

[0023] The Softmax operation module searches for the maximum value of the output vector, and calls the feature cache data in the feature cache area to perform a normalized exponential operation to obtain a first operation result;

[0024] The mean and variance of each row element of the output vector are calculated by the LayerNorm operation module, and the bias matrix cache data in the bias cache area and the parameter cache data in the parameter cache area are called to perform a layer normalization function operation to obtain a second operation result;

[0025] The first operation result and the second operation result are output.

[0026] The beneficial effects of the present invention are as follows: the present invention provides a universal configurable Transformer hardware accelerator and an implementation method thereof, wherein linear layer operations, self-attention layer operations, fully connected layer operations and residual connection operations can be realized through a computing engine array, and nonlinear layer operations can be realized through a Softmax operation module and a LayerNorm operation module, thereby meeting all the operation requirements of the entire network layer; the computing engine array improves the computing parallelism in the accelerator, and while accelerating the computing speed, the utilization rate of data loaded into the internal cache of the accelerator is improved; through each storage area of ​​the on-chip cache module, feature data, bias matrix data, weight data and parameter data can be stored separately, thereby improving the efficiency of subsequent data reading, reducing the delay of the memory access process, and improving the operation efficiency; the parallelism of the computing engine, the number of computing engines, and the operation parallelism of the nonlinear layer can all be configured through a control module, and the deployment and implementation of the Transformer model can be completed according to different computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the structure of a universal and configurable Transformer hardware accelerator provided in an embodiment of the present invention;

[0028] Figure 2A schematic diagram of the structure of a computing engine provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of the Softmax operation module provided in an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a LayerNorm operation module provided in an embodiment of the present invention;

[0031] Figure 5 A flowchart of the steps of a method for implementing a universal and configurable Transformer hardware accelerator provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0033] In the description of the present invention, the meaning of "a plurality" is more than two. If there is a description of "a first" or "a second", it is only used to distinguish the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this article have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments, not for limiting the present invention.

[0034] Reference Figure 1 , an embodiment of the present invention provides a universal and configurable Transformer hardware accelerator, comprising:

[0035] A computing engine array, the computing engine array includes multiple configurable computing engines, the computing engines include configurable levels of addition trees, accumulators, ReLU units and multiple parallel multipliers, and the computing engine array is used to perform linear layer operations, self-attention layer operations, fully connected layer operations and residual connection operations in parallel;

[0036] An on-chip cache module, which includes a weight cache area, a bias cache area, a parameter cache area, and a plurality of feature cache areas. The computing engine array is connected to the off-chip storage through the on-chip cache module. The on-chip cache module is used to realize data interaction between the computing engine array and the off-chip storage;

[0037] A nonlinear layer acceleration module, the nonlinear layer acceleration module includes a Softmax operation module and a LayerNorm operation module, the input end of the Softmax operation module and the input end of the LayerNorm operation module are both connected to the output end of the computing engine array, the Softmax operation module and the LayerNorm operation module are both connected to the on-chip cache module, and the Softmax operation module and the LayerNorm operation module are used to perform nonlinear layer operations;

[0038] The control module, the computing engine array, the Softmax computing module and the LayerNorm computing module are all connected to the control module, and the control module is used to perform computing configuration and memory access logic scheduling control on the computing engine array, the Softmax computing module and the LayerNorm computing module.

[0039] Specifically, the Transformer hardware accelerator of the embodiment of the present invention includes:

[0040] 1) General-purpose vector multiplication-based computing engine array: The computing engine array consists of multiple configurable computing engines, and the number of computing engines can also be configured; each computing engine contains multiple parallel multipliers and adders, accumulators, and ReLU units, which are combined into a pipeline structure to achieve parallel computing of large amounts of matrix data. Matrix operations of the self-attention layer, linear layer, and fully connected layer operations can all be accelerated in parallel on this computing array.

[0041] 2) On-chip cache module: The on-chip cache module is divided into multiple feature cache areas, weight buffer areas, bias and parameter cache areas, etc., which are used to realize the interaction between off-chip data storage and computing engine arrays, cache the data that will be involved in the calculation and the data that has been calculated to the on-chip cache, reduce the memory access bandwidth, realize internal high-bandwidth access, shorten the running time, and reduce the running power consumption.

[0042] 3) Nonlinear layer acceleration module: The Softmax operation module is used for the softmax nonlinear operation in the network layer, and the LayerNorm operation module is used to implement LayerNorm-related nonlinear operations and perform normalization. The time used for some calculations of the two can overlap with the running time of the computing engine array, and the other part of the calculation can also be implemented in parallel accelerated operation.

[0043] 4) Control module: The control module is responsible for receiving communication information from the host, including control signals and configuration information. In addition, the control module also includes scheduling control of different computing types and memory access logic in the network layer, maximizing the hiding of memory access time, reducing the computing delay of the entire network layer, and improving computing efficiency.

[0044] As a further optional implementation, the parallelism of the computing engine array is controlled by configuring the number of computing engines and parallel multipliers.

[0045] The embodiment of the present invention can map different operators of the Transformer model to corresponding hardware modules to achieve efficient operations and shorten the overall running time. The parallelism configuration is as follows: a computing engine contains N multipliers, there are M computing engines in total, and the representation bit width of a data is W.

[0046] The computing engine array is the core computing part of the hardware accelerator, occupying most of the computing resources. The design of the computing engine matches the vector operation. Each computing engine contains multiple parallel multipliers and a configurable adder, accumulator, and Relu unit. According to the function of the internal sub-controller, matrix multiplication of different sizes, residual structure addition, bias addition, and transposition of the output matrix in the Transformer algorithm can be realized. The transposition here is related to the data arrangement in the cache, which is convenient for data access and calculation. The degree of parallelism can be obtained by configuring the number of multipliers and computing engines. The input of a computing unit is a set of NW-bit input feature map vectors and a set of NW-bit weight vectors. After the operation of the computing engine array, the output is a MW-bit output vector, which is stored in the data cache unit in sequence through data rearrangement.

[0047] The computing engine can be configured to meet different matrix operation types in various Transformer structures under the action of the controller, such as:

[0048] 1) The first layer of the feed-forward layer in the Transformer model contains Relu operations. You can enable Relu to support the calculation of this type of layer.

[0049] 2) In some operations, in addition to accumulating multiple calculation results, it is also necessary to add bias data, residual data from the previous layer, or even both. These functions can be implemented through multi-way selection in the accumulation stage, classifying various types of calculations in the Transformer model to adapt to different configurations of the calculation array.

[0050] The architecture of the computing engine of the embodiment of the present invention is as follows Figure 2 As shown, different types of operations can be switched through the control module, which has strong flexibility and high utilization rate of the operation unit.

[0051] Reference Figure 1 As an optional implementation, the Transformer hardware accelerator further includes a data rearrangement module, through which the computing engine array is connected to the on-chip cache module, and the data rearrangement module is used to rearrange the output vectors of the computing engine array and transmit them to each cache area of ​​the on-chip cache module.

[0052] Specifically, in addition to considering the core computing units of the computing engine array itself, the embodiments of the present invention also consider the problem of input and output data arrangement of the computing engine. Since the matrix output in the calculation of the self-attention layer may also be the weight output of the next calculation, in order to maximize the use of the computing unit, improve the hardware reuse rate, and reduce the idle time of the hardware, the output data needs to be dispersed and rearranged to different caches in preparation for the next access calculation. In addition, the processing of matrix transposition is also designed in different matrix calculations. If all are handed over to a single hardware transposition module, it will consume more time and resources. Therefore, the embodiment of the present invention adds a data rearrangement module between the computing engine array and the on-chip cache module, and optimizes the transposition feature by performing special rearrangement through the data rearrangement module. The price that needs to be sacrificed is the cutting and combination of the cache unit. It is necessary to use a smaller bit width cache to form a large cache unit to meet the selective writing requirements of certain bit widths in the large bit width cache.

[0053] Reference Figure 1 , further as an optional implementation, the feature cache area includes a first feature cache area and a second feature cache area, the storage bit width of the first feature cache area is consistent with the number of bits of the feature map vector input to the computing engine array, and the storage bit width of the second feature cache area is consistent with the number of bits of the output vector of the computing engine array.

[0054] Specifically, the Transformer model itself has a variety of calculation types, and in the general field of natural language processing, the storage required for feature maps is relatively small. In order to improve the computing efficiency of the hardware accelerator and reduce the interaction with off-chip storage, and because the Transformer structure involves multiple output matrices that need to be calculated again, a simple ping-pong cache design still needs to temporarily store the data back to the off-chip storage, which cannot meet the demand. The embodiment of the present invention uses multiple data caches to temporarily store intermediate data to achieve a high bandwidth on-chip data transmission. In order to better match the computing array and reduce data re-arrangement, the size and bit width of the data cache are related to the configuration of the computing engine array.

[0055] There are two types of feature map caches: the first type of feature cache has a storage bit width of NWbits for each address. The data of a row are arranged from left to right first, and then the next row of data is arranged from left to right, and so on. The second type of feature cache has a feature cache data arrangement order that remains unchanged, but the storage bit width changes, which is designed to be MW bits and is mainly used for the addition operation of the residual connection part.

[0056] The computing array contains multiple computing engines to achieve higher parallelism. In order to obtain multiple groups of weight values ​​and send them to the computing array in parallel at the same time, it is necessary to increase the on-chip cache bandwidth. The embodiment of the present invention adopts a multi-group weight cache method, each group of sub-weight cache accesses part of the weight data, and the total weight number of multiple groups of sub-weight caches is equal to the number required for the entire weight cache. The first type of weight cache is a large block of storage space, and the other type is multiple groups of small caches spliced ​​to form a large cache space. The reason is that the values ​​after some matrix calculations are completed need to be used as weights for calculation again next time. In order to reduce the operation of data transposition and rearrangement and reduce the delay of network layer operations, special data rearrangement operations are performed in the output stage of the computing array, and the control signals and address generation modules for data writing need to be additionally designed. If the above requirements need to be met, it is necessary to selectively write the W bits of each storage block, then consider splicing the caches of N W bits to form a complete cache space. When the next matrix calculation needs to read data from the cache, NWbits of data can also be obtained at the same time.

[0057] The parameter buffer and bias buffer mainly store parameter data and bias data, occupying a small amount of memory. They are also divided into two categories according to the bit width of the storage unit, one with a bit width of NW bits and the other with a bit width of MW bits. The specific selection criteria correspond to the network structure layer.

[0058] Reference Figure 3 As an optional implementation, the Softmax operation module includes a maximum value search unit, a maximum value cache unit and a Softmax calculation unit. The maximum value search unit is used to find the maximum value of each row element of the output vector of the calculation engine array. The maximum value cache module is used to cache the maximum value. The Softmax calculation unit is used to obtain feature cache data from the feature cache area and perform normalized exponential operation based on the feature cache data and the maximum value.

[0059] Specifically, the Softmax operation module is located in the self-attention layer. The specific method of the nonlinear function is to transform the result into an exponential function to ensure the non-negativity of the probability, and then divide the transformed result by the sum of all transformed results, which can be understood as the percentage occupied, thereby obtaining an approximate probability. In order to avoid overflow in the fixed-point representation of large inputs, the method of subtracting the maximum value of the input vector from the dot product value being processed is adopted. The Softmax operation is on the critical path of the calculation. The embodiment of the present invention mainly adopts two optimization techniques: one is to hide the calculation acquisition time of the maximum value into the calculation time of the previous matrix, and the other is that the Softmax calculation module adopts a pipeline method to increase the throughput, and multiple parallel computing units are used to further reduce the operation delay.

[0060] like Figure 3The figure shows a schematic diagram of the structure of the Softmax operation module provided by an embodiment of the present invention. It can be understood that the computing engine array obtains the value of the output matrix in sequence, and sends multiple output values ​​to the maximum value search unit of the Softmax operation module at the same time, obtains the maximum value of each row of the feature matrix and temporarily stores it in the maximum value cache unit; after the computing engine array completes the matrix operation before the Softmax operation, it starts to calculate the Softmax value of each data in the matrix, and the maximum value of the row is obtained from the maximum value cache unit. The value at an address in the feature cache area can be obtained in each clock cycle. The value includes multiple data in the matrix value, which can be sent to the Softmax operation module in parallel. After several cycles, multiple Softmax values ​​can be obtained in parallel; a batch of new calculation values ​​can be sent in each clock cycle, which flow into the Softmax operation module in sequence to form a pipeline, thereby improving the throughput of the module.

[0061] Reference Figure 4 As an optional implementation, the LayerNorm operation module includes a mean calculation unit, a mean cache unit, a variance calculation unit, a variance cache unit and a LayerNorm calculation unit. The mean calculation unit is used to calculate the mean of each row element of the output vector of the calculation engine array, the mean cache unit is used to cache the mean, the variance calculation unit is used to calculate the variance of each row element of the output vector of the calculation engine array, the variance cache module is used to cache the variance, and the LayerNorm calculation unit is used to obtain the bias matrix cache data and the parameter cache data from the bias cache area and the parameter cache area, and perform a layer normalization function operation according to the bias matrix cache data, the parameter cache data, the mean and the variance.

[0062] Specifically, in the Transformer model, a LayerNorm operation module is added to ensure the stability of data feature distribution. The LayerNorm operation module needs to calculate the mean and variance on each sample. Its structure is as follows: Figure 4 As shown, it can be understood that the LayerNorm operation module directly obtains data from the computing engine array and then sends it to the mean calculation unit and the variance calculation unit. Therefore, when calculating the mean and variance, the calculation time can be hidden in the previous matrix operation, and the delay on the critical path of the data flow will not be increased; in addition, the variance calculation and the mean calculation can be executed in parallel. Since the variance calculation also includes a multiplication at the previous level of the addition tree, the mean can be obtained one cycle in advance when the variance value is finally obtained, which can just meet the timing requirements of data acquisition in the variance calculation formula; the parallelism of the mean and variance calculations is consistent with the number of computing engines, which is M.

[0063] After obtaining the variance and mean, it is necessary to perform LayerNorm operation on each data point. The pipeline data flow mode is also adopted, and multiple multipliers and adders operate in parallel. The degree of parallelism is consistent with the data parallelism in the cache, which is N. When calculating the LayerNorm output, it is necessary to read the corresponding data from the mean cache, variance cache, and the bias cache and parameter cache in the on-chip cache module.

[0064] As a further optional implementation, the control module is also used to receive communication information from the host, and perform operation configuration and memory access logic scheduling control on the computing engine array, the Softmax operation module, and the LayerNorm operation module according to the communication information.

[0065] It should be recognized that the hardware architecture of the embodiment of the present invention can be used to accelerate the operation of any model containing a Transformer structure or a neural network model containing a common operator structure; the multipliers in a computing engine can be increased to improve the parallelism, thereby obtaining better acceleration performance; the embodiment of the present invention is not only suitable for deployment on FPGA devices, but also can be used to ASIC the hardware accelerator.

[0066] The above is a description of the structure and working principle of the digital converter in the embodiment of the present invention. It can be seen that the embodiment of the present invention has the following advantages:

[0067] 1) The Transformer network structure has various layers, including linear layer operations, self-attention layer operations, fully connected layer operations, residual connections, nonlinear operations and many other types of operations. Most of the first four operations can be implemented by the computing engine array of the embodiment of the present invention, and the calculation of the nonlinear layer can be deployed to the Softmax operation module and LayerNorm operation module of the embodiment of the present invention. Therefore, the architecture of the embodiment of the present invention meets all the computing requirements of the entire network layer;

[0068] 2) The vector computing engine array improves the computational parallelism in the accelerator, which speeds up the computation and improves the utilization of the data loaded into the accelerator's internal cache;

[0069] 3) The accelerator contains a data rearrangement module that can adapt to the changes in the order of data arrangement to perform classified partition caching, increase the integrity of acquired data, improve the efficiency of data acquisition, and reduce the delay of the memory access process;

[0070] 4) The parallelism of the internal units of the computing engine, the number of computing engines, and the parallelism of operations in the nonlinear layer module can all be configured, and the model can be deployed and implemented according to different computing resources.

[0071] Reference Figure 5The embodiment of the present invention provides a method for implementing a universal configurable Transformer hardware accelerator, which is implemented by the universal configurable Transformer hardware accelerator, and includes the following steps:

[0072] S101, performing operation configuration and memory access logic scheduling control on a computing engine array, a Softmax operation module, and a LayerNorm operation module through a control module;

[0073] S102, inputting the feature map vector and the weight vector into the computing engine array to obtain an output vector;

[0074] S103, rearrange the data of the output vector and store them in each buffer area of ​​the on-chip buffer module in sequence;

[0075] S104, input the output vector to the Softmax operation module and the LayerNorm operation module;

[0076] S105, searching for the maximum value of the output vector through the Softmax operation module, and calling the feature cache data in the feature cache area to perform a normalized exponential operation to obtain a first operation result;

[0077] S106, calculating the mean and variance of each row element of the output vector through the LayerNorm operation module, and calling the bias matrix cache data in the bias cache area and the parameter cache data in the parameter cache area to perform layer normalization function operation to obtain a second operation result;

[0078] S107, output the first operation result and the second operation result.

[0079] Specifically, the embodiment of the present invention can realize linear layer operations, self-attention layer operations, fully connected layer operations and residual connection operations through the computing engine array, and can realize nonlinear layer operations through the Softmax operation module and the LayerNorm operation module, thereby meeting all the operation requirements of the entire network layer; the computing engine array improves the computing parallelism in the accelerator, and improves the utilization rate of data loaded into the internal cache of the accelerator while speeding up the calculation speed; the various storage areas of the on-chip cache module can store feature data, bias matrix data, weight data and parameter data separately, improve the efficiency of subsequent data reading, reduce the delay of the memory access process, and improve the operation efficiency; the parallelism of the computing engine, the number of computing engines, and the operation parallelism of the nonlinear layer can all be configured through the control module, and the deployment and implementation of the Transformer model can be completed according to different computing resources.

[0080] It can be understood that the contents of the above system embodiments are all applicable to the present method embodiments, the functions specifically implemented by the present method embodiments are the same as those of the above system embodiments, and the beneficial effects achieved are also the same as those achieved by the above system embodiments.

[0081] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The above methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner, according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0082] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer programs described above include a plurality of instructions that may be executed by one or more processors.

[0083] Further, the above method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself.

[0084] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0085] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A universal and configurable Transformer hardware accelerator, characterized in that: include: A computing engine array, the computing engine array comprising a plurality of configurable computing engines, the computing engines comprising a configurable number of adder trees, accumulators, ReLU units, and a plurality of parallel multipliers, the computing engine array being used to perform linear layer operations, self-attention layer operations, fully connected layer operations, and residual connection operations in parallel; An on-chip cache module, the on-chip cache module comprising a weight cache area, a bias cache area, a parameter cache area, and a plurality of feature cache areas, the computing engine array is connected to an off-chip storage via the on-chip cache module, and the on-chip cache module is used to implement data interaction between the computing engine array and the off-chip storage; A nonlinear layer acceleration module, the nonlinear layer acceleration module comprising a Softmax operation module and a LayerNorm operation module, the input end of the Softmax operation module and the input end of the LayerNorm operation module are both connected to the output end of the computing engine array, the Softmax operation module and the LayerNorm operation module are both connected to the on-chip cache module, and the Softmax operation module and the LayerNorm operation module are used to perform nonlinear layer operations; A control module, the computing engine array, the Softmax operation module and the LayerNorm operation module are all connected to the control module, and the control module is used to perform operation configuration and memory access logic scheduling control on the computing engine array, the Softmax operation module and the LayerNorm operation module.

2. A universal configurable Transformer hardware accelerator according to claim 1, characterized in that: The parallelism of the computing engine array is controlled by configuring the number of the computing engines and the parallel multipliers.

3. A universal configurable Transformer hardware accelerator according to claim 1, characterized in that: The Transformer hardware accelerator also includes a data rearrangement module, through which the computing engine array is connected to the on-chip cache module, and the data rearrangement module is used to rearrange the output vectors of the computing engine array and transmit them to each cache area of ​​the on-chip cache module.

4. The universal configurable Transformer hardware accelerator according to claim 1, characterized in that: The feature cache area includes a first feature cache area and a second feature cache area. The storage bit width of the first feature cache area is consistent with the number of bits of the feature map vector input to the computing engine array, and the storage bit width of the second feature cache area is consistent with the number of bits of the output vector of the computing engine array.

5. The universal configurable Transformer hardware accelerator according to claim 1, characterized in that: The Softmax operation module includes a maximum value search unit, a maximum value cache unit and a Softmax calculation unit. The maximum value search unit is used to find the maximum value of each row element of the output vector of the calculation engine array, the maximum value cache unit is used to cache the maximum value, and the Softmax calculation unit is used to obtain feature cache data from the feature cache area and perform normalized exponential operation based on the feature cache data and the maximum value.

6. A universal configurable Transformer hardware accelerator according to claim 1, characterized in that: The LayerNorm operation module includes a mean calculation unit, a mean cache unit, a variance calculation unit, a variance cache unit and a LayerNorm calculation unit. The mean calculation unit is used to calculate the mean of each row element of the output vector of the computing engine array, the mean cache unit is used to cache the mean, the variance calculation unit is used to calculate the variance of each row element of the output vector of the computing engine array, the variance cache module is used to cache the variance, and the LayerNorm calculation unit is used to obtain bias matrix cache data and parameter cache data from the bias cache area and the parameter cache area, and perform layer normalization function operation according to the bias matrix cache data, the parameter cache data, the mean and the variance.

7. A universal configurable Transformer hardware accelerator according to any one of claims 1 to 6, characterized in that: The control module is also used to receive communication information from the host, and perform operation configuration and memory access logic scheduling control on the computing engine array, the Softmax operation module and the LayerNorm operation module according to the communication information.

8. A method for implementing a universal configurable Transformer hardware accelerator, for implementing the universal configurable Transformer hardware accelerator according to any one of claims 1 to 7, characterized in that: The following steps are involved: The control module is used to configure the operation and control the memory access logic scheduling of the computing engine array, the Softmax operation module, and the LayerNorm operation module; Inputting the feature map vector and the weight vector into the computing engine array to obtain an output vector; Rearranging the data of the output vectors and storing them in sequence in each buffer area of ​​the on-chip buffer module; Input the output vector into the Softmax operation module and the LayerNorm operation module; The Softmax operation module searches for the maximum value of the output vector, and calls the feature cache data in the feature cache area to perform a normalized exponential operation to obtain a first operation result; The mean and variance of each row element of the output vector are calculated by the LayerNorm operation module, and the bias matrix cache data in the bias cache area and the parameter cache data in the parameter cache area are called to perform a layer normalization function operation to obtain a second operation result; The first operation result and the second operation result are output.

Images