Nonlinear layer acceleration method and device of Transform model, accelerator and storage medium

Abstract

The invention provides a nonlinear layer acceleration method and device of a Transform model, an accelerator and a storage medium. The method comprises the following steps: acquiring an input vector of a Softmax network layer; performing grouping sparse processing on the input vector to obtain an output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; and performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain an output vector of the LayerNorm network layer. According to the method, the sparse processing is performed on the input vector of the Softmax network layer, and the dual-path sparse processing is performed on the input vector of the LayerNorm network layer, so that the calculation efficiency of the Transform model for processing the long sequence data is remarkably improved, and the corresponding memory overhead is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a nonlinear layer acceleration method, device, accelerator and storage medium of a Transformer model. Background Art

[0002] The Transformer model has demonstrated powerful capabilities in natural language processing (NLP), traditional computer vision (CV) and other fields. However, the cost of its high performance is much higher than the computational complexity and storage complexity of traditional AI algorithms. For example, compared with the ResNet-152 model, ViT (Vision Transformer) has increased the computational complexity by 152 times, but when inferring on a GPU, it may consume more than 10GB of storage when the sequence length is greater than 8192.

[0003] Among them, Softmax and LayerNorm are the core nonlinear operators in the Transformer algorithm. When the input of the Transformer model is a particularly long sequence, processing these two types of nonlinear operators faces the problems of low processing efficiency and high memory overhead. This is because the attention mechanism of the standard Transformer model has a quadratic complexity, which leads to slow processing of long sequences and huge memory consumption.

[0004] In this context, the prior art has proposed some hardware-software co-design methods to improve the energy efficiency of Softmax and LayerNorm, but the potential sparsity of Softmax and LayerNorm has not been fully explored.

[0005] Therefore, how to solve the problem of low computational efficiency and high storage overhead of traditional Transformer models when processing long sequence data is an important issue that needs to be urgently addressed in the field of natural language processing. Summary of the invention

[0006] The present invention provides a nonlinear layer acceleration method, device, accelerator and storage medium of a Transformer model, which are used to solve the defects of low computational efficiency and high storage overhead of traditional Transformer models in the prior art when processing long sequence data, significantly improve the computational efficiency of the Transformer model in processing long sequence data, and reduce memory overhead.

[0007] On the one hand, the present invention provides a nonlinear layer acceleration method of a Transformer model, wherein the Transformer model includes an encoder and a decoder, wherein the encoder and the decoder both include a Softmax network layer and a LayerNorm network layer, and the method includes: obtaining an input vector of the Softmax network layer; performing grouping sparse processing on the input vector to obtain an output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain an output vector of the LayerNorm network layer.

[0008] Furthermore, the grouping and sparse processing of the input vector to obtain the output vector of the Softmax network layer includes: grouping the input vector to obtain a plurality of group vectors; converting the original exponent in the Softmax operation into a set exponent, and independently calculating the plurality of group vectors to obtain the output vector of the Softmax network layer; wherein the set exponent is smaller than the original exponent.

[0009] Furthermore, the multiple group vectors are calculated independently, specifically including: taking a set index for each data in the group vector to obtain a group calculation result; based on a pre-stored global maximum value, shift compensating the group calculation result to obtain a target group calculation result; normalizing the target group calculation result to obtain an output vector of the Softmax network layer.

[0010] Furthermore, the step of obtaining the global maximum value includes: determining a group maximum value in the group calculation result; when the group maximum value is greater than a pre-stored global maximum value, determining the group maximum value as the global maximum value; when the group maximum value is less than a pre-stored global maximum value, determining the pre-stored global maximum value as the global maximum value.

[0011] Furthermore, the obtaining of the group calculation result then includes: determining a difference between the group calculation result and a set threshold; when the difference is lower than the set difference, the group calculation result no longer participates in subsequent calculation processes.

[0012] Furthermore, the dual-path sparse processing is performed on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer, including: identifying important dimensions in the input vector of the LayerNorm network layer, using full-precision calculation for the important dimensions, and using half-precision calculation for other dimensions other than the important dimensions to obtain the output vector of the LayerNorm network layer.

[0013] Furthermore, the identifying of important dimensions in the input vector of the LayerNorm network layer includes: calculating variance values ​​corresponding to different dimensions in the input vector of the LayerNorm network layer; when the variance value is higher than a set variance value, determining the dimension corresponding to the variance value as an important dimension; when the variance value is lower than the set variance value, determining the dimension corresponding to the variance value as a non-important dimension, and the non-important dimension is a dimension other than the important dimension.

[0014] In a second aspect, the present invention also provides a nonlinear layer acceleration device for a Transformer model, wherein the Transformer model includes an encoder and a decoder, wherein the encoder and the decoder both include a Softmax network layer and a LayerNorm network layer, and the device includes: an input vector acquisition module, used to obtain the input vector of the Softmax network layer; a Softmax network layer acceleration processing module, used to perform grouping sparse processing on the input vector to obtain an output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; and a LayerNorm network layer acceleration processing module, used to perform dual-path sparse processing on the input vector of the LayerNorm network layer to obtain an output vector of the LayerNorm network layer.

[0015] In a third aspect, the present invention also provides a nonlinear layer accelerator for a Transformer model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the nonlinear layer acceleration method for the Transformer model as described in any one of the above is implemented.

[0016] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a nonlinear layer acceleration method for a Transformer model as described in any one of the above.

[0017] The nonlinear layer acceleration method of the Transformer model provided by the present invention obtains the input vector of the Softmax network layer and performs group sparse processing on the input vector to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer, and then the input vector of the LayerNorm network layer is subjected to dual-path sparse processing to obtain the output vector of the LayerNorm network layer. The method performs sparse processing on the input vector of the Softmax network layer and performs dual-path sparse processing on the input vector of the LayerNorm network layer. This process not only significantly improves the computational efficiency of the Transformer model in processing long sequence data, but also reduces the corresponding memory overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of a nonlinear layer acceleration method of a Transformer model provided in an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the processing flow of the Softmax network layer provided by an embodiment of the present invention.

[0021] Figure 3 This is one of the schematic diagrams of skipping non-critical calculations of the Softmax network layer provided by an embodiment of the present invention.

[0022] Figure 4 This is the second schematic diagram of skipping non-critical calculations of the Softmax network layer provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the overall process of the Softmax network layer provided by an embodiment of the present invention.

[0024] Figure 6 FIG. 1 is a schematic diagram of the processing flow of the LayerNorm network layer provided in an embodiment of the present invention.

[0025] Figure 7 It is a schematic diagram of the overall process of the LayerNorm network layer provided by an embodiment of the present invention.

[0026] Figure 8It is a structural schematic diagram of a nonlinear layer acceleration device of a Transformer model provided in an embodiment of the present invention.

[0027] Fig. 9 It is a structural schematic diagram of a nonlinear layer accelerator of a Transformer model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that the prior art has proposed some software-hardware co-design methods to improve the energy efficiency of Softmax and LayerNorm, but the potential sparsity of Softmax and LayerNorm has not been fully explored.

[0030] In summary, the existing Transformer model still has the following defects.

[0031] On the one hand, the processing efficiency of long sequence input data is low. Specifically, the existing Transformer model has very low computational efficiency when processing long sequence data, especially the two main nonlinear layers of Softmax and LayerNorm contained in the Transformer model.

[0032] On the other hand, high storage overhead. The demand for on-chip memory in existing Transformer model accelerators increases dramatically as the length of the input sequence increases, making it difficult to process ultra-long sequences on-chip.

[0033] In view of this, the present invention proposes a nonlinear layer acceleration method for the Transformer model. Specifically, Figure 1 A schematic flow chart of a nonlinear layer acceleration method for a Transformer model provided in an embodiment of the present invention is shown.

[0034] like Figure 1 As shown, the method includes steps S110-S130, and these steps will be described in detail below.

[0035] S110, obtaining an input vector of the Softmax network layer.

[0036] It is easy to understand that the Transformer model is an encoder-decoder structure. The encoder module includes a multi-head attention mechanism, a LayerNorm network layer, and a feedforward network layer. The multi-head attention mechanism includes a Softmax network layer. Similarly, the decoder module also includes a multi-head attention mechanism, a LayerNorm network layer, and a feedforward network layer, but the number of network layers and the specific calculation method are different.

[0037] In summary, the Transformer model includes an encoder and a decoder, and both the encoder and the decoder include a Softmax network layer and a LayerNorm network layer. It can be seen that the Softmax network layer and the LayerNorm network layer are two crucial nonlinear layer operators in the Transformer model, and the embodiments of the present invention are mainly aimed at improving the processing of the two.

[0038] It should be noted that the model structure of the Transformer model is already a relatively mature content in the existing technology, and will not be elaborated on in detail here.

[0039] In this step, the input vector of the Softmax network layer is obtained. Specifically, it is obtained from the upper network layer connected to the Softmax network layer. The layer output vector of the upper network layer connected to the Softmax network layer is the input vector of the Softmax network layer in this step.

[0040] It should also be noted that, in this embodiment, the processing object of the Transformer model can be an image, or it can be text or other types of data, which can be adjusted according to actual conditions and is not specifically limited here.

[0041] S120, performing group sparse processing on the input vector to obtain an output vector of a Softmax network layer; the output vector of the Softmax network layer is an input vector of a LayerNorm network layer.

[0042] It can be understood that, based on obtaining the input vector of the Softmax network layer in step S110, further, step S120 performs grouping and sparse processing on the input vector of the Softmax network layer to obtain the output vector of the Softmax network layer.

[0043] Specifically, the input vectors of the Softmax network layer can be grouped, and when processing the grouped vectors, the e index in the Softmax operation is converted to the set index, and then the subsequent Softmax operation is performed.

[0044] The purpose of grouping the input vectors of the Softmax network layer here is to simplify the calculation process and allow parallel processing.

[0045] At the same time, in order to ensure that the calculation results after grouping the input vector are still highly accurate, after performing the Softmax operation on the grouped vector, a shift operation is performed to compensate for the error introduced by the Softmax network layer due to grouping.

[0046] It is also worth mentioning that, when performing the Softmax operation on the grouped vectors, considering the data amplification characteristics of the Softmax operation, this embodiment chooses to skip the subsequent Softmax calculation steps of some grouped vectors to significantly reduce the overall calculation burden while having little effect on the accuracy of the calculation results.

[0047] It should be noted that in this step, the output vector of the Softmax network layer is limited to the input vector of the LayerNorm network layer. This is assuming that the Softmax network layer and the LayerNorm network layer are interconnected front and back network layers. In addition, the nonlinear layer acceleration method of the Transformer model provided in this embodiment can also be applied to the Softmax network layer and the LayerNorm network layer respectively. When they are applied separately (there is no connection relationship between the Softmax network layer and the LayerNorm network layer), the input vectors of the Softmax network layer and the LayerNorm network layer are both the output vectors of the previous network layer connected to them.

[0048] S130, performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain an output vector of the LayerNorm network layer.

[0049] It can be understood that, based on the grouping and sparse processing of the input vector of the Softmax network layer in step S120 to obtain the output vector of the Softmax network layer, further, step S130 performs dual-path sparse processing on the output vector of the Softmax network layer (that is, the input vector of the LayerNorm network layer) to obtain the output vector of the LayerNorm network layer.

[0050] Specifically, dual-path sparse processing means that for the input vector of the LayerNorm network layer, it is first necessary to identify the important dimensions and unimportant dimensions in the input vector, and then use different algorithms to process the important dimensions and unimportant dimensions respectively. For example, full-precision calculation is used for the data of important dimensions, and low-precision / half-precision calculation is used for the data of unimportant dimensions.

[0051] Among them, important dimensions have a greater impact on the accuracy of the final calculation results, while unimportant dimensions have little impact on the accuracy of the final calculation results, and can even be ignored.

[0052] Important dimensions and unimportant dimensions can be determined based on the variance value and / or activation times of the vector dimensions.

[0053] In the above, steps S110 - S130 are the nonlinear layer acceleration process of the Transformer model.

[0054] It should be noted that the nonlinear layer acceleration method of the Transformer model provided by the present invention is mainly aimed at the nonlinear layers in the Transformer model, namely the Softmax network layer and the LayerNorm network layer, and the connection relationship between other network layers and the Softmax network layer and the LayerNorm network layer is not affected.

[0055] In this embodiment, the input vector of the Softmax network layer is obtained, and the input vector is subjected to group sparse processing to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer, and then the input vector of the LayerNorm network layer is subjected to dual-path sparse processing to obtain the output vector of the LayerNorm network layer. This method performs sparse processing on the input vector of the Softmax network layer and dual-path sparse processing on the input vector of the LayerNorm network layer. This process not only significantly improves the computational efficiency of the Transformer model in processing long sequence data, but also reduces the corresponding memory overhead.

[0056] On the basis of the above embodiments, the data processing process of the Softmax network layer will be described in detail below.

[0057] The input vector is subjected to group sparse processing to obtain an output vector of the Softmax network layer, including: performing vector grouping on the input vector to obtain a plurality of group vectors; converting an original index in the Softmax operation into a set index, and performing independent calculations on the plurality of group vectors to obtain an output vector of the Softmax network layer; wherein the set index is smaller than the original index.

[0058] Figure 2 A schematic diagram of the processing flow of the Softmax network layer provided by an embodiment of the present invention is shown.

[0059] like Figure 2As shown, first, the input vector is divided into multiple groups, so that the multiple group vectors obtained by grouping are calculated independently, thereby simplifying the calculation process and allowing parallel processing.

[0060] The grouping rules for the input vectors can be set according to actual conditions, and the size of the group vectors obtained by grouping can be configured according to the requirements of the Transformer model, which are not specifically limited here.

[0061] After grouping, independent calculations are performed for each group of vectors. During the calculation process, the original exponent in the Softmax operation is converted to a set exponent. This conversion makes the algorithm more suitable for hardware implementation because the exponential operation of the set exponent is simpler and more efficient in hardware than the exponential operation of the original exponent.

[0062] In a specific embodiment, the original exponent is e and the set exponent is 2. The original exponent e is converted to the set exponent 2 to take advantage of the efficiency of exponential operations of 2 in digital logic, which is simpler on most hardware than exponential operations of the natural logarithm e.

[0063] In the conversion process from the original exponent e to the set exponent 2, the loss of precision needs to be considered, and the conversion process usually involves a lookup table (LUT) or an approximation algorithm to be efficiently implemented in hardware.

[0064] Then, each group of vectors is calculated independently. Specifically, a set index is taken for each data in the group vector to obtain the group calculation result; based on the pre-stored global maximum value, the group calculation result is shift-compensated to obtain the target group calculation result; the target group calculation result is normalized to obtain the output vector of the Softmax network layer.

[0065] That is to say, when calculating the group vector, the set index value is first obtained for each data in the group vector, and the corresponding group calculation result can be obtained. The group calculation result includes the calculation results of the set index values ​​corresponding to the data in the multiple groups.

[0066] Then, the group maximum value in the group calculation result is determined; when the group maximum value is greater than the pre-stored global maximum value, the group maximum value is determined as the global maximum value; when the group maximum value is less than the pre-stored global maximum value, the pre-stored global maximum value is determined as the global maximum value.

[0067] Next, based on the determined global maximum value, the group calculation results are shift-compensated to ensure that each group calculation result remains highly accurate under simplified processing.

[0068] It is also worth mentioning that after obtaining the group calculation results, the difference between the group calculation results and the set threshold can be determined; when the difference is lower than the set difference, the group calculation results will no longer participate in the subsequent calculation process. This step is mainly based on the data amplification characteristics of the Softmax operation. Choosing to skip some non-critical intermediate or subsequent Softmax calculation steps can significantly reduce the overall calculation burden, while having little effect on the accuracy of the calculation results.

[0069] For example, Figure 3 One of the schematic diagrams of skipping non-critical calculations of the Softmax network layer provided by an embodiment of the present invention is shown, and examples are given for skipping certain non-critical intermediate Softmax calculation steps.

[0070] like Figure 3 As shown in the figure, this is an example of intermediate Softmax skipping. First, two groups of four data are processed. The maximum value of the two groups of data is calculated, and each group of data and the maximum value of the group are processed with a subtractor. If the subtraction result is greater than the user-defined threshold, the data is directly set to zero and the subsequent calculation is skipped, that is, 1 is skipped in the middle. Otherwise, the result data is processed by the LUT to calculate the numerically stable power of 2 result. When the calculation is completed, the global maximum value is updated, and the other two groups of data and their group maximum values ​​are calculated. Since the difference between the current global maximum value and the group maximum value of group 3 is greater than the user-defined threshold, the subtraction and subsequent operations of group 3 are skipped. At the same time, since the difference between the group maximum value and the data of group 4 is greater than the threshold, the calculation of the two data is also skipped. When the results of group 3 and group 4 are calculated, the global maximum value is updated with the group maximum value of group 4. After all calculations are completed, the data of all groups are right-shifted according to the difference between the global maximum value and the group maximum value. Since the difference between the global maximum value and the group maximum value of group 2 is greater than the threshold, all data of group 2 are set to zero and the subsequent calculation is skipped, that is, 2 is skipped in the middle. After all compensations are completed, addition and division operations are calculated to obtain the final result. By using the skipping approach, we achieve an overall sparsity of 68.75% in this example.

[0071] For example, Figure 4 The second schematic diagram of skipping non-critical calculations of the Softmax network layer provided in an embodiment of the present invention is shown, and examples are given for some non-critical subsequent Softmax calculation steps.

[0072] like Figure 4As shown in Figure 1, after all data are calculated, the result Si is used to multiply with Vi to calculate the context Ci. Therefore, the generated zeros in Si can be skipped by masking to improve performance and energy efficiency. Based on the above principle, 68.75% of the multiplication and accumulation operations can be skipped, that is, post-skipping. By using this strategy, the computational overhead of the Softmax layer and the subsequent linear layer is reduced due to sparsity.

[0073] in addition, Figure 5 The figure shows the overall process diagram of the Softmax network layer provided by the embodiment of the present invention.

[0074] like Figure 5 As shown in FIG. 1 , the Softmax network layer receives the corresponding input vector, performs group sparse processing on it, and finally outputs the output vector and sparse mask of the Softmax network layer.

[0075] Specifically, the processing process of the Softmax network layer includes four stages, namely, the grouping result calculation stage, the global result compensation stage, the normalization stage, and the output stage.

[0076] First, in the grouping result calculation stage, the input vectors are grouped, and the multiple group vectors obtained by the grouping are subjected to exponential operation to obtain the grouping calculation result, and then the group maximum value in the grouping calculation result is obtained.

[0077] After all data are calculated, the group maximum value in each group of vectors is compared with the previously stored global maximum value. When the group maximum value is greater than the pre-stored global maximum value, the group maximum value is determined as the global maximum value and the global maximum value buffer is updated; when the group maximum value is less than the pre-stored global maximum value, the pre-stored global maximum value is determined as the global maximum value and continues to be stored.

[0078] Then, in the global result compensation stage, the calculated group calculation results are shift-compensated based on the global maximum value to obtain the target group calculation results. After all group vectors are calculated, the group maximum value will be subtracted from the global maximum value, and if the difference between the group calculation result and the set threshold is lower than the set difference, the group calculation result will skip the subsequent calculation and will no longer participate in the subsequent calculation process.

[0079] Finally, in the normalization stage, the target group calculation results are normalized to obtain the output vector of the Softmax network layer. Specifically, the data in all group vectors are accumulated in the adder, and the final accumulated result is stored in the register. Further, each data in all buffers is output, and the 32-bit accumulated result is divided to calculate and store the normalized data, which is the final output result / output vector of the Softmax network layer.

[0080] It should be noted that the accelerated processing of the Softmax network layer is to solve the high computational and storage cost problems of the traditional Softmax operation in the processing of very long sequences. The grouped sparse processing provided in this embodiment can significantly reduce the computational burden of the Softmax layer and subsequent linear layers.

[0081] In this embodiment, the input vector is grouped to obtain multiple group vectors, and the original index in the Softmax operation is converted into a set index, and then the multiple group vectors are independently calculated to obtain the output vector of the Softmax network layer, so as to perform dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer. This method performs sparse processing on the input vector of the Softmax network layer and dual-path sparse processing on the input vector of the LayerNorm network layer. This process not only significantly improves the computational efficiency of the Transformer model in processing long sequence data, but also reduces the corresponding memory overhead.

[0082] On the basis of the above embodiments, the data processing process of the LayerNorm network layer will be described in detail below.

[0083] The input vector of the LayerNorm network layer is subjected to dual-path sparse processing to obtain the output vector of the LayerNorm network layer, including: identifying important dimensions in the input vector of the LayerNorm network layer, using full-precision calculation for the important dimensions, and using half-precision calculation for the dimensions other than the important dimensions to obtain the output vector of the LayerNorm network layer.

[0084] Figure 6 A schematic diagram of the processing flow of the LayerNorm network layer provided by an embodiment of the present invention is shown.

[0085] like Figure 6 As shown, first, the important dimensions and unimportant dimensions in the input vector of the LayerNorm network layer are identified.

[0086] Identifying important dimensions in an input vector of a LayerNorm network layer includes: calculating variance values ​​corresponding to different dimensions in the input vector of the LayerNorm network layer; when the variance value is higher than a set variance value, determining the dimension corresponding to the variance value as an important dimension; when the variance value is lower than the set variance value, determining the dimension corresponding to the variance value as a non-important dimension, wherein the non-important dimension is a dimension other than the important dimension.

[0087] Of course, when identifying important dimensions and unimportant dimensions, in addition to selecting based on variance values, you can also select based on the number of times each dimension has been activated in the previous layer. For example, if and only if the number of times the dimension has been activated in the previous layer reaches a set number of times, the dimension is determined to be an important dimension, otherwise, the dimension is determined to be an unimportant dimension.

[0088] The variance value and the number of times can be set according to actual needs and are not specifically limited here.

[0089] It should be noted that the identified important dimensions have a greater impact on the accuracy of the final results, while the unimportant dimensions have little impact on the accuracy of the final results, and can even be ignored.

[0090] After identifying the important and unimportant dimensions in the input vector of the LayerNorm network layer, different precision algorithms are used for different dimension types. Specifically, full precision is used for data in important dimensions, and low precision / half precision / sparse calculations are used for data in unimportant precision. This is to simplify the calculations on unimportant dimensions, processing only the key parts and ignoring those parts that have little impact on the final result.

[0091] Different processing methods are used for data of important dimensions and unimportant dimensions, which greatly reduces the complexity of calculation and has little impact on the accuracy of the final calculation results.

[0092] It should be noted that the LayerNorm network layer also adopts specific computing optimization strategies when processing data. For example, in a specific embodiment, parallel processing and hardware acceleration functions are used. Through this optimization, LayNorm operations can be performed quickly and efficiently even when processing large amounts of data.

[0093] in addition, Figure 7 The figure shows the overall process diagram of the LayerNorm network layer provided by the embodiment of the present invention.

[0094] like Figure 7 As shown in Figure 1, the processing of the LayerNorm network layer includes four stages, namely, the accumulation calculation stage, the mean and variance calculation stage, the normalization stage, and the output stage.

[0095] First, in the accumulation calculation stage, the core is to complete the residual addition operation and calculate the data sum of all data and the sum of squares of all data.

[0096] The input vector of the LayerNorm network layer comes from the row vector of the input matrix of the input buffer and the calculation row of the context matrix of the attention fusion buffer. All data is processed by the adder to complete the residual addition operation and stored in the data buffer.

[0097] At the same time, the data is sent to two paths for sparse computing, one of which is for data computing of important dimensions, and the other is for data computing of unimportant dimensions.

[0098] Then, in the mean and variance calculation stage, after completing the data accumulation calculation, the mean and variance of the dimension vector are calculated based on the accumulated data and its square sum.

[0099] The data sum of all data is first shifted and divided by the feature dimension to calculate the mean, then stored in the mean register and passed through a multiplier to calculate its square value.

[0100] At the same time, the sum of the squares of all the data is also shifted and divided by the feature dimension, and the calculation result is subtracted from the square of the mean through the adder to obtain the variance. Finally, the variance is stored in the anti-erasure register.

[0101] Next, in the normalization stage, each data in the data buffer is normalized by the calculated mean and variance. Specifically, the data is first subtracted from the mean through an adder; then, each data is divided by the square root of the variance; finally, all data are multiplied and added by predefined parameters, and the result is shifted back to 8 bits and stored in the data buffer.

[0102] Finally, in the output stage, the final calculation result stored in the output data buffer is output, that is, the output vector of the LayerNorm network layer.

[0103] It should be noted that the accelerated processing of the LayerNorm network layer is to solve the high computational complexity problem of the LayerNorm operation when processing very long sequences. The dual-path sparse processing provided in this embodiment can effectively improve the computational efficiency of the LayerNorm operation.

[0104] In this embodiment, the output vector of the LayerNorm network layer is obtained by identifying the important dimensions in the input vector of the LayerNorm network layer, using full-precision calculation for the important dimensions, and using half-precision calculation for the dimensions other than the important dimensions. This method performs sparse processing on the input vector of the Softmax network layer and dual-path sparse processing on the input vector of the LayerNorm network layer. This process not only significantly improves the computational efficiency of the Transformer model in processing long sequence data, but also reduces the corresponding memory overhead.

[0105] In some embodiments, the hardware implementation technical details of the nonlinear layer acceleration method of the Transformer model provided by the present invention are given.

[0106] The group sparse processing of the Softmax network layer and the dual-path sparse processing of the LayerNorm network layer focus on the attention fusion sparse nonlinear architecture of the Transformer model.

[0107] (1) The maximum supported sequence length is 32768, and the length of each sequence slice is 64. This design is to adapt to the limited on-chip storage size and effectively process ultra-long sequences on hardware.

[0108] (2) The dimension of the input sequence is 1024, and the precision of activation and weight is 8 bits. This precision setting balances computational efficiency and result accuracy.

[0109] (3) The intermediate precision of the linear layer is 32 bits, and the intermediate precision of the Softmax and LayerNorm layers is 32 bits and 64 bits respectively. The FQ-ViT quantization method is used to optimize storage and computational efficiency.

[0110] (4) Hardware resources include 64-bit VecMul engine input size and 4 vector engines.

[0111] (5) The input buffer size is 80KB and the weight buffer size is 68KB.

[0112] (6) Using TSMC 65nm GP technology node, the supply voltage is 1.0V and the clock frequency is 200MHz.

[0113] (7) The attention fusion buffer size is 32KB.

[0114] In some other embodiments, the advantages of the nonlinear layer acceleration method of the Transformer model provided by the embodiments of the present invention are given.

[0115] (1) Significantly improved computational efficiency.

[0116] By introducing grouped sparse Softmax and dual-path sparse LayerNorm algorithms, the present invention significantly reduces the amount of computation required. These sparse algorithms can achieve a 4.45-fold and 7.59-fold reduction in Softmax computation and LayerNorm computation, thereby significantly improving the computational efficiency when processing long sequence data.

[0117] (2) Reduced memory requirements.

[0118] The hardware architecture design enables it to process longer sequences with a smaller memory footprint. It can support input lengths up to 32,768 and requires only 186KB of chip memory, significantly reducing memory requirements compared to existing technologies.

[0119] (3) Accelerate processing speed.

[0120] In group sparse Softmax processing and dual-path sparse LayerNorm processing, the present invention achieves processing speed acceleration of 1.75x-1.98x and 3.22x-4.32x, which means that when processing ultra-long sequences, the present invention can complete the same task faster than the prior art.

[0121] (4) Optimize hardware utilization.

[0122] The attention-fusion sparse nonlinear architecture reduces the need for large chip memory and makes more efficient use of hardware resources, which not only improves hardware efficiency but also reduces costs.

[0123] Corresponding to the nonlinear layer acceleration method of the Transformer model described in the above embodiments, an embodiment of the present invention further provides a nonlinear layer acceleration device of the Transformer model, specifically, Figure 8 A schematic diagram of the structure of a nonlinear layer acceleration device of a Transformer model provided by an embodiment of the present invention is shown.

[0124] like Figure 8 As shown, the device includes: an input vector acquisition module 810, used to obtain the input vector of the Softmax network layer; a Softmax network layer acceleration processing module 820, used to perform grouping sparse processing on the input vector to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; and a LayerNorm network layer acceleration processing module 830, used to perform dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer.

[0125] It should be noted that the nonlinear layer acceleration device of the Transformer model provided in the embodiment of the present invention and the nonlinear layer acceleration method of the Transformer model described in the above embodiments can be referred to each other, and will not be repeated here.

[0126] In this embodiment, the input vector of the Softmax network layer is obtained by the input vector acquisition module 810, and the Softmax network layer acceleration processing module 820 performs group sparse processing on the input vector to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer, and then the LayerNorm network layer acceleration processing module 830 performs dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer. The device performs sparse processing on the input vector of the Softmax network layer and dual-path sparse processing on the input vector of the LayerNorm network layer. This process not only significantly improves the computational efficiency of the Transformer model in processing long sequence data, but also reduces the corresponding memory overhead.

[0127] Fig. 9 The following is a schematic diagram of the physical structure of a nonlinear layer accelerator of a Transformer model. Fig. 9 As shown, the nonlinear layer accelerator of the Transformer model may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the nonlinear layer acceleration method of the Transformer model, the method comprising: obtaining the input vector of the Softmax network layer; performing group sparse processing on the input vector to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer.

[0128] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0129] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the nonlinear layer acceleration method of the Transformer model provided by the above-mentioned methods. The method includes: obtaining the input vector of the Softmax network layer; performing group sparse processing on the input vector to obtain the output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nonlinear layer acceleration method for a Transformer model, characterized in that: The Transformer model includes an encoder and a decoder, each of which includes a Softmax network layer and a LayerNorm network layer. The method includes: Get the input vector of the Softmax network layer; Performing group sparse processing on the input vector to obtain an output vector of a Softmax network layer; the output vector of the Softmax network layer is an input vector of a LayerNorm network layer; The input vector of the LayerNorm network layer is subjected to dual-path sparse processing to obtain an output vector of the LayerNorm network layer.

2. The nonlinear layer acceleration method of the Transformer model according to claim 1, characterized in that: The step of performing grouping and sparse processing on the input vector to obtain an output vector of a Softmax network layer includes: Grouping the input vectors to obtain a plurality of group vectors; Convert the original index in the Softmax operation to the set index, and independently calculate multiple group vectors to obtain the output vector of the Softmax network layer; Wherein, the set index is smaller than the original index.

3. The nonlinear layer acceleration method of the Transformer model according to claim 2, characterized in that: The independent calculation of the plurality of group vectors specifically includes: Take the set index for each data in the group vector to obtain the group calculation result; Based on the pre-stored global maximum value, shift compensation is performed on the group calculation result to obtain a target group calculation result; The target group calculation results are normalized to obtain the output vector of the Softmax network layer.

4. The nonlinear layer acceleration method of the Transformer model according to claim 3, characterized in that: The step of obtaining the global maximum value comprises: Determine the group maximum value in the group calculation result; When the group maximum value is greater than the pre-stored global maximum value, determining the group maximum value as the global maximum value; In the case that the group maximum value is smaller than the pre-stored global maximum value, the pre-stored global maximum value is determined as the global maximum value.

5. The nonlinear layer acceleration method of the Transformer model according to claim 3, characterized in that: The obtaining of the group calculation results then includes: Determine the difference between the group calculation result and a set threshold; When the difference is lower than the set difference, the group calculation result will no longer participate in the subsequent calculation process.

6. The nonlinear layer acceleration method of the Transformer model according to claim 1, characterized in that: The step of performing dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer includes: Identify important dimensions in the input vector of the LayerNorm network layer, use full-precision calculation for the important dimensions, and use half-precision calculation for other dimensions other than the important dimensions, so as to obtain an output vector of the LayerNorm network layer.

7. The nonlinear layer acceleration method of the Transformer model according to claim 6, characterized in that: The identifying important dimensions in the input vector of the LayerNorm network layer includes: Calculate the variance values ​​corresponding to different dimensions in the input vector of the LayerNorm network layer; When the variance value is higher than the set variance value, determining the dimension corresponding to the variance value as an important dimension; When the variance value is lower than the set variance value, the dimension corresponding to the variance value is determined as a non-important dimension, where the non-important dimension is a dimension other than the important dimension.

8. A nonlinear layer acceleration device for a Transformer model, characterized in that: The Transformer model includes an encoder and a decoder, each of which includes a Softmax network layer and a LayerNorm network layer. The device includes: Input vector acquisition module, used to obtain the input vector of the Softmax network layer; A Softmax network layer acceleration processing module is used to perform grouping and sparse processing on the input vector to obtain an output vector of the Softmax network layer; the output vector of the Softmax network layer is the input vector of the LayerNorm network layer; The LayerNorm network layer acceleration processing module is used to perform dual-path sparse processing on the input vector of the LayerNorm network layer to obtain the output vector of the LayerNorm network layer.

9. A nonlinear layer accelerator for a Transformer model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the nonlinear layer acceleration method of the Transformer model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the nonlinear layer acceleration method of the Transformer model according to any one of claims 1 to 7 is implemented.

Images