Multi-core vector processor-oriented sparse bert model acceleration method, apparatus and device, and medium

Abstract

The invention discloses a sparse bert model acceleration method and device for a multi-core vector processor, equipment and a medium, and relates to the technical field of vector processors, and the method comprises the steps: constructing a pre-training language model for adjustment and optimization; determining the type of the graph optimization operation; the type is operator fine tuning, and the graph optimization operation of the target application scene is realized by utilizing the optimized pre-training language model and a multi-core vector processor in combination with layer normalization and matrix transposition; the type is operator fusion, operator fusion is carried out on each layer of encoders in the optimized pre-training language model based on a multi-core vector processor, and graph optimization operation on a target application scene is realized; the type is storage optimization, data storage structure optimization and data reuse design are carried out on the optimized pre-training language model based on the multi-core vector processor, graph optimization operation on a target application scene is achieved, the parallel computing capacity of the multi-core vector processor can be brought into full play, and the data reusability and the data storage effect are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vector processors, and in particular to a sparse BERT model acceleration method, device, equipment and medium for a multi-core vector processor. Background Art

[0002] In recent years, the Transformer (neural network) model has become a hot topic in the field of advanced deep learning and deep neural networks. By introducing the self-attention mechanism, it has achieved significant performance improvements in processing long-distance dependencies and capturing complex semantic relationships and contextual information. However, the Transformer model generally has a large number of parameters, which poses a considerable challenge to both memory efficiency and computing efficiency. Traditional graph optimization methods have a lot of redundant calculations and cannot fully utilize the vectorized computing capabilities of the accelerator. In addition, the number of data transmissions is high, and the data storage effect is poor.

[0003] As can be seen from the above, how to give full play to the parallel computing capabilities of multi-core vector processors and improve data reusability and data storage effects is a problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a sparse BERT model acceleration method, device, equipment and medium for multi-core vector processors, which can give full play to the parallel computing capabilities of multi-core vector processors and improve data reusability and data storage effects. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a sparse BERT model acceleration method for a multi-core vector processor, comprising:

[0006] Constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain an optimized pre-trained language model;

[0007] Determine the type of graph optimization operation to be performed on a target application scenario; the target application scenario is a scenario that meets a preset low latency condition;

[0008] If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used in combination with layer normalization and matrix transposition operations to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario;

[0009] If the type is operator fusion, a target operator fusion mode is determined, and based on the multi-core vector processor and in accordance with the target operator fusion mode, the operator fusion is performed on the encoders of each layer in the optimized pre-trained language model to implement a graph optimization operation of the type of operator fusion for the target application scenario;

[0010] If the type is storage optimization, data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of storage optimization type for the target application scenario.

[0011] Optionally, the constructing a pre-trained language model based on semi-structured sparse matrix multiplication includes:

[0012] The semi-structured N:M sparse pattern is used to perform sparsification design on matrix multiplication to obtain semi-structured sparse matrix multiplication.

[0013] A pre-trained language model is constructed based on the semi-structured sparse matrix multiplication; the pre-trained language model includes an embedding layer, an encoder layer of a plurality of encoder stacks with the same structure, and a pooling layer.

[0014] Optionally, the multi-core vector processor includes: a central processing unit, a digital signal computing cluster; the digital signal computing cluster includes a digital signal computing core and a global shared memory; the digital signal computing core includes a scalar storage space and a vector storage space.

[0015] Optionally, if the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used in combination with layer normalization and matrix transposition operations to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario, including:

[0016] If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used, combined with the matrix device operation and based on the layer normalization calculation formula to calculate the input sample matrix, and the matrix dimension of the layer normalization calculation is switched to obtain the output result, so as to complete the graph optimization operation of the target application scenario of the type of operator fine-tuning; the layer normalization calculation formula is:

[0017] ;

[0018] in, To output the result, is the input sample matrix, and They are The mean and standard deviation of is a constant, and are learnable scaling and shift parameters, and m is the number of elements.

[0019] Optionally, determining the target operator fusion method includes:

[0020] A target operator fusion method is selected from all operator fusion methods; the operator fusion methods include SpMM and Bias operator fusion, SpMM and Bias and Residual operator fusion, SpMM and Bias and GELU operator fusion, GEMM and Add_mask and Softmax operator fusion.

[0021] Optionally, the performing operator fusion on encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion method to implement a graph optimization operation of the type of operator fusion for the target application scenario includes:

[0022] Filtering operators with data dependency from encoders of each layer in the optimized pre-trained language model;

[0023] Based on the multi-core vector processor and in accordance with the target operator fusion method, the operators with data dependencies in the optimized pre-trained language model are fused to implement a graph optimization operation of the type of operator fusion for the target application scenario.

[0024] Optionally, if the type is storage optimization, data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of storage optimization type for the target application scenario, including:

[0025] If the type is storage optimization, the data storage structure of the encoder in the optimized pre-trained language model is optimized based on the storage structure characteristics of the multi-core vector processor, and the cache memory is optimized according to the self-attention layer in the optimized pre-trained language model, so as to realize the graph optimization operation of storage optimization for the target application scenario.

[0026] In a second aspect, the present application discloses a sparse BERT model acceleration device for a multi-core vector processor, comprising:

[0027] A construction and optimization module, used for constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain an optimized pre-trained language model;

[0028] A type determination module, used to determine the type of graph optimization operation to be performed on a target application scenario; the target application scenario is a scenario that meets a preset low latency condition;

[0029] An operator fine-tuning module, for, if the type is operator fine-tuning, using the optimized pre-trained language model and multi-core vector processor, and combining layer normalization and matrix transposition operations, to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario;

[0030] An operator fusion module, for determining a target operator fusion mode if the type is operator fusion, and performing operator fusion on encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion mode, so as to implement a graph optimization operation of the type of operator fusion for the target application scenario;

[0031] A storage optimization module is used to optimize the data storage structure and design data reuse for the optimized pre-trained language model based on the multi-core vector processor if the type is storage optimization, so as to implement a graph optimization operation of the type of storage optimization for the target application scenario.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned sparse BERT model acceleration method for multi-core vector processors.

[0035] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed sparse BERT model acceleration method for a multi-core vector processor are implemented.

[0036] It can be seen that the present application provides a sparse BERT model acceleration method for a multi-core vector processor, including constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain the optimized pre-trained language model; determining the type of graph optimization operation for the target application scenario; the target application scenario is a scenario that meets the preset low latency condition; if the type is operator fine-tuning, the optimized pre-trained language model and the multi-core vector processor are used, and layer normalization and matrix transposition operations are combined to achieve a graph optimization operation of the type of operator fine-tuning for the target application scenario; if the type is operator fusion, a target operator fusion mode is determined, and based on the multi-core vector processor and according to the target operator fusion mode, the encoders of each layer in the optimized pre-trained language model are subjected to operator fusion to achieve a graph optimization operation of the type of operator fusion for the target application scenario; if the type is storage optimization, the data storage structure optimization and data reuse design of the optimized pre-trained language model are performed based on the multi-core vector processor to achieve a graph optimization operation of the type of storage optimization for the target application scenario. This application constructs a pre-trained language model based on semi-structured sparse matrix multiplication, adjusts and optimizes the model, determines the type of graph optimization operation for the target application scenario, and if the type is operator fine-tuning, uses the optimized pre-trained language model and multi-core vector processor, and combines layer normalization and matrix transposition operations to implement the corresponding graph optimization operation, so that the redundant operations of transposition can be simplified and the vectorized computing capabilities of the multi-core vector processor can be fully utilized. If the type is operator fusion, the encoders of each layer in the optimized pre-trained language model are fused by operators to reduce the number of data transmissions, improve data reusability, and reduce redundant computing operations. If the type is storage optimization, the data storage structure of the optimized pre-trained language model is optimized and the data reuse design is performed to fully utilize the storage structure of the multi-core vector processor and improve the data storage effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of a sparse BERT model acceleration method for a multi-core vector processor disclosed in this application;

[0039] Figure 2 A structural diagram of a pre-trained language model disclosed in this application;

[0040] Figure 3 A schematic diagram of data storage and calculation in a dense matrix multiplication disclosed in this application on a domestic vector processor FT-M7032;

[0041] Figure 4 A schematic diagram of data storage and calculation in a sparse matrix multiplication disclosed in this application on a domestic vector processor FT-M7032;

[0042] Figure 5 A diagram showing the structural changes of a BERT model after introducing sparse matrix multiplication disclosed in this application;

[0043] Figure 6 The architecture diagram of a domestic vector processor FT-M7032 disclosed in this application;

[0044] Figure 7 A schematic diagram of combining layer normalization and transposition operations disclosed in the present application;

[0045] Figure 8 A schematic diagram of the completion of encoder operator fusion of a BERT model disclosed in this application;

[0046] Fig. 9 This is a schematic diagram of an I×Wq data storage disclosed in this application;

[0047] Fig.10 This is a schematic diagram of an I×Wk data storage disclosed in this application;

[0048] Fig.11 This is a schematic diagram of an I×Wv data storage disclosed in this application;

[0049] Fig.12 This is a schematic diagram of the structure of a sparse BERT model acceleration device for a multi-core vector processor disclosed in this application;

[0050] Fig.13 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In recent years, the Transformer model has become a hot topic in the field of advanced deep learning and deep neural networks. By introducing the self-attention mechanism, it has achieved significant performance improvements in processing long-distance dependencies and capturing complex semantic relationships and contextual information. However, the Transformer model generally has a large number of parameters, which poses a considerable challenge to both memory efficiency and computing efficiency. Traditional graph optimization methods have a lot of redundant calculations and cannot fully utilize the vectorized computing capabilities of the accelerator. In addition, the number of data transmissions is high and the data storage effect is poor. As can be seen from the above, how to fully utilize the parallel computing capabilities of multi-core vector processors and improve data reusability and data storage effects are issues to be solved in this field.

[0053] See also Figure 1 As shown, the embodiment of the present invention discloses a sparse BERT model acceleration method for a multi-core vector processor, which may specifically include:

[0054] Step S11: constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain the optimized pre-trained language model.

[0055] In this embodiment, a semi-structured N:M sparse pattern is used to perform a sparse design on matrix multiplication to obtain a semi-structured sparse matrix multiplication; a pre-trained language model is constructed based on the semi-structured sparse matrix multiplication, and the pre-trained language model is adjusted and optimized to obtain the optimized pre-trained language model; the pre-trained language model includes an embedding layer, an encoder layer of a plurality of encoder stacks with the same structure, and a pooling layer.

[0056] The implementation of semi-structured pruning used in this application is N:M semi-structured pruning. In this sparse model, there are at most N non-zero elements in each consecutive M weights. Compared with structured pruning, semi-structured pruning can obtain higher model accuracy and has more computational parallelism. However, the research and optimization acceleration of semi-structured sparse pruning methods are still in the preliminary stage on general platforms. This patent will perform semi-structured sparse optimization on BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on Transformer) for domestic DSP (Digital Signal Processing).

[0057] The BERT model is mainly composed of an embedding layer, an encoder layer, and a pooling layer. The embedding layer mainly performs table lookup operations, and the pooling layer has a low computational load, so the computational bottleneck is mainly in the encoder layer. The structure of the BERT model is as follows: Figure 2As shown in the figure, for BERT, the encoder layer is a stacked structure of multiple encoders with the same structure, with a total of 12 layers, corresponding to N in the figure. The structure of each layer is consistent, consisting of a multi-head self-attention mechanism layer, a feedforward network layer, a normalization layer, and a residual layer, but the weights of each encoder are not shared. The reasoning process of each encoder of BERT involves operations such as matrix multiplication, vector reduction, Softmax (normalized exponential function), LayerNorm (Layer Normalization), GELU (Gaussian Error Linear Units, an activation function based on the Gaussian error function), and matrix transposition. The sparse processing of the encoder is to replace all matrix multiplication calculations with sparse matrix multiplication calculations with weighted data, and other operators still maintain dense format calculations. In the sparse processing, because the specific calculation implementation is different from the general matrix multiplication, the data format of the sparse matrix multiplication changes, which leads to some subtle changes in the structure of the BERT model, as detailed below:

[0058] For the matrix multiplication of the original BERT model, taking the self-attention mechanism layer as an example, let the input matrix be I, the weight matrix be W, and the output matrix be O. The specifications of each matrix in the dense matrix multiplication are designed as input matrix I: [Seq_length][hid_size], weight matrix W: [hid_size][hid_size], output matrix O: [Seq_length][hid_size], and the data storage and calculation on the domestic vector processor FT-M7032 in the dense matrix multiplication are shown as follows Figure 3 As shown in the figure, the input matrix I will be transferred to the SM space in the core by DMA, the weight matrix W will be transferred to the AM space in the core, the register reads the I matrix data from the SM scalar and vectorizes it into a set of identical data, and then the register reads a set of W matrix data from the AM vector, the two are multiplied correspondingly and accumulated in a loop in the hid_size dimension until the calculation is completed.

[0059] In the semi-structured sparse matrix multiplication, the input, weight, and output matrices are also Is, Ws, and Os. Since Ws is a sparse compressed matrix, and the decoding operation of the Ws matrix can only be implemented one by one through scalar instructions, and cannot be batched in the vector space, the Ws matrix can only be placed in the SM, so the matrix multiplication formula in the original model is: O = I × W, and the calculation formula of the sparse matrix multiplication in the sparse model is: Os = Ws × Is, where the data format of the Ws, Is, and Os matrices are all transposed with the original matrices W, I, and O, specifically Is: [hid_size][Seq_length], Ws: [Seq_length][Seq_length], O: [Seq_length][hid_size], the data storage and calculation diagram of the sparse matrix multiplication on the domestic vector processor FT-M7032 is as follows Figure 4 shown.

[0060] Since sparse matrix multiplication and matrix multiplication are calculated differently, the matrix data format changes. Assuming that the weight matrix is ​​transposed before entering the model, after the sparse operation is introduced in the BERT model, the input and output matrices before and after the sparse matrix multiplication (SpMM) calculation also need to be transposed. The BERT model calculation graph will change. After the introduction of sparse matrix multiplication, the BERT model structure changes as follows: Figure 5 shown.

[0061] Step S12: Determine the type of graph optimization operation to be performed on the target application scenario; the target application scenario is a scenario that meets a preset low-latency condition.

[0062] Based on the above background, this application will fine-tune and optimize the encoder structure based on the data format changes caused by sparse matrix multiplication, and perform the following graph optimization operations for low-latency application scenarios based on the processor's multi-level storage structure and model reasoning framework.

[0063] Step S13: If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used in combination with layer normalization and matrix transposition operations to implement a graph optimization operation of operator fine-tuning for the target application scenario.

[0064] Among them, the multi-core vector processor includes: a central processing unit, a digital signal computing cluster; the digital signal computing cluster includes a digital signal computing core and a global shared memory; the digital signal computing core includes a scalar storage space and a vector storage space. In recent years, DSP vector processors have been increasingly used in supercomputing, AI (Artificial Intelligence) acceleration and other scenarios. Their architectural features make them also suitable for computationally intensive tasks, such as deep neural network reasoning. The development hardware platform for this patent is FT-M7032, which is a domestically produced multi-core DSP vector processor consisting of a 16-core CPU (Central Processing Unit) and four DSP computing clusters. The architecture of the domestically produced vector processor FT-M7032 is as follows: Figure 6 As shown. Each DSP computing cluster consists of 8 general-purpose DSP computing cores and a 6MB GSM (Global Shared Memory). The GSM is accessed by all DSP cores in the same cluster. Each cluster can access the corresponding local DDR (Double Data Rate) memory space. Each DSP computing core contains 64KB of SM (scalar storage space) and 768KB of vector storage space. DDR, GSM and SM / AM (vector storage space) in the core can communicate through a cross switch network. DMA is responsible for data transmission between the multi-level storage structure composed of DDR, GSM, SM / AM. The DSP computing core of the chip supports FP64, FP32 and FP16 operations. This application is developed and designed based on the FP16 data type.

[0065] In this embodiment, if the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used, combined with the matrix device operation and based on the layer normalization calculation formula to calculate the input sample matrix, and the matrix dimension of the layer normalization calculation is switched to obtain the output result, so as to complete the graph optimization operation of the target application scenario of the type of operator fine-tuning; the layer normalization calculation formula is:

[0066] ;

[0067] in, To output the result, is the input sample matrix, and They are The mean and standard deviation of is a constant, and are learnable scaling and shift parameters, and m is the number of elements.

[0068] In this embodiment, Figure 5 It can be seen that there are a lot of transposition operations in the BERT model that introduces sparse matrix multiplication. The number of transpositions of each encoder increases from 1 in the original model to 7. Although sparse operations can improve the running speed of the model, multiple transpositions affect the better performance of the model to a certain extent. In order to reduce redundant and repeated transposition operations, this application proposes the idea of ​​merging the LayerNorm and Transpose operations for calculation.

[0069] Transpose is to swap the order of the matrix dimensions without any calculation operation, while LayerNorm is to normalize each sample. The combined layer normalization and transpose operation is shown below: Figure 7 As shown, for a three-dimensional matrix of size [batch][seq_length][hid_size], Layernorm performs normalization calculation on the hid_size dimension in units of samples, as follows: Figure 7 In the left figure, the calculation direction is transverse. When the three operations of Transpose, LayerNorm and Transpose exist continuously, Transpose can be skipped directly and simplified to Figure 7 In the right figure, the three-in-one operation is renamed as LayerNorm_H operator. Its input and output matrix format is [batch][hid_size][seq_length]. The layer normalization operation is still performed in the hid_size dimension, but its calculation direction is vertical cutting according to the different data storage formats. The specific calculation formula is shown above. and The reduction calculation is performed on the second dimension in the matrix, that is, the seq_length dimension. and Applied to each input, Usually take , for numerical stability.

[0070] Through this calculation fine-tuning action, the number of transpositions in the sparse encoder is reduced from 7 to 3, and the encoder layer contains 12 encoders, the overall transposition operation is reduced by 48 times, and the operating efficiency of the sparse model is improved. The implementation process of the LayerNorm_H half-precision operator is as follows:

[0071] Step 1: Obtain the logical number of each DSP computing core according to the number of startup cores and the startup status, and execute the following procedures;

[0072] Step 2: Allocate corresponding storage space in the SM space for the weights and bias parameters calculated by LayerNorm_H, and transfer the half-precision weights and bias parameters from DDR to the SM space through DMA;

[0073] Step 3: Allocate corresponding storage space in the AM space for the half-precision input and output data calculated by LayerNorm_H, and transfer the input data from the DDR to the AM space via DMA;

[0074] Step 4: Each core performs parallel calculations in the batch dimension with a block size of 1, calling the microkernel assembly function, and the calculation specification for each calculation is hid_size×seq_length;

[0075] Step 5: Enter the inline assembly function and perform LayerNorm_H calculation on the microkernel hid_size×seq_length matrix;

[0076] Step 6: Calculate the mean value in the hid_size dimension, initialize i=0, where i is the index of the hid_size dimension of the microkernel data block;

[0077] Step 7: Vector loads 32 rows of half-precision data on the hid_size dimension at index i-i+31 To different vector registers VR0-VR63, the data size is 32×seq_length;

[0078] Step 8. Replace the Accumulate by row. Note that in order to reduce the loss of half-precision, the row accumulation process needs to be added in pairs according to the binary method to avoid a large difference between the values ​​of the two addends. After the calculation, the intermediate result of length seq_length is obtained and stored in the corresponding AM temporary space;

[0079] Step 9: Let i = i + 32;

[0080] Step 10: Determine whether i is less than hid_size. If so, return to step 7. If not, execute sequentially.

[0081] Step 11, obtain multiple 32 rows of accumulated intermediate results stored on AM in the hid_size dimension, and similarly add the data in pairs by row to obtain an accumulated result with a length of seq_length;

[0082] Step 12: Multiply the accumulated result of the previous step by 1 / hid_size to get the average value, which corresponds to the calculation formula , and temporarily store it in the AM space;

[0083] Step 13: Calculate the sum of squares in the hid_size dimension, initialize j = 0, where j is the index of the hid_size dimension of the microkernel data block;

[0084] Step 14: vector loads 16 rows of half-precision data xi in hid_size dimension at index j - j+15 to different vector registers VR0-VR31, and the data size is 16×seq_length;

[0085] Step 15: Change the data in VR0-VR31 With the mean Perform vector subtraction row by row to obtain the difference of each element and continue to save it in VR0-VR31;

[0086] Step 16: To reduce subsequent precision loss, the half-precision difference result is converted into single-precision data in two steps, with 8 rows of data converted at a time.

[0087] Step 17: Get the sum of the squares of the single-precision difference results of the 16 rows, obtain the sum of the squares of the length seq_length, and accumulate it to the AM temporary space;

[0088] Step 18: Let j = j + 16;

[0089] Step 19, determine whether j is less than hid_size, if so, return to step 14, if not, execute sequentially;

[0090] Step 20, obtain the single-precision difference square sum result on the hid_size dimension and assign it to VR0-VR3;

[0091] Step 21, initialize the VR62 register to a single-precision value 1 / hid_size;

[0092] Step 22, perform single-precision floating-point multiplication on the single-precision difference square sum result in VR0-VR3 and 1 / hid_size in VR62 and store the result in VR0-VR3;

[0093] Step 23, take the inverse square root of the data in VR0-VR3 and store it in VR4-VR7;

[0094] Step 24: Convert the single-precision square root inverse into half-precision data to obtain , stored in VR62-VR63;

[0095] Step 25, initialize k = 0, k is the index of the hid_size dimension of the microkernel data block;

[0096] Step 26, vector loads 16 rows of half-precision data xi on hid_size dimensions at indexes k - k+15 to different vector registers VR0-VR31;

[0097] Step 27: Vector loading mean into registers VR54-VR55, which are loaded into VR62-VR63;

[0098] Step 28: Data With the mean Do vector floating point subtraction and then Perform vector floating-point multiplication and store the result in VR0-VR31;

[0099] Step 29: Load the normalized parameters corresponding to rows k - k+15 in sequence and , and perform operations such as low-order expansion, high-order expansion, and broadcasting on it to transform it into vector data with the same value;

[0100] Step 30: Compare the intermediate data in VR0-VR31 with the normalization parameters and Perform vector floating-point multiplication and addition operations to obtain the final calculation results of each element, and store them in the corresponding storage space of the output data on AM in sequence;

[0101] Step 31, let k = k + 16;

[0102] Step 32, determine whether k is less than hid_size, if so, return to step 26, if not, execute sequentially;

[0103] Step 33, at this point, the calculation of LayerNorm_H of the microkernel calculation with a data block size of 1×hid_size×seq_length is completed;

[0104] Step 34: The output result calculated by the microkernel and stored on the AM is transferred back to the DDR space via DMA.

[0105] Step 35: Calculation is completed, fence synchronization is performed, and the function is exited after waiting for all DSP computing cores to arrive.

[0106] Step S14: If the type is operator fusion, determine the target operator fusion method, perform operator fusion on the encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion method, and implement a graph optimization operation of the type of operator fusion for the target application scenario.

[0107] In this embodiment, if the type is operator fusion, a target operator fusion method is screened out from all operator fusion methods; the operator fusion method includes SpMM and Bias operator fusion, SpMM and Bias and Residual operator fusion, SpMM and Bias and GELU operator fusion, GEMM and Add_mask and Softmax operator fusion, and operators with data dependence are screened out from the encoders of each layer in the optimized pre-trained language model. Based on the multi-core vector processor and according to the target operator fusion method, the operators with data dependence in the optimized pre-trained language model are fused to realize a graph optimization operation of the type of operator fusion for the target application scenario.

[0108] Operator fusion is the main optimization method for many inference frameworks to improve efficiency. It combines operators with data dependencies on the computational graph to eliminate unnecessary instantiation of intermediate results of some operators, reduce memory access pressure and computational complexity, and thus achieve the purpose of performance improvement. Operator fusion can reduce memory usage, improve data memory access locality, and alleviate memory bottlenecks and bandwidth pressure to a certain extent; the fusion operation merges operators with data dependencies, reduces the reading and writing of intermediate variables, improves the utilization efficiency of resources such as memory and registers, eliminates redundant operations in the computational graph, and reduces the complexity of the computational graph, which also improves computational efficiency; in addition, the fused computational graph reduces the number of kernel calls, reduces some startup overhead, stack access overhead, etc. Based on the principle of operator fusion, the encoder inference computational graph can mainly implement four operator fusion methods, as follows:

[0109] (1) Fusion of SpMM and Bias operator;

[0110] (2) Fusion of SpMM, Bias and Residual operators;

[0111] (3) Fusion of SpMM, Bias and GELU operators;

[0112] (4) Operator fusion of GEMM, Add_mask and Softmax.

[0113] The first three categories are fusion optimizations based on sparse matrix multiplication calculations. In this model structure, sparse matrix multiplication is a computationally intensive operation, while addition, activation, and normalized exponential functions are memory-intensive operations. The amount of computation itself is not large. For accelerators with low memory bandwidth, if they are calculated separately, most of their running time will be consumed in data memory access, which will lead to very low computational efficiency. Combining it with sparse matrix multiplication can reduce the transmission of overall input and output data, which will benefit the model execution efficiency to a certain extent. There are two fusion implementations in specific operator fusion, one is in-core fusion and the other is microkernel fusion. In-core fusion means that from the perspective of out-of-core operations, operators are merged into one interface. After multiple operators are merged, the instantiation of intermediate results is omitted. The fusion operator only needs to move and transmit the input data of the first execution operator and the output data of the last execution operator in and out of the core. Microkernel fusion means that on the basis of intra-kernel fusion, the inline assembly of operators is further integrated, and the assembly functions are merged into one interface. Programmers need to design the corresponding blocks of AM and SM storage space according to the input parameters and intermediate results of each operator, and uniformly implement the calculation of multiple operators in the assembly. For programmers, microkernel fusion is the highest level of fusion, and the premise of microkernel fusion is intra-kernel fusion. Based on the above fusion ideas, the various layers of the BERT encoder are fused to obtain the fused calculation diagram as shown below: Figure 8 The following takes the operator fusion of SpMM, Bias and GELU as an example to illustrate the specific implementation of this operator fusion on the domestic multi-core DSP vector processor FT-M7032:

[0114] In the operator fusion of SpMM, Bias and GELU, since the calculation formula of Bias is relatively simple, this patent application designs the fusion of SpMM and Bias as a microkernel fusion; and since the calculation formula of GELU includes an exponential function, its inline assembly implementation is relatively complex, so the microkernel assembly of GELU will still be implemented separately, and the GELU microkernel assembly function is merged with the SpMM and Bias microkernel assembly functions in the kernel functions at the data block and call level. Since this patent application is aimed at the BERT-Base model specifications and low-latency application scenarios, the batchsize value in the calculation is 1, the input matrix specification is set to [hid_size2][seq_length], the weight matrix specification is [hid_size1][hid_size2], and the bias parameter specification is [hid_size1]. The specific implementation is as follows:

[0115] Step 1: Obtain the logical number of each DSP computing core according to the number of startup cores and the startup status, and execute the following procedures;

[0116] Step 2: According to the size of the weight matrix, allocate corresponding storage space SM_weight and SM_bias in the SM space for the weights calculated by SpMM and the bias parameters calculated by Bias;

[0117] Step 3, according to the size of the input matrix and the output matrix, allocate corresponding storage spaces AM_input, AM_output and AM_gelu_output in the AM space for the input data calculated by SpMM, the output data calculated by Bias and the output data of GELU;

[0118] Step 4, set the block sizes of the matrix in hid_size1 dimension and hid_size2 dimension to block_hid1 and block_hid2 respectively, and the number of blocks is loop_hid1 = hid_size1 / block_hid1, loop_hid2 = hid_size2 / block_hid2;

[0119] Step 5: Transfer the bias parameters from DDR to the SM specified space SM_bias by DMA (since the amount of bias parameters is small, this part of the data transfer can be completed at one time);

[0120] Step 6: Initialize i = 0, where i is the index of the input block data in the hid_size2 dimension;

[0121] Step 7: Transfer the input block data indexed as i in the hid_size2 dimension from DDR to the AM space AM_input by DMA;

[0122] Step 8: Initialize j = 0, where j is the index of the weighted block data in the hid_size1 dimension;

[0123] Step 9: Transfer the weight block data with index j on hid_size1 dimension and index i on hid_size2 dimension from DDR to SM space SM_weight by DMA;

[0124] Step 10: Input data, weight data and bias data are ready. Call the microkernel assembly function of SpMM+Bias fusion to perform the corresponding calculation operation. The calculation specification of each time is block_hid1×block_hid2×seq_length. The data result of block_hid1×seq_length is obtained and saved in the AM space AM_output.

[0125] Step 11, load the calculation result of SpMM+Bias in AM_output, call the GELU microkernel assembly function, perform the corresponding calculation, obtain the data result of block_hid1×seq_length, and save the result to AM space AM_gelu_output;

[0126] Step 12: Determine whether i is equal to loop_hid2-1. If yes, move the data result in AM_gelu_output to the corresponding DDR. If no, skip;

[0127] Step 13: Let j = j + 1;

[0128] Step 14: Determine whether j is less than loop_hid1. If so, return to step 9. If not, execute sequentially.

[0129] Step 15: Let i = i + 1;

[0130] Step 16, determine whether i is less than loop_hid2, if so, return to step 7, if not, execute sequentially;

[0131] Step 17: Calculation is completed, fence synchronization is performed, and all computing DSP cores are waited for to reach the exit function.

[0132] Step S15: If the type is storage optimization, data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of storage optimization type for the target application scenario.

[0133] In this embodiment, if the type is storage optimization, the data storage structure of the encoder in the optimized pre-trained language model is optimized based on the storage structure characteristics of the multi-core vector processor, and the cache memory is optimized according to the self-attention layer in the optimized pre-trained language model, so as to implement a graph optimization operation of the type of storage optimization for the target application scenario.

[0134] This application optimizes data storage from two dimensions: the storage structure characteristics of the processor and the BERT encoder structure. From the perspective of the storage structure of the processor, the FT-M7032 processor has a multi-level storage structure, which is expressed as DDR-GSM-AM / SM. Among them, the DDR memory is connected to the GSDSP and the CPU through a cross-switch network. The GSDSP contains 4 clusters, each cluster contains 8 computing cores, and each computing core has a scalar storage space SM and a vector storage space AM. The computing cores in the same cluster share a shared storage space GSM. The transmission bandwidth from GSM to the GSDSP computing core is much higher than the bandwidth from DDR to GSDSP. Therefore, in order to make full use of the high bandwidth within the cluster and reduce the memory access overhead, the input and output data in each encoder layer in the BERT model are temporarily stored in the GSM to reduce the interaction between data and the storage space outside the core. During the calculation, the input of the first encoder and the output of the last encoder are removed to ensure that other calculation results are not output.

[0135] In addition, the present application performs data reuse design based on the self-attention layer of the BERT encoder. Data reuse design is a common inference performance optimization technology that improves inference performance by exchanging space for time without affecting any accuracy. In the multi-head self-attention mechanism of the BERT model, the input matrix is ​​I, and each self-attention layer involves four weight matrices and four sets of bias parameters, namely Wq, Wk, Wv, Wo and Bq, Bk, Bv, Bo, corresponding to query, key, value, and output matrices. For each self-attention layer, the input I needs to be mapped to the query matrix, key matrix, and value matrix through three weight matrices Wq, Wk, and Wv, and then added as the matrix multiplication result of the residual block and the Wo weight matrix after a series of calculations. Therefore, in each encoder, the same input matrix I will participate in four calculations, three matrix multiplications and one residual addition.

[0136] Therefore, when constructing a sparse BERT model in a domestic DSP accelerator, when the input matrix I is transmitted to the GSM, I will participate in three SpMMs and one residual addition calculation. This part of the data needs to be kept intact in the GSM before the calculation is completed. When the DSP performs SpMM and residual addition calculations, the input matrix needs to be moved to the AM space. Therefore, in order to improve the reusability of the input matrix, this design continues to perform cache optimization on the input matrix in the self-attention layer in the core. Three SpMM calculations are performed continuously in the self-attention layer, and then operators such as GEMM and Softmax are calculated, and finally added to the input matrix, that is, residual addition. It can be seen that there are many operator operations between residual addition and SpMM, which requires a large space to store the intermediate layer results. It is difficult to ensure that the intermediate input matrix is ​​not destroyed. Therefore, the cache optimization in the self-attention layer is only for three consecutive SpMM calculations. Similarly, a corresponding fixed space size is allocated to the input matrix I in the storage space AM in the core. Before the SpMM calculation, the input matrix I is transferred to the AM space at one time, and the SpMM calculation is performed three times in succession. During the calculation process, the input matrix is ​​ensured not to be destroyed in the AM storage space. The I×Wq (sparse matrix multiplication) data storage diagram is shown as follows: Fig. 9 As shown, the I×Wk (sparse matrix multiplication) data storage diagram is as follows Fig.10 As shown, the I×Wv (sparse matrix multiplication) data storage diagram is as follows Fig.11 As shown in the figure. The cache optimization design of the self-attention layer sacrifices part of the AM space to reduce the number of data transfers, so that the number of transmissions of the input matrix I in the original model encoder is reduced from 4 times to 2 times, which fully improves the reusability of the data, reduces the memory access pressure of the model, and improves the reasoning performance. The specific implementation steps are as follows.

[0137] Step 1: In GSM, the corresponding data space is allocated according to the data size, namely GSM_I, GSM_q, GSM_k, and GSM_v, for the transposed result of the input matrix I and the output result of SpMM+Bias.

[0138] Step 2: After entering the encoder, perform a transpose operation on the input matrix I and store the result in GSM_I;

[0139] Step 3: According to the size of the input matrix I, allocate a space AM_I in the AM space;

[0140] Step 4: The transposed input matrix X is transferred from the DDR space to AM_I through the DMA channel;

[0141] Step 5: The weight matrix Wq and bias data Bq are transferred from the DDR space to the SM through the DMA channel, and the SpMM+Bias calculation of the transposed input matrix I and Wq and Bq is performed, and the calculation result is transferred to the GSM_q space;

[0142] Step 6: The weight matrix Wk and bias data Bk are transferred from the DDR space to the SM through the DMA channel, and the SpMM+Bias calculation of the transposed input matrix I and Wk and Bk is performed, and the calculation result is transferred to the GSM_k space;

[0143] Step 7: The weight matrix Wv and the bias data Bv are transferred from the DDR space to the SM through the DMA channel, and the SpMM+Bias calculation of the transposed input matrix I and Wv and Bv is performed, and the calculation result is transferred to the GSM_v space;

[0144] Step 8: Continue with other operations in the encoder, and the intermediate calculation results are also saved in the GSM until the last calculation of the last layer of the encoder, and the final calculation result is transmitted to the corresponding DDR space.

[0145] This application provides a sparse BERT model acceleration method for domestic DSP processors, which performs a sparse design for matrix multiplication based on a semi-structured N:M sparse pattern, and rebuilds the BERT model based on the computational characteristics of semi-structured sparse matrix multiplication. The overall structure of the model is adjusted and optimized under the premise of computational consistency, and a series of graph optimization operations are deployed for BERT-Base model specifications and low-latency computing scenarios, including: operator fine-tuning, operator fusion, storage optimization, etc. The beneficial technical effects of this application are as follows: the DSP multi-core vector processor in the design combines its hardware characteristics to cleverly decode the semi-structured sparse model, which reduces the decoding overhead of the sparse compression model on general general-purpose platforms, and gives full play to the parallel computing capabilities of the DSP architecture based on the BERT model. As the sparsity of the model increases, it can bring considerable acceleration benefits. At the same time, operator fine-tuning in graph optimization makes rational use of the changes brought about by encoder sparsity and integrates transposition and normalization operations, which not only simplifies the redundant operations of transposition, but also fits the accelerator structure more closely to give full play to its vectorized computing capabilities; operator fusion avoids the instantiation of intermediate results, reduces the number of data transmissions, improves data reusability and reduces redundant computing operations; storage optimization makes full use of the storage structure of the accelerator and efficiently utilizes high-bandwidth storage, which improves data access efficiency overall. The above series of optimization acceleration methods bring considerable performance benefits to the operation of the BERT model on domestic accelerators.

[0146] In this embodiment, a pre-trained language model is constructed based on semi-structured sparse matrix multiplication, and the pre-trained language model is adjusted and optimized to obtain the optimized pre-trained language model; the type of graph optimization operation for the target application scenario is determined; the target application scenario is a scenario that meets the preset low latency condition; if the type is operator fine-tuning, the optimized pre-trained language model and the multi-core vector processor are used, and layer normalization and matrix transposition operations are combined to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario; if the type is operator fusion, a target operator fusion mode is determined, and based on the multi-core vector processor and according to the target operator fusion mode, the encoders of each layer in the optimized pre-trained language model are subjected to operator fusion to implement a graph optimization operation of the type of operator fusion for the target application scenario; if the type is storage optimization, the data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of the type of storage optimization for the target application scenario. This application constructs a pre-trained language model based on semi-structured sparse matrix multiplication, adjusts and optimizes the model, determines the type of graph optimization operation for the target application scenario, and if the type is operator fine-tuning, uses the optimized pre-trained language model and multi-core vector processor, and combines layer normalization and matrix transposition operations to implement the corresponding graph optimization operation, so that the redundant operations of transposition can be simplified and the vectorized computing capabilities of the multi-core vector processor can be fully utilized. If the type is operator fusion, the encoders of each layer in the optimized pre-trained language model are fused by operators to reduce the number of data transmissions, improve data reusability, and reduce redundant computing operations. If the type is storage optimization, the data storage structure of the optimized pre-trained language model is optimized and the data reuse design is performed to fully utilize the storage structure of the multi-core vector processor and improve the data storage effect.

[0147] See also Fig.12 As shown, the embodiment of the present invention discloses a sparse BERT model acceleration device for a multi-core vector processor, which may specifically include:

[0148] A construction and optimization module 11 is used to construct a pre-trained language model based on semi-structured sparse matrix multiplication, and adjust and optimize the pre-trained language model to obtain an optimized pre-trained language model;

[0149] A type determination module 12 is used to determine the type of graph optimization operation to be performed on a target application scenario; the target application scenario is a scenario that meets a preset low latency condition;

[0150] An operator fine-tuning module 13 is used to use the optimized pre-trained language model and multi-core vector processor, and combine layer normalization and matrix transposition operations to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario if the type is operator fine-tuning;

[0151] An operator fusion module 14 is used to determine a target operator fusion mode if the type is operator fusion, and perform operator fusion on encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and according to the target operator fusion mode, so as to implement a graph optimization operation of the type of operator fusion for the target application scenario;

[0152] The storage optimization module 15 is used to optimize the data storage structure and design data reuse for the optimized pre-trained language model based on the multi-core vector processor if the type is storage optimization, so as to implement a graph optimization operation of the type of storage optimization for the target application scenario.

[0153] In this embodiment, a pre-trained language model is constructed based on semi-structured sparse matrix multiplication, and the pre-trained language model is adjusted and optimized to obtain the optimized pre-trained language model; the type of graph optimization operation for the target application scenario is determined; the target application scenario is a scenario that meets the preset low latency condition; if the type is operator fine-tuning, the optimized pre-trained language model and the multi-core vector processor are used, and layer normalization and matrix transposition operations are combined to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario; if the type is operator fusion, a target operator fusion mode is determined, and based on the multi-core vector processor and according to the target operator fusion mode, the encoders of each layer in the optimized pre-trained language model are subjected to operator fusion to implement a graph optimization operation of the type of operator fusion for the target application scenario; if the type is storage optimization, the data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of the type of storage optimization for the target application scenario. This application constructs a pre-trained language model based on semi-structured sparse matrix multiplication, adjusts and optimizes the model, determines the type of graph optimization operation for the target application scenario, and if the type is operator fine-tuning, uses the optimized pre-trained language model and multi-core vector processor, and combines layer normalization and matrix transposition operations to implement the corresponding graph optimization operation, so that the redundant operations of transposition can be simplified and the vectorized computing capabilities of the multi-core vector processor can be fully utilized. If the type is operator fusion, the encoders of each layer in the optimized pre-trained language model are fused by operators to reduce the number of data transmissions, improve data reusability, and reduce redundant computing operations. If the type is storage optimization, the data storage structure of the optimized pre-trained language model is optimized and the data reuse design is performed to fully utilize the storage structure of the multi-core vector processor and improve the data storage effect.

[0154] In some specific embodiments, the construction and optimization module 11 may specifically include:

[0155] A sparse design module, used for performing sparse design on matrix multiplication using a semi-structured N:M sparse pattern to obtain a semi-structured sparse matrix multiplication;

[0156] A pre-trained language model construction module is used to construct a pre-trained language model based on the semi-structured sparse matrix multiplication; the pre-trained language model includes an embedding layer, an encoder layer of a plurality of encoder stacks with the same structure, and a pooling layer.

[0157] In some specific embodiments, the multi-core vector processor includes: a central processing unit, a digital signal computing cluster; the digital signal computing cluster includes a digital signal computing core and a global shared memory; the digital signal computing core includes a scalar storage space and a vector storage space.

[0158] In some specific embodiments, the operator fine-tuning module 13 may specifically include:

[0159] The sample matrix calculation module is used to calculate the input sample matrix by using the optimized pre-trained language model and multi-core vector processor, combining the matrix device operation and based on the layer normalization calculation formula if the type is operator fine-tuning, and switching the matrix dimension of the layer normalization calculation to obtain the output result, so as to complete the graph optimization operation of the type of operator fine-tuning for the target application scenario; the layer normalization calculation formula is:

[0160] ;

[0161] in, To output the result, is the input sample matrix, and They are The mean and standard deviation of is a constant, and are learnable scaling and shift parameters, and m is the number of elements.

[0162] In some specific embodiments, the operator fusion module 14 may specifically include:

[0163] The operator fusion mode screening module is used to screen out the target operator fusion mode from all operator fusion modes; the operator fusion modes include SpMM and Bias operator fusion, SpMM and Bias and Residual operator fusion, SpMM and Bias and GELU operator fusion, GEMM and Add_mask and Softmax operator fusion.

[0164] In some specific embodiments, the storage optimization module 15 may specifically include:

[0165] An operator screening module, used to screen out operators with data dependency from encoders of each layer in the optimized pre-trained language model;

[0166] An operator fusion module is used to fuse the operators with data dependencies in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion method, so as to implement a graph optimization operation of the operator fusion type for the target application scenario.

[0167] In some specific embodiments, the storage optimization module 15 may specifically include:

[0168] The optimization module is used to optimize the data storage structure of the encoder in the optimized pre-trained language model based on the storage structure characteristics of the multi-core vector processor if the type is storage optimization, and optimize the cache memory according to the self-attention layer in the optimized pre-trained language model to achieve a graph optimization operation of the type of storage optimization for the target application scenario.

[0169] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the sparse BERT model acceleration method for a multi-core vector processor executed by the electronic device disclosed in any of the aforementioned embodiments.

[0170] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0171] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0172] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20 to realize the operation and processing of the data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the sparse BERT model acceleration method for multi-core vector processors executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to including data transmitted from an external device received by the sparse BERT model acceleration device for multi-core vector processors, the data 223 can also include data collected by its own input and output interface 25, etc.

[0173] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0174] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the sparse BERT model acceleration method for a multi-core vector processor disclosed in any of the aforementioned embodiments are implemented.

[0175] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0176] The above is a detailed introduction to a sparse BERT model acceleration method, device, equipment and storage medium for a multi-core vector processor provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A sparse BERT model acceleration method for multi-core vector processors, characterized in that: include: Constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain an optimized pre-trained language model; Determine the type of graph optimization operation to be performed on a target application scenario; the target application scenario is a scenario that meets a preset low latency condition; If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used in combination with layer normalization and matrix transposition operations to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario; If the type is operator fusion, a target operator fusion mode is determined, and based on the multi-core vector processor and in accordance with the target operator fusion mode, the operator fusion is performed on the encoders of each layer in the optimized pre-trained language model to implement a graph optimization operation of the type of operator fusion for the target application scenario; If the type is storage optimization, data storage structure optimization and data reuse design are performed on the optimized pre-trained language model based on the multi-core vector processor to implement a graph optimization operation of storage optimization type for the target application scenario.

2. The sparse BERT model acceleration method for multi-core vector processors according to claim 1 is characterized in that: The method of constructing a pre-trained language model based on semi-structured sparse matrix multiplication includes: The semi-structured N:M sparse pattern is used to perform sparsification design on matrix multiplication to obtain semi-structured sparse matrix multiplication. A pre-trained language model is constructed based on the semi-structured sparse matrix multiplication; the pre-trained language model includes an embedding layer, an encoder layer of a plurality of encoder stacks with the same structure, and a pooling layer.

3. The sparse BERT model acceleration method for multi-core vector processors according to claim 1 is characterized in that: The multi-core vector processor comprises: a central processing unit and a digital signal computing cluster; the digital signal computing cluster comprises a digital signal computing core and a global shared memory; the digital signal computing core comprises a scalar storage space and a vector storage space.

4. The sparse BERT model acceleration method for multi-core vector processors according to claim 1 is characterized in that: If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used, and layer normalization and matrix transposition operations are combined to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario, including: If the type is operator fine-tuning, the optimized pre-trained language model and multi-core vector processor are used, combined with the matrix device operation and based on the layer normalization calculation formula to calculate the input sample matrix, and the matrix dimension of the layer normalization calculation is switched to obtain the output result, so as to complete the graph optimization operation of the target application scenario of the type of operator fine-tuning; the layer normalization calculation formula is: ； in, To output the result, is the input sample matrix, and They are The mean and standard deviation of is a constant, and are learnable scaling and shift parameters, and m is the number of elements.

5. The sparse BERT model acceleration method for multi-core vector processors according to claim 1 is characterized in that: The determining of the target operator fusion method includes: A target operator fusion method is selected from all operator fusion methods; the operator fusion methods include SpMM and Bias operator fusion, SpMM and Bias and Residual operator fusion, SpMM and Bias and GELU operator fusion, GEMM and Add_mask and Softmax operator fusion.

6. The sparse BERT model acceleration method for multi-core vector processors according to claim 1, characterized in that: The method of performing operator fusion on encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion method, so as to implement a graph optimization operation of the type of operator fusion on the target application scenario, includes: Filtering operators with data dependency from encoders of each layer in the optimized pre-trained language model; Based on the multi-core vector processor and in accordance with the target operator fusion method, the operators with data dependencies in the optimized pre-trained language model are fused to implement a graph optimization operation of the type of operator fusion for the target application scenario.

7. The sparse BERT model acceleration method for multi-core vector processors according to any one of claims 1 to 6, characterized in that: If the type is storage optimization, then based on the multi-core vector processor, data storage structure optimization and data reuse design are performed on the optimized pre-trained language model to implement a graph optimization operation of storage optimization type for the target application scenario, including: If the type is storage optimization, the data storage structure of the encoder in the optimized pre-trained language model is optimized based on the storage structure characteristics of the multi-core vector processor, and the cache memory is optimized according to the self-attention layer in the optimized pre-trained language model, so as to realize the graph optimization operation of storage optimization for the target application scenario.

8. A sparse BERT model acceleration device for multi-core vector processors, characterized in that: include: A construction and optimization module, used for constructing a pre-trained language model based on semi-structured sparse matrix multiplication, and adjusting and optimizing the pre-trained language model to obtain an optimized pre-trained language model; A type determination module, used to determine the type of graph optimization operation to be performed on the target application scenario; The target application scenario is a scenario that meets the preset low-latency condition; An operator fine-tuning module, for, if the type is operator fine-tuning, using the optimized pre-trained language model and multi-core vector processor, and combining layer normalization and matrix transposition operations, to implement a graph optimization operation of the type of operator fine-tuning for the target application scenario; An operator fusion module, for determining a target operator fusion mode if the type is operator fusion, and performing operator fusion on encoders of each layer in the optimized pre-trained language model based on the multi-core vector processor and in accordance with the target operator fusion mode, so as to implement a graph optimization operation of the type of operator fusion for the target application scenario; A storage optimization module is used to optimize the data storage structure and design data reuse for the optimized pre-trained language model based on the multi-core vector processor if the type is storage optimization, so as to implement a graph optimization operation of the type of storage optimization for the target application scenario.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the sparse BERT model acceleration method for a multi-core vector processor as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the sparse BERT model acceleration method for a multi-core vector processor as described in any one of claims 1 to 7 is implemented.

Images