A self-attention mechanism neural network acceleration method, device and medium

Abstract

This application provides a method, device, and medium for accelerating a self-attention mechanism neural network, relating to the field of artificial intelligence model optimization technology. The method includes: constructing a Transformer network structure and initializing the embedding layer and positional encoding layer to obtain an initialized Transformer network; training and parameter fixing based on training data; and accelerating inference of the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships. This application addresses the technical problem in existing technologies where the high computational complexity and large memory consumption of Transformers lead to low inference efficiency and difficulty in real-time deployment in embedded environments, further affecting operational efficiency. By reconstructing the Transformer network, efficient deployment in embedded environments is achieved, improving operational efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model optimization technology, specifically to a method, device, and medium for accelerating a self-attention mechanism neural network. Background Technology

[0002] The Transformer model, due to its powerful sequence modeling capabilities and parallelization advantages, has become the mainstream architecture for various deep learning tasks. However, it suffers from high computational complexity and large memory consumption, facing severe performance bottlenecks, especially when deployed on embedded devices. Current methods typically reduce model size through pruning and quantization, but these methods sacrifice model expressiveness and fail to optimize Transformer's operational efficiency at the architectural level. Some methods have attempted to improve inference speed through structural simplification or the design of dedicated hardware, but the former sacrifices accuracy, and the latter relies on specific chip platforms, resulting in poor versatility.

[0003] In summary, existing technologies suffer from the technical problem that the high computational complexity and large memory consumption of Transformer lead to low inference efficiency in embedded environments, making real-time deployment difficult and further affecting operational efficiency. Summary of the Invention

[0004] This application provides a method, device, and medium for accelerating neural networks using a self-attention mechanism, which addresses the technical problem in the prior art where the high computational complexity and large memory consumption of Transformer lead to low inference efficiency in embedded environments, making real-time deployment difficult and further affecting operational efficiency.

[0005] In view of the above problems, this application provides a method, device and medium for accelerating a self-attention mechanism neural network.

[0006] In a first aspect, this application provides a method for accelerating a self-attention mechanism neural network. This method is implemented using a self-attention mechanism neural network acceleration device. The method includes: constructing a Transformer network structure comprising a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections; initializing the embedding layer and the positional encoding layer to obtain an initialized Transformer network; training and parameter fixing of the initialized Transformer network based on training data to obtain a trained Transformer network; and accelerating inference of the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships.

[0007] Optionally, in the multi-head self-attention layer, a sparse attention mechanism is introduced to replace the traditional multi-head attention module with a sparse attention module. A sparse mask matrix is ​​constructed based on the activation information of each token in the input sequence to limit the scope of attention calculation. In the sparse attention module, the attention weight matrix is ​​divided into blocks, and a low-rank decomposition interface is reserved for each sub-block to obtain a block-based low-rank attention matrix. A channel attention mechanism is introduced in the feedforward neural network, and a learnable channel importance parameter is set to control the channel output of the feedforward neural network. An inter-layer shared projection matrix mechanism is introduced to establish a projection matrix sharing relationship between multiple Transformer network layers, so that at least two Transformer network layers share the query matrix, key matrix, and value matrix. A hybrid positional encoding is constructed in the positional encoding layer and added to the embedding layer to obtain the initialized Transformer network.

[0008] Optionally, the attention weights of the sparse attention module are: ;in, Let M represent the Hadamard product, and SparseAttn(Q,K,V) be the attention weights, where Q is the query matrix, K is the key matrix, V is the value matrix, and M is the sparse mask matrix. This is the scaling factor.

[0009] Optionally, the attention weight matrix is ​​divided into blocks to obtain multiple sub-blocks; the singular value decomposition function is used to traverse the multiple sub-blocks to perform singular value decomposition, thereby obtaining a block-based low-rank attention matrix.

[0010] Optionally, the singular value decomposition function is: Among them, A ij Attention weight matrix The i-th and j-th sub-blocks, where n is a positive integer, U i For A ij The left singular vector matrix obtained by singular value decomposition, For A ij The singular value diagonal matrix obtained by singular value decomposition, For A ij The transpose of the right singular vector matrix obtained by singular value decomposition.

[0011] The block-based low-rank attention matrix is ​​as follows: ;in, V is the approximate attention matrix composed of the low-rank approximation results of all sub-blocks, and V is the value matrix in the self-attention mechanism. j To be with sub-block A ij The value submatrix corresponding to the column direction.

[0012] Optionally, initialize a sine or cosine position encoding matrix; introduce a learnable scalar parameter α, where α is automatically optimized during training via gradient descent; obtain a hybrid encoding formula, wherein the hybrid encoding formula is: .

[0013] .

[0014] in, The values ​​of the hybrid position encoding matrix at position pos and the 2i-th dimension are given. Let d be the value of the hybrid positional encoding matrix at position pos and the (2i+1)th dimension, where pos is the position index of the token in the input sequence and d is the total number of dimensions of the positional encoding vector.

[0015] Optionally, based on the channel attention mechanism, a set of learnable channel importance parameters is set, corresponding one-to-one with the channels of the feedforward neural network. During training, the set of channel importance parameters participates in the forward propagation calculation and is updated synchronously with the network weight parameters during the backpropagation process. During the model inference stage, the channels in the feedforward neural network are sorted according to the absolute value of the set of channel importance parameters, and the top k% of the feature channels in the sorting result are retained to obtain a set of retained feature channels, where k is a positive integer. Based on the set of retained feature channels, the corresponding weight matrix in the feedforward neural network is pruned according to the channel dimension.

[0016] Optionally, the channel importance parameter is used to characterize the importance of each feature channel.

[0017] Secondly, this application also provides a self-attention mechanism neural network acceleration device for executing a self-attention mechanism neural network acceleration method as described in the first aspect, wherein the self-attention mechanism neural network acceleration device includes: an architecture building module for constructing a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initializing an embedding layer and a position encoding layer to obtain an initialized Transformer network; a parameter fixing module for training and fixing the parameters of the initialized Transformer network based on training data to obtain a trained Transformer network; and an inference acceleration module for accelerating inference of the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships.

[0018] Thirdly, a computer-readable storage medium storing a computer program that, when executed, implements the steps of the self-attention mechanism neural network acceleration method described in any one of the first aspects above.

[0019] One or more technical solutions provided in this application have at least the following beneficial effects: An initialized Transformer network is obtained by constructing a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initializing the embedding layer and positional encoding layer; the initialized Transformer network is trained and its parameters are fixed based on training data to obtain a trained Transformer network; and inference is accelerated on the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships. In other words, by reconstructing and training the Transformer network, and based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships, the trained model is accelerated, significantly reducing the computational complexity, memory usage, and inference latency of the Transformer model without sacrificing model accuracy, achieving efficient deployment in embedded environments, and improving operational efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a self-attention mechanism neural network acceleration method according to this application.

[0022] Figure 2 This is a schematic diagram of the structure of a self-attention mechanism neural network acceleration device according to this application.

[0023] Figure labeling: Architecture building module 11, parameter fixing module 12, inference acceleration module 13. Detailed Implementation

[0024] This application provides a method, device, and medium for accelerating self-attention mechanism neural networks, addressing the technical problem in existing technologies where the high computational complexity and large memory footprint of Transformers lead to low inference efficiency in embedded environments, difficulty in real-time deployment, and further impact on operational efficiency. By reconstructing and training the Transformer network, based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships, the trained model is accelerated. This significantly reduces the computational complexity, memory footprint, and inference latency of the Transformer model without sacrificing model accuracy, enabling efficient deployment in embedded environments and improving operational efficiency.

[0025] Example 1, as Figure 1 As shown, this application provides a method for accelerating a self-attention mechanism neural network, wherein the method is applied to a self-attention mechanism neural network acceleration device, and the method specifically includes the following steps: A Transformer network structure consisting of a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections is constructed, and the embedding layer and position encoding layer are initialized to obtain the initialized Transformer network.

[0026] Furthermore, this application also includes the following steps: In the multi-head self-attention layer, a sparse attention mechanism is introduced to replace the traditional multi-head attention module with a sparse attention module, and a sparse mask matrix is ​​constructed based on the activation information of each token in the input sequence to limit the scope of attention calculation; In the sparse attention module, the attention weight matrix is ​​divided into blocks, and a low-rank decomposition interface is reserved for each sub-block to obtain a block-based low-rank attention matrix; A channel attention mechanism is introduced in the feedforward neural network, and a learnable channel importance parameter is set to control the channel output of the feedforward neural network; An inter-layer shared projection matrix mechanism is introduced to establish a projection matrix sharing relationship between multiple Transformer network layers, so that at least two Transformer network layers share the query matrix, key matrix, and value matrix; A hybrid positional encoding is constructed in the positional encoding layer, and the hybrid positional encoding is added to the embedding layer to obtain the initialized Transformer network.

[0027] Specifically, the Transformer network structure is constructed, including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections. The multi-head self-attention layer allows each element in the sequence to simultaneously focus on information from all other elements in the sequence, and captures the complex relationships between elements from different perspectives through multiple sets of parallel attention heads.

[0028] Feedforward neural network layers are used to perform nonlinear transformations and refinements on the features output by the self-attention layer, enhancing the model's representational power. The Transformer's feedforward neural network (FFN) consists of two linear transformations plus activation functions: Here, the intermediate dimension is typically set to 4 times the input dimension; the feedforward neural network is a function representation of the feedforward neural network, with input x and output the new feature representation after transformation by this layer; W1 is the weight matrix of the first linear transformation layer, used to project the input features from the dimension of the input features to a higher intermediate dimension; b1 is the bias vector corresponding to the first linear transformation layer, with a length equal to the intermediate dimension; GELU is an activation function that performs a non-linear mapping on the high-dimensional features output by the first linear layer; W2 is the weight matrix of the second linear transformation layer, which reprojects the non-linearly activated high-dimensional features (intermediate dimension) back to the original model dimension (input dimension); b2 is the bias vector corresponding to the second linear transformation layer, with a length equal to the input dimension.

[0029] Layer normalization layers are used to standardize the outputs of neurons in a single layer of a neural network, adjusting the mean and variance of all outputs of that layer to a stable range, thereby accelerating model training convergence and improving training stability. Residual connections provide a shortcut path where the input to a layer is directly added to its output, effectively alleviating the gradient vanishing problem in deep networks and making it possible to build deep networks with dozens or even hundreds of layers.

[0030] The multi-head self-attention layer allows all elements within the sequence to interact globally. The output of this layer is not directly passed; instead, it is first added to its own input via residual connections, preserving the original information. The result of this addition is immediately fed into the layer normalization module for standardization, yielding stable features. Next, these features enter the feedforward neural network layer, where in-depth nonlinear transformations are performed on the features at each position. Similarly, the output of the feedforward neural network layer is also added to its input (i.e., the layer-normalized features) via residual connections, and then undergoes another layer normalization.

[0031] Set the model hyperparameters, and determine the number of layers L (usually 6-12 layers), the number of attention heads h (usually 8-16 heads), the feature dimension d (usually 512 or 768), and the feedforward network dimension d. ff (Typically 4D). Construct the basic framework of a standard Transformer encoder / decoder, including a multi-head attention layer, a feedforward neural network layer, a layer normalization module, and residual connections. Initialize the embedding layer, mapping the input tokens to a d-dimensional vector space, with the weight matrix as follows. Among them, W e`mb` is the weight matrix of the embedding layer, a learnable parameter matrix, and `V` is the vocabulary size, i.e., the number of all unique tokens the model can recognize. The positional encoding layer is initialized to reserve an interface for subsequent hybrid positional encoding.

[0032] A sparse attention mechanism is introduced into the multi-head self-attention layer, replacing the traditional multi-head attention module with a sparse attention module. For a given input sequence, the activation information of each token is computed. For each target token for which attention is to be computed, the module employs a hybrid strategy to select the source tokens it should focus on: first, global importance filtering, selecting the K tokens with the strongest activation information in the entire sequence (Top-K); second, local context preservation, retaining all tokens within a fixed window (e.g., W tokens before and after) adjacent to the target token. The selected positions are marked as 1 in the mask matrix, and the rest are marked as 0. This generates an input-dependent, dynamic mask that significantly reduces the number of connections for which attention weights need to be computed for each token.

[0033] In the sparse attention module, a block low-rank approximation strategy is introduced to divide the matrix into multiple smaller sub-matrices. Then, a low-rank decomposition interface is provided for each sub-block, which is to approximate it as the product of several smaller matrices to obtain the block low-rank attention matrix, simulating the operation of the original matrix with fewer parameters and less computation.

[0034] A channel attention mechanism is introduced into the feedforward neural network, setting a learnable channel importance parameter, initially set to 1. This parameter is updated synchronously during backpropagation. During inference, the channels are sorted according to the absolute values ​​of the synchronously updated importance parameters, retaining the highest-ranking channels; the corresponding weight matrix is ​​also pruned column-by-column, significantly reducing the computational cost of the feedforward neural network with almost no loss of accuracy.

[0035] A mechanism for sharing projection matrices between layers is introduced, where the same projection matrix is ​​shared between certain layers, thereby reducing the total number of parameters and computational load. Formally, if layer l shares W with layer l+1... Q W K W V ,but , This sharing method reduces redundant parameter duplication, making it particularly suitable for shallow networks or scenarios where consecutive layers have similar features. The inter-layer shared projection matrix mechanism reduces the total number of model parameters and memory access overhead during computation by allowing multiple Transformer network layers to share the same parameter matrix. In the standard Transformer, each layer has its own independent query, key, and value projection matrix; however, under the inter-layer shared projection matrix mechanism, some layers will share the same set of matrices. Instead of independently initializing the Q, K, and V matrices for each layer, only one set of matrices is initialized for each shared group. A mapping table is also established from layer indexes to shared group IDs. During model runtime, when the data flow reaches a certain layer (e.g., layer l) and self-attention needs to be calculated, that layer no longer uses its own stored projection matrix. Instead, it queries the corresponding shared group ID according to the mapping table and retrieves the Q, K, and V matrices of that group from the shared parameter pool for the projection calculation of the current layer.

[0036] A hybrid positional encoding mechanism is used in the positional encoding layer, combining the advantages of sine and cosine positional encoding. A learnable α parameter is used to scale and fuse the traditional sine-cosine encoding. The embedded representation enhanced by hybrid positional encoding is then input into the network body, which integrates various optimization techniques, to obtain the initialized Transformer network. Sparse attention and low-rank decomposition target the computationally intensive attention module; channel pruning optimizes the feedforward neural network module, which has both a large number of parameters and computational cost; parameter sharing directly reduces storage and memory bandwidth usage. Multiple techniques work together to alleviate the deployment burden from different dimensions.

[0037] Furthermore, this application also includes the following steps: The attention weights of the sparse attention module are: ;in, Let M represent the Hadamard product, and SparseAttn(Q,K,V) be the attention weights, where Q is the query matrix, K is the key matrix, V is the value matrix, and M is the sparse mask matrix. This is the scaling factor.

[0038] Specifically, the computational complexity of the multi-head attention mechanism in the traditional Transformer is O(n^2). 2 ·d), where, The length of the input sequence is a positive integer, and d is the feature dimension. A dynamic sparse attention mechanism is introduced to reduce the redundancy of global attention.

[0039] Let the original input sequence be... Where d is the feature dimension. The traditional query, key, and value projection matrices are Q = XW. Q K=XWK V=XW V Then the attention output is .

[0040] Define a sparse mask matrix This is used to dynamically select key context positions. It is then applied to the attention weights. ,in, This represents the Hadamard product. The sparse mask M is generated as follows: For each token, the top k relevant positions are selected based on the activation values ​​of the previous layer; a sliding window is used to limit the attention within a local range; the sparsity of the mask can be dynamically adjusted with the layer depth, significantly reducing invalid interactions in attention computation.

[0041] Define the sparsity parameter, set the number of Top-k selections k, and initialize it to a value of 0. Between n / 4, the sliding window size w is recommended to be 16-64. The traditional Multi-Head-Attention module is replaced with a SparseAttention module, retaining the original Q, K, V projections, but introducing a sparse mask when calculating the attention weights. The mask generator is implemented by calculating the activation energy of each token. , where h i-1 E is in the hidden state of the previous layer. i It activates energy; for each position i, select the k positions with the highest energy. The `argsort(E)` function returns an ascending list of indices of an energy array `E` (containing the energy values ​​of all tokens in the sequence). `[-k:]` indicates retrieving the last `k` elements of this list, representing the indices of the `k` tokens with the highest energy values; it also preserves the positions within the sliding window. , is a continuous integer interval centered at the current target token position i, extending w positions before and after it, representing the local context positions that must be retained; generate a binary mask matrix. ,if If M[i,j]=1, it means that M[i,j] is 1 only when position j simultaneously satisfies one of the following two conditions: the global Top-k related position of the i-th token or the position within the local sliding window, allowing attention calculation; otherwise, it is 0 and is masked. Configure the sparsity decay strategy for each layer: use higher sparsity (smaller k) for shallow layers (layers 1-3), and gradually reduce sparsity (k gradually increases) for deeper layers (layer 4 and beyond) to balance computational efficiency and expressive power.

[0042] By introducing a sparse attention mechanism, the redundancy of global attention is reduced.

[0043] Furthermore, this application also includes the following steps: dividing the attention weight matrix into blocks to obtain multiple sub-blocks; using a singular value decomposition function, traversing the multiple sub-blocks to perform singular value decomposition to obtain a block-based low-rank attention matrix.

[0044] Furthermore, this application also includes the following step: the singular value decomposition function is: Among them, A ij Attention weight matrix The i-th and j-th sub-blocks, where n is a positive integer, U i For A ij The left singular vector matrix obtained by singular value decomposition, For A ij The singular value diagonal matrix obtained by singular value decomposition, For A ij The transpose of the right singular vector matrix obtained by singular value decomposition; the block-based low-rank attention matrix is: ;in, V is the approximate attention matrix composed of the low-rank approximation results of all sub-blocks, and V is the value matrix in the self-attention mechanism. j To be with sub-block A ij The value submatrix corresponding to the column direction.

[0045] Specifically, singular value decomposition (SVD) is a powerful matrix factorization method that decomposes any matrix into the product of three specific matrices: the left singular vector matrix U, the singular value diagonal matrix Σ, and the transpose of the right singular vector matrix V. By retaining only the largest few singular values ​​and their corresponding vectors, a high-quality low-rank approximation of the original matrix can be obtained. Attention matrix The original attention matrix exhibits certain structural characteristics, and a block-based low-rank approximation method is used for compression. The original attention matrix is ​​divided into multiple sub-blocks A. ij Perform singular value decomposition on each sub-block. Among them, A ij Attention weight matrix The i-th and j-th sub-blocks, where n is a positive integer; U i For A ij The left singular vector matrix obtained by singular value decomposition, , i For A ij The singular value diagonal matrix obtained by singular value decomposition; , For A ijThe transpose of the right singular vector matrix obtained by singular value decomposition. Here, r << p, q, where p is the number of rows of a single sub-block matrix after the original attention matrix is partitioned, q is the number of columns of a single sub-block matrix after the original attention matrix is partitioned, and r is the number of singular values retained for compressing the matrix during singular value decomposition, also known as the rank, which determines the accuracy and compression rate of the low-rank approximation. The larger the retained r, the more accurate the approximation, but the weaker the compression effect; the smaller r, the higher the compression rate, but more information may be lost.

[0046] The block low-rank attention matrix is an approximate representation of the entire attention weight matrix formed by partitioning the attention weight matrix, independently performing low-rank approximation on each sub-block, and then combining all the approximated sub-blocks in their original positions.

[0047] The block low-rank attention matrix is: ; where is the approximate attention matrix formed by combining the low-rank approximation results of all sub-blocks, V is the value matrix in the self-attention mechanism, and V j is the value sub-matrix corresponding to sub-block A ij in the column direction. By retaining the first several principal components, the computational cost of matrix multiplication is effectively reduced while maintaining high representational ability.

[0048] Furthermore, this application also includes the following steps: initializing a sine or cosine position encoding matrix; introducing a learnable scalar parameter α, where α is automatically optimized through gradient descent during training; obtaining a hybrid encoding formula, where the hybrid encoding formula is: .

[0049] .

[0050] where is the value of the hybrid position encoding matrix at position pos and the 2i-th dimension, is the value of the hybrid position encoding matrix at position pos and the 2i + 1-th dimension, pos is the position index of the token in the input sequence, and d is the total dimension number of the position encoding vector.

[0051] Specifically, the sine / cosine positional encoding matrix is ​​a fixed-positional encoding method that uses a set of numerical matrices based on sine and cosine functions to generate a unique encoded value for each feature dimension at each position in the sequence. During model construction, a standard sine-cosine base positional encoding matrix is ​​pre-computed based on the preset maximum sequence length and model hidden dimensions. Even-numbered dimensions of this matrix are generated using sine functions, and odd-numbered dimensions using cosine functions, with the same formula as the traditional Transformer. Simultaneously, the model creates a trainable parameter called a learnable scalar parameter α, typically initialized to 1.0, which is automatically optimized during training using gradient descent. The learnable scalar parameter α serves as a regular parameter of the model, participating in every forward and backward propagation.

[0052] During forward propagation, the model uses the current data in real time. Value, according to the mixed coding formula , The basic sine and cosine components are recombine to generate the final positional encoding used in the current iteration. This final positional encoding is added to the word embedding vectors and fed into the network. During backpropagation, the gradient of α is calculated based on the model's total loss (e.g., prediction error) and updated by the optimizer Adam. The entire process is data-driven; the model automatically learns the most suitable positional encoding mixture for the current task. During inference, the α value after training convergence is fixed as a constant. The model uses this optimal α along with the fixed sine function to generate the final mixed positional encoding for any input sequence. The values ​​of the hybrid position encoding matrix at position pos and the 2i-th dimension are given. Let d be the value of the hybrid positional encoding matrix at position pos and the (2i+1)th dimension, where pos is the position index of the token in the input sequence and d is the total number of dimensions of the positional encoding vector.

[0053] After constructing the hybrid positional encoding, the optimized Transformer model structure is initialized. The model includes five core modules: dynamic sparse attention, low-rank decomposition reserved interface, adaptive channel pruning, inter-layer parameter sharing, and hybrid positional encoding.

[0054] Furthermore, this application also includes the following steps: based on the channel attention mechanism, a set of learnable channel importance parameters is set, corresponding one-to-one with the channels of the feedforward neural network; during training, the set of channel importance parameters participates in the forward propagation calculation and is updated synchronously with the network weight parameters during the backpropagation process; during the model inference stage, the channels in the feedforward neural network are sorted according to the absolute value of the set of channel importance parameters, and the top k% of the feature channels in the sorting result are retained to obtain a set of retained feature channels, where k is a positive integer; based on the set of retained feature channels, the corresponding weight matrix in the feedforward neural network is pruned according to the channel dimension.

[0055] Furthermore, this application also includes the following steps: the channel importance parameter is used to characterize the importance of each feature channel.

[0056] Specifically, during model initialization, a channel importance parameter 'a' is introduced for each feedforward neural network channel. Initialize it as a vector of all 1s, i.e., a i =1, For a feedforward neural network with a hidden dimension of d and an intermediate dimension of 4*d, a channel attention module is inserted after its first linear transformation layer and before the activation function. This module contains a learnable vector of length 4*d, which is usually initialized as a vector of all 1s, indicating that all channels are equally important initially.

[0057] During model training, channel importance parameters participate in forward propagation and are optimized. A standard feedforward neural network is... , modified to ,in This represents element-wise multiplication, that is, multiplying each channel after GELU activation by its importance coefficient α. i During backpropagation, the gradient of 'a' is calculated from the partial derivative of the loss function with respect to 'a', and is updated by the optimizer along with the network weights. To make 'a' more sparse, i.e., to make the coefficients of unimportant channels close to 0, an L1 regularization term with respect to 'a' is usually added to the loss function. , where λ is the regularization coefficient, set to 0.0001-0.001, to make the vector a sparse.

[0058] After training and before inference deployment, channel pruning based on importance ranking is performed. For each feedforward neural network channel, its channel importance vector 'a' after training convergence is extracted, and the absolute value 'a' of each channel importance score is calculated. i According to a i Sort all channels from largest to smallest. Based on a preset retention rate of k%, such as 50%, select the top k% of channel indices to form the set of retained feature channels.

[0059] Based on the set of retained feature channels, the corresponding weight matrices in the feedforward neural network are pruned according to the channel dimension. For example, the W1 matrix is ​​pruned by columns (output channel dimension), and the W2 matrix is ​​pruned by rows (input channel dimension), while the bias vector b1 is also pruned accordingly. By incorporating channel importance evaluation into the training process and guiding sparsity with L1 regularization, the model can automatically learn to distinguish the importance of channels. Retaining only key channels during inference significantly reduces parameters and computational cost while keeping the loss of task accuracy within a very small range.

[0060] The initialized Transformer network is trained and its parameters are fixed based on the training data to obtain the trained Transformer network.

[0061] Specifically, select an appropriate dataset based on the application scenario; for example, use large-scale text corpora for natural language processing tasks and speech datasets for speech recognition tasks. Segment the text, construct a vocabulary, and unify the sequences to a length of n.

[0062] Set training hyperparameters: Initial learning rate is 1e. 4 The number of steps is 10% of the total steps, followed by cosine decay. The batch size is adjusted based on GPU / CPU memory, typically 16-64. The optimizer used is AdamW, with β1=0.9, β2=0.999, and weight_decay=0.01. The training epochs are 50-100, continuing until the validation set loss no longer decreases. A loss function is defined; for language modeling tasks, cross-entropy loss is used, along with an L1 regularization term for channel importance. Input data in batches, calculate the loss using forward propagation, and update parameters, including model weights and channel importance vectors, using backpropagation.

[0063] An L1 regularization term is added to the loss function of each feedforward neural network layer to cause the channel importance vectors of unimportant channels to tend towards 0. During backpropagation, the channel importance vectors are updated synchronously, and the importance distribution of each channel is monitored: every few epochs, the distribution of the α vectors in each layer is statistically recorded, and histograms are plotted to observe the degree of sparsity. The mean, variance, and proportion of non-zero elements of each layer's vectors are saved for subsequent pruning decisions.

[0064] Evaluate model performance on the validation set for different sparsities k: keeping other parameters constant, at k... A grid search is performed on discrete values ​​such as {16, 32, 64, 128, 256}. For each k value, the validation set accuracy and inference latency are recorded. The k value that satisfies the condition of accuracy loss being less than a threshold (e.g., 1%) and minimum latency is found. Layer sparsity is set, with the k value gradually increasing in deeper layers, such as: k=32 for layers 1-2 (high sparsity); k=64 for layers 3-4 (medium sparsity); and k=128 for layers 5-6 (low sparsity).

[0065] Record the optimal sparse configuration for each layer in a configuration file and save it in JSON format.

[0066] Collect statistical information of the attention matrices of each layer during training. In the last few epochs of training, periodically sample several batches and record the attention weight matrix of each layer. Perform singular value decomposition on the collected attention matrices, calculate singular values, plot the arrangement of singular values ​​from largest to smallest, and observe the energy distribution. Set an energy retention threshold, such as retaining 95% of the energy, i.e., find the minimum rank. Determine the rank of each layer based on the threshold. Different layers may have different optimal rank; shallower layers with simpler information can use smaller rank, while deeper layers with richer semantics may require larger rank. Record the optimal rank of each layer to a configuration file.

[0067] Analyze the similarity of projection matrices between adjacent layers: Calculate cosine similarity, and set a similarity threshold θ, such as θ=0.85. If the cosine similarity is greater than the similarity threshold θ, the two layers are considered to share parameters. Based on the similarity matrix, use a clustering algorithm to group similar layers together. Record the grouping results in a configuration file, which will not be changed in subsequent training. Multiple layers in the same group are treated as using the same set of parameters, and the gradients of each layer are accumulated when calculating gradients.

[0068] Learnable scalar parameters are used as model parameters during training: PE is calculated using learnable scalar parameters in each forward propagation. hybridDuring backpropagation, the learnable scalar parameters are updated. The change curves of the learnable scalar parameters during training are recorded, and their stability is observed. If the task emphasizes local information, such as text classification, the learnable scalar parameters tend to decrease (0.5-0.8), reducing the weight of the cosine term; if the task requires long-range dependencies, such as machine translation, the learnable scalar parameters tend to increase (1.2-2.0), enhancing position awareness. After training, the learnable scalar parameter values ​​are fixed, and the converged learnable scalar parameter values ​​are saved to the configuration file. After training is complete, the model's performance on the test set is evaluated, calculating task-related metrics such as accuracy, F1 score, and BLEU score. All model parameters are frozen, and the optimization configuration file is exported. The final values ​​of the channel importance vectors are saved, and a complete channel importance vector is saved for each layer for pruning during the inference phase. The complete model weight file is saved, including the parameter matrices of all layers and the optimized configuration. Accuracy and inference latency are verified on an independent test set to ensure that the design goals are met. The model training and tuning phases are completed, resulting in a trained Transformer network.

[0069] Based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships, the inference speed of the trained Transformer network is accelerated.

[0070] Specifically, the pre-trained model weights are loaded from the checkpoint file. The optimization configuration file is read, the JSON file is parsed, and information such as sparsity k, rank r, pruning ratio p, channel α value, and shared mapping is obtained. The mapping relationship of the shared projection matrix is ​​loaded, and a mapping table from layer number to parameter group is established. The inference engine is initialized, the model is set to inference mode, gradient calculation is disabled, and training-related modules are shut down. The runtime environment is configured, selecting a suitable backend based on the deployment platform (CPU / GPU / embedded device), and setting the number of threads, memory allocation strategy, etc.

[0071] Based on the saved channel importance vectors, sort each layer of the feedforward neural network: calculate the absolute value of the channel importance vectors and sort them from largest to smallest. Select the top k% of channel indices with the largest channel importance vectors: prune the first-layer weight matrix W1 by column and the second-layer weight matrix W2 by row. Prune the bias vector b1. Replace the original parameters with the pruned parameters, removing redundant parameters to free up memory. Verify the correctness of the pruned calculations by testing on a small batch of data to ensure the correct output dimensions and values.

[0072] For each attention head, the pre-computed SVD decomposition results are applied, and the corresponding layer's U is read from the configuration file. V Matrix. Replace the original path with an efficient computational path based on low-rank decomposition. Optimize the computation order: First step: Calculate V. ·V, complexity O(r·n·d); Second step: Calculate Σ·V ·V, complexity O(r) 2 ·d); Step 3: Calculate the result U, with a complexity of O(n·r·d); Total complexity: O(n·r·d), where r is much smaller than n, and much smaller than the original O(n 2 ·d). Pre-compute and cache the decomposition matrix: Before the first inference, calculate and cache the U values ​​of all layers. V Load the data into memory / video memory to avoid redundant calculations. Apply low-rank decomposition independently to each head, and finally concatenate the outputs of all heads.

[0073] For the input sequence X, calculate the activation energy of each token. For each position i, select the k positions with the highest energy. Simultaneously, apply a sliding window constraint, retaining positions within the range [iw, i+w], where w is the sliding window size, determining the local context range that each token must focus on. Generate a sparse mask and apply it when calculating the attention score, only calculating the positions M[i,j]=1 and skipping elements M[i,j]=0, using a sparse matrix operation library for acceleration. That is, when calculating attention, only the attention scores of positions marked as 1 in the mask M are calculated. For these computationally required parts, efficient computation is performed using a pre-loaded low-rank factorization matrix, completely skipping a large number of invalid dense matrix calculations.

[0074] The shared mapping table is queried based on layer number l, the corresponding projection matrix is ​​loaded from the shared parameter pool, and the projection is calculated using the shared parameters. For multiple layers within the same group, parameters are loaded into memory / cache only once, reducing memory bandwidth usage. On embedded devices, reading parameters from external storage is a bottleneck; the sharing mechanism significantly reduces I / O operations.

[0075] The input sequence is processed through an embedding layer of dimension n×d. Hybrid positional encoding is added. An optimized Transformer is executed layer by layer. For each layer, multi-head sparse attention, residual connections and layer normalization, pruned feedforward neural network, residual connections and layer normalization are applied. Loading is based on a shared configuration. The process involves calculating Q, K, and V to generate a sparse mask M (based on Top-k and a sliding window). Low-rank decomposition is applied to compute attention, and the attention output is calculated. Multiple heads are concatenated and projected, followed by residual connections and layer normalization. Using pruned W1, W2, and b1, a pruned feedforward neural network is computed, with residual connections and layer normalization performed. The final hidden state sequence is output, with dimensions n×d.

[0076] Post-processing is performed according to the task type: for classification tasks, the vector corresponding to the token is taken, and the class probability is obtained through the classification head; for generation tasks, the next token is predicted through the language model head; and for sequence labeling, the vector at each position is classified.

[0077] Measure the time of a single forward propagation using a high-precision timer. Monitor memory usage, including peak GPU and peak RAM usage (CPU). Count floating-point operations (FLOPs) and analyze the model's computational complexity using tools such as fvcore and thop. Calculate the computational speedup, aiming for 2-3 times. Evaluate the accuracy loss, requiring it to be <1.5%. If the accuracy loss exceeds the threshold, adjust the sparsity k, rank r, or pruning ratio p, and retrain and fine-tune. Export the optimized model to a mobile format and validate its performance on a real embedded device.

[0078] The inference phase optimization is complete, achieving efficient Transformer inference acceleration in resource-constrained environments. Through the combined effect of multiple technologies, inference acceleration and significant memory reduction are achieved on embedded devices, making it feasible to deploy large Transformer models that were previously difficult to deploy.

[0079] In summary, the self-attention mechanism neural network acceleration method provided in this application has the following beneficial effects: By constructing a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initializing the embedding layer and positional encoding layer, an initialized Transformer network is obtained. The initialized Transformer network is then trained and its parameters are fixed based on training data to obtain a trained Transformer network. Inference is accelerated using channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships. In other words, by reconstructing and training the Transformer network, and leveraging channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships, the trained model is accelerated. This significantly reduces the computational complexity, memory usage, and inference latency of the Transformer model without sacrificing model accuracy, enabling efficient deployment in embedded environments and improving operational efficiency.

[0080] Example 2: Based on the same inventive concept as the self-attention mechanism neural network acceleration method in Example 1, this application also provides a self-attention mechanism neural network acceleration device. Please refer to the appendix. Figure 2 The self-attention mechanism neural network acceleration device includes: Architecture building module 11 is used to build a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initialize the embedding layer and the position encoding layer to obtain an initialized Transformer network; parameter solidification module 12 is used to train and solidify the parameters of the initialized Transformer network based on training data to obtain a trained Transformer network; inference acceleration module 13 is used to accelerate the inference of the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships.

[0081] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used for: introducing a sparse attention mechanism in the multi-head self-attention layer to replace the traditional multi-head attention module with a sparse attention module, constructing a sparse mask matrix based on the activation information of each token in the input sequence to limit the attention calculation range; dividing the attention weight matrix into blocks in the sparse attention module and reserving a low-rank decomposition interface for each sub-block to obtain a block-based low-rank attention matrix; introducing a channel attention mechanism in the feedforward neural network, setting a learnable channel importance parameter to control the channel output of the feedforward neural network; introducing an inter-layer shared projection matrix mechanism to establish a projection matrix sharing relationship between multiple Transformer network layers, so that at least two Transformer network layers share the query matrix, key matrix, and value matrix; constructing a hybrid positional encoding in the positional encoding layer, adding the hybrid positional encoding to the embedding layer to obtain an initialized Transformer network.

[0082] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used to: the attention weights of the sparse attention module are: ;in, Let M represent the Hadamard product, and SparseAttn(Q,K,V) be the attention weights, where Q is the query matrix, K is the key matrix, V is the value matrix, and M is the sparse mask matrix. This is the scaling factor.

[0083] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used to: divide the attention weight matrix into blocks to obtain multiple sub-blocks; and use a singular value decomposition function to traverse the multiple sub-blocks to perform singular value decomposition to obtain a block-based low-rank attention matrix.

[0084] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used for: the singular value decomposition function being: Among them, A ij Attention weight matrix The i-th and j-th sub-blocks, where n is a positive integer, U i For A ij The left singular vector matrix obtained by singular value decomposition, For A ij The singular value diagonal matrix obtained by singular value decomposition, For A ij The transpose of the right singular vector matrix obtained by singular value decomposition.

[0085] The block-based low-rank attention matrix is ​​as follows: ;in, V is the approximate attention matrix composed of the low-rank approximation results of all sub-blocks, and V is the value matrix in the self-attention mechanism. j To be with sub-block A ij The value submatrix corresponding to the column direction.

[0086] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used for: initializing a sine or cosine position encoding matrix; introducing a learnable scalar parameter α, wherein α is automatically optimized through gradient descent during training; and obtaining a hybrid encoding formula, wherein the hybrid encoding formula is: .

[0087] .

[0088] in, The values ​​of the hybrid position encoding matrix at position pos and the 2i-th dimension are given. Let d be the value of the hybrid positional encoding matrix at position pos and the (2i+1)th dimension, where pos is the position index of the token in the input sequence and d is the total number of dimensions of the positional encoding vector.

[0089] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used for: setting a set of learnable channel importance parameters that correspond one-to-one with the channels of the feedforward neural network based on the channel attention mechanism; during training, the set of channel importance parameters participates in the forward propagation calculation and is updated synchronously with the network weight parameters during the backpropagation process; during the model inference stage, the channels in the feedforward neural network are sorted according to the absolute value of the set of channel importance parameters, and the top k% of the feature channels in the sorting result are retained to obtain a set of retained feature channels, where k is a positive integer; and the corresponding weight matrix in the feedforward neural network is pruned according to the channel dimension based on the set of retained feature channels.

[0090] Furthermore, the architecture building module 11 in the self-attention mechanism neural network acceleration device is also used for: channel importance parameters to characterize the importance of each feature channel.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The self-attention mechanism neural network acceleration method and specific examples in the foregoing embodiment one are also applicable to the self-attention mechanism neural network acceleration device in this embodiment. Through the foregoing detailed description of the self-attention mechanism neural network acceleration method, those skilled in the art can clearly understand the self-attention mechanism neural network acceleration device in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0092] In embodiment three, based on the same inventive concept as the self-attention mechanism neural network acceleration method in embodiment one, this application also provides a computer-readable storage medium storing a computer program, which, when executed, implements the steps of the self-attention mechanism neural network acceleration method described in any one of embodiments one above.

[0093] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0094] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for accelerating a self-attention mechanism neural network, characterized in that, include: Construct a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initialize the embedding layer and the position encoding layer to obtain the initialized Transformer network; The initialized Transformer network is trained and its parameters are fixed based on the training data to obtain the trained Transformer network. Based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships, the inference speed of the trained Transformer network is accelerated.

2. The method for accelerating a self-attention mechanism neural network as described in claim 1, characterized in that, Construct a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and initialize the embedding layer and the positional encoding layer to obtain the initialized Transformer network, including: In the multi-head self-attention layer, a sparse attention mechanism is introduced to replace the traditional multi-head attention module with a sparse attention module. A sparse mask matrix is ​​constructed based on the activation information of each token in the input sequence to limit the scope of attention calculation. In the sparse attention module, the attention weight matrix is ​​divided into blocks, and a low-rank decomposition interface is reserved for each sub-block to obtain the block-based low-rank attention matrix. A channel attention mechanism is introduced into the feedforward neural network, and a learnable channel importance parameter is set to control the channel output of the feedforward neural network. An inter-layer shared projection matrix mechanism is introduced to establish a projection matrix sharing relationship between multiple Transformer network layers, enabling at least two Transformer network layers to share the query matrix, key matrix, and value matrix; A hybrid positional encoding is constructed in the positional encoding layer, and then added to the embedding layer to obtain the initialized Transformer network.

3. The method for accelerating a self-attention mechanism neural network as described in claim 2, characterized in that, The attention weights of the sparse attention module are: ； in, Let M represent the Hadamard product, and SparseAttn(Q,K,V) be the attention weights, where Q is the query matrix, K is the key matrix, V is the value matrix, and M is the sparse mask matrix. This is the scaling factor.

4. The method for accelerating a self-attention mechanism neural network as described in claim 2, characterized in that, In the sparse attention module, the attention weight matrix is ​​divided into blocks, and a low-rank decomposition interface is reserved for each sub-block to obtain the block-based low-rank attention matrix, including: The attention weight matrix is ​​divided into blocks to obtain multiple sub-blocks; Using the singular value decomposition function, the multiple sub-blocks are traversed to perform singular value decomposition, resulting in a block-based low-rank attention matrix.

5. The method for accelerating a self-attention mechanism neural network as described in claim 4, characterized in that, The singular value decomposition function is: ； Among them, A ij Attention weight matrix The i-th and j-th sub-blocks, where n is a positive integer, U i For A ij The left singular vector matrix obtained by singular value decomposition, For A ij The singular value diagonal matrix obtained by singular value decomposition, For A ij The transpose of the right singular vector matrix obtained by singular value decomposition; The block-based low-rank attention matrix is ​​as follows: ; in, V is the approximate attention matrix composed of the low-rank approximation results of all sub-blocks, and V is the value matrix in the self-attention mechanism. j To be with sub-block A ij The value submatrix corresponding to the column direction.

6. The method for accelerating a self-attention mechanism neural network as described in claim 2, characterized in that, A hybrid positional encoding is constructed in the positional encoding layer, and then added to the embedding layer to obtain the initialized Transformer network, including: Initialize the sine or cosine position encoding matrix; A learnable scalar parameter α is introduced, which is automatically optimized during training via gradient descent. Obtain the hybrid encoding formula, wherein the hybrid encoding formula is: ； ； in, The values ​​of the hybrid position encoding matrix at position pos and the 2i-th dimension are given. Let d be the value of the hybrid positional encoding matrix at position pos and the (2i+1)th dimension, where pos is the position index of the token in the input sequence and d is the total number of dimensions of the positional encoding vector.

7. The method for accelerating a self-attention mechanism neural network as described in claim 2, characterized in that, A channel attention mechanism is introduced into the feedforward neural network, setting learnable channel importance parameters to control the channel output of the feedforward neural network, including: Based on the channel attention mechanism, a set of learnable channel importance parameters is set up that corresponds one-to-one with the channels of the feedforward neural network; During training, the set of channel importance parameters participates in the forward propagation calculation and is updated synchronously with the network weight parameters during the back propagation process; During the model inference phase, the channels in the feedforward neural network are sorted according to the absolute value of the channel importance parameter set, and the top k% of the feature channels in the sorting result are retained to obtain the retained feature channel set, where k is a positive integer; Based on the set of retained feature channels, the corresponding weight matrix in the feedforward neural network is pruned according to the channel dimension.

8. The method for accelerating a self-attention mechanism neural network as described in claim 7, characterized in that, The channel importance parameter is used to characterize the importance of each feature channel.

9. A self-attention mechanism neural network acceleration device, characterized in that, The step of implementing the self-attention mechanism neural network acceleration method according to any one of claims 1 to 8, wherein the self-attention mechanism neural network acceleration device comprises: The architecture building module is used to construct a Transformer network structure including a multi-head self-attention layer, a feedforward neural network layer, a layer normalization module, and residual connections, and to initialize the embedding layer and the positional encoding layer to obtain the initialized Transformer network. The parameter fixing module is used to train and fix the parameters of the initialized Transformer network based on the training data, so as to obtain the trained Transformer network. The inference acceleration module is used to accelerate the inference of the trained Transformer network based on channel importance parameters, sparse attention parameters, low-rank decomposition parameters, and projection matrix sharing relationships.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the steps of the self-attention mechanism neural network acceleration method according to any one of claims 1 to 8.

Images