Method, system, device and medium for implementing linear complexity self-attention mechanism

Abstract

The present invention relates to the field of artificial intelligence. The present invention proposes a method, system, device and medium for realizing a linear complexity self-attention mechanism. The method comprises: position encoding the sequence data to be processed and applying feature mapping to the query matrix and the key matrix; calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix; augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix; transposing the sequence self-attention matrix as sequence data and repeating the above steps, using the sequence self-attention matrix obtained again as a feature self-attention matrix, and obtaining a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix. The present invention realizes linear self-attention while better maintaining the model performance and having stronger scalability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, system, device and medium for implementing a linear complexity self-attention mechanism. Background Art

[0002] Recurrent neural networks (RNNs) perform well in processing sequence data, but they are prone to gradient vanishing or gradient exploding problems when facing long sequences, and their computational efficiency is low. Convolutional neural networks (CNNs) perform well in feature extraction, but their receptive field is limited by the size and number of convolution kernels, making it difficult to capture long-distance dependencies. The self-attention mechanism was first proposed in the paper "Attention is All You Need" and quickly became a milestone in the field of NLP. It solves the shortcomings of traditional models in processing long sequence data, and captures the dependencies between information more efficiently and accurately by dynamically assigning different weights to different positions in the sequence. The self-attention mechanism has been widely used in recent years, and has been widely used because of its lack of prior assumptions, global characteristics, and wide applicability.

[0003] Although the self-attention mechanism has significant advantages in processing long sequences, due to the computational characteristics of the self-attention mechanism, each point in the sequence needs to be similar to all points in the sequence, so the computational complexity is O(N^2), where n is the length of the sequence. As the length of the sequence increases, the amount of computation increases dramatically, which leads to a high computational cost, greatly increasing the inference time and memory usage of most models based on the self-attention mechanism, greatly increasing the training and inference costs of the model, and limiting its deployment and application on devices with lower computing power. A series of methods including Linformer reduce its computational complexity to linear, but are limited by issues such as approximation accuracy, resulting in a large performance loss. Summary of the invention

[0004] In order to solve one or more of the above-mentioned technical problems, the present invention proposes a method for implementing a linear complexity self-attention mechanism, including:

[0005] S1: Position encoding of the sequence data to be processed and feature mapping applied to the query matrix and key matrix;

[0006] S2: Calculate based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix;

[0007] S3: Augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0008] S4: Transpose the sequence self-attention matrix as sequence data and repeat S1-S3, and use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix.

[0009] In some embodiments, the step of calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix comprises:

[0010] Selecting row submatrices from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix;

[0011] Taking the product of the query matrix and the transpose of the key row submatrix as the low-rank query matrix;

[0012] The product of the key matrix and the transpose of the query row submatrix is ​​used as the low-rank key matrix;

[0013] The product of the query matrix and the transpose of the key matrix is ​​taken as the low-rank generalized inverse matrix.

[0014] In some embodiments, the step of selecting a row submatrix from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix comprises:

[0015] Calculating the row norm of each row of the query matrix and the key matrix;

[0016] Apply Softmax to all row norms of each matrix and transform them into probability distribution;

[0017] According to the probability distribution, row sampling is performed;

[0018] The selected rows are extracted and combined into corresponding query row sub-matrices and key row sub-matrices.

[0019] In some embodiments, the step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix includes:

[0020] 1 is used as a constant term vector and combined with the value matrix to form an augmented matrix;

[0021] Left-multiply the augmented matrix and the low-rank query matrix, and calculate the dot product element by element to obtain a first matrix;

[0022] Multiply the first matrix by the low-rank generalized inverse matrix on the left, and calculate the dot product element by element to obtain a second matrix;

[0023] The second matrix and the low-rank query matrix are left-multiplied, and the dot product is calculated element by element to obtain a self-attention matrix, and a sequence self-attention matrix is ​​obtained based on the self-attention matrix.

[0024] In some embodiments, the step of obtaining a sequence self-attention matrix based on the self-attention matrix comprises:

[0025] Selecting the maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the maximum normalization term;

[0026] Selecting a minimum value of an intersection matrix of the low-rank query matrix and the low-rank key matrix as a minimum normalization term;

[0027] Scaling all elements of the self-attention matrix to the range of [0, 1] by the maximum normalization term and the minimum normalization term to obtain a normalized matrix;

[0028] The normalized matrix is ​​used as the sequence self-attention matrix.

[0029] In some embodiments, the step of obtaining a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix comprises:

[0030] The transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix are concatenated and output as a self-attention matrix of linear complexity.

[0031] In some embodiments, the step of position encoding the sequence data to be processed and applying feature mapping to the query matrix and the key matrix includes:

[0032] generating a positional encoding according to the length of the sequence data and the dimensions of the query matrix and the key matrix;

[0033] The position codes are added to the corresponding positions in the query matrix and the key matrix.

[0034] The present invention proposes a system for implementing a linear complexity self-attention mechanism, comprising:

[0035] A preprocessing unit configured to position encode the sequence data to be processed and apply feature mapping to the query matrix and the key matrix;

[0036] A low-rank unit, configured to calculate based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix;

[0037] An approximation unit configured to augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0038] The interactive unit is configured to transpose the sequence self-attention matrix as sequence data to repeatedly call the preprocessing unit, the low-rank unit and the approximation unit, use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix.

[0039] The present invention provides a computer device, comprising:

[0040] At least one processor; and a memory storing a computer program executable on the processor, wherein the processor executes the steps of the method implemented by the linear complexity self-attention mechanism when executing the program.

[0041] The present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method for implementing the linear complexity self-attention mechanism are performed.

[0042] The present invention has at least the following beneficial technical effects:

[0043] The present invention proposes a method, system, device and medium for implementing a linear complexity self-attention mechanism, the method comprising: S1: position encoding the sequence data to be processed and applying feature mapping to a query matrix and a key matrix; S2: calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix; S3: augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix; S4: transposing the sequence self-attention matrix as sequence data and repeating S1-S3, using the sequence self-attention matrix obtained again as a feature self-attention matrix, obtaining a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix and outputting it.

[0044] The present invention utilizes the low-rank characteristics of the self-attention mechanism and fully utilizes the key information in the low-rank matrix by decomposing it. While achieving linear self-attention, it also maintains the model performance well. The amount of calculation is linearly related to the sequence length, and it has stronger scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flow chart of the method for implementing the linear complexity self-attention mechanism provided by the present invention;

[0047] Figure 2 A system module diagram for implementing the linear complexity self-attention mechanism provided by the present invention;

[0048] Figure 3 A self-attention mechanism diagram of an embodiment of a method for implementing a linear complexity self-attention mechanism provided by the present invention;

[0049] Figure 4 A comparison chart of quantitative indicators of an embodiment of a method for implementing a linear complexity self-attention mechanism provided by the present invention;

[0050] Figure 5 A schematic diagram of the structure of an embodiment of a computer device provided by the present invention;

[0051] Figure 6 A schematic diagram of the structure of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0053] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0054] This paper proposes a method to implement the linear complexity self-attention mechanism, please refer to Figure 1 ,include:

[0055] S1: Position encoding of the sequence data to be processed and feature mapping applied to the query matrix and key matrix;

[0056] S2: Calculate based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix;

[0057] S3: Augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix, the low-rank key matrix, the low-rank generalized inverse matrix, and the low-rank query matrix;

[0058] S4: Transpose the sequence self-attention matrix as sequence data and repeat S1 - S3. Use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix with linear complexity based on the sequence self-attention matrix and the feature self-attention matrix.

[0059] The implementation process of the present invention is based on CUR decomposition, and its decomposition process is as follows:

[0060] Obtain the sequence data (N, C) to be processed, where N represents the sequence length and C represents the feature dimension. Encode the sequence data to obtain query (Q), key (K), and value (V) matrices, and apply positional encoding to Q and K;

[0061] Apply Softmax to the row norms of Q and K, and select the row sub-matrices of Q and K according to the probability distribution, satisfying (d << N, d << C), where d represents the number of selected rows;

[0062] Calculate the low rank of where

[0063] Approximate Augment V to get (V; 1), calculate CUR(V; 1) from right to left, and normalize the rows of CURV to get O to achieve information transfer in the tokens of the sequence;

[0064] Transpose O (C, N), repeat the above process to get {O} to achieve interaction between features;

[0065] Output + 。

[0066] The proposed method for calculating the self-attention mechanism with linear complexity significantly reduces the computational complexity of the self-attention mechanism and improves the computational efficiency when processing long sequences. Through low-rank approximation and vector quantization techniques, the present invention can effectively reduce computational resources and time consumption while maintaining the model performance, and is applicable to various sequence processing tasks such as natural language processing, computer vision, and speech recognition, etc., with broad application prospects. In some embodiments, the aforementioned sequence data can be, for example, text, time series data, and sequence data converted from non-sequence data such as images and audio.

[0067] In some embodiments, please refer to Figure 1, the step of calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix comprises:

[0068] Selecting row submatrices from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix;

[0069] Taking the product of the query matrix and the transpose of the key row submatrix as the low-rank query matrix;

[0070] The product of the key matrix and the transpose of the query row submatrix is ​​used as the low-rank key matrix;

[0071] The product of the query matrix and the transpose of the key matrix is ​​taken as the low-rank generalized inverse matrix.

[0072] Compared with the standard CUR decomposition, the present invention replaces the U matrix in the CUR decomposition by constructing an approximation of a low-rank generalized inverse matrix. The calculation is simple and fast, and the original complexity of O( ) is reduced to O(n), where n is the sequence length. This is because the rank k of the low-rank matrix is ​​much smaller than n, so that when calculating the attention score, the full matrix multiplication operation can be avoided, thus significantly improving the computational efficiency.

[0073] A low-rank matrix means that there is a strong correlation or redundancy between the rows or columns in the matrix. Low-rank approximation can remove this redundant information and retain only the main features. Each row or column of a low-rank matrix can be linearly represented by other rows or columns. This redundancy allows the model to focus more on extracting the truly important information in the data while ignoring noise or minor information. In applications such as image processing, the low-rank property of low-rank matrices can be used to remove noise from images. Similarly, in natural language processing, low-rank approximation can also be seen as a denoising process that helps the model better capture the essential characteristics of the language rather than being disturbed by the noise in the data. Since low-rank matrices contain fewer free parameters (i.e., the rank k is smaller), overfitting is less likely to occur during training. This helps the model maintain good generalization capabilities on unseen data.

[0074] In some embodiments, see Figure 1 The step of selecting a row submatrix from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix comprises:

[0075] Calculating the row norm of each row of the query matrix and the key matrix;

[0076] Apply Softmax to all row norms of each matrix and transform them into probability distribution;

[0077] Sample the row submatrix according to the probability distribution and fill up k rows;

[0078] The selected rows are extracted and combined into corresponding query (Q) row submatrix and key (K) row submatrix.

[0079] In the matrix, the self-attention mechanism generates a self-attention probability distribution by calculating the interaction between the query matrix, the key matrix, and the value matrix, and then selects a row submatrix. The self-attention probability distribution assigns different weights to each input element. These weights reflect the importance or relevance between elements. By selecting a high-weight row submatrix, it is possible to focus on the key information in the input sequence and ignore the unimportant parts.

[0080] When processing long sequences, the self-attention mechanism can capture long-distance dependencies between elements, which is difficult for traditional RNN models to do. By selecting row submatrices containing these dependency information, the context of the entire sequence can be better understood. At the same time, probability distribution is a way of information aggregation. It recombines the information of the entire sequence according to weights. The model can use this information more effectively, thereby improving the prediction accuracy and generalization ability of the model.

[0081] In some embodiments, see Figure 1 , the step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix includes:

[0082] 1 is used as a constant term vector and combined with the value matrix to form an augmented matrix;

[0083] Left-multiply the augmented matrix and the low-rank query matrix, and calculate the dot product element by element to obtain a first matrix;

[0084] Left-multiply the first matrix and the low-rank generalized inverse matrix, and calculate the dot product element by element to obtain a second matrix;

[0085] The second matrix and the low-rank query matrix are left-multiplied, and the dot product is calculated element by element to obtain a self-attention matrix, and a sequence self-attention matrix is ​​obtained based on the self-attention matrix.

[0086] The computational complexity of the self-attention mechanism is usually high when processing long sequences, especially when the sequence length n increases, the amount of computation increases quadratically (O(n^2)). By using low-rank matrices for approximation, the computational complexity can be reduced to a linear level (O(n) or lower), thereby significantly improving computational efficiency. The rank k of a low-rank matrix is ​​much smaller than n, and the amount of computation can be reduced when using a low-rank matrix for multiplication.

[0087] The left multiplication operation of the low-rank matrix is ​​actually a compression or dimensionality reduction process of the original self-attention matrix, which retains the main features and structural information in the original matrix. It has high flexibility and adaptability and can be applied to various types of tasks and data sets. By adjusting the parameters such as the rank k of the low-rank matrix and the approximate algorithm, the complexity and performance of the model can be flexibly controlled. This makes this method widely applicable and valuable for promotion in practical applications.

[0088] In some embodiments, see Figure 1 , the step of obtaining a sequence self-attention matrix based on the self-attention matrix comprises:

[0089] Selecting the maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the maximum normalization term;

[0090] Selecting a minimum value of an intersection matrix of the low-rank query matrix and the low-rank key matrix as a minimum normalization term;

[0091] Scaling all elements of the self-attention matrix to the range of [0, 1] by the maximum normalization term and the minimum normalization term to obtain a normalized matrix;

[0092] The normalized matrix is ​​used as the sequence self-attention matrix.

[0093] Features at different positions may have different effects on the results. Normalization can ensure that each feature has a relatively equal contribution in the calculation of the attention score, avoiding the situation where some features are over-emphasized or ignored due to differences in data magnitude.

[0094] Due to the existence of the chain rule, the gradient may increase or decrease cumulatively during the back propagation process, leading to gradient explosion or gradient vanishing problems. Normalization can alleviate this problem to a certain extent by limiting the magnitude range of the data and enhancing the stability of the model. Normalization can also reduce the model's dependence on a specific data distribution, making it easier for the model to adapt to different data distributions, thereby improving the model's generalization ability.

[0095] The normalized data has a unified dimension and magnitude range, which makes subsequent calculations and processing easier and more efficient. When calculating the attention score, the normalized data can be more directly applied to functions such as softmax, avoiding calculation errors caused by differences in data magnitude.

[0096] In some embodiments, see Figure 1 The step of transposing the sequence self-attention matrix as sequence data and repeating S1-S3 to obtain a feature self-attention matrix includes:

[0097] Transposing the sequence self-attention matrix to obtain a transposed matrix;

[0098] Taking the transposed matrix as sequence data, selecting the position code of the transposed matrix and mapping it into the query matrix and the key matrix;

[0099] After transforming the query matrix and the key matrix, the feature self-attention matrix is ​​obtained by normalization.

[0100] The sequence self-attention matrix is ​​again subjected to CUR decomposition, and the sequence self-attention matrix is ​​approximated by selecting representative rows and columns in the matrix (i.e., C and R matrices), and the intersections between them (i.e., U matrix). This process is actually extracting key features from the sequence self-attention matrix, which are crucial for understanding the context of the sequence and the dependencies between elements. The extracted features are re-used as a new matrix, i.e., the feature self-attention matrix.

[0101] By enhancing feature interactions, the model can capture the information in the sequence more comprehensively and improve its performance on various tasks. The enhanced feature interactions help the model better understand the semantics and contextual information of the sentence.

[0102] In some embodiments, see Figure 1 The step of position encoding the sequence data to be processed and applying feature mapping to the query matrix and the key matrix includes:

[0103] generating a positional encoding according to the length of the sequence data and the dimensions of the query matrix and the key matrix;

[0104] The position codes are added to the corresponding positions in the query matrix and the key matrix.

[0105] Position encoding is able to perceive the position of elements in the input sequence. This is crucial for processing data with clear sequential relationships such as natural language and time series. By introducing position, the context and interdependence of elements in the sequence can be understood more accurately, thereby improving performance on various tasks such as text classification, machine translation, time series prediction, etc. Position encoding adds a feature representation of the position dimension to each element of the query matrix, allowing the model to comprehensively consider the content and position information of the element when performing self-attention calculations. This multi-dimensional feature representation helps the model capture the information in the sequence more comprehensively. Since position encoding is encoded based on the inherent properties of the sequence (i.e., position), it has universality and generalization capabilities, which enables the model to maintain stable performance when processing sequences of different lengths.

[0106] In some embodiments, the step of obtaining a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix comprises:

[0107] The transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix are concatenated and output as a self-attention matrix of linear complexity.

[0108] After the sequence self-attention matrix is ​​calculated, in order to enhance feature interaction, the sequence self-attention matrix is ​​selected to perform the method of the present invention again to obtain a result that pays more attention to the features, that is, the feature self-attention matrix, and then the two matrices are transposed and added together for output. This can not only ensure that the output result has linear complexity and avoid exponential complexity, but also achieve that the output result is infinitely close to the original self-attention matrix, thereby improving computational efficiency.

[0109] This paper proposes a system that implements the linear complexity self-attention mechanism, please refer to Figure 2 ,include:

[0110] A preprocessing unit 100 is configured to perform position encoding on the sequence data to be processed and apply feature mapping to the query matrix and the key matrix;

[0111] A low-rank unit 200, configured to calculate based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix;

[0112] An approximation unit 300 is configured to augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0113] The interactive unit 400 is configured to transpose the sequence self-attention matrix as sequence data to repeatedly call the preprocessing unit, the low-rank unit and the approximation unit, use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix and output it.

[0114] The present invention utilizes the low-rank characteristics of the self-attention mechanism and fully utilizes the key information in the low-rank matrix by decomposing it. While achieving linear self-attention, it also maintains the model performance well. The amount of calculation is linearly related to the sequence length, and it has stronger scalability.

[0115] In some embodiments, see Figure 1 , Figure 3 and Figure 4 In steps S2 and S3, the idea of ​​skeleton decomposition is used to decompose the matrix into multiple low-rank matrices, and operations are performed on the low-rank matrices, thereby reducing the complexity and ensuring that no The intermediate result of Calculation expenses.

[0116] First, for the lth self-attention layer, let its input be , the output is , ignoring the normalization layer, its output can be calculated by the following formula:

[0117]

[0118] in, Representative The feature map matrix of the layer, these two matrices can be the same. Usually there is , this is to reduce computational overhead. , , , are all learnable parameters used to Integration is performed on the channel dimension.

[0119] in, is called the self-attention matrix because it computes the similarity between every pair of tokens. However, computing this matrix is ​​expensive because its computational complexity is . Since arrive The process does not involve self-attention, so the expression of the self-attention mechanism is rewritten as follows:

[0120]

[0121] Similarly, for the multi-head self-attention mechanism (MHSA), the input is divided into multiple slices in the feature dimension, which can be written as follows:

[0122]

[0123] in is the number of heads, Indicates the first slices. It can be seen that for each head, the self-attention calculation method is no different from the single-head self-attention. After discussing the single-head self-attention mechanism, it can be directly transplanted to each head of the multi-head self-attention mechanism, so the formula of the self-attention mechanism is rewritten as:

[0124]

[0125] in is a system where all elements are of dimension column vector, represents a diagonal matrix spanned by column vectors, is a calculation for matrix elements. By multiplying two matrices ( ) instead of Calculation of and The meaning is exactly the same. But in this way, Replaced by matrix operations. is a diagonal matrix whose diagonal elements are The row sum of . Here we call the matrix is the self-attention matrix.

[0126] Regarding skeleton decomposition, in addition to obtaining the row-column submatrix through probability distribution in step S2, it can also be obtained by the following method: given , Indicates that the original matrix A is indexed , the obtained row submatrix; Represents the original matrix Indexed , and obtain the column submatrix. There are two ways to select row and column indexes: indexing based on vector norm and random index selection.

[0127] So the skeleton decomposition of A by index can be written as a formula:

[0128]

[0129] After decomposing the matrix A, the formula of A can be rewritten as:

[0130]

[0131] Among them, The row sampling of sampling of Column sampling is equivalent to of which and are all column vectors, so the formula is:

[0132]

[0133] Since the exponential power operation and normalization here are both calculations for matrix elements, after considering the exponential power operation and normalization operations, the formula is as follows:

[0134]

[0135] ;

[0136] In order to avoid repeated calculations, the self-attention mechanism based on skeleton decomposition is rewritten as follows:

[0137]

[0138] In step S2, in addition to the method of calculating the low-rank generalized inverse matrix in step 2, two alternative generalized inverse approximation methods can also be used.

[0139] The first method is the permutation matrix method, which replaces the matrix M with a similar permutation matrix and regards the generalized inverse of the permutation matrix as First, select the global largest element , set all elements in row i and column j to 0, and then search The global maximum value other than , and set the corresponding rows and columns to zero again, and repeat the above process until there is no row (column) with more than one non-zero element.

[0140] The second method is to replace the above process with an iterative method as follows:

[0141]

[0142] Due to the exponential operation, There are very large elements in , and directly approximating its generalized inverse through iteration may cause numerical overflow. ,when is a non-zero scalar, we have , so by a larger number right Normalize and then Approximation is an optional method. So there is the following formula:

[0143]

[0144] Among them, there are the following formulas:

[0145]

[0146] In step S3, when normalizing, directly set as a normalization term.

[0147] In summary, the following decomposition process can be extended in step S2 and step S3:

[0148] (1) Select appropriate row and column indices and , and meet ;

[0149] (2) Calculate based on the selected row and column indexes ;

[0150] (3) Calculation ;

[0151] (4) Calculate the final output ;

[0152] Among them, according to the associative law of matrix multiplication, we can first calculate , and then calculate Until the final result is obtained.

[0153] although is a , but it is 0 except for the diagonal elements, so it is not necessary to obtain it in actual calculations , only need to No. Multiply the row elements by No. The reciprocal of the elements.

[0154] Therefore, in this process, no The intermediate result of Calculation expenses.

[0155] Finally, in Figure 4 It can be seen that when comparing the quantitative indicators of the algorithms, SPCT is a baseline model using the standard self-attention mechanism, Reformer, Linformer and Deformer are several variants of the Transformer model that are improved or optimized in the field of deep learning, especially in natural language processing (NLP) and computer vision (CV). The prediction accuracy of the present invention is the highest, and the performance of the model is improved most significantly.

[0156] Based on the same inventive concept, according to another aspect of the present invention, Figure 5 As shown, an embodiment of the present invention further provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, the steps of the above method are performed.

[0157] Based on the same inventive concept, according to another aspect of the present invention, Figure 6 As shown, an embodiment of the present invention further provides a computer-readable storage medium 40, which stores a computer program 410 for executing the above method when executed by a processor.

[0158] The embodiment of the present invention may also include a corresponding computer device. The computer device includes a memory, at least one processor, and a computer program stored in the memory and executable on the processor, and the processor executes any one of the above methods when executing the program.

[0159] The memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules in the embodiments of the present application. The processor executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory, that is, implementing the above method.

[0160] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In an embodiment, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] Finally, it should be noted that a person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding above-mentioned arbitrary method embodiments.

[0162] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0163] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0164] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0165] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. The method for implementing the linear complexity self-attention mechanism is characterized by: include: S1: Position encoding is performed on the sequence data to be processed and feature mapping is applied to the query matrix and the key matrix, wherein the sequence data to be processed includes natural language and images, a position encoding for position dimension feature representation is generated according to the length of the sequence data and the dimensions of the query matrix and the key matrix, and the position encoding is added to the corresponding positions in the query matrix and the key matrix; S2: Calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix of the sequence data, wherein the product of the transpose of the query matrix and the key matrix is ​​used as the low-rank generalized inverse matrix; S3: Augmenting the value matrix to obtain an augmented matrix, and performing low-rank matrix operations based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix to obtain a sequence self-attention matrix for denoising, wherein the step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix includes: taking 1 as a constant term vector and combining it with the value matrix to form an augmented matrix, multiplying the augmented matrix and the low-rank query matrix on the left, calculating the dot product element by element, and obtaining a first matrix, multiplying the first matrix and the low-rank generalized inverse matrix on the left, calculating the dot product element by element, and obtaining a second matrix, multiplying the second matrix and the low-rank query matrix on the left, calculating the dot product element by element, and obtaining a self-attention matrix, and obtaining a sequence self-attention matrix based on the self-attention matrix, The step of obtaining a sequence self-attention matrix based on the self-attention matrix includes: selecting the maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the maximum normalization term, selecting the minimum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the minimum normalization term, scaling all elements of the self-attention matrix to the range of [0, 1] by the maximum normalization term and the minimum normalization term to obtain a normalized matrix, and using the normalized matrix as the sequence self-attention matrix; S4: Transpose the sequence self-attention matrix as sequence data and repeat S1-S3, and use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix.

2. The method for implementing the linear complexity self-attention mechanism according to claim 1, characterized in that: The step of calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix comprises: Selecting row submatrices from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix; Taking the product of the query matrix and the transpose of the key row submatrix as the low-rank query matrix; The product of the key matrix and the transpose of the query row submatrix is ​​taken as the low-rank key matrix.

3. The method for implementing the linear complexity self-attention mechanism according to claim 2, characterized in that: The step of selecting row submatrices from the query matrix and the key matrix by probability distribution to obtain a query row submatrix and a key row submatrix comprises: Calculating the row norm of each row of the query matrix and the key matrix; Apply Softmax to all row norms of each matrix and transform them into probability distribution; According to the probability distribution, row sampling is performed; The selected rows are extracted and combined into corresponding query row sub-matrices and key row sub-matrices.

4. The method for implementing the linear complexity self-attention mechanism according to claim 1, characterized in that: The step of obtaining a self-attention matrix with linear complexity based on the sequence self-attention matrix and the feature self-attention matrix comprises: The transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix are concatenated and output as a self-attention matrix of linear complexity.

5. A system implemented with a linear complexity self-attention mechanism, characterized in that: include: A preprocessing unit is configured to perform position encoding on the sequence data to be processed and apply feature mapping to the query matrix and the key matrix, wherein the sequence data to be processed includes natural language and images, generate position encoding for feature representation of position dimension according to the length of the sequence data and the dimensions of the query matrix and the key matrix, and add the position encoding to the corresponding positions in the query matrix and the key matrix; a low-rank unit configured to calculate, based on the query matrix and the key matrix, a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix of the sequence data, wherein a product of the query matrix and the transpose of the key matrix is ​​used as the low-rank generalized inverse matrix; An approximation unit is configured to augment a value matrix to obtain an augmented matrix, and perform low-rank matrix operations based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix to obtain a sequence self-attention matrix for denoising, wherein the step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix includes: taking 1 as a constant term vector and combining it with the value matrix to form an augmented matrix, multiplying the augmented matrix and the low-rank query matrix on the left, calculating the dot product element by element, and obtaining a first matrix, multiplying the first matrix and the low-rank generalized inverse matrix on the left, calculating the dot product element by element, and obtaining a second matrix, multiplying the second matrix and the low-rank query matrix on the left, calculating the dot product element by element, and obtaining a self-attention matrix, and obtaining a sequence self-attention matrix based on the self-attention matrix, The step of obtaining a sequence self-attention matrix based on the self-attention matrix includes: selecting the maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the maximum normalization term, selecting the minimum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the minimum normalization term, scaling all elements of the self-attention matrix to the range of [0, 1] by the maximum normalization term and the minimum normalization term to obtain a normalized matrix, and using the normalized matrix as the sequence self-attention matrix; The interactive unit is configured to transpose the sequence self-attention matrix as sequence data to repeatedly call the preprocessing unit, the low-rank unit and the approximation unit, use the sequence self-attention matrix obtained again as the feature self-attention matrix, and obtain a self-attention matrix of linear complexity based on the sequence self-attention matrix and the feature self-attention matrix.

6. A computer device comprising: at least one processor; and a memory storing a computer program executable on the processor, wherein the processor executes the steps of the method for implementing the linear complexity self-attention mechanism as described in any one of claims 1 to 4 when executing the program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for implementing the linear complexity self-attention mechanism as described in any one of claims 1 to 4 are performed.

Images