Method and system for implementing self-attention mechanism having linear complexity, and device and medium

Abstract

The present application relates to the field of artificial intelligence. Provided in the present application are a method and system for implementing a self-attention mechanism having a linear complexity, and a device and a medium. The method comprises: performing positional encoding on sequence data to be processed, and performing feature mapping to apply the positional encoding to a query matrix and a key matrix; on the basis of the query matrix and the key matrix, performing calculation to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix; augmenting a value matrix to obtain an augmented matrix, and on the basis of the augmented matrix, the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix, obtaining a sequence self-attention matrix; and transposing the sequence self-attention matrix, using the transposed matrix as sequence data to repeat the foregoing steps, using the newly obtained sequence self-attention matrix as a feature self-attention matrix, and on the basis of the sequence self-attention matrix and the feature self-attention matrix, obtaining a self-attention matrix having a linear complexity. The present application implements linear self-attention, better maintains the model performance and has relatively good expansibility.

Description

Method, system, device and medium for implementing linear complexity self-attention mechanism

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202411296820.X, filed on September 18, 2024, and entitled “Method, system, device and medium for implementing linear complexity self-attention mechanism”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to a method, system, device and medium for implementing linear complexity self-attention mechanism. BACKGROUND

[0004] Recurrent neural networks (RNNs) perform well in processing sequential data, but are prone to gradient vanishing or gradient explosion when dealing with long sequences, and are computationally inefficient. Convolutional neural networks (CNNs) perform well in feature extraction, but their receptive field is limited by the size of the convolution kernel and the number of layers, making it difficult to capture long-range dependencies. The self-attention mechanism was first proposed in the paper “Attention is All You Need” and quickly became a milestone in the field of NLP. It addresses the shortcomings of traditional models in processing long sequence data by dynamically assigning different weights to different positions in the sequence, more efficiently and accurately capturing dependencies between information. In recent years, the self-attention mechanism has been widely applied due to its lack of prior assumptions, global characteristics, and wide applicability.

[0005] Although the self-attention mechanism has significant advantages in processing long sequences, due to the computational characteristics of the self-attention mechanism, each point in the sequence needs to be calculated with all points in the sequence, resulting in a computational complexity of O(N^2), where n is the length of the sequence. As the length of the sequence increases, the amount of computation increases dramatically, which results in a high computational cost, greatly increasing the inference time and memory usage of most models based on the self-attention mechanism, greatly increasing the training and inference cost of the model, limiting its deployment and application on devices with lower computing power. A series of methods, including Linformer, reduce the computational complexity to linear, but are limited by approximation accuracy and other issues, resulting in significant performance loss. SUMMARY

[0006] To solve one or more of the aforementioned technical problems, the present application provides a method for implementing a linear complexity self-attention mechanism, comprising:

[0007] S1: encode the sequence data to be processed by position and feature mapping and apply to the query matrix and the key matrix;

[0008] S2: calculate based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix;

[0009] S3: augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0010] S4: transpose the sequence self-attention matrix as a sequence data, repeat S1-S3, take the sequence self-attention matrix obtained again as a feature self-attention matrix, and obtain a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix.

[0011] In some embodiments, the step of calculating based on the query matrix and the key matrix to obtain a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix comprises:

[0012] Selecting a row sub-matrix in the query matrix and the key matrix by a probability distribution to obtain a query row sub-matrix and a key row sub-matrix;

[0013] Taking the product of the transpose of the query row sub-matrix and the key matrix as the low-rank query matrix;

[0014] Taking the product of the transpose of the query row sub-matrix and the key matrix as the low-rank query matrix;

[0015] Taking the product of the transpose of the query row sub-matrix and the key matrix as the low-rank query matrix.

[0016] In some embodiments, the step of selecting a row sub-matrix in the query matrix and the key matrix by a probability distribution to obtain a query row sub-matrix and a key row sub-matrix comprises:

[0017] Calculating the row norm of each row of the query matrix and the key matrix;

[0018] Applying Softmax to all row norms of each matrix to convert to a probability distribution;

[0019] According to the probability distribution, row sampling is performed;

[0020] Extracting the selected row combination into the corresponding query row sub-matrix and key row sub-matrix.

[0021] In some embodiments, the step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix comprises:

[0022] Combine the 1 as a constant term vector and the value matrix into an augmented matrix;

[0023] Left multiply the augmented matrix and the low-rank query matrix, and calculate the dot product element by element to obtain a first matrix;

[0024] Left multiply the first matrix and the low-rank generalized inverse matrix, and calculate the dot product element by element to obtain a second matrix;

[0025] Left multiply the second matrix and the low-rank query matrix, and calculate the dot product element by element to obtain a self-attention matrix, and obtain a sequence self-attention matrix based on the self-attention matrix.

[0026] In some embodiments, the step of obtaining the sequence self-attention matrix based on the self-attention matrix comprises:

[0027] Select the intersection matrix maximum value of the low-rank query matrix and the low-rank key matrix as the maximum normalization term;

[0028] Select the intersection matrix minimum value of the low-rank query matrix and the low-rank key matrix as the minimum normalization term;

[0029] Scale all elements of the self-attention matrix to the range of [0, 1] through the maximum normalization term and the minimum normalization term to obtain a normalized matrix;

[0030] Take the normalized matrix as the sequence self-attention matrix.

[0031] In some embodiments, the step of obtaining the linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix comprises:

[0032] Concatenate the transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix as the linear complexity self-attention matrix and output.

[0033] In some embodiments, the step of performing position encoding and feature mapping of the to-be-processed sequence data into the query matrix and the key matrix comprises:

[0034] Generate position encoding according to the length of the sequence data and the dimensions of the query matrix and the key matrix;

[0035] Add the position encoding to the corresponding positions in the query matrix and the key matrix.

[0036] The present application proposes a system for implementing a linear complexity self-attention mechanism, comprising:

[0037] A preprocessing unit configured to perform position encoding and feature mapping of the to-be-processed sequence data into the query matrix and the key matrix;

[0038] The low-rank unit is configured to calculate a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix based on the query matrix and the key matrix;

[0039] The approximation unit is configured to augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0040] The interaction unit is configured to transpose the sequence self-attention matrix as sequence data to repeatedly call the preprocessing unit, the low-rank unit and the approximation unit, and obtain a feature self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix to obtain a linear complexity self-attention matrix.

[0041] The application provides a computer device, comprising:

[0042] At least one processor; and a memory storing a computer program capable of running on the processor, and the processor executes the steps of the method for implementing the linear complexity self-attention mechanism when executing the program.

[0043] The application provides a computer non-volatile readable storage medium, which stores a computer program, and the steps of the method for implementing the linear complexity self-attention mechanism are executed when the computer program is executed by the processor.

[0044] The application has at least the following beneficial technical effects:

[0045] The application provides a method, system, device and medium for implementing a linear complexity self-attention mechanism, the method comprising: S1: performing position encoding and feature mapping on sequence data to be processed and applying the sequence data to a query matrix and a key matrix; S2: calculating a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix based on the query matrix and the key matrix; S3: augmenting a value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix; S4: transposing the sequence self-attention matrix as sequence data to repeatedly perform S1-S3, and obtaining a feature self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix to obtain a linear complexity self-attention matrix and output.

[0046] The application utilizes the low-rank property of the self-attention mechanism, fully utilizes the key information in the low-rank matrix by decomposing the low-rank matrix, maintains the model performance while realizing linear self-attention, and has a linear relationship between the calculation amount and the sequence length, and has stronger scalability. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0048] Fig. 1 is a flow chart of the method for implementing the linear complexity self-attention mechanism provided by the present application;

[0049] Fig. 2 is a system block diagram of the method for implementing the linear complexity self-attention mechanism provided by the present application;

[0050] Fig. 3 is a self-attention mechanism diagram of an embodiment of the method for implementing the linear complexity self-attention mechanism provided by the present application;

[0051] Fig. 4 is a comparison diagram of quantitative indicators of an embodiment of the method for implementing the linear complexity self-attention mechanism provided by the present application;

[0052] Fig. 5 is a structural schematic diagram of an embodiment of the computer device provided by the present application;

[0053] Fig. 6 is a structural schematic diagram of an embodiment of the computer non-volatile readable storage medium provided by the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the embodiments of the present application in combination with specific embodiments and with reference to the drawings.

[0055] It should be noted that all the expressions of "first" and "second" in the embodiments of the present application are used to distinguish two same name different entities or different parameters. It can be seen that "first" and "second" are only for the convenience of description, and should not be understood as a limitation of the embodiments of the present application. The subsequent embodiments will not be described one by one.

[0056] The present application proposes a method for implementing the linear complexity self-attention mechanism, please refer to Fig. 1, which comprises:

[0057] S1: position encoding and feature mapping of the to-be-processed sequence data are applied to the query matrix and the key matrix;

[0058] S2: based on the query matrix and the key matrix, low-rank query matrix, low-rank key matrix and low-rank generalized inverse matrix are obtained by calculation;

[0059] S3: the value matrix is augmented to obtain an augmented matrix, and based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix, a sequence self-attention matrix is obtained;

[0060] S4: transpose the sequence self-attention matrix as sequence data, repeat S1-S3, take the sequence self-attention matrix obtained again as a feature self-attention matrix, and obtain a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix.

[0061] The implementation process of the present application is based on CUR decomposition (matrix decomposition technology), and the decomposition process is as follows:

[0062] Obtain the sequence data (N, C) to be processed, where N represents the sequence length and C represents the feature dimension. Encode the sequence data to obtain the query (Q), key (K), and value (V) matrices. Apply position encoding to Q and K;

[0063] Perform Softmax (normalization) on the norm of each row of Q and K, and select the row sub-matrix of Q and K according to the probability distribution, which satisfies (d << N, d << C), where d represents the number of rows selected;

[0064] Calculate QK T of low rank through the row sub-matrix of Q and K, where QK T is approximated as CUR;

[0065] Augment V to obtain (V; 1), and calculate CUR (V; 1) from right to left. Through CUR1 -1 Perform row normalization on CURV to obtain O, and realize information transmission in the token of the sequence;

[0066] Transpose O (C, N), repeat the above process, and obtain {O} to realize the interaction between features;

[0067] Output O T + {O} T .

[0068] The linear complexity self-attention mechanism calculation method proposed in the present application significantly reduces the computational complexity of the self-attention mechanism and improves the computational efficiency when processing long sequences. Through low-rank approximation and vector quantization technology, the present application can effectively reduce the computational resources and time consumption while maintaining the performance of the model, and is suitable for various sequence processing tasks such as natural language processing, computer vision, and speech recognition, etc., and has a wide application prospect. In some embodiments, the aforementioned sequence data may, for example, be text, time series data, and sequence data converted from non-sequence data such as images and audio.

[0069] In some embodiments, referring to FIG. 1, the steps of calculating the low-rank query matrix, the low-rank key matrix, and the low-rank generalized inverse matrix based on the query matrix and the key matrix include:

[0070] selecting row sub-matrices in the query matrix and the key matrix by probability distribution to obtain query row sub-matrices and key row sub-matrices;

[0071] multiplying the product of the query matrix and the transpose of the key row sub-matrices as a low-rank query matrix;

[0072] multiplying the product of the key matrix and the transpose of the query row sub-matrices as a low-rank key matrix;

[0073] multiplying the product of the query matrix and the transpose of the key matrix as a low-rank generalized inverse matrix.

[0074] Compared with the standard CUR decomposition, the present application uses the construction of the low-rank generalized inverse matrix to replace the U matrix in the CUR decomposition, which is simple and fast in calculation. Through the low-rank approximation, the self-attention calculation with the original complexity of O(n 2 ) is reduced to O(n), where n is the sequence length. This is because the rank k of the low-rank matrix is much smaller than n, so that the full matrix multiplication operation can be avoided when calculating the attention score, thereby significantly improving the calculation efficiency.

[0075] The low-rank matrix means that there is a strong correlation or redundancy between the rows or columns in the matrix. Through the low-rank approximation, these redundant information can be removed, and only the main features are retained. Each row or column of the low-rank matrix can be linearly represented by other rows or columns. This redundancy allows the model to focus more on extracting the truly important information in the data, while ignoring noise or secondary information. In image processing applications, the low-rank property of the low-rank matrix can be used to remove noise in the image. Similarly, in natural language processing, the low-rank approximation can also be regarded as a denoising process, which helps the model to better capture the essential features of the language, rather than being disturbed by the noise in the data. Since the low-rank matrix contains fewer free parameters (i.e., the rank k is small), it is less likely to overfit during the training process. This helps the model to maintain good generalization ability on unseen data.

[0076] In some embodiments, referring to FIG. 1, the step of selecting row sub-matrices in the query matrix and the key matrix by probability distribution to obtain query row sub-matrices and key row sub-matrices includes:

[0077] calculating the row norm of each row of the query matrix and the key matrix;

[0078] applying Softmax transformation to all row norms of each matrix to form a probability distribution;

[0079] sampling row sub-matrices according to the probability distribution, and sampling k rows;

[0080] extracting the selected row combinations to form corresponding query (Q) row sub-matrices and key (K) row sub-matrices.

[0081] In the matrix, the self-attention mechanism generates a self-attention probability distribution by calculating the interaction between the query matrix, the key matrix, and the value matrix. When selecting the row sub-matrix, the self-attention probability distribution assigns different weights to each input element, which reflects the importance or relevance between elements. By selecting the row sub-matrix with high weights, the model can focus on the key information in the input sequence and ignore the unimportant parts.

[0082] When processing long sequences, the self-attention mechanism can capture long-distance dependencies between elements, which is difficult for traditional RNN models. By selecting the row sub-matrix containing these dependency information, the model can better understand the context of the entire sequence. At the same time, the probability distribution is a way of information aggregation, which recombines the information of the entire sequence according to the weights. The model can more effectively utilize these information, thereby improving the prediction accuracy and generalization ability of the model.

[0083] In some embodiments, referring to FIG. 1, the value matrix is augmented to obtain an augmented matrix, and the steps of obtaining the sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix, and the low-rank query matrix include:

[0084] The constant term vector 1 is combined with the value matrix to form an augmented matrix;

[0085] The augmented matrix is multiplied by the low-rank query matrix on the left, and the dot product is calculated element by element to obtain a first matrix;

[0086] The first matrix is multiplied by the low-rank generalized inverse matrix on the left, and the dot product is calculated element by element to obtain a second matrix;

[0087] The second matrix is multiplied by the low-rank query matrix on the left, and the dot product is calculated element by element to obtain a self-attention matrix, and the sequence self-attention matrix is obtained based on the self-attention matrix.

[0088] When processing long sequences, the self-attention mechanism usually has high computational complexity, especially when the sequence length n increases, the computational amount will increase quadratically (O(n^2)). By using low-rank matrices for approximation, the computational complexity can be reduced to linear level (O(n) or lower), thereby significantly improving the computational efficiency. The rank k of the low-rank matrix is much smaller than n, and the use of low-rank matrices for multiplication can reduce the amount of calculation.

[0089] The left multiplication operation of the low-rank matrix actually performs a compression or dimensionality reduction on the original self-attention matrix, preserving the main features and structural information in the original matrix. It has high flexibility and adaptability, and can be applied to various types of tasks and datasets. By adjusting the rank k and approximation algorithm parameters of the low-rank matrix, the complexity and performance of the model can be flexibly controlled. This makes the method widely applicable and valuable in practical applications.

[0090] In some embodiments, referring to FIG. 1, the step of obtaining a sequence self-attention matrix based on the self-attention matrix comprises:

[0091] selecting the maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the maximum normalization term;

[0092] selecting the minimum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix as the minimum normalization term;

[0093] scaling all elements of the self-attention matrix to the range of [0, 1] through the maximum normalization term and the minimum normalization term to obtain a normalized matrix;

[0094] taking the normalized matrix as the sequence self-attention matrix.

[0095] The influence of features at different positions on the result can be different. The normalization operation can ensure that each feature has a relatively equal contribution in the calculation of attention scores, avoiding the situation that some features are overemphasized or ignored due to the difference in the magnitude of data.

[0096] Due to the existence of the chain rule, the gradient can be accumulated to increase or decrease during backpropagation, causing the problem of gradient explosion or gradient vanishing. The normalization operation can alleviate this problem to some extent by limiting the range of the magnitude of data, enhancing the stability of the model. The normalization operation can also reduce the dependence of the model on a specific data distribution, making the model more adaptable to different data distributions, thereby improving the generalization ability of the model.

[0097] The normalized data has a unified dimension and magnitude range, which makes subsequent calculations and processing more convenient and efficient. When calculating the attention scores, the normalized data can be more directly applied to functions such as softmax, avoiding calculation errors caused by the difference in the magnitude of data.

[0098] In some embodiments, referring to FIG. 1, the step of obtaining a feature self-attention matrix by transposing the sequence self-attention matrix as sequence data S1-S3 comprises:

[0099] transposing the sequence self-attention matrix to obtain a transposed matrix;

[0100] taking the transposed matrix as sequence data, selecting the position encoding of the transposed matrix and mapping it to the query matrix and the key matrix;

[0101] After transformation of the query matrix and the key matrix, the normalized processing obtains the feature self-attention matrix.

[0102] The sequence self-attention matrix is decomposed again by CUR, and the sequence self-attention matrix is approximated by selecting representative rows and columns in the matrix (that is, the C and R matrices), and the intersection points between them (that is, the U matrix). This process actually extracts key features in the sequence self-attention matrix, which are essential to understanding the context and dependencies between elements in the sequence. The extracted features are reassembled into a new matrix, which is the feature self-attention matrix.

[0103] By enhancing feature interaction, the model can more comprehensively capture information in the sequence, improve its performance on various tasks, and enhanced feature interaction helps the model better understand the semantic and contextual information of the sentence.

[0104] In some embodiments, referring to FIG. 1, the steps of position encoding and feature mapping the sequence data to be processed applied to the query matrix and the key matrix include:

[0105] Generating position encoding according to the length of the sequence data and the dimensions of the query matrix and the key matrix;

[0106] Adding position encoding to the corresponding positions in the query matrix and the key matrix.

[0107] Position encoding can perceive the position of elements in the input sequence. This is crucial for processing data with a clear sequential relationship, such as natural language, time series, etc. By introducing position, the context and mutual dependence of elements in the sequence can be more accurately understood, thereby improving performance on various tasks such as text classification, machine translation, time series prediction, etc. Position encoding adds a position-dimension feature representation to each element of the query matrix, allowing the model to consider both content and position information when performing self-attention calculations. This multi-dimensional feature representation helps the model more comprehensively capture information in the sequence, and since position encoding is based on the inherent properties of the sequence (i.e. position), it has universality and generalization ability, which allows the model to maintain stable performance when processing sequences of different lengths.

[0108] In some embodiments, the step of obtaining a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix includes:

[0109] Concatenating the transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix as the linear complexity self-attention matrix and outputting.

[0110] After the sequence self-attention matrix is calculated, in order to enhance the feature interaction, the sequence self-attention matrix is selected again to perform the method in the application to obtain a result of paying more attention to the features, that is, a feature self-attention matrix, and then the two matrices are transposed and added together for output, which can not only ensure that the output result has a linear complexity and avoid exponential complexity, but also can realize that the output result is infinitely close to the original self-attention matrix, thereby improving the calculation efficiency.

[0111] The application provides a system for implementing a linear complexity self-attention mechanism, and refer to FIG. 2, which comprises:

[0112] The preprocessing unit 100 is configured to perform position encoding and feature mapping on the sequence data to be processed and apply the sequence data to the query matrix and the key matrix;

[0113] The low-rank unit 200 is configured to calculate a low-rank query matrix, a low-rank key matrix and a low-rank generalized inverse matrix based on the query matrix and the key matrix;

[0114] The approximation unit 300 is configured to augment the value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix and the low-rank query matrix;

[0115] The interaction unit 400 is configured to transpose the sequence self-attention matrix as sequence data to repeatedly call the preprocessing unit, the low-rank unit and the approximation unit, take the sequence self-attention matrix obtained again as a feature self-attention matrix, and obtain a self-attention matrix with linear complexity based on the sequence self-attention matrix and the feature self-attention matrix and output the self-attention matrix.

[0116] The application utilizes the low-rank characteristic of the self-attention mechanism, fully utilizes the key information in the low-rank matrix through decomposition of the low-rank matrix, well maintains the model performance while realizing the linear self-attention, and has a linear relationship between the calculation amount and the sequence length and strong scalability.

[0117] In some embodiments, refer to FIG. 1, FIG. 3 and FIG. 4, the skeleton decomposition idea is used in the steps of S2 and S3, the matrix is decomposed into a plurality of low-rank matrices, and the operation is performed on the low-rank matrices, so that the complexity is reduced, it is ensured that no N×N intermediate result is generated, and the calculation cost of O(N 2 ) is avoided.

[0118] Firstly, for the lth self-attention layer, the input is denoted as F l , the output is denoted as F l+1 , and the normalization layer is not considered, and the output can be calculated by the following formula:

[0119] wherein, is the feature map matrix representing the first layer, and the two matrices can be the same. Usually, d r , which is to reduce the computational cost. W1, W2, b1, b2 are learnable parameters, which are used to map the input to the output. The integration is done in the channel dimension.

[0120] where, is called the self-attention matrix, because it calculates the similarity between each pair of tokens. However, calculating this matrix is very expensive, because its computational complexity is O(N 2 ). Since the process from to does not involve self-attention, the expression of the self-attention mechanism can be rewritten as follows:

[0121] Similarly, for the multi-head self-attention mechanism (MHSA), which splits the input into multiple slices in the feature dimension, the formula can be written as follows:

[0122] where M is the number of heads, X i represents the i-th slice of X in the channel dimension. It can be seen that for each head, its self-attention calculation method is no different from that of single-head self-attention. After discussing the single-head self-attention mechanism, it can be directly transplanted into each head of the multi-head self-attention mechanism, so the formula of the self-attention mechanism is rewritten as:

[0123] where 1 N is an N-dimensional column vector with all elements equal to 1, diag(·) represents a diagonal matrix spanned by the column vector, and exp(·) is a calculation for matrix elements. By replacing the Softmax calculation with the form of multiplying two matrices (D -1 A), D -1 A has exactly the same meaning as . However, in this way, the Softmax is replaced by matrix operations. D is a diagonal matrix, whose diagonal elements are the row sums of A. Here, the matrix D -1 A is called the self-attention matrix.

[0124] Regarding the skeleton decomposition, in addition to obtaining the row and column sub-matrices through the probability distribution in step S2, it can also be obtained by the following method, given represents the row sub-matrix obtained by indexing A with r . represents the column sub-matrix obtained by indexing A with c, the obtained column sub-matrix. There are two ways to select the row and column indexes: the index selection based on the vector norm and the random index selection.

[0125] Therefore, the skeleton decomposition of A through the index can be written as the following formula:

[0126] After the decomposition of the matrix A, the formula of A is rewritten as:

[0127] where the row sampling of A is equivalent to the sampling of , and the column sampling of A is equivalent to the sampling of , where q_i (i = 1, 2, …, N) and k j (j = 1, 2, …, N) are column vectors, and the formula obtained is:

[0128] Since the exponential operation and the normalization operation here are the calculations of the matrix elements, after considering the exponential operation and the normalization operation, the formula is as follows:

[0129] In order to avoid repeated calculations, the self-attention mechanism based on the skeleton decomposition is rewritten as follows:

[0130] In step S2, in addition to the method of calculating the low-rank generalized inverse matrix through step 2, two alternative approximate methods of generalized inverse can also be used.

[0131] The first method is a permutation matrix method, that is, the matrix M is replaced by a permutation matrix similar to it, and the generalized inverse of the permutation matrix is regarded as the generalized inverse of M. First, the globally largest element M i,j is selected, and all elements in the i-th row and the j-th column are set to 0, and then the globally largest value other than M i,j is searched, and the corresponding row and column are set to 0 again, and the above process is repeated until there is no row (column) with more than one non-zero element.

[0132] The second method is that the above process can be replaced by an iterative method, as shown below:

[0133] Since the exponential operation is performed, there are very large elements in S , and directly approximating the generalized inverse of S by the iterative method can cause numerical overflow. Since for the matrix S, when k is a non-zero scalar, there is therefore, by normalizing S by a larger number γ and then approximating S , it is a method in the embodiments of the present application. Therefore, the following formula is obtained:

[0134] wherein the following equations are respectively:

[0135] In step S3, when normalization is performed, directly set as the normalization term.

[0136] In summary, in step S2 and step S3, the following process for decomposition can be extended:

[0137] (1) Select appropriate row index and column index Γ r and Γ c , and satisfy |Γ r |=|Γ c |=k<N;

[0138] (2) According to the selected row and column index, calculate

[0139] (3) Calculate

[0140] (4) Calculate the final output

[0141] Wherein, according to the associative law of matrix multiplication, RV can be calculated first, and then U(RV) can be calculated until the final result is obtained.

[0142] Although is an N×N matrix, it is 0 except for the diagonal elements, so in actual calculation, CURV only needs to multiply the reciprocal of the i-th element of CUR1 N .

[0143] Therefore, in this process, there is no N×N intermediate result, thereby avoiding the O(N 2 ) calculation cost.

[0144] Finally, in FIG. 4, when comparing the algorithm quantification indicators, SPCT is the baseline model using the standard self-attention mechanism, Reformer, Linformer and Deformer are several variants in the field of deep learning, especially in natural language processing (NLP) and computer vision (CV), which are improved or optimized for the Transformer model, and the prediction accuracy of the present application is the highest, and the performance of the model is the most obvious.

[0145] Based on the same inventive concept, according to another aspect of the present application, the embodiments of the present application further provide a computer device 30, as shown in Fig. 5, comprising a processor 310 and a memory 320, wherein the memory 320 stores a computer program 321 capable of running on the processor, and the processor 310 executes the steps of the above method when executing the program.

[0146] Based on the same inventive concept, according to another aspect of the present application, the embodiments of the present application further provide a computer nonvolatile readable storage medium 40, as shown in Fig. 6, which stores a computer program 410 capable of executing the above method when executed by a processor.

[0147] The embodiments of the present application can also include corresponding computer devices. The computer device comprises a memory, at least one processor, and a computer program stored in the memory and capable of running on the processor, and the processor executes the program to execute any of the above methods.

[0148] The memory, as a nonvolatile computer nonvolatile readable storage medium, can be used to store nonvolatile software programs, nonvolatile computer executable programs and modules, such as program instructions / modules in the embodiments of the present application. The processor executes the various functions of the device and data processing by running the nonvolatile software programs, instructions and modules stored in the memory, that is, implements the above method.

[0149] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other nonvolatile solid-state storage device. In embodiments, the memory can include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the local module through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0150] Finally, it should be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The storage medium of the program can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding any of the above-mentioned method embodiments.

[0151] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0152] The above are exemplary embodiments of the present disclosure, but it should be noted that various changes and modifications can be made without departing from the scope of the present embodiments defined by the claims. The functions, steps and / or actions of the method claims described above need not be performed in any particular order. Unless explicitly stated, the ordinal use of terms such as "first", "second", etc. does not imply a requirement for any particular order. Furthermore, although the elements of the present embodiments disclosed herein can be described or claimed in individual forms, other embodiments can include a combination of those elements absent otherwise explicitly stated (e.g., processes can include elements of different embodiments).

[0153] It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", or "includes" and / or "including" when used in this specification, specify the presence of stated features, regions, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components, and / or groups thereof.

[0154] Those skilled in the art will appreciate that the above discussions of various embodiments are merely exemplary and not intended to be limiting of the scope of the present embodiments (including claims) which are defined by the appended claims. Those skilled in the art will further appreciate that the various embodiments described above and illustrated in the drawings can be modified or combined into other embodiments, and that many other embodiments are possible within the scope of the present embodiments as defined by the appended claims. Accordingly, the above embodiments are to be construed as illustrative only and not restrictive.

Claims

1. A method for implementing a linear-complexity self-attention mechanism, characterized in that, Comprise: S1: position coding and feature mapping of the to-be-processed sequence data are applied to the query matrix and the key matrix; wherein, the to-be-processed sequence data comprises natural language and image; S2: low-rank query matrix, low-rank key matrix and low-rank generalized inverse matrix are obtained based on the query matrix and the key matrix; wherein, the low-rank query matrix, the low-rank key matrix and the low-rank generalized inverse matrix are configured to denoise the natural language and the image; S3: the value matrix is augmented to obtain an augmented matrix, and a sequence self-attention matrix is obtained based on the augmented matrix and the low-rank key matrix, low-rank generalized inverse matrix and low-rank query matrix; S4: the sequence self-attention matrix is transposed as a sequence data, and S1-S3 are repeated, the sequence self-attention matrix obtained again is taken as a feature self-attention matrix, and a linear complexity self-attention matrix is obtained based on the sequence self-attention matrix and the feature self-attention matrix.

2. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The step of obtaining the low-rank query matrix, the low-rank key matrix and the low-rank generalized inverse matrix based on the query matrix and the key matrix comprises: In the query matrix and the key matrix, a row submatrix is selected by a probability distribution to obtain a query row submatrix and a key row submatrix; The product of the query matrix and the transpose of the key row submatrix is taken as the low-rank query matrix; The product of the key matrix and the transpose of the query row submatrix is taken as the low-rank key matrix; The product of the query matrix and the transpose of the key matrix is taken as the low-rank generalized inverse matrix.

3. The method of linear complexity self-attention mechanism implementation according to claim 2, wherein, The method further comprises that the query matrix and the key matrix obtain the query row submatrix and the key row submatrix according to a row index.

4. The method of linear complexity self-attention mechanism implementation according to claim 3, wherein, The row index is obtained by an index mode based on a vector norm, or is obtained by random index selection.

5. The method of linear complexity self-attention mechanism implementation according to claim 2, wherein, The step of selecting a row submatrix in the query matrix and the key matrix by a probability distribution to obtain a query row submatrix and a key row submatrix comprises: The row norm of each row of the query matrix and the key matrix is calculated; Softmax is applied to all row norms of each matrix to be converted into a probability distribution; According to the probability distribution, row sampling is performed; The selected row combination is extracted into the corresponding query row submatrix and key row submatrix.

6. The method of linear complexity self-attention mechanism implementation according to claim 5, wherein, The probability distribution is generated by the linear complexity self-attention mechanism by calculating the interaction between the query matrix, the key matrix and the value matrix.

7. The method of linear complexity self-attention mechanism implementation according to claim 5, wherein, The method further comprises that in the process of extracting the selected row combination into the corresponding query row submatrix and key row submatrix, the probability distribution assigns different weights to each input element, and the weights represent the importance or correlation between the input elements.

8. The method for implementing linear complexity self-attention mechanism according to claim 1, wherein, The step of augmenting the value matrix to obtain an augmented matrix, and obtaining a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, low-rank generalized inverse matrix and low-rank query matrix comprises: 1 is taken as a constant term vector, and the value matrix is combined into an augmented matrix; The augmented matrix is left multiplied by the low-rank query matrix, and the dot product is calculated element by element to obtain a first matrix; The first matrix is left multiplied by the low-rank generalized inverse matrix, and the dot product is calculated element by element to obtain a second matrix; Multiplying the second matrix and the low-rank query matrix left, calculating the dot product element by element, obtaining a self-attention matrix, and obtaining a sequence self-attention matrix based on the self-attention matrix.

9. The method of linear complexity self-attention mechanism implementation according to claim 8, wherein, The left multiplication means that the original matrix is compressed or reduced in dimension on the basis of preserving the main features and structure information in the original matrix.

10. The method of linear complexity self-attention mechanism implementation according to claim 8, wherein, The step of obtaining the sequence self-attention matrix based on the self-attention matrix comprises: The maximum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix is selected as the maximum normalization item; The minimum value of the intersection matrix of the low-rank query matrix and the low-rank key matrix is selected as the minimum normalization item; All elements of the self-attention matrix are scaled into the range of [0, 1] through the maximum normalization item and the minimum normalization item, and a normalized matrix is obtained; The normalized matrix is taken as a sequence self-attention matrix.

11. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The step of obtaining the linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix comprises: The transpose of the sequence self-attention matrix and the transpose of the feature self-attention matrix are spliced as a linear complexity self-attention matrix and output.

12. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The step of performing position encoding and feature mapping of the to-be-processed sequence data into the query matrix and the key matrix comprises: Generating position encoding according to the length of the sequence data and the dimension of the query matrix and the key matrix; Adding the position encoding to the corresponding position of the query matrix and the key matrix.

13. The method of linear complexity self-attention mechanism implementation according to claim 12, wherein, The position encoding represents the features of each element of the query matrix in the position dimension.

14. The method of linear complexity self-attention mechanism implementation according to claim 12, wherein, The position encoding is encoded based on the position attribute of the sequence data.

15. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The step of transposing the sequence self-attention matrix as sequence data to repeat S1-S3 to obtain a feature self-attention matrix comprises: Transposing the sequence self-attention matrix to obtain a transposed matrix; Taking the transposed matrix as sequence data, selecting the position encoding of the transposed matrix and mapping it into the query matrix and the key matrix; After transformation of the query matrix and the key matrix, a feature self-attention matrix is obtained through normalization processing.

16. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The to-be-processed sequence data is composed of sequence length and encoding of the sequence data in the feature dimension.

17. The method of linear complexity self-attention mechanism implementation according to claim 1, wherein, The method further comprises obtaining the low-rank generalized inverse matrix through a permutation matrix method or an iterative method.

18. A system for implementing a linear-complexity self-attention mechanism, characterized in that, Comprise: A preprocessing unit configured to perform position encoding and feature mapping of the to-be-processed sequence data into the query matrix and the key matrix; A low-rank unit configured to calculate a low-rank query matrix, a low-rank key matrix, and a low-rank generalized inverse matrix based on the query matrix and the key matrix; An approximation unit configured to augment a value matrix to obtain an augmented matrix, and obtain a sequence self-attention matrix based on the augmented matrix and the low-rank key matrix, the low-rank generalized inverse matrix, and the low-rank query matrix; An interaction unit configured to transpose the sequence self-attention matrix as sequence data to repeat calling the preprocessing unit, the low-rank unit, and the approximation unit, take the sequence self-attention matrix obtained again as a feature self-attention matrix, and obtain a linear complexity self-attention matrix based on the sequence self-attention matrix and the feature self-attention matrix.

19. A computer device comprising: at least one processor; and a memory storing a computer program executable on the processor, characterized in that the processor, when executing the program, performs the steps of the method for implementing a linear-complexity self-attention mechanism according to any one of claims 1 to 17.

20. A computer non-transitory readable storage medium, the computer non-transitory readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, performs the steps of the method for implementing a linear-complexity self-attention mechanism according to any one of claims 1 to 17.

Images