Low time complexity self-attention operation method and device

Abstract

This invention discloses a low-time-complexity self-attention computation method and apparatus, belonging to the field of artificial intelligence technology. The method includes: encoding a token sequence using input data from a sequence modeling model, where the input data includes at least one of text, image, and audio; mapping the query vector and key vector of the token sequence to a one-dimensional query sequence and key sequence, respectively; for the i-th position in the query sequence, selecting k candidate elements from the key sequence based on binary search to obtain the i-th candidate element set for the i-th position in the query sequence; calculating the sparse matrix at the i-th position; and performing a dot product operation between the sparse matrix at the i-th position and the value vector to obtain the self-attention score for the i-th position in the query sequence. This method effectively reduces the time complexity of the self-attention computation process in the sequence modeling model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a self-attention computation method and apparatus with low time complexity. Background Technology

[0002] Sequence modeling models include Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and Transformers. Self-attention mechanisms are commonly used in sequence modeling.

[0003] Taking the Transformer architecture as an example, the time complexity of the self-attention mechanism in the Transformer architecture is quadratic. ( (This refers to the length of the token sequence). When processing long sequence data such as long text and long videos, the model's resource consumption increases exponentially with the increase in sequence length.

[0004] The inference process of a large language model based on the Transformer architecture mainly consists of two stages: a pre-filling stage and a decoding stage. In the pre-filling stage, the large language model receives the user's input text, converts it into a token sequence via a token segmenter, maps it to a word vector matrix, and then uses the Transformer to compute the probability distribution of the first output token in parallel. In the inference / decoding stage, the model enters an autoregressive generation mode. Each time a new token is generated, it is appended to the end of the existing sequence, and the entire updated sequence is input into the Transformer again to predict the next token. This process is repeated until a stop symbol is generated. Due to the quadratic complexity of the self-attention mechanism, as the sequence length accumulates in the decoding stage, both the initial computational load of the pre-filling stage and the iterative efficiency of the inference / decoding stage are severely constrained. Summary of the Invention

[0005] This invention provides a self-attention computation method and apparatus with low time complexity, which can effectively reduce the time complexity of the self-attention computation process in sequence modeling models. The technical solution includes at least the following: Firstly, a low-time-complexity self-attention operation method is provided, comprising: obtaining a query vector, a key vector, and a value vector of a token sequence, wherein the token sequence is encoded through input data of a sequence modeling model, and the input data of the sequence modeling model includes at least one of text, image, and audio; mapping the query vector and the key vector to a one-dimensional query sequence and a key sequence, respectively; for the i-th position in the query sequence, selecting k candidate elements from the key sequence based on the absolute difference between the values ​​of each element in the key sequence and the value of the i-th position, thereby obtaining the i-th candidate element set for the i-th position in the query sequence, wherein the i-th position is the position for which self-attention operation is required; calculating the sparse matrix of the i-th position based on the absolute difference between the value of the i-th position and the value of each candidate element in the i-th candidate element set; and performing a dot product operation between the sparse matrix of the i-th position and the value vector to obtain the self-attention score of the i-th position in the query sequence.

[0006] Optionally, during the inference phase, the i-th position in the query sequence is the position of the last token in the token sequence. The step of filtering k candidate elements from the key sequence based on the absolute difference between the value of each element in the key sequence and the value at the i-th position, to obtain the i-th candidate element set at the i-th position in the query sequence, includes: sorting the elements in the key sequence to obtain a globally ordered sequence; and using binary search to find the k elements in the globally ordered sequence with the smallest absolute difference from the i-th position in the query sequence, to obtain the i-th candidate element set.

[0007] Optionally, during the training phase, the i-th position is not the position of the last token in the token sequence, but any position in the token sequence. The step of filtering k candidate elements from the key sequence based on the absolute difference between the values ​​of each element in the key sequence and the value at the i-th position, to obtain the i-th candidate element set at the i-th position in the query sequence, includes: dividing the key sequence into m blocks, independently sorting the elements in each block, and merging and sorting the blocks where each element has obtained a candidate element set to obtain a globally ordered sequence; using binary search. Find the 2k elements adjacent to the i-th position in the query sequence in the globally ordered sequence to obtain a first element set; for the element at the i-th position in the b-th block and the elements before the i-th position, select the k elements with the smallest absolute difference from the i-th position in the query sequence to obtain a second element set, where the i-th position is in the b-th block; merge the first element set and the second element set to obtain a third element set; select the k candidate elements with the smallest difference from the value at the i-th position in the query sequence from the third element set to obtain the i-th candidate element set.

[0008] Optionally, the step of using binary search to find the 2k elements adjacent to the i-th position in the query sequence in the globally ordered sequence to obtain the first element set includes: using binary search in the globally ordered sequence to find the target sorting position of the value at the i-th position in the query sequence in the globally ordered sequence; and taking the first k elements and the last k elements adjacent to the target sorting position in the globally ordered sequence as the first element set.

[0009] Optionally, the sparse matrix is ​​calculated using the following formula:

[0010] in, For the sparse matrix, For the query sequence The value at the i-th position in the array. The value at the j-th position in the key sequence. For temperature coefficient, Let be the set of the i-th candidate elements.

[0011] Optionally, mapping the query vector and the key vector to one-dimensional query sequences and key sequences respectively includes: using a learnable scalar matrix to map the query vector and the key vector to one-dimensional query sequences and key sequences respectively.

[0012] Secondly, a low-time-complexity self-attention computing device is also provided, comprising: an acquisition module for acquiring a query vector, a key vector, and a value vector of a token sequence, wherein the token sequence is encoded through input data of a sequence modeling model, and the input data of the sequence modeling model includes at least one of text, image, and audio; a one-dimensional mapping module for mapping the query vector and the key vector to a one-dimensional query sequence and a key sequence, respectively; and a candidate element filtering module for, for the i-th position in the query sequence, selecting the candidate element based on the value of each element in the key sequence and the i-th position. The absolute difference between the values ​​is used to filter k candidate elements from the key sequence based on binary search, resulting in the i-th candidate element set at the i-th position in the query sequence, where the i-th position is the position where self-attention operation needs to be performed; a sparse matrix calculation module is used to calculate the sparse matrix at the i-th position based on the absolute difference between the value at the i-th position and the value of each candidate element in the i-th candidate element set; a self-attention operation module is used to perform a dot product operation on the sparse matrix at the i-th position and the value vector to obtain the self-attention score at the i-th position in the query sequence.

[0013] Optionally, during the inference phase, the i-th position in the query sequence is the position of the last token in the token sequence. The candidate element filtering module is also used to sort the elements in the key sequence to obtain a globally ordered sequence. A binary search is used to find the k elements in the globally ordered sequence that have the smallest absolute difference with the i-th position in the query sequence to obtain the i-th candidate element set.

[0014] Optionally, during the training phase, the i-th position is not the position of the last token in the token sequence, but any position in the token sequence. The candidate element filtering module is further configured to divide the key sequence into m blocks, perform sorting operations independently on the elements in each block, and merge and sort the blocks where each element has obtained a candidate element set to obtain a globally ordered sequence; use binary search to find 2k elements adjacent to the i-th position in the query sequence in the globally ordered sequence to obtain a first element set; for the element at the i-th position in the b-th block and the elements before the i-th position, filter out the k elements with the smallest absolute difference from the i-th position in the query sequence to obtain a second element set, where the i-th position is in the b-th block; merge the first element set and the second element set to obtain a third element set; filter out the k candidate elements with the smallest difference from the value at the i-th position in the query sequence from the third element set to obtain the i-th candidate element set.

[0015] Optionally, the candidate element filtering module is further configured to use binary search in the globally ordered sequence to find the target sorting position of the value at the i-th position in the query sequence in the globally ordered sequence; and to use the first k elements and the last k elements in the globally ordered sequence that are adjacent to the target sorting position as the first element set.

[0016] Optionally, in the sparse matrix calculation module, the sparse matrix is ​​calculated using the following formula:

[0017] in, For the sparse matrix, For the query sequence The value at the i-th position in the array. The value at the j-th position in the key sequence. For temperature coefficient, Let be the set of the i-th candidate elements.

[0018] Optionally, the one-dimensional mapping module is further configured to map the query vector and the key vector into a one-dimensional query sequence and key sequence, respectively, using a learnable scalar matrix.

[0019] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the low-time-complexity self-attention computation method described in the above embodiments.

[0020] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the low-time-complexity self-attention operation method described in the above embodiments.

[0021] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0022] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment, by mapping the query vector and key vector of the token sequence to a one-dimensional sequence, "similar scalar values" are equated to "semantic relevance of tokens." This replaces the traditional high-dimensional vector inner product calculation with scalar difference calculation during attention weight computation, fundamentally changing the way attention weights are calculated. This significantly reduces the complexity of attention operations and solves the technical problems of high computational complexity, large memory consumption, slow processing speed, high energy consumption, and large response latency of self-attention mechanisms with long input sequences. Consequently, under the same hardware conditions, the computer system can process longer input sequences, significantly shortening the time spent on model training and inference, ultimately improving the overall resource utilization efficiency and throughput of the computing system. The token sequence in this embodiment is obtained by encoding the input data of the sequence modeling model. This sequence modeling model can be used in technical fields such as text question answering, text translation, speech recognition, speech synthesis, and text classification. Since this low-time-complexity self-attention operation method can reduce the time complexity of the sequence modeling model in the self-attention operation process, when the sequence modeling model is applied to the corresponding technical field, it can reduce the time complexity of the sequence modeling model in that field and improve the computational efficiency of the sequence modeling model in that technical field. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of a low-time-complexity self-attention computation method provided by an exemplary embodiment of the present invention is shown; Figure 2 A flowchart of a low-time-complexity self-attention computation method provided by another exemplary embodiment of the present invention is shown; Figure 3 This is a schematic diagram illustrating the selection of candidate element sets during the training phase; Figure 4 This is the data flow diagram of the low-time-complexity self-attention computation method in this invention; Figure 5 A schematic diagram of the structure of a self-attention computing device with low time complexity provided in an exemplary embodiment of the present invention is shown. Figure 6 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0025] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0027] Example 1.

[0028] Figure 1 A flowchart illustrating a low-time-complexity self-attention computation method provided by an exemplary embodiment of the present invention is shown, which can be executed by a computer device. See also Figure 1 The method includes: In step 101, the query vector, key vector, and value vector of the token sequence are obtained.

[0029] The token sequence is obtained by encoding the input data of the sequence modeling model, which includes at least one of text, images, and audio. The images here can be still images captured by a photographic device or video frames.

[0030] The sequence modeling model in this embodiment is a Transformer. The input data for this sequence modeling model can be unimodal data or multimodal data.

[0031] When the input data is unimodal, the modality of the input data can be any one of text, image, or audio.

[0032] When the input data is multimodal, the input data can be at least two of the following: text, image, or audio.

[0033] After the input data is input into the sequence modeling model, the input data is encoded using an encoder corresponding to the input data modality to obtain the token sequence.

[0034] For example, when the input data is unimodal, such as when the input data is an image, an image encoder is used to encode the input data to obtain a token sequence. In this case, the token sequence belongs to the visual token sequence.

[0035] When the input data is multimodal, for example, when the input data includes first modal data and second modal data, a first modal encoder can be used to encode the first modal data to obtain a first token sequence, and then a second modal encoder can be used to encode the second modal data to obtain a second token sequence. Finally, the first token sequence and the second token sequence are fused to obtain a fused token sequence. In this embodiment, the token sequence can be any one of the first token sequence, the second token sequence, or the fused token sequence. The first modal data and the second modal data are data of two different modalities, and both the first modal data and the second modal data are either text, image, or audio.

[0036] The sequence modeling model in this embodiment can be used in technical fields such as text question answering, text translation, speech recognition, speech synthesis, and text classification. The low-time-complexity self-attention operation method in this embodiment can reduce the time complexity of the sequence modeling model during the self-attention operation process. When this sequence modeling model is applied to the corresponding domain, it can reduce the time complexity of the sequence modeling model in that domain and improve its computational efficiency. (For example, when applying the sequence modeling model to the text question answering domain, it can reduce the time complexity of the text question answering process and improve the efficiency of text question answering).

[0037] Before executing step 101, it is necessary to first obtain the token sequence, which is the token sequence for which self-attention operations will be performed. During the training phase, the token sequence is transformed from the input data in the training set. During the inference phase, the token sequence is generated one by one by the large model. Here, during the inference phase, the large model generates tokens one by one. Each time a new token is generated, self-attention is calculated on the new token based on all the generated tokens. This sequence of all generated tokens (including historical tokens and newly generated tokens) is the token sequence.

[0038] When acquiring the token sequence during the training phase, the input data requiring self-attention operations can be converted into a token sequence. Each token in the token sequence is a d-dimensional vector, and the length of the token sequence is n.

[0039] Since self-attention operations are performed, the query vector, key vector, and value vector are derived from the token sequence. In implementation, a query projection matrix can be used to convert the token sequence into a query vector, a key projection matrix into a key vector, and a value projection matrix into a value vector.

[0040] There are many implementation methods for query projection matrices, key projection matrices, and value projection matrices in related technologies, which will not be detailed here.

[0041] In the attention model, the query vector consists of several q vectors, and the key vector consists of several k vectors. The number of q vectors and k vectors corresponds to the length of the token sequence, meaning that each q vector and each k vector corresponds to one token.

[0042] When the sequence length is n, the query vector can be represented as: The key vector can be represented as , , It is the dimension of each q vector in the query vector. It is also the dimension of each k vector in the key vector.

[0043] For any token in the token sequence, the q-vector represents the semantic features of that token, and the key vector represents the semantic features of each token in the sequence. The core function of the q-vector and the key vector is to calculate the semantic similarity (or semantic distance) between the current token and other (or historical) tokens in the token sequence, and then assign attention weights based on this similarity.

[0044] Generally, the higher the semantic similarity between the k-vector of a given token and the q-vector currently used for self-attention calculation, the greater its corresponding attention weight; that is, the more significant the semantic contribution of that token to the current token. In standard attention mechanisms, the dimensionality of the q-vector and k-vector is usually high. To calculate the self-attention weight between the query vector and the key vector, it is necessary to perform an inner product operation between each q-vector in the query vector and the entire key vector. This operation results in high time and space complexity for attention computation, making it difficult to adapt to long sequence scenarios.

[0045] In step 102, the query vector and key vector are mapped to a one-dimensional query sequence and key sequence, respectively.

[0046] Optionally, step 102 includes: mapping the query vector and key vector to a one-dimensional query sequence and key sequence, respectively, using a learnable scalar matrix.

[0047] For example, a learnable scalar matrix The dimension is .

[0048] In this case, step 102 can be represented by the following formula (1).

[0049] (1) In formula (1), For query sequence, , where n is the length of the token sequence (i.e., the number of tokens in the token sequence). For query vector, It is a learnable scalar matrix. For key sequences, , For value vectors, It is a non-linear activation function.

[0050] In the above transformation process, without a nonlinear activation function, the mapping process in formula (1) is merely a linear transformation, resulting in insufficient accuracy of the final query sequence and key sequence. By introducing a nonlinear activation function (or other activation functions) into the mapping process, the fitting accuracy of the model parameters can be improved. This invention utilizes a learnable scalar matrix... The vector corresponding to each token in the query vector and key vector is compressed and converted into a single scalar value. The learnable scalar matrix in the mapping process... It can be trained and optimized with tasks, significantly improving the quality of historical token candidates.

[0051] In step 103, for the i-th position in the query sequence, based on the absolute difference between the values ​​of each element in the key sequence and the value at the i-th position, k candidate elements are selected from the key sequence using binary search to obtain the i-th candidate element set at the i-th position in the query sequence.

[0052] The i-th position is the position in the token sequence where self-attention operation needs to be performed.

[0053] During the inference phase, tokens in the token sequence are generated one by one, and only the self-attention of the newly generated token needs to be calculated. In the inference phase, the i-th position in the query sequence is the position of the last token in the token sequence, which is the token preceding the newly generated token (which has not yet been added to the token sequence).

[0054] In this case, step 103 includes steps 1031 to 1032 as follows.

[0055] Step 1031: Sort the elements in the key sequence to obtain a globally ordered sequence.

[0056] In this embodiment, the elements of the key sequence are sorted according to the size of the value of each element in the key sequence. Since the query sequence and the key sequence are one-dimensional scalar sequences, each element in the query sequence and the key sequence is a numerical value (usually a value between 0 and 9).

[0057] The original order of the elements in the key sequence is the token order (each element corresponds to one token). For example, the original order of the elements in the key sequence is 2, 8, 5, 6, 3, with each value corresponding to one token. These elements can then be sorted according to their numerical values, resulting in the sequence 2, 3, 5, 6, 8, which is a globally ordered sequence. Of course, the original position of each element in the key sequence is also stored during the sorting process for easy restoration.

[0058] For a key sequence of length n, several comparison-based sorting algorithms (such as quicksort, mergesort, and heapsort) can be used to sort the key sequence into a globally ordered sequence. The average and worst-case time complexities of these sorting algorithms are both n. Therefore, the time complexity of step 1031 can be considered to be O(n log n). .

[0059] Step 1032: Use binary search to find the k elements in the globally ordered sequence that have the smallest absolute difference with the i-th position in the query sequence, and obtain the i-th candidate element set.

[0060] The i-th position in the query sequence is also an element (or a value). Since it's self-attention (the query sequence and the key sequence are essentially the same), the element at the i-th position in the query sequence must also exist in the globally ordered sequence. For example, if the original order of elements in the key sequence is 2, 8, 5, 6, 3, and the original order of elements in the query sequence is also 2, 8, 5, 6, 3, then the i-th position in the query sequence is the element corresponding to the last token, which is 3. When the globally ordered sequence of the key sequence is 2, 3, 5, 6, 8, step 1032 actually requires querying the position of 3 in the globally ordered sequence. After finding the position of 3, starting from that position, select 0.5k elements before and after it. This yields the k elements with the smallest absolute difference from the i-th position in the query sequence, i.e., the i-th candidate element set. For example, if k is 2, the i-th candidate element set is {2, 5}. Usually, k is a positive integer. When k is even, the above operation can be performed directly. When k is odd, 0.5k is no longer an integer. In this case, we can select 0.5(k+1) elements before and after the i-th position in the globally ordered sequence to filter out k+1 elements. Then, we can select the k elements with the smallest absolute difference from the i-th position in the query sequence from these k+1 elements to obtain the i-th candidate element set.

[0061] In this embodiment, binary search is used to implement step 1032. For a key sequence of length n, the time complexity of binary search is O(n^2). .

[0062] In summary, the total time complexity of step 103 is... This is a standard operation in time complexity arithmetic, where only the higher-order term is retained and the lower-order term is ignored when considering addition in terms of time complexity. Since the higher-order term in the above operation is... ,and This is a relatively small term, therefore the total time complexity of step 103 is O(n). .

[0063] In step 104, the sparse matrix at the i-th position is calculated based on the absolute difference between the value at the i-th position and the value of each candidate element in the i-th candidate element set.

[0064] The candidate element value is the value of an element in the candidate element set.

[0065] Alternatively, the sparse matrix can be calculated using the following formula (2).

[0066] (2) In formula (2), It is a sparse matrix. For query sequence The value at the i-th position in the array. Let j be the value at the j-th position in the key sequence. For temperature coefficient, Let i be the set of candidate elements of the i-th generation. During the training phase, the value of i ranges from 1 to n. However, during the inference phase, the i-th position of the query sequence is fixed as the position of the last token in the token sequence. Therefore, during the training phase, it is necessary to calculate the cases where i takes n different values ​​to obtain the final sparse matrix, but during the inference phase, it is only necessary to calculate the cases where i takes only one value.

[0067] Therefore, during the inference phase, sparse matrices Attention weights used to indicate the last token in the token sequence; during training, a sparse matrix Used to indicate the attention weight of the entire token sequence.

[0068] The value of the i-th row and j-th column in the sparse matrix (i.e., the attention weight of the i-th position in the query sequence to the j-th position in the key sequence) is represented by formula (3).

[0069] (3) In formula (3), To determine the attention weight of the i-th position in the query sequence to the j-th position in the key sequence, For query sequence The value of the i-th position in The value at the j-th position in the key sequence The negative absolute difference between them is expressed as Formulas (2) and (3) can be converted to each other. During the training phase, this conversion process is expressed as follows: In the reasoning stage, this process of mutual conversion is represented as... , where i indicates the position of the last token in the current token sequence.

[0070] The essence of the above formulas (2) and (3) is to equate "similar scalar values" with "semantic related tokens", thereby replacing the traditional high-dimensional vector inner product calculation with scalar difference calculation in the process of calculating attention weights, fundamentally changing the way attention weights are calculated and greatly reducing the complexity of attention operations.

[0071] Related technologies also employ dimensionality reduction of key and value vectors to decrease the complexity of self-attention computation. The core purpose of this dimensionality reduction in related technologies is to reduce the storage pressure on the KV cache caused by the continuous increase in sequence length during the model inference stage, without changing the core calculation logic of attention weights. However, this embodiment maps query vectors and key vectors to query sequences and key sequences using scalar matrices. The core purpose is to optimize the calculation process of attention weights and explicitly reduce the dimensionality of query and key vectors to 1 dimension, thereby enabling rapid calculation of semantic similarity based on scalar differences. This approach not only further reduces computational complexity but also forms a completely different attention calculation path from existing technologies (i.e., it calculates attention weights not by multiplying query and key vectors, but by calculating the difference between scalars).

[0072] In step 105, a dot product operation is performed on the sparse matrix and value vector at the i-th position to obtain the self-attention score at the i-th position in the query sequence.

[0073] Optionally, step 105 can be represented by the following formula (4).

[0074] (4) In formula (4), during the inference stage, due to the sparse matrix The attention weight used to indicate the last token in the token sequence, therefore during the inference phase. To query the self-attention score at position i in the sequence; during the training phase, due to the sparse matrix The attention weights used to indicate the entire token sequence are therefore used during the training phase. The self-attention score for the entire token sequence. It is a value vector.

[0075] The dot product operation in step 105 also has a time complexity of O(n log n). However, since the token sequence is often quite long during the large language model operation, the time complexity in step 103 becomes dominant, and the time complexity of the dot product operation is not considered a major term. Therefore, the time complexity of the dot product operation is ignored in the time complexity calculation, and the overall time complexity of the inference stage remains the same as that in step 103. .

[0076] In this embodiment, by mapping the query vector and key vector to a one-dimensional sequence, "similar scalar values" are equated to "semantic relevance of tokens." This allows scalar difference calculations to replace traditional high-dimensional vector inner product calculations during attention weight computation, fundamentally changing the way attention weights are calculated and significantly reducing the complexity of attention operations. During the inference phase, the time complexity of self-attention can be reduced to [missing information]. During the training phase, the time complexity of self-attention computation can be reduced to [missing information]. Due to the reduction in time complexity, the amount of memory access and video memory usage during the computation process is also reduced accordingly.

[0077] This invention addresses the technical problems of high computational complexity, large memory consumption, slow processing speed, high energy consumption, and large response latency of self-attention mechanisms when dealing with long input sequences by reducing the time complexity of the self-attention mechanism. As a result, under the same hardware conditions, the computer system can process longer input sequences, significantly shorten the time spent on model training and inference, and ultimately improve the overall resource utilization efficiency and throughput of the computing system.

[0078] Example 2.

[0079] Figure 2 A flowchart illustrating a low-time-complexity self-attention computation method provided by another exemplary embodiment of the present invention is shown, which can be executed by a computer device. See also Figure 2 The method includes: In step 201, the query vector, key vector, and value vector of the token sequence are obtained.

[0080] The token sequence is obtained by encoding the input data of the sequence modeling model, which includes at least one of text, image, and audio.

[0081] In step 202, the query vector and key vector are mapped to one-dimensional query sequences and key sequences, respectively.

[0082] The contents of steps 201 to 202 are the same as those of steps 101 to 102 mentioned above, and are omitted here in detail.

[0083] In step 203, for the i-th position in the query sequence, based on the absolute difference between the values ​​of each element in the key sequence and the value at the i-th position, k candidate elements are selected from the key sequence using binary search to obtain the i-th candidate element set at the i-th position in the query sequence.

[0084] The i-th position is the position in the token sequence where self-attention operation needs to be performed.

[0085] This embodiment describes step 203 in the context of the training phase. During the training phase, the length of the token sequence is fixed, and the token sequence is encoded from external input data. Therefore, each position in the token sequence requires self-attention. When the length of the token sequence is n, the value of i ranges from 1 to n, meaning that each position in the token sequence requires self-attention during the training phase.

[0086] In this case, step 203 includes steps 2031 to 2035 as follows. Figure 3 This is a diagram illustrating the selection of candidate element sets during the training phase. The following section combines... Figure 3 Step 203 will be explained.

[0087] Step 2031: Divide the key sequence into m blocks, perform sorting operations independently on the elements in each block, and merge and sort the blocks that have obtained candidate element sets to obtain a globally ordered sequence.

[0088] Let the size of each block be s, and the length of the key sequence (token sequence) be n, then , This indicates the rounding up operation.

[0089] Let the original order of the b-th block in the key sequence be... The original global index of each element in the b-th block is The scalar sequence obtained after sorting the b-th block is: Then, the sorting of each element in the b-th block within the block can be represented as follows: That is, after sorting the s elements within each block, the original global index of each element is still stored to ensure that the position of the element can be accurately located from the original sequence later.

[0090] When filtering the candidate element set, the elements in the key sequence (original sequence) are filtered one by one in their original order. When a candidate element set has been obtained for each element in a block, the block is considered processed and can be merged into the globally ordered sequence.

[0091] In the initial case (before the first block is processed), the global ordered sequence does not exist. As candidate element sets are gradually filtered for each element in the key sequence, blocks will be added to the global ordered sequence until all blocks and all elements have been filtered to obtain candidate element sets. At this point, step 204 can be executed.

[0092] like Figure 3As shown, the key sequence is divided into 3 blocks, with elements of the same color belonging to the same block. Initially, all three blocks are sorted within each block. The i-th position of the candidate element set to be searched is the element "9" in the third block. This means that each element in the first two blocks has already obtained its corresponding candidate element set, thus the elements in the first two blocks are sorted into a globally ordered sequence.

[0093] Step 2031 mainly includes two operations: one is to divide the key sequence into blocks and sort each block within the block, and the other is to sort the globally ordered sequence.

[0094] For intra-block sorting, if the key sequence is divided into m blocks, and each block contains s elements (the last block may contain fewer than s elements, but we estimate it as s), then the time complexity of intra-block sorting for a single block is O(n). The total time complexity of sorting each of the m blocks is O(m). .because Therefore, the time complexity of the sorting operation within the block can be written as: .

[0095] When processing the b-th block, the elements of the first b-1 blocks need to be arranged into a globally ordered sequence. Given a total of t elements to be sorted, the time complexity of sorting t elements into a globally ordered sequence is O(t). If all m blocks are arranged into a globally ordered sequence, then the total number of elements that need to be processed in the total ordering is approximately Therefore, the time complexity of arranging all m blocks into a globally ordered sequence can be denoted as . Considering that constant terms are usually omitted during time complexity calculations, the time complexity of arranging all m blocks into a globally ordered sequence can be simplified to... .

[0096] Step 2032: Use binary search to find the 2k elements that are adjacent to the i-th position in the query sequence in the globally ordered sequence, and obtain the first element set.

[0097] Optionally, step 2032 includes the following two steps.

[0098] The first step is to use binary search in the globally ordered sequence to find the target sorting position of the value at position i in the query sequence within the globally ordered sequence.

[0099] As mentioned in step 1032 above, using binary search to search for the i-th position in the globally ordered sequence is actually finding the sorting position of the i-th position in the query sequence within the globally ordered sequence, i.e., the target sorting position.

[0100] like Figure 3As shown, the globally ordered sequence is: 1, 2, 2, 3, 3, 4, 5, 5, 6, 8. The i-th position in the query sequence is 9. Using binary search, we can find that the target sorting position of 9 in the globally ordered sequence is the last one, represented as: 1, 2, 2, 3, 3, 4, 5, 5, 6, 8, 9.

[0101] The second step is to take the first k elements and the last k elements in the globally ordered sequence that are adjacent to the target sorting position as the first element set.

[0102] If there are only a elements before the target sorting position and a is less than k, then the second step becomes: take the first a elements and the last 2k-a elements of the globally ordered sequence that are adjacent to the target sorting position as the first element set.

[0103] If there are only c elements after the target sorting position and c is less than k, then the second step becomes: take the last c elements and the first 2k-c elements in the globally ordered sequence that are adjacent to the target sorting position as the first element set.

[0104] like Figure 3 In the case where k is 1, there are only 0 elements after the target sorting position, meaning c is 0 and c is less than k. Therefore, we need to take the first 2k elements adjacent to the target sorting position in the globally ordered sequence as the first element set. Figure 3 The final set of elements obtained is {6, 8}.

[0105] Since the size of a single block is s, and the current block is the b-th block, there are b-1 blocks in the globally ordered sequence, and the length of the globally ordered sequence is... At this point, the time complexity of performing a binary search in the globally ordered sequence in step 2032 is . And because Therefore, the time complexity of step 2032 can be simplified to: .

[0106] Step 2033: For the element at position i in the b-th block and the elements before position i, select the k elements with the smallest absolute difference from position i in the query sequence to obtain the second element set.

[0107] The i-th position is within the b-th block.

[0108] It should be noted that the i-th position within the b-th block mentioned in step 2033 refers to the i-th position in the original order of the b-th block.

[0109] like Figure 3As shown, b is 3, k is 1, and the original order of the b-th block is 5, 9, 2, 0, 3. Then the element at position i within the b-th block and the elements preceding it are 5 and 9. With k = 1, the element with the smallest absolute difference between 5 and 9 and the 9 at position i in the query sequence is 9. Therefore, this... Figure 3 The second set of elements in is {9}.

[0110] In obtaining the second element set, it is necessary to traverse at most s elements in the second element set, therefore the time complexity of step 2033 is O(n log n). .

[0111] In summary, for a single position in the query sequence, the time complexity required to filter out the first set of elements and the second set of elements to obtain the candidate set is O(n). Therefore, during the training phase, for the entire token sequence of length n, the time complexity required to filter out the first and second element sets and thus obtain the candidate element set is O(n). .

[0112] Related technologies also include causal autoregressive models, which enable efficient autoregressive training and also require self-attention operations. The core of a causal autoregressive model is the causal attention mask, which masks information about future positions, ensuring that the current step can only access historical information (i.e., strict temporal causality). For example, when the model generates the t-th word, it can only see the previous t-1 words, ensuring autoregressivity.

[0113] In this embodiment, step 203 ensures that each candidate element set selected during the training phase conforms to causal constraints. Here, the candidate element sets selected during the training phase are essentially obtained from the first element set and the second element set. The first element set is selected from a globally ordered sequence, which consists of elements from blocks preceding the current block (the b-th block). Therefore, each element in the first element set must be before the i-th position. Furthermore, the second element set itself only acquires elements within the b-th block that are before the i-th position; thus, the second element set also conforms to causal constraints. Therefore, the candidate element sets selected based on the first and second element sets also conform to causal constraints. Thus, this embodiment achieves self-attention computation that satisfies causal constraints; that is, the self-attention operation method in this embodiment can be adapted to self-attention operations in causal autoregressive models.

[0114] Existing linear attention mechanisms cannot efficiently adapt to causal masks during the training of causal autoregressive models. In the aforementioned embodiment one, the self-attention operation during the inference phase naturally conforms to causal constraints since the i-th position is the last position in the token sequence, requiring no additional processing. In this embodiment (embodiment two), steps 2031 to 2035 ensure that the i-th candidate element set conforms to causal constraints. Therefore, whether in the training or inference phase, the self-attention process of this invention can satisfy causal constraints and adapt to causal masks in causal regression models, thereby reducing computational complexity while ensuring the model's performance in long sequence tasks.

[0115] Step 2034: After merging the first set of elements and the second set of elements, a third set of elements is obtained.

[0116] like Figure 3 As shown, the first set of elements is {6, 8}, the second set of elements is {9}, and the third set of elements obtained by merging the first and second sets of elements is {6, 8, 9}.

[0117] Step 2035: Select the k candidate elements with the smallest difference from the value at the i-th position in the query sequence from the third element set to obtain the i-th candidate element set.

[0118] The first element set contains 2k elements, the second element set contains a minimum of 0 elements and a maximum of s elements, therefore the total number of elements in the third element set must be greater than k. Thus, step 2035 is needed to select the k candidate elements with the smallest difference between the values ​​at the i-th position in the query sequence from the third set, obtaining the i-th candidate element set.

[0119] For example, such as Figure 3 As shown, when k is 1, we need to select the element with the smallest difference from the i-th position "9" in the query sequence from the third element set {6, 8, 9}, which is the element "9". At this time, the i-th candidate element set is {9}.

[0120] Combining the time complexity terms of step 203 above, the total time complexity of step 203 can be expressed as follows: When s takes When choosing the optimal block size, , ,and , All below Therefore, these two items can also be recorded as Based on the above derivation, the total time complexity of step 203 can be simplified to... Based on this, since the constant in the time complexity calculation is negligible, the total time complexity of step 203 can be finally obtained as follows: .

[0121] Optionally, when implementing step 103 or step 203, an approximate globally optimal filtering scheme, or a combination of local window sequences and historical sequence candidates, can be used to filter the i-th candidate element set. The filtering only needs to satisfy the following conditions: each candidate element in the i-th candidate element set is one of the top k elements with the smallest absolute difference to the i-th position in the query sequence, and the causal constraint is met. The final implementation effect of this type of scheme is close to that of the current scheme.

[0122] By performing step 203 above on the values ​​at each position in the query sequence in a homogeneous manner, a set of candidate elements at each position in the query sequence can be obtained.

[0123] In step 204, the sparse matrix of the query sequence is calculated based on the absolute difference between the value at each position in the query sequence and the value of each candidate element in the candidate element set corresponding to each position.

[0124] Optionally, the sparse matrix during the training phase is calculated using formula (5).

[0125] (5) The meaning of the parameter in formula (5) is the same as that of the parameter in formula (2) and formula (3), and will not be elaborated here.

[0126] In step 205, a dot product operation is performed on the sparse matrix and value vector of the query sequence to obtain the self-attention score of the query sequence.

[0127] The details of step 205 are the same as those of step 105 mentioned above, and will not be described in detail here.

[0128] The dot product operation in step 205 also has a time complexity of O(n log n). However, since the token sequence is often quite long during the large language model operation, the time complexity in step 203 becomes dominant, and the time complexity of the dot product operation is not considered a major term. Therefore, the time complexity of the dot product operation is ignored in the time complexity calculation, and the overall time complexity during the training phase remains the same as that in step 203. .

[0129] The low-time-complexity self-attention computation method in this invention is compared with other attention computation schemes in related technologies.

[0130] (1) The time complexity of standard self-attention is In long sequence scenarios, training and inference consume enormous resources. The low-time-complexity self-attention computation method in this invention has a time complexity of O(n log n) during the inference phase. The time complexity during the training phase is All of these are superior to standard self-attention, and perform even better in long sequence scenarios (such as when n is much greater than 1000).

[0131] (2) NSA (Native Sparse Attention) has a complex structure and low robustness, integrating multiple architectures such as compression, selection, and sparsity within a single module. The low-time-complexity self-attention operation method in this invention has a simple and reliable architecture, requiring only three architectures in total: step 102, which uses a learnable scalar matrix to map the query vector and key vector to a one-dimensional query sequence and key sequence, respectively; steps 103 and 203, which construct a globally ordered sequence; and steps 104 to 105, which involve sparse matrix multiplication.

[0132] (3) DSA (DeepSeek Sparse Attention) reduces the time complexity of self-attention operations by introducing a sparse indexer, but the time complexity of the sparse indexer in DSA is still O(n log n). The low-time-complexity self-attention computation method in this invention has a time complexity of O(n) during the inference phase. The time complexity during the training phase is Compared to DSA, the time complexity is significantly reduced.

[0133] (4) The internal state space of linear attention is limited in size, which restricts the performance of long sequences. The internal state space of the low-time-complexity self-attention operation method in this invention is the entire KV cache (e.g., a globally ordered sequence), which can still retrieve historical related information in long sequence scenarios.

[0134] (5) Sparse attention algorithms based on KV caching (such as Minference and Tokenselect) can only accelerate the inference stage, and their performance is easily affected by manually constructed token selection strategies. The low-time-complexity self-attention operation method in this invention can reduce the time complexity not only in the inference stage, but also in the training stage, achieving dual acceleration in both the inference and training stages.

[0135] Figure 4 This is the data flow diagram of the low-time-complexity self-attention computation method in this invention. For example... Figure 4As shown, this invention only performs dimensionality reduction on the query vector and key vector, while the value vector remains unchanged. The sparse matrix is ​​also obtained through absolute negative difference calculation, which is equivalent to optimizing the original method of obtaining the sparse matrix (dot product operation between query vector and key vector) into difference operation, thereby significantly reducing the time complexity.

[0136] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0137] Example 3.

[0138] Figure 5 A schematic diagram of a low-time-complexity self-attention computing device provided by an exemplary embodiment of the present invention is shown. See also Figure 5 The low-time-complexity self-attention computing device 500 includes: an acquisition module 501, a one-dimensional mapping module 502, a candidate element filtering module 503, a sparse matrix calculation module 504, and a self-attention computing module 505.

[0139] The acquisition module 501 is used to acquire the query vector, key vector and value vector of the token sequence. The token sequence is obtained by encoding the input data of the sequence modeling model. The input data of the sequence modeling model includes at least one of text, image and audio. The one-dimensional mapping module 502 is used to map the query vector and the key vector into a one-dimensional query sequence and a key sequence, respectively; The candidate element filtering module 503 is used to filter k candidate elements from the key sequence based on the absolute difference between the value of each element in the key sequence and the value of the i-th position in the query sequence, and obtain the i-th candidate element set at the i-th position in the query sequence. The i-th position is the position where self-attention operation needs to be performed. The sparse matrix calculation module 504 is used to calculate the sparse matrix at the i-th position based on the absolute difference between the value at the i-th position and the value of each candidate element in the i-th candidate element set; The self-attention operation module 505 is used to perform a dot product operation on the sparse matrix and value vector at the i-th position to obtain the self-attention score at the i-th position in the query sequence.

[0140] Optionally, during the inference phase, the i-th position in the query sequence is the position of the last token in the token sequence. The candidate element filtering module 503 is also used to sort the elements in the key sequence to obtain a globally ordered sequence. A binary search is used to find the k elements in the globally ordered sequence that have the smallest absolute difference with the i-th position in the query sequence to obtain the i-th candidate element set.

[0141] Optionally, during the training phase, the i-th position is not the position of the last token in the token sequence, but any position in the token sequence. The candidate element filtering module 503 is also used to divide the key sequence into m blocks, perform sorting operations independently on the elements in each block, and merge and sort the blocks where each element has obtained a candidate element set to obtain a globally ordered sequence; use binary search to find the 2k elements adjacent to the i-th position in the query sequence in the globally ordered sequence to obtain the first element set; for the element at the i-th position in the b-th block and the elements before the i-th position, filter out the k elements with the smallest absolute difference from the i-th position in the query sequence to obtain the second element set, where the i-th position is in the b-th block; merge the first element set and the second element set to obtain the third element set; filter out the k candidate elements with the smallest difference from the value at the i-th position in the query sequence from the third element set to obtain the i-th candidate element set.

[0142] Optionally, the candidate element filtering module 503 is also used to find the target sorting position of the value at position i in the query sequence in the global ordered sequence by using binary search in the global ordered sequence; and to take the first k elements and the last k elements in the global ordered sequence that are adjacent to the target sorting position as the first element set.

[0143] Optionally, in the sparse matrix calculation module 504, the sparse matrix is ​​calculated using the following formula:

[0144] in, It is a sparse matrix. For query sequence The value at the i-th position in the array. Let j be the value at the j-th position in the key sequence. For temperature coefficient, Let i be the set of candidate elements.

[0145] Optionally, the one-dimensional mapping module 502 is also used to map the query vector and key vector into a one-dimensional query sequence and key sequence, respectively, using a learnable scalar matrix.

[0146] It should be noted that the low-time-complexity self-attention computing device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the low-time-complexity self-attention computing device and the low-time-complexity self-attention computing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0147] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of the invention can be integrated into a single processor, exist as separate physical entities, or consist of two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0148] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the method of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] Figure 6 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. For example... Figure 6 As shown, the computer device 600 includes a processor 601 and a memory 602.

[0150] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0151] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to implement the low-time-complexity self-attention computation method provided in the embodiments of the present invention.

[0152] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0153] This invention also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of a computer device, enables the computer device to execute the low-time-complexity self-attention computation method provided in this invention.

[0154] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the low-time-complexity self-attention computation method provided in this invention.

[0155] The above description is merely an optional embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A self-attention computation method with low time complexity, characterized in that, The method includes: Obtain the query vector, key vector, and value vector of the token sequence, wherein the token sequence is encoded through the input data of the sequence modeling model, and the input data of the sequence modeling model includes at least one of text, image, and audio. The query vector and the key vector are mapped to a one-dimensional query sequence and a key sequence, respectively; For the i-th position in the query sequence, based on the absolute difference between the value of each element in the key sequence and the value of the i-th position, k candidate elements are selected from the key sequence using binary search to obtain the i-th candidate element set for the i-th position in the query sequence, where the i-th position is the position where self-attention operation needs to be performed. Calculate the sparse matrix at the i-th position based on the absolute difference between the value at the i-th position and the value of each candidate element in the i-th candidate element set; Perform a dot product operation between the sparse matrix at the i-th position and the value vector to obtain the self-attention score at the i-th position in the query sequence.

2. The self-attention computation method with low time complexity according to claim 1, characterized in that, During the inference phase, the i-th position in the query sequence is the position of the last token in the token sequence. Based on the absolute difference between the values ​​of each element in the key sequence and the value at the i-th position, k candidate elements are selected from the key sequence using binary search, resulting in the set of candidate elements at the i-th position in the query sequence, including: Sort the elements in the key sequence to obtain a globally ordered sequence; A binary search is used to find the k elements in the globally ordered sequence that have the smallest absolute difference with the i-th position in the query sequence, thus obtaining the i-th candidate element set.

3. The low-time-complexity self-attention computation method according to claim 1, characterized in that, During the training phase, the i-th position is not the position of the last token in the token sequence, but any position in the token sequence. Based on the absolute difference between the values ​​of each element in the key sequence and the value at the i-th position, k candidate elements are selected from the key sequence using binary search, resulting in the set of candidate elements at the i-th position in the query sequence, including: The key sequence is divided into m blocks. The elements in each block are sorted independently. The blocks in which each element has obtained a candidate element set are merged and sorted to obtain a globally ordered sequence. A binary search is used to find the 2k elements that are adjacent to the i-th position in the query sequence in the globally ordered sequence, and the first element set is obtained. For the element at position i within the b-th block and the elements preceding position i, select the k elements with the smallest absolute difference from position i in the query sequence to obtain the second element set, wherein position i is within the b-th block; After merging the first set of elements with the second set of elements, a third set of elements is obtained; From the third element set, select the k candidate elements with the smallest difference from the value at the i-th position in the query sequence to obtain the i-th candidate element set.

4. The low-time-complexity self-attention computation method according to claim 3, characterized in that, The binary search is used to find the 2k elements adjacent to the i-th position in the query sequence in the globally ordered sequence, resulting in a first set of elements including: A binary search is used to find the target sorting position of the value at the i-th position in the query sequence within the globally ordered sequence. The first set of elements is the first set of elements consisting of the first k elements and the last k elements in the globally ordered sequence that are adjacent to the target sorting position.

5. The low-time-complexity self-attention computation method according to any one of claims 1 to 4, characterized in that, The sparse matrix is ​​calculated using the following formula: in, For the sparse matrix, For the query sequence The value at the i-th position in the array. The value at the j-th position in the key sequence. For temperature coefficient, Let be the set of the i-th candidate elements.

6. The low-time-complexity self-attention computation method according to any one of claims 1 to 4, characterized in that, The step of mapping the query vector and the key vector to a one-dimensional query sequence and key sequence, respectively, includes: The query vector and the key vector are mapped to one-dimensional query sequences and key sequences respectively using learnable scalar matrices.

7. A self-attention computing device with low time complexity, characterized in that, The device includes: The acquisition module is used to acquire the query vector, key vector, and value vector of the token sequence. The token sequence is obtained by encoding the input data of the sequence modeling model, and the input data of the sequence modeling model includes at least one of text, image, and audio. A one-dimensional mapping module is used to map the query vector and the key vector into a one-dimensional query sequence and a key sequence, respectively; The candidate element filtering module is used to filter k candidate elements from the key sequence based on the absolute difference between the value of each element in the key sequence and the value of the i-th position in the query sequence, and obtain the i-th candidate element set at the i-th position in the query sequence, where the i-th position is the position where self-attention operation needs to be performed. The sparse matrix calculation module is used to calculate the sparse matrix at the i-th position based on the absolute difference between the value at the i-th position and the value of each candidate element in the i-th candidate element set; The self-attention operation module is used to perform a dot product operation between the sparse matrix at the i-th position and the value vector to obtain the self-attention score at the i-th position in the query sequence.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the low-time-complexity self-attention computation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the low-time-complexity self-attention computation method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the low-time-complexity self-attention computation method according to any one of claims 1 to 6.

Images