Efficient sparse attention algorithm optimization methods, devices, equipment and media

Abstract

This application relates to an efficient sparse attention algorithm optimization method, apparatus, device, and medium. The method includes: acquiring data to be processed and performing an importance-aware layernorm calculation on the data to obtain normalized data; performing row-product sparse matrix multiplication and group hashing multi-head attention processing on the normalized data to obtain an attention score matrix; performing a shift-based softmax calculation on the attention score matrix to obtain a softmax calculation result; and performing row-product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result. Throughout this process, the sparse attention algorithm is optimized through improved layernorm, softmax, and matrix multiplication, reducing the amount of data processed by the attention algorithm and significantly improving its efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an efficient sparse attention algorithm optimization method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] Attention algorithms are mechanisms used in sequence models to calculate the importance of each element in an input sequence. They allow the model to dynamically focus on information from different parts of the sequence while processing it. The core idea is to calculate a weight, or attention score, at each position in the sequence, representing the importance of that position in the current context.

[0003] However, attention algorithms require a lot of computational resources. Standard attention mechanisms have quadratic time complexity and require considerable storage and computational overhead, which makes their deployment on edge devices and efficient inference challenging.

[0004] Therefore, there is an urgent need for an effective optimization scheme for attention algorithms to significantly improve their efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a highly efficient sparse attention algorithm optimization method, device, computer equipment, storage medium, and computer program product that can significantly improve the efficiency of attention algorithms, addressing the aforementioned technical problems.

[0006] Firstly, this application provides an efficient optimization method for sparse attention algorithms. The method includes:

[0007] Acquire the data to be processed, and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data;

[0008] The normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix;

[0009] Based on the attention score matrix, a softmax calculation based on a shift operation is performed to obtain the softmax calculation result;

[0010] The attention calculation result is obtained by performing row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix.

[0011] In one embodiment, the step of acquiring the data to be processed and performing an importance-aware layernorm calculation on the data to be processed to obtain normalized data includes:

[0012] Obtain the data to be processed;

[0013] Calculate the absolute value of each element in the data to be processed;

[0014] The elements with the pre-defined percentage before the absolute value are retained, and the remaining elements are set to zero to obtain the adjusted data;

[0015] The adjusted data is subjected to layernorm calculation to obtain normalized data.

[0016] In one embodiment, performing layernorm calculation on the adjusted data to obtain normalized data includes:

[0017] Identify the non-zero elements in the adjusted data and calculate the square of the non-zero elements;

[0018] If the square of the current non-zero element is less than the preset first threshold, then the current non-zero element is set to zero.

[0019] The sum of the element values ​​and squares of the non-zeroed elements after adjustment;

[0020] Based on the element value and the square value, the normalized layernorm output result for each element is obtained.

[0021] In one embodiment, the step of performing row-directed product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain an attention score matrix includes:

[0022] Apply a linear transformation to the normalized data to obtain the query vector, key vector, and value vector;

[0023] CSR compression is performed on the query vector, key vector, and value vector;

[0024] Calculate the dot product matrix of the query and the key based on the query vector and the key vector;

[0025] The dot product matrix is ​​hashed and grouped. The query vector and key vector in each hash group are multiplied by row product sparse matrix multiplication and summed to obtain a row of the output matrix.

[0026] The output matrix is ​​rearranged to obtain the attention score matrix.

[0027] In one embodiment, hashing the dot product matrix includes:

[0028] Hash the non-zero query vectors and key vectors in the dot product matrix multiple times;

[0029] The query vector and key vector are assigned to different hash groups without duplication.

[0030] In one embodiment, the step of performing a shift-based softmax calculation based on the attention score matrix to obtain the softmax calculation result includes:

[0031] The attention score matrix is ​​divided into several groups;

[0032] Find the maximum value within each group and obtain a preset second threshold, which is determined based on the overall tensor corresponding to all data within the group;

[0033] Calculate the difference between each element in the group and the maximum value in the group. If the difference is greater than the preset second threshold, set the corresponding element to zero.

[0034] Calculate the difference raised to the power of 2 to obtain an intermediate result;

[0035] Find the maximum value among all elements based on the maximum value within each group;

[0036] The intermediate results are compensated by the maximum value among all elements, and the softmax calculation is completed by a shift operation to obtain the softmax calculation result.

[0037] In one embodiment, dividing the attention score matrix into several groups includes:

[0038] The attention score matrix is ​​scaled by dividing it by the dimension of the key matrix to obtain the scaled attention score matrix.

[0039] The scaled attention score matrix is ​​divided into several groups.

[0040] Secondly, this application also provides an efficient sparse attention algorithm optimization device. The device includes:

[0041] The normalization module is used to acquire the data to be processed and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data.

[0042] The attention processing module is used to perform row product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain an attention score matrix;

[0043] The softmax calculation module is used to perform softmax calculation based on the attention score matrix and obtain the softmax calculation result.

[0044] The attention calculation module is used to perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result.

[0045] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0046] Acquire the data to be processed, and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data;

[0047] The normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix;

[0048] Based on the attention score matrix, a softmax calculation based on a shift operation is performed to obtain the softmax calculation result;

[0049] The attention calculation result is obtained by performing row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix.

[0050] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0051] Acquire the data to be processed, and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data;

[0052] The normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix;

[0053] Based on the attention score matrix, a softmax calculation based on a shift operation is performed to obtain the softmax calculation result;

[0054] The attention calculation result is obtained by performing row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix.

[0055] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0056] Acquire the data to be processed, and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data;

[0057] The normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix;

[0058] Based on the attention score matrix, a softmax calculation based on a shift operation is performed to obtain the softmax calculation result;

[0059] The attention calculation result is obtained by performing row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix.

[0060] The aforementioned efficient sparse attention algorithm optimization method, apparatus, computer device, storage medium, and computer program product acquire the data to be processed and perform importance-aware layernorm calculation on the data to be processed to obtain normalized data; perform row product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain an attention score matrix; perform shift-based softmax calculation on the attention score matrix to obtain the softmax calculation result; and perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result. Throughout the process, by optimizing the sparse attention algorithm through improved layernorm, softmax, and matrix multiplication, the amount of data processed by the attention algorithm is reduced, significantly improving the efficiency of the attention algorithm. Attached Figure Description

[0061] Figure 1 This is a diagram illustrating the application environment of an efficient sparse attention algorithm optimization method in one embodiment.

[0062] Figure 2 This is a flowchart illustrating an efficient sparse attention algorithm optimization method in one embodiment;

[0063] Figure 3 This is a flowchart illustrating an efficient sparse attention algorithm optimization method in another embodiment;

[0064] Figure 4 This is a block diagram of a high-efficiency sparse attention algorithm optimization device in one embodiment;

[0065] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] The efficient sparse attention algorithm optimization method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 sends an attention algorithm optimization request to server 104. Server 104 pre-loads the original attention algorithm. In response to the attention algorithm optimization request, server 104 obtains the data to be processed and performs layernorm calculation based on importance awareness to obtain normalized data. It then performs row product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain an attention score matrix. Based on the attention score matrix, it performs softmax calculation based on shift operations to obtain the softmax calculation result. Finally, it performs row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0068] In one embodiment, such as Figure 2 As shown, an efficient optimization method for sparse attention algorithms is provided, which is then applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0069] S200: Obtain the data to be processed and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data.

[0070] The data to be processed is typically sequential data, such as text or time series, represented as tensors. Before LayerNorm, the weights of different elements could be determined based on certain characteristics of the data (such as location or contextual relevance), thus guiding LayerNorm to focus more on important elements. Specifically, the absolute value of each element in the data could be calculated to determine its importance. Then, data filtering was performed based on importance, selecting the most important data for layernorm calculation to obtain normalized data. Layernorm specifically refers to layer normalization of the input data, which helps accelerate the training process and improve model stability. Importance-aware LayerNorm can introduce importance weights by modifying the calculation method of the mean and variance in the normalization process.

[0071] S400: Perform row product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain the attention score matrix.

[0072] Row-product sparse matrix multiplication: In attention mechanisms, this typically involves multiplying a large key-value matrix and a query matrix. Sparse matrix multiplication reduces computation by only computing the important or expectedly large pairs of elements. The row-product is a specific form of matrix multiplication where the computation of each row can be performed independently. Grouped hashing multi-head attention: This is a variation of the multi-head attention mechanism where memory usage and computational complexity are reduced by grouping the input data and using a hash function to map the elements of each group to different "heads." Each head independently computes an attention score, and these scores are then combined to generate the final attention output. After the above processing, an attention score matrix is ​​obtained.

[0073] S600: Perform softmax calculation based on shift operation according to the attention score matrix to obtain the softmax calculation result.

[0074] Softmax is commonly used to convert attention scores into probability distributions, but standard softmax computations are typically very time-consuming on large-scale datasets. To address this time-consuming nature of traditional softmax computations, a shift operation is performed here to significantly reduce data processing volume and improve efficiency. Specifically, by leveraging the properties of the softmax function (such as maximum value shift invariance), the computation is simplified by first finding the maximum value in the score vector and then shifting all scores accordingly (i.e., subtracting the maximum value). Next, efficient exp and sum operations are performed using the shifted scores, and finally, a single division operation is used to reconstruct the probability distribution.

[0075] S800: Perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result.

[0076] The softmax result (i.e., the attention weights) is multiplied by the value vectors in the attention matrix to obtain the final attention calculation result. Since both the softmax result and the value vectors may be sparse, sparse matrix multiplication can be used to speed up the computation.

[0077] The aforementioned efficient sparse attention algorithm optimization method acquires the data to be processed and performs importance-aware layernorm calculation on the data to obtain normalized data. Then, it performs row-product sparse matrix multiplication and group hashing multi-head attention processing on the normalized data to obtain an attention score matrix. Based on the attention score matrix, it performs shift-based softmax calculation to obtain the softmax result. Finally, it performs row-product sparse matrix multiplication on the softmax result and the median vector of the attention matrix to obtain the attention calculation result. Throughout this process, by optimizing the sparse attention algorithm through improved layernorm, softmax, and matrix multiplication, the amount of data processed by the attention algorithm is reduced, significantly improving the efficiency of the attention algorithm.

[0078] In practical applications, suppose we have a text processing task where the input is a sequence containing word IDs. We need to calculate the attention representation of this sequence. The specific processing steps of the efficient sparse attention algorithm optimized in this application are as follows:

[0079] 1. Data preprocessing: Convert word IDs into embedding vectors and apply location-aware importance-aware LayerNorm.

[0080] 2. Sparse Multi-Head Attention: Divide the embedding vector into multiple heads; use a hash function to assign different keys and values ​​to each head; perform sparse matrix multiplication on each head to calculate the dot product between the query and the key to obtain an attention score; combine the scores and apply the grouped hash multi-head attention mechanism.

[0081] 3. Softmax based on shift operations; shift the maximum value of the attention score for each head; apply exp and sum operations, and then divide by the sum to obtain the probability distribution.

[0082] 4. Result Calculation: Perform sparse matrix multiplication on the softmax result (attention weights) and the corresponding value vector. Combine the outputs of all heads to obtain the final attention calculation result.

[0083] like Figure 3 As shown, in one embodiment, S200 includes:

[0084] S220: Obtain the data to be processed.

[0085] The data to be processed is usually sequential data, such as text and time series, and is represented in the form of tensors.

[0086] S240: Calculate the absolute value of each element in the data to be processed.

[0087] To assess the importance of each element, we first calculate the absolute value of each element. The magnitude of the absolute value can be considered a simple measure of the element's "importance".

[0088] S260: Retain the element with the preset percentage position before the absolute value, and set the remaining elements to zero to obtain the adjusted data.

[0089] The elements are sorted based on their absolute values, and the elements with the largest absolute values ​​that make up a predetermined percentage (e.g., the top 10%, 20%, 30%) are retained, while the values ​​of other elements are set to zero. The purpose of this step is to highlight the "important" elements while ignoring those that are relatively unimportant. It's important to note that "importance" here is defined based on absolute value and may not fully reflect the actual importance of an element in the data.

[0090] S280: Perform layernorm calculation on the adjusted data to obtain normalized data.

[0091] Finally, LayerNorm is calculated on the adjusted data. LayerNorm typically normalizes each sample (or each element in a "layer") to have zero mean and unit variance (or adjusted as needed). Since the data has already been adjusted according to the importance of the elements, this normalization step focuses more on those important elements that are retained.

[0092] Furthermore, performing layernorm calculations on the adjusted data to obtain normalized data includes: identifying non-zero elements in the adjusted data and calculating the square of the non-zero elements; if the square of the current non-zero element is less than a preset first threshold, then setting the current non-zero element to zero; accumulating the element values ​​and square values ​​of the adjusted non-zero elements; and obtaining the normalized layernorm output result for each element based on the element values ​​and square values.

[0093] In a specific application example, the entire layernorm calculation process includes the following steps:

[0094] 1. After inputting the data, calculate the absolute value of each element, keep the first 20% of the data, and set the rest of the data to zero to skip subsequent calculations. The sparsity here can be adjusted according to the data distribution.

[0095] 2. Calculate the square of each non-zero element, x. 2 (After squaring the data, the differences between the elements are amplified. In the subsequent normalization operation, only a few data points contribute significantly to the result.) If the square value of an element is less than the threshold, the element is set to zero and the calculation of subsequent steps is skipped.

[0096] 3. Accumulate all input element values ​​and calculate Ex.

[0097] 4. Sum the squares of the non-zero elements to calculate E(x). 2 ).

[0098] 5. When all elements have been calculated, i.e., Ex and E(x) 2 After outputting, we obtain the normalized layernorm output results for each element.

[0099] In one embodiment, the normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix including:

[0100] Step 1: Apply a linear transformation to the normalized data to obtain the query vector, key vector, and value vector.

[0101] Normalized data (typically an embedded representation of sequence data) is transformed into three distinct spaces to generate query, key, and value vectors. This is the standard procedure for multi-head attention mechanisms. Different linear layers (typically fully connected layers) are used to transform the normalized data, each with different weights and biases. Specifically, the attention matrix is ​​directly obtained after applying the linear transformation.

[0102] Step 2: Perform CSR compression on the query vector, key vector, and value vector.

[0103] CSR compression is performed on the attention matrix containing query vectors, key vectors, and value vectors. Specifically, the entire CSR compression process includes: 1) Representing each element in the matrix with a tuple (value, column number) and a row offset (distinct from the triple storage used in COO). The value refers to the non-zero value in the matrix, the column number refers to the column containing the non-zero value, and the row offset represents the starting offset position of the first element in a certain row within the value tuple. 2) First, fill the non-zero values ​​of the first row into the value tuple, and simultaneously fill in the corresponding column number in the column number tuple; fill in the starting offset position of the first element in each row within the value tuple in the row offset tuple, and finally append the total number of elements in the matrix to the end of the row offset tuple.

[0104] Step 3: Calculate the dot product matrix of the query vector and the key vector.

[0105] The similarity between query vectors and key vectors is evaluated by calculating the dot product between them, which is the basis of the attention score; for each query vector, it is multiplied by all key vectors to generate a dot product matrix.

[0106] Step 4: Perform hash grouping on the dot product matrix, perform row product sparse matrix multiplication on the query vector and key vector in each hash group, and sum them to obtain one row of the output matrix.

[0107] Query and key vectors are grouped using hashing to reduce computation, and sparse matrix multiplication is performed within each hash group to approximate the calculation of the attention score. A hash function is applied to the index (or some kind of hashable identifier) ​​of each query and key vector, assigning them to different groups. Within each hash group, only the dot product of the query vector and key vector within that group is calculated, and sparse matrix multiplication is used (if the data is sparse) to reduce unnecessary computation. The results of the calculations for each hash group are summed (or merged) to form a row (or a portion) of the attention score matrix.

[0108] In practical applications, the specific process of the row-product sparse matrix multiplication is as follows: 1) Create an array of length n based on the number of columns n of the output matrix. 2) Based on the row offset of the left-multiplied matrix CSR, extract all elements of the first row from the value-column pair, and then extract the elements of each row of the right-multiplied matrix corresponding to the first column number. 3) Multiply the column number value of the left-multiplied element with the corresponding row of the right-multiplied element, and store the corresponding value in the array. 4) After the product operation is completed for all non-zero elements in the first row of the left-multiplied matrix, the first row of the output matrix is ​​obtained. 5) After the product operation is completed for all non-zero elements in all rows of the left-multiplied matrix (parallel computation can be performed here), all rows of the output matrix are obtained. Here, the left-multiplied matrix and the right-multiplied matrix refer to the query matrix and the key matrix, respectively.

[0109] Step 5: Rearrange each row of the output matrix to obtain the attention score matrix.

[0110] Since hash grouping may scramble the order of rows in the output matrix, it needs to be rearranged back to the original order. Based on the original order index before hash grouping, the rows of the accumulated attention score matrix are rearranged to generate the final attention score matrix.

[0111] Furthermore, hashing the dot product matrix involves: hashing the non-zero query vectors and key vectors in the dot product matrix multiple times; and assigning the query vectors and key vectors to different hash groups without repetition.

[0112] In practical applications, the entire process of hashing and grouping the dot product matrix includes the following steps: 1) Hash the non-zero Q and K vectors multiple times (random projection) and assign the vectors to unique hash groups. 2) For each query vector, calculate the attention score only for the key vectors in the same hash group, i.e., matrix multiplication. 3) After grouping and calculation, rearrange the outputs to form a unified attention map.

[0113] In one embodiment, a shift-based softmax calculation is performed based on the attention score matrix to obtain the softmax calculation result, including:

[0114] Step 1: Divide the attention score matrix into several groups.

[0115] Divide the attention score matrix into several groups. This can be achieved by slicing along the sequence length (seq_length) or the number of heads (num_heads) dimension.

[0116] Step 2: Find the maximum value within each group and obtain the preset second threshold. The preset second threshold is determined based on the overall tensor corresponding to all data in the group.

[0117] For each group of scores, find the maximum value x. gmax i The search process can be parallelized. The preset second threshold T can be dynamically determined based on the statistical properties of the overall tensor, but more commonly a fixed proportion (such as a fractional multiple of the maximum value) is used as the threshold.

[0118] Step 3: Calculate the difference between each element in the group and the maximum value in the group. If the difference is greater than the preset second threshold, set the corresponding element to zero.

[0119] The difference x between the maximum value of each group and each element is calculated. gmax i -x, if the result is greater than the threshold, i.e., x gmax i If -x > T, the element is set to zero (which has little impact on the result), and subsequent calculations are skipped.

[0120] Step 4: Calculate the difference raised to the power of 2 to obtain an intermediate result.

[0121] Calculate 2 (x-xgmax i) This yields an intermediate result. This intermediate result still needs to be processed by x... max The final result of the compensation.

[0122] Step 5: Based on the maximum value within each group, find the maximum value among all elements.

[0123] When all groups x gmax i After the search is complete, find the maximum value x among all elements. max .

[0124] Step 6: Compensate for intermediate results by finding the maximum value among all elements, and complete the softmax calculation through a shift operation to obtain the softmax result.

[0125] Through x maxTo compensate for intermediate results, since 2 is the base of the exponent, only a simple shift operation is needed here.

[0126] In one embodiment, the attention score matrix is ​​divided into several groups, including:

[0127] The attention score matrix is ​​scaled by dividing it by the dimension of the key matrix to obtain the scaled attention score matrix; the scaled attention score matrix is ​​then divided into several groups.

[0128] Scale the attention score matrix S and divide by Where dk is the dimension of the key), the scaled attention score matrix is ​​then divided into several groups to reduce the amount of data processing. Here, the specific row-direction product sparse matrix multiplication is described.

[0129] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0130] Based on the same inventive concept, this application also provides an efficient sparse attention algorithm optimization device for implementing the efficient sparse attention algorithm optimization method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the efficient sparse attention algorithm optimization device provided below can be found in the limitations of the efficient sparse attention algorithm optimization method described above, and will not be repeated here.

[0131] In one embodiment, such as Figure 4 As shown, an efficient sparse attention algorithm optimization device is provided, comprising:

[0132] The normalization module 200 is used to acquire the data to be processed and perform layernorm calculation based on importance awareness on the data to be processed to obtain normalized data.

[0133] The attention processing module 400 is used to perform row product sparse matrix multiplication and group hash multi-head attention processing on normalized data to obtain an attention score matrix.

[0134] The softmax calculation module 600 is used to perform softmax calculation based on the attention score matrix and obtain the softmax calculation result.

[0135] The attention calculation module 800 is used to perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result.

[0136] In one embodiment, the normalization module 200 is further configured to acquire the data to be processed; calculate the absolute value of each element in the data to be processed; retain the elements with the preset percentage position before the absolute value and set the remaining elements to zero to obtain the adjusted data; and perform layernorm calculation on the adjusted data to obtain normalized data.

[0137] In one embodiment, the normalization module 200 is further configured to identify non-zero elements in the adjusted data, calculate the square of the non-zero elements; if the square of the current non-zero element is less than a preset first threshold, then set the current non-zero element to zero; accumulate the element value and square value of the adjusted non-zero elements; and obtain the normalized layernorm output result for each element based on the element value and square value.

[0138] In one embodiment, the attention processing module 400 is further configured to apply a linear transformation to the normalized data to obtain a query vector, a key vector, and a value vector; perform CSR compression on the query vector, key vector, and value vector; calculate the dot product matrix of the query and key vectors based on the query vector and key vectors; perform hash grouping on the dot product matrix; perform row product sparse matrix multiplication on the query vector and key vector in each hash group; accumulate to obtain a row of the output matrix; and output a rearranged row of the output matrix to obtain the attention score matrix.

[0139] In one embodiment, the attention processing module 400 is further configured to perform multiple hashes on the non-zero query vectors and key vectors in the dot product matrix; and to assign the query vectors and key vectors to different hash groups without repetition.

[0140] In one embodiment, the softmax calculation module 600 is further configured to divide the attention score matrix into several groups; find the maximum value in each group and obtain a preset second threshold, the preset second threshold being determined based on the overall tensor corresponding to all data in the group; calculate the difference between each element in the group and the maximum value in the group, and if the difference is greater than the preset second threshold, set the corresponding element to zero; calculate the difference raised to the power of 2 to obtain an intermediate result; find the maximum value among all elements based on the maximum values ​​in different groups; compensate the intermediate result with the maximum value among all elements; complete the softmax calculation through a shift operation to obtain the softmax calculation result.

[0141] In one embodiment, the softmax calculation module 600 is further configured to scale the attention score matrix by dividing it by the dimension of the key matrix to obtain a scaled attention score matrix; and to divide the scaled attention score matrix into several groups.

[0142] Each module in the aforementioned high-efficiency sparse attention algorithm optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0143] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores preset data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an efficient sparse attention algorithm optimization method.

[0144] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described efficient sparse attention algorithm optimization method.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described efficient sparse attention algorithm optimization method.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described efficient sparse attention algorithm optimization method.

[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An efficient sparse attention algorithm optimization method, characterized in that, The method includes: Obtain the text data to be processed, and perform layernorm calculation based on importance awareness on the text data to be processed to obtain normalized data; The normalized data is subjected to row product sparse matrix multiplication and group hash multi-head attention processing to obtain an attention score matrix; Based on the attention score matrix, a softmax calculation based on a shift operation is performed to obtain the softmax calculation result; Perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result; The step of performing row-directed product sparse matrix multiplication and grouped hashing multi-head attention processing on the normalized data to obtain an attention score matrix includes: applying a linear transformation to the normalized data to obtain a query vector, a key vector, and a value vector; performing CSR compression on the query vector, key vector, and value vector; calculating the dot product matrix of the query and key vectors based on the query vector and the key vector; hashing the dot product matrix, performing row-directed product sparse matrix multiplication on the query vector and key vector in each hash group, and summing them to obtain a row of the output matrix; and rearranging each row of the output matrix to obtain the attention score matrix. The step of performing a softmax calculation based on a shift operation on the attention score matrix to obtain the softmax calculation result includes: dividing the attention score matrix into several groups; finding the maximum value in each group and obtaining a preset second threshold, the preset second threshold being determined based on the overall tensor corresponding to all data in the group; calculating the difference between each element in the group and the maximum value in the group, and if the difference is greater than the preset second threshold, setting the corresponding element to zero; calculating the difference raised to the power of 2 to obtain an intermediate result; finding the maximum value among all elements based on the maximum values ​​in different groups; compensating the intermediate result with the maximum value among all elements; and completing the softmax calculation through a shift operation to obtain the softmax calculation result.

2. The method according to claim 1, characterized in that, The process of acquiring the data to be processed and performing an importance-aware layernorm calculation on the data to be processed to obtain normalized data includes: Obtain the data to be processed; Calculate the absolute value of each element in the data to be processed; The elements with the pre-defined percentage before the absolute value are retained, and the remaining elements are set to zero to obtain the adjusted data; The adjusted data is subjected to layernorm calculation to obtain normalized data.

3. The method according to claim 2, characterized in that, The process of performing layernorm calculation on the adjusted data to obtain normalized data includes: Identify the non-zero elements in the adjusted data and calculate the square of the non-zero elements; If the square of the current non-zero element is less than the preset first threshold, then the current non-zero element is set to zero. The sum of the element values ​​and squares of the non-zeroed elements after adjustment; Based on the element value and the square value, the normalized layernorm output result for each element is obtained.

4. The method according to claim 1, characterized in that, Hash grouping of the dot product matrix includes: Hash the non-zero query vectors and key vectors in the dot product matrix multiple times; The query vector and key vector are assigned to different hash groups without duplication.

5. The method according to claim 1, characterized in that, The step of dividing the attention score matrix into several groups includes: The attention score matrix is ​​scaled by dividing it by the dimension of the key matrix to obtain the scaled attention score matrix. The scaled attention score matrix is ​​divided into several groups.

6. A highly efficient sparse attention algorithm optimization device, characterized in that, The device includes: The normalization module is used to acquire the text data to be processed and perform layernorm calculation based on importance awareness on the text data to be processed to obtain normalized data. The attention processing module is used to perform row product sparse matrix multiplication and group hash multi-head attention processing on the normalized data to obtain an attention score matrix; The softmax calculation module is used to perform softmax calculation based on the attention score matrix and obtain the softmax calculation result. The attention calculation module is used to perform row product sparse matrix multiplication on the softmax calculation result and the median vector of the attention matrix to obtain the attention calculation result. The attention processing module is further configured to apply a linear transformation to the normalized data to obtain a query vector, a key vector, and a value vector; perform CSR compression on the query vector, key vector, and value vector; calculate the dot product matrix of the query and key vectors based on the query vector and the key vector; perform hash grouping on the dot product matrix; perform row product sparse matrix multiplication on the query vector and key vector within each hash group; accumulate the results to obtain a row of the output matrix; and output a rearranged row of the output matrix to obtain the attention score matrix. The softmax calculation module is also used to divide the attention score matrix into several groups; find the maximum value in each group and obtain a preset second threshold, which is determined based on the overall tensor corresponding to all data in the group; calculate the difference between each element in the group and the maximum value in the group, and if the difference is greater than the preset second threshold, set the corresponding element to zero; calculate the difference raised to the power of 2 to obtain an intermediate result; find the maximum value among all elements based on the maximum values ​​in different groups; compensate the intermediate result with the maximum value among all elements; and complete the softmax calculation through a shift operation to obtain the softmax calculation result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images