Data processing methods, apparatus, electronic devices and readable storage media

By dividing the matrix of the attention computation thread block in the Transformer model into blocks and processing it in a private register group, the computational bottleneck and memory pressure issues are resolved, thereby improving computational efficiency and bandwidth utilization.

CN123115206APending Publication Date: 2026-11-17龙芯中科(合肥)技术有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611628373.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

The Transformer model faces computational bottlenecks and memory pressure in scenarios such as long texts and high-resolution images, resulting in excessive computational latency and bandwidth consumption.

Method used

By dividing the query matrix, key matrix, and value matrix corresponding to the attention computation thread block into blocks and performing the computation in a private register group, the frequent access to computation parameters from shared memory is avoided, and data processing is performed using a private register group.

Benefits of technology

It reduces computational latency and bandwidth usage, improves data processing efficiency, reduces video memory usage, and increases the utilization rate of computing units.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115206A_ABST
    Figure CN123115206A_ABST
Patent Text Reader

Abstract

This invention provides a data processing method, apparatus, electronic device, and readable storage medium. The method includes: dividing the query matrix, key matrix, and value matrix corresponding to the input sequence into blocks based on a private register group corresponding to the attention calculation thread block, obtaining sub-query matrices, sub-key matrices, and sub-value matrices; loading any of the sub-query matrices and each of the sub-key matrices into the private register group corresponding to the sub-query matrix; obtaining the sub-score matrix corresponding to the sub-query matrix based on the sub-query matrix and each of the sub-key matrices in the private register group, and writing it into the private register group; performing row-level reduction on each of the sub-score matrices within the private register group, and calculating the normalized exponential function based on the reduction result and the sub-value matrix, until all sub-query matrices are processed to obtain the target attention matrix. This invention can improve the speed of attention calculation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] The Transformer model has become a fundamental architecture in the field of deep learning, with its core being the self-attention mechanism. In standard implementations, the input vector or sequence is typically first transformed into corresponding query, key, and value matrices. Then, multiple matrix operations and exponential normalization are performed on these matrices to determine the attention weights of each element in the vector or sequence. As the length of the input sequence increases, the computational and memory overhead of the model increases quadratically, creating bottlenecks in scenarios such as long text and high-resolution images. This results in significant computational bottlenecks and memory pressure for the Transformer model in these scenarios, leading to high computational latency. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data processing method that overcomes or at least partially solves the above problems. Attention calculation can be performed between registers, avoiding frequent access to calculation parameters from shared memory, reducing computational latency and bandwidth occupation caused by data movement during the calculation process, and improving data processing efficiency.

[0004] Accordingly, embodiments of the present invention also provide an apparatus, an electronic device, and a computer program product to ensure the implementation and application of the above methods.

[0005] In a first aspect, embodiments of the present invention disclose a data processing method, the method comprising: Based on the private register group corresponding to the attention computation thread block, the query matrix, key matrix and value matrix corresponding to the input sequence are divided into blocks to obtain sub-query matrix, sub-key matrix and sub-value matrix; Load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; Based on the subquery matrix and each subkey matrix in the private register group, the sub-fraction matrix corresponding to the subquery matrix is ​​obtained, and the sub-fraction matrix is ​​written into the private register group; Within the private register group, row-level reduction is performed on each of the sub-score matrices, and the normalized exponential function is calculated based on the reduction result and the sub-value matrix until all sub-query matrices are processed, thus obtaining the target attention matrix.

[0006] In a second aspect, embodiments of the present invention disclose a data processing apparatus, the apparatus comprising: The matrix partitioning module is used to divide the query matrix, key matrix, and value matrix corresponding to the input sequence into sub-query matrix, sub-key matrix, and sub-value matrix based on the private register group corresponding to the attention computation thread block. The block loading module is used to load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; The score acquisition module is used to obtain the sub-score matrix corresponding to the sub-query matrix based on the sub-query matrix and each sub-key matrix in the private register group, and write the sub-score matrix into the private register group; The result generation module is used to perform row-level reduction on each of the sub-score matrices within the private register group, and to calculate the normalized exponential function based on the reduction result and the sub-value matrix, until all sub-query matrices have been processed to obtain the target attention matrix.

[0007] Thirdly, embodiments of the present invention disclose an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform the steps of any of the data processing methods described above.

[0008] Fourthly, embodiments of the present invention disclose a readable storage medium storing a program or instructions, which, when executed by a processor, can implement any of the data processing methods described above.

[0009] Fifthly, embodiments of the present invention disclose a computer program product, including a computer program, wherein when the computer program is executed by a processor, it performs the steps of any of the data processing methods described above.

[0010] The embodiments of the present invention have the following advantages: First, the matrices involved in the attention calculation are partitioned according to the private register set corresponding to the attention calculation thread block. This includes partitioning the query matrix, key matrix, and value matrix, ensuring that the size of each sub-matrix matches the private register set. Therefore, the partitioned sub-matrices (such as sub-query matrices, sub-key matrices, and sub-value matrices) can be loaded into the private register set, and attention calculation can then be performed within that set. During the calculation, intermediate parameters are written to the current private register set instead of being read from shared memory. Since the speed at which the computation unit reads and writes data in the private register (set) is typically faster than the speed at which it reads and writes data in shared memory or video memory, performing attention calculations based on the parameters stored in the private register set reduces computational latency and improves computational efficiency. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of an exemplary attention calculation process; Figure 2 This is a flowchart illustrating the steps of an embodiment of the attention calculation method of this application; Figure 3 This is a schematic diagram of the execution logic of the attention calculation method in an example of an embodiment of the present invention; Figure 4 This is a logical diagram of layout rearrangement in one example of an embodiment of the present invention; Figure 5 This is a structural block diagram of an embodiment of a data processing device according to the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] The Transformer model has become a fundamental architecture in the field of deep learning. Its core attention mechanism, however, has high computational complexity and has consistently been a bottleneck for inference acceleration. (See diagram for single-head computation illustration.) Figure 1 Specifically, first calculate the complete query matrix. (N×N matrix), K is the key matrix, then perform Softmax on the query matrix S to obtain the probability matrix P, and finally calculate V is the value matrix, resulting in the final attention matrix Y. Taking an example where all elements in the matrix used for calculation are of type float, the overall process overhead is analyzed as follows: Computational complexity: The computational complexity of matrix S is: The softmax value is approximately: The last matrix multiplication (i.e. The computational complexity is: The total computational workload is: .in, This is the unit of numerical calculation corresponding to the float format, representing the number of floating-point operations.

[0014] Inventory access: Total Q, K, V matrices read The number of elements (the actual matrix multiplication and block memory access will far exceed this value, based on the lowest possible count), and the number of reads and writes to the intermediate matrix P×Q are calculated. Each element, output Y matrix There are 10 elements. Each float requires 4 bytes, for a total of 4 bytes to read and write. .

[0015] Memory usage of the intermediate matrix: Matrices S and P' need to be stored completely in video memory simultaneously, with a length of: When N=4096, these two matrices alone occupy 128 MB of video memory (and this is for a single-head operation scenario). In multi-head attention or batch inference, this can easily exceed the GPU's (Graphics Processing Unit) video memory capacity.

[0016] Taking a certain model of GPU as an example, this GPU has a computing power of FP32 and 19.5 TFLOPS, meaning it can theoretically perform 19.5 trillion FP32 single-precision floating-point operations per second; its HBM bandwidth is 2TB / s, meaning its video memory can move approximately 2 trillion bytes of data per second. The estimated overhead for different sequence lengths is as follows:

[0017] As shown in the table above, when N=1024, the computation time and memory access time are similar, and the system can still maintain a certain level of computational efficiency. When N increases to 4096, the memory access time (approximately 0.273 ms) exceeds the computation time (approximately 0.22 ms), and bandwidth becomes the main performance bottleneck. Simultaneously, the memory usage of the intermediate matrices S and P increases with the square of N, putting enormous pressure on the GPU memory and severely limiting the inference efficiency and scalability in long sequence scenarios.

[0018] In summary, the traditional attention implementation of the Transformer model has two major drawbacks: High memory usage: The memory requirements of the S and P matrices are... When the sequence length N is large, these two intermediate matrices alone may exhaust the GPU memory, making long sequence inference impossible to execute.

[0019] High memory access pressure: HBM's memory access volume is also high. As N increases, bandwidth gradually becomes a performance bottleneck, and the utilization rate of computing units decreases significantly.

[0020] Among the related technologies, the FlashAttention calculation method was proposed. Its core lies in the online softmax (normalized exponential function) technique, which allows the softmax calculation to be performed in blocks.

[0021] The fundamental reason why traditional attention implementations cannot be segmented is that the softmax operation requires all elements of a complete row to calculate the normalized result, and a row result requires traversing all columns of the K matrix, which severely restricts the implementation of segmentation strategies. FlashAttention, by maintaining the historical maximum value and historical exponent sum of each row during the segmentation iteration process, enables online correction of the calculated partial results when processing each new block, thereby breaking the dependence of Softmax on complete rows.

[0022] Based on online softmax technology, FlashAttention divides Q, K, and V into small blocks and performs attention calculations in (on-chip) shared memory (SRAM), avoiding the generation of the complete attention matrix at once. This avoids reading and writing the complete N×N intermediate matrix to high-bandwidth memory (HBM), which is commonly used as video memory. It employs a two-loop structure: the outer loop iterates through the blocks (K_j, V_j) of K and V and loads them into SRAM; within each loop, the inner loop iterates through the blocks (Q_i) of Q and loads them into SRAM. Here, i and j are integers.

[0023] Load the Q_i block, K_j block, and V_j block into shared memory respectively; load the Q_i and K_j matrices from shared memory and calculate the local attention score matrix. .

[0024] Use online Softmax technology to maintain the historical maximum value (max), historical exponent, and sum for each row, and perform... Solving the probability matrix .

[0025] Then V_j is loaded from video memory, and the accumulated fragment is calculated and output. Recalculate This continues until the entire line of V_j has been traversed. , (equal to elements in V_j), ultimately yielding the result of this calculation. matrix.

[0026] Next, repeat the above steps for the next Q_i block and the K_j and V_j matrices in the graph, and then use the results of this calculation. And the result of the previous calculation Cumulative result of this calculation And the result of the previous calculation The size of the matrix is ×d, until the calculation of all Q_i blocks is completed.

[0027] For a single Q_i block, after traversing all K and V blocks, Y_i = / sum, and you will get the final attention output Y_i corresponding to Q_i; after repeating the above steps for all Q_i blocks, you will get the target attention matrix Y.

[0028] This scheme addresses the high memory consumption issue caused by storing complete S and P matrices in traditional schemes through a block-based strategy. All intermediate small matrices S_ij and P_ij are cached in shared memory SRAM, with a data access rate more than 10 times that of HBM, resulting in a significant performance improvement. However, this intermediate data still needs to be frequently exchanged with shared memory, consuming considerable shared memory space and bandwidth resources.

[0029] Reference Figure 2 The diagram illustrates a flowchart of an embodiment of a data processing method according to the present invention, the method comprising the following steps: Step 101: Based on the private register group corresponding to the attention calculation thread block, divide the query matrix, key matrix and value matrix corresponding to the input sequence into blocks to obtain sub-query matrix, sub-key matrix and sub-value matrix respectively; Step 102: Load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; Step 103: Based on the subquery matrix and each subkey matrix in the private register group, obtain the sub-fraction matrix corresponding to the subquery matrix and write it into the private register group; Step 104: Within the private register group, perform row-level reduction on each of the sub-score matrices, and calculate the normalized exponential function based on the reduction result and the sub-value matrix, until all sub-query matrices have been processed to obtain the target attention matrix.

[0030] A single attention computation thread block contains multiple independent attention computation threads. These threads can collaborate through shared memory and synchronization mechanisms to complete a large task (such as attention computation for a subquery matrix).

[0031] Understandably, during the attention calculation process, each row of the query matrix Q is a query vector, used to represent the information that needs attention at each position in the input sequence; each row of the key matrix K is a key vector, used to represent the feature label at each position in the input sequence; and each row of the value matrix V is a value vector, used to represent the actual content at each position in the input sequence.

[0032] The attention calculation process can be summarized as follows: The Q matrix and the transpose of the K matrix are transposed and then productd to obtain the similarity score between the query vector and the key vector at each position. This similarity score is then processed by a normalized exponential function to obtain the attention weights. Finally, the value matrix is ​​weighted and summed according to these attention weights to calculate the final attention result for the input sequence (i.e., the target attention matrix in this example). The similarity score represents the relevance between the feature labels of the key vectors at each position in the input sequence and the information that needs attention, as indicated by each query vector. The final attention result is the output after weighting and adjusting the actual content (i.e., the V matrix) at each position in the input sequence based on this relevance. This output is a new representation of each position in the input sequence after fusion with its context (i.e., the content of the preceding and following positions). Each position in this output contains the content of all positions in the input sequence, providing a more comprehensive representation for subsequent layers.

[0033] The input sequence can be a feature sequence formed after preprocessing of text, image, or speech data; the query matrix, key matrix, and value matrix are matrices formed by the ordered arrangement of three feature vectors obtained by feature extraction from the input sequence through different weight matrices.

[0034] In this embodiment of the invention, the matrices involved in the calculation are first divided into regular blocks according to the private register group corresponding to the attention calculation thread block, resulting in sub-matrices (including sub-query matrices, sub-key matrices, and sub-value matrices) matching each private register group. The private register groups formed by the private registers corresponding to multiple attention calculation threads within the attention calculation thread block allow the sub-matrices to be loaded into these groups, serving as the basis for completing the attention calculation within them. Subsequently, the corresponding sub-score matrices are calculated in each private register group based on the currently processed sub-query matrix. Reduction calculations are then performed on each sub-score matrix among the private registers within the private register group, ultimately generating and outputting the target attention matrix. This embodiment of the invention supports performing the complete attention matrix calculation within the private register group, eliminating the need to write intermediate calculation parameters (such as sub-score matrices) to shared memory or video memory and read them for subsequent calculations and processing. Reading and writing data in registers is generally faster than in shared memory and video memory, thus significantly improving the speed of attention calculation. Note that the private registers are only allowed to be read and written by their corresponding attention calculation threads and are not supported by other attention calculation threads. Meanwhile, the number of private registers corresponding to an attention computation thread is usually not fixed. This number is determined by the hardware architecture, kernel implementation, and compilation options, and generally varies within a wide range.

[0035] It is understandable that the private registers are specifically manifested as vector registers allocated to each attention computation thread. Therefore, the matrix involved in the computation can be divided into blocks based on the size of each vector register.

[0036] Simultaneously, the parallelism of each attention calculation thread block can be considered when dividing the matrices involved in the calculation. Specifically, if the blocks of matrices involved in the calculation are too large, each attention calculation thread will require more vector registers (i.e., private registers) for processing. With a limited number of vector registers, it will be impossible to parallelize a large number of attention calculation threads, i.e., a large number of attention calculation thread blocks, thus affecting the parallelism of the attention calculation thread blocks. Specifically, a single attention calculation thread block processes one subquery matrix at a time; processing all subquery matrices simultaneously requires multiple attention calculation thread blocks to run in parallel. When the parallelism of the attention calculation thread blocks is affected, the processing efficiency of the subquery matrices will decrease.

[0037] Specifically, for step 101, the query matrix, key matrix, and value matrix in the shared memory are first divided into blocks based on the private register group corresponding to the attention calculation thread block, so as to obtain sub-query matrix, sub-key matrix, and sub-value matrix respectively. This allows the divided sub-query matrix and sub-key matrix to be loaded into the private register respectively, and then the subsequent attention calculation process can be completed in the private register.

[0038] Understandably, in this application, the complete query matrix, key matrix, and value matrix originally stored in video memory are divided into regular blocks to obtain sub-matrices, which are then loaded into shared memory. Subsequently, each sub-matrix is ​​further loaded from shared memory into the private register set corresponding to the attention calculation thread block, and the calculation is completed in the private register set. When resources in shared memory are scarce, after matrix block division in video memory, the sub-matrices obtained from the block division are directly loaded into the private register set, without needing to be transferred through shared memory.

[0039] For example, each attention calculation thread block can calculate one subquery matrix at a time. Therefore, during the calculation process, the private register group corresponding to the attention calculation thread block needs to load one subquery matrix. Simultaneously, when calculating the sub-score matrix corresponding to this subquery matrix, matrix multiplication needs to be performed on the subquery matrix and each subkey matrix to obtain a single sub-score matrix. Since the matrix multiplication process requires performing matrix multiplication on the subquery matrix and all subkey matrices, it is possible to choose to load the subkey matrices sequentially into the private register group and perform matrix multiplication with the subquery matrix, keeping only one subkey matrix being calculated in the private register group at a time, thus reducing storage space usage.

[0040] Optionally, attention computation is typically performed collaboratively by multiple attention computation thread blocks, each with its own private register set. In this example, each private register set stores the sub-query matrix Qi and sub-key matrix Kj, which are computed by the attention computation thread block, and further performs attention computation. During attention computation, the computation unit only reads and writes data to the private register set. Correspondingly, the shared memory is only used to store the query matrix Q, key matrix K, and other parameters during the computation process (such as the value matrix V), and does not receive intermediate parameters obtained during attention computation, such as the score matrix S. That is, the computation unit does not need to frequently interact with the shared memory, improving computational efficiency. The computation unit can be, for example, a stream processor in a GPU (Graphics Processing Unit).

[0041] In some examples, the step of dividing the query matrix, key matrix, and value matrix corresponding to the input sequence into sub-query matrices, sub-key matrices, and sub-value matrices based on the private register set includes: S11, determine the row block specification and column block specification based on the storage space size of the private register group; S12, the query matrix is ​​divided into multiple sub-query matrices according to the row block specification, the key matrix is ​​divided into multiple sub-key matrices according to the column block specification, and the value matrix is ​​divided into multiple sub-value matrices according to the column block specification.

[0042] like Figure 3 As shown, each subquery matrix, subkey matrix, and subvalue matrix includes a different number of complete matrix rows. The specific number of matrix rows is determined by the attention computation thread block currently being computed.

[0043] For example, the size of the storage space in the private register group corresponding to the attention calculation thread block (or the size of the space divided for storing related sub-matrices), as well as the number of sub-matrices to be stored / the amount of data in a single row of the sub-matrices, can be analyzed to jointly determine the row block specification Br and the column block specification Bc.

[0044] In this case, when the dimension of a single matrix row is d (i.e., the number of elements in a single row is d), the size of the subquery matrix is ​​Br×d, the size of the subkey matrix is ​​Bc×d, and the size of the subvalue matrix is ​​Bc×d. It is understandable that, since matrix multiplication is required, the dimensions of the query matrix and the key matrix must be the same to perform the dot product calculation; therefore, both are set to the dimension of d. The dimension of the value matrix is ​​also usually set to d to maintain consistent calculation dimensions and facilitate subsequent processing.

[0045] Furthermore, the naming of the column partitioning specification Bc for the key matrix does not refer to partitioning the key matrix K according to the columns based on this parameter. Rather, when performing matrix multiplication between the query matrix and the key matrix, since the key matrix needs to be transposed before participating in the calculation, its partitioning specification Bc will correspond to the column specification parameter of the matrix multiplication calculation result.

[0046] In this example, each matrix is ​​divided according to its complete rows. Each submatrix retains its complete rows. Under the premise of scaling down each complete matrix, it can still support the original Transformer rules that require row-by-row attention calculation, avoid complicating the computational logic, and maintain the accuracy of the final attention result.

[0047] After obtaining the subquery matrix Qi and the subkey matrix Kj, the subquery matrix Qi and the subkey matrix Kj are loaded into the private register group respectively, so that the computing unit can calculate the subfraction matrix Sij based on the subquery matrix Qi and the subkey matrix Kj in the private register group.

[0048] Specifically, each attention calculation thread block corresponds to a subquery matrix Qi. After calculating the subquery matrix Qi to be processed and each subkey matrix Kj, all corresponding subscore matrices Sij are obtained, and j subscore matrices Sij form a subscore block row Si (e.g., ...). Figure 3 As shown in the embodiment of this application, it is not necessary to calculate the complete sub-fraction block row Si. After calculating the sub-fraction matrix Sij in the sub-fraction block row Si, the corresponding subsequent processing flow can be executed. After obtaining the new sub-fraction matrix Sij, it is supported to perform iterative updates based on the relevant parameters in the latest sub-fraction matrix Sij, which can reduce the calculation delay of intermediate parameters, such as the fraction matrix S, and reduce the space occupied by intermediate parameters.

[0049] For step 104, after obtaining the sub-score matrix Sij based on the subquery matrix Qi and the subkey matrix Kj, row-level reduction is performed on the sub-score matrix Sij in the private register group that processes the subquery matrix Qi. Based on the reduction result, a normalized exponential function is calculated until all subquery matrices are processed, resulting in the target attention matrix. Different subquery matrices are processed in parallel by different attention calculation thread blocks. For example, subquery matrix Q(i+1) will be processed by attention calculation thread block T1, while at the same time, subquery matrix Q(i+2) may be processed by attention calculation thread block T2. It is not a single attention calculation thread block that processes all subquery matrices sequentially.

[0050] Specifically, reduction is an operation that repeatedly combines a set of elements through binary operations to obtain one or a few results. Examples include summing, averaging, finding maximum and minimum values, multiplying, calculating variance / standard deviation, and intersecting / unifying sets.

[0051] The reductions in this application are all row-level reductions, meaning that the reduction needs to be performed separately for each fractional row vector in the sub-fractional matrix. A fractional row vector is a matrix row in the sub-fractional matrix Sij. If it is the r-th fractional row vector in the sub-fractional matrix Sij, then the row-level reduction is to perform the reduction for the r-th row in the sub-fractional matrix Sij.

[0052] In this implementation, reduction is performed within a private register set, followed by normalization exponent calculation to obtain the final target attention matrix. For example, a fractional row vector is generated by a local attention computation thread block. This local attention computation thread block includes multiple attention computation threads (typically, an attention computation thread block may include multiples of 32 attention computation threads). That is, when a fractional row vector is generated collaboratively by multiple attention computation threads, reduction can be performed between the private registers corresponding to at least two attention computation threads using an intra-bunch shuffle instruction (Warp Shuffle), and the reduction result is broadcast to all private registers in the attention computation thread block for storage. It is understood that a local attention computation thread block refers to a portion of the attention computation threads within the attention computation thread block mentioned in step 101. Since one attention calculation thread block corresponds to processing one subquery matrix Qi, and one subquery matrix Qi contains multiple matrix rows, there are multiple local attention calculation thread blocks in the attention calculation thread block. Each local attention calculation thread block includes several attention calculation threads (e.g., 4). Each local attention calculation thread block corresponds to processing one row (one matrix row) in the subquery matrix Qi, which will generate one row in the sub-score matrix Sij, i.e., a score row vector.

[0053] Furthermore, after the reduction is completed within the private register group, the normalized exponential function, i.e., softmax, can be calculated based on the reduction result and the subvalue matrix, and finally the target attention matrix is ​​calculated.

[0054] In some related technologies, the fractional matrix S' calculated from the predefined query matrix segment Q' and key matrix segment K' is stored in shared memory. Intermediate parameters (such as probability matrix segment P') obtained based on the fractional matrix S' are also stored in shared memory. These parameters are then retrieved from shared memory during subsequent calculations to complete the attention calculation. This resource access process consumes significant bandwidth and shared memory resources, resulting in high computational latency.

[0055] In this embodiment, the matrices involved in the attention calculation are divided according to the private register set corresponding to the attention calculation thread block. This includes dividing the query matrix, key matrix, and value matrix separately, ensuring that the size of each sub-matrix matches the private register set. Therefore, the divided sub-matrices (such as sub-query matrices, sub-key matrices, and sub-value matrices) can be loaded into the private register set first, and then attention calculation can be performed within the private register set. During the calculation, the generated intermediate parameters are written to the current private register set instead of reading and writing through shared memory. Since the speed at which the computing unit reads and writes data in the private register (set) is usually faster than the speed at which it reads and writes data in shared memory or video memory, performing attention calculation based on the parameters stored in the private register set will reduce computation latency and improve computational efficiency.

[0056] In an optional embodiment of the present invention, the normalized exponential function is calculated based on the reduction result and the sub-value matrix until all sub-query matrices are processed, resulting in the target attention matrix, including: S21, perform layout rearrangement on the reduction result according to a preset rule, the preset rule being used to make the reduction result after layout rearrangement compatible with matrix multiplication and accumulation instructions; S22, execute a matrix multiplication and accumulation instruction on the reduction result after the layout rearrangement and the subvalue matrix, and complete the calculation of the normalized exponential function to obtain the target attention matrix corresponding to the input sequence.

[0057] Specifically, the attention calculation process proposed in this application requires two sets of matrix multiplications. The first set involves multiplying the subquery matrix and the subkey matrix. The second set involves multiplying the reduced result of the sub-fraction matrix and the sub-value matrix. The second set can be derived by obtaining a sub-probability matrix from the reduced result of the sub-fraction matrix. Matrix multiplication is then performed on each sub-probability matrix and sub-value matrix to obtain the sub-target attention matrix. Finally, the target attention matrix is ​​obtained by concatenating and combining the sub-target attention matrices corresponding to all subquery matrices.

[0058] To further ensure the invention's objective of "improving the computational efficiency of attention calculation," this example uses the Matrix Multiply Accumulate (MMA) instruction to complete two sets of matrix multiplications. Compared to using multiplication and addition instructions, the MMA instruction has higher throughput, enabling matrix multiplication to be completed with a single instruction, rather than requiring multiple multiplication instructions to multiply the elements within the matrix and then further executing addition or concatenation instructions to obtain the final matrix multiplication result. This improves the computational efficiency of attention calculation.

[0059] Therefore, during the execution of two sets of matrix multiplications in this application, matrix parameters conforming to the matrix multiplication accumulation instruction input format need to be provided to ensure that matrix multiplication can be completed and to avoid data corruption.

[0060] It is understandable that during the execution of the first set of matrix multiplications, the input parameters of the matrix multiplication accumulation instruction consist of the subquery matrix and the subkey matrix obtained by regular division. Moreover, both the subquery matrix and the subkey matrix are directly loaded from shared memory or video memory. This loading instruction can be the loading instruction corresponding to the matrix multiplication accumulation instruction MMA. Therefore, the input parameters can be adapted to the matrix multiplication accumulation instruction.

[0061] However, for the sub-fractional matrix obtained after the first set of multiplications, the output matrix and input matrix of the matrix multiplication accumulation instruction have different formats. Specifically, due to hardware limitations in executing the matrix multiplication accumulation (MMA) instruction, the row and column positions, number, and data types of the elements in the registers storing the input matrix WA, input matrix WB, and output matrix WC are all different for the same thread executing matrix multiplication (e.g., thread 0). The expected input matrix format for the next matrix accumulation accumulation (MMA) instruction is still the original input matrix WA or input matrix WB, and therefore, matrix WC cannot be directly used to execute the next MMA instruction. A simple example of the impact of MMA on format is as follows: When the input matrix is ​​WA: Input matrix WB: The output matrix WC is obtained by performing matrix multiplication according to MMA: In this case.

[0062] Thread 0 is responsible for wa1 and wa2 in matrix WA, and wb1 and wb2 in matrix WB. After MMA instructions, it will calculate wc1 in matrix WC. When performing the next matrix multiplication based on MMA, thread 0 needs to provide two elements, with the same type as wa1 and wa2 (when matrix WC is used as matrix WA in the next MMA calculation), or the same type as wb1 and wb2 (when matrix WC is used as matrix WB in the next MMA calculation). However, thread 0 can only provide one element. In some actual execution processes, the elements in matrices WA and WB during the MMA calculation are of type char, while the elements in matrix WC are of type INT32. Therefore, the number and type of elements that thread 0 can provide based on matrix WC do not meet expectations, making it impossible to perform subsequent MMA calculations.

[0063] In this context, it can be understood that the private registers corresponding to thread 0, which is the attention calculation thread, only include the private registers storing wa1, wa2, wb1, wb2, and wc1. The private register stored in wc2 has no corresponding relationship with thread 0, so thread 0 cannot access it.

[0064] Referring to the example above, the subquery matrix sub-block Qi-a in the subquery matrix Qi (a subquery matrix consists of multiple subquery matrix sub-blocks; since the MMA instruction cannot process the complete subquery matrix Qi at once, it is divided into multiple subquery matrix sub-blocks for separate processing) and the subkey matrix sub-block Ki-a in the subkey matrix Ki (same as above, a subkey matrix consists of multiple subkey matrix sub-blocks; since the MMA instruction cannot process the complete subkey matrix Kj at once, it is divided into multiple subkey matrix sub-blocks for separate processing), i.e., the input matrix WA and input matrix WB in the example above, can correctly execute the first MMA instruction to obtain the subfraction matrix sub-block Sij-a (same as above, obtained after executing the MMA instruction on the subquery matrix sub-block and the subkey matrix sub-block; a subfraction matrix includes multiple subfraction matrix sub-blocks), i.e., the output matrix WC. At this time, the subfraction matrix sub-block Sij-a, which is the output matrix WC, cannot directly execute the next MMA instruction and needs to be rearranged. Correspondingly, the subprobability matrix obtained based on the subfraction matrix subblock Sij-a and the first-level reduction result cannot directly execute MMA instructions because only numerical adjustments have been made.

[0065] Therefore, before performing matrix multiplication on each sub-probability matrix and sub-value matrix according to the MMA instruction, it is necessary to rearrange the layout of the sub-probability matrix so that its specific arrangement is updated to conform to the input format of the matrix cumulative multiplication and addition instruction, so as to support subsequent matrix multiplication calculation through the matrix cumulative multiplication and addition instruction.

[0066] In some examples, such as in attention mechanisms like FlashAttention, intermediate local score fragments and local probability fragments are typically written to shared memory. Shared memory, as a shared storage space, allows threads within the same thread block / group to read and write at arbitrary addresses. Therefore, it can act as a layout transformation medium on top of temporary storage. During the second set of matrix multiplication calculations, each calculation thread can read the required parameter values ​​as needed. That is, in the example above, if matrix WC is placed in shared memory, thread 0 can read both wc1 and wc2 arbitrarily, without being restricted to reading only its corresponding wc1, so the input format of the matrix multiplication accumulation instruction does not need to be considered. Furthermore, while supporting reading wc1 and wc2, the format can also be adjusted as needed according to relevant instructions. This application proposes implementing the attention calculation process within a private register set. Therefore, the intermediate parameters obtained are stored in different private registers within the private register set. Each private register can only be accessed by its corresponding attention calculation thread. This may lead to a situation where the thread performing the second set of matrix multiplication calculations cannot access the parameters required for the second set of matrix multiplication operations, or even if it does access them, they may not conform to the input parameter format of the matrix multiplication and addition instruction. Consequently, matrix multiplication of the subprobability matrix and the subvalue matrix cannot be performed, and attention calculation cannot be completed.

[0067] For example, the reduction result obtained by reducing a subfractional matrix can be rearranged according to a preset rule, and the rearranged result can be written into the target register in the current private register group to execute the matrix multiplication and addition instruction. The target register can be one or more specified private registers that need to be accessed to execute the matrix multiplication and addition instruction.

[0068] Specifically, the MMA calculation is illustrated using an input data format of int8 (usually char type) and an output data format of int32.

[0069] like Figure 4 As shown, the subquery matrix subblock Qi-a can be stored in private register A0, the subkey matrix subblock Kj-a can be stored in private register B0, and the subfraction matrix subblock Sij-a obtained by performing matrix multiplication according to MMA is stored in private registers C0 and C1. Private registers A0, B0, C0 and C1 are all different private registers in the private register group corresponding to the current attention calculation thread block.

[0070] In this example, the Qi-a matrix has a size of 8×16, that is, 8 rows, with 16 elements in each row, and all data types are char. regA0_0 represents thread number 0 in private register A0, regA0_1 represents thread number 1 in private register A0, and so on. regA0_0~regA0_3 hold 16 elements per row in the sub-block of the subquery matrix Qi-a (in this example, each thread can store 32 bits; with each char element being 8 bits, one thread can hold 4 adjacent elements, meaning each row of the sub-block of the subquery matrix corresponds to 16 / 4 = 4 threads).

[0071] In this example, the Kj-a matrix is ​​also 8×16, i.e., 8 rows, with 16 elements per row, and the data type is char. regB0_0 represents thread number 0 in private register B0, regB0_1 represents thread number 1 in private register B0, and so on. regB0_0~regB0_3 hold 16 elements per row of the subkey matrix Kj-a (each thread can store 32 bits; with each char element being 8 bits, a thread can hold 4 adjacent elements).

[0072] Matrix multiplication is performed based on the aforementioned Qi-a and the transposed Kj-a matrix to obtain the sub-fractional matrix sub-block Sij-a. Based on matrix multiplication rules, the sub-fractional matrix sub-block Sij-a is now 8×8, i.e., 8 rows, each row containing 8 elements, and the data type is int32 (input elements are of type char, output elements are of type int32, which is the specification of the MMA instruction in this example). regC0_0 represents thread 0 in private register C0, regC1_0 represents thread 0 in private register C1, and so on. regC0_0~regC1_3 hold 8 elements per row in the sub-key matrix sub-block Sij-a (each thread can still store 32 bits; with each element being int32, i.e., 32 bits, one thread can hold one element, so one row requires 8 threads to hold it together. However, in the original register format, there were only 4 threads per row, so two registers (C0 and C1) are needed to hold the currently output Sij-a matrix).

[0073] In the above example, all threads ending in _0 ("Thread 0") can collectively form thread 0 for attention calculation, and all threads ending in _1 ("Thread 1") can collectively form thread 1 for attention calculation. That is, the above calculation process can be regarded as the matrix multiplication calculation process of threads 0 to 31 for attention calculation in registers A0, B0, C0 and C1.

[0074] like Figure 4 As shown, the arrangement of parameters in registers C0 and C1, which ultimately store Sij-a, is different from that in registers A0 and B0. When performing the second set of matrix multiplications based on MMA, the Sij-a matrix format stored in registers C0 and C1 needs to be converted to a form consistent with that in the original registers A0 or B0.

[0075] For example, referring to the forms in A0 and B0 above, if an MMA operation requires 16 elements per row, then at least the Sij-a stored in registers C0 and C1 and the other Sij-b stored in another set of registers C2 and C3 need to be concatenated to ensure that there are 16 elements per row, thus meeting the 8×16 specification required for the first matrix multiplication. Simultaneously, the int32 type elements also need to be converted to char format. The aforementioned other Sij-b is, for example, the sub-fraction matrix sub-block Sij-b calculated based on the subquery matrix sub-block Qi-b and the subkey matrix sub-block Kj-b (i.e., other sub-blocks in the subquery matrix Qi and other sub-blocks in the subkey matrix Kj). Figure 4 As shown, referring to the z1-z4 identifiers in Sij-a and Sij-b that refer to different columns, they can be concatenated into the exemplary empty matrix form at the bottom of the diagram, which can match the calculation format in the original A0 and B0 to complete the MMA calculation.

[0076] Optionally, for example, it may be necessary to concatenate the elements in regC0_0, regC1_0, regC0_1, and regC1_1 into a form that the original regA0_0 thread can process. This concatenation process is a layout rearrangement, and the rearranged matrix format can meet the input parameters for MMA execution.

[0077] It is understandable that the above example uses Sij-a as an example for illustration. In actual execution, the sub-probability matrices Pij (sub-blocks) with the same format as each Sij-a but different values ​​can be rearranged so that the rearranged matrix can be directly multiplied with the sub-value matrix Vj according to the MMA instruction. Finally, based on the rearranged sub-probability matrix (reduction result) and the sub-value matrix, matrix multiplication is performed based on the matrix multiplication accumulation instruction MMA, and the complete normalized exponential function calculation is completed to obtain the target attention matrix for use in attention calculation.

[0078] Furthermore, as in the example above, data interaction between threads (such as regC0_0, regC1_0, regC0_1, and regC1_1, representing elements distributed across attention calculation threads 0 and 1) is required to complete the final layout rearrangement. However, the computation process in this application is executed within a private register set. Different threads can only access their corresponding private registers and cannot directly perform data interaction between threads or between private registers across threads. Therefore, this example supports using the Warp Shuffle instruction within the bundle to perform cross-thread and cross-register access, thus achieving the above layout rearrangement process.

[0079] In this example, matrix multiplication-accumulation instructions are first selected to further improve the speed of target attention calculation. Simultaneously, considering that the proposed method of completing attention calculation entirely within a private register set is not directly compatible with matrix multiplication-accumulation instructions, a layout rearrangement of the parameters involved in the matrix multiplication calculation is performed before the execution of the second set of matrix multiplication-accumulation instructions. This rearrangement ensures that the parameters match the matrix multiplication-accumulation instructions, guaranteeing their correct execution. This approach not only improves the speed of attention calculation based on matrix multiplication-accumulation instructions but also ensures their smooth execution, maintaining the efficiency of attention calculation.

[0080] In some examples, the reduction result includes a first-level reduction result and a second-level reduction result, and the step of performing layout rearrangement on the reduction result according to preset rules includes: S31, obtain the sub-probability matrix according to each of the first-level reduction results; S32, the sub-probability matrix is ​​rearranged according to a preset rule to obtain the target sub-probability matrix; the target sub-probability matrix is ​​adapted to the matrix multiplication and accumulation instruction; Then, the execution of the matrix multiplication and accumulation instruction, and the completion of the normalized exponential function calculation to obtain the target attention matrix, includes: S33, Execute the matrix multiplication and accumulation instruction on the target sub-probability matrix and the sub-value matrix to obtain the primary sub-attention matrix; S34, obtain the sub-target attention matrix based on the primary sub-attention matrix and the secondary reduction result, and obtain the target attention matrix based on each sub-target attention matrix.

[0081] Specifically, in this example, the calculation of the normalization exponential function is divided into two stages. The first stage involves obtaining the sub-probability matrix based on the first-level reduction result; the output of the first stage is the sub-probability matrix. Then, matrix multiplication is performed between the sub-probability matrix and the sub-value matrix to obtain the sub-primary attention matrix. The second stage involves normalizing the sub-primary attention matrix based on the second-level reduction result to obtain the sub-target attention matrix. Finally, the sub-target attention matrices corresponding to each sub-query matrix are concatenated to obtain the target attention matrix, which is then output. Figure 3 As shown, a subquery matrix Qi will correspond to a subtarget attention matrix Yi. By concatenating all the Yi corresponding to Qi, the final target attention matrix Y can be obtained.

[0082] The second-order reduction result is used to complete the calculation of the normalized exponential function.

[0083] The preset rules are used to adapt the data arrangement of the reduction result after layout rearrangement to the input format of the matrix multiplication and accumulation instruction.

[0084] Optionally, the first-level reduction result is the maximum value of the matrix rows (i.e., the fractional row vectors), and the second-level reduction result is the sum of the exponents of the matrix rows (i.e., the fractional row vectors).

[0085] Understandably, in this example, the first-level and second-level reduction results only need to be obtained before their use, and are not required to be obtained simultaneously. For example, only the first-level reduction result can be obtained before the sub-probability matrix is ​​generated, and the second-level reduction result can be obtained before the normalization of the primary sub-attention matrix is ​​performed, such as before or after the matrix multiplication involving the sub-value matrix. This allows for more flexible execution of the complete calculation scheme and can be adjusted to meet users' temporary and sudden needs.

[0086] In some examples, where the first-level reduction result is the maximum value of a matrix row, the first-level reduction result is the maximum value of each element in each fractional row vector of the sub-fractional matrix Sij. In step S31, for each fractional row vector in the sub-fractional matrix, the first-level reduction result (i.e., the maximum value of the current matrix row) can be subtracted from all elements, and then the exponent can be calculated to obtain the matrix row of the sub-probability matrix corresponding to the current row. By integrating the matrix rows of the sub-probability matrices obtained from all rows, the sub-probability matrix Pij corresponding to the sub-fractional matrix Sij can be obtained.

[0087] For example, the elements in rows Sx and Sy of the sub-fraction matrix Sij are: Sx1, Sx2, Sx3, Sx4; Sy1, Sy2, Sy3, Sy4; Given that the first-order reduction results (maximum row values) are Sx1 and Sy3 respectively, the matrix order for calculating the corresponding sub-probability matrix for row Sx is: exp(Sx1-Sx1), exp(Sx2-Sx1), exp(Sx3-Sx1), exp(Sx4-Sx1); and the matrix order for calculating the corresponding sub-probability matrix for row Sy is: exp(Sy1-Sy3), exp(Sy2-Sy3), exp(Sy3-Sy3), exp(Sy4-Sy3). Integrating the calculation results of the two rows of parameters yields the Px and Py rows in the corresponding sub-probability matrix Pij. exp(Sx1- Sx1), exp(Sx2- Sx1), exp(Sx3- Sx1), exp(Sx4- Sx1); exp(Sy1- Sy3), exp(Sy2- Sy3), exp(Sy3- Sy3), exp(Sy4- Sy3).

[0088] The exp() function performs the exponentiation operation. Subtracting the maximum value of the current row before calculating the exponent for each element stabilizes the value and prevents data overflow during exponentiation, avoiding results such as infinity (inf).

[0089] After obtaining the sub-probability matrix, it can be rearranged according to preset rules to obtain the target sub-probability matrix. The data arrangement and storage of the target sub-probability matrix meet the requirements of matrix multiplication and accumulation instructions, and matrix multiplication can be performed with the sub-value matrix according to the matrix multiplication and accumulation instructions.

[0090] For example, the matrix multiplication-accumulation instruction (MMA) requires data input in a fixed format. Therefore, in this example, the results of the previous matrix multiplication are rearranged to fit the MMA instruction.

[0091] After rearranging the layout to obtain the target sub-probability matrix, it can be multiplied and accumulated with the sub-value matrix Vj using a matrix multiplication instruction to obtain the primary sub-attention matrix. For example, a single target sub-probability matrix P'ij needs to be multiplied with its corresponding single sub-value matrix Vj to obtain the primary sub-attention contribution matrix. O, contribution matrix for all primary sub-attentions O is accumulated to obtain the primary sub-attention matrix Oi corresponding to the current sub-query matrix Qi; where, .like Figure 3 As shown, the target sub-probability matrix has a size of Br×Bc, and the size of a single sub-value matrix Vj in the value matrix V is Bc×d. Therefore, the primary sub-attention matrix has a size of Br×d. The primary sub-attention contribution matrix is ​​calculated from all the sub-score matrices Sij (the resulting target sub-probability matrix P'ij) corresponding to a single sub-query matrix Qi. O, and the primary sub-attention contribution matrix The primary sub-attention matrix obtained after combining O is also of size Br×d. That is, the primary sub-attention contribution matrix, the primary sub-attention matrix, and the sub-target attention matrix have the same size; only the primary sub-attention contribution matrix... The parameters in O only contain information from their corresponding subkey matrices. The primary sub-attention matrix Oi obtained after accumulation includes information from the subquery matrix Qi and all its corresponding subkey matrices Kj. The sub-target attention matrix Yi is calculated by normalizing Oi. The sub-target attention matrices Yi corresponding to all subquery matrices Qi can constitute the target attention matrix.

[0092] For example, when Br is 2 and d is 3, a primary sub-attention contribution matrix The elements in On are:

[0093] Another primary sub-attention contribution matrix The elements in Om are:

[0094] After summing the two primary sub-attention contribution matrices above, the resulting primary sub-attention matrix Oi (= On+ Om) is:

[0095] With a normalization coefficient of f, normalizing the primary sub-attention matrix Oi yields the sub-target attention matrix Yi, which can be:

[0096] Therefore, each primary sub-attention matrix Oi and each primary sub-attention contribution matrix O and the attention matrix Yi of each sub-target have the same matrix specifications, the difference being that the elements include the influence of different sub-key matrices, and whether normalization has been completed based on the second-level reduction result.

[0097] After obtaining the primary sub-attention matrix Oi, normalization needs to be performed based on the secondary reduction result. Specifically, the j target sub-probability matrices Pij are multiplied by each sub-value matrix Vj in the complete value matrix to obtain the j primary sub-attention contribution matrices. O, summing them up yields the primary sub-attention matrix Oi, which is then normalized to obtain the sub-target attention matrix Yi. The sub-target attention matrix is ​​obtained from the primary sub-attention matrix using the second-level reduction result l, for example, Yi = Oi / l.

[0098] Specifically, since the primary sub-attention matrix Oi is a contribution matrix of multiple primary sub-attention matrices... The accumulated value of O, and the second-order reduction result corresponding to the primary sub-attention matrix Oi also changes in real time during the accumulation process (because each primary sub-attention contribution matrix O has a corresponding second-order reduction result, that is, the second-order reduction result corresponding to the accumulated primary sub-attention matrix Oi also has a certain iterative process, for example, Oi= O_old+ O_new, O_new represents the matrix product of the single target sub-probability matrix and the single sub-value matrix calculated in this operation. O_old represents the cumulative value of the matrix product of a single target sub-probability matrix and a single sub-value matrix (i.e., the historical primary sub-attention contribution matrix), which is the real-time primary sub-attention matrix before this calculation. Furthermore, since the sub-score matrix Sij does not include the complete row vectors in the sub-score block rows Si (which are obtained by horizontally concatenating j sub-score matrices Sij), the maximum value of each row may be updated during the iteration process based on each sub-score matrix Sij. Since the first-level and second-level reduction results are calculated based on the real-time complete rows (i.e., the real-time complete row vectors), this will affect the first-level and second-level reduction results, and consequently, the historical cumulative value. O_old ultimately affects the target attention matrix. That is, it requires accumulating parameters from historical data. O_old adjusts its parameters based on a correction factor to adapt to the real-time first-level reduction result (the maximum value in the fractional row vector) and second-level reduction result (the exponential sum in the fractional row vector). Meanwhile, because... O_new is calculated based on the real-time first-level and second-level reduction results, so no further adjustment is needed based on the correction factor.

[0099] Specifically, for example, the sub-fraction matrix S 11 The maximum value of row 3 is obtained by maximum reduction, which is a1. However, in the iterative processing of the sub-fraction matrix S... 12 At that time, S 12 If the maximum value of the third row in the fraction matrix S is a2, and a1 < a2, then for the first fraction block row S1 in the fraction matrix S, the real-time maximum value of the third row (i.e., the third row vector) within the block is a2. The final sub-target attention matrix Y1 needs to correspond to the current first fraction block row S1 (i.e., to a single subquery matrix Q1), not to the sub-fraction matrix S. 11 Therefore, the previously accumulated values ​​need to be updated based on the real-time maximum values.

[0100] For example, at this time, the sub-target attention matrix Y1 = ( O_old / α+ O_new) / (l_old / α+l_new), where α is a correction factor used to reflect the iterative update caused by the change in the maximum value. O_old and l_old are based on the previous round S 11 The calculated primary sub-attention matrix and the sum of the exponents of each row of the matrix. O_new and l_new are based on the current S 12 The corresponding calculated primary sub-attention contribution matrix and S 12 The sum of the exponents of each row of the matrix, Y1 is based on S 12The calculation results are updated in real time to the sub-target attention matrix, which is the result of normalizing the real-time primary sub-attention matrix based on the exponent and the result.

[0101] This process is repeated for all subquery matrices until all subquery matrices have been processed. The resulting sub-target attention matrices are then integrated in sequence to form the final target attention matrix.

[0102] It is understandable that, such as Figure 3 As shown, a subquery matrix Qi is multiplied by all subkey matrices Kj, resulting in a Si. The product of one Qi and one Kj is a Sij (shaded). Furthermore, each Sij corresponds to a Pij, so a Si including all Sij will correspond to a row matrix composed of all Pij (identified by the blank and shaded squares in the diagram). Subsequently, the P matrix needs to be multiplied by the V matrix, for example, by multiplying the shaded Pij by the shaded Vj. Simultaneously, one-to-one matrix multiplications are performed between the Pij and Vj in the blank squares. After multiplying each row of Pij by the complete V matrix, the resulting Yi is shown in the diagram. In other words, a subquery matrix Qi will generate a subtarget block matrix Yi.

[0103] In this example, the layout of each sub-probability matrix is ​​rearranged so that matrix multiplication between the sub-probability matrix and the sub-value matrix can be performed using the matrix multiplication and addition instruction (MMA). The result of the matrix multiplication is then normalized based on the second-level reduction result (i.e., the exponent sum of each row), completing the normalized exponent function (i.e., softmax) calculation in the attention calculation process. Specifically, the unnormalized target sub-probability matrix and sub-value matrix are multiplied first. The normalization process is then performed after the matrix multiplication with the sub-value matrix. This means that the exponent sum of the current row does not need to be calculated simultaneously with the calculation of the sub-probability matrix Pij (i.e., before the matrix multiplication with the sub-value matrix), and each Pij is normalized based on its corresponding exponent sum. This ensures that the second-level reduction result (i.e., the exponent sum of each row) is obtained before the normalization calculation of the primary sub-attention matrix. This embodiment saves memory and reduces the division steps used for normalization (normalization is not performed for each Pij, only for the final Oi), further reducing computational load and improving computational efficiency.

[0104] For example, the reduction includes maximum value reduction and exponential sum reduction; then, row-level reduction is performed on each of the sub-fraction matrices, including: S41, For each fraction row vector in the sub-fraction matrix, a comparison operation is performed in conjunction with the bundle shuffle instruction to determine the maximum value of each element in the fraction row vector, which is used as the first-level reduction result; S42, based on each element in the fractional row vector and the first-level reduction result, calculate the exponent sum of the fractional row vector as the second-level reduction result.

[0105] Specifically, in this application, the attention calculation thread block includes multiple attention calculation threads, and an attention calculation thread block includes at least one warp thread bundle, that is, at least 32 attention calculation threads, and the threads in the same warp thread bundle support intra-warp communication.

[0106] Here, the fractional row vector is a row parameter in the subfractional matrix.

[0107] Taking an attention calculation thread block that includes a warp thread bundle as an example, if four attention calculation threads can work together to process a row vector, and a warp thread bundle typically includes 32 threads, then eight rows of data in the matrix can be processed simultaneously, that is, eight row vectors can be processed simultaneously.

[0108] The first-level reduction result can be calculated using the following exemplary instructions: float local_max = thread_local_max; local_max = max(local_max, __shfl_down_sync(FULL_MASK, local_max,2)); local_max = max(local_max, __shfl_down_sync(FULL_MASK, local_max,1)); float row_max = __shfl_sync(FULL_MASK, local_max, row_base_lane); Here, `float local_max` means that the local maximum value `thread_local_max` held by each attention calculation thread, which was calculated in the steps not shown in the previous section, is assigned to its corresponding `local_max`, and is presented in float format.

[0109] `local_max = max(local_max, __shfl_down_sync(FULL_MASK, local_max,2));` is used to compare the local maximum values ​​between two attention calculation threads with a step size of 2, and update the previous thread with the maximum value. For example, if there are four threads (threads 0-3), since the step size is 2, the interaction occurs between threads 0 and 2, writing the maximum value to thread 0; the interaction occurs between threads 1 and 3, writing the maximum value to thread 1. `_shfl_down_sync` is a shuffle-down instruction, indicating a downward offset, i.e., obtaining the parameter value from the adjacent thread with a step size of 2 and writing it to the previous thread (i.e., thread 0 obtains the local maximum value from thread 2). `FULL_MASK` indicates that the current (2-step size comparison and offset) instruction is executed on all attention calculation threads processing the current row. `max(local_max,...)` means comparing the parameters to find the maximum value; that is, in thread 0, the original local maximum value is compared with the obtained local maximum value from thread 2, and the maximum value is used to update and store the result.

[0110] Here, shfl_down_sync is a shuffle down instruction, which means that the target parameter is placed in the front thread. It can be replaced with shfl_up_sync as an up shuffle instruction, which means that the target parameter is placed in the back thread, depending on the actual usage requirements.

[0111] `local_max = max(local_max, __shfl_down_sync(FULL_MASK, local_max,1));` is used to compare the local maximum values ​​between two threads with a step size of 1, and update the previous thread with the larger of the two values. For example, when interacting between threads 0 and 1, the maximum value of the two is written to thread 0; when interacting between threads 2 and 3, the maximum value of the two is written to thread 2. At this point, thread 0 stores the maximum values ​​of the four threads.

[0112] Then, according to the instruction float row_max = __shfl_sync(FULL_MASK, local_max, row_base_lane), the maximum value in thread 0 is broadcast so that all current attention computation threads know the real-time maximum value of this row, i.e., the first-level reduction result.

[0113] Here, `__shfl_sync` indicates broadcasting, and `row_base_lane` is used to support threads storing the maximum value, with the constraint `row_base_lane = tid - tid % 4`. `tid - tid % 4` indicates that the maximum value among every four threads should be written to the first thread (in this example, a single-row matrix is ​​processed collaboratively by four threads, meaning the maximum value of each row is written to the first thread among every four threads), i.e., thread 0 in this example.

[0114] `tid` is a unique identifier for each thread in a warp. For a warp with 32 threads, the `tid` number ranges from 0 to 31. Further, with limitations: tid = 0, row_base_lane = 0 tid = 1, row_base_lane = 0 tid = 2, row_base_lane = 0 tid = 3, row_base_lane = 0 tid = 4, row_base_lane = 4 tid = 5, row_base_lane = 4 tid = 6, row_base_lane = 4 tid = 7, row_base_lane = 4 ... Therefore, the row_base_lane corresponding to threads 0, 1, 2, and 3 is thread 0. This means the maximum value among threads 0-3 is written to thread 0 and broadcast; the maximum value among threads 4-7 is written to thread 4 and broadcast, and so on. It can be understood that __shfl_ is a prefix for the bundle shuffle instruction, meaning all the above steps are completed using the bundle shuffle instruction.

[0115] Furthermore, based on each element in the matrix and the first-level reduction result, the exponent sum of the fractional row vectors is calculated as the second-level reduction result. That is, for each element, the difference is first calculated with the first-level reduction result (i.e., the maximum value of the current row), then the exponent is calculated, and the exponents of the entire row are summed to obtain the exponent sum of each row, which is the second-level reduction result.

[0116] Specifically, this can be achieved through the following exemplary instructions: float sum_exp = 0.0f; for each element in thread's part: float p = expf(S - m_new); sum_exp += p; The above instruction blocks are used to calculate the exponent corresponding to each element, including: first, initializing the exponent and sum_exp to 0.0, and making them floating-point numbers. Then, for each thread (i.e., for each element in thread'spart), the exponent corresponding to the partial elements of that thread is calculated: float p = expf(S - m_new), that is, for each element S in the sub-fraction matrix, the real-time maximum value m_new is subtracted and the exponent is calculated, finally obtaining the local exponents corresponding to the four threads. Then, the in-bundle exponents are summed, for example: sum_exp += __shfl_down_sync(FULL_MASK, sum_exp, 2); sum_exp += __shfl_down_sync(FULL_MASK, sum_exp, 1); float row_sum = __shfl_sync(FULL_MASK, sum_exp, row_base_lane); As mentioned above, __shfl_down_sync(FULL_MASK, sum_exp, 2) is used to retrieve the value of a thread with a step size of 2. In this case, no comparison is required; the value can be retrieved simply by using __shfl_down_sync, thus obtaining the sum of the two threads.

[0117] Specifically, `sum_exp += __shfl_down_sync(FULL_MASK, sum_exp, 2);` is used to calculate the exponential sum of threads 0 and 2 and write it to thread 0, and to calculate the exponential sum of threads 1 and 3 and write it to thread 1. `sum_exp += __shfl_down_sync(FULL_MASK, sum_exp, 1);` is used to calculate the exponential sum of threads 0 and 1 and write it to thread 0, and to calculate the exponential sum of threads 2 and 3 and write it to thread 2. At this point, thread 0 contains the exponential sum of the current four threads. Then, a broadcast is performed using `float row_sum = __shfl_sync(FULL_MASK, sum_exp, row_base_lane)` to make the real-time exponential sum of this row known to all currently focused computation threads.

[0118] Here, `row_base_lane` can also be limited according to the example above: `row_base_lane = tid - tid% 4`. That is, the `row_base_lane` corresponding to threads 0, 1, 2, and 3 is thread 0, meaning the sum of the exponents of threads 0, 1, 2, and 3 is written to thread 0 and broadcast; the sum of the exponents of threads 4-7 is written to thread 4 and broadcast, and so on.

[0119] In this example, the in-bundle shuffle instruction enables communication between the private registers corresponding to each attention calculation thread. That is, data interaction is completed in the warp communication domain corresponding to multiple attention calculation threads, without the need for shared memory as an intermediary. This ensures that the attention calculation process is completed entirely within the private register group, avoiding data interaction with shared memory and maintaining the speed of attention calculation.

[0120] In one example, the attention calculation thread block includes multiple attention calculation threads, each attention calculation thread corresponds to multiple private registers, and the multiple private registers constitute a private register group; Then, the method may further include: S51, if the bit width of the effective data in a single attention calculation thread is less than the target bit width, the effective data of the attention calculation thread is concatenated in a preset private register in the private register group to achieve the target bit width; S52, using the target bit width as the basic calculation bit width for the bundle shuffle instruction, and executing the normalized exponential function calculation based on the reduction result and the subvalue matrix.

[0121] In this example, the attention computation thread block includes multiple attention computation threads, each with its own private register. The parameters in the sub-probability matrix may be regularly distributed across these private registers. To address this, this application employs an intra-bundle shuffle instruction to perform intra-bundle parameter exchange between the attention computation threads within the current attention computation thread block, directly achieving layout rearrangement without requiring shared memory as an intermediary.

[0122] In some cases, layout rearrangement can be achieved using the following directives, for example: / / Based on the default swizzle layout requirements of the MMA directives, the original (packed) elements are redistributed across different threads. / / Data rearrangement between threads is accomplished using instructions such as _shfl_up_sync (shuffle up) and shfl_down_sync (shuffle down). uint32_t final_value = shuffle_redistribute(packed); p_frag[index] = final_value; Where uint32_t represents a 32-bit unsigned integer; final_value is the final variable name of the target sub-probability matrix; p_frag[] represents the register set or pointer parameter, and index is used to identify the target register for storing the rearranged parameters. shuffle_redistribute() is used to refer to a custom rearrangement function, which can be replaced during actual execution with a code segment containing shuffling instructions within the warp shuffle bundle, such as _shfl_up_sync / _shfl_down_sync, to complete the custom rearrangement.

[0123] Furthermore, the shuffling instruction within the bundle has a base startup or effective computation bit width (i.e., target bit width), for example, 32 bits. This means that if the effective data in a single attention computation thread is less than 32 bits wide, the shuffling instruction within the bundle cannot be executed correctly, and the parameters may be rolled back to shared memory to complete subsequent computations. In order to avoid using shared memory as an intermediary and to maintain the correct execution of the shuffling instruction within the bundle, this application needs to concatenate the effective data from the attention computation thread to solve the bit width problem of the effective data.

[0124] To address this, this example proposes concatenating valid data from different attention calculation threads to achieve the target bit width, ensuring correct execution of the shuffle instruction within the bundle. Furthermore, after execution, the layout rearrangement result can be broadcast to the private registers corresponding to all attention calculation threads. Even if the current thread or register did not participate in the layout rearrangement of the shuffle instruction within the bundle, it can still receive the correct layout information. Simultaneously, the preset private register for concatenating each valid data is also a register in the current private register group, which is within the read / write range of the shuffle instruction within the bundle, ensuring that the attention calculation process resides within the current attention calculation thread block.

[0125] Furthermore, the step of concatenating the valid data in the attention calculation thread to achieve the target bit width includes: concatenating the valid data in adjacent threads to achieve the target bit width; the adjacent threads are the attention calculation threads adjacent to the attention calculation thread corresponding to the preset private register. That is, in this example, concatenating the valid data in adjacent attention calculation threads, compared to randomly selecting attention calculation threads for concatenation, can maintain the original parameter order, and the addresses of adjacent parameters are continuous, simplifying access rules and improving access efficiency; in addition, it can also reduce memory access errors such as parameter out-of-bounds errors.

[0126] Specifically, effective data concatenation can be achieved through the following exemplary instructions: First, the scattered valid data is aggregated into a preset private register, specifically the preset private register corresponding to the preset attention calculation thread. For example, the attention calculation threads are divided into odd-numbered threads and even-numbered threads, with even-numbered threads serving as the preset attention calculation threads. Taking an even-numbered thread and an adjacent odd-numbered thread as an example, if each thread holds two quantized values ​​(UINT8, which are the element values ​​in the quantized sub-probability matrix), then the following can be executed: uint8_t byte_a = quantize(softmax_out0); This function quantizes the element value softmax_out0 in the first sub-probability matrix of the current thread (even-numbered threads), converting it from a floating-point element value to a UINT8 value, which is then used as byte_a; the quantize() function is used to complete the quantization. uint8_t byte_b = quantize(softmax_out1); This is used to quantize the element value softmax_out1 in the second sub-probability matrix of this thread, converting it from a floating-point element value to a UINT8 value, which is then used as byte_b; that is, the above two steps are used to make the current thread hold two quantized values.

[0127] Subsequently, the quantized value is obtained from the adjacent thread (odd-numbered thread) through a Warp Shuffle shuffle.

[0128] uint8_t byte_c = __shfl_down_sync(FULL_MASK, byte_a, 1); uint8_t byte_d = __shfl_down_sync(FULL_MASK, byte_b, 1); The `__shfl_down_sync()` instruction is a shuffle instruction used to retrieve values. `uint8_t byte_c = __shfl_down_sync(FULL_MASK, byte_a, 1);` means retrieving `byte_a` from the adjacent thread (odd-numbered thread) with a step size of 1 as `byte_c` for the current thread (even-numbered thread); then retrieving `byte_b` from the adjacent thread as `byte_d` for the current thread.

[0129] After obtaining valid data, the concatenation and packaging process is performed in this thread (even-numbered thread). Specifically, the following instructions can be executed: `uint32_t packed = 0;` is used to initialize a 32-bit unsigned integer. if (is_even_thread) { packed = (byte_a) | (byte_b<<8) | (byte_c<<16) | (byte_d<<24); } `is_even_thread` is a boolean condition; subsequent steps will only be executed if the condition of an even number of threads (`even_thread`) is met. That is, the concatenation and packing process is only executed by an even number of threads (the preset attention calculation thread corresponding to the preset private register).

[0130] Specifically, (byte_a) | (byte_b<<8) | (byte_c<<16) | (byte_d<<24) is a bitwise OR operation used to place the four quantized values ​​into different 8-bit segments of a 32-bit unsigned integer. Since each byte occupies a unique space, they are directly concatenated. The final concatenated value, `packed`, is used for layout reordering, which can then be referred to the aforementioned layout reordering instructions. uint32_t final_value = shuffle_redistribute(packed); p_frag[index] = final_value; Execution.

[0131] In this example, placement rearrangement is implemented using an in-bundle shuffle instruction. This allows the placement rearrangement to be performed within a private register set, eliminating the need for shared memory or other intermediaries. Furthermore, to accommodate the in-bundle shuffle instruction, effective data concatenation is considered to ensure it meets the target bit width of the in-bundle shuffle instruction, thereby guaranteeing its execution efficiency and maintaining attention calculation speed.

[0132] It is understandable that after the current data concatenation step is executed, a second matrix multiplication (i.e., the second MMA process) will be performed on the concatenation result and the sub-value matrix. That is, the current concatenation step corresponds to the step in the previous example of "concatenating the elements in regC0_0, regC1_0, regC0_1 and regC1_1 into a form that the original regA0_0 thread can process".

[0133] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0134] Reference Figure 5The diagram illustrates a structural block diagram of an embodiment of a data processing apparatus according to the present invention. The data processing apparatus may include: The matrix partitioning module 201 is used to divide the query matrix, key matrix and value matrix corresponding to the input sequence into sub-query matrix, sub-key matrix and sub-value matrix based on the private register group corresponding to the attention calculation thread block; The block loading module 202 is used to load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; The score acquisition module 203 is used to obtain the sub-score matrix corresponding to the sub-query matrix based on the sub-query matrix and each sub-key matrix in the private register group, and write the sub-score matrix into the private register group; The result generation module 204 is used to perform row-level reduction on each of the sub-score matrices within the private register group, and to calculate the normalized exponential function based on the reduction result and the sub-value matrix to obtain the target attention matrix.

[0135] Optionally, the result generation module 204 includes: The rearrangement module is used to perform layout rearrangement on the reduction result according to preset rules, wherein the preset rules are used to make the reduction result after layout rearrangement compatible with matrix multiplication and accumulation instructions. The normalization module is used to perform matrix multiplication and accumulation instructions on the reduction result after the layout rearrangement and the subvalue matrix, and to complete the calculation of the normalization exponential function to obtain the target attention matrix.

[0136] Optionally, the reduction result includes a first-level reduction result and a second-level reduction result, and the rearrangement module includes: The probability acquisition module is used to obtain the sub-probability matrix based on the first-level reduction result; The layout rearrangement module is used to perform layout rearrangement on the sub-probability matrix according to preset rules to obtain the target sub-probability matrix; the target sub-probability matrix is ​​adapted to matrix multiplication and accumulation instructions. Then, the normalization module includes: The matrix multiplication module is used to perform matrix multiplication and accumulation instructions on the target sub-probability matrix and the sub-value matrix to obtain the primary sub-attention matrix; The target acquisition module obtains a sub-target attention matrix from the primary sub-attention matrix based on the secondary reduction result, and obtains the target attention matrix based on each sub-target attention matrix; wherein, the secondary reduction result is used to complete the calculation of the normalized exponential function.

[0137] Optionally, the result generation module 204 further includes: The first reduction matrix is ​​used to perform comparison operations on each fraction row vector in the sub-fraction matrix using a bundle shuffle instruction to determine the maximum value of each element in the fraction row vector, which is then used as the first-level reduction result. The second reduction matrix is ​​used to calculate the sum of the exponents of the fractional row vector based on each element in the fractional row vector and the first-level reduction result, as the second-level reduction result.

[0138] Optionally, the attention calculation thread block includes multiple attention calculation threads, each attention calculation thread corresponding to multiple private registers, and the multiple private registers constitute the private register group; the layout rearrangement module specifically includes: The sub-probability matrix is ​​rearranged according to preset rules using the in-bundle shuffle command; Then, the device further includes: The splicing module is used to splice the valid data in the attention calculation thread in a preset private register in the private register group to achieve the target bit width when the bit width of the valid data in a single attention calculation thread is less than the target bit width; wherein, the target bit width is the basic calculation bit width of the bundle shuffle instruction.

[0139] Optionally, the splicing module includes: The effective data in adjacent threads are concatenated to achieve the target bit width. The adjacent threads are the attention calculation threads adjacent to the attention calculation threads corresponding to the preset private register.

[0140] Optionally, the matrix partitioning module 201 includes: The specification confirmation module is used to determine the row block specification and column block specification based on the storage space size of the private register group corresponding to the attention calculation thread block; The rule partitioning module is used to partition the query matrix corresponding to the input sequence into multiple sub-query matrices according to the row partitioning specification, partition the key matrix corresponding to the input sequence into multiple sub-key matrices according to the column partitioning specification, and partition the value matrix corresponding to the input sequence into multiple sub-value matrices according to the column partitioning specification.

[0141] This invention provides a data processing apparatus. First, the matrices involved in attention calculation are divided according to the private register set corresponding to the attention calculation thread block. This includes dividing the query matrix, key matrix, and value matrix, ensuring that the size of each sub-matrix matches the private register set. Therefore, the divided sub-matrices (such as sub-query matrices, sub-key matrices, and sub-value matrices) can be loaded into the private register set, and attention calculation can then be performed within the private register set. During the calculation, intermediate parameters are written to the current private register set instead of being read from shared memory. Since the speed at which the computing unit reads and writes data in the private register (set) is usually faster than the speed at which it reads and writes data in shared memory or video memory, performing attention calculation based on the parameters stored in the private register set will reduce computation latency and improve computational efficiency.

[0142] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0143] Reference Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 6 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the steps of the data processing method of the aforementioned embodiment.

[0144] This invention provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a program or processor of a terminal, enables the terminal to perform the steps of the data processing method described in the foregoing embodiments.

[0145] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: Based on the private register group corresponding to the attention computation thread block, the query matrix, key matrix and value matrix corresponding to the input sequence are divided into blocks to obtain sub-query matrix, sub-key matrix and sub-value matrix; Load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; Based on the subquery matrix and each subkey matrix in the private register group, the sub-fraction matrix corresponding to the subquery matrix is ​​obtained and written into the private register group; Within the private register group, row-level reduction is performed on each of the sub-score matrices, and the normalized exponential function is calculated based on the reduction result and the sub-value matrix until all sub-query matrices are processed, thus obtaining the target attention matrix.

2. The data processing method according to claim 1, characterized in that, The row-level reduction of each of the sub-fraction matrices includes: For each fraction row vector in the sub-fraction matrix, a comparison operation is performed in conjunction with the bundle shuffle instruction to determine the maximum value of each element in the fraction row vector, which is used as the first-level reduction result; Based on each element in the fractional row vector and the first-level reduction result, the exponent sum of the fractional row vector is calculated as the second-level reduction result.

3. The data processing method according to claim 2, characterized in that, The calculation of the normalized exponential function based on the reduction result and the sub-value matrix includes: The reduction result is rearranged according to a preset rule; the preset rule is used to make the reduction result after the rearranged layout compatible with matrix multiplication and accumulation instructions; The matrix multiplication and accumulation instruction is executed on the reduced result after the layout rearrangement and the subvalue matrix, and the normalized exponential function is calculated to obtain the target attention matrix corresponding to the input sequence.

4. The data processing method according to claim 3, characterized in that, The step of rearranging the layout of the reduction result according to preset rules includes: The sub-probability matrices are obtained based on the first-order reduction results described above; The sub-probability matrix is ​​rearranged according to a preset rule to obtain the target sub-probability matrix; the target sub-probability matrix is ​​adapted to the matrix multiplication and accumulation instruction. Then, executing the matrix multiplication and accumulation instruction and completing the normalized exponential function calculation includes: The matrix multiplication and accumulation instruction is executed on the target sub-probability matrix and the sub-value matrix to obtain the primary sub-attention matrix; The sub-target attention matrix is ​​obtained based on the primary sub-attention matrix and the secondary reduction result, and the target attention matrix is ​​obtained by concatenating the sub-target attention matrices.

5. The data processing method according to claim 1, characterized in that, The attention calculation thread block includes multiple attention calculation threads, each attention calculation thread corresponds to multiple private registers, and the multiple private registers constitute the private register group; Then, the method further includes: when the bit width of the effective data in a single attention calculation thread is less than the target bit width, concatenating the effective data in the attention calculation thread in a preset private register in the private register group to achieve the target bit width; The target bit width is used as the base calculation bit width for the bundle shuffle instruction, and the normalized exponential function is calculated based on the reduction result and the subvalue matrix.

6. The data processing method according to claim 5, characterized in that, The step of concatenating the valid data in the attention calculation thread to achieve the target bit width includes: The effective data in adjacent threads are concatenated to achieve the target bit width; the adjacent threads are the attention calculation threads adjacent to the attention calculation threads corresponding to the preset private register.

7. The data processing method according to claim 1, characterized in that, The step of dividing the query matrix, key matrix, and value matrix corresponding to the input sequence into blocks to obtain sub-query matrix, sub-key matrix, and sub-value matrix includes: Based on the storage space size of the private register group, determine the row block size and column block size; The query matrix is ​​divided into multiple sub-query matrices according to the row block specification, the key matrix is ​​divided into multiple sub-key matrices according to the column block specification, and the value matrix is ​​divided into multiple sub-value matrices according to the column block specification.

8. A data processing apparatus, characterized in that, The device includes: The matrix partitioning module is used to divide the query matrix, key matrix, and value matrix corresponding to the input sequence into sub-query matrix, sub-key matrix, and sub-value matrix based on the private register group corresponding to the attention computation thread block. The block loading module is used to load any of the subquery matrices and each of the subkey matrices into the private register group corresponding to the subquery matrix; The score acquisition module is used to obtain the sub-score matrix corresponding to the sub-query matrix based on the sub-query matrix and each sub-key matrix in the private register group, and write it into the private register group; The result generation module is used to perform row-level reduction on each of the sub-score matrices within the private register group, and to calculate the normalized exponential function based on the reduction result and the sub-value matrix, until all sub-query matrices have been processed to obtain the target attention matrix.

9. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the data processing method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 7.