Attention computation optimization methods, electronic devices, storage media, and application products

Abstract

This invention relates to the field of artificial intelligence technology, providing an attention computation optimization method, electronic device, storage medium, and program product. The method includes: generating a bounding box list of an attention matrix based on mask information for attention computation, wherein the bounding box list includes at least one valid bounding box, which defines the spatial range of a target data block in the attention matrix to be subject to attention computation; loading the target data block corresponding to each valid bounding box from a query matrix, a key matrix, and a value matrix based on each valid bounding box in the bounding box list, and performing attention computation on the loaded target data block to obtain the attention output corresponding to each valid bounding box. This invention, by generating a bounding box list and loading and performing attention computation only on data blocks within valid bounding boxes in the list, avoids invalid computation and waste of computing resources, thereby improving overall computational performance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an attention calculation optimization method, electronic device, storage medium, and program product. Background Technology

[0002] In existing mainstream model architectures such as Transformer, the attention mechanism is a core component. In the attention calculation process, mask filling is a key step. It fills specific positions (such as future positions or filling positions) in the attention score matrix with a large negative value, so that the weights of these positions are zeroed after subsequent Softmax normalization. This precisely controls the information flow of the model and ensures that the model does not pay attention to invalid information.

[0003] To improve computational efficiency, especially when processing long sequences, block-based attention methods, such as FlashAttention, have been proposed. These methods divide the massive attention matrix into several data blocks and utilize cache to perform computation in blocks, thereby reducing the number of accesses to high-latency memory and significantly accelerating the computation process.

[0004] However, existing block attention methods have certain drawbacks when combined with masking mechanisms. When all positions within a data block need to be masked, existing methods still mechanically perform complete data loading, matrix multiplication, and Softmax operations on the invalid block, producing a series of intermediate results that are ultimately discarded. This inefficient computation is particularly problematic when processing long sequences, resulting in a significant waste of computational resources and limiting the overall performance of model training and inference. Summary of the Invention

[0005] This invention provides an attention computation optimization method, electronic device, storage medium, and program product to address the shortcomings of masked attention computation in related technologies, such as wasted computing resources and low efficiency in model training and inference.

[0006] This invention provides an attention calculation optimization method, comprising:

[0007] Based on the mask information of attention computation, a list of bounding boxes for attention matrix is ​​generated. The list of bounding boxes includes at least one valid bounding box, which is used to define the spatial range of the target data block to be performed on attention computation in the attention matrix.

[0008] Based on each valid bounding box in the bounding box list, the target data block corresponding to each valid bounding box is loaded from the query matrix, key matrix, and value matrix, and attention calculation is performed on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0009] According to an attention calculation optimization method provided by the present invention, generating a bounding box list of attention matrices based on mask information from attention calculation includes:

[0010] Based on the mask information, at least one effective computation region in the attention matrix is ​​determined;

[0011] Based on the position of each effective computing region in the attention matrix, corresponding boundary parameters are generated for each effective computing region, and the boundary parameters are used to limit the spatial range of the effective computing region.

[0012] The bounding box list is generated based on the boundary parameters of each effective calculation region.

[0013] According to an attention calculation optimization method provided by the present invention, the step of loading target data blocks corresponding to each valid bounding box from a query matrix, a key matrix, and a value matrix based on each valid bounding box in the bounding box list, and performing attention calculation on the loaded target data blocks, includes:

[0014] Traverse each of the valid bounding boxes;

[0015] Based on the boundary parameters of the currently valid bounding boxes traversed, the corresponding query blocks, key blocks, and value blocks are loaded from the query matrix, the key matrix, and the value matrix;

[0016] Attention calculations are performed on the loaded query blocks, key blocks, and value blocks.

[0017] According to an attention calculation optimization method provided by the present invention, the boundary parameters include a starting x-coordinate, a starting y-coordinate, an ending x-coordinate, and an ending y-coordinate. The method of loading corresponding query blocks, key blocks, and value blocks from the query matrix, the key matrix, and the value matrix based on the boundary parameters of the currently valid bounding boxes includes:

[0018] Based on the starting and ending ordinates in the boundary parameters, the corresponding query blocks are loaded from the query matrix;

[0019] Based on the starting and ending x-coordinates in the boundary parameters, the corresponding key blocks and value blocks are loaded from the key matrix and the value matrix.

[0020] According to an attention calculation optimization method provided by the present invention, the bounding box list further includes at least one invalid bounding box, which is used to define the spatial range of the data block to be skipped in the attention calculation in the attention matrix.

[0021] The attention calculation optimization method provided by the present invention further includes:

[0022] Set the output corresponding to each invalid bounding box to zero.

[0023] The attention output corresponding to each valid bounding box is combined with the zero value corresponding to each invalid bounding box to obtain the attention result matrix.

[0024] According to an attention calculation optimization method provided by the present invention, the mask information includes at least one of causal mask, sliding window mask, and padding mask.

[0025] The present invention also provides an attention calculation optimization device, comprising:

[0026] A bounding box generation unit is used to generate a list of bounding boxes for the attention matrix based on the mask information of the attention calculation. The list of bounding boxes includes at least one valid bounding box, which is used to define the spatial range of the target data block to be subjected to attention calculation in the attention matrix.

[0027] The attention calculation unit is used to load the target data block corresponding to each valid bounding box from the query matrix, key matrix and value matrix based on each valid bounding box in the bounding box list, and perform attention calculation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the attention computation optimization method as described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attention computation optimization method as described above.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the attention computation optimization method as described above.

[0031] The attention computation optimization method, electronic device, storage medium, and program product provided by this invention accurately delineate the effective region where attention computation needs to be performed by dynamically generating at least one effective bounding box of the attention matrix based on mask information. This avoids the mechanical processing of all data blocks by traditional block-based attention, reducing the loading and storage of invalid data from the source. Subsequently, based on the generated effective bounding box, target data blocks are loaded only from the query matrix, key matrix, and value matrix, and attention computation is performed only on these loaded data blocks. This means that attention computation is not required for data blocks outside the effective bounding box, thereby avoiding invalid computation and waste of computing resources. This significantly improves the utilization rate of hardware resources and the overall performance of attention computation, especially when processing long sequence tasks containing a large number of mask regions, where its advantages are even more prominent. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the attention block calculation process provided by the present invention;

[0034] Figure 2 This is a schematic diagram of the attention matrix provided by the present invention. Figure 2 (a) Figure 2 (b) Figure 2 (c) The attention matrices for different sliding window mask scenarios are shown respectively;

[0035] Figure 3 This is a schematic diagram of the structure of the artificial intelligence chip provided by the present invention;

[0036] Figure 4 This is a flowchart illustrating the attention calculation optimization method provided by the present invention;

[0037] Figure 5 This is a schematic diagram of the bounding box provided by the present invention. Figure 5 (a) Figure 5 (b) Figure 5 (c) The bounding boxes of the attention matrix are shown for different sliding window mask scenarios;

[0038] Figure 6 This is a flowchart illustrating the attention block calculation method based on bounding boxes provided by the present invention.

[0039] Figure 7This is a schematic diagram of the attention calculation optimization device provided by the present invention;

[0040] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] The Transformer model has become the cornerstone architecture for many artificial intelligence tasks, such as natural language processing and computer vision. One of its core components is the attention mechanism, which assigns weights to values ​​(V) by calculating the similarity between the query (Q) and the key (K), thereby enabling the model to dynamically focus on relevant parts of the input sequence.

[0043] In standard attention computation, a key operation is masking. This operation is typically performed after the original attention score matrix (obtained through the dot product of Q and K) is computed and before Softmax normalization. The core purpose of masking is to control the model's information flow, preventing it from "seeing" information it shouldn't be paying attention to. For example, in autoregressive language models, causal masks are used to mask future tokens (the smallest unit the model processes in text, which can be understood as a fragment of a character, word, or symbol), ensuring that predictions for the current position depend only on tokens at previous positions. As another example, in scenarios requiring local awareness or limited by computational resources, sliding window masks are used to restrict attention to key tokens within a fixed-size window before and after each query token, thus compressing the attention scope from global to local.

[0044] The masking operation described above is implemented by adding a mask matrix (e.g., an upper triangular matrix or a Boolean matrix) to the attention score matrix. In the mask matrix, the positions to be masked are assigned an extremely large negative value (theoretically negative infinity, i.e., -inf). When this matrix containing extremely negative values ​​is processed by the subsequent Softmax function, the exponent of the masked positions will approach zero, thus rendering them ineffective in the final weighted summation and achieving precise masking.

[0045] To address the increasing length of sequences and improve computational efficiency, block attention methods, such as FlashAttention, have been proposed. These methods divide a large attention matrix into several smaller data blocks and utilize the memory hierarchy of AI chips (such as caches) to reduce the number of accesses to high-latency memory, thereby significantly accelerating computation and reducing memory usage.

[0046] Figure 1 This is a schematic diagram of the attention block calculation process provided by the present invention, as follows: Figure 1 As shown, in the Attention calculation process, the input sequence (such as a text sequence) is first converted into an embedding representation. That is, each token in the text sequence is mapped to a fixed-dimensional vector through an embedding layer, and then this vector is converted into Q, K, and V matrices through a linear transformation. Next, the Q, K, and V matrices are segmented to obtain their respective blocks. Then, all Q blocks are traversed, and the currently traversed Q block (denoted as Qi) is loaded. For each Qi, all K and V blocks are traversed, and the currently traversed K and V blocks (denoted as Ki and Vi) are loaded for Attention calculation. During the calculation, matrix multiplication is first performed on Qi and Ki to obtain the first result (a sub-matrix in the attention score matrix). Then, a Softmax operation is performed on this first result to obtain the second result (a sub-matrix in the attention weight matrix). Finally, matrix multiplication is performed on the second result and Vi to obtain the current intermediate result. This process continues to traverse the next K and V block until all K and V blocks have been traversed, obtaining the calculation result corresponding to the current block Qi. Repeat the above steps until all Q-blocks have been traversed to obtain the final calculation result.

[0047] While block-based attention methods like FlashAttention improve computational efficiency, they exhibit a significant drawback when combined with masking mechanisms: their process for handling block computation is static and lacks adaptability. Specifically, when all elements within a block of the attention matrix (i.e., the attention score matrix) need to be masked (e.g., the entire block in the upper triangular region in a causal mask), that block is effectively a "fully masked block." Ideally, the most efficient approach for such "fully masked blocks" is to skip all computations. However, current methods lack the ability to dynamically identify and skip these blocks; they still mechanically execute the complete computation process. The following example illustrates this using the attention computation scenario of a sliding window mask.

[0048] Sliding window masking is an optimization strategy in attention mechanisms, aiming to improve efficiency and performance by limiting the computational scope of attention, especially effective when processing long sequences. Its core principle is to divide the input sequence (such as text sequences or image patches) into multiple non-overlapping local windows, and compute attention independently within each window; that is, each token only performs attention computation with tokens within a fixed-size window nearby. The computational complexity of attention within a window is reduced from O(N²) to O(NW), where N is the sequence length and W is the window size, which is typically less than N. Here, O(N²) indicates that the algorithm's running time is proportional to the square of the sequence length N, and O(NW) indicates that the algorithm's running time is proportional to the product of the sequence length N and the window size W.

[0049] For example, when processing long sequences, global attention in long texts is computationally expensive and may introduce noise. Using a sliding window mask restricts attention to each token only within its neighboring window, reducing computational complexity while preserving key long-distance dependencies. As another example, in multimodal scenarios (such as image-text matching), sequences of different modalities need to be processed. Applying sliding window masks to images and text separately enables local modeling of each, preventing information confusion in the early stages. It should be understood that by adopting the sliding window mask strategy, the huge overhead and noise from global computation can be avoided, significantly improving the efficiency and effectiveness of long sequence modeling and multimodal learning.

[0050] Figure 2 This is a schematic diagram of the attention matrix provided by the present invention, as shown below. Figure 2 As shown, Figure 2 (a) Figure 2 (b) Figure 2(c) illustrates the attention matrices under several different sliding window masking scenarios. The entire attention matrix can be visualized as a large grid divided into smaller matrix blocks for efficient computation on AI chips. The numbers 0-15 in the figure represent these matrix blocks. It should be understood that each row element in the attention matrix is ​​the dot product (i.e., relevance score) of a Q vector and all K vectors, and each column element is the dot product of a K vector and all Q vectors. Therefore, in the attention matrix, the vertical axis (row direction) corresponds to the sequence position of Q (i.e., the i-th Q token), and the horizontal axis (column direction) corresponds to the sequence position of K (i.e., the j-th K token). When the attention matrix is ​​divided into multiple matrix blocks, the vertical axis corresponds to the Q blocks along the sequence length dimension, and the horizontal axis corresponds to the K blocks along the sequence length dimension. Each matrix block numbered 0-15 in the figure represents the pairwise attention computation between all tokens in the corresponding Q block and K block. A white matrix block in each attention matrix indicates that the block will be masked, meaning the final attention weight of the elements within that block will approach 0; hence, it is also called a zero-block. The attention between the Q token and K token within these matrix blocks is completely masked, and theoretically, no attention calculation is needed. Blue matrix blocks indicate that attention calculation is required between the Q token and K token within these blocks, and the final calculated weight values ​​are retained; therefore, they are also called non-zero blocks.

[0051] The following is based on Figure 2 (a) will be used as an example to illustrate the concept. Figure 2 The principles of (b) and 2(c) are similar and will not be elaborated here. Suppose the model needs to process a text sequence of length 2048 (denoted as A). During attention calculation, it can be divided into two non-overlapping local windows, each with a sequence length of 1024. That is, the entire sequence A is divided into two sub-sequences A1 and A2, and attention is calculated independently for each sub-sequence within its own window. When using the FlashAttention block attention method, the entire sequence is divided into four blocks: block one and block two belong to sub-sequence A1, and block three and block four belong to sub-sequence A2. Each block has a sequence length of 512. (Corresponding to...) Figure 2 In (a), it is assumed that the query blocks Q0, Q1, Q2 and Q3 are respectively from top to bottom along the vertical axis, and the key blocks K0, K1, K2 and K3 are respectively from left to right along the horizontal axis. Q0 and K0 correspond to the first sequence block after the sequence is split, Q1 and K1 correspond to the second sequence block after the sequence is split, and so on.

[0052] like Figure 2As shown in (a), the blue matrix blocks in the upper left corner (i.e., matrix block 10, matrix block 21, matrix block 34, and matrix block 45) correspond to the effective attention calculation between the query blocks (i.e., Q0 and Q1) and key blocks (i.e., K0 and K1) of subsequence A1; the blue matrix blocks in the lower right corner (i.e., matrix block 510, matrix block 611, matrix block 714, and matrix block 815) correspond to the effective attention calculation between the query blocks (i.e., Q2 and Q3) and key blocks (i.e., K2 and K3) of subsequence A2. Since attention calculation is performed independently only within the window of each subsequence, the white matrix blocks in the upper right and lower left corners, representing cases where the query block comes from one subsequence and the key block comes from another subsequence, need to be masked to prevent attention calculation across subsequences.

[0053] In traditional block attention calculation methods, for Figure 2 Each matrix block in the attention matrix shown in (a) mechanically executes the complete computation process because it lacks the ability to dynamically identify and skip "full mask blocks" (i.e., the white matrix blocks shown in the figure). Specifically, this includes:

[0054] 1. Load Data: Load the Q-block and K-block data corresponding to the "full mask block" from memory into the cache.

[0055] 2. Invalid computation: Perform matrix multiplication on the loaded Q and K blocks, even if the subsequent results are completely masked.

[0056] 3. Redundant operation: The invalid calculation result is combined with the mask information to obtain a score matrix block in which all elements are extremely negative (-inf), and then the Softmax operation is performed on it. Performing exponentiation and normalization operations on a matrix full of -inf is a pure waste of computational resources.

[0057] This redundant computation is particularly problematic when dealing with long input sequences. As the sequence length increases, the size of the attention matrix grows quadratically, and the number of matrix blocks that are segmented also increases, with "full-mask blocks" potentially making up a significant proportion. Repeatedly performing ineffective computations on each such matrix block leads to severe waste of computational power and inefficient use of hardware resources, thereby limiting the overall performance of model training and inference.

[0058] To address the aforementioned problems, this invention provides an optimized attention computation method, aiming to solve the computational waste caused by the static computation process in traditional block-based attention methods. This method first generates one or more bounding boxes to define the effective computation range based on the mask information from attention computation. This step transforms the previously passive masking operation into an active, pre-emptive computational planning, creating a precise "execution map" for subsequent computations. Then, based on the generated bounding box list, only target data blocks corresponding to the effective bounding boxes are loaded from the query matrix, key matrix, and value matrix, and attention computation is performed only on these loaded data blocks. Through this process, for all "full-masked blocks" not within the effective bounding boxes, the corresponding data loading, matrix multiplication, and even Softmax normalization computation steps are fundamentally skipped. Therefore, this invention transforms the previously fixed and non-adaptive computation mode into a dynamic, on-demand, and efficient mode, thereby accurately avoiding redundant computations and significantly improving the utilization of hardware resources and the overall performance of attention computation, especially when processing long sequence tasks containing a large number of masked regions.

[0059] Figure 3 This is a schematic diagram of the structure of the artificial intelligence chip provided by the present invention, as shown below. Figure 3 As shown, the execution subject of the attention calculation optimization method provided by the present invention can be an artificial intelligence chip. The artificial intelligence chip 300 includes multiple streaming processor clusters (SPCs) 301 and video memory 302. Each streaming processor cluster 301 includes multiple computing units 303, and each computing unit 303 includes at least an on-chip cache 304 and a register 305.

[0060] Video memory 302 can be off-chip HBM (High Bandwidth Memory) or other types of memory. On-chip cache 304 is a temporary memory with a smaller capacity than video memory 302, but a faster data exchange speed. On-chip cache 304 can be a Gemm Main Buffer (GMB). Compared to on-chip cache 304, register 305 has a smaller capacity but a faster data exchange speed. Register 305 can specifically be a TLR (Thread Local Register).

[0061] It should be noted that the artificial intelligence chip 300 in this invention may include other structures besides those described above, and this invention does not specifically limit these. The artificial intelligence chip 300 may be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), etc.

[0062] based on Figure 3 The diagram shows the architecture of an artificial intelligence chip. This invention provides an attention calculation optimization method, applicable to various scenarios such as text processing, image processing, and speech processing. In different application scenarios, the data used to perform attention calculations (i.e., the target data block) has different physical meanings. For example, in a text processing scenario, the target data block can be text data used in tasks such as text generation and text recognition. Similarly, in an image processing scenario, the target data block can be image data used in tasks such as image preprocessing, image segmentation, and object detection. And in a speech processing scenario, the target data block can be speech data used in tasks such as speech recognition and speech synthesis. The specific process of the attention calculation optimization method provided by this invention is described below.

[0063] Figure 4 This is a flowchart illustrating the attention calculation optimization method provided by the present invention, as shown below. Figure 4 As shown, the method includes:

[0064] Step 410: Based on the mask information of attention calculation, generate a bounding box list of attention matrix. The bounding box list includes at least one valid bounding box, which is used to define the spatial range of the target data block to be performed on attention calculation in the attention matrix.

[0065] It's important to note that the core of this step lies in pre-analyzing and planning the computational task to identify the regions that truly require computation. The masking information for attention computation is a data structure used to guide the attention calculation process. It is typically represented as a two-dimensional Boolean matrix or numerical matrix of the same size as the attention matrix (i.e., the attention score matrix). Its function is to identify which positions in the attention computation are valid and need to be calculated, and which positions are invalid and need to be ignored or masked.

[0066] For example, in the causal masking scenario, the mask information for attention computation can be an upper triangular matrix used to prevent the current position from paying attention to information about future positions; in the sliding window masking scenario, the mask information is used to block out information outside the window. In this mask matrix, the value 0 (or False) is usually used to represent valid positions, and the value 1 (or True) is used to represent invalid positions that need to be blocked.

[0067] Specifically, by analyzing the mask information of attention computation, the effective regions in the attention matrix that need to be actually computed can be dynamically identified and marked, thereby generating a list of bounding boxes containing these regions. Here, the bounding box list refers to a set consisting of one or more effective bounding boxes. An effective bounding box (i.e., the bounding box of the effective computation region) is a data structure used to describe the computation region. It precisely defines the rectangular spatial range of one or more consecutive target data blocks containing at least one effective location in the attention matrix grid. It can be represented by a set of coordinates, such as (h_start, w_start, h_end, w_end), where the h dimension corresponds to the sequence length of the Q matrix, the w dimension corresponds to the sequence length of the K matrix, and start and end represent the starting and ending block coordinates (or indices) of the rectangular range in the corresponding dimensions, respectively.

[0068] Understandably, in block attention methods (such as FlashAttention), to utilize caching, large Q, K, V matrices are not processed all at once, but are instead divided into multiple smaller data blocks. The target data block refers to these sub-blocks (also called chunks) of the Q, K, V matrices, which serve as the basic computational units after being segmented. The attention score matrix is ​​thus correspondingly divided into a grid composed of calculations performed on these sub-blocks.

[0069] Specifically, based on the mask information for attention calculation, a list of bounding boxes for the attention matrix is ​​generated. This can be achieved through the following steps: First, obtain the complete mask matrix required for this attention calculation based on the mask information, treating this mask matrix as a grid corresponding to the attention matrix partitioning method. Then, traverse each cell in the grid (each cell corresponds to the computational region of a data block), checking whether the mask information covered by the cell contains at least one valid value (e.g., value 0). If it does, mark the cell as a valid block; otherwise, if all values ​​in a cell are invalid (e.g., value 1), mark it as an invalid block or a fully masked block. Finally, scan and merge all cells marked as valid blocks. A greedy algorithm can be used, starting from an unprocessed valid block and expanding outwards (e.g., to the right and below) to form a rectangular region that is as large as possible and contains only valid blocks. The coordinates of the four corners of this rectangular region constitute a valid bounding box. Add this generated bounding box to the bounding box list, and mark all valid blocks covered by this bounding box as processed. Repeat this scanning and merging process until all valid blocks are covered by a bounding box. Finally, you will get a list of bounding boxes containing all regions to be calculated.

[0070] Step 420: Based on each valid bounding box in the bounding box list, load the target data block corresponding to each valid bounding box from the query matrix, key matrix, and value matrix, and perform attention calculation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0071] Specifically, after generating the bounding box list, the computation process no longer blindly traverses all data blocks in the entire attention matrix. Instead, it iterates through each valid bounding box in the list. For each valid bounding box being processed, the system precisely loads its corresponding target data block from video memory (such as HBM) into a cache (such as an on-chip cache) based on its coordinate range. Data whose coordinates are outside the valid bounding box range will not be loaded. This avoids loading and transferring invalid data blocks, and naturally, there is no need to compute these data blocks subsequently. This completely skips the invalid computation process for the full mask block, thus avoiding waste of computing resources and improving the utilization of hardware resources.

[0072] Here, for each valid bounding box (assuming a coordinate range of (h_start, w_start, h_end, w_end)), the corresponding target data block refers to all Q blocks in the query matrix Q with block indices hi satisfying h_start ≤ hi < h_end, and all K and V blocks in the key matrix K and value matrix V with block indices wi satisfying w_start ≤ wi < w_end. All Q, K, and V blocks with coordinates outside this range will not be loaded in this loop, thus avoiding data transmission of invalid blocks.

[0073] For target data blocks already loaded into the cache, standard attention calculations can be performed. Specifically, in the block attention implementation, the attention calculation process for each Q block is as follows: A Q block Qi is retrieved from the cache (where the block index hi of Qi is in the range h_start to h_end), and all corresponding K blocks Ki are traversed (where the block index wi of Ki is in the range w_start to w_end). Then, attention scores are calculated based on Qi and Ki, generating a submatrix of the score matrix. The Softmax function is applied to the submatrix of the score matrix to obtain a submatrix of the attention weight matrix. Finally, a weighted sum is calculated based on the submatrix of the weight matrix and Vi (where Vi is the V block corresponding to Ki), and this weighted value is accumulated and added to the final output corresponding to the Qi block. After completing the above inner loop (i.e., traversing all K and V blocks corresponding to the Qi block), the output obtained is the final output of that Qi block.

[0074] After traversing all Q-blocks and K / V-blocks defined by a valid bounding box, the resulting set of outputs Oi constitutes the attention output corresponding to that valid bounding box. These output blocks logically correspond to the regions defined by the bounding box in the final complete attention output matrix. Since the valid bounding box has already defined the effective region to be calculated, there is no need to consider mask zeroing operations during the calculation process, avoiding redundant calculations in the full matrix operation. For all regions not within any valid bounding box, their output is naturally zero, requiring no calculation or storage. It should be understood that this embodiment of the invention only performs data loading and calculation on the effective regions defined by the bounding box list, completely skipping invalid operations on the full mask blocks, thereby achieving a significant optimization of attention calculation.

[0075] The method provided in this invention accurately delineates the effective regions requiring attention computation by dynamically generating at least one effective bounding box of the attention matrix based on mask information. This avoids the mechanical processing of all data blocks by traditional block-based attention, reducing the loading and storage of invalid data from the source. Subsequently, based on the generated effective bounding boxes, target data blocks are loaded only from the query matrix, key matrix, and value matrix, and attention computation is performed only on these loaded data blocks. This means that attention computation is not required for data blocks outside the effective bounding boxes, thereby avoiding invalid computation and wasting computing resources. This significantly improves the utilization rate of hardware resources and the overall performance of attention computation, especially when processing long sequence tasks containing a large number of masked regions, where its advantages are even more prominent.

[0076] Based on any of the above embodiments, step 410 specifically includes:

[0077] Step 411: Based on the mask information, determine at least one valid computation region in the attention matrix.

[0078] It's important to note that in block-based attention computation, the entire attention matrix is ​​viewed as a grid composed of multiple data blocks. A valid computation region refers to a set of one or more spatially adjacent data blocks within this grid that are not themselves "full-mask blocks." In other words, a data block is valid as long as there is at least one point that needs to be computed within its corresponding region, and a valid computation region is a connected component formed by these valid blocks.

[0079] Specifically, when determining one or more valid computational regions in the attention matrix based on the mask information, the input mask information (a large Boolean or numerical matrix) can first be logically gridded according to the data block size (or block size) of the sequence. Then, each cell in the grid is traversed, and for each cell, the mask information it covers is checked. If all mask values ​​in the cell represent "masked" (e.g., all 1s), the cell is marked as an invalid block; otherwise, if at least one non-masked value exists (e.g., value 0), the cell is marked as a valid block.

[0080] Cluster all cells marked as valid blocks to form one or more valid computation regions. One feasible implementation is to initialize an empty list of regions, then, starting with the first valid block in the grid, find all directly or indirectly adjacent valid blocks, group these blocks together into a single valid computation region, and mark all blocks in this region as processed. Continue scanning the grid to find the next unprocessed valid block and repeat the search process to form new valid computation regions. This process continues until all valid blocks in the grid have been grouped into a single valid computation region. After these steps, several independent regions requiring computation, i.e., valid computation regions, can be identified from the original mask information.

[0081] Step 412: Based on the position of each effective computing region in the attention matrix, generate corresponding boundary parameters for each effective computing region. The boundary parameters are used to limit the spatial range of the effective computing region.

[0082] Specifically, after identifying one or more valid computation regions in the attention matrix, corresponding boundary parameters can be generated for each valid computation region. Here, the boundary parameters are a set of coordinate values ​​used to accurately describe the rectangular box occupied by a valid computation region. In this embodiment of the invention, this set of parameters can be a quadruple (h_start, w_start, h_end, w_end). Here, h_start and w_start represent the starting block index of the rectangular box in the row (corresponding to the Q matrix sequence) and column (corresponding to the K matrix sequence) directions, respectively; h_end and w_end represent the ending block index of the rectangular box in the row and column directions, respectively.

[0083] Specifically, for each valid computation region obtained in step 411, the boundary parameters are generated as follows: Traverse all valid blocks in the region set and record the row index h_i and column index w_i of each block. Then, based on the row and column indices of each block, the four boundaries of the region can be determined, where h_start is the minimum value of all h_i in the set, w_start is the minimum value of all w_i in the set, h_end is the maximum value of all h_i in the set, and w_end is the maximum value of all w_i in the set. By performing the above operation on each valid computation region, a unique and accurate set of boundary parameters can be generated for each region.

[0084] Step 413: Generate the bounding box list based on the boundary parameters of each effective calculation region.

[0085] Specifically, after generating the boundary parameters for each effective computation region, the bounding box corresponding to each effective computation region can be determined based on these boundary parameters. These bounding boxes are then integrated to form a bounding box list. Specifically, first, an empty data container, such as a list or array, can be initialized; this container serves as the initial bounding box list. Then, all boundary parameter sets generated for each effective computation region are traversed. For each set of boundary parameters, it is encapsulated into a "bounding box" object or structure. This newly created "bounding box" object is then appended to the bounding box list. Once all boundary parameters have been processed and encapsulated, the bounding box list is complete. It contains the precise range of all effective computation tasks required for this attention computation and can be directly passed to step 420 to guide subsequent data loading and computation.

[0086] Figure 5 This is a schematic diagram of the bounding box provided by the present invention, as shown below. Figure 5 As shown, Figure 5 (a) Figure 5 (b) Figure 5 (c) The bounding boxes of the attention matrices under several different sliding window mask scenarios are shown. The region containing the blue matrix block in each attention matrix is ​​the effective computation region, and the bounding box corresponding to this region is identified by boundary parameters. For each effective computation region (or effective bounding box) shown in the figure, the red dot marks the upper left corner (including the starting ordinate and starting abscissa) of each effective bounding box, and the green dot marks the lower right corner (including the ending ordinate and ending abscissa). By identifying the effective computation regions and generating corresponding boundary parameters for each effective computation region, the corresponding bounding boxes can be determined based on these boundary parameters, thus forming a bounding box list. After generating the bounding box list, each effective bounding box can be traversed, thereby performing attention calculations only on the data blocks within the effective bounding boxes, significantly reducing a large amount of redundant computation and avoiding waste of computing resources.

[0087] Based on any of the above embodiments, step 420 specifically includes:

[0088] Step 421: Traverse each of the valid bounding boxes;

[0089] Step 422: Based on the boundary parameters of the currently valid bounding boxes traversed, load the corresponding query blocks, key blocks, and value blocks from the query matrix, the key matrix, and the value matrix.

[0090] Specifically, after generating the bounding box list, each valid bounding box in the list can be traversed using a loop. In each iteration of the loop, the single bounding box object retrieved from the list is the current valid bounding box. This current valid bounding box encapsulates its corresponding boundary parameters, namely the quadruple (h_start, w_start, h_end, w_end) used to accurately describe its spatial extent as described in the above embodiment.

[0091] For each currently valid bounding box encountered, the corresponding Q, K, and V blocks (i.e., target data blocks) can be loaded based on their boundary parameters. Specifically, the system will perform a precise, on-demand data loading operation based on these boundary parameters. This operation loads the following data from video memory (such as HBM) to cache (such as on-chip cache): loading query blocks Qi in query matrix Q where all block indices hi satisfy the condition h_start ≤ hi < h_end; loading key blocks Ki in key matrix K where all block indices wi satisfy the condition w_start ≤ wi < w_end; and loading value blocks Vi in value matrix V where all column block indices wi satisfy the condition w_start ≤ wi < w_end. It should be understood that this selective loading process ensures that only data necessary for effective computation is transferred to the cache, while all data blocks outside the currently valid bounding box (i.e., data corresponding to "full mask blocks") are not accessed at all, thus avoiding bandwidth waste at the source.

[0092] Step 423: Perform attention calculation on the loaded query block, key block, and value block.

[0093] It should be noted that once the query, key, and value chunks related to the current valid bounding box are successfully loaded into the cache, standard attention calculations can be performed on this data. This calculation process itself follows the basic principles of chunked attention, but its scope is strictly limited to the currently loaded data chunk.

[0094] Specifically, the process of performing attention calculations on the loaded blocks can be viewed as a nested loop structure. The outer loop iterates through the currently loaded query block Qi (where the range of the block index hi of Qi is limited by h_start and h_end), while the inner loop iterates through all loaded key / value blocks Ki / Vi (where the range of the block index wi of Ki / Vi is limited by w_start and w_end). For each query block Qi, the calculation process is as follows:

[0095] First, an output accumulator matrix Oi (usually initialized to zero) is initialized for the Qi block to store the final result. Then, the inner loop calculation is performed, iterating through all corresponding Ki and Vi blocks. In each inner loop, matrix multiplication between Qi and Ki is performed to obtain the attention numerator matrix. The Softmax function is applied to this numerator matrix to obtain the attention weight submatrix. The obtained weight submatrix is ​​used to perform a weighted summation on the corresponding value block Vi, and the weighted result is accumulated into the output accumulator matrix Oi.

[0096] After the inner loop iterates through all corresponding Ki / Vi blocks within the current valid bounding box, the content stored in the output accumulator matrix Oi is the final attention calculation result for the query block Qi. After the outer loop iterates through all corresponding Qi blocks within the current valid bounding box, the attention output corresponding to that valid bounding box is obtained. When the outermost loop iterates through all valid bounding boxes in the bounding box list, the entire attention calculation process is complete, and the final result is a complete attention output matrix logically combined from the attention outputs of each valid calculation region.

[0097] Based on any of the above embodiments, the boundary parameters include the starting x-coordinate, the starting y-coordinate, the ending x-coordinate, and the ending y-coordinate. Accordingly, step 422 specifically includes:

[0098] Step 4221: Based on the starting and ending ordinates in the boundary parameters, load the corresponding query blocks from the query matrix;

[0099] Step 4222: Based on the starting and ending x-coordinates in the boundary parameters, load the corresponding key blocks and value blocks from the key matrix and the value matrix.

[0100] It should be noted that in attention calculation, the final score matrix is ​​a two-dimensional structure, with its rows determined by the sequence elements of the query matrix Q and its columns determined by the sequence elements of the key matrix K. To intuitively describe the position of the bounding box on this two-dimensional plane, this embodiment of the invention introduces the concepts of ordinate and abscissa. The ordinate is associated with the rows of the attention matrix, and its index range corresponds to the sequence dimension of the query matrix Q; the abscissa is associated with the columns of the attention matrix, and its index range corresponds to the sequence dimension of the key matrix K (and the value matrix V).

[0101] Therefore, the boundary parameter quadruple (h_start, w_start, h_end, w_end) described in the above embodiments can be understood more specifically as follows: the starting ordinate (i.e., h_start) represents the starting row block index of the region covered by the bounding box; the ending ordinate (i.e., h_end) represents the ending row block index of the region covered by the bounding box; the starting abscissa (i.e., w_start) represents the starting column block index of the region covered by the bounding box; and the ending abscissa (i.e., w_end) represents the ending column block index of the region covered by the bounding box.

[0102] Specifically, since the ordinates correspond to the sequence dimensions of the Q matrix, the corresponding query blocks can be loaded from the Q matrix based on the start and end ordinates in the boundary parameters. Specifically, the calculation of a specific row in the attention matrix depends entirely on the data of the corresponding row in the query matrix Q. Therefore, the loading operation is as follows: The system obtains the start and end ordinates h_start and h_end of the current valid bounding box. Then, the system iterates through all row block indices hi that satisfy the condition h_start ≤ hi < h_end. For each index hi that satisfies the condition, the system locates and loads the hi-th query block (Qi) from the query matrix Q in video memory (such as HBM) into the cache.

[0103] Similarly, since the x-coordinates correspond to the sequence dimensions of the K / V matrix, the corresponding key and value blocks can be loaded from the K / V matrix based on the start and end x-coordinates in the boundary parameters. Specifically, the calculation of a specific column in the attention matrix depends on the data in the corresponding row of the key matrix K, while the final output weighting requires the data in the corresponding row of the value matrix V (the sequence dimensions and block methods of the K and V matrices are consistent). Therefore, the loading operation is as follows: the system obtains the start x-coordinate w_start and end x-coordinate w_end of the current valid bounding box, and then the system traverses all column block indices wi that satisfy the condition w_start≤wi<w_end. For each index wi that satisfies the condition, the system simultaneously locates and loads the wi-th key block Ki and the wi-th value block Vi from video memory (such as HBM) into the cache.

[0104] Based on any of the above embodiments, the bounding box list further includes at least one invalid bounding box, which is used to define the spatial range of the data block in the attention matrix to be skipped in the attention calculation.

[0105] Specifically, for matrix blocks identified as fully masked in the attention matrix, although the attention calculation for such matrix blocks will be skipped directly, the bounding box metadata (i.e., the corresponding boundary parameters) of these fully masked blocks can be retained. This allows the weight values ​​within the bounding boxes containing these fully masked blocks to be set to 0 when integrating the final complete attention output. The bounding boxes containing these fully masked blocks are called invalid bounding boxes, and therefore the bounding box list can also include one or more invalid bounding boxes.

[0106] Here, the invalid bounding box is the corresponding concept to the valid bounding box. It is used to define a continuous rectangular region in the attention matrix that consists entirely of fully masked blocks and does not require any attention calculation. Its determination process complements the generation of valid bounding boxes. Specifically, after the system meshes the masking information of the attention matrix and identifies all valid blocks, those cells that are not marked as valid blocks are naturally invalid blocks or fully masked blocks. The system can perform operations similar to steps 411 and 412 on these invalid blocks, that is, cluster spatially adjacent invalid blocks into regions and generate corresponding boundary parameters for these regions, thereby forming one or more invalid bounding boxes.

[0107] Understandably, although the system does not perform calculations on regions within invalid bounding boxes, the purpose of identifying and recording these regions is to completely reconstruct the final output matrix. Invalid bounding boxes provide necessary location metadata, precisely telling the system where zero values ​​should be filled in the final output matrix, thus enabling the final reconstruction stage to be performed efficiently and accurately.

[0108] Based on any of the above embodiments, the method further includes:

[0109] Set the output corresponding to each invalid bounding box to zero.

[0110] The attention output corresponding to each valid bounding box is combined with the zero value corresponding to each invalid bounding box to obtain the attention result matrix.

[0111] Specifically, to obtain the final complete attention result matrix, the output corresponding to invalid bounding boxes can be set to zero. This means that for each spatial region defined by an invalid bounding box, its corresponding value in the final output result is predetermined to be zero. This "zero value" is a logical result directly derived from the mask information (because the attention weight of a fully masked region must be zero), and does not require any actual matrix multiplication or Softmax operation. This is the core of the invention's computational efficiency savings.

[0112] Understandably, the attention result matrix refers to the final generated complete output matrix that has the same dimensions and structure as the output of the standard attention mechanism, and it can be directly input into the next layer of the Transformer model.

[0113] To obtain the final attention result matrix, the following steps can be taken: First, the system allocates a memory space to store the final attention result matrix based on the complete dimensions of this attention calculation (e.g., the output dimensions corresponding to the sequence lengths of Q and K). A crucial step is to initialize this memory space entirely to zero. Next, the system iterates through each valid bounding box in the list. For each valid bounding box, the system retrieves its corresponding attention output (a small block of the result matrix) that has already been calculated in step 423. Subsequently, the system writes or copies this block of the result matrix to the corresponding position in the allocated attention result matrix based on the boundary parameters of the valid bounding box (i.e., the start and end x and y coordinates). Once the attention outputs corresponding to all valid bounding boxes have been filled into the correct positions in the result matrix, the combination process is complete. Since the entire result matrix was initially initialized to zero, the regions defined by invalid bounding boxes naturally retain their zero values, requiring no additional operations.

[0114] The embodiments of the present invention efficiently combine the discrete effective output results obtained through optimized calculations with the invalid regions that have been skipped in the calculations, ultimately generating an attention result matrix that is completely consistent with the results of the prior art and can be used directly.

[0115] Based on any of the above embodiments, the mask information includes at least one of causal mask, sliding window mask, and padding mask.

[0116] It should be noted that the method provided in this embodiment of the invention can be applied to various computational patterns defined by masks, such as causal masks, sliding window masks, and padding masks. Among them, causal masks are mainly used in autoregressive models and are the foundation for tasks such as text generation. Their core function is to ensure that when the model predicts a certain position (token) in a sequence, it can only focus on the information before that position and cannot "see" the information in the future, thereby maintaining the causal dependency of the model.

[0117] In implementation, a causal mask is typically represented as a triangular matrix. For an attention score matrix of size N×N (where N is the sequence length), this mask marks all positions above its diagonal as invalid and needs to be masked. When the sequence length N is very large, this masked triangular region can cover a large number of data blocks, forming a huge invalid computation region that can be identified and encapsulated as an invalid bounding box by this invention. This invention significantly improves the training and inference efficiency of autoregressive models when dealing with long sequences by generating valid bounding boxes that define the valid lower triangular region and directly skipping the computation of the invalid upper triangular region.

[0118] Sliding window masking is a key technique in long sequence processing models, designed to address the quadratic increase in computational and memory usage that standard global attention mechanisms incur when processing extremely long sequences. It restricts global attention to a local, fixed-size window. For each position in the sequence, it is only allowed to focus on other positions within a certain range to its left and right (i.e., the window size). This invention can efficiently identify and encapsulate regions outside the window into several large invalid bounding boxes, while defining regions within the window as one or more valid bounding boxes. This avoids invalid computations on distant, irrelevant locations, making the computation of long sequence models more efficient and feasible.

[0119] Padding masks are essential for processing batch data, resolving the conflict between the demands of parallel computing and the varying lengths of input sequences. To leverage the parallel computing capabilities of GPUs and other hardware, multiple sequences of varying lengths are typically padded to the same length (usually the length of the longest sequence in the batch) before being input into the model as a single batch. The purpose of a padding mask is to inform the model which data is original and valid, and which are meaningless padding tokens added for alignment. It masks all attention calculations related to padding tokens. Specifically, if the m-th position in a sequence is a padding token, then in the attention matrix, the m-th row (indicating that the padding token will not attract any other tokens) and the m-th column (indicating that no other tokens will attract the padding token) will be completely marked as invalid.

[0120] Understandably, in masking scenarios, when the sequence lengths within a batch vary significantly, this creates large, rectangular invalid regions on the right and bottom of the attention matrix. These naturally occurring rectangular invalid regions correspond to the invalid bounding boxes in this invention. This invention can directly identify these regions as invalid bounding boxes and skip them entirely, thereby greatly reducing redundant computation when processing batches of variable-length sequences.

[0121] Based on any of the above embodiments, this invention provides a method for optimizing the computational efficiency of the attention mechanism in the Transformer architecture. This method optimizes block attention algorithms such as FlashAttention by introducing the concept of dynamic bounding boxes, and is particularly suitable for block computation of mask attention. Figure 6 This is a flowchart illustrating the attention block calculation method based on bounding boxes provided by the present invention, as shown below. Figure 6 As shown, the method includes the following steps:

[0122] Step S0: Pre-generate a list of bounding boxes, each bounding box being labeled with the block space range for which attention computation needs to be performed in the format (h_start, w_start, h_end, w_end).

[0123] Step S1: Read each bounding box in the bounding box list. For the current bounding box, parse its coordinate range and load the start coordinates (h_start, w_start) and end coordinates (h_end, w_end). Based on the loaded start and end coordinates, initialize the position parameters hi=h_start and wi=w_start of the current processing block, where hi is the block index in the Q sequence length dimension and wi is the block index in the K sequence length dimension.

[0124] Step S2: Load the Q-block data corresponding to the current bounding box into the cache using the Load Q(hi) instruction.

[0125] Step S3: Traverse all K and V blocks corresponding to the current bounding box. During each traversal, load the K and V blocks within the corresponding coordinate range using the Load K(wi) and Load V(wi) instructions respectively to execute the standard attention calculation process. The block index wi of the valid K and V blocks satisfies the condition w_start≤wi<w_end.

[0126] Step S4: Traverse all Q blocks corresponding to the current bounding box, and execute steps S2 and S3 above during each traversal.

[0127] Step S5: Continue to traverse the next bounding box in the bounding box list and repeat steps S1 to S4 to perform attention calculations on the matrix blocks (i.e., the blocks after the attention matrix is ​​divided) within the bounding box.

[0128] For example, with Figure 5Taking the attention matrix shown in (a) as an example, the bounding box list includes two valid bounding boxes, which are the bounding boxes corresponding to the regions where the blue matrix blocks are located in the upper left corner and the lower right corner respectively. Among them, for the valid bounding box in the upper left corner (including matrix block one 0, matrix block two 1, matrix block three 4, and matrix block four 5), its boundary parameters are h_start = 0, w_start = 0, h_end = 1, w_end = 1. The block index hi of the query block Qi within this bounding box satisfies h_start ≤ hi < h_end (i.e., 0 ≤ hi < 1), and the block indices wi of the key block ki and the value block Qi satisfy w_start ≤ wi < w_end (i.e., 0 ≤ wi < 1). The matrix blocks within this bounding box are traversed in a loop according to these conditions.

[0129] Specifically, first initialize the position parameters of the first matrix block (i.e., matrix block one 0) as hi = 0, wi = 0. Therefore, load the query block Q0, the key block K0, and the value block V0, and perform attention calculation based on these blocks. Thus, the output result corresponding to matrix block one 0 in the attention matrix can be obtained. Since the index wi of the current key block and value block is wi = 0 < w_end, for the current query block Q0, continue to traverse the next K and V blocks within this bounding box. At this time, wi is updated to wi = wi + chunk (where chunk represents the block size, and the value of chunk here is 1). The updated wi = 0 + 1 = 1. According to the updated block index wi, load the key block K1 and the value block V1, and perform attention calculation with the current query block Q0. Thus, the output result corresponding to matrix block two 1 in the attention matrix can be obtained.

[0130] Subsequently, since the index wi of the current key block and value block is wi = 1 (not satisfying wi < w_end), the calculation process for the query block Q0 has been completed. In this case, it is necessary to reset the index wi of the K and V blocks to wi = w_start to process the next query block within this bounding box. The index hi of the next query block is updated to hi = hi + chunk, that is, hi = 0 + 1 = 1. According to the updated index hi, load the query block Q1, and traverse all the K and V blocks within this bounding box (i.e., K0 / V0 and K1 / V1). Perform attention calculation based on the query block Q1, the key block K0, and the value block V0, and the output result corresponding to matrix block three 4 in the attention matrix can be obtained; similarly, perform attention calculation based on the query block Q1, the key block K1, and the value block V1, and the output result corresponding to matrix block four 5 in the attention matrix can be obtained. After completing the attention calculation for all matrix blocks within the current bounding box, the next bounding box can be traversed and processed until the attention calculation for all matrix blocks within all valid bounding boxes is completed.

[0131] The method provided in this invention, by generating a bounding box list, can directly skip the calculation of full mask blocks (i.e., matrix blocks), avoiding a large number of invalid matrix multiplications and Softmax operations, thereby significantly reducing the total computational load. Simultaneously, by skipping invalid blocks (i.e., invalid matrix blocks), it avoids loading the Q and K data corresponding to these matrix blocks from high-latency GPU memory to low-latency cache, reducing memory bandwidth usage. Furthermore, the longer the processed sequence, the larger the proportion of the masked region, and the more severe the waste in traditional methods. The advantages of this invention become more prominent as the sequence length increases. It moves masking information from a step in the computation process to metadata that determines the computation flow, thereby achieving fine-grained management and optimization of computing resources and hardware memory access.

[0132] The attention calculation optimization device provided by the present invention is described below. The attention calculation optimization device described below and the attention calculation optimization method described above can be referred to in correspondence.

[0133] Based on any of the above embodiments Figure 7 This is a schematic diagram of the attention calculation optimization device provided by the present invention, as shown below. Figure 7 As shown, the device includes:

[0134] The bounding box generation unit 710 is used to generate a bounding box list of the attention matrix based on the mask information of the attention calculation. The bounding box list includes at least one valid bounding box, which is used to define the spatial range of the target data block to be performed on the attention matrix.

[0135] The attention calculation unit 720 is used to load the target data block corresponding to each valid bounding box from the query matrix, key matrix and value matrix based on each valid bounding box in the bounding box list, and perform attention calculation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0136] The apparatus provided in this invention accurately delineates the effective region requiring attention computation by dynamically generating at least one effective bounding box of the attention matrix based on mask information. This avoids the mechanical processing of all data blocks by traditional block-based attention, reducing the loading and storage of invalid data from the source. Subsequently, based on the generated effective bounding boxes, target data blocks are loaded only from the query matrix, key matrix, and value matrix, and attention computation is performed only on these loaded data blocks. This means that attention computation is not required for data blocks outside the effective bounding boxes, thereby avoiding invalid computation and wasting computing resources. This significantly improves the utilization rate of hardware resources and the overall performance of attention computation, especially when processing long sequence tasks containing a large number of masked regions, where its advantages are even more prominent.

[0137] Based on any of the above embodiments, the bounding box generation unit 710 is specifically used for:

[0138] Based on the mask information, at least one effective computation region in the attention matrix is ​​determined;

[0139] Based on the position of each effective computing region in the attention matrix, corresponding boundary parameters are generated for each effective computing region, and the boundary parameters are used to limit the spatial range of the effective computing region.

[0140] The bounding box list is generated based on the boundary parameters of each effective calculation region.

[0141] Based on any of the above embodiments, the attention calculation unit 720 includes:

[0142] Traverse sub-units to traverse each valid bounding box;

[0143] The loading subunit is used to load the corresponding query block, key block and value block from the query matrix, the key matrix and the value matrix based on the boundary parameters of the currently valid bounding box traversed.

[0144] The computation subunit is used to perform attention computation on the loaded query block, key block, and value block.

[0145] Based on any of the above embodiments, the boundary parameters include a starting x-coordinate, a starting y-coordinate, an ending x-coordinate, and an ending y-coordinate. Accordingly, the loading subunit is specifically used for:

[0146] Based on the starting and ending ordinates in the boundary parameters, the corresponding query blocks are loaded from the query matrix;

[0147] Based on the starting and ending x-coordinates in the boundary parameters, the corresponding key blocks and value blocks are loaded from the key matrix and the value matrix.

[0148] Based on any of the above embodiments, the bounding box list further includes at least one invalid bounding box, which is used to define the spatial range of the data block in the attention matrix to be skipped in the attention calculation.

[0149] Based on any of the above embodiments, the device further includes a result integration unit, the result integration unit being used for:

[0150] Set the output corresponding to each invalid bounding box to zero.

[0151] The attention output corresponding to each valid bounding box is combined with the zero value corresponding to each invalid bounding box to obtain the attention result matrix.

[0152] Based on any of the above embodiments, the mask information includes at least one of causal mask, sliding window mask, and padding mask.

[0153] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an attention calculation optimization method. This method includes: generating a bounding box list of an attention matrix based on mask information for attention calculation, the bounding box list including at least one valid bounding box, the valid bounding box being used to define the spatial range of the target data block in the attention matrix to be subject to attention calculation; loading the target data block corresponding to each valid bounding box from a query matrix, a key matrix, and a value matrix based on each valid bounding box in the bounding box list, and performing attention calculation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0154] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the attention calculation optimization method provided by the above methods. The method includes: generating a bounding box list of an attention matrix based on the mask information of attention calculation, wherein the bounding box list includes at least one valid bounding box, which is used to define the spatial range of the target data block to be subjected to attention calculation in the attention matrix; loading the target data block corresponding to each valid bounding box from a query matrix, a key matrix, and a value matrix based on each valid bounding box in the bounding box list, and performing attention calculation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0156] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an attention computation optimization method provided by the above methods. The method includes: generating a bounding box list of an attention matrix based on mask information for attention computation, wherein the bounding box list includes at least one valid bounding box, the valid bounding box being used to define the spatial range of a target data block in the attention matrix to be subjected to attention computation; loading the target data block corresponding to each valid bounding box from a query matrix, a key matrix, and a value matrix based on each valid bounding box in the bounding box list, and performing attention computation on the loaded target data block to obtain the attention output corresponding to each valid bounding box.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An attention calculation optimization method, characterized in that, include: Based on the mask information of attention computation, a list of bounding boxes for attention matrix is ​​generated. The list of bounding boxes includes at least one valid bounding box, which is used to define the spatial range of the target data block to be performed on attention computation in the attention matrix. Based on each valid bounding box in the bounding box list, the target data block corresponding to each valid bounding box is loaded from the query matrix, key matrix and value matrix stored in the video memory into the cache, and attention calculation is performed on the loaded target data block to obtain the attention output corresponding to each valid bounding box. The list of bounding boxes for generating the attention matrix based on the mask information calculated by attention includes: Based on the mask information, at least one effective computation region in the attention matrix is ​​determined; Based on the position of each effective computing region in the attention matrix, corresponding boundary parameters are generated for each effective computing region, and the boundary parameters are used to limit the spatial range of the effective computing region. The bounding box list is generated based on the boundary parameters of each effective calculation region.

2. The attention calculation optimization method according to claim 1, characterized in that, The step of loading target data blocks corresponding to each valid bounding box from the query matrix, key matrix, and value matrix based on each valid bounding box in the bounding box list, and performing attention calculation on the loaded target data blocks, includes: Traverse each of the valid bounding boxes; Based on the boundary parameters of the currently valid bounding boxes traversed, the corresponding query blocks, key blocks, and value blocks are loaded from the query matrix, the key matrix, and the value matrix; Attention calculations are performed on the loaded query blocks, key blocks, and value blocks.

3. The attention calculation optimization method according to claim 2, characterized in that, The boundary parameters include a starting x-coordinate, a starting y-coordinate, an ending x-coordinate, and an ending y-coordinate. The loading of corresponding query blocks, key blocks, and value blocks from the query matrix, the key matrix, and the value matrix based on the boundary parameters of the currently valid bounding boxes includes: Based on the starting and ending ordinates in the boundary parameters, the corresponding query blocks are loaded from the query matrix; Based on the starting and ending x-coordinates in the boundary parameters, the corresponding key blocks and value blocks are loaded from the key matrix and the value matrix.

4. The attention calculation optimization method according to claim 1, characterized in that, The list of bounding boxes also includes at least one invalid bounding box, which is used to define the spatial range of the data block in the attention matrix that needs to skip attention calculation.

5. The attention calculation optimization method according to claim 4, characterized in that, Also includes: Set the output corresponding to each invalid bounding box to zero. The attention output corresponding to each valid bounding box is combined with the zero value corresponding to each invalid bounding box to obtain the attention result matrix.

6. The attention calculation optimization method according to any one of claims 1 to 5, characterized in that, The mask information includes at least one of causal mask, sliding window mask, and padding mask.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the attention calculation optimization method as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the attention calculation optimization method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the attention calculation optimization method as described in any one of claims 1 to 6.

Images