An attention acceleration method and device based on a diffusion model, equipment and medium

Abstract

This invention discloses an attention acceleration method, apparatus, device, and medium based on a diffusion model. The method includes: acquiring the input feature set, current time step, and local window parameters corresponding to a target multi-head attention module in a target diffusion model; reshaping the shape of each input feature in the input feature set based on a preset tensor format to obtain a three-dimensional feature set; determining a feature recombination strategy corresponding to the three-dimensional feature set based on the current time step, and recombinating the three-dimensional feature set based on the feature recombination strategy to obtain a target feature result set corresponding to the target multi-head attention module; and performing local window attention calculation on the target feature result set based on the local window parameters and the feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module. Through the technical solution of this invention, efficient acceleration can be achieved while maintaining generation quality, improving the actual utilization rate of hardware.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an attention acceleration method, apparatus, device, and medium based on a diffusion model. Background Technology

[0002] With the gradual development of the computer field, diffusion models, which are the result of the deep integration of diffusion probability models and neural network architectures based on self-attention mechanisms, have become the mainstream model architecture in practical applications such as image generation and video generation. However, as users' demands for high-resolution image and long-duration video generation continue to increase, the computational cost of attention calculation in diffusion models increases quadratically with the amount of input data, resulting in a quadratic complexity problem and extremely high inference latency, posing a severe challenge to the computational efficiency of diffusion models.

[0003] In existing technologies, to alleviate the quadratic complexity problem of attention mechanisms, linear attention methods, sparse attention methods, or token merging methods are typically used to implement the attention computation process. Linear attention methods reduce the theoretical complexity of attention computation to linear by introducing kernel function approximation or low-rank projection. Sparse attention methods reduce computational cost by limiting the number of key-value pairs in each query interaction. Token merging methods reduce computational overhead by reducing the number of tokens involved in the attention computation.

[0004] However, existing linear attention, sparse attention, and token merging methods are highly architecture-dependent and cannot be plugged and played on pre-trained models, severely limiting their practical deployment value and reducing generation quality. Furthermore, existing sparse attention and token merging methods typically produce irregular memory access patterns, meaning the key-value vectors after token selection are scattered in memory. This contradicts the efficient computational requirements of modern graphics processors for contiguous, aligned data blocks, resulting in actual speedups far lower than theoretical values ​​and low hardware utilization.

[0005] Therefore, how to achieve end-to-end inference acceleration and improve the actual utilization of hardware while maintaining the quality of generation is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides an attention acceleration method, apparatus, device, and medium based on a diffusion model, which can solve the problems of high computational overhead, low hardware utilization, and easy degradation of generation quality in the attention calculation process of the diffusion model in the prior art.

[0007] According to one aspect of the present invention, an attention acceleration method based on a diffusion model is provided, comprising: Obtain the input feature set, current time step, and local window parameters corresponding to the multi-head attention module in the target diffusion model; Based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain the three-dimensional feature set corresponding to the input feature set. Based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and the three-dimensional feature set is recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; Based on the local window parameters and feature recombination strategy, local window attention calculation is performed on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module.

[0008] According to another aspect of the present invention, an attention acceleration device based on a diffusion model is provided, comprising: The data acquisition module is used to acquire the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. The shape reshaping module is used to reshape each input feature in the input feature set based on a preset tensor format to obtain a three-dimensional feature set corresponding to the input feature set. The feature recombination module is used to determine the feature recombination strategy corresponding to the three-dimensional feature set based on the current time step, and to recombine the features of the three-dimensional feature set based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; The result generation module is used to perform local window attention calculation on the target feature result set based on the local window parameters and feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the attention acceleration method based on the diffusion model as described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the diffusion-based attention acceleration method according to any embodiment of the present invention.

[0011] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the attention acceleration method based on a diffusion model as described in any embodiment of the present invention.

[0012] The technical solution of this invention obtains the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. Then, based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain a three-dimensional feature set corresponding to the input feature set. Further, based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and features are recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module. Finally, based on the local window parameters and the feature recombination strategy, local window attention is calculated on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module. Due to the spatial locality of images, a method combining reduced attention sequence length and local attention is used, significantly reducing computational load while ensuring the fidelity and detail consistency of the generated samples. By using continuous memory access and a well-organized computation process, unnecessary data movement and waiting time are reduced, allowing computational acceleration to truly translate into end-to-end inference latency reduction. It solves the problems of high computational overhead, low hardware utilization, and easy degradation of generation quality in the attention calculation process of existing technologies in diffusion models. It can achieve end-to-end actual inference acceleration while maintaining generation quality and improve the actual hardware utilization.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of an attention acceleration method based on a diffusion model according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of an attention acceleration method based on a diffusion model according to Embodiment 2 of the present invention; Figure 3This is a schematic diagram of input feature shape reshaping according to Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of token merging according to Embodiment 2 of the present invention; Figure 5 This is a flowchart of a token merging method provided in Embodiment 2 of the present invention; Figure 6 This is a flowchart of a local window attention calculation method according to Embodiment 2 of the present invention; Figure 7 This is a flowchart of an optional attention acceleration method based on a diffusion model provided according to Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of an attention acceleration device based on a diffusion model according to Embodiment 3 of the present invention; Figure 9 This is a schematic diagram of the structure of an electronic device that implements the attention acceleration method based on the diffusion model according to an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Example 1 Figure 1This is a flowchart of an attention acceleration method based on a diffusion model provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the attention calculation process of a diffusion model needs to be accelerated. This method can be executed by an attention acceleration device based on a diffusion model, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model.

[0019] In this context, a diffusion model can refer to a generative model that generates images or videos by progressively adding noise to data and then learning an inverse denoising process. A target diffusion model can refer to the specific diffusion model instance selected for acceleration and optimization. For example, the target diffusion model can be a pre-trained diffusion model or a diffusion model for a specific task. Typically, the target diffusion model is determined based on the actual application requirements. A multi-head attention module can refer to a module that splits input features into multiple independent attention heads, each head calculates attention separately, and then concatenates or merges the results. Typically, the input of a multi-head attention module contains three tensors: query (Q), key (K), and value (V). A target multi-head attention module can refer to a specific multi-head attention layer within the target diffusion model. Input features can refer to the original tensors input to the multi-head attention module. Typically, each tensor contains dimensional information such as sequence length and number of heads. The input feature set can refer to the set composed of the query, key, and value input features. For example, the input feature set can be represented as {Q,K,V}. A time step refers to a discrete step in the denoising process of the diffusion model. Generally, a smaller time step, such as one close to 0, indicates a clearer generated image. Conversely, a larger time step, such as one close to the total number of steps, indicates more noise. The current time step refers to the specific denoising step that the target diffusion model is currently performing in the inference process. Local window parameters refer to hyperparameters used to control the local window attention calculation. Optionally, local window parameters include a first window parameter and a second window parameter, where the first window parameter is smaller than the second window parameter. The first window parameter can refer to a hyperparameter used to divide non-overlapping query blocks. The second window parameter can refer to a hyperparameter used to determine the range of keys or values ​​that each query block can interact with.

[0020] S120. Based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain the three-dimensional feature set corresponding to the input feature set.

[0021] The preset tensor format refers to a predefined standard reshaping format. For example, the preset tensor format could be: [Batch×Heads,H,W,Dim]. Batch represents the batch size, i.e., the number of independent samples processed in parallel at one time. Examples include multiple images, multiple video frames, or text sequences. Heads represents the number of heads in a multi-head attention module. Typically, each head independently learns different attention patterns. H represents the height of the feature map. Typically, H is measured in pixels or the number of spatial locations. W represents the width of the feature map. Dim represents the feature dimension of each head, i.e., the length of the feature vector at each location within each head. Shape reshaping refers to the operation of converting the input features from their original shape to a preset three-dimensional format. Three-dimensional features refer to feature maps with three spatial or tensor dimensions obtained after shape reshaping. For example, if the original shape of the input features is [Batch,Heads,H×W,Dim], then the three-dimensional feature shape obtained after shape reshaping based on the preset tensor format could be [Batch×Heads,H,W,Dim]. The first dimension is H, the second dimension is W, and the third dimension is Dim. A three-dimensional feature set can refer to a set composed of the query, key, and value features. For example, a three-dimensional feature set can be represented as {Q', K', V'}.

[0022] S130. Based on the current time step, determine the feature recombination strategy corresponding to the three-dimensional feature set, and perform feature recombination on the three-dimensional feature set based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module.

[0023] Feature restructuring can refer to operations that adjust the spatial structure of features or change the number of tokens. Feature restructuring strategy can refer to the decision rules determined by the current time step to instruct the feature restructuring process. Target feature result can refer to the 3D feature map prepared for local window attention computation after feature restructuring. For example, the target feature result can include target query features, target key features, and target value features. Target query features can refer to the 3D feature map obtained after reshaping and restructuring the query features in the input feature set. Target key features can refer to the 3D feature map obtained after reshaping and restructuring the key features in the input feature set. Target value features can refer to the 3D feature map obtained after reshaping and restructuring the value features in the input feature set. The target feature result set can refer to the set of target feature results corresponding to the same input feature set.

[0024] S140. Based on the local window parameters and feature recombination strategy, perform local window attention calculation on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module.

[0025] In this context, local window attention computation can refer to approximate attention methods used to limit the scope of attention interactions. The attention result can refer to the feature map output by the local window attention computation. The target attention result can refer to the final output tensor of the target multi-head attention module obtained after output reshaping. Typically, the shape of the target attention result is consistent with the original shape corresponding to the input feature set.

[0026] The technical solution of this invention obtains the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. Then, based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain a three-dimensional feature set corresponding to the input feature set. Further, based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and features are recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module. Finally, based on the local window parameters and the feature recombination strategy, local window attention is calculated on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module. Due to the spatial locality of images, a method combining reduced attention sequence length and local attention is used, significantly reducing computational load while ensuring the fidelity and detail consistency of the generated samples. By using continuous memory access and a well-organized computation process, unnecessary data movement and waiting time are reduced, allowing computational acceleration to truly translate into end-to-end inference latency reduction. It solves the problems of high computational overhead, low hardware utilization, and easy degradation of generation quality in the attention calculation process of existing technologies in diffusion models. It can achieve end-to-end actual inference acceleration while maintaining generation quality and improve the actual hardware utilization.

[0027] Example 2 Figure 2 This is a flowchart of an attention acceleration method based on a diffusion model provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Specifically, it refines the step of "determining the feature recombination strategy corresponding to the three-dimensional feature set based on the current time step". Specifically, it may include: determining a preset time step threshold corresponding to the target multi-head attention module based on a preset time step length corresponding to the target diffusion model; performing a threshold judgment on the current time step based on the preset time step threshold, and determining a merging identifier corresponding to the three-dimensional feature set based on the threshold judgment result; if the merging identifier is true, then using a preset token merging strategy as the feature recombination strategy corresponding to the three-dimensional feature set; if the merging identifier is false, then using a preset data preservation strategy as the feature recombination strategy corresponding to the three-dimensional feature set. Figure 2 As shown, the method includes: S210. Obtain the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model.

[0028] S220. Based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain the three-dimensional feature set corresponding to the input feature set.

[0029] Figure 3 The diagram illustrates an input feature shape reshaping method according to an embodiment of the present invention. Specifically, taking the input feature as an input query tensor as an example, firstly, the input query tensor is reshaped from its shape [Batch,Heads,H×W,Dim] to [Batch,Heads,H,W,Dim], that is, the sequence length is restored to the height H and width W in two-dimensional space, where H×W is equal to the original sequence length; then, the batch size dimension and the head number dimension are merged to obtain a three-dimensional spatial feature map tensor with the shape [Batch×Heads,H,W,Dim]. The first dimension of this tensor represents the independent sample of each attention head, and the last three dimensions are the height, width, and feature vector dimension of each spatial location, respectively.

[0030] In this embodiment of the invention, identical reshaping operations are performed on the key tensors and value tensors in the input feature set to obtain the corresponding three-dimensional spatial feature maps. This reshaping process allows all subsequent spatial dimension-based operations to be uniformly applied to the merged batch dimension, achieving efficient parallel computation.

[0031] S230. Determine the preset time step threshold corresponding to the target multi-head attention module based on the preset time step corresponding to the target diffusion model.

[0032] The preset time step can refer to the total number of denoising steps corresponding to the pre-defined target diffusion model. Typically, the preset time step determines the precision of the denoising process. The preset time step threshold can refer to a value determined in advance based on the preset time step for evaluating the current time step. For example, the standard for generating the preset time step threshold can be: using one-quarter of the preset time step as the preset time step threshold. For instance, if the preset time step is 20, then the preset time step threshold can be 5.

[0033] S240. Based on the preset time step threshold, a threshold judgment is made on the current time step, and based on the threshold judgment result, the merging identifier corresponding to the three-dimensional feature set is determined.

[0034] The threshold judgment result can refer to the logical result obtained by comparing the current time step with a preset time step threshold. For example, the threshold judgment result can be either the current time step exceeding the preset time step threshold or the current time step being less than the preset time step threshold. Typically, if the current time step exceeds the preset time step threshold, it is considered to be in an early stage requiring token merging. Conversely, if the current time step is less than the preset time step threshold, it is considered to be in a later stage requiring preservation of the original resolution. The merging flag can refer to a Boolean flag corresponding to the threshold judgment result. For example, if the threshold judgment result is that the current time step exceeds the preset time step threshold, the merging flag can be true. Conversely, if the threshold judgment result is that the current time step is less than the preset time step threshold, the merging flag can be false.

[0035] S250. If the merge identifier is true, then the preset token merge strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set.

[0036] The preset token merging strategy refers to a pre-defined rule used to instruct the token merging operation. Typically, a preset token merging strategy can reduce the number of tokens participating in subsequent attention calculations, thereby reducing the computational load.

[0037] S260. If the merging identifier is false, then the preset data preservation strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set.

[0038] Among them, the preset data preservation strategy refers to a strategy that does not perform any token merging or resolution change, and directly uses the input 3D features as the output feature set. Typically, the preset data preservation strategy is used in the later stage of denoising.

[0039] Specifically, during the inference process of the target diffusion model, when accelerating a multi-head attention module at a certain layer, the input query, key-value features, and current time step of that target multi-head attention module are first obtained. After reshaping the input features into a three-dimensional spatial format, it is necessary to determine whether to perform token merging based on the current time step. That is, the preset time step threshold corresponding to the target multi-head attention module is determined according to the preset time step length corresponding to the target diffusion model. The current time step is compared with the preset time step threshold: if the current time step ≥ the preset time step threshold, it is considered that the current stage is still in the early stage of capturing the global structure, and the merging flag is set to true, and the preset token merging strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set; if the current time step < the preset time step threshold, it is considered that the later stage of refining details has been entered, the merging flag is set to false, and the preset data preservation strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set. Thus, by introducing a time step-aware merging flag, it is possible to dynamically adapt to the different needs of the diffusion model for global or local information at different denoising stages, providing an effective basis for subsequent operations.

[0040] S270. If the feature recombination strategy is a preset token merging strategy, then the target three-dimensional features in the three-dimensional feature set are token-merged based on the preset merging factor and the preset row index rule to obtain the target feature result set corresponding to the target multi-head attention module.

[0041] The preset merging factor can refer to the number of adjacent tokens participating in each merging operation in a pre-defined token merging operation. For example, in this embodiment of the invention, the preset merging factor can be fixed at 2, meaning that each time, tokens from two adjacent column indices are concatenated along the feature dimension to form a new token. This factor selection ensures that the width of the merged feature is halved, the computational load is halved, and it is easy to implement in hardware. The preset row index rule can refer to a pre-defined rule used to instruct different processing operations to be performed on different rows when performing token merging.

[0042] In an optional implementation, the step of token merging of the target 3D features in the 3D feature set based on a preset merging factor and a preset row indexing rule to obtain the target feature result set corresponding to the target multi-head attention module includes: Step a1: Based on the preset row index rules, the target three-dimensional features in the three-dimensional feature set are divided into even row groups and odd row groups according to the parity of the row index.

[0043] In this context, the target 3D feature can refer to the specific feature tensor selected for data processing within the 3D feature set. For example, the target 3D feature can be a query feature, key feature, or value feature. The row index can refer to the number of each row in the feature graph. Typically, the row index ranges from (0, H-1). Row indices are unique within the same feature graph. Parity refers to the remainder when the row index is divided by 2. Typically, rows with a remainder of 0 are even-numbered rows (indexes 0, 2, 4, etc.), and rows with a remainder of 1 are odd-numbered rows (indexes 1, 3, 5, etc.). An even-numbered row group can refer to the set of all rows with even-numbered row indices. An odd-numbered row group can refer to the set of all rows with odd-numbered row indices.

[0044] Step a2: Based on the preset merging factor, concatenate the adjacent target column indices in the even-numbered row group along the feature dimension to obtain the first token with a feature dimension twice that of the original dimension.

[0045] Here, the column index refers to the number of each column in the feature map. Typically, the column index ranges from (0, W-1). Within the same feature map, each row has a unique column index. The target column index refers to the index of the adjacent column pairs to be concatenated during the merge operation. For example, adjacent column pairs could be (0,1), (2,3), or (4,5). The first token refers to the new token generated by concatenating every two adjacent tokens along the feature dimension according to the target column index for each row in the even-numbered row group. Each first token has a feature dimension of 2×Dim, and its spatial position corresponds to the center of the original two columns.

[0046] Step a3: Mirror the odd-numbered row group to obtain a flipped row group, and concatenate the adjacent target column indices in the flipped row group along the feature dimension based on a preset merging factor to obtain the second token.

[0047] Mirroring can refer to the operation of reversing the column index order of all tokens in a row. For example, mirroring can move the token at the original column index j to the new column index W-1-j, essentially mirroring it horizontally. A flipped row group can refer to the new set of rows obtained after mirroring each row in an odd-numbered row group. Typically, the flipped rows are completely opposite in memory layout to the original rows. A second token can refer to a new token generated for each row in the flipped row group, following the same concatenation rules as the even-numbered rows.

[0048] Step a4: Merge the first token and the second token according to the row index order to obtain the target feature results corresponding to the target 3D features, and summarize the target feature results corresponding to the multi-head attention module of the same target to obtain the target feature result set.

[0049] Figure 4 The diagram illustrates a token merging method according to an embodiment of the present invention. Specifically, it uses the target three-dimensional features in the three-dimensional feature set as... Figure 3 As shown, taking a preset merging factor of 2 as an example, rows T0 and T2 are marked as even-numbered row groups, and rows T1 and T3 are marked as odd-numbered row groups. Then, the two adjacent column indices of each row in the even-numbered row groups are concatenated along the feature dimension, such as (T... 0,0 ,T 0,1 ), (T) 0,2 ,T 0,3 ), (T) 0,4 ,T 0,5 ) and (T 2,0 ,T 2,1 ), (T) 2,2 ,T 2,3 ), (T) 2,4 ,T 2,5 ), resulting in a first token (T) with a feature dimension twice that of the original dimension.0,0 ,T 0,2 ,T 0,4 ) and (T 2,0 ,T 2,2 ,T 2,4 Simultaneously, the odd-numbered row groups are mirrored to obtain flipped row groups, and the two adjacent column indices of each row in the flipped row groups are concatenated along the feature dimension to obtain the second token (T). 1,1 ,T 1,3 ,T 1,5 ) and (T 3,1 ,T 3,3 ,T 3,5 ).

[0050] Figure 5 The diagram shows a flowchart of a token merging process provided by an embodiment of the present invention. Specifically, firstly, the height dimension H of the target 3D feature in the 3D feature set is traversed, and all row indices 0 to H-1 are divided into even-numbered row groups (indices 0, 2, 4, ...) and odd-numbered row groups (indices 1, 3, 5, ...). Then, for each even-numbered row, the columns are traversed from left to right with a step size of 2. For each pair of adjacent columns (2j, 2j+1), the tokens at these two positions are directly concatenated along the feature dimension to generate a new token, the first token, whose dimension becomes 2×Dim. After processing, the original width W of the row becomes W / 2, and the j-th column in the new row corresponds to the information of the original columns 2j and 2j+1. Simultaneously, for each odd-numbered row, a mirror flip operation is performed: the token at column index j is swapped with the token at column index W-1-j, effectively reversing the entire row horizontally. After the flip is complete, the same adjacent column concatenation rules as for the even-numbered rows are applied to the row to generate a second token. The dimension of each second token is also 2×Dim. Furthermore, following the original row order (row 0, row 1, row 2, etc.), the rows containing the first token and the second token are arranged sequentially to form the target feature result corresponding to the target 3D feature. This target feature result has the shape [Batch×Heads,H,W / 2,2×Dim]. Finally, after independently performing the above operations on the query, key, and value tensors to obtain their respective target 3D features, they are collectively combined into the target feature result set {Q_t,K_t,V_t}, which is used for subsequent local window attention calculations.

[0051] S280. If the feature recombination strategy is a preset data preservation strategy, then the three-dimensional feature set is used as the target feature result set corresponding to the target multi-head attention module.

[0052] Specifically, after determining the feature recombination strategy, feature recombination operations can be performed according to the corresponding feature recombination strategy, thereby obtaining the target feature result set.

[0053] S290. Flatten the target query features, target key features and target value features in the target feature result set according to the preset dimension format to obtain the corresponding flattened query features, flattened key features and flattened value features.

[0054] The preset dimension format refers to the standard format used to flatten a 3D spatial feature map into a 1D sequence. For example, the preset dimension format could be [Batch×Heads, H×W / 2, 2×Dim]. Dimension flattening refers to the operation of merging the height and width spatial dimensions of the feature map into a single sequence length dimension. Flattened query feature refers to the tensor obtained after performing dimension flattening on the target query feature. For example, the shape of the flattened query feature is [Batch×Heads, H×W / 2, 2×Dim]. Flattened key feature refers to the tensor obtained after performing dimension flattening on the target key feature. Flattened value feature refers to the tensor obtained after performing dimension flattening on the target value feature.

[0055] S2100. Based on the first window parameters, the flattened query feature is divided into windows to obtain the query feature sequence corresponding to the flattened query feature; wherein, the query feature sequence contains at least one query feature block; one query feature block corresponds to one target partitioning window.

[0056] In this context, the target partitioning window refers to the computational unit comprised of each query feature block and its corresponding key and value feature blocks within the local window attention. Typically, each partitioning window contains one query feature block, one key feature block, and one value feature block. A query feature block refers to the window partitioning result obtained after partitioning the flattened query feature according to the first window parameter. Typically, the length of each query feature block is the first window parameter, and the query feature blocks corresponding to the same flattened query feature do not overlap. A query feature sequence refers to the sequence structure composed of all query feature blocks arranged in their original order.

[0057] S2110. Based on the query feature sequence and the second window parameters, the flattened key features are divided into windows to obtain a key feature sequence, and based on the query feature sequence and the second window parameters, the flattened value features are divided into windows to obtain a value feature sequence; wherein, the key feature sequence contains at least one key feature block, the value feature sequence contains at least one value feature block, and the same target partitioning window contains a set of query feature blocks, key feature blocks, and value feature blocks.

[0058] Here, a key feature block can refer to a subsequence extracted from flattened key features. A key feature sequence can refer to a sequence structure composed of all key feature blocks arranged in the order of their corresponding query blocks. A value feature block can refer to a subsequence extracted from flattened value features. A value feature sequence can refer to a sequence structure composed of all value feature blocks arranged in the order of their corresponding query blocks.

[0059] In an optional implementation, the step of dividing the flattened key features into windows based on the query feature sequence and the second window parameters to obtain the key feature sequence includes: Step b1: Obtain the target query feature block in the query feature sequence, and determine the basic key feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameters.

[0060] The target query feature block can refer to the specific query block selected for data processing within the query feature sequence. The range of the base key feature block can be defined based on the starting index i and the second window parameter of the target query feature block. The initially calculated key window range. For example, the calculation formula for the basic key feature block division range can include a start position calculation formula and an end position calculation formula. The start position calculation formula can be expressed as: The formula for calculating the ending position can be expressed as: .

[0061] Step b2: Based on the dimensional parameters of the flattened key features, perform data judgment on the division range of the basic key feature blocks, and determine the target key feature block division range corresponding to the division range of the basic key feature blocks according to the judgment result.

[0062] The dimension parameter of the flattening key feature can refer to the total length of the sequence corresponding to the flattening key feature. For example, the dimension parameter of the flattening key feature can be represented as H×W / 2. The judgment result can refer to the logical result obtained after checking the boundaries of the basic key feature block division range. For example, the judgment result can be the starting index. A value less than 0 can be the end index. The dimensional parameters beyond the flattening key features can also be within the range [0, dimensional parameters of the flattening key features). The target key feature block division range can refer to the final key window interval obtained after boundary clipping and adjustment. For example, if the judgment result is the starting index... If it is less than 0, then let =0, and = If the judgment result is the end index. Dimensional parameters beyond the flattening bond features can then allow =H×W / 2, and = - If the judgment result is that both the starting index and the ending index are within the range of [0, the dimension parameter of the flattened key feature), then the range of the basic key feature block division is taken as the range of the target key feature block division.

[0063] Step b3: Based on the target key feature block division range, the flattened key features are divided into windows to obtain a key feature sequence.

[0064] Specifically, assuming the total length of the flattened bond feature is L, and the second window parameter is... Each target query feature block is extracted sequentially from the query feature sequence. Let its starting index be i, and the index range occupied by this query block be [i, i+]. First, calculate the range of the basic key feature block partition: starting index. End of index Then, obtain the dimension parameter L of the flattening key feature for boundary judgment: if <0 indicates that the window goes out of bounds on the left, and the final window range will be adjusted to... =0, = .like >L indicates that the window goes out of bounds to the right, in which case the final window range will be adjusted to... =L- , =L. Otherwise, the window is valid and remains. , Finally, the range is divided according to the adjusted target key feature block. , Extract the corresponding continuous subsequence from the flattened key features to serve as the key feature block for that query block. Perform the above operation on all query blocks sequentially to obtain the complete key feature sequence.

[0065] Accordingly, the step of dividing the flattened value features into windows based on the query feature sequence and the second window parameters to obtain the value feature sequence includes: Step c1: Obtain the target query feature block in the query feature sequence, and determine the basic value feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameter.

[0066] The range of the basic value feature block can refer to the starting index i and the second window parameter of the same target query feature block, calculated using the exact same method as the range of the basic key feature block. The calculated value window range. Typically, the key and value use the same window to divide the range.

[0067] Step c2: Based on the dimensional parameters of the flattened value feature, perform data judgment on the division range of the basic value feature block, and determine the target value feature block division range corresponding to the division range of the basic value feature block according to the judgment result.

[0068] The dimension parameter of the flattened value feature can refer to the total length of the sequence corresponding to the flattened value feature. The target value feature block partition range can refer to the final value window interval obtained after processing with the same boundary pruning logic. Usually, the target value feature block partition range is completely consistent with the target key feature block partition range.

[0069] Step c3: Based on the target value feature block division range, the flattened value feature is divided into windows to obtain a value feature sequence.

[0070] Specifically, the process for dividing the value window is exactly the same as that for the key window. For the same target query feature block with a starting index of i, firstly, the range of the basic value feature block is calculated using the same formula: , Then, based on the dimension parameter L of the flattened value feature, the same boundary judgment logic is performed: if the left boundary is exceeded, the window is shifted to the leftmost end [0, ... If it crosses the boundary to the right, it will shift to the rightmost end [L-]. If valid, the original window is maintained. After obtaining the target value feature block division range, the corresponding continuous interval is extracted from the flattened value features as the value feature block of the query block. All query blocks are traversed to form a value feature sequence.

[0071] It is worth noting that, in this embodiment of the invention, since the key and value use the same window boundary rules and the query block has the same index, the key feature block and value feature block corresponding to the same query block have the same index range. This ensures that the query block, key block and value block are strictly aligned in spatial position during subsequent attention calculation within the window.

[0072] S2120. Perform scaling dot product attention calculation on the query feature blocks, key feature blocks and value feature blocks within the same target partition window to obtain the basic attention result.

[0073] Here, scaling dot product attention calculation refers to the calculation operation based on the standard attention calculation formula. The basic attention result refers to the output obtained after performing scaling dot product attention calculation on the query feature block, key feature block, and value feature block within each target partitioning window. For example, continuing from the above example, taking the first window parameter as... For example, the shape of the basic attention result can be represented as: [Batch×Heads, ,2×Dim].

[0074] S2130. Based on the block order of each query feature block in the query feature sequence, the basic attention result is processed to obtain the candidate attention result.

[0075] Here, the candidate attention result can refer to the tensor obtained by concatenating all the basic attention results according to the block order of the query feature block. For example, following the example above, the shape of the candidate attention result can be: [Batch×Heads,H×W / 2,2×Dim].

[0076] Figure 6 The diagram shows a flowchart of a local window attention calculation method provided in an embodiment of the present invention. Specifically, the input query, key, and value feature map of shape [Batch×Heads,H,W / 2,2×Dim] is first flattened into a one-dimensional sequence, and the parameters of the first window are set. Second window parameters Next, according to The query sequence is divided into non-overlapping query blocks, and the starting index of each block is recorded. Then, each query block is traversed, and the final window partition range of the corresponding key block and value block is calculated based on the starting index i of the current query block, thus obtaining the key block and value block corresponding to that query block. Subsequently, scaling dot product attention calculations are performed within a local window on the query block and its corresponding key and value blocks. The above traversal is repeated until all query blocks have been processed. Finally, the outputs of all query blocks are concatenated in their original order to obtain the candidate attention result.

[0077] S2140. Adjust the candidate attention results based on the feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

[0078] Specifically, after determining the feature recombination strategy and obtaining the target feature result set, the target query features, target key features, and target value features in the target feature result set can be flattened dimensionally to obtain the corresponding flattened query features, flattened key features, and flattened value features. Then, the flattened query features are windowed with the first window parameter as the step size to obtain the query feature sequence corresponding to the flattened query features. Next, each query feature block is traversed, and the flattened key features are windowed according to the second window parameter to obtain the key feature sequence. Similarly, the flattened value features are windowed to obtain the value feature sequence. Further, scaling dot product attention is performed on the query feature blocks, key feature blocks, and value feature blocks within the same target partition window to obtain the basic attention result. Finally, the basic attention result is concatenated based on the block order of each query feature block in the query feature sequence to obtain the candidate attention result. Thus, by replacing global attention with local window attention, each query interacts only with the key values ​​of the second window parameter, significantly reducing the computational load.

[0079] In an optional implementation, the step of adjusting the candidate attention results based on a feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module includes: Step d1: If the feature reorganization strategy is a preset token merging strategy, then the candidate attention results are reorganized in dimensions to obtain intermediate attention results, and the intermediate attention results are restored by token operation to obtain the target attention result corresponding to the target multi-head attention module.

[0080] Dimensional restructuring refers to the operation of restoring the flattened candidate attention results from a one-dimensional sequence to a three-dimensional feature map. Intermediate attention results refer to the feature map obtained after dimensional restructuring of the candidate attention results when the feature restructuring strategy is a preset token merging strategy. For example, the shape of the intermediate attention result can be: [Batch×Heads,H,W / 2,2×Dim]. Token restoration refers to the operation used to restore the merged feature map to its original resolution. For example, the token restoration operation can be: For even rows: split each 2×Dim token into two adjacent Dim tokens, and put them back in left-to-right order. For odd rows: first split each 2×Dim token into two Dim tokens, then mirror the entire row to restore the original order of the odd rows before merging. The final feature map has the shape [Batch×Heads,H,W,Dim].

[0081] Step d2: If the feature reorganization strategy is a preset data preservation strategy, then the candidate attention results are reorganized in dimensions to obtain the target attention result corresponding to the target multi-head attention module.

[0082] Specifically, after obtaining the candidate attention results through local window attention calculation, data adjustment can be performed according to the previously determined feature reorganization strategy: If the feature reorganization strategy is a preset token merging strategy: First, the candidate attention results are dimensionally reorganized, reshaping them into intermediate attention results. Then, a token restoration operation is performed on the intermediate attention results. Finally, the target attention result with the shape [Batch×Heads,H,W,Dim] is obtained. If the feature reorganization strategy is a preset data preservation strategy: The candidate attention results are directly dimensionally reorganized, reshaping them from [Batch×Heads,H×W,Dim] to [Batch×Heads,H,W,Dim], without performing a token restoration operation, and are directly output as the target attention result.

[0083] The technical solution of this invention involves obtaining the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. The shape of each input feature in the input feature set is reshaped based on a preset tensor format to obtain a three-dimensional feature set corresponding to the input feature set. Then, a preset time step threshold corresponding to the target multi-head attention module is determined based on a preset time step length corresponding to the target diffusion model. A threshold judgment is performed on the current time step based on the preset time step threshold, and a merging identifier corresponding to the three-dimensional feature set is determined based on the threshold judgment result. If the merging identifier is true, a preset token merging strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set. If the merging identifier is false, a preset data preservation strategy is used as the feature recombination strategy corresponding to the three-dimensional feature set. Further, if the feature recombination strategy is a preset token merging strategy, token merging is performed on the target three-dimensional features in the three-dimensional feature set based on a preset merging factor and a preset row index rule to obtain the target feature result set corresponding to the target multi-head attention module. If the feature recombination strategy is a preset data preservation strategy, the three-dimensional feature set is used as the target feature result set corresponding to the target multi-head attention module. Furthermore, based on a preset dimensionality format, the target query features, target key features, and target value features in the target feature result set are dimensionally flattened to obtain the corresponding flattened query features, flattened key features, and flattened value features. The flattened query features are then windowed based on the first window parameter to obtain the corresponding query feature sequence. The flattened key features are then windowed based on the query feature sequence and the second window parameter to obtain the key feature sequence, and the flattened value features are then windowed based on the query feature sequence and the second window parameter to obtain the value feature sequence. Scaling dot product attention calculations are performed on the query feature blocks, key feature blocks, and value feature blocks within the same target partition window to obtain the basic attention result. Finally, the basic attention result is concatenated based on the block order of each query feature block in the query feature sequence to obtain the candidate attention result. The candidate attention result is then adjusted based on a feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module. Due to the spatial locality of images, a method combining reduced attention sequence length and local attention is adopted, significantly reducing computational load while ensuring the fidelity and detail consistency of the generated samples. By employing sequential memory access and a streamlined computation process, unnecessary data movement and waiting time are reduced, enabling computational acceleration to truly translate into end-to-end inference latency reduction. This addresses the issues of high computational overhead, low hardware utilization, and potential degradation in generation quality inherent in existing technologies for attention computation in diffusion models. It achieves end-to-end inference acceleration while maintaining generation quality, thus improving actual hardware utilization.

[0084] Figure 7The diagram shows a flowchart of an optional attention acceleration method based on a diffusion model provided by an embodiment of the present invention. Specifically, firstly, three feature maps—the input query, key, and value—are obtained from the multi-head attention module, with shapes [Batch, Heads, H×W, Dim]. Then, each feature map is spatially reshaped to obtain a three-dimensional spatial feature map with the shape [Batch×Heads, H, W, Dim]. Next, it is determined whether token merging needs to be performed based on the current time step: if merging is required, the query, key, and value feature maps are token-merged according to a preset merging factor M, making the feature map size [Batch×Heads, H, W / M, M×Dim]; if merging is not required, proceed directly to the next step. Then, local window attention calculation is performed on the processed feature map, outputting an intermediate attention result with the shape [Batch×Heads, H, W / M, M×Dim]. Finally, if token merging was performed previously, a token recovery operation is performed to restore the feature map to its original resolution of [Batch×Heads, H, W, Dim], and this is output as the final attention result. Therefore, if the shape of an attention head is [1, H × W, Dim], its original computational cost should be: Assuming the token merging factor is M, the computational complexity after token merging is: After token merging, the shape of the attention head becomes [1, H × W / M, Dim]. Assuming that in local window attention, the first window parameter of the query feature sequence is... The second window parameters corresponding to the key feature sequence and the value feature sequence are: The computational cost after applying local window attention is: This greatly reduces the amount of computation.

[0085] Example 3 Figure 8 This is a schematic diagram of an attention acceleration device based on a diffusion model provided in Embodiment 3 of the present invention. Figure 8 As shown, the device includes: a data acquisition module 310, a shape reshaping module 320, a feature recombination module 330, and a result generation module 340; Among them, the data acquisition module 310 is used to acquire the input feature set, current time step and local window parameters corresponding to the target multi-head attention module in the target diffusion model; The shape reshaping module 320 is used to reshape each input feature in the input feature set based on a preset tensor format to obtain a three-dimensional feature set corresponding to the input feature set. The feature recombination module 330 is used to determine the feature recombination strategy corresponding to the three-dimensional feature set based on the current time step, and to recombine the three-dimensional feature set based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; The result generation module 340 is used to perform local window attention calculation on the target feature result set based on the local window parameters and feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

[0086] The technical solution of this invention obtains the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. Then, based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain a three-dimensional feature set corresponding to the input feature set. Further, based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and features are recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module. Finally, based on the local window parameters and the feature recombination strategy, local window attention is calculated on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module. Due to the spatial locality of images, a method combining reduced attention sequence length and local attention is used, significantly reducing computational load while ensuring the fidelity and detail consistency of the generated samples. By using continuous memory access and a well-organized computation process, unnecessary data movement and waiting time are reduced, allowing computational acceleration to truly translate into end-to-end inference latency reduction. It solves the problems of high computational overhead, low hardware utilization, and easy degradation of generation quality in the attention calculation process of existing technologies in diffusion models. It can achieve end-to-end actual inference acceleration while maintaining generation quality and improve the actual hardware utilization.

[0087] Optionally, the feature recombination module 330 can be used for: The preset time step threshold corresponding to the target multi-head attention module is determined based on the preset time step corresponding to the target diffusion model. The current time step is judged based on the preset time step threshold, and the merging identifier corresponding to the three-dimensional feature set is determined based on the threshold judgment result. If the merge identifier is true, then the preset token merging strategy will be used as the feature recombination strategy corresponding to the three-dimensional feature set; If the merging identifier is false, then the preset data preservation strategy will be used as the feature recombination strategy corresponding to the three-dimensional feature set.

[0088] Optionally, the feature recombination module 330 can be used for: If the feature recombination strategy is a preset token merging strategy, then the target three-dimensional features in the three-dimensional feature set are token-merged based on the preset merging factor and the preset row indexing rule to obtain the target feature result set corresponding to the target multi-head attention module; If the feature recombination strategy is a preset data preservation strategy, then the three-dimensional feature set is used as the target feature result set corresponding to the target multi-head attention module.

[0089] Optionally, the feature recombination module 330 can be used for: Based on the preset row indexing rules, the target three-dimensional features in the three-dimensional feature set are divided into even row groups and odd row groups according to the parity of the row index; Based on a preset merging factor, the adjacent target column indices in the even-numbered row group are concatenated along the feature dimension to obtain a first token with a feature dimension twice that of the original dimension. The odd-numbered row groups are mirrored to obtain a flipped row group. Adjacent target column indices in the flipped row group are then concatenated along the feature dimension based on a preset merging factor to obtain a second token. The first token and the second token are merged and processed according to the row index order to obtain the target feature results corresponding to the target 3D features. The target feature results corresponding to the multi-head attention module of the same target are then summarized and processed to obtain the target feature result set.

[0090] Optionally, the target feature result set includes target query features, target key features, and target value features; the local window parameters include a first window parameter and a second window parameter, wherein the first window parameter is smaller than the second window parameter; Result generation module 340 can be used specifically for: Based on a preset dimension format, the target query features, target key features and target value features in the target feature result set are dimensionally flattened to obtain the corresponding flattened query features, flattened key features and flattened value features. The flattened query feature is divided into windows based on the first window parameters to obtain a query feature sequence corresponding to the flattened query feature; wherein, the query feature sequence contains at least one query feature block; one query feature block corresponds to one target partitioning window; The flattened key features are divided into windows based on the query feature sequence and the second window parameters to obtain a key feature sequence. The flattened value features are also divided into windows based on the query feature sequence and the second window parameters to obtain a value feature sequence. The key feature sequence contains at least one key feature block, the value feature sequence contains at least one value feature block, and the same target partitioning window contains a set of query feature blocks, key feature blocks, and value feature blocks. Calculate scaling dot product attention for query feature blocks, key feature blocks, and value feature blocks within the same target partitioning window to obtain the basic attention result; Based on the block order of each query feature block in the query feature sequence, the basic attention result is concatenated to obtain the candidate attention result; The candidate attention results are adjusted based on a feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

[0091] Optionally, the result generation module 340 can be used for: Obtain the target query feature block in the query feature sequence, and determine the basic key feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameters; The data judgment is performed on the division range of the basic key feature block based on the dimensional parameters of the flattened key feature, and the target key feature block division range corresponding to the division range of the basic key feature block is determined according to the judgment result. Based on the target key feature block division range, the flattened key features are divided into windows to obtain a key feature sequence; Correspondingly, the result generation module 340 can be used for: Obtain the target query feature block in the query feature sequence, and determine the basic value feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameter; The data judgment is performed on the division range of the basic value feature block based on the dimension parameters of the flattened value feature, and the target value feature block division range corresponding to the division range of the basic value feature block is determined according to the judgment result. The flattened value features are divided into windows based on the target value feature block division range to obtain a value feature sequence.

[0092] Optionally, the result generation module 340 can be used for: If the feature reorganization strategy is a preset token merging strategy, then the candidate attention results are reorganized in dimensions to obtain intermediate attention results, and the intermediate attention results are restored by token operation to obtain the target attention result corresponding to the target multi-head attention module. If the feature reorganization strategy is a preset data preservation strategy, then the candidate attention results are reorganized in dimensions to obtain the target attention result corresponding to the target multi-head attention module.

[0093] The attention acceleration device based on the diffusion model provided in the embodiments of the present invention can execute the attention acceleration method based on the diffusion model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0094] Example 4 Figure 9 A schematic diagram of an electronic device 410 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0095] like Figure 9 As shown, the electronic device 410 includes at least one processor 420 and a memory, such as a read-only memory (ROM) 430 or a random access memory (RAM) 440, communicatively connected to the at least one processor 420. The memory stores computer programs executable by the at least one processor. The processor 420 can perform various appropriate actions and processes based on the computer program stored in the ROM 430 or loaded into the RAM 440 from storage unit 490. The RAM 440 may also store various programs and data required for the operation of the electronic device 410. The processor 420, ROM 430, and RAM 440 are interconnected via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0096] Multiple components in electronic device 410 are connected to I / O interface 460, including: input unit 470, such as keyboard, mouse, etc.; output unit 480, such as various types of monitors, speakers, etc.; storage unit 490, such as disk, optical disk, etc.; and communication unit 4100, such as network card, modem, wireless transceiver, etc. Communication unit 4100 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] Processor 420 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 420 performs the various methods and processes described above, such as attention acceleration methods based on diffusion models.

[0098] The method includes: Obtain the input feature set, current time step, and local window parameters corresponding to the multi-head attention module in the target diffusion model; Based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain the three-dimensional feature set corresponding to the input feature set. Based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and the three-dimensional feature set is recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; Based on the local window parameters and feature recombination strategy, local window attention calculation is performed on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module.

[0099] In some embodiments, the diffusion-based attention acceleration method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 410 via ROM 430 and / or communication unit 4100. When the computer program is loaded into RAM 440 and executed by processor 420, one or more steps of the diffusion-based attention acceleration method described above can be performed. Alternatively, in other embodiments, processor 420 can be configured to perform the diffusion-based attention acceleration method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0105] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0106] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the diffusion-based attention acceleration method provided in any embodiment of this application. This program product shares the same inventive concept as the diffusion-based attention acceleration method disclosed in the embodiments of this application, and therefore will not be described further here.

[0107] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An attention acceleration method based on a diffusion model, characterized in that, include: Obtain the input feature set, current time step, and local window parameters corresponding to the multi-head attention module in the target diffusion model; Based on a preset tensor format, the shape of each input feature in the input feature set is reshaped to obtain the three-dimensional feature set corresponding to the input feature set. Based on the current time step, a feature recombination strategy corresponding to the three-dimensional feature set is determined, and the three-dimensional feature set is recombined based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; Based on the local window parameters and feature recombination strategy, local window attention calculation is performed on the target feature result set to obtain the target attention result corresponding to the target multi-head attention module.

2. The method according to claim 1, characterized in that, The feature recombination strategy for determining the three-dimensional feature set based on the current time step includes: The preset time step threshold corresponding to the target multi-head attention module is determined based on the preset time step corresponding to the target diffusion model. The current time step is judged based on the preset time step threshold, and the merging identifier corresponding to the three-dimensional feature set is determined based on the threshold judgment result. If the merge identifier is true, then the preset token merging strategy will be used as the feature recombination strategy corresponding to the three-dimensional feature set; If the merging identifier is false, then the preset data preservation strategy will be used as the feature recombination strategy corresponding to the three-dimensional feature set.

3. The method according to claim 2, characterized in that, The step of reorganizing the three-dimensional feature set based on the feature reorganization strategy to obtain the target feature result set corresponding to the target multi-head attention module includes: If the feature recombination strategy is a preset token merging strategy, then the target three-dimensional features in the three-dimensional feature set are token-merged based on the preset merging factor and the preset row indexing rule to obtain the target feature result set corresponding to the target multi-head attention module; If the feature recombination strategy is a preset data preservation strategy, then the three-dimensional feature set is used as the target feature result set corresponding to the target multi-head attention module.

4. The method according to claim 3, characterized in that, The token merging of the target 3D features in the 3D feature set based on a preset merging factor and a preset row indexing rule yields the target feature result set corresponding to the target multi-head attention module, including: Based on the preset row indexing rules, the target three-dimensional features in the three-dimensional feature set are divided into even row groups and odd row groups according to the parity of the row index; Based on a preset merging factor, the adjacent target column indices in the even-numbered row group are concatenated along the feature dimension to obtain a first token with a feature dimension twice that of the original dimension. The odd-numbered row groups are mirrored to obtain a flipped row group. Adjacent target column indices in the flipped row group are then concatenated along the feature dimension based on a preset merging factor to obtain a second token. The first token and the second token are merged and processed according to the row index order to obtain the target feature results corresponding to the target 3D features. The target feature results corresponding to the multi-head attention module of the same target are then summarized and processed to obtain the target feature result set.

5. The method according to claim 2, characterized in that, The target feature result set includes target query features, target key features, and target value features; The local window parameters include a first window parameter and a second window parameter, wherein the first window parameter is smaller than the second window parameter; The step of performing local window attention calculation on the target feature result set based on the local window parameters and feature reorganization strategy to obtain the target attention result corresponding to the target multi-head attention module includes: Based on a preset dimension format, the target query features, target key features and target value features in the target feature result set are dimensionally flattened to obtain the corresponding flattened query features, flattened key features and flattened value features. The flattened query feature is divided into windows based on the first window parameters to obtain a query feature sequence corresponding to the flattened query feature; wherein, the query feature sequence contains at least one query feature block; one query feature block corresponds to one target partitioning window; The flattened key features are divided into windows based on the query feature sequence and the second window parameters to obtain a key feature sequence. The flattened value features are also divided into windows based on the query feature sequence and the second window parameters to obtain a value feature sequence. The key feature sequence contains at least one key feature block, the value feature sequence contains at least one value feature block, and the same target partitioning window contains a set of query feature blocks, key feature blocks, and value feature blocks. Calculate scaling dot product attention for query feature blocks, key feature blocks, and value feature blocks within the same target partitioning window to obtain the basic attention result; Based on the block order of each query feature block in the query feature sequence, the basic attention result is concatenated to obtain the candidate attention result; The candidate attention results are adjusted based on a feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

6. The method according to claim 5, characterized in that, The step of dividing the flattened key features into windows based on the query feature sequence and the second window parameters to obtain the key feature sequence includes: Obtain the target query feature block in the query feature sequence, and determine the basic key feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameters; The data judgment is performed on the division range of the basic key feature block based on the dimensional parameters of the flattened key feature, and the target key feature block division range corresponding to the division range of the basic key feature block is determined according to the judgment result. Based on the target key feature block division range, the flattened key features are divided into windows to obtain a key feature sequence; The step of dividing the flattened value features into windows based on the query feature sequence and the second window parameters to obtain the value feature sequence includes: Obtain the target query feature block in the query feature sequence, and determine the basic value feature block division range corresponding to the target query feature block based on the index value of the target query feature block and the second window parameter; The data judgment is performed on the division range of the basic value feature block based on the dimension parameters of the flattened value feature, and the target value feature block division range corresponding to the division range of the basic value feature block is determined according to the judgment result. The flattened value features are divided into windows based on the target value feature block division range to obtain a value feature sequence.

7. The method according to claim 5, characterized in that, The feature-based reorganization strategy is used to adjust the candidate attention results to obtain the target attention result corresponding to the target multi-head attention module, including: If the feature reorganization strategy is a preset token merging strategy, then the candidate attention results are reorganized in dimensions to obtain intermediate attention results, and the intermediate attention results are restored by token operation to obtain the target attention result corresponding to the target multi-head attention module. If the feature reorganization strategy is a preset data preservation strategy, then the candidate attention results are reorganized in dimensions to obtain the target attention result corresponding to the target multi-head attention module.

8. An attention acceleration device based on a diffusion model, characterized in that, include: The data acquisition module is used to acquire the input feature set, current time step, and local window parameters corresponding to the target multi-head attention module in the target diffusion model. The shape reshaping module is used to reshape each input feature in the input feature set based on a preset tensor format to obtain a three-dimensional feature set corresponding to the input feature set. The feature recombination module is used to determine the feature recombination strategy corresponding to the three-dimensional feature set based on the current time step, and to recombine the features of the three-dimensional feature set based on the feature recombination strategy to obtain the target feature result set corresponding to the target multi-head attention module; The result generation module is used to perform local window attention calculation on the target feature result set based on the local window parameters and feature recombination strategy to obtain the target attention result corresponding to the target multi-head attention module.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the attention acceleration method based on the diffusion model according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the attention acceleration method based on the diffusion model as described in any one of claims 1-7.

Images