Feature processing method and device based on multi-head attention network in model, and equipment

Abstract

The application discloses a feature processing method and device based on a multi-head attention network in a model and equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining input data of a multi-head attention network, and obtaining query feature representations corresponding to a plurality of attention heads in an attention group and key-value feature representations corresponding to the plurality of attention heads based on the input data; for each attention head in the attention group, dividing the query feature representations into a plurality of query blocks and dividing the key-value feature representations into a plurality of key-value blocks; for the query blocks at the same position of each query feature representation, determining a shared feature block from the key-value blocks corresponding to the plurality of attention heads and determining private feature blocks of each attention head from the key-value blocks of each attention head; and based on each attention head and each query block, querying the shared feature block and the private feature blocks respectively to obtain attention feature representations. The feature processing efficiency is improved evenly, and the differentiated attention ability to features is retained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a feature processing method, apparatus and device based on a multi-head attention network in a model. Background Technology

[0002] Sparse attention mechanisms are used to provide deep learning models with a computationally less complex attention query strategy to improve feature processing efficiency, thereby enabling deep learning models to perform prediction or generation tasks, such as video generation.

[0003] In related technologies, at least one selected feature block is selected from multiple feature blocks corresponding to the input feature representation through a sparse attention mechanism; at least one selected feature block is independently linearly projected through a multi-head attention mechanism, and attention calculation is performed to obtain the feature processing result.

[0004] However, in the above feature processing methods, the selected feature blocks of different attention heads are the same, and the semantic attention of different attention heads is quite different, resulting in low feature processing accuracy and low accuracy of feature processing result indication generation. Summary of the Invention

[0005] This application provides a feature processing method, apparatus, and device based on a multi-head attention network in a model. The technical solution provided by this application includes the following aspects.

[0006] According to one aspect of the embodiments of this application, a feature processing method based on a multi-head attention network in a model is provided, the method comprising: The input data of the multi-head attention network in the model is obtained, and the query feature representations and key-value feature representations corresponding to the multiple attention heads in the attention group are obtained based on the input data. For each attention head in the attention group, the query feature representation is divided into multiple query blocks, and the key value feature representation is divided into multiple key value blocks; For query blocks at the same position represented by each query feature, a shared feature block is determined from the key-value blocks corresponding to the multiple attention heads, and a private feature block corresponding to each attention head is determined from the key-value blocks corresponding to each attention head. Based on each query block corresponding to each attention head, the shared feature block and the private feature block are queried respectively to obtain the attention feature representation output by the multi-head attention network.

[0007] According to one aspect of the embodiments of this application, a feature processing apparatus based on a multi-head attention network in a model is provided, the apparatus comprising: The feature mapping module is configured to acquire the input data of the multi-head attention network in the model, and map the input data to obtain the query feature representations corresponding to the multiple attention heads in the attention group and the key-value feature representations corresponding to the multiple attention heads. The block segmentation module is configured to divide the query feature representation into multiple query blocks and the key value feature representation into multiple key value blocks for each attention head. The routing module is configured to determine shared feature blocks from the key-value blocks corresponding to multiple attention heads for query blocks at the same position divided by each query feature representation, and to determine the private feature blocks corresponding to each attention head from the key-value blocks corresponding to each attention head. The feature processing module is configured to query the shared feature block and the private feature block respectively based on each query block corresponding to each attention head, so as to obtain the attention feature representation output by the multi-head attention network.

[0008] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described feature processing method based on a multi-head attention network in the model.

[0009] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described feature processing method based on a multi-head attention network in the model.

[0010] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described feature processing method based on a multi-head attention network in the model.

[0011] The technical solutions provided in this application can bring at least the following beneficial effects: By mapping the input data to the feature spaces corresponding to multiple attention heads in an attention group, and dividing it into multiple query blocks and multiple key-value blocks, the feature processing objects for the block sparse attention mechanism are provided. For each position of the query block in the different query feature representations corresponding to different attention heads, the key-value blocks that indicate the focus of the query block are selected from the multiple key-value blocks. These include shared feature blocks shared by multiple attention heads at that position, and private feature blocks corresponding to each attention head. Shared feature blocks simplify the routing process of attention calculation and improve feature processing efficiency, while private feature blocks are used to retain the differentiated attention capabilities of different attention heads to the input feature representation, improve the accuracy of feature processing, and thus improve the efficiency and accuracy of subsequent calculations based on attention feature representations. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a feature processing system based on a multi-head attention network in a model provided in an exemplary embodiment of this application; Figure 2 This is a flowchart of a feature processing method based on a multi-head attention network in a model provided in an exemplary embodiment of this application; Figure 3 This is a comparative schematic diagram of the selected feature block selection method provided in an exemplary embodiment of this application; Figure 4 This is a flowchart of a feature processing method based on a multi-head attention network in a model, provided in another exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the selection method between shared feature blocks and private feature blocks provided in an exemplary embodiment of this application; Figure 6 This is a flowchart of a feature processing method based on a multi-head attention network in a model provided in another exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating a selected feature block quantity budget configuration method provided in an exemplary embodiment of this application; Figure 8 This is a schematic diagram of a feature processing flow provided in an exemplary embodiment of this application; Figure 9 This is a structural block diagram of a feature processing device based on a multi-head attention network in a model provided in an exemplary embodiment of this application; Figure 10 This is a schematic block diagram of a computer device provided in an exemplary embodiment of this application; Figure 11 This is a schematic block diagram of a computer device provided in another exemplary embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] The Transformer model is a deep learning model based on an attention mechanism. It is used to perform prediction tasks such as natural language processing and computer vision processing by processing feature vectors corresponding to multimodal data. The sparse attention mechanism provides a computationally less complex attention query strategy for deep learning models to improve feature processing efficiency, thereby enabling deep learning models to perform prediction or generation tasks, such as video generation.

[0015] The standard self-attention mechanism in the Transformer model performs feature processing by performing attention calculations on all query terms and all key terms. This involves a large number of terms, leading to high computational costs and high GPU memory access overhead. Illustratively, when the Transformer model is implemented as a video generation model, the number of basic processing units for video data is even greater, and it has both temporal and spatial dimensions. This makes the attention calculation stage a significant bottleneck in inference speed and GPU memory usage during video generation.

[0016] In related technologies, a block sparse attention mechanism divides the input feature representation of the input data into multiple query blocks and multiple key-value blocks. For each query block, a subset of the key-value blocks is selected as the query block, and precise attention calculations are performed only on the selected key-value blocks. This reduces computational and memory overhead while maintaining the quality of the deep learning model corresponding to feature processing. Alternatively, a multi-attention head mechanism can be used, where multiple attention heads share the selected key-value blocks for their respective query blocks and perform attention calculations separately.

[0017] However, in the aforementioned feature processing methods based on multi-head attention networks in the model, when different attention heads select the same key value block, their semantic attention varies significantly. For example, different attention heads may focus on different objects, different motion regions, different local textures, or different temporal locations. When all attention heads are forced to focus on the exact same key value block, key value blocks that are focused on by a few attention heads but have a high weight influencing the model's generation quality may be missed during feature processing, resulting in low feature processing accuracy.

[0018] To illustrate, taking the Transformer model as an example of a video generation model, the following situations may occur where key-value blocks are missed during the feature processing: 1. Multiple object loss: Multiple objects or subjects that should exist in the generated video data are missing, with only some objects appearing. 2. Object confusion: Information such as the appearance, color, or position of different objects or subjects is mixed together, making them difficult to distinguish. 3. Unstable local texture: Details on the surface of video objects (such as stripes, spots, etc.) flicker or change abruptly over time, lacking temporal consistency of texture. 4. Discontinuous motion: The motion trajectory of objects in the video data jumps, stops, or reverses, which does not conform to the laws of physics. 5. Block artifacts: Obvious rectangular grid boundaries or blockiness appear in the generated image, especially at the boundaries of attention blocks. 6. Decreased detail fidelity: The clarity or accuracy of high-frequency information (such as local texture, edge information, small-sized targets, etc.) in the generated video data is low, resulting in a blurry and rough image.

[0019] To address the aforementioned issues, this application provides a feature processing method based on a multi-head attention network in the model. This method divides the input feature representation corresponding to the input information into multiple query blocks and multiple key-value blocks after mapping with multiple attention heads. For each query block corresponding to each attention head, a key-value block indicating the attention head's focus is selected from the multiple key-value blocks. This achieves a sparse attention method for video generation models based on a shared-private hybrid key-value block approach, solving the problem of balancing feature processing efficiency and quality in existing shared-block sparse attention methods. The selected key-value blocks include shared feature blocks shared by attention heads within the attention group, and private feature blocks corresponding to each attention head. Shared feature blocks simplify the routing process of attention calculation and improve feature processing efficiency, while private feature blocks preserve the differentiated attention capabilities of different attention heads to the input feature representation, improving the accuracy of feature processing.

[0020] Please refer to Figure 1 This illustration shows a schematic diagram of a feature processing system based on a multi-head attention network in a model, provided in an exemplary embodiment of this application. The computer system 100 includes: a terminal 120, or, a terminal 120 and a first server 140.

[0021] The feature processing method based on multi-head attention networks in a model provided in this application embodiment can be a computer device in a computer system 100, wherein the computer device is deployed with a model training device that applies the model training method. In some embodiments, the computer device includes edge devices, hybrid input devices, wearable devices, Internet of Things devices, etc. Optionally, the aforementioned computer device can be a terminal 120 or a first server 140, etc. This application embodiment does not limit the executing entity of each step of the feature processing method based on multi-head attention networks in a model. The feature processing method based on multi-head attention networks in a model can be executed independently by the terminal 120, independently by the first server 140, or jointly by the terminal 120 and the first server 140. In some embodiments, the computer system 100 can be implemented as a system architecture for the feature processing method based on multi-head attention networks in a model.

[0022] The device types of terminal 120 include at least one of the following: smartphones, desktop computers, laptop computers, personal computers (PCs), tablet computers, e-book readers, smart robots, augmented reality (AR) devices, virtual reality (VR) devices, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, smart TVs, in-vehicle terminals, wearable devices, and other electronic devices. This application embodiment does not limit the types of devices included in the embodiments.

[0023] Terminal 120 is connected to the first server 140 via a wireless network or a wired network.

[0024] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0025] The first server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The first server 140 provides functional services for implementing model training. Optionally, the first server 140 undertakes the primary computational work, and the terminal 120 undertakes the secondary computational work; or, the first server 140 undertakes the secondary computational work, and the terminal 120 undertakes the primary computational work; or, the first server 140 and the terminal 120 collaborate on computation using a distributed computing architecture.

[0026] In one example, a feature processing method based on a multi-head attention network in the model is implemented collaboratively by a terminal 120 and a first server 140. Illustratively, input data is acquired through the terminal 120; optionally, the input data includes multimodal data. The first server 140 receives the input data sent by the terminal 120 and maps the input data into input feature representations corresponding to multiple attention heads, including query feature representations and key-value feature representations. For each attention head in the attention group, the query feature representation is divided into multiple query blocks, and the key-value feature representation is divided into multiple key-value blocks. A pre-trained deep learning model is deployed in the first server 140, which is used to process the input data and perform attention feature processing on the input feature representations.

[0027] In the above example, when the first server 140 obtains multiple query blocks and multiple key-value blocks, it determines shared feature blocks from the key-value blocks corresponding to multiple attention heads for query blocks at the same position divided by each query feature representation, and determines the private feature blocks corresponding to each attention head from the key-value blocks corresponding to each attention head; it then uses an attention mechanism to query the shared and private feature blocks corresponding to each query block to obtain the attention feature representation. Optionally, based on the attention feature representation, generated content corresponding to the input data can be generated, or the input data can be identified to obtain a recognition result.

[0028] Optionally, in the above example, when the first server 140 generates content corresponding to the input data, the first server 140 sends the generated content to the terminal 120 and displays it or performs further processing.

[0029] It is worth noting that the aforementioned terminal 120 refers to an electronic device with output display capabilities and input control capabilities. The aforementioned first server 140 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0030] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0031] In some embodiments, the first server 140 described above can also be implemented as a node in a blockchain system.

[0032] Please refer to Figure 2 This document illustrates a flowchart of a feature processing method based on a multi-head attention network in a model, provided in an exemplary embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method is executed by either the terminal 120 or the first server 140 shown, or it is executed jointly by the terminal and the server through interaction. In this embodiment, the method is taken as being executed by the terminal as an example. Figure 2 As shown, the method may include steps 210 to 240.

[0033] Step 210: Obtain the input data of the multi-head attention network in the model, and map the input data to obtain the query feature representations and key-value feature representations corresponding to the multiple attention heads in the attention group.

[0034] In some embodiments, there is a correspondence between the input data and the task to be performed by the deep learning model corresponding to feature processing, including but not limited to at least one of the following.

[0035] 1. When the task to be performed is a content generation task, the input data includes prompt information, which is used to indicate the content generation requirements; the attention feature representation is the feature representation obtained by the attention layer in the content generation task; the content generation task includes at least one of video generation task, image generation task, audio generation task, and text generation task.

[0036] 2. Input data includes image data, which is the data to be recognized; attention feature representation is the feature representation obtained by the attention layer in the image recognition task.

[0037] 3. Input data includes audio data, which is the data to be recognized; attention feature representation is the feature representation obtained by the attention layer in the audio recognition task.

[0038] Taking the task to be executed as a video generation task as an example, the prompt information is used to indicate the video generation requirements. The prompt information is the input data of the video generation model; optionally, the prompt information corresponding to the video generation model includes, but is not limited to, at least one conditional information such as text prompts, image prompts, and video clip prompts, or the prompt information includes unconditional information such as noise data, wherein the noise data is a random tensor independently sampled from a standard high-speed distribution, and the noise data does not include semantic and structural information; the dimensions of each dimension of the noise data are the same as the dimensions of each dimension of the target video data corresponding to the video generation requirements.

[0039] In some embodiments, the input data can be implemented as multimodal data, which includes at least two modalities interacting, i.e., performing task prediction, data recognition, or content recognition based on multimodal data. Optionally, when obtaining input data of different modalities, the feature dimensions are supplemented by uniformly or dynamically adjusting the encoding rules, so that the dimensions of each dimension of the input data are the same as the dimensions of each dimension of the target data corresponding to the task requirements. Here, the task requirements refer to the feature processing requirements corresponding to the input data.

[0040] Schematic representation: The multimodal data corresponding to the input data includes, but is not limited to, relevant open-source data, network-acquired data, manually collected and labeled data, real-world scene-collected data, historical model-generated data, and user interaction input data. In some embodiments, based on the physical source and perceptual dimension of the multimodal data, the multimodal data includes, but is not limited to, at least one of the following: 1. Visual modal data: including image data, video data, 3D model data, etc. 2. Text modal data: including natural language text, code, structured data, etc., presented in the form of symbolic sequences. 3. Auditory modal data: including waveform audio such as speech and voiceprints, and symbolic audio such as musical scores. 4. Physical modal data: including sensor signal data such as pressure and texture, and motion data such as acceleration. 5. Spatiotemporal modal data: including time series, spatial coordinates, etc.

[0041] In some embodiments, taking the task to be performed as a video generation task as an example, a pre-trained video generation model receives prompt information and obtains input feature representation based on the prompt information through the encoding module in the video generation model; wherein, the video generation model includes, but is not limited to, at least one neural network model such as a video diffusion model, an image / video Transformer model, a large language model, or a multimodal model.

[0042] In the video generation model, the latent space corresponding to the input feature representation of the input data is a low-dimensional feature space obtained by compression and mapping after the encoding module of the video generation model. The input feature representation retains the semantic and structural information of the prompts, but its dimensionality is lower than that of the original prompts, which reduces the computational cost in feature processing and video generation. Illustratively, the video generation model performs attention calculations and denoising on the input feature representation, and then uses the decoding module to restore the generated results to video data.

[0043] It is worth noting that this application does not limit the input data, the model of the task to be executed, the type of the task to be executed, or the specific implementation method.

[0044] The feature transformation process, which obtains input feature representations based on input data, is implemented as an encoding process. There is a correspondence between this encoding process and the video generation model that performs video generation. In the Transformer model, the input feature representations include query feature representations (Query, Q), key feature representations (Key, K), and value feature representations (Value, V), obtained through linear projection of different weight matrices; among these, there is a correspondence between the key feature representations and the value feature representations.

[0045] Where Q is used for addressing, K is used for responding to addressing, and V is used to provide the content to be read; schematically, the query feature represents the "content that the current feature unit wants to find", which is the encoding of the feature unit's attention needs; the key feature represents the "features that each feature unit in the input data has that can be matched", which is the encoding of the feature unit's potential to be noticed; the value feature represents the "actual content information carried by each feature unit in the input data", which is the specific value contributed by the feature unit after it is selected for query.

[0046] Indicatively, the first Each attention point uses its own independent projection matrix (query weight matrix) Key weight matrix Value weight matrix A linear transformation is performed on the same input feature representation A to generate the feature mapping result of the attention head, including the query feature representation. Key feature representation Value feature representation ; where the shape of each projection matrix is ,in H represents the number of attention heads. The projection processes of all attention heads are independent and performed in parallel on the same input feature representation, yielding... Different feature mapping results.

[0047] Step 220: For each attention head in the attention group, divide the query feature representation into multiple query blocks and the key value feature representation into multiple key value blocks.

[0048] The input feature representation includes multiple adjacent feature units. These feature units (tokens) include, but are not limited to, at least one of the following: text tokens corresponding to text feature representations, image blocks corresponding to image feature representations, spatiotemporal blocks or frame-by-frame image blocks corresponding to video feature representations, and spectral time segments corresponding to audio feature representations. Optionally, when the prompt information is implemented as multimodal data, the feature units of the input feature representation include vector sequences encoded into different modalities of uniform length. Taking a spatiotemporal block as an example, each feature unit corresponds to multi-dimensional positional information, including temporal position (time frame number) and spatial position (two-dimensional coordinates indicating pixel height and width).

[0049] Optionally, the aforementioned feature blocks include, but are not limited to, at least one of query blocks and key-value blocks. The number of query blocks is consistent with the way key-value blocks are divided, and correspondingly, the number of query blocks is consistent with the number of key-value blocks.

[0050] The input feature representation is divided into multiple feature blocks according to fixed and / or dynamic partitioning rules, and each feature block has the same size; for example, a feature block includes multiple feature units with a width of 4 pixels, a height of 4 pixels, and a time frame range of 4.

[0051] The Multi-Head Attention (MHA) mechanism assigns an independent key-value attention head to each query attention head. The input feature representation includes a query feature representation and key-value feature representations (including key and value representations). The query feature representation and key-value feature representation correspond at the attention head level; that is, the key-value feature representation mapped by the key-value attention head corresponding to the query attention head is to be queried by the query feature representation mapped by that query attention head. Correspondingly, all feature blocks corresponding to the input feature representation of the input data include query blocks and the key-value blocks being queried, where the key-value blocks include key feature blocks and value feature blocks.

[0052] The query feature representation, key feature representation, and value feature representation are obtained through the attention layer of the video generation model. Illustratively, when the input feature representation is implemented as a video feature representation, the input feature representation includes temporal dimension, spatial dimension, channel dimension, and attention head dimension.

[0053] In some embodiments, under the MHA mechanism, the feature processing in the video generation process can also be implemented as at least one shared key-value structure, such as Grouped-Query Attention (GQA) mechanism or Multi-Query Attention (MQA) mechanism. The GQA mechanism assigns the same key-value attention head to each query attention head, and the MQA mechanism assigns the same key-value attention head to multiple query attention heads within the same attention group.

[0054] Step 230: For query blocks at the same position in each query feature representation, determine the shared feature block from the key-value blocks corresponding to the multiple attention heads respectively, and determine the private feature block corresponding to each attention head from the key-value blocks corresponding to each attention head.

[0055] Taking video generation as an example, the video generation model may optionally include an input and block partitioning module for receiving the input feature representation of the attention layer and dividing it into feature blocks; a shared routing module for selecting shared feature blocks within the attention group; a private supplementary routing module for selecting private feature blocks for each attention head; a routing merging module for generating routing information for the selected key-value blocks of the sparse attention; and a sparse attention execution module for performing precise attention calculations within the selected feature blocks corresponding to each attention head.

[0056] The block sparse attention mechanism divides the query feature representation and key-value feature representation into fixed-size query blocks and key-value blocks, respectively. It then performs attention computation on the selected key-value blocks using query blocks to achieve feature processing. The key-value feature representation includes both key and value feature representations. Selecting a key-value block can be implemented as a routing process; the routing information corresponding to the selected queried key-value block can be implemented as a sparse block selection graph and / or a block sparse route set. Each query block corresponds to routing information, which is used to query the selected key-value block during the attention feature processing.

[0057] Optionally, the selected key-value block is called the selected feature block. The selected feature block includes shared feature blocks shared by multiple attention heads within an attention group and private feature blocks corresponding to each attention head. Shared feature blocks within an attention group refer to query blocks at the same position partitioned for the query feature representations corresponding to multiple attention heads within the attention group. These query blocks at the same position query their respective corresponding key-value blocks at the same position. The selected feature block provides feature information for the actual attention computation during feature processing.

[0058] Different attention heads exhibit semantic focus differences, including but not limited to: different attention heads focusing on different objects, where objects are semantic entities of the same type in the input feature representation (e.g., vehicles in a video); different attention heads focusing on different motion regions, where motion regions are dynamic parts of the input feature representation (e.g., objects moving in a video); different attention heads focusing on different local textures, where local textures are high-frequency details in the input feature representation (e.g., stripes, grids, water ripples on an object's surface), used to maintain visual fidelity; and different attention heads focusing on different temporal positions, where temporal positions are implemented as the sequence numbers of time frames in the input feature representation, used to determine the temporal dependencies between different feature blocks.

[0059] By dividing the selected feature blocks ultimately accessible to each attention head into shared and private portions using shared and private feature blocks corresponding to each attention head, the total available feature blocks for each attention head are divided into shared and private portions. Shared feature blocks provide the same feature information that different attention heads commonly focus on for feature processing; private feature blocks provide independent feature information that different attention heads focus on for feature processing. The process by which different attention heads determine their respective private feature blocks is independent of each other, and the private feature blocks corresponding to different attention heads may be the same or different.

[0060] By expanding the selected feature blocks into shared feature blocks and private feature blocks, and using a sparse routing method for the expanded blocks, the backbone structure of the model to be executed can be reused. There is no need to adjust the main query projection layer, primary key projection layer, and primary value projection layer in the encoding module. The routing method of shared feature blocks and private feature blocks is embedded into the model.

[0061] In some embodiments, the block-based sparse attention mechanism includes at least one of the following feature processing methods: 1. Per-head feature block selection: Each attention head independently selects at least one selected feature block. Compared to selected feature blocks from different attention heads, this method has higher semantic fidelity, but the overhead of selecting, storing, and performing sparse attention computation is higher. 2. Shared-head feature block selection: The same group of attention heads shares the same feature blocks. This method has higher feature processing efficiency, but it is prone to losing the differentiated attention of each attention head, compressing the differences between attention heads, and causing important feature blocks that a few attention heads focus on to be missed. 3. Post-aggregation shared selected feature block selection: First, the selected feature block index of each attention head is calculated. Then, the selected feature block indices of multiple attention heads are aggregated by averaging, maximizing, voting, etc. Finally, the shared feature blocks that meet the preset conditions are selected as the selected feature blocks. 4. Increase the number of shared feature blocks in the selected feature block selection method: By increasing the preset number of selected feature blocks to cover more feature blocks, the computational cost of sparse attention is directly increased, and it cannot be guaranteed that the feature blocks that are truly missing from the semantic perspective of each attention head will be replenished.

[0062] For illustrative purposes, please refer to the following: Figure 3 , Figure 3 This is a comparative schematic diagram of the selected feature block selection method provided in an exemplary embodiment of this application, such as... Figure 3 As shown, for query blocks at the same position represented by each query feature, in the completely independent head selection feature block selection method 310, multiple selected feature blocks for feature processing are determined sequentially from multiple feature blocks by attention heads; that is, the process of different attention heads determining selected feature blocks is completely independent, and the multiple first selected feature blocks selected by the first attention head, the multiple second selected feature blocks selected by the second attention head, and the multiple third selected feature blocks selected by the third attention head have no dependency relationship with each other.

[0063] like Figure 3 As shown, in the fully shared head selected feature block selection method 320, multiple shared feature blocks for feature processing are jointly determined from multiple feature blocks; that is, the process of different attention heads determining the selected feature blocks is completely shared, and the multiple shared feature blocks 321 selected by the first attention head, the second attention head and the third attention head are the same, wherein the multiple shared feature blocks 321 include multiple different feature blocks.

[0064] like Figure 3As shown, in the shared feature block + private feature block selection method 330, multiple shared feature blocks are first determined from multiple feature blocks, and then multiple private feature blocks are determined from each of the multiple feature blocks by attention heads. Feature processing is then performed based on the shared feature blocks and the private feature blocks. That is, the process of different attention heads determining the selected feature blocks is partially shared. The shared feature blocks selected by the first attention head, the second attention head, and the third attention head are the same. When there is more than one shared feature block, the shared feature blocks include multiple different feature blocks. The process of different attention heads determining the selected feature blocks also includes independently selecting private feature blocks. At least one first private feature block 331 selected by the first attention head, at least one second private feature block 332 selected by the second attention head, and at least one third private feature block 333 selected by the third attention head have no dependency relationship with each other.

[0065] Step 240: Based on each query block corresponding to each attention head, query the shared feature block and the private feature block respectively to obtain the attention feature representation.

[0066] For each attention head, the query feature representation (Q), key feature representation (K), and value feature representation (V) are obtained. The selected key-value block is obtained based on the routing information corresponding to the shared feature block and the private feature block, including the key feature block obtained by partitioning the key feature representation and the value feature block obtained by partitioning the value feature representation.

[0067] For each pair of query and key feature blocks, an attention score is calculated on a feature block basis to obtain the original attention score matrix. Illustratively, the attention score includes, but is not limited to, the QK dot product or its scaling value. For each query block, the attention score vector is normalized to probability weights using a softmax function. Based on these probability weights, a weighted sum is applied to all value feature blocks to obtain the attention feature representation for that query block location. Illustratively, the formula for obtaining the attention feature representation by querying selected feature blocks based on each query block using the attention mechanism is shown below.

[0068]

[0069] in, Let Q be the dimension of the key vector K, Q be the query vector (query block), K be the key vector (key feature block), and V be the value vector (value feature block). The softmax function is used to convert any real number vector into a probability distribution vector. This involves performing an exponentiation operation on each element in the input feature representation to convert the vector elements into positive numbers; dividing the result of the exponentiation operation corresponding to each vector element by the sum of the exponents of the input vector elements, so that the sum of the processed vector elements is 1, and each vector element is between 0 and 1.

[0070] Taking video generation as an example, a pre-trained video generation model, starting with the input feature representation in the compressed latent space, progressively outputs the generated results through a diffusion model or an autoregressive model. The generated results are then input into the decoding module of the video generation model. The decoding module maps the generated results back to the original space corresponding to the prompt information, obtaining the video data. Schematically, attention feature representation is used as the initial input to the decoding module, and autoregressive decoding prediction yields the predicted video data. The video data includes, but is not limited to, continuous video frame sequences.

[0071] In some embodiments, the video generation model outputs the corresponding generation result, i.e., video data, through an autoregressive generation mechanism based on the prompt information. The video generation process includes, but is not limited to, at least one of the following steps.

[0072] 1. Through feature processing, the prompt information is converted into an input feature representation that can be processed by the video generation model.

[0073] 2. The encoded input feature representation is input into the Transformer architecture of the video generation model. Through a multi-layer self-attention mechanism feedforward network, the input feature representation is mapped to the feature space corresponding to each of the multiple attention heads, resulting in query feature representations and key-value feature representations corresponding to each attention head. The query feature representation is divided into multiple query blocks, and the key-value feature representation is divided into multiple key-value blocks. For the same attention head, based on the pairwise relationships between multiple query blocks and multiple key-value blocks, the shared feature blocks shared by query blocks at the same position are determined from the multiple key-value blocks, and the private feature blocks corresponding to the query blocks of each attention head are determined from the multiple key-value blocks. Based on each query block corresponding to each attention head, the shared feature blocks and private feature blocks are queried respectively to obtain the attention feature representation.

[0074] 3. The video generation model is based on attention feature representation and predicts the probability distribution of the next feature unit for each feature unit. At each prediction step, the feature unit with the highest probability is selected, or the target feature unit is selected as the newly generated feature unit according to the sampling strategy. The newly generated feature unit is appended to the attention feature representation as the input for the next step. The process is iterated until a terminator is generated, such as the end of sequence (EOS), or the maximum length limit is reached, resulting in a generated sequence composed of newly generated feature units arranged in order.

[0075] 4. The generated sequence is restored to video data through the decoding module.

[0076] It is worth noting that the above video generation process is only an illustrative example, and the specific implementation of the attention feature representation subsequent processing in this application embodiment is not limited.

[0077] In summary, the feature processing method based on multi-head attention networks in this application maps the input data to the feature spaces corresponding to multiple attention heads in the attention group, and divides them into multiple query blocks and multiple key-value blocks to provide processing objects for the feature processing of the block sparse attention mechanism. For each position of the query block in the different query feature representations corresponding to different attention heads, a key-value block indicating the focus of the query block is selected from the multiple key-value blocks. This includes a shared feature block shared by multiple attention heads at that position, and a private feature block corresponding to each attention head. The shared feature block simplifies the routing process of attention calculation and improves the efficiency of feature processing, while the private feature block is used to retain the differentiated attention capabilities of different attention heads to the input feature representation, improving the accuracy of feature processing, thereby improving the efficiency and accuracy of subsequent calculations based on attention feature representations.

[0078] In some embodiments, the number of multiple attention heads is H, where H is an integer greater than 1. The key-value block includes a key feature block and a value feature block corresponding to the key feature block. For the query feature representation corresponding to each attention head, the process of determining the selected feature block from multiple key-value blocks is divided into determining the shared feature block of the common attention region and determining the private feature block of the characteristic attention region. Based on the selected feature block, feature processing can be performed on the selected key-value block based on the query block through a multi-attention mechanism and a block sparse attention mechanism, thereby performing at least one model task such as content generation, data recognition, and task prediction based on the result of feature processing, i.e., attention feature representation.

[0079] Please refer to Figure 4 This illustrates a flowchart of a feature processing method based on a multi-head attention network in a model, provided in another exemplary embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method is executed by either the terminal 120 or the first server 140 shown, or it is executed jointly by the terminal and the server through interaction. In this embodiment, the method is taken as being executed by the terminal as an example. Figure 4 As shown, step 230 above can also be implemented as at least one of steps 231 to 235 below.

[0080] Step 231: Generate a block-level representation for each attention head based on the query block and key feature block.

[0081] The input feature representation includes multiple feature blocks, and each feature block includes multiple feature units; wherein, obtaining the block-level representation of each feature block includes, but is not limited to, obtaining the block-level representation of each query block and key-value block.

[0082] In some embodiments, a plurality of query feature units in a query block are determined, and a plurality of key feature units in a key feature block are determined; aggregation processing is performed on the plurality of query feature units to obtain a block-level query feature representation corresponding to the query block, and aggregation processing is performed on the plurality of key feature units to obtain a block-level key feature representation corresponding to the key feature block; a block-level representation is determined based on the block-level query feature representation and the block-level key feature representation. The block-level representation is used as a basis for deciding and selecting a feature block, which reduces routing overhead so as to improve feature processing efficiency, thereby improving video generation efficiency.

[0083] Wherein, the aggregation processing for obtaining the block-level query feature representation of the query block and / or the block-level key-value feature representation of the key-value block includes but is not limited to at least one of the following lightweight aggregation manners: 1. Perform averaging processing or weighted averaging processing on the features of the feature units within the feature block (the query block and / or the key-value block). 2. Extract representative feature units within the feature block.

[0084] It is worth noting that the block-level representation may be used only in the selection process of selecting the feature block, that is, the routing process, and the final attention calculation may still use the feature representations of the original feature units within the selected feature block to complete feature processing. Illustratively, in the case that the attention head index is h, the query block index is q, and the key-value block index is b, the block-level representation may be denoted as S(h,q,b), that is, the block-level representation of the q-th query block of the h-th attention head for the b-th key-value block is S(h,q,b), where h, q, and b are all positive integers, h≤H, q,b≤N, and N is the total number of a plurality of feature blocks corresponding to the input feature representation.

[0085] Step 232: For query blocks at the same position divided by each query feature representation, determine a sharing indicator corresponding to each of the plurality of key feature blocks based on the block-level representation.

[0086] The sharing indicator is used to indicate the common attention degree of the query block to the key feature block from the attention perspectives respectively indicated by the plurality of attention heads. In the case that a shared feature block is determined from the plurality of feature blocks based on the sharing indicator, the plurality of attention heads corresponding to the sharing indicator take the shared feature block as a shared feature processing object, that is, for the query blocks at the same position divided by the query feature representations respectively corresponding to the plurality of attention heads, the plurality of selected feature blocks to be queried by each query block include the shared feature block.

[0087] In some embodiments, the sharing indicator is a group-level sharing indicator. All the attention heads are divided into M attention groups, where M is a positive integer, M<H, a plurality of attention heads in the attention group share the aforementioned shared feature block for the query blocks at the same position divided by each query feature representation, and different attention groups correspond to different shared feature blocks; based on the block-level representation, the sharing indicator of each of the plurality of feature blocks corresponding to the i-th attention group is obtained, where i is a positive integer, i≤M.

[0088] When all attention heads are divided into one or more attention groups, attention heads within the same attention group share a portion of selected feature blocks through shared feature blocks, and each attention head in the same attention group retains a small number of private feature blocks. It is worth noting that the grouping of all attention heads can occur before obtaining the block-level representation, where the block-level representation can be implemented as the block-level feature representation corresponding to the key feature block.

[0089] Schematic, with the group index of the attention group being g, the query block index being q, and the key value block index being b, the group-level sharing index of the attention group is P(g,q,b)=Agg(S), where the block level is represented by S=S(h,q,b), g, q, and b are all positive integers, g≤M, q,b≤N, and N is the total number of input feature blocks corresponding to multiple feature blocks.

[0090] Schematic, the grouping methods for all attention heads include, but are not limited to, at least one of the following: grouping attention heads according to their index adjacency, such as grouping them according to consecutive attention heads; grouping attention heads that have an existing key-value sharing relationship with each other into the same attention group; grouping attention heads based on the query attention head in the grouped query structure; randomly dividing all attention heads into at least one attention group based on a preset group size; grouping based on the similarity between the response values ​​corresponding to attention heads; grouping attention heads based on their attention tendency determined by the weight matrix, where the attention tendency determines the semantic functional similarity between different attention heads; and grouping based on hardware execution convenience, such as dividing attention heads with consecutive memory access into the same attention group.

[0091] Schematic, when attention heads are grouped based on their key-value sharing relationships, multiple query attention heads within the same attention group share at least one shared feature block. Specifically, for each query attention block, a shared metric is calculated for the shared feature block on a per-attention-group basis, and a shared feature block is selected to cover the main regions that multiple query attention heads within the attention group commonly focus on. Correspondingly, a block-level independent metric is calculated for each query attention head, and a private feature block is selected from key-value blocks that are not covered by the shared feature block.

[0092] The set of selected feature blocks accessed by each query block of each attention head consists of shared feature blocks and private feature blocks for each query attention head. The sparse attention execution module loads the key-value blocks corresponding to each attention head based on this set of selected feature blocks and performs precise attention calculations within the selected feature blocks, thus naturally integrating the feature processing process with grouped query attention or multi-query attention structures. Since most selected feature blocks are still shared within the attention group, the improved memory access and indexing efficiency brought by the shared key-value structure can be preserved. Furthermore, the private feature blocks corresponding to each query attention head alleviate the problem of lost head differences caused by the fully shared head routing method, improving the accuracy of feature processing.

[0093] Without completely reverting to a fully independent header routing approach, for query blocks at the same position in each query feature representation, the shared metrics of each key feature block corresponding to multiple attention heads in the same attention group are obtained. The shared feature blocks are then determined through the shared metrics, allowing multiple attention heads in the same attention group to share most of the selected feature blocks, while allowing each attention head to retain a small number of private feature blocks. This reduces routing overhead in the feature processing process and improves the efficiency and accuracy of feature processing.

[0094] In some embodiments, when the input feature representation is divided into N feature blocks, for the i-th attention group, based on the block-level representation, the shared metrics of the multiple key-value blocks corresponding to the i-th attention group are obtained, including: for each query block at the same position, based on the block-level representation, the shared metrics of the j-th key-value block corresponding to the i-th attention group are obtained, where j is a positive integer, j≤N, and N is a positive integer.

[0095] In some embodiments, for query blocks at the same position divided by each query feature representation, the process of obtaining shared metrics includes, but is not limited to, at least one of the following.

[0096] 1. For query blocks at the same position in each query feature representation, perform block-level representation aggregation processing on the key feature blocks at the same position in each key value feature representation to obtain response aggregation values ​​corresponding to key feature blocks at multiple positions. Shared metrics include response aggregation values.

[0097] The block-level response value can be a block-level representation or a block-level attention score. The attention score is the dot product of the query feature representation and the key feature representation, or a variant thereof. Illustratively, the shared routing module calculates the attention score of each query block corresponding to each attention head on each key feature block based on multiple attention heads within the attention group. The attention scores corresponding to multiple attention heads are then aggregated, for example, by averaging or maximizing, to obtain the aggregated response value.

[0098] Schematic, when the input feature representation is divided into N feature blocks, for each query block at the same position, based on the block-level representation, the block-level response value, i.e., the block-level representation, of the j-th key-value block corresponding to the multiple attention heads in the i-th attention group is obtained, where j is a positive integer, j≤N, and N is a positive integer; based on the block-level response value of the j-th key-value block corresponding to the multiple attention heads in the i-th attention group, the response aggregation value of the j-th key-value block corresponding to the i-th attention group is obtained, thus obtaining the shared index of the j-th key-value block corresponding to the i-th attention group.

[0099] 2. For query blocks at the same position in each query feature representation, determine the voting results corresponding to multiple key feature blocks based on the block-level representation, and share the voting results as indicators.

[0100] The voting result indicates the degree of attention paid to the key-value block by query blocks at the same position within the attention group. By aggregating the response values ​​of each query block corresponding to each attention head to the key-value block, at least one shared candidate block is selected, and a vote is cast for that key-value block (e.g., adding a voting flag or updating the flag value). The voting result is obtained after the voting process for multiple attention heads within the attention group is completed. Optionally, the voting result includes, but is not limited to, the sum of votes or voting scores.

[0101] Schematic, when the input feature representation is divided into N feature blocks, for each query block at the same position, based on the block-level representation, the voting result of the j-th key-value block corresponding to the i-th attention group is obtained. The voting result is used to indicate the degree of attention of each query block in the i-th attention group to the j-th key-value block, where j is a positive integer, j≤N, and N is a positive integer.

[0102] 3. For query blocks at the same position in each query feature representation, determine the similarity scores corresponding to multiple key feature blocks based on the block-level representation, and share indicators including similarity scores.

[0103] The similarity score indicates the degree of similarity between the key feature blocks corresponding to the feature blocks in the attention heads of the attention group. The similarity score includes, but is not limited to, at least one of the following: cosine similarity, Kullback-Leibler divergence (KL) and cross-correlation coefficient between the key feature blocks corresponding to any two attention heads.

[0104] Schematic, when the input feature representation is divided into N feature blocks, for each query block at the same position, based on the block-level representation, the similarity score of the j-th key value block corresponding to the i-th attention group is obtained. The similarity score is used to indicate the degree of similarity of the j-th key value block from the perspective of the query block corresponding to each of the multiple attention heads in the i-th attention group, where j is a positive integer, j≤N, and N is a positive integer.

[0105] 4. For query blocks at the same position in each query feature representation, obtain the shared metrics that have been obtained in the historical time period for each of the multiple key-value blocks corresponding to the i-th attention group, and use them as the current shared metrics; illustratively, when feature processing is repeatedly performed on the input feature representation through the attention head, the current shared metrics can reuse the previously obtained shared metrics.

[0106] It is worth noting that the above-mentioned implementation methods of shared indicators (such as aggregated response values, voting results, etc.) are merely illustrative examples, and this application does not limit the specific methods for obtaining shared indicators. When the process of obtaining shared indicators does not rely on model parameter training, it can be directly applied to the feature processing stage of deep learning models as an improved method for training-free inference, resulting in high feature processing efficiency and accuracy. Optionally, the process of obtaining shared indicators also includes generating shared indicators based on block-level representation through a pre-trained indicator model.

[0107] Step 233: Determine the shared feature blocks according to the shared indicators.

[0108] In some embodiments, shared feature blocks whose shared metrics meet a first condition are determined from a plurality of key-value blocks. Optionally, the first condition includes the shared feature block being located first among the plurality of key-value blocks in descending order of shared metrics. indivual, is a positive integer.

[0109] Based on the shared metrics corresponding to each key-value block, at least one shared feature block is selected for each query block. The shared feature block is shared by query blocks at the same location corresponding to multiple attention heads within the same attention group; that is, the routing information of the shared feature block is shared by multiple attention heads. The shared feature block is used to cover the main body region, main motion path, important background context, or other feature regions that are jointly concerned by multiple attention heads in the input feature representation. Sharing the shared feature block within the group reduces the overhead of redundant index construction and redundant key-value block selection.

[0110] In some embodiments, the first condition includes a budget for the number of shared feature blocks, such as a first quantity threshold. Shared feature blocks are acquired when the determined number of shared feature blocks reaches the first quantity threshold.

[0111] Determining the shared feature blocks whose sharing metrics meet the first condition from the key-value blocks respectively corresponding to a plurality of attention heads includes, but is not limited to, at least one of the following.

[0112] 1. Sorting the plurality of key-value blocks in descending order according to the sharing metrics of the key feature blocks to obtain a shared candidate block list; determining the top ones based on the shared candidate list key-value blocks to obtain the shared feature blocks, wherein is the first quantity threshold, is a positive integer; in the case where the input feature representation is divided into N feature blocks, < N, where N is a positive integer. Illustratively, the shared feature blocks are determined by TopK(P, ), where P is the sharing metric corresponding to each feature block.

[0113] 2. Acquiring a sharing threshold corresponding to the first condition, and determining at least one shared feature block of which the sharing metric of the key feature block among the plurality of key-value blocks reaches the sharing threshold, wherein the sharing threshold may be preset or dynamically determined; optionally, in the case that the number of the shared feature blocks reaches the first quantity threshold, stopping acquiring the key-value blocks whose sharing metrics reach the sharing threshold as the shared feature blocks.

[0114] 3. Acquiring key-value blocks near the multi-dimensional position of the query block as the shared feature blocks.

[0115] It should be noted that the foregoing manner of determining the shared feature blocks based on the sharing metrics is only an illustrative example, and the present application does not limit the implementation manner of the first condition.

[0116] For the query blocks at each same position, the shared feature blocks are used to cover the main areas jointly focused by multiple attention heads in the group, so as to maintain the processing efficiency of shared routing; the block indexes of the shared feature blocks are shared, and the shared feature blocks are loaded and share feature processing scheduling, which reduces routing overhead and execution overhead compared with the routing manner of completely independent heads.

[0117] In some embodiments, the proxy index predicts the importance degree of key-value blocks through a lightweight surrogate metric (such as a sharing metric) during the routing process, and generates the first routing information corresponding to the shared feature block accordingly, that is, obtaining the shared feature block; the proxy index is used to guide sparse attention calculation, and the generation overhead is much lower than that of full attention calculation. Optionally, with a feature processing structure that combines a training-free proxy scorer or a block-level proxy indexer, the sharing metric can be generated by the proxy scorer. Illustratively, the sharing metric I_proxy(g,q,b) is obtained, where g is the group index of the attention head group, q is the query block index, and b is the key-value block index; the block-level proxy indexer generates the first routing information of the shared feature block; or generates shared candidate blocks, and then further determines the shared feature block from the shared candidate blocks. Exemplarily, the first routing information of the shared feature block corresponding to the h-th attention head is Ω_shared = TopK(I_proxy).

[0118] Step 234: For each query block under the h-th attention head, determine the independent metric corresponding to each of the plurality of key feature blocks based on the block-level representation.

[0119] The independent metric is used to indicate the degree of attention that the query block pays to the characteristics of the key feature block from the attention perspective indicated by the h-th attention head, where h is a positive integer and 1<h≤H. In the case where private feature blocks are determined from a plurality of key-value blocks based on the independent metric, the plurality of selected feature blocks of each attention head each include a private feature block. The private feature block is a key-value block selected according to the independence of each attention head itself after excluding the shared feature block. In the case where the block-level representation is obtained, the independent metric of each key feature block is obtained for each attention head, and the routing preference of each attention head is retained.

[0120] In some embodiments, the process of obtaining the independent metric of the key feature block for each query block includes but is not limited to at least one of the following.

[0121] 1. For each query block under the h-th attention head, the block-level representation is determined as the independent metric. Based on the block-level representation, the block-level response value of each of the plurality of key feature blocks corresponding to each query block corresponding to the h-th attention head is obtained.

[0122] 2. For each query block under the h-th attention head, a difference score corresponding to each of the plurality of key feature blocks is determined based on the block-level representation, and the difference score is used to indicate the degree of difference between the block-level representation and the sharing metric. Based on the block-level representation, the difference score of each of the plurality of key feature blocks corresponding to each query block corresponding to the h-th attention head is obtained.

[0123] 3. For each query block under the h-th attention head, determine the local feature scores corresponding to multiple key feature blocks based on the block-level representation. The local feature scores are used to indicate the degree of attention the query block pays to the key feature blocks in terms of local details. Based on the block-level representation, obtain the local feature scores of each of the multiple key feature blocks corresponding to each query block under the h-th attention head. The local feature scores are used to indicate the degree of attention the feature blocks receive in terms of local details, which include at least one of local texture information, motion boundary information, and small-scale target information.

[0124] It is worth noting that the implementation methods of the aforementioned independent indicators are merely illustrative examples, and this application does not limit the specific implementation methods of the independent indicators corresponding to each attention head. Optionally, the selection of private feature blocks can be limited from multiple perspectives through independent indicators.

[0125] By selecting private feature blocks to preserve unique feature representations for each attention head, the freedom in selecting feature blocks is increased. Private feature blocks are used to fill in important regions unique to each attention head, reducing the risk of missing details due to shared routing. Without significantly increasing the final number of selected feature blocks, the coverage quality of key-value feature representations by the selected feature blocks relevant to the query block is improved. Compared to simply increasing the budget for the number of selected feature blocks, allocating a portion of the budget for the selected feature blocks to the private feature blocks corresponding to each attention head more specifically fills in missing regions of feature representation.

[0126] Step 235: Determine the private feature block corresponding to the h-th attention head according to the independent index.

[0127] In some embodiments, for the h-th attention head, private feature blocks whose independent indices satisfy a second condition are determined from a plurality of key-value blocks. Optionally, the second condition includes that the private feature block ranks first among the plurality of key-value blocks in descending order of their independent indices. indivual, is a positive integer.

[0128] Private feature blocks differ from shared feature blocks. Based on the independent metrics corresponding to each key feature block, the private supplementary routing module first selects private feature blocks from the key-value blocks that are not covered by shared feature blocks for each attention head, thereby replenishing the important feature regions unique to each attention head. By identifying private feature blocks whose independent metrics of the key feature blocks meet the second condition, each attention head obtains a small number of additional private feature blocks that match its own focus characteristics, in addition to the shared feature blocks.

[0129] Schematically, in the process of determining the private feature blocks according to the independent index of each key feature block of a single attention head itself, if the current candidate key-value block already belongs to a shared feature block, the candidate key-value block may be skipped, and the judgment on whether the independent index of the next candidate key-value block meets the second condition is continued, so as to determine the private feature block, preventing the budget for the number of selected feature blocks from being repeatedly occupied by the shared feature blocks.

[0130] In some embodiments, the second condition includes a budget for the number of private feature blocks, such as a second quantity threshold. When the number of determined private feature blocks reaches the second quantity threshold, the private feature blocks are acquired. On the premise that the number of finally selected feature blocks is fixed or approximately fixed, the effective coverage of selected feature block selection is improved.

[0131] Determining private feature blocks whose independent indexes of key feature blocks meet the second condition from a plurality of key-value blocks includes but is not limited to at least one of the following.

[0132] 1. For the h-th attention head, sort the plurality of key-value blocks in descending order according to the independent indexes of the key feature blocks, to obtain a private candidate block list; determine the top from the private candidate list key-value blocks to obtain private feature blocks, wherein is the second quantity threshold is a positive integer; in the case where the input feature representation is divided into N feature blocks, and shared feature blocks are determined <N- , N, are positive integers <N. Schematically, the shared feature blocks are determined by TopK( , ), wherein are the independent indexes respectively corresponding to the plurality of key-value blocks excluding the shared feature blocks corresponding to the h-th attention head.

[0133] 2. An independent threshold corresponding to the second condition is acquired, and at least one private feature block whose independent index reaches the independent threshold is determined from the plurality of key-value blocks, wherein the independent threshold may be preset or dynamically determined; optionally, when the number of private feature blocks reaches the second quantity threshold, the acquisition of key-value blocks whose independent indexes reach the independent threshold as private feature blocks is stopped.

[0134] 3. Key-value blocks near the multi-dimensional position of the query block are acquired as private feature blocks.

[0135] It should be noted that the above manner of determining private feature blocks based on the independent index is only a schematic example, and the present application does not limit the implementation manner of the second condition.

[0136] For illustrative purposes, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram illustrating the selection method between shared feature blocks and private feature blocks provided in an exemplary embodiment of this application, as shown below. Figure 5 As shown, taking the implementation of the input feature representation as a spatiotemporal three-dimensional feature representation corresponding to video data as an example, the input feature representation includes a time dimension for indicating the time frame of the video data and a spatial dimension for indicating the two-dimensional image of the video data. The spatial dimension includes a height dimension and a width dimension. The input feature representation is mapped to the feature space corresponding to each of the multiple attention heads to obtain the feature mapping results of the input feature representation in the feature space corresponding to each of the multiple attention heads. The feature mapping result of each attention head includes a query feature representation and a key-value feature representation 500. For each attention head, the query feature representation is divided into multiple query blocks, and the key-value feature representation 500 is divided into multiple key-value blocks.

[0137] like Figure 5 As shown, for query blocks at the same position divided by each query feature representation, based on the block-level representations corresponding to multiple key feature blocks, a shared feature block 510 is determined from multiple key-value blocks, which are shared by multiple attention heads corresponding to multiple query blocks at the same position. The private feature blocks corresponding to the query blocks of each attention head are also determined. Different attention heads may focus on the same or different aspects, and correspondingly, the private feature blocks corresponding to different attention heads may be the same or different. The private feature blocks corresponding to each query block of each attention head include, but are not limited to, the private feature block 520 of the first attention head, the private feature block 530 of the second attention head, and the private feature block 540 of the third attention head. Optionally, the shared feature block 510 is a key-value block shared by multiple attention heads within one or more attention groups.

[0138] like Figure 5 As shown, when the shared feature block 510 and private feature block corresponding to each attention head are determined, the multiple key-value blocks also include optional neighborhood feature blocks 550. The optional neighborhood feature blocks 550 can be key-value blocks that were not selected during the initial selection of selected feature blocks. When merging the shared feature block 510 and private feature block corresponding to each attention head, if the number of selected feature blocks corresponding to an attention head does not reach the total threshold, supplementary feature blocks can be determined from the optional neighborhood feature blocks 550 to improve the routing selection of the attention head.

[0139] In summary, the feature processing method based on multi-head attention networks in this application maps the input data to the feature spaces corresponding to multiple attention heads in the attention group, and divides them into multiple query blocks and multiple key-value blocks to provide processing objects for the feature processing of the block sparse attention mechanism. For each position of the query block in the different query feature representations corresponding to different attention heads, a key-value block indicating the focus of the query block is selected from the multiple key-value blocks. This includes a shared feature block shared by multiple attention heads at that position, and a private feature block corresponding to each attention head. The shared feature block simplifies the routing process of attention calculation and improves the efficiency of feature processing, while the private feature block is used to retain the differentiated attention capabilities of different attention heads to the input feature representation, improving the accuracy of feature processing, thereby improving the efficiency and accuracy of subsequent calculations based on attention feature representations.

[0140] The method provided in this embodiment, while maintaining the efficiency of shared routing through shared feature blocks, retains a small number of private feature selection degrees of freedom for each attention head through private feature blocks. With the total budget remaining constant or approximately constant, it improves the coverage of multi-interest targets, local textures, and motion details by the shared sparse attention mechanism, reduces the routing overhead and indexing costs of the fully attention-head-by-attention routing method, and reduces the risk of quality degradation in task results (such as video generation results). By retaining a small number of privately selected feature blocks, it improves the coverage of local detail features and cross-frame correlated feature regions during feature processing, thereby reducing the occurrence of missed feature blocks and improving the accuracy of video generation.

[0141] Please refer to Figure 6 This illustrates a flowchart of a feature processing method based on a multi-head attention network in a model, provided in yet another exemplary embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method is executed by either the terminal 120 or the first server 140 shown, or it is executed jointly by the terminal and the server through interaction. In this embodiment, the method is taken as being executed by the terminal as an example. Figure 6 As shown, step 240 above can also be implemented as at least one of steps 242 to 248 below.

[0142] Step 242: For the p-th query block corresponding to the h-th attention head, determine the first routing information corresponding to the shared feature block and the second routing information corresponding to the private feature block.

[0143] Where p is a positive integer.

[0144] In some embodiments, before determining shared feature blocks from key-value blocks corresponding to multiple attention heads and determining private feature blocks corresponding to each attention head from key-value blocks at the same position for each query feature representation, the method further includes: determining a total number threshold and determining a quantity ratio coefficient; the total number threshold is used to constrain the total number of shared feature blocks and private feature blocks determined for a single attention head and a single query block, and the quantity ratio coefficient is used to constrain the quantity ratio relationship between shared feature blocks and private feature blocks determined for a single attention head and a single query block; a first quantity threshold for shared feature blocks and a second quantity threshold for private feature blocks are determined based on the total number threshold and the quantity ratio coefficient, the first quantity threshold is used to constrain the number of shared feature blocks determined for a single attention head and a single query block, and the second quantity threshold is used to constrain the number of private feature blocks determined for a single attention head and a single query block.

[0145] The total number threshold and / or the number ratio coefficient are quantity budget configurations set before the routing process. These can be pre-fixed configuration parameters or dynamically adaptively determined configuration parameters based on the input feature representation. When determining the selected feature blocks corresponding to each query block for each attention head, the specific quantity budget for the selected feature blocks includes a first quantity threshold and a second quantity threshold. By constraining the number of selected feature blocks through the total number threshold, and under the premise of fixing or approximately fixing the quantity budget corresponding to each attention head, the quantity budget is divided into a shared portion and a head-private portion. This ensures that most selected feature blocks are shared within the attention group, and a small number of selected feature blocks are used to indicate the recovery of semantic details related to the attention head.

[0146] In some embodiments, obtaining the quantity ratio coefficient between shared feature blocks and private feature blocks includes at least one of the following: determining the quantity ratio coefficient based on the difference between key-value feature representations among multiple attention heads; determining the quantity ratio coefficient based on the network layer position corresponding to the input feature representation; and determining the quantity ratio coefficient based on the complexity of feature processing requirements corresponding to the input data.

[0147] That is, in the method of dynamically and adaptively determining the quantity budget configuration based on the input feature representation, the configuration basis includes, but is not limited to, at least one of the following factors: 1. The degree of difference between key-value feature representations among multiple attention heads, used to indicate the attention difference between attention heads: the degree of divergence between input feature representation heads; at least one of the response value entropy, overlap rate, and boundary error of key feature blocks. 2. The network layer position corresponding to the input feature representation: including but not limited to the number of network layers of the deep learning model; the time step parameters of the deep learning model; the generation stage corresponding to feature processing, etc. 3. The complexity of feature processing requirements: the complexity of the current content to be processed indicated by the input data; taking the task to be executed as a video generation model, and the deep learning model implementation as a video generation model as an example, the complexity of feature processing requirements is the complexity of the current generated content indicated by the prompt information, the feature processing requirements correspond to the video generation requirements, and the complexity of video generation requirements includes video resolution. 4. Hardware load.

[0148] It is worth noting that, based on the configuration, at least one of the total threshold and the quantity ratio coefficient can be adjusted. Optionally, the total thresholds corresponding to different attention points may be the same or different.

[0149] The inter-head divergence of the input feature representation refers to the degree of difference in the attention distribution or feature processing output results produced by different attention heads for the same feature representation within the same feature processing layer. It is used to quantify the diversity of attention patterns of each attention head. Optionally, the inter-head divergence can be implemented as at least one of KL divergence, cosine similarity, or variance. Illustratively, a higher inter-head divergence indicates that each attention head focuses on different features or regions. The system can also set an adaptive budget mechanism.

[0150] For example, when the total number threshold for each attention head is determined to be 16 based on the configuration and the quantity ratio coefficient is 3, sixteen selected feature blocks are accessed during feature processing, including twelve shared feature blocks and four private feature blocks; the twelve shared feature blocks are used by all attention heads in the attention group, and the four private feature blocks are used by each attention head separately.

[0151] To illustrate, if multiple attention heads within an attention group give high response values ​​to the same key-value block for query blocks at the same location, it indicates that the attention heads in that attention group focus on a relatively consistent area. In this case, the number of shared feature blocks can be increased, while the number of private feature blocks can be decreased. Conversely, if different attention heads within an attention group give high response values ​​to different feature blocks, it indicates a clear division of attention among the attention heads in that group. In this case, the number of shared feature blocks can be decreased, while the number of private feature blocks can be increased.

[0152] For example, when the total threshold is kept at 16, if the attention areas of the attention heads within an attention group are relatively consistent, the quantity ratio coefficient can be set to 7, that is, the first quantity threshold is 14 and the second quantity threshold is 2, and fourteen shared feature blocks and two private feature blocks are used for feature processing. When the attention areas of the attention heads within a group are significantly different, the quantity ratio coefficient can be set to 5 / 3, that is, the first quantity threshold is 10 and the second quantity threshold is 6, and ten shared feature blocks and six private feature blocks are used for querying.

[0153] It is worth noting that the above configuration criteria include, but are not limited to, the influencing factors corresponding to the quantity ratio coefficient and the influencing factors corresponding to the total number threshold. The reasons for the influence of each configuration criterion on the quantity budget configuration may be the same or different. By expanding the different acquisition methods of the quantity ratio coefficient between shared feature blocks and private feature blocks, an adaptive balance is achieved between computational efficiency and feature processing quality based on factors such as the complexity of feature processing requirements and the degree of attention difference among multiple attention heads. This improves the adaptability between the quantity budget configuration of the selected feature block and the current input feature representation and the focus of each attention head.

[0154] Step 244: Merge the first routing information and the second routing information to obtain the selected key-value block routing information for the p-th query block corresponding to the h-th attention head.

[0155] The route merging module merges the shared and private feature blocks corresponding to each attention head to form the final sparse route set for each attention head, which is to obtain the set of selected feature blocks finally used by each query block and each attention head.

[0156] Optionally, during the merging process, at least one of the following processes is performed on shared and private feature blocks: 1. Sorting: Shared and private feature blocks are uniquely identified (e.g., one-dimensional feature block numbers arranged in row-major order) according to their multi-dimensional positions in the multiple key-value blocks in the key-value feature representation and then sorted. 2. Deduplication: Key-value blocks with the same multi-dimensional positions are excluded from the sorted shared and private feature blocks. 3. Completion: If the number of selected feature blocks after deduplication does not reach the target number (total threshold), the feature processing system based on the multi-head attention network in the model can complete the completion from the backup candidate pool according to the order of independent indicators; wherein, the backup candidate pool can be implemented as a set of unselected key-value blocks determined during the process of determining private feature blocks, such as a list of private candidate blocks sorted according to independent indicators; if the number of selected feature blocks reaches the target number, pruning can be performed according to at least one constraint such as response value, spatiotemporal coverage, or execution cost of feature processing. 4. Format conversion: The selected feature blocks are unified into the feature representation format required for attention calculation, including but not limited to unified block identifiers.

[0157] It is worth noting that the above-described method of merging selected feature blocks is only an illustrative example. This application does not limit the process and implementation of obtaining selected feature blocks. In illustrative terms, the specific operation and process of merging can be replaced. When merging shared feature blocks and private feature blocks, deduplication can be performed directly, or the blocks can be sorted uniformly first and then deduplicated.

[0158] Optionally, the feature processing method provided in this application can be used in combination with other sparse attention enhancement techniques. Indicatively, sparse attention enhancement techniques include, but are not limited to, deduplication of selected feature blocks; diversity preservation, such as adjusting selected feature blocks based on spatiotemporal coverage; local window supplementation, such as forcibly retaining at least one key-value block adjacent to each query block for querying, to ensure the local context is reflected in the feature processing; setting global anchor blocks as the queried key-value blocks, such as the beginning or end key-value blocks of the key-value feature representation; low-bit quantization, such as reducing the activation values ​​or weight matrices in attention calculation from 16-bit floating-point (FP16) or 32-bit floating-point (FP32) data formats to low-precision data formats such as 8-bit integer (INT8), reducing storage space occupation and computational load; and developing architecture-optimized kernels for specific hardware devices to accelerate matrix operations corresponding to block sparse attention using hardware characteristics, thereby further improving feature processing efficiency and / or processing quality.

[0159] In some embodiments, for query blocks at the same position in each query feature representation, when a shared feature block is determined from the key-value blocks corresponding to multiple attention heads, first routing information corresponding to each query block corresponding to each attention head is obtained. The first routing information is used to indicate the position of the shared feature block in the key-value feature representation. When a private feature block corresponding to each attention head is determined from the key-value blocks corresponding to each attention head, second routing information corresponding to each query block corresponding to each attention head is obtained. The second routing information is used to indicate the position of the private feature block in the key-value feature representation. Based on the first routing information and the second routing information, routing information for a selected key-value block is obtained. Based on the routing information for the selected key-value block, the attention feature representation is obtained by querying the selected key-value block using an attention mechanism.

[0160] Optionally, the implementation of the first routing information, the second routing information, and the routing information of the selected key-value block can be consistent, and the scope of intuitive attention calculation can be used by determining the location information of the key-value block in multiple dimensions.

[0161] The selected key-value block routing information includes, but is not limited to, at least one of the following implementations: 1. The block mask is implemented as a binary matrix, used to explicitly specify the feature blocks to be computed during the block sparse attention computation. Indicatively, the matrix shape corresponding to the block mask is [N, N], with a total of N... 2 There are 1 binary element, where N is the number of feature blocks. The binary element at (j1, j2) is 1 to indicate that the j1-th query block and the j2-th key-value block are selected for attention calculation. That is, the selected feature block corresponding to the j1-th query block includes the j2-th key-value block, where j1 and j2 are positive integers, and j1, j2 ≤ N. A binary element of 0 indicates that the key-value block corresponding to that element is masked for attention calculation. 2. The index is a set of block identifiers that directly determine the selected feature blocks. Optionally, a block mask can be generated based on the index.

[0162] Schematic, the routing information of the selected key-value block corresponding to the h-th attention head is implemented as Ωh = Ωshared ∪ Ω(private, h), where Ωshared is the first routing information corresponding to the shared feature block of the attention group to which the h-th attention head belongs, and Ω(private, h) is the second routing information corresponding to the private feature block of the h-th attention head.

[0163] Optionally, the implementation of attention computation by selecting key-value block routing information can reuse the block mask path, that is, reuse the sparse attention operator already acquired during feature processing. Alternatively, the feature processing process can be extended to a hybrid execution method of performing feature processing based on shared feature block indexes and performing feature processing based on private feature block indexes. Here, the block mask path refers to the complete data flow and control flow from generating the block mask to actually performing attention computation in block sparse attention computation, which can reduce routing overhead and improve feature processing efficiency.

[0164] Based on the routing information of selected key-value blocks, an inference-time routing method is implemented that requires no model parameter training or only minimal configuration. This method makes the selection of feature blocks compatible with the execution path of existing block sparse attention mechanisms, thereby improving the efficiency of real-time feature processing. The shared feature blocks determined based on the first routing information mainly improve the efficiency of feature processing, while the private feature blocks determined based on the second routing information restore the differentiated attention capabilities of each attention head.

[0165] In some embodiments, after merging the first routing information and the second routing information to obtain the selected key value block routing information for the p-th query block corresponding to the h-th attention head, the method further includes: for each query block corresponding to each attention head, if the total number of shared feature blocks and private feature blocks indicated by the selected key value block routing information does not reach the total number threshold, determining supplementary feature blocks from the key value blocks corresponding to each attention head, and updating the routing information of the supplementary feature blocks to the selected key value block routing information.

[0166] The process of completing shared and private feature blocks for each query block includes determining supplementary feature blocks. Illustratively, methods for determining supplementary feature blocks include, but are not limited to: re-determining supplementary feature blocks from the unselected feature blocks among multiple key-value blocks corresponding to the key-value feature representation based on independent indicators; or, obtaining the list of private candidate blocks saved during the determination of private feature blocks, and continuing to select supplementary feature blocks based on the candidate block list.

[0167] Schematic illustration: When the shared feature blocks and private feature blocks corresponding to each attention head are determined through the routing process, the actual first number of shared feature blocks may not reach a first threshold, and the second number of private feature blocks may not reach a second threshold. The ratio and ratio coefficient between the first and second numbers may be different. If the sum of the first and second numbers does not reach the total number threshold, supplementary feature blocks are determined from the unselected key-value blocks to ensure that the total number of selected feature blocks corresponding to the shared feature blocks, private feature blocks, and supplementary feature blocks reaches the total number threshold.

[0168] In some embodiments, determining the private feature blocks and supplementary feature blocks for each query block is the second routing stage in the routing process. The routing process includes two routing stages: 1. First, select shared feature blocks from multiple key-value blocks based on group-level shared indicators; 2. Then, select a small number of private feature blocks (and supplementary feature blocks) from the key-value blocks other than the shared feature blocks based on the independent indicators of each attention head itself, to supplement the semantic information of the attention head characteristics (head-specific); optionally, the feature information of the attention head characteristics includes the feature information indicated by the private feature blocks and the supplementary feature blocks.

[0169] By using a pre-obtained total threshold to indicate the attention budget for each query block corresponding to each attention head, on the one hand, the total threshold is used as a constraint during the routing process to avoid insufficient number of selected feature blocks; on the other hand, if there are insufficient selected feature blocks, private feature blocks corresponding to each attention head are supplemented. Feature processing is performed based on fully obtaining the feature information required by each attention head in the input feature representation, thereby improving the accuracy of feature processing with a fixed routing mechanism.

[0170] For illustrative purposes, please refer to the following: Figure 7 , Figure 7 This is a schematic diagram illustrating a selected feature block quantity budget configuration method provided in an exemplary embodiment of this application, such as... Figure 7 As shown, when the selected feature block quantity budget is configured in a fixed budget manner 710, Kselected = Kshared + Kprivate, where Kselected is the total threshold; Kshared is the first quantity threshold for shared feature blocks, and optionally, Kshared = ... K_private is the second threshold for the number of private feature blocks; optionally, K_private = Kselected, Kshared, and Kprivate are all positive integers. Optionally, the ratio coefficient between the number of shared and private feature blocks is the same when configuring the number of selected feature blocks for different attention heads. Correspondingly, when the total threshold is fixed, the first and second quantity thresholds for different attention heads are the same.

[0171] like Figure 7As shown, when the selected feature block quantity budget is configured in the adaptive dynamic budget method 720, K_private = f(inter-head divergence), where K_private is the second quantity threshold of private feature blocks, K_private is a positive integer, and f is used to indicate the mapping relationship between the second quantity threshold and the inter-head divergence, that is, the second quantity threshold can be determined based on the inter-head divergence. When the inter-head divergence increases, the quantity budget for the private feature blocks corresponding to each attention head is increased, which is also the second quantity threshold. Optionally, the mapping relationship between the second quantity threshold and the inter-head divergence is related to at least one of the quantity ratio coefficient between shared feature blocks and private feature blocks and the total quantity threshold corresponding to shared feature blocks and private feature blocks, where the quantity ratio coefficient and the total quantity threshold are each adaptive parameters determined based on the inter-head divergence.

[0172] Step 246: Based on the selected key-value block routing information, query the key feature block and value feature block for the p-th query block corresponding to the h-th attention head, and generate the sub-attention feature representation.

[0173] In some embodiments, for query blocks at the same position in each query feature representation, when the selected key-value block routing information corresponding to each attention head is obtained based on the first routing information and the second routing information, the sparse attention execution module loads only the selected key-value blocks, including the queried key feature blocks and value feature blocks, according to the selected key-value block routing information, and performs precise attention calculation within the selected feature blocks. The shared feature blocks indicated by the first routing information in the selected key-value block routing information can be used by multiple attention heads within the attention group. The private feature blocks indicated by the second routing information in the selected key-value block routing information serve as the additional calculation basis for each attention head, and attention calculation is performed independently. Based on each query block corresponding to each attention head, the shared feature blocks and private feature blocks are queried respectively to obtain the attention feature representation, and the attention feature representation is written back to the network layer used to execute the task according to the pre-set subsequent processing flow of the deep learning model.

[0174] Step 248: Aggregate the sub-attention feature representations of each attention head corresponding to each query block to obtain the attention feature representation.

[0175] In some embodiments, for each attention head, the selected feature block is queried separately using the attention mechanism to query the multiple query blocks corresponding to the query feature representation, and feature processing is performed independently to obtain the sub-attention feature representation in the feature space corresponding to each attention head; the sub-attention feature representations corresponding to multiple attention heads are concatenated, and the feature information of the feature space corresponding to each attention head is retained to obtain the attention feature representation.

[0176] Schematically, the output result of performing feature processing by each attention head is a sub-attention feature representation on the respective feature space corresponding to each attention head; concatenation processing is performed on the sub-attention feature representations corresponding to each subspace, and a preset output weight matrix is used to obtain the attention feature representation. Exemplarily, the attention feature representation can be obtained through concatenation processing (Concat) by the following formula.

[0177]

[0178] Wherein, the sub-attention feature representation is obtained through attention calculation (Attention) based on the query block and the key-value block corresponding to the selected feature block, that is, the sub-attention feature representation , h is a positive integer, the range of h is [1, H], the positive integer H is the number of the multiple attention heads, and H>1; the query block Q=A , the key feature block , the value feature block , the weight matrix is used to project any query block A among the multiple query blocks corresponding to the query feature representation to the query representation space, and the weight matrix and are respectively used to project any key-value block B in the selected feature block to the key representation space and the value representation space. is the output weight matrix. Schematically, when the dimension of the input feature representation is , the dimension of the sub-attention feature representation in each subspace is / H, is a positive integer, 0< <H; after the sub-attention feature representations are concatenated, linear projection is performed to obtain the attention feature representation, and the dimension of the attention feature representation is H.

[0179] Schematically, please refer to Figure 8 , Figure 8 which is a schematic diagram of a feature processing procedure provided by an exemplary embodiment of the present application, such as Figure 8As shown, after obtaining the input feature representation corresponding to the input data, the query feature representation, key feature representation, and value feature representation corresponding to the input feature representation are obtained. Step 810 is executed to divide the input feature representation into multiple feature blocks and perform attention head grouping. The feature blocks include query blocks obtained based on the query feature representation and key-value blocks obtained based on the key-value feature representation. A feature block is the basic unit for performing feature processing on the input feature representation. Multiple feature units (including multiple query feature units and multiple key-value feature units) are obtained from each feature block. The multiple query feature units are aggregated to obtain the block-level query feature representation corresponding to the query block, and the multiple key feature units are aggregated to obtain the block-level key feature representation corresponding to the key feature block. The block-level representation is determined based on the block-level query feature representation and the block-level key feature representation.

[0180] like Figure 8 As shown, step 820 is executed to obtain at least one group-level shared metric for an attention group. This is the shared metric corresponding to each key feature block. The implementation of the group-level shared metric includes, but is not limited to, at least one of aggregated response values, voting results, and proxy indexes. The group-level shared metric is used to indicate the shared feature blocks corresponding to multiple attention heads in the attention group, determined from the key value blocks corresponding to multiple attention heads respectively. Before determining the shared feature blocks based on the group-level shared metric, step 830 is executed to obtain the quantity budget (total quantity threshold = first quantity threshold + second quantity threshold) for each of the shared and private feature blocks. Optionally, the total quantity threshold and the quantity ratio coefficient between the shared and private feature blocks are obtained; the first quantity threshold for the shared feature blocks and the second quantity threshold for the private feature blocks are obtained based on the total quantity threshold and the quantity ratio coefficient.

[0181] like Figure 8 As shown, in step 840, based on a first quantity threshold, shared feature blocks shared by multiple attention heads within an attention group are determined from the key-value blocks corresponding to each of the multiple attention heads. Specifically, shared feature blocks whose shared metrics meet a first condition are determined; optionally, the first condition indicates the prior acquisition... The key-value blocks corresponding to the shared indicators are used as shared feature blocks, where This is the first quantity threshold. Step 850 is executed, based on the second quantity threshold, to determine the private feature blocks corresponding to each attention head from the key-value blocks corresponding to each attention head. Specifically, private feature blocks whose independent metrics meet the second condition are determined; in the process of determining private feature blocks, private feature blocks are determined from multiple key-value blocks excluding shared feature blocks. Optionally, the second condition indicates the acquisition... Each independent indicator corresponds to a key-value block as a private feature block, where It is the second quantity threshold.

[0182] like Figure 8 As shown, in step 860, the shared feature blocks and private feature blocks corresponding to each attention head are merged to generate selected key-value block routing information. Optionally, the first routing information corresponding to the shared feature block and the second routing information corresponding to the private feature block are obtained, and the selected key-value block routing information is obtained based on the first and second routing information; the implementation form of the selected key-value block routing information includes, but is not limited to, at least one of block mask, index identifier, etc. In step 870, feature processing is performed based on the selected key-value block routing information through a block sparse attention mechanism. Optionally, for each attention head, the selected feature block is queried separately using the block sparse attention mechanism with multiple query blocks corresponding to the query feature representation, and feature processing is performed to obtain the attention feature representation; wherein, the selected feature block includes at least one of the merged shared feature block and the private feature block.

[0183] In summary, the feature processing method based on multi-head attention networks in this application maps the input data to the feature spaces corresponding to multiple attention heads in the attention group, and divides them into multiple query blocks and multiple key-value blocks to provide processing objects for the feature processing of the block sparse attention mechanism. For each position of the query block in the different query feature representations corresponding to different attention heads, a key-value block indicating the focus of the query block is selected from the multiple key-value blocks. This includes a shared feature block shared by multiple attention heads at that position, and a private feature block corresponding to each attention head. The shared feature block simplifies the routing process of attention calculation and improves the efficiency of feature processing, while the private feature block is used to retain the differentiated attention capabilities of different attention heads to the input feature representation, improving the accuracy of feature processing, thereby improving the efficiency and accuracy of subsequent calculations based on attention feature representations.

[0184] The method provided in this embodiment merges the shared and private feature blocks corresponding to multiple attention heads into a selected feature block. It performs preliminary feature processing fusion at the attention head level through an attention mechanism, and aggregates the sub-attention feature representations corresponding to each attention head. Attention is calculated independently from a low dimension first, and then concatenated to obtain the attention feature representation. By aggregating multiple different feature extraction perspectives, the efficiency and accuracy of feature processing are improved while reducing computational complexity. Furthermore, by merging to obtain the selected feature block, the processing efficiency corresponding to the shared feature block and the semantic attention details corresponding to the private feature block are balanced, thereby improving the efficiency of video generation.

[0185] Figure 9 This is a structural block diagram of a feature processing device based on a multi-head attention network in a model provided in an exemplary embodiment of this application, as shown below. Figure 9 As shown, the device includes: The feature mapping module 910 is configured to acquire the input data of the multi-head attention network in the model, and map the input data to obtain the query feature representations corresponding to the multiple attention heads in the attention group and the key-value feature representations corresponding to the multiple attention heads. The block segmentation module 920 is configured to divide the query feature representation into multiple query blocks and the key value feature representation into multiple key value blocks for each attention head. The routing module 930 is configured to determine shared feature blocks from key-value blocks corresponding to multiple attention heads for query blocks at the same position divided by each query feature representation, and to determine private feature blocks corresponding to each attention head from key-value blocks corresponding to each attention head. The feature processing module 940 is configured to query shared feature blocks and private feature blocks based on each query block corresponding to each attention head, so as to obtain the attention feature representation output by the multi-head attention network.

[0186] In an optional embodiment, the number of multiple attention heads is H, where H is an integer greater than 1, and the key value block includes a key feature block and a value feature block corresponding to the key feature block; The routing module 930 is also configured to generate block-level representations for each attention head based on query blocks and key feature blocks; The routing module 930 is also configured to determine the shared indicators corresponding to multiple key feature blocks based on the block-level representation for query blocks at the same position. The shared indicators are used to indicate the degree of common attention of the query blocks to the key feature blocks from the attention angles indicated by multiple attention heads. The routing module 930 is also configured to determine shared feature blocks based on shared metrics; The routing module 930 is also configured to determine independent metrics corresponding to multiple key feature blocks based on block-level representation for each query block under the h-th attention head. These independent metrics indicate the degree of attention the query block pays to the key feature block's characteristics from the perspective indicated by the h-th attention head, where h is a positive integer. <h≤H; The routing module 930 is also configured to determine the private feature block corresponding to the h-th attention head according to independent metrics.

[0187] In an optional embodiment, the routing module 930 is further configured to perform block-level representation aggregation processing on the query blocks at the same position of each query feature representation, and to obtain response aggregation values ​​corresponding to the key feature blocks at the same position of each key value representation, and the shared metrics include the response aggregation values. The routing module 930 is also configured to determine the voting results corresponding to multiple key feature blocks based on the block-level representation for query blocks at the same position, and the shared metrics include the voting results; The routing module 930 is also configured to determine the similarity scores corresponding to multiple key feature blocks based on the block-level representation for query blocks at the same position, and the shared metrics include the similarity scores.

[0188] In an optional embodiment, the routing module 930 is further configured to determine the block-level representation as an independent metric for each query block under the h-th attention head; The routing module 930 is also configured to determine the difference scores corresponding to multiple key feature blocks based on the block-level representation for each query block under the h-th attention head. The difference scores are used to indicate the degree of difference between the block-level representation and the shared metrics. The routing module 930 is also configured to determine the local feature scores corresponding to multiple key feature blocks based on the block-level representation for each query block under the h-th attention head. The local feature scores are used to indicate the degree to which the query block pays attention to the key feature blocks in terms of local details.

[0189] In an optional embodiment, the routing module 930 is further configured to determine a plurality of query feature units in a query block, and to determine a plurality of key feature units in a key feature block. The routing module 930 is also configured to aggregate multiple query feature units to obtain a block-level query feature representation corresponding to a query block, and to aggregate multiple key feature units to obtain a block-level key feature representation corresponding to a key feature block; The routing module 930 is also configured to determine the block-level representation based on the block-level query feature representation and the block-level key feature representation.

[0190] In an optional embodiment, the routing module 930 is further configured to determine first routing information corresponding to a shared feature block and second routing information corresponding to a private feature block for the p-th query block corresponding to the h-th attention head, where p is a positive integer; The routing module 930 is also configured to merge the first routing information and the second routing information to obtain the selected key-value block routing information for the p-th query block corresponding to the h-th attention head; The feature processing module 940 is also configured to query the key feature block and value feature block for the p-th query block corresponding to the h-th attention head based on the selected key-value block routing information, and generate a sub-attention feature representation; The feature processing module 940 is also configured to aggregate the sub-attention feature representations of each query block corresponding to each attention head to obtain the attention feature representation.

[0191] In an optional embodiment, the routing module 930 is further configured to determine a total number threshold and a quantity ratio coefficient; the total number threshold is used to constrain the total number of shared feature blocks and private feature blocks determined for a single attention head and a single query block, and the quantity ratio coefficient is used to constrain the quantity ratio relationship between shared feature blocks and private feature blocks determined for a single attention head and a single query block. The routing module 930 is also configured to determine a first quantity threshold for shared feature blocks and a second quantity threshold for private feature blocks based on a total quantity threshold and a quantity ratio coefficient. The first quantity threshold is used to constrain the number of shared feature blocks determined for a single attention head and a single query block, and the second quantity threshold is used to constrain the number of private feature blocks determined for a single attention head and a single query block.

[0192] In an optional embodiment, the routing module 930 is further configured to, for each query block corresponding to each attention head, determine supplementary feature blocks from the key-value blocks corresponding to each attention head if the total number of shared feature blocks and private feature blocks indicated by the selected key-value block routing information does not reach the total number threshold, and update the routing information of the supplementary feature blocks to the selected key-value block routing information.

[0193] In an optional embodiment, the routing module 930 is further configured to determine a quantity scaling factor based on the degree of difference in key-value feature representations among multiple attention heads; The routing module 930 is also configured to determine the quantity ratio coefficient based on the network layer position corresponding to the input data; The routing module 930 is also configured to determine the quantity ratio coefficient based on the complexity of the feature processing requirements corresponding to the input data.

[0194] In an optional embodiment, the input data includes prompts indicating content generation requirements; the attention feature representation is a feature representation obtained by the attention layer in the content generation task; the content generation task includes at least one of video generation, image generation, audio generation, and text generation tasks. The input data includes image data, which is the data to be recognized; the attention feature representation is the feature representation obtained by the attention layer in the image recognition task. The input data includes audio data, which is the data to be recognized; the attention feature representation is the feature representation obtained by the attention layer in the audio recognition task.

[0195] In summary, the feature processing device based on the multi-head attention network in the model of this application maps the input data to the feature space corresponding to multiple attention heads in the attention group, and divides it into multiple query blocks and multiple key-value blocks to provide processing objects for the feature processing of the block sparse attention mechanism. For each position of the query block in the different query feature representations corresponding to different attention heads, the key-value blocks that indicate the focus of the query block are selected from the multiple key-value blocks. These include shared feature blocks shared by multiple attention heads at that position, and private feature blocks corresponding to each attention head. The shared feature blocks simplify the routing process of attention calculation and improve the efficiency of feature processing, while the private feature blocks are used to retain the differentiated attention capabilities of different attention heads to the input feature representation, improve the accuracy of feature processing, and thus improve the efficiency and accuracy of subsequent calculations based on attention feature representations.

[0196] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0197] Figure 10 This is a schematic block diagram of a computer device provided in an exemplary embodiment of this application. The computer device is a second server 1000. Typically, the second server 1000 includes a first processor 1001 and a first memory 1002.

[0198] The first processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The first processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The first processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the first processor 1001 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the first processor 1001 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0199] The first memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The first memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the first memory 1002 are used to store at least one instruction, which is executed by the first processor 1001 to implement the texture processing method provided in the method embodiments of this application.

[0200] In some embodiments, the second server 1000 may optionally include an input interface 1003 and an output interface 1004. The first processor 1001, the first memory 1002, and the input interface 1003 and output interface 1004 can be connected via a bus or signal line. Various peripheral devices can be connected to the input interface 1003 and output interface 1004 via a bus, signal line, or circuit board. The input interface 1003 and output interface 1004 can be used to connect at least one input / output (I / O) related peripheral device to the first processor 1001 and the first memory 1002. In some embodiments, the first processor 1001, the first memory 1002, and the input interface 1003 and output interface 1004 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the first processor 1001, the first memory 1002, and the input interface 1003 and output interface 1004 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0201] Figure 11 This is a schematic block diagram of a computer device provided in another exemplary embodiment of this application. Optionally, the computer device 1100 is a terminal.

[0202] The computer device 1100 can be a portable mobile terminal, also referred to as a mobile terminal in this embodiment. Examples include: smartphones, tablets, laptops, e-book readers, MP3 players, MP4 players, etc. The computer device 1100 may also be referred to as user equipment, portable terminal, or other names.

[0203] Typically, computer device 1100 includes: a second processor 1101 and a second memory 1102.

[0204] The second processor 1101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The second processor 1101 may be implemented using at least one hardware form selected from DSP, FPGA, and PLA. The second processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the second processor 1101 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, the second processor 1101 may also include an AI processor, which is used to handle computational operations related to machine learning.

[0205] The second memory 1102 may include one or more computer-readable storage media, which may be tangible and non-transitory. The second memory 1102 may also include high-speed random access memory devices and non-volatile storage devices, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the second memory 1102 are used to store at least one instruction, which is executed by the second processor 1101 to implement the texture processing methods provided in the various method embodiments of this application.

[0206] In some embodiments, the computer device 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1104, a touch display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1108. The computer device 1100 also includes one or more sensors 1109. The one or more sensors 1109 include, but are not limited to: an accelerometer 1110, a gyroscope 1111, a pressure sensor 1112, an optical sensor 1113, and a proximity sensor 1114.

[0207] Those skilled in the art will understand that Figure 10 , Figure 11 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0208] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor, implements the feature processing method based on the multi-head attention network in the model described above. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0209] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the feature processing method based on the multi-head attention network in the model described above.

[0210] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the operational data and account information involved in this application were obtained with full authorization.

[0211] To further clarify, this application can display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data (such as input data). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without confirmation from the user), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of this user data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0212] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0213] Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in the order of their numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0214] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0215] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0216] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A feature processing method based on a multi-head attention network in a model, characterized in that, The method includes: The input data of the multi-head attention network in the model is obtained, and the query feature representations and key-value feature representations corresponding to the multiple attention heads in the attention group are obtained based on the input data. For each attention head in the attention group, the query feature representation is divided into multiple query blocks, and the key value feature representation is divided into multiple key value blocks; For query blocks at the same position represented by each query feature, a shared feature block is determined from the key-value blocks corresponding to the multiple attention heads, and a private feature block corresponding to each attention head is determined from the key-value blocks corresponding to each attention head. Based on each query block corresponding to each attention head, the shared feature block and the private feature block are queried respectively to obtain the attention feature representation output by the multi-head attention network.

2. The method according to claim 1, characterized in that, The number of the plurality of attention heads is H, where H is an integer greater than 1, and the key value block includes a key feature block and a value feature block corresponding to the key feature block; The process of determining shared feature blocks from the key-value blocks corresponding to the multiple attention heads for query blocks at the same positions based on each query feature representation, and determining the private feature blocks corresponding to each attention head from the key-value blocks corresponding to each attention head, includes: For each attention head, a block-level representation is generated based on the query block and the key feature block; For query blocks at the same position in each query feature representation, a shared index is determined based on the block-level representation. The shared index is used to indicate the degree of common attention of the query block to the key feature block from the attention angles indicated by the multiple attention heads. The shared feature block is determined according to the shared index; For each query block under the h-th attention head, independent metrics are determined based on the block-level representation for each of the multiple key feature blocks. These independent metrics indicate the degree of attention the query block pays to the characteristics of the key feature blocks from the perspective indicated by the h-th attention head, where h is a positive integer. <h≤H; The private feature block corresponding to the h-th attention head is determined according to independent indicators.

3. The method according to claim 2, characterized in that, The method for determining shared metrics corresponding to the multiple key feature blocks based on the block-level representation of query blocks at the same position for each query feature representation includes at least one of the following: For query blocks at the same position divided by each query feature representation, the block-level representation corresponding to the key feature blocks at the same position divided by each key value feature representation is aggregated to obtain the response aggregate value corresponding to the key feature blocks at the multiple positions respectively. The shared index includes the response aggregate value. For query blocks at the same position in each query feature representation, the voting results corresponding to the multiple key feature blocks are determined based on the block-level representation, and the shared index includes the voting results; For query blocks at the same position in each query feature representation, a similarity score is determined based on the block-level representation for each of the multiple key feature blocks, and the shared metric includes the similarity score.

4. The method according to claim 2, characterized in that, For each query block under the h-th attention head, the independent indicators corresponding to the multiple key feature blocks are determined based on the block-level representation, including at least one of the following: For each query block under the h-th attention head, the block-level representation is determined as the independent indicator; For each query block under the h-th attention head, the difference score corresponding to each of the multiple key feature blocks is determined based on the block-level representation. The difference score is used to indicate the degree of difference between the block-level representation and the shared index. For each query block under the h-th attention head, the local feature scores corresponding to the multiple key feature blocks are determined based on the block-level representation. The local feature scores are used to indicate the degree to which the query block pays attention to the local detail information of the key feature block.

5. The method according to claim 2, characterized in that, The step of generating a block-level representation for each attention head based on the query block and the key feature block includes: Determine multiple query feature units in the query block, and determine multiple key feature units in the key feature block; The plurality of query feature units are aggregated to obtain the block-level query feature representation corresponding to the query block, and the plurality of key feature units are aggregated to obtain the block-level key feature representation corresponding to the key feature block; The block-level representation is determined based on the block-level query feature representation and the block-level key feature representation.

6. The method according to claim 1, characterized in that, The process involves querying the shared feature block and the private feature block based on each query block corresponding to each attention head, to obtain the attention feature representation output by the multi-head attention network, including: For the p-th query block corresponding to the h-th attention head, determine the first routing information corresponding to the shared feature block and the second routing information corresponding to the private feature block, where p is a positive integer; Merge the first routing information and the second routing information to obtain the selected key-value block routing information for the p-th query block corresponding to the h-th attention head; Based on the selected key-value block routing information, the key feature block and value feature block for the p-th query block corresponding to the h-th attention head are obtained, and a sub-attention feature representation is generated; The attention feature representation is obtained by aggregating the sub-attention feature representations corresponding to each query block for each attention head.

7. The method according to claim 6, characterized in that, Before determining the shared feature block from the key-value blocks corresponding to the multiple attention heads, and before determining the private feature block corresponding to each attention head from the key-value blocks corresponding to each attention head, the method further includes: A total threshold is determined, and a quantity ratio coefficient is determined; the total threshold is used to constrain the total number of the shared feature blocks and the private feature blocks determined for a single attention head and a single query block, and the quantity ratio coefficient is used to constrain the quantity ratio relationship between the shared feature blocks and the private feature blocks determined for a single attention head and a single query block. The first quantity threshold of the shared feature blocks and the second quantity threshold of the private feature blocks are determined based on the total number threshold and the quantity ratio coefficient. The first quantity threshold is used to constrain the number of the shared feature blocks determined for a single attention head and a single query block, and the second quantity threshold is used to constrain the number of the private feature blocks determined for a single attention head and a single query block.

8. The method according to claim 7, characterized in that, After merging the first routing information and the second routing information to obtain the selected key-value block routing information for the p-th query block corresponding to the h-th attention head, the method further includes: For each query block corresponding to each attention head, if the total number of shared feature blocks and private feature blocks indicated by the selected key value block routing information does not reach the total threshold, a supplementary feature block is determined from the key value blocks corresponding to each attention head, and the routing information of the supplementary feature block is updated to the selected key value block routing information.

9. The method according to claim 7, characterized in that, The determination of the quantity ratio coefficient includes at least one of the following: The quantity ratio coefficient is determined based on the degree of difference in key-value feature representations among the multiple attention heads; The quantity ratio coefficient is determined based on the network layer position corresponding to the input data; The quantity ratio coefficient is determined based on the complexity of the feature processing requirements corresponding to the input data.

10. The method according to claim 1, characterized in that, The input data includes prompt information, which is used to indicate content generation requirements; the attention feature representation is a feature representation obtained by the attention layer in the content generation task; the content generation task includes at least one of video generation task, image generation task, audio generation task, and text generation task. The input data includes image data, which is the data to be recognized; the attention feature representation is the feature representation obtained by the attention layer processing in the image recognition task. The input data includes audio data, which is the data to be recognized; the attention feature representation is the feature representation obtained by the attention layer in the audio recognition task.

11. A feature processing device based on a multi-head attention network in a model, characterized in that, The device includes: The feature mapping module is configured to acquire the input data of the multi-head attention network in the model, and map the input data to obtain the query feature representations corresponding to the multiple attention heads in the attention group and the key-value feature representations corresponding to the multiple attention heads. The block segmentation module is configured to divide the query feature representation into multiple query blocks and the key value feature representation into multiple key value blocks for each attention head. The routing module is configured to determine shared feature blocks from the key-value blocks corresponding to multiple attention heads for query blocks at the same position divided by each query feature representation, and to determine the private feature blocks corresponding to each attention head from the key-value blocks corresponding to each attention head. The feature processing module is configured to query the shared feature block and the private feature block respectively based on each query block corresponding to each attention head, so as to obtain the attention feature representation output by the multi-head attention network.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the feature processing method based on a multi-head attention network in a model as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the feature processing method based on a multi-head attention network in a model as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the feature processing method based on a multi-head attention network in a model as described in any one of claims 1 to 10.

Images