Methods, devices, storage media, and electronic devices for converting multi-head attention models

Abstract

This application provides a method, apparatus, storage medium, and electronic device for converting a multi-head attention model, relating to the computer field. The method includes: acquiring a multi-head attention model; traversing at least one attention layer; obtaining a target distance based on a first query weight matrix, a first key weight matrix, and a first value weight matrix corresponding to the current attention layer; obtaining the attention weight corresponding to each attention head based on the output projection matrix corresponding to the current attention layer; grouping multiple attention heads based on all target distances corresponding to the current attention layer and the attention weight corresponding to each attention head to obtain at least one target attention head group corresponding to the current attention layer; and converting the target multi-head attention model based on the at least one target attention head group to obtain a target model. This application addresses the problem of low inference output accuracy of the converted model in related technologies.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for converting a multi-head attention model. Background Technology

[0002] In recent years, with the rapid development of Natural Language Processing (NLP) technology, large-scale pre-trained models based on the Transformer architecture have become a key driving force in the field of NLP. These models rely on the Multi-Head Attention (MHA) mechanism to capture complex dependencies in the input data, significantly improving model performance. However, the MHA mechanism faces the challenge of high computational cost during the inference stage, especially on resource-constrained devices or in scenarios with large-scale concurrent access. High memory consumption limits the deployment and application scope of the model.

[0003] To reduce the existing overhead of multi-head attention models, related techniques employ a uniform grouping strategy for the multiple attention heads of the model. However, this may lead to a loss of the model's expressive power, especially when dealing with highly complex language tasks. As a result, when using the uniform grouping strategy for multiple attention heads to obtain the transformed model for inference, the error in the results is relatively large, leading to low accuracy of the inference output of the transformed model. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and electronic device for converting a multi-head attention model, in order to at least solve the technical problem of low output accuracy of the converted model used in related technologies.

[0005] According to one embodiment of this application, a method for converting a multi-head attention model is provided, comprising: obtaining a target multi-head attention model, wherein the target multi-head attention model includes at least one attention layer, and the attention layer includes multiple attention heads; traversing at least one attention layer and obtaining at least one target distance based on a first query weight matrix, a first key weight matrix, and a first value weight matrix corresponding to the current attention layer, wherein the target distance is used to indicate the similarity between a first attention head and a second attention head, and the multiple attention heads include the first attention head and the second attention head; obtaining the attention weight corresponding to each attention head in the current attention layer based on the output projection matrix corresponding to the current attention layer, wherein the attention weight is used to indicate the importance of the attention head; grouping the multiple attention heads based on all target distances corresponding to the current attention layer and the attention weight corresponding to each attention head in the current attention layer to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group includes at least one attention head; and converting the target multi-head attention model based on the at least one target attention head group corresponding to the current attention layer to obtain a target model.

[0006] According to another embodiment of this application, a multi-head attention model conversion apparatus is provided, comprising: a first acquisition unit, configured to acquire a target multi-head attention model, wherein the target multi-head attention model includes at least one attention layer, and the attention layer includes a plurality of attention heads; a second acquisition unit, configured to traverse at least one attention layer and acquire at least one target distance based on a first query weight matrix, a first key weight matrix, and a first value weight matrix corresponding to the current attention layer, wherein the target distance is used to indicate the similarity between a first attention head and a second attention head, and the plurality of attention heads includes the first attention head and the second attention head; and a third acquisition unit, configured to acquire based on the above... The output projection matrix corresponding to the current attention layer is used to obtain the attention weight of each attention head in the current attention layer, wherein the attention weight is used to indicate the importance of the attention head; a grouping unit is used to group multiple attention heads based on all target distances corresponding to the current attention layer and the attention weight of each attention head in the current attention layer to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group includes at least one attention head; a transformation unit is used to transform the target multi-head attention model based on at least one target attention head group corresponding to the current attention layer to obtain the target model.

[0007] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0008] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0009] The embodiments provided in this application significantly reduce the computational cost of the model's inference stage by performing uneven grouping of attention heads in a target multi-head attention model based on similarity and importance, while preserving the original performance of the model to the maximum extent. Specifically, the similarity between attention heads is measured by obtaining the target distance between them, and then the importance of each attention head is evaluated using the attention weights in the output projection matrix. By combining these two factors, it is possible to determine which attention heads can be merged, which need to be kept independent, and which attention heads with high importance and similarity are prioritized for retention. This effectively prevents a significant drop in model performance while reducing computational costs, thereby improving the accuracy of inference when using a model transformed from a multi-head attention model. This solves the technical problem of low inference output accuracy of transformed models in related technologies. Attached Figure Description

[0010] Figure 1 This is a hardware structure block diagram of a multi-head attention model conversion method according to an embodiment of this application;

[0011] Figure 2 This is a flowchart of a conversion method for a multi-head attention model according to an embodiment of this application;

[0012] Figure 3 This is one of the schematic diagrams of a conversion method for a multi-head attention model according to an embodiment of this application;

[0013] Figure 4 This is a second schematic diagram of a conversion method for a multi-head attention model according to an embodiment of this application;

[0014] Figure 5 This is a structural block diagram of a multi-head attention model conversion device according to an embodiment of this application. Detailed Implementation

[0015] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a multi-head attention model conversion method according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0018] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the mapping relationship determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0019] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0020] As an alternative approach, the conversion method for multi-head attention models, such as... Figure 2 As shown, the specific steps include:

[0021] S202, Obtain the target multi-head attention model, wherein the target multi-head attention model includes at least one attention layer, and the attention layer includes multiple attention heads;

[0022] S204, traverse at least one attention layer, and obtain at least one target distance based on the first query weight matrix, the first key weight matrix and the first value weight matrix corresponding to the current attention layer, wherein the target distance is used to indicate the similarity between the first attention head and the second attention head, and the multiple attention heads include the first attention head and the second attention head;

[0023] S206, Based on the output projection matrix corresponding to the current attention layer, obtain the attention weight corresponding to each attention head in the current attention layer, where the attention weight is used to indicate the importance of the attention head;

[0024] S208, based on all target distances corresponding to the current attention layer and the attention weights corresponding to each attention head in the current attention layer, group multiple attention heads to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group includes at least one attention head;

[0025] S210, based on at least one target attention head group corresponding to the current attention layer, transform the target multi-head attention model to obtain the target model.

[0026] Optionally, in this embodiment, the multi-head attention model (MHA) is a component based on the Transformer architecture. It captures the dependencies in input data such as text and image feature sequences from different dimensions through multiple independent attention heads, thereby extracting information more comprehensively and improving the model's ability to capture complex patterns.

[0027] Optionally, in this embodiment, the attention layer is the basic building block of the target multi-head attention model. Each attention layer contains a complete process including Q-projection, K-projection, V-projection, multi-head segmentation, attention calculation of each head, multi-head merging, and output projection. It is responsible for completing a complete attention mechanism calculation and converting the input feature matrix into an output feature matrix containing attention weights.

[0028] Optionally, in this embodiment, the query weight matrix may be, but is not limited to, a matrix used to map input features to query vectors, representing the model's query intent for input data; the key weight matrix may be, but is not limited to, a matrix used to map input features to key vectors, representing the identification information of input data; and the weight matrix is ​​used to map input features to value vectors, representing the content information of input data. The query weight matrix, key weight matrix, and value weight matrix together support the calculation of attention scores and are the foundation for implementing the attention mechanism.

[0029] Optionally, in this embodiment, the target distance can be, but is not limited to, an indicator used to quantify the similarity between two attention heads. For example, the similarity between a first attention head and a second attention head can be, but is not limited to, calculated by taking the distance between the Q, K, and V submatrices corresponding to the two attention heads and then weighted and summing the distances between the submatrices. The smaller the target distance, the more similar the dimensions of information captured by the two attention heads are, and the more suitable they are to be grouped together; conversely, the lower the similarity, the more suitable they are to be grouped together, thus providing a quantitative basis for the reasonable grouping of attention heads.

[0030] Optionally, in this embodiment, the output projection matrix may be, but is not limited to, a parameter matrix in the attention layer used to perform linear transformation on the feature matrix after merging multiple attention heads. It can map the feature matrix after combining the outputs of multiple attention heads to the target dimension of the model. At the same time, through the element weights in the matrix, it indirectly reflects the importance of the output results of different attention heads. The larger the element value of the output projection matrix corresponding to a certain attention head, the higher the contribution of the output of that attention head to the final feature representation.

[0031] Optionally, in this embodiment, the attention weight can be, but is not limited to, an indicator calculated based on the output projection matrix to quantify the importance of each attention head. The larger the attention weight, the higher the contribution of the attention head's output to the final model result. When grouping, its merging with other attention heads should be minimized to avoid the loss of important information, thus providing a basis for prioritizing the importance of attention head grouping.

[0032] Optionally, in this embodiment, the target attention head group is a set obtained by grouping multiple attention heads in the attention layer. Each group contains at least one attention head, the attention heads in the group have high similarity, and the importance distribution of the attention heads in the group is reasonable. For example, the group containing a high-importance attention head contains a small number of attention heads, which can provide a grouping basis for the subsequent generation of a shared key-value matrix. This reduces the KV caching cost during model inference while reducing the loss of important information.

[0033] The target model is a non-uniformly grouped query attention model derived from the target multi-head attention model through operations such as attention head grouping and generating a shared key-value matrix. Compared with the original MHA model, the target model reduces the number of key-value caches while maintaining similar inference quality, thus lowering the consumption of hardware memory or system memory during model inference. At the same time, through the non-uniform grouping strategy, it takes into account the similarity and importance of attention heads, and its function can achieve a balance between model inference cost and inference quality in resource-constrained scenarios.

[0034] To further illustrate, suppose an original MHA model contains 12 attention layers, each with 16 attention heads. During inference, each layer needs to cache 16 sets of key-value matrices, resulting in significant GPU memory consumption. After converting it to a target model, each layer is divided into 5 target attention head groups based on the target distance and attention weights of the attention heads. Each layer only needs to cache 5 sets of shared key-value matrices, reducing memory consumption to 5 / 16 of the original. Simultaneously, because the similarity and importance of attention heads are considered during grouping, the loss in inference quality is minimal, meeting the deployment requirements of edge devices.

[0035] Optionally, in this embodiment, it is first necessary to obtain the original multi-head attention model to be transformed and clarify the basic structure of the model, such as including one or more attention layers, and each attention layer contains two or more attention heads. These attention heads are the core objects of subsequent grouping operations, thereby determining the input objects and operation scope of the transformation, and providing a basis for subsequent distance calculation, weight evaluation and grouping operations based on attention layers and attention heads.

[0036] Next, each attention layer of the target multi-head attention model is processed sequentially. For the attention layer currently being processed, its corresponding first query weight matrix, first key weight matrix, and first value weight matrix are extracted, and the target distance indicating the similarity between the two attention heads is obtained based on these matrices, thereby obtaining the target distance between any two attention heads.

[0037] Then, for the attention layer being processed, its output projection matrix is ​​extracted to obtain the attention weight corresponding to each attention head. The larger the weight value, the higher the contribution of the output of the attention head to the final result of the model, and the stronger its importance. This provides an important basis for attention head grouping, ensuring that attention heads with high importance can be protected first when grouping them in the future, such as reducing their merging with other heads, avoiding the loss of important information due to grouping, and thus maintaining the quality of model inference.

[0038] Then, based on the target distance between all pairs of attention heads in the current attention layer and the attention weight of each attention head, multiple attention head groups are grouped to obtain at least one target attention head group. Each group contains at least one attention head, and the heads within the group have high similarity and reasonable importance distribution.

[0039] Finally, based on at least one attention head group, the target multi-head attention model is transformed to obtain an unevenly grouped query attention model.

[0040] Understandably, by acquiring the target multi-head attention model and clarifying its attention layer and attention head structure, a clear operational object is provided for subsequent transformation; by calculating the target distance and attention weight, similarity and importance criteria are provided for grouping, respectively, to ensure the rationality of grouping; finally, based on the grouping results, the target multi-head attention model is transformed to obtain the target model, thereby reducing the KV caching cost during the inference of the target multi-head attention model while maintaining the model inference quality to the maximum extent, enabling the model to adapt to resource-constrained scenarios.

[0041] The embodiments provided in this application significantly reduce the computational cost of the model's inference stage by performing uneven grouping of attention heads in a target multi-head attention model based on similarity and importance, while preserving the original performance of the model to the maximum extent. Specifically, the similarity between attention heads is measured by obtaining the target distance between them, and then the importance of each attention head is evaluated using the attention weights in the output projection matrix. By combining these two factors, it is possible to determine which attention heads can be merged, which need to be kept independent, and which attention heads with high importance and similarity are prioritized for retention. This effectively prevents a significant drop in model performance while reducing computational costs, thereby improving the accuracy of inference when using a model transformed from a multi-head attention model. This solves the technical problem of low inference output accuracy of transformed models in related technologies.

[0042] As an optional approach, based on the first query weight matrix, the first key weight matrix, and the first value weight matrix corresponding to the current attention layer, at least one target distance is obtained, including:

[0043] Based on multiple attention heads, the first query weight matrix, the first key weight matrix, and the first value weight matrix are segmented to obtain multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices corresponding to the current attention layer;

[0044] Based on the second query weight matrix corresponding to the first attention head, the query distance is obtained, where the query distance is used to indicate the similarity between the second query weight matrix corresponding to the first attention head and the second query weight matrix corresponding to the second attention head.

[0045] Based on the second key weight matrix corresponding to the first attention head, the key distance is obtained, where the key distance is used to indicate the similarity between the second key weight matrix corresponding to the first attention head and the second key weight matrix corresponding to the second attention head.

[0046] Based on the second value weight matrix corresponding to the first attention head, the value distance is obtained, where the value distance is used to indicate the similarity between the second value weight matrix corresponding to the first attention head and the second value weight matrix corresponding to the second attention head.

[0047] Based on query distance, key distance, and value distance, obtain the target distance between the first query header and the second query header.

[0048] Optionally, in this embodiment, the second query weight matrix may be, but is not limited to, a sub-matrix obtained by dividing the first query weight matrix according to the number of attention heads. Each second query weight matrix corresponds to one attention head, thereby realizing the splitting of query features in different attention head dimensions, so that each attention head can capture the dependency relationship of input data from an independent query perspective, and provide data units for subsequent calculation of the similarity between individual attention heads.

[0049] Optionally, in this embodiment, the second key weight matrix may be, but is not limited to, a submatrix resulting from dividing the first key weight matrix according to the number of attention heads, with each submatrix corresponding to one attention head. Its function is to split the features of the key matrix into different attention heads, enabling each attention head to capture the identification information of the input data from an independent key dimension. This provides a basis for calculating the similarity of key features between individual attention heads and ensures that feature consistency in the key dimension is considered during grouping.

[0050] Optionally, in this embodiment, the second value weight matrix may be, but is not limited to, a submatrix resulting from dividing the first value weight matrix according to the number of attention heads, with each submatrix corresponding to one attention head. Its function is to split the features of the value matrix into different attention heads, enabling each attention head to extract content information from the input data from an independent value dimension. This provides data support for calculating the similarity of value features between individual attention heads and ensures the continuity of value dimension features during grouping.

[0051] Optionally, in this embodiment, the query distance can be, but is not limited to, a metric used to quantify the similarity between two attention heads in the query dimension. The smaller the query distance, the more similar the feature mapping rules of the two attention heads are in the query dimension, the more consistent the direction of capturing the query intent of the input data, and the more suitable they are to be grouped together to reduce information loss.

[0052] Optionally, in this embodiment, the key distance can be, but is not limited to, a metric for measuring the similarity between two attention heads in the key dimension. The smaller the key distance, the more similar the feature mapping rules of the two attention heads in the key dimension, the more uniform the way they capture the identification information of the input data, and the better they can maintain the feature consistency in the key dimension when grouping them, thus reducing the deviation in attention score calculation.

[0053] Optionally, in this embodiment, the value distance can be, but is not limited to, an indicator for evaluating the similarity between two attention heads in the value dimension. The smaller the value distance, the more similar the information extraction rules of the two attention heads in the value dimension, the more similar the output features of the generated attention heads, the less information loss caused when sharing the value matrix after grouping, and the better the model inference performance can be maintained.

[0054] Optionally, in this embodiment, the target distance is a quantitative indicator of the overall similarity between two attention heads, which combines query distance, key distance, and value distance. The smaller the target distance, the higher the overall similarity between the two attention heads in the three dimensions of query, key, and value. It is the core criterion for grouping attention heads and can ensure that the grouping takes into account the feature consistency of the three dimensions.

[0055] Optionally, in this embodiment, the total number of attention heads in the current attention layer is first determined. The first query weight matrix, first key weight matrix, and first value weight matrix are then split into multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices based on the attention heads. Each submatrix uniquely corresponds to one attention head. This splits the weight matrix from the layer level to the attention head level, providing independent data units for subsequent calculations of the similarity between individual attention heads, which is a prerequisite for subsequent distance calculations. Without this splitting, it is impossible to evaluate the similarity of features for individual attention heads, and therefore, it is impossible to achieve reasonable grouping based on attention head features.

[0056] Next, select any two attention heads, extract their corresponding second query weight matrices, and calculate the query distance between these two submatrices. The query distance is negatively correlated with similarity; the smaller the distance, the higher the similarity. By quantifying the similarity of two attention heads in the query dimension, a basis for ensuring the consistency of query dimension features is provided for grouping. Since the feature mapping rules of different attention heads in the query dimension directly affect the capture of the query intent of the input data, the query distance ensures that the feature similarity of the query dimension is taken into account during grouping, avoiding information loss after grouping due to excessive differences in the query dimension.

[0057] For the first and second attention heads, extract their respective second key weight matrices and calculate the key distance between these two sub-matrices. The smaller the key distance, the more similar the feature mapping rules of the two attention heads in the key dimension, and the more consistent their methods of capturing input data identification information.

[0058] Next, for the first and second attention heads, extract their respective second key weight matrices and calculate the value distance between these two sub-matrices. The smaller the value distance, the more similar the information extraction rules of the two attention heads in the value dimension, the more similar the output features of the generated attention heads, and the smaller the information loss of the shared value matrix after grouping.

[0059] Finally, based on query distance, key distance, and value distance, the target distance between the first and second attention heads is obtained. The target distance integrates similarity information from three dimensions, comprehensively reflecting the overall feature similarity between the two attention heads.

[0060] Understandably, by dividing the hierarchical weight matrix into attention head-level sub-matrices, the similarity calculation is refined, providing independent data units for subsequent inter-head similarity evaluation. By calculating the distances in the query, key, and value dimensions separately, the similarity differences of attention heads across different feature dimensions are comprehensively captured. The target distance is obtained through the distances in the query, key, and value dimensions, integrating multi-dimensional similarity information and avoiding the one-sidedness of single-dimensional evaluation. This achieves the technical effect of providing a scientific and comprehensive similarity basis for attention head grouping, reducing information loss due to unreasonable grouping, and laying the foundation for subsequent conversion to a non-uniform grouping query attention model and balancing inference cost and inference quality.

[0061] The embodiments provided in this application achieve refined similarity calculation by dividing the hierarchical weight matrix into attention head-level sub-matrices, providing independent data units for subsequent inter-head similarity evaluation. By calculating the distances in the query, key, and value dimensions separately, the similarity differences of attention heads in different feature dimensions are comprehensively captured. The target distance is obtained through the distances in the query, key, and value dimensions, integrating multi-dimensional similarity information and avoiding the one-sidedness of single-dimensional evaluation. This achieves the technical effect of providing a scientific and comprehensive similarity basis for attention head grouping, reducing information loss caused by unreasonable grouping, and laying the foundation for subsequent conversion to a non-uniform grouping query attention model and balancing inference cost and inference quality.

[0062] As an optional approach, the query distance is obtained based on the second query weight matrix corresponding to the first attention head, including:

[0063] The difference between the element located in the i-th row and j-th column of the second query weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second query weight matrix corresponding to the second attention head is determined as the first element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0064] The square root of the sum of the squares of the differences of all first elements is taken as the query distance.

[0065] Optionally, in this embodiment, the first element difference can be, but is not limited to, an indicator used to quantify the difference between the elements of the first attention head and the second attention head at the corresponding positions in the second query weight matrix. It is obtained by extracting elements of the same row and the same column in the two second query weight matrices and calculating the difference, which serves as the basic data unit for subsequent calculation of query distance. Each first element difference reflects the difference in feature mapping of the two attention heads at a specific position in the query dimension. The larger the difference, the more significant the difference in feature mapping rules at that position.

[0066] To illustrate further, suppose the element value in the 10th row and 20th column of the second query weight matrix corresponding to the first attention head is 0.82, and the element value in the 10th row and 20th column of the second query weight matrix corresponding to the second attention head is 0.57, then the difference between the first elements of the two is 0.25.

[0067] Optionally, in this embodiment, the query distance is the core indicator for measuring the overall similarity between the first attention head and the second attention head in the query dimension. Its calculation logic focuses on the square root of the sum of squares of the differences between corresponding elements of the second query weight matrix. It can integrate the differences between elements at all positions of the two second query weight matrices into a single value, thereby quantifying the similarity of the query dimension and providing a basis for query dimension consistency for subsequent target distance calculation and attention head grouping. The smaller the distance, the more similar the feature mapping rules of the two attention heads in the query dimension.

[0068] Understandably, by traversing all elements of the second query weight matrix to calculate the difference of the first element, a comprehensive capture of the element-level differences between the two attention heads in the query dimension is achieved, preventing local differences from being ignored. By squaring the difference of the first element, summing it, and taking the square root, the discrete element differences are integrated into an overall distance metric, while amplifying the impact of significant differences to ensure that the query distance accurately reflects the similarity at the matrix level. This provides an accurate and reliable basis for query dimension similarity in subsequent target distance calculations, assists in the reasonable grouping of attention heads, reduces the loss of grouping information caused by query dimension similarity assessment bias, and lays the foundation for transforming unevenly grouped query attention models.

[0069] The embodiments provided in this application calculate the difference of the first element by traversing all element positions of the second query weight matrix, thereby achieving a comprehensive capture of the element-level differences between the two attention heads in the query dimension. This avoids ignoring local differences, thus providing an accurate and reliable basis for query dimension similarity for subsequent target distance calculation. It also helps to group attention heads reasonably, reduces the loss of grouping information caused by query dimension similarity evaluation bias, and lays the technical foundation for transforming unevenly grouped query attention models.

[0070] As an optional approach, the key distance is obtained based on the second key weight matrix corresponding to the first attention head, including:

[0071] The difference between the element located in the i-th row and j-th column of the second key weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second key weight matrix corresponding to the second attention head is determined as the second element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0072] The bond distance is determined by taking the square root of the sum of the squares of all the differences of the second element.

[0073] Optionally, in this embodiment, the second element difference is a core indicator used to quantify the difference between the elements of the first attention head and the second attention head at the corresponding positions in the second key weight matrix. It is obtained by extracting the elements of the same row and the same column in the two second key weight matrices and calculating the difference. The overall difference between the two second key weight matrices can be decomposed into the difference of individual elements, providing basic data units for subsequent integration and calculation of key distance. The magnitude of each second element difference directly reflects the difference in feature mapping rules of the two attention heads at a specific position in the key dimension. The larger the difference, the more significant the feature mapping difference at that position.

[0074] Understandably, by traversing all elements of the second key weight matrix and calculating the difference between the second elements, the element-level differences between the two attention heads in the key dimension are comprehensively captured, avoiding the neglect of key local differences. By squaring, summing, and taking the square root of the difference between the second elements, discrete element differences are transformed into a holistic and comparable key distance metric, while amplifying the impact of significant differences to ensure that the key distance accurately reflects the similarity of the feature mapping rules of the two attention heads in the key dimension. This provides an accurate and reliable basis for subsequent target distance calculation, helping attention heads to achieve reasonable grouping based on similarity priority, reducing the loss of grouping information caused by bias in key dimension similarity assessment, transforming the multi-head attention model into a non-uniform grouping query attention model, and balancing inference cost and inference quality.

[0075] The embodiments provided in this application calculate the difference of the second element by traversing all element positions of the second key weight matrix, comprehensively capturing the element-level differences between the two attention heads in the key dimension, avoiding the neglect of local key differences. By squaring, summing, and taking the square root of the difference of the second element, the discrete element differences are transformed into a global, comparable key distance index, while amplifying the impact of significant differences, ensuring that the key distance accurately reflects the similarity of the feature mapping rules of the two attention heads in the key dimension. This achieves the goal of providing accurate and reliable key-dimensional similarity basis for subsequent target distance calculation, helping attention heads to achieve reasonable grouping based on similarity priority, reducing the loss of grouping information caused by bias in key-dimensional similarity evaluation, and realizing the technical effect of converting the multi-head attention model into a non-uniform grouping query attention model, balancing inference cost and inference quality.

[0076] As an optional approach, the value distance is obtained based on the second value weight matrix corresponding to the first attention head, including:

[0077] The difference between the element located in the i-th row and j-th column of the second value weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second value weight matrix corresponding to the second attention head is determined as the third element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0078] The value obtained by taking the square root of the sum of the squares of the differences of all third elements is determined as the value distance.

[0079] Optionally, in this embodiment, the third element difference is the core indicator for quantifying the difference between the first attention head and the second attention head at the corresponding position of the second value weight matrix. By extracting the elements of the same row and the same column in the two second value weight matrices and calculating the difference, the overall difference between the two second value weight matrices can be decomposed into the concrete difference of individual elements, providing the underlying data unit for subsequent integration and calculation of value distance. The magnitude of each third element difference directly reflects the difference in information extraction rules of the two attention heads at a specific position in the value dimension. The larger the difference, the more significant the difference in information extraction logic at that position, and the higher the risk of information loss at that position when sharing the value matrix after grouping.

[0080] Understandably, by traversing all elements of the second-value weight matrix and calculating the difference of the third element, the element-level differences between the two attention heads in the value dimension are comprehensively captured, avoiding the neglect of differences in local key information extraction rules. By squaring, summing, and taking the square root of the difference of the third element, the discrete element differences are transformed into a holistic, comparable value distance indicator, while amplifying the impact of significant differences to ensure that the value distance accurately reflects the similarity of the information extraction rules of the two attention heads in the value dimension. This provides an accurate and reliable basis for subsequent target distance calculation, helping attention heads to achieve reasonable grouping based on similarity priority, reducing the loss of grouping information due to bias in value dimension similarity assessment, and achieving the technical effect of maintaining high-quality inference while reducing inference costs in converting the multi-head attention model into a non-uniform grouping query attention model.

[0081] The embodiments provided in this application calculate the difference of the third element by traversing all element positions of the second value weight matrix, comprehensively capturing the element-level differences between the two attention heads in the value dimension, thus avoiding the neglect of differences in local key information extraction rules. By squaring, summing, and taking the square root of the difference of the third element, the discrete element differences are transformed into a holistic and comparable value distance index, while amplifying the impact of significant differences, ensuring that the value distance accurately reflects the similarity of the information extraction rules of the two attention heads in the value dimension. This provides an accurate and reliable basis for subsequent target distance calculation, helping attention heads to achieve reasonable grouping based on similarity priority, reducing the loss of grouping information due to bias in value dimension similarity assessment, and achieving the technical effect of maintaining high-quality inference while reducing inference costs in converting multi-head attention models into non-uniform grouping query attention models.

[0082] As an optional approach, the target distance between the first query header and the second query header is obtained based on query distance, key distance, and value distance, including:

[0083] Based on the type of target multi-head attention model, the first weight coefficient, the second weight coefficient, and the third weight coefficient are obtained. The first weight coefficient is used to indicate the importance of query distance, the second weight coefficient is used to indicate the importance of key distance, and the third weight coefficient is used to indicate the importance of value distance.

[0084] The target distance between the first query header and the second query header is obtained by summing the product of the query distance and the first weight coefficient, the product of the key distance and the second weight coefficient, and the product of the value distance and the third weight coefficient.

[0085] Optionally, in this embodiment, the type of target multi-head attention model refers to the classification of target multi-head attention models based on the applicable scenarios, core tasks, or architectural features. Different types of models have different degrees of feature dependence in the three dimensions of query, key, and value, providing a basis for determining the first, second, and third weight coefficients, ensuring that the weight allocation is adapted to the actual functional requirements of the model, and avoiding similarity evaluation bias caused by uniform weights. For further examples, text generation models may rely more on information extraction in the value dimension to ensure the accuracy of the generated content, in which case value distance is more important; semantic understanding models may focus more on the association and matching of query and key dimensions to capture contextual logic, in which case key distance is more important.

[0086] Optionally, in this embodiment, the first weight coefficient is a parameter used to quantify the proportion of importance of query distance in the calculation of target distance, with a value range of [0,1], and the sum of it with the second and third weight coefficients is 1. Its function is to adjust the contribution of query distance to target distance according to the type of target multi-head attention model. The larger the first weight coefficient, the more significant the impact of the similarity of the query dimension on the overall similarity of the two attention heads, and the greater the priority given to the size of query distance when grouping, ensuring that the grouping strategy matches the model's dependence on the query dimension.

[0087] Optionally, in this embodiment, the second weighting coefficient is a parameter used to quantify the proportion of importance of key distance in the target distance calculation. Its value ranges from [0,1], and the sum of the second and third weighting coefficients is 1. Its function is to adjust the contribution of key distance to the target distance based on the type of the target multi-head attention model. The larger the second weighting coefficient, the more critical the similarity of the key dimension is to the overall similarity of the attention heads. This means that more emphasis is placed on the size of the key distance during grouping, ensuring that the grouping strategy matches the model's dependence on the key dimension.

[0088] Optionally, in this embodiment, the third weighting coefficient is a parameter used to quantify the proportion of importance of value distance in the target distance calculation. Its value range is [0,1], and the sum of the third weighting coefficient and the first and second weighting coefficients is 1. Its function is to adjust the contribution of value distance to the target distance according to the type of the target multi-head attention model. The larger the third weighting coefficient, the more central the influence of value dimension similarity on the overall similarity of the attention heads. Therefore, the value distance is given priority during grouping, making the grouping strategy consistent with the model's dependence on the value dimension.

[0089] Optionally, in this embodiment, the core tasks and applicable scenarios of the target multi-head attention model are first analyzed to determine its type; then, according to the dimensional dependency characteristics corresponding to the model type, the first, second, and third weight coefficients are assigned. If the model is more dependent on a certain dimension, the weight coefficient of the corresponding dimension is larger, and the sum of the first, second, and third weight coefficients must be 1; finally, three weight coefficients are obtained, which are used to quantify the importance ratio of query distance, key distance, and value distance in the target distance calculation.

[0090] Next, the first weight coefficient, the second weight coefficient, the third weight coefficient, and the query distance, key distance, and value distance between the first attention head and the second attention head calculated above are used. Then, query distance × first weight coefficient, key distance × second weight coefficient, and value distance × third weight coefficient are calculated respectively. Finally, the three products are added together, and the sum is the target distance between the first attention head and the second attention head.

[0091] Understandably, this embodiment discloses a complete method for calculating the target distance between the first attention head and the second attention head. The core process consists of two steps: First, based on the type of the target multi-head attention model, the model's dependence on the three dimensions of query, key, and value is analyzed, and the first, second, and third weight coefficients are used to quantify the importance of the distance in each of the three dimensions. Second, the distances in the three dimensions are multiplied by their corresponding weight coefficients, and the products are summed to obtain the target distance, which reflects the overall similarity index of the attention heads and reflects the model's characteristics. By assigning the first, second, and third weight coefficients based on the type of the target multi-head attention model, accurate matching between the weights and the model's dimensional dependence characteristics is achieved, avoiding mismatches in dimensional importance caused by uniform weights. By multiplying the distances in each dimension by their corresponding weight coefficients and then summing them, multi-dimensional similarity information is integrated, highlighting the influence of the model's dependent dimensions, and generating a scientific target distance. This achieves the technical effect of providing an overall similarity basis that adapts to the model's needs for attention head grouping, ensuring that the grouping strategy conforms to the model's functional characteristics, reducing the loss of grouping information due to biases in dimensional importance assessment, and providing key support for converting multi-head attention models into non-uniformly grouped query attention models and balancing inference cost and inference quality.

[0092] The embodiments provided in this application achieve precise matching between weights and model dimensional dependence characteristics by assigning first, second, and third weight coefficients based on the type of the target multi-head attention model, avoiding dimensional importance mismatch caused by uniform weights. By multiplying the distance of each dimension with its corresponding weight coefficient and then summing the results, multi-dimensional similarity information is integrated, highlighting the influence of model-dependent dimensions, and generating a scientific target distance. This achieves the technical effect of providing an overall similarity basis adapted to model requirements for attention head grouping, ensuring that the grouping strategy conforms to the model's functional characteristics, reducing the loss of grouping information due to dimensional importance assessment bias, and providing key support for converting multi-head attention models into non-uniform grouping query attention models and balancing inference cost and inference quality.

[0093] As an optional approach, after segmenting the first query weight matrix, the first key weight matrix, and the first value weight matrix based on multiple attention heads to obtain multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices corresponding to the current attention layer, it further includes at least one of the following:

[0094] Perform range normalization on the multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices corresponding to the current attention layer;

[0095] Logical functions are applied to the multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices corresponding to the current attention layer.

[0096] The multiple second query weight matrices, multiple second key weight matrices, and multiple second value weight matrices corresponding to the current attention layer are pruned and then linearly normalized.

[0097] Optionally, in this embodiment, range normalization is a linear normalization method that maps data to a specific interval. It is used to eliminate similarity calculation bias caused by differences in the value range of elements in different second weight matrices, so that the element values ​​of all matrices are in a uniform and comparable interval, providing a standardized data basis for subsequent distance calculation, and avoiding the dominance of similarity evaluation results by a certain matrix element value being too large or too small.

[0098] Optionally, in this embodiment, the logistic function processing refers to using the logistic function to perform a nonlinear transformation on the elements of the normalized second weight matrix. By compressing the extreme differences in element values ​​through nonlinear mapping, the element values ​​that were originally too large or too small converge to the middle of the interval, while preserving the relative differences between elements. This avoids excessive influence of individual extreme values ​​on subsequent distance calculations and enhances the robustness of similarity assessment.

[0099] Optionally, in this embodiment, the linear normalization process after pruning involves first truncating the element values ​​that exceed the preset range after the logical function processing, and then performing a second linear normalization operation on the pruned data to further eliminate extreme values ​​that may still exist after the logical function processing. At the same time, the second normalization ensures the stability of the data range, providing more balanced and reliable data for subsequent calculation of the distance between attention points, and avoiding distortion of similarity assessment due to residual extreme values.

[0100] Understandably, by sequentially performing range normalization, logistic function processing, and pruning followed by linear normalization on the second weight matrix, the differences in data value ranges and the interference of extreme values ​​are gradually eliminated, ensuring that the element values ​​of all second weight matrices are evenly distributed within the [0,1] interval and that their relative differences are stable. This provides a standardized, low-interference data foundation for subsequent calculations of query distance, key distance, and value distance, avoiding biases in attention head similarity assessment due to data quality issues, ensuring accurate and reliable target distance calculations, and providing crucial data support for the conversion between reasonable and unevenly grouped attention head query attention models.

[0101] Through the embodiments provided in this application, it can be understood that by sequentially performing range normalization, logistic function processing, and pruning followed by linear normalization on the second weight matrix, the differences in data value ranges and interference from extreme values ​​are eliminated, ensuring that the element values ​​of all second weight matrices are evenly distributed and relatively stable within the [0,1] interval. This provides a standardized, low-interference data foundation for subsequent calculations of query distance, key distance, and value distance, avoiding bias in attention head similarity evaluation due to data quality issues, ensuring accurate and reliable target distance calculation, and providing crucial data support for the conversion between reasonable and unevenly grouped attention head query attention models.

[0102] As an optional approach, based on the output projection matrix corresponding to the current attention layer, the attention weights corresponding to each attention head in the current attention layer are obtained, including:

[0103] Based on multiple attention heads, the output projection matrix corresponding to the current attention layer is divided to obtain multiple weight sub-matrices corresponding to the current attention layer. The number of multiple weight sub-matrices matches the number of multiple attention heads.

[0104] Based on multiple weight sub-matrices corresponding to the current attention layer, obtain the attention weight corresponding to each attention head in the current attention layer.

[0105] Optionally, in this embodiment, the weight submatrix is ​​a submatrix obtained by dividing the output projection matrix of the current attention layer by rows. Each weight submatrix uniquely corresponds to one attention head, and the number of weight submatrixes perfectly matches the number of attention heads. Its function is to separate the parameters related to a single attention head in the output projection matrix, providing independent parameter units for subsequent calculation of the attention weight of each attention head. The values ​​of the elements in the weight submatrix directly reflect the contribution of the output of the corresponding attention head to the final feature representation.

[0106] Optionally, in this embodiment, the attention weight can be, but is not limited to, an indicator used to quantify the importance of each attention head, calculated based on a weight submatrix, and obtained by calculating the square root of the sum of the squares of all elements in a single weight submatrix. Its function is to clarify the contribution of each attention head to the final output of the model. The larger the attention weight, the higher the proportion of the corresponding attention head's output in the model's feature representation, and the stronger its importance. In subsequent attention head grouping, its merging with other attention heads should be minimized to avoid the loss of important information. This is a key criterion for implementing the importance-priority grouping strategy.

[0107] Optionally, in this embodiment, the output projection matrix is ​​divided by rows according to the number of attention heads and the dimension of a single attention head in the current attention layer, resulting in a weight submatrix whose number perfectly matches the number of attention heads, with each submatrix corresponding to one attention head. The second step involves obtaining attention weights based on each weight submatrix, quantifying the importance of each attention head, and providing crucial importance basis for the grouping strategy of the unevenly grouped query attention model. By dividing the output projection matrix based on the number of attention heads, precise separation of parameters related to a single attention head is achieved, providing independent data units for quantifying the importance of each attention head individually. By calculating and normalizing the Euclidean norm of the weight submatrix, the parameter features are transformed into intuitive attention weights, clarifying the importance differences of each attention head. This ensures that high-importance attention heads are prioritized during grouping, reducing the loss of important information, and laying the foundation for converting the multi-head attention model into an unevenly grouped query attention model and balancing inference cost and inference quality.

[0108] The embodiments provided in this application achieve precise separation of parameters related to a single attention head by dividing the output projection matrix based on the number of attention heads, providing an independent data unit for quantifying the importance of each attention head. By calculating and normalizing the Euclidean norm of the weight submatrix, the parameter features are transformed into intuitive attention weights, clarifying the differences in importance of each attention head. This ensures that high-importance attention heads are prioritized for protection during grouping, reducing the loss of important information. This lays the foundation for converting multi-head attention models into non-uniform grouping query attention models and balancing inference cost and inference quality.

[0109] As an optional approach, based on multiple weight sub-matrices corresponding to the current attention layer, the attention weights corresponding to each attention head in the current attention layer are obtained, including:

[0110] Traverse multiple weight submatrices and take the square root of the sum of squares of all elements in the current weight submatrix to obtain the attention weights corresponding to the current attention head, where the current attention head matches the current weight submatrix.

[0111] Optionally, in this embodiment, multiple weight sub-matrices of the current attention layer are selected sequentially, with the number of weight sub-matrices matching the number of attention heads. Each weight sub-matrix corresponds to one attention head. Two operations are performed on each selected current weight sub-matrix: First, all elements of the matrix are traversed, the square of each element is calculated, and the sum is obtained to get the sum of squares of the elements. Second, the square root of the sum of squares is taken to obtain the Euclidean norm of the current weight sub-matrix. Finally, the norm is normalized to obtain the attention weights corresponding to the current attention heads that match the current weight sub-matrix. The normalization operation can ensure that the weights of different attention heads are in the same comparable range, intuitively reflecting the differences in importance.

[0112] Understandably, by traversing multiple weight submatrices, the importance of all attention heads is ensured to be quantified one by one without omission. By taking the square root of the sum of the squares of the elements in the current weight submatric, the overall contribution strength of the weight submatric is accurately captured, highlighting the influence of high-value elements while eliminating the interference of matrix size on the quantification results. Attention weights are obtained through normalization, ensuring that the importance indicators of different attention heads can be directly compared. This provides accurate and comparable importance criteria for attention head grouping, ensuring that high-importance attention heads are prioritized during grouping, reducing the loss of important information, and laying a crucial foundation for converting multi-head attention models into non-uniform grouping query attention models, maintaining high-quality inference performance while reducing inference costs.

[0113] The embodiments provided in this application ensure that the importance of all attention heads is quantified one by one without omission by traversing multiple weight sub-matrices; by taking the square root of the sum of the squares of the elements of the current weight sub-matrix, the overall contribution intensity of the weight sub-matrix is ​​accurately captured, highlighting the influence of high-value elements while eliminating the interference of matrix size on the quantification results; and attention weights are obtained through normalization, ensuring that the importance indicators of different attention heads can be directly compared. This achieves the technical effect of providing accurate and comparable importance criteria for attention head grouping, ensuring priority protection of high-importance attention heads during grouping, reducing the loss of important information, and laying a key foundation for converting multi-head attention models into non-uniform grouping query attention models while maintaining high-quality inference performance.

[0114] As an optional approach, the method is characterized by grouping multiple attention heads based on all target distances corresponding to the current attention layer and the attention weights corresponding to each attention head in the current attention layer, to obtain at least one target attention head group corresponding to the current attention layer, including:

[0115] Obtain at least one initial attention head group corresponding to the current attention layer, wherein the initial attention head group includes at least one attention head;

[0116] Obtain the target loss value corresponding to at least one attention head group, wherein the target loss value is determined based on at least one target distance and the attention weight corresponding to each attention head;

[0117] Regroup each attention head in the current attention layer until the target loss value reaches the preset convergence condition.

[0118] Optionally, in this embodiment, the initial attention head group is a set of initial grouping schemes of attention heads in the current attention layer. Each initial attention head group contains at least one attention head, and the total number of attention heads in all initial attention head groups is consistent with the total number of attention heads in the current attention layer. This provides initial input for the optimization process of attention head grouping and serves as the starting point for subsequent calculation of the target loss value and adjustment of the grouping scheme. The initial grouping scheme is generated by random allocation to ensure that the subsequent optimization process can search for the optimal grouping from diverse starting points and avoid getting trapped in local optima.

[0119] The target loss value is a core indicator used to quantify the quality of the current attention head grouping scheme. It provides a quantitative judgment standard for optimizing the grouping scheme. The smaller the target loss value, the more the grouping scheme conforms to the principle of high intra-group similarity and fewer merging of high-importance heads. It is the core basis for guiding grouping optimization.

[0120] Optionally, in this embodiment, the preset convergence condition is a standard used to determine whether the attention head grouping optimization process has stopped. It may include, but is not limited to, two situations: First, the temperature parameter used in the optimization process drops below a preset threshold, such as the temperature T in the simulated annealing algorithm; Second, after a preset number of consecutive optimization iterations, the target loss value has not improved at all. This avoids the grouping optimization process from getting stuck in infinite iteration, ensuring that a stable and high-quality grouping scheme is obtained within a reasonable time. At the same time, it balances the optimization effect and computational efficiency, ensuring that the grouping scheme is good enough while avoiding the waste of computational resources caused by excessive iteration.

[0121] Optionally, in this embodiment, the total number of attention heads h and the preset number of groups g in the current attention layer are first determined. Then, the h attention heads are divided into g non-overlapping groups by random allocation. Each group contains at least one attention head, and finally g initial attention head groups are obtained. The attention head set of all groups covers all attention heads in the current layer.

[0122] Next, information about all groups in the current attention head grouping scheme is collected: first, the target distance between all pairs of attention heads in each group; second, the attention weight corresponding to each attention head. Then, the target loss value corresponding to the current grouping scheme is calculated.

[0123] Then, the simulated annealing algorithm is used to iteratively optimize the current grouping scheme. First, the target loss value is calculated based on the current grouping. If the loss value does not reach the convergence condition, the grouping is re-organized until the convergence condition is reached.

[0124] Understandably, randomly generating initial attention head groups provides diverse starting points for grouping optimization, avoiding local optima; calculating the target loss value by fusing target distance and attention weights achieves objective quantification of the grouping scheme; and iteratively regrouping using simulated annealing algorithm combined with preset convergence conditions gradually finds the optimal grouping scheme while avoiding over-iteration. This achieves a reasonable grouping scheme for the current attention layer, grouping similar heads together and minimizing the merging of important heads, thus reducing information loss caused by grouping.

[0125] Through the embodiments provided in this application, it can be understood that by randomly generating initial attention head groups, diverse starting points are provided for grouping optimization, avoiding local optima; by fusing target distance and attention weights to calculate the target loss value, objective quantification of the grouping scheme is achieved; and by iteratively regrouping using simulated annealing algorithm combined with preset convergence conditions, the optimal grouping scheme is gradually found while avoiding over-iteration. This achieves a reasonable grouping scheme for the current attention layer, grouping similar heads together and minimizing the merging of important heads, thus reducing information loss caused by grouping.

[0126] As an optional approach, the target loss value corresponding to at least one attention head group is obtained, including:

[0127] Iterate through each initial attention head group and sum the initial distances that match at least one target distance of the current initial attention head group to obtain the initial distance sum that matches the current initial attention head group;

[0128] The first loss value is obtained based on the sum of all initial distances, and the second loss value is obtained based on the sum of all initial distances and the attention weights corresponding to each attention head.

[0129] Based on the first loss value and the second loss value, obtain the target loss value corresponding to at least one initial attention head group.

[0130] Optionally, in this embodiment, the initial distance sum is a value obtained by summing the target distances between all pairs of attention heads in a single initial attention head group. It quantifies the overall similarity of attention heads within a single initial attention head group. The smaller the initial distance sum, the higher the similarity between pairs of attention heads within the group and the stronger the consistency of features within the group, thus providing basic data units for subsequent calculation of the first loss value.

[0131] Optionally, in this embodiment, the first loss value may be, but is not limited to, the value obtained by summing the initial distances of all current initial attention head groups. It is a global quantification of the similarity loss of all initial attention head groups. The smaller the first loss value, the higher the overall similarity of the attention heads in all groups.

[0132] Optionally, in this embodiment, the second loss value is an importance penalty term calculated based on all initial distances and corresponding grouping structures, combined with the attention weight of each attention head. It is a loss of the grouping scheme in terms of the principle of protecting important heads. The smaller the second loss value, the fewer heads the group of the attention heads with high importance belongs to, and the lower the risk of loss of important information due to merging.

[0133] Optionally, in this embodiment, all current initial attention head groups are selected in sequence. For each current initial attention head group, the pre-calculated target distances between all attention heads in the group are collected first, and then all these target distances are added together to obtain the initial distance sum specific to the group, ensuring that each initial attention head group has a corresponding initial distance sum.

[0134] Next, when calculating the first loss value, the initial distances of all initial attention head groups are summed to obtain the global first loss value; when calculating the second loss value, the number of heads and the maximum attention weight within each initial attention head group are first determined, then the product of the number of heads in each group and the maximum attention weight is calculated, and finally the products of all groups are summed to obtain the second loss value.

[0135] Finally, based on the first loss value and the second loss value, the target loss value corresponding to at least one initial attention head group is obtained.

[0136] Understandably, by traversing the initial attention head groups and calculating the initial distance sum, the similarity of a single group is quantified; by calculating the first and second loss values, the grouping loss is evaluated from the dimensions of similarity and importance, respectively; and by integrating the two loss values ​​to obtain the target loss value, a comprehensive quantification of the grouping scheme is achieved. This provides an objective and comprehensive criterion for judging the quality of the initial attention head groups, offering a clear direction for subsequent iterative optimization of the grouping scheme (such as adjusting groupings using simulated annealing), and ensuring that the final grouping scheme balances similarity and importance.

[0137] The embodiments provided in this application quantify the similarity of a single group by traversing the initial attention head groups and calculating the initial distance sum; evaluate the grouping loss from the dimensions of similarity and importance by calculating the first and second loss values ​​respectively; and achieve comprehensive quantification of the grouping scheme by integrating the two loss values ​​to obtain the target loss value. This provides an objective and comprehensive criterion for judging the quality of the initial attention head groups, provides a clear direction for subsequent iterative optimization of the grouping scheme (such as adjusting the grouping using simulated annealing), and ensures that the final grouping scheme takes into account both similarity and importance.

[0138] As an optional approach, based on the first loss value and the second loss value, a target loss value corresponding to at least one initial attention head group is obtained, including:

[0139] Obtain the adjustment coefficient, which is used to control the importance of the second loss value;

[0140] The target loss value is determined by summing the first loss value with the product of the adjustment coefficient and the second loss value.

[0141] Optionally, in this embodiment, the adjustment coefficient is a parameter used to balance the importance of the first and second loss values ​​in the target loss value, and its value range is λ>0. Its core function is to flexibly adjust the contribution ratio of the two loss values ​​to the target loss value according to the application scenario and demand preferences of the target multi-head attention model. If the model pays more attention to inference quality, the adjustment coefficient can be increased; if it pays more attention to grouping efficiency, the adjustment coefficient can be decreased, ensuring that the calculation of the target loss value can adapt to the optimization needs of different models.

[0142] To further illustrate, if the target multi-head attention model is used for medical text analysis, which requires extremely high inference accuracy and where information from important attention heads cannot be lost, the adjustment coefficient can be set to 1.5. In this case, the second loss value has a higher weight in the target loss value, and the merging of highly important attention heads will be reduced more preferentially during group optimization. If the model is used for general text classification, which requires moderate inference accuracy, intra-group similarity can be given appropriate priority. The adjustment coefficient λ can be set to 0.8, which reduces the weight of the second loss value and focuses more on reducing information loss by improving intra-group similarity.

[0143] Optionally, in this embodiment, the specific value of the adjustment coefficient is determined based on the core task, application scenario, and performance requirements of the target multi-head attention model.

[0144] Next, the calculated first and second loss values ​​are obtained. Then, the product of the adjustment coefficient and the second loss value is calculated. Finally, the first loss value is added to this product to obtain the target loss value. The smaller the target loss value, the better the overall performance of the grouping scheme in terms of both similarity and importance protection.

[0145] Understandably, by obtaining appropriate adjustment coefficients based on model requirements, the calculation of the target loss value achieves flexibility for different scenarios. By weighting the second loss value with the adjustment coefficients and summing it with the first loss value, the losses from both similarity and importance protection dimensions are integrated, ensuring that the target loss value guides the grouping scheme towards optimization in the direction required by the model. This achieves the goal of enabling the grouping scheme to meet the model's core requirements for inference quality or computational efficiency while also considering performance in another dimension.

[0146] The embodiments provided in this application achieve flexible adaptation of the target loss value calculation to different scenarios by obtaining suitable adjustment coefficients according to model requirements. By weighting the second loss value with the adjustment coefficients and summing it with the first loss value, the losses of both similarity and importance protection dimensions are integrated, ensuring that the target loss value can guide the grouping scheme to optimize in the direction required by the model. Thus, the grouping scheme can meet the model's core requirements for inference quality or computational efficiency while also taking into account the performance of another dimension.

[0147] As an optional approach, the attention heads in the current attention layer are regrouped until the target loss value reaches a preset convergence condition, including:

[0148] The first attention head in the first attention head group is moved to the second attention head group to obtain at least one current attention head group, wherein at least one initial attention head group includes the first attention head group and the second attention head group, and the first attention head group includes multiple attention heads;

[0149] Based on at least one current attention head group, at least one target attention head group is determined.

[0150] Optionally, in this embodiment, the first attention head group is any one of at least one initial attention head group, and this group contains multiple attention heads, providing movable attention heads and a basis for generating new grouping schemes. Since this group contains multiple attention heads, removing one still ensures that there are remaining attention heads in the group, avoiding empty groups and ensuring the effectiveness of the grouping scheme.

[0151] Optionally, in this embodiment, the second attention head group is any group different from the first attention head group among at least one initial attention head group. It is the target group for the attention head movement operation, receives the attention heads moved from the first attention head group, and cooperates with the first attention head group to complete the adjustment of the grouping structure and generate a new grouping scheme, thereby providing new samples for subsequent evaluation of the grouping quality and optimization of the grouping scheme.

[0152] Optionally, in this embodiment, the current attention head group is a new set of attention head groups formed by moving the first attention head in the first attention head group to the second attention head group. The total number of attention heads in the current attention head group is the same as the total number of the initial attention head group, and each group contains at least one attention head. This serves as an intermediate candidate scheme in the grouping optimization process and provides a basis for calculating the target loss value and judging whether the grouping is better. By continuously generating the current attention head group and evaluating its merits, the grouping scheme is gradually approached.

[0153] Optionally, in this embodiment, firstly, a group containing multiple attention heads is selected from at least one initial attention head group as a first attention head group, and then another different group is selected as a second attention head group; then, an attention head is randomly selected from the first attention head group as the first attention head, removed from the first attention head group, and added to the second attention head group; finally, a new set of attention head groups is formed as the current attention head group, and it must be ensured that each group in the current attention head group contains at least one attention head. Next, based on at least one current attention head group, at least one target attention head group is determined.

[0154] Understandably, by moving attention heads from the first attention head group to the second attention head group, flexible adjustments to the grouping structure are achieved, providing diverse candidate solutions for grouping optimization. Through iterative evaluation of the current attention head group and selection of the optimal solution, it is ensured that the final target attention head group minimizes the target loss value, satisfying the principles of high intra-group similarity and minimal merging of highly important heads. This provides the optimal grouping basis for unevenly grouped query attention models, reducing information loss due to unreasonable grouping, and maximizing the maintenance of model inference quality while reducing model inference costs.

[0155] The embodiments provided in this application achieve flexible adjustment of the grouping structure by moving attention heads from the first attention head group to the second attention head group, providing diverse candidate solutions for grouping optimization. By iteratively evaluating the current attention head group and selecting the optimal solution, it ensures that the final target attention head group minimizes the target loss value and meets the principles of high intra-group similarity and minimal merging of highly important heads. This provides the optimal grouping basis for unevenly grouped query attention models, reduces information loss caused by unreasonable grouping, and maximizes the maintenance of model inference quality while reducing model inference cost.

[0156] As an optional approach, based on at least one current attention head group, at least one target attention head group is determined, including:

[0157] Obtain the target loss value corresponding to at least one current attention head group, and determine the difference between the target loss value corresponding to at least one current attention head group and the target loss value corresponding to at least one initial attention head group as the loss value difference;

[0158] If the difference in loss values ​​is less than or equal to zero, at least one current attention head group is identified as at least one target attention head group;

[0159] If the difference in loss values ​​is greater than zero, the target probability is obtained based on the difference in loss values, where the target probability is used to indicate the probability of identifying at least one current attention head group as at least one target attention head group;

[0160] If the target probability satisfies the condition of no preset probability, the second attention head in the third attention head group is moved to the fourth attention head group, wherein at least one current attention head group includes the third attention head group and the fourth attention head group, and the third attention head group includes multiple attention heads;

[0161] If the target probability meets the preset probability condition, at least one current attention head group is determined as at least one target attention head group.

[0162] Optionally, in this embodiment, the loss difference is the difference between the target loss value corresponding to the current attention head group and the target loss value corresponding to the initial attention head group, which can quantify the advantages and disadvantages of the current grouping scheme compared to the initial grouping scheme.

[0163] Optionally, in this embodiment, the target probability is the probability value used to determine whether to accept the current attention head group when the difference in loss values ​​is greater than zero. It can be calculated based on the probability formula of the simulated annealing algorithm, but is not limited to, to avoid the grouping optimization getting stuck in a local optimum. Even if the current grouping scheme is temporarily inferior, there is still a certain probability of acceptance, reserving the possibility of exploring a better grouping scheme in the future. The higher the temperature, the greater the target probability and the higher the possibility of accepting the temporarily inferior scheme; the lower the temperature, the smaller the target probability and the lower the possibility of accepting the temporarily inferior scheme.

[0164] Optionally, in this embodiment, the preset probability condition is a standard used to determine whether to temporarily degrade the current attention head group. Specifically, it generates a random number r between 0 and 1. If r is less than the target probability, the preset probability condition is met; if r is greater than the target probability, the preset probability condition is not met.

[0165] Optionally, in this embodiment, the third attention head group is a source group used to readjust the grouping when the target probability does not meet the preset probability condition. It belongs to any group in the current attention head group and contains multiple attention heads. Its function is similar to that of the first attention head group. It provides movable attention heads as second attention heads, provides operation objects for generating new current attention head groups, and ensures that the grouping optimization can continuously explore new solutions and avoid optimization stagnation due to the rejection of a single temporarily inferior solution.

[0166] Optionally, in this embodiment, the fourth attention head group can be, but is not limited to, a group that receives the target group moved from the third attention head group when the target probability does not meet the preset probability condition. This target group belongs to any group in the current attention head group that is different from the third attention head group. Its function is to cooperate with the third attention head group to complete the new grouping adjustment, generate a new current attention head group, and ensure that the grouping optimization process can continue until an optimal solution that meets the convergence condition is found.

[0167] Optionally, in this embodiment, a method for iteratively optimizing and determining a target attention head group based on the difference in loss values ​​and probability judgment is provided. The core process consists of five steps: First, calculate the difference in loss values ​​between the current attention head group and the benchmark group to quantify the quality of the current group; Second, if the difference is ≤0, accept the current group as the target group; Third, if the difference is >0, calculate the target probability of accepting the current suboptimal group; Fourth, if the probability does not meet the preset conditions, move the attention head again to generate a new current group; Fifth, if the probability meets the preset conditions, accept the current suboptimal group as the target group.

[0168] Understandably, calculating the difference in loss values ​​clarifies the merits of the current grouping compared to the baseline grouping, providing an objective basis for acceptance or rejection; accepting better groupings ensures optimization efficiency; calculating target probabilities and probability judgments suggests the possibility of retaining temporarily inferior groupings, avoiding local optima; and readjusting groupings ensures continuous optimization progress. This achieves the goal of exploring globally superior target attention head groups while maintaining grouping optimization efficiency, ensuring that the grouping scheme satisfies both high intra-group similarity and minimal merging of highly important heads.

[0169] The embodiments provided in this application clarify the merits of the current grouping compared to the baseline grouping by calculating the difference in loss values, providing an objective basis for acceptance or rejection; ensuring optimization efficiency by accepting better groups; avoiding local optima by calculating target probabilities and probability judgments to retain temporarily inferior groups; and ensuring continuous optimization by readjusting groupings. This achieves the goal of exploring globally superior target attention head groups while maintaining grouping optimization efficiency, ensuring that the grouping scheme satisfies both high intra-group similarity and minimal merging of highly important heads.

[0170] As an optional approach, based on at least one target attention head group corresponding to the current attention layer, the target multi-head attention model is transformed to obtain the target model, including:

[0171] Based on the elements in the key weight matrix corresponding to each attention head in the fifth attention head group, the shared key weight matrix corresponding to the fifth attention head group is obtained. The first target element in the shared key weight matrix is ​​the average value of the elements at the same position as the first target element in the key weight matrix corresponding to each attention head.

[0172] Replace the key weight matrix of all attention heads included in the fifth attention head group with a shared key weight matrix;

[0173] Based on the elements in the value weight matrix corresponding to each attention head in the sixth attention head group, the shared value weight matrix corresponding to the sixth attention head group is obtained. The second target element in the shared value weight matrix is ​​the average value of the elements at the same position as the second target element in the value weight matrix corresponding to each attention head.

[0174] Replace the value weight matrix of all attention heads contained in the sixth attention head group with a shared value weight matrix.

[0175] Optionally, in this embodiment, the shared key weight matrix is ​​a key weight matrix generated based on the second key weight matrices of all attention heads in the fifth attention head group, which is shared by all attention heads in the group. Its dimension is the same as that of a single second key weight matrix, and the first target element value at any position is the arithmetic mean of the elements at the same position in all second key weight matrices in the group. This replaces the independent second key weight matrices of all attention heads in the group, so that all attention heads in the group share the same key matrix, reducing the number of KV caches and reducing the consumption of hardware video memory or memory during model inference. At the same time, because the similarity of the heads in the group is high, the average generated shared matrix can retain the original key features to the maximum extent and reduce the loss of inference quality.

[0176] Optionally, in this embodiment, the shared value weight matrix can be, but is not limited to, a value weight matrix generated based on the second value weight matrices of all attention heads in the sixth attention head group, which is shared by all attention heads in the group. The second target element value at any position is the arithmetic mean of the elements at the same position in all second value weight matrices in the group, replacing the independent second value weight matrices of all attention heads in the group. This allows attention heads in the group to share the same set of value matrices, further reducing the number of KV caches and lowering the inference cost. At the same time, because the heads in the group have high similarity, the shared matrix can retain the original value features and maintain the inference quality.

[0177] It should be noted that this embodiment provides a method for generating a shared key-value weight matrix for a target attention head group and replacing the original independent matrices. The core process consists of four steps: First, based on the second key weight matrices of all attention heads in the fifth attention head group, a shared key weight matrix is ​​generated by averaging the element positions; Second, the independent second key weight matrices of all attention heads in the group are replaced with the shared key weight matrix; Third, based on the second value weight matrices of all attention heads in the sixth attention head group, a shared value weight matrix is ​​generated by averaging the element positions; Fourth, the independent second value weight matrices of all attention heads in the group are replaced with the shared value weight matrix.

[0178] Understandably, by generating shared key-value weight matrices for the fifth and sixth attention head groups, the number of key-value caches is reduced from the total number of attention heads to the target number of attention head groups, while ensuring feature consistency within the group. Replacing independent matrices with shared matrices directly reduces the hardware memory or GPU memory usage of the key-value cache during model inference. This achieves a significant reduction in inference cost while maintaining model inference quality, enabling the transformed unevenly grouped query attention model to adapt to resource-constrained scenarios.

[0179] The embodiments provided in this application generate shared key-value weight matrices for the fifth and sixth attention head groups, reducing the number of KV caches from the total number of attention heads to the target number of attention head groups while ensuring feature consistency within the group. By replacing independent matrices with shared matrices, the hardware memory or system memory usage of KV caches during model inference is directly reduced. This achieves a significant reduction in inference cost while maintaining model inference quality, enabling the transformed uneven grouping query attention model to adapt to resource-constrained scenarios.

[0180] As an optional approach, after transforming the target multi-head attention model based on at least one target attention head group corresponding to the current attention layer to obtain the target model, it includes at least one of the following:

[0181] Obtain the first prompt text and input the prompt text into the target model to obtain the first result output by the target model, wherein the first result matches the prompt text;

[0182] The image information and the second prompt text are acquired and input into the target model to obtain a second result output by the target model, wherein the second result matches the image information and the second prompt text.

[0183] Optionally, in this embodiment, the first prompt text may be, but is not limited to, text-based input information entered into the target model to guide the model in generating specific results. Its content must be compatible with the model's core task. Its function is to provide the target model with clear task instructions or contextual information, enabling the model to output expected text results based on the input prompt logic. It is the core input carrier for the target model in pure text task scenarios.

[0184] Optionally, in this embodiment, the first result is the text output by the target model after receiving the first prompt text, based on its own attention mechanism and weight parameters, and after inference calculation. This result matches the task requirements and contextual information of the first prompt text. Its purpose is to demonstrate the reasoning ability of the target model in a pure text task scenario and to verify the effectiveness of the model after conversion. If the first result can accurately respond to the requirements of the first prompt text, it indicates that the model maintains good text processing performance while reducing inference costs.

[0185] Optionally, the image information may be, but is not limited to, non-textual input data input into the target model. It usually exists in the form of an image feature matrix and can be used in conjunction with the second cue text to guide the model in processing cross-modal tasks, providing the target model with visual dimension information, enabling the model to combine text cue to achieve cross-modal information understanding and result generation.

[0186] Optionally, in this embodiment, the second prompt text is a text instruction input into the target model in conjunction with the image information. It clarifies the specific requirements of the cross-modal task, and its content must be compatible with the scene of the image information and the cross-modal processing capabilities of the target model. Its function is to provide textual guidance for the cross-modal task to the target model, enabling the model to combine the visual features of the image information with the task requirements of the text prompt to generate accurately matched cross-modal results, thus avoiding output deviations caused by ambiguous task instructions.

[0187] Optionally, in this embodiment, a first prompt text that meets the task requirements is constructed or obtained according to the core text task of the target model; then the first prompt text is converted into an input format that the target model can accept; the processed input feature matrix is ​​input into the target model, and the model performs inference calculation through a non-uniform grouping query attention mechanism; finally, a first result that matches the task requirements and context of the first prompt text is output.

[0188] Optionally, in this embodiment, image information that the target model can process is obtained, and a second prompt text is also obtained; then, cross-modal input processing is performed on the image information and the second prompt text; the cross-modal input matrix is ​​input into the target model, and the model completes inference by adapting to the cross-modal non-uniform grouping query attention mechanism; finally, a second result that matches both the visual content of the image information and the task requirements of the second prompt text is output.

[0189] To illustrate further, suppose the target model is a cross-modal image-text description model, and the acquired image information is "a child is drawing with colored pens at a desk." The second prompt text is "please describe in detail the actions of the person in the image, the environment, and the items used." The target model then outputs the second result: "The person in the image is a child who is sitting at a desk, holding a colored pen in his right hand and drawing on paper. The environment is an indoor desk scene with drawing paper and a colored pen case on the desk. The items used include colored pens, drawing paper, and a desk."

[0190] It should be noted that by implementing the target model in inference applications for pure text tasks, we can verify the performance effectiveness of the model after conversion in basic text scenarios, ensuring that the text processing capability is not lost while reducing inference costs; by extending the target model to cross-modal tasks, we can verify the model's feature fusion and inference capabilities in multi-input type scenarios, thereby expanding the application scope of the model.

[0191] Through the embodiments of this application, by implementing the inference application of the target model in pure text tasks, the performance effectiveness of the model after conversion in basic text scenarios can be verified, ensuring that the inference cost is reduced without losing the text processing capability; by extending the target model to cross-modal tasks, the feature fusion and inference capabilities of the model in multi-input type scenarios can be verified, thereby expanding the application scope of the model.

[0192] As an alternative approach, the aforementioned multi-head attention model conversion method is applied to a scenario where a multi-head attention model is converted to a non-uniform grouping query attention model. In this scenario, the MHA model has already been trained, but the hardware configuration for inference is insufficient, requiring a reduction in inference cost while ensuring that the inference quality does not degrade too much. This issue has attracted significant attention in resource-constrained edge computing scenarios and multi-user concurrent access scenarios.

[0193] One method provided in this embodiment is to convert the trained MHA model into GQA. The key to this process is how to group multiple attention heads. There are currently two grouping strategies: (1) grouping adjacent heads together; (2) expanding the corresponding part of each head in the weight matrix into a vector, and then grouping them according to the cosine similarity between the vectors, that is, grouping similar attention heads into the same group. After the grouping is determined, the average value of the corresponding parts in the weight matrix of a group is taken, and the average value is used as the shared KV of the group.

[0194] However, this technical solution has the following problems:

[0195] (1) GQA is average grouping, but the similarity of the weight matrix after training may not be suitable for average grouping. Forced average grouping will increase the loss of information.

[0196] (2) Current methods can only choose one of the three when calculating similarity, that is, grouping based on the similarity of Q, K, or V, and cannot take into account the similarity of all three.

[0197] (3) The role of the output projection matrix is ​​not considered in the grouping process. The parameters of this matrix represent which head is more important. The existing grouping method does not consider the differences in importance of different heads.

[0198] Therefore, this embodiment provides a method for converting from a multi-head attention model to a non-uniform grouping query attention model, which can reduce the computational cost of large models with high computational costs during the inference stage. In the grouping strategy, the traditional uniform grouping is changed to non-uniform grouping, while taking into account the similarity of the three weight matrices QKV and the importance of the different heads represented by the output projection matrix, so as to achieve more reasonable grouping.

[0199] like Figure 3 The diagram shown is a block diagram of this embodiment. A loss function is constructed by calculating the distance between attention heads and the importance based on the output projection matrix, and the annealing search algorithm is simulated. The distance calculation between attention heads includes Q-weight matrix distance calculation, K-weight matrix distance calculation, and V-weight matrix distance calculation.

[0200] Let the total number of query heads be The number of groups is In the MHA model, the attention head is segmented as follows:

[0201] Q-segmentation is (The dimension of each submatrix is...) );

[0202] K-segmentation (The dimension of each submatrix is...) );

[0203] V is divided into (The dimension of each submatrix is...) ).

[0204] What we need to do is... Each attention point is divided into The grouping strategy proposed in this embodiment is described in four parts: distance calculation between attention heads, importance calculation of attention heads, loss function design, and search algorithm.

[0205] First, we need to calculate the distance between attention heads, for any two Q submatrices. Calculate its Frobenius distance, as shown in formula (1):

[0206] (1);

[0207] in express The first in the matrix Line 1 The elements of the column. The distances between all submatrices of Q are formed. The matrix is ​​given by the matrix $\matrix$, where the element in the $i$-th row and $j$-th column is the Frobenius distance between the $i$-th matrix and the $j$-th matrix. The values ​​in the matrix are normalized to [0,1] using the Min-Max method (normalizing over all elements of the entire matrix), as shown in the formula:

[0208] (2);

[0209] in, Represents the minimum value in the matrix. Indicates the maximum value. Represents the original value. This represents the value after normalization. (Used) This represents the resulting matrix.

[0210] Using the same method, and The calculations are performed to obtain the distance matrix between the normalized submatrices of K. Distance matrix between and normalized submatrices of V .

[0211] Define the distance matrix between different heads as follows: ,in In this embodiment, we take . This represents the distance between attention heads i and j. The larger the distance, the less similar the two attention heads are.

[0212] Next, the importance of attention is calculated. The elements in the matrix can be used to evaluate the importance of each attention head. Intuitively, if an attention head is... If the corresponding element in the matrix has a large value, the result of the multiplication will be amplified.

[0213] Will The matrix is ​​divided into rows. There are several parts, and the dimensions of each part are... Each of these corresponds to a different attention head, with the i-th head corresponding to... Elements of all rows within the range.

[0214] For each attention head i, use its in The importance of corresponding elements in the matrix is ​​calculated using the Euclidean norm, which is the square root of the sum of the squares of all elements, as shown in formula (3):

[0215] (3);

[0216] in Indicates attention head i is The corresponding elements in the matrix. Finally, consider the importance of all attention points. Normalize to [0,1].

[0217] Next, the loss function is designed. The grouping principles include two points: first, similar attention heads (i.e., those with small distances) should be in the same group; second, the group containing important attention heads should contain fewer attention heads (the most important attention heads should even be in their own group). To satisfy these two principles, the loss function is as follows:

[0218] First, consider the distance between attention heads: For group m, let the set of attention heads it contains be... The sum of the distances between all the attention heads in this group is shown in formula (4):

[0219] (4);

[0220] The sum of the distances for all groups is shown in formula (5):

[0221] (5);

[0222] The smaller the value of C, the better the grouping effect.

[0223] Next, consider the importance of attention: design the importance penalty items for all groups as shown in formula (6):

[0224] (6);

[0225] in It is the set of attention heads contained in group m. for The number of attention heads in a group m. The logic behind this term is that if the maximum importance of attention heads in a group m is high and the number of attention heads is large, then this penalty term will be large, forcing groups with high-importance attention heads to reduce the number of attention heads they contain.

[0226] The total loss function is shown in formula (7):

[0227] (7);

[0228] in is the adjustment coefficient ( ), the closer it is to 0, the more attention is paid to the similarity between attention heads, and vice versa, the more attention is paid to the importance of attention heads. In this embodiment, is taken.

[0229] Finally, the search algorithm is as follows. In this embodiment, the simulated annealing algorithm is used for searching. Set the initial temperature , the temperature reduction coefficient , and the steps are as follows:

[0230] (a) Randomly divide attention heads into groups, with at least one attention head in each group, and calculate the value of the loss function .

[0231] (b) At temperature T, repeat the following steps 100 times:

[0232] Randomly select a group with more than 1 attention heads, and randomly select one attention head from it;

[0233] Randomly move this attention head to another group to generate a new grouping scheme, and calculate its loss function ;

[0234] Calculate and the difference between the current loss function value;

[0235] If [[ID=...]] It seems there is an issue with the provided text as some tags are not properly closed or there are consecutive tags without proper separation. Please check and correct the original text for a more accurate translation. The above translation is based on the available and understandable parts. , then accept the new grouping scheme;

[0236] If , then accept the new grouping scheme with a certain probability, as shown in formula (8):

[0237] (8);

[0238] The specific method is to randomly generate a number r between 0 and 1. If r < P, then accept the new grouping scheme; otherwise, retain the original grouping scheme.

[0239] When accepting the scheme, update the value of the loss function .

[0240] (c) After completing 100 iterations in (b), update the temperature . Repeat step (b).

[0241] (d) Stop searching when or there is no improvement under 50 consecutive temperatures .

[0242] When the simulated annealing algorithm stops iterating, it returns the final grouping scheme. Note that the process does not require average grouping. Whether the grouping is average is not a principle in the search process. The algorithm only focuses on reducing the value of the loss function, so the result is not average.

[0243] After obtaining the grouping scheme, the K-weight matrix of each attention head in each group is averaged according to its position to form a shared K-weight matrix for that group, as shown in formula (9):

[0244] (9);

[0245] in This represents the value in the a-th row and b-th column of the shared weight matrix. This represents the set of attention heads contained in group m. The V weight matrix is ​​processed in the same way as the K weight matrix.

[0246] like Figure 4 The flowchart shown is a flowchart of this embodiment, which includes: reading the model file, processing the i-th layer, obtaining the QKV weight matrix and W_O matrix, performing attention head segmentation and grouping, merging the KV matrices in the same group as a shared matrix, and then checking whether all layers have been processed. If so, inference is performed or the model is saved to a file; otherwise, the i-th layer is processed.

[0247] In summary, this invention begins by reading the MHA model file. After processing, the MHA model is converted into a non-uniformly grouped query attention model. The converted model is then used for inference, or saved to a file for subsequent model inference. Specifically, after reading the model, each layer is traversed sequentially (large language models consist of multiple layers, each containing a separate attention mechanism; in this invention, each layer is processed), obtaining the QKV matrix and the matrix. Based on the user-defined number of groups and the number of attention heads in the model, the models are grouped using the method described above, generating a shared KV matrix for each group. Once all layers have been processed, the converted model can be used for inference or saved to a file.

[0248] Through the embodiments provided in this application, the present invention enables collaboration between the hyperconverged platform and the storage service cluster. By defining different upgrade states for upgrade nodes, it controls the upgrade process of each node in the storage cluster. Utilizing the ETCD database event detection mechanism, it can effectively and promptly identify various fault types occurring during the upgrade process. In the event of a single node failure during the upgrade, as long as the hardware is sound, the upgrade process can ensure uninterrupted business I / O and complete normally without manual intervention. When multiple nodes fail, to ensure the security of the storage cluster data, the cluster's storage service needs to be paused. Global data recovery is then performed after each node in the storage cluster restarts with the new version. Once data recovery is complete, the entire cluster upgrade is finished. In this extreme failure scenario, although the storage service is paused, data security is effectively guaranteed, and subsequent upgrades of each node individually are no longer required, shortening the overall online upgrade time. Therefore, the method provided by this invention can automatically handle most faults occurring during the upgrade process while ensuring data security, providing a reliable upgrade service.

[0249] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0250] This embodiment also provides a multi-head attention model conversion device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0251] Figure 5 This is a structural block diagram of a multi-head attention model conversion device according to an embodiment of this application. The device includes:

[0252] The first acquisition unit 502 is used to acquire a target multi-head attention model, wherein the target multi-head attention model includes at least one attention layer, and the attention layer includes multiple attention heads;

[0253] The second acquisition unit 504 is used to traverse at least one attention layer and acquire at least one target distance based on the first query weight matrix, the first key weight matrix and the first value weight matrix corresponding to the current attention layer, wherein the target distance is used to indicate the similarity between the first attention head and the second attention head, and the multiple attention heads include the first attention head and the second attention head;

[0254] The third acquisition unit 506 is used to acquire the attention weight of each attention head in the current attention layer based on the output projection matrix corresponding to the current attention layer, wherein the attention weight is used to indicate the importance of the attention head;

[0255] Grouping unit 508 is used to group multiple attention heads based on all target distances corresponding to the current attention layer and the attention weights corresponding to each attention head in the current attention layer, to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group includes at least one attention head;

[0256] The conversion unit 510 is used to convert the target multi-head attention model based on at least one target attention head group corresponding to the current attention layer to obtain the target model.

[0257] As an optional scheme, the second acquisition unit 504 includes: a segmentation module, used to segment the first query weight matrix, the first key weight matrix and the first value weight matrix based on multiple attention heads, to obtain multiple second query weight matrices, multiple second key weight matrices and multiple second value weight matrices corresponding to the current attention layer;

[0258] The first acquisition module is used to acquire the query distance based on the second query weight matrix corresponding to the first attention head, wherein the query distance is used to indicate the similarity between the second query weight matrix corresponding to the first attention head and the second query weight matrix corresponding to the second attention head.

[0259] The second acquisition module is used to acquire the key distance based on the second key weight matrix corresponding to the first attention head, wherein the key distance is used to indicate the similarity between the second key weight matrix corresponding to the first attention head and the second key weight matrix corresponding to the second attention head;

[0260] The third acquisition module is used to acquire the value distance based on the second value weight matrix corresponding to the first attention head, wherein the value distance is used to indicate the similarity between the second value weight matrix corresponding to the first attention head and the second value weight matrix corresponding to the second attention head.

[0261] The fourth acquisition module is used to obtain the target distance between the first query header and the second query header based on the query distance, key distance, and value distance.

[0262] As an optional scheme, the first acquisition module includes: a first determination submodule, used to determine the difference between the element located in the i-th row and j-th column of the second query weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second query weight matrix corresponding to the second attention head as the first element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0263] The second determination submodule is used to determine the query distance by taking the square root of the sum of the squares of the differences of all the first elements.

[0264] As an optional scheme, the second acquisition module includes: a third determining submodule, used to determine the difference between the element located in the i-th row and j-th column of the second key weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second key weight matrix corresponding to the second attention head as the second element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0265] The fourth determination submodule is used to determine the bond distance by taking the square root of the sum of the squares of the differences of all second elements.

[0266] As an optional solution, the third acquisition module includes: a fifth determination submodule, used to determine the difference between the element located in the i-th row and j-th column of the second value weight matrix corresponding to the first attention head and the element located in the i-th row and j-th column of the second value weight matrix corresponding to the second attention head as the third element difference, where i is an integer greater than or equal to 1 and j is an integer greater than or equal to 1.

[0267] The sixth submodule is used to determine the value distance by taking the square root of the sum of the squares of the differences of all third elements.

[0268] As an optional solution, the fourth acquisition module includes: a first acquisition submodule, used to acquire a first weight coefficient, a second weight coefficient, and a third weight coefficient based on the type of the target multi-head attention model, wherein the first weight coefficient is used to indicate the importance of query distance, the second weight coefficient is used to indicate the importance of key distance, and the third weight coefficient is used to indicate the importance of value distance;

[0269] The seventh determination submodule is used to sum the product of the query distance and the first weight coefficient, the product of the key distance and the second weight coefficient, and the product of the value distance and the third weight coefficient to obtain the target distance between the first query head and the second query head.

[0270] As an optional solution, the segmentation module includes: a first processing submodule, used to perform range normalization processing on multiple second query weight matrices, multiple second key weight matrices and multiple second value weight matrices corresponding to the current attention layer;

[0271] The second processing submodule is used to perform logical function processing on the multiple second query weight matrices, multiple second key weight matrices and multiple second value weight matrices corresponding to the current attention layer.

[0272] The third processing submodule is used to perform linear normalization processing on the multiple second query weight matrices, multiple second key weight matrices and multiple second value weight matrices corresponding to the current attention layer after pruning.

[0273] As an optional solution, the third acquisition unit 506 includes: a partitioning module, used to partition the output projection matrix corresponding to the current attention layer based on multiple attention heads to obtain multiple weight sub-matrices corresponding to the current attention layer, wherein the number of multiple weight sub-matrices matches the number of multiple attention heads;

[0274] The fifth acquisition module is used to obtain the attention weight of each attention head in the current attention layer based on multiple weight sub-matrices corresponding to the current attention layer.

[0275] As an optional approach, the fifth acquisition module includes: a processing submodule, which is used to traverse multiple weight submatrices and take the square root of the sum of squares of all elements in the current weight submatric to obtain the attention weight corresponding to the current attention head, wherein the current attention head is matched with the current weight submatric.

[0276] As an optional solution, the grouping unit 508 includes: a sixth acquisition module, used to acquire at least one initial attention head group corresponding to the current attention layer, wherein the initial attention head group includes at least one attention head;

[0277] The seventh acquisition module is used to acquire the target loss value corresponding to at least one attention head group, wherein the target loss value is determined based on at least one target distance and the attention weight corresponding to each attention head;

[0278] The grouping module is used to regroup the attention heads in the current attention layer until the target loss value reaches the preset convergence condition.

[0279] As an optional approach, the seventh acquisition module includes: a summation submodule, which is used to traverse each initial attention head group and sum based on at least one target distance that matches the current initial attention head group to obtain the initial distance sum that matches the current initial attention head group;

[0280] The second acquisition submodule is used to acquire a first loss value based on all initial distances and a second loss value based on all initial distances and the attention weights corresponding to each attention head.

[0281] The third acquisition submodule is used to acquire the target loss value corresponding to at least one initial attention head group based on the first loss value and the second loss value.

[0282] As an optional solution, the third acquisition submodule includes: a first acquisition subunit, used to acquire an adjustment coefficient, wherein the adjustment coefficient is used to control the importance of the second loss value;

[0283] The first determining subunit is used to determine the target loss value by summing the product of the first loss value and the adjustment coefficient and the second loss value.

[0284] As an optional approach, the grouping module includes: a moving submodule, used to move a first attention head in a first attention head group to a second attention head group to obtain at least one current attention head group, wherein at least one initial attention head group includes a first attention head group and a second attention head group, and the first attention head group includes multiple attention heads;

[0285] The eighth determination submodule is used to determine at least one target attention head group based on at least one current attention head group.

[0286] As an optional solution, the eighth determining submodule includes: a second acquisition subunit, used to acquire at least one target loss value corresponding to a current attention head group, and to determine the difference between the target loss value corresponding to at least one current attention head group and the target loss value corresponding to at least one initial attention head group as the loss value difference;

[0287] The second determining subunit is used to determine at least one current attention head group as at least one target attention head group when the difference in loss values ​​is less than or equal to zero.

[0288] The third acquisition subunit is used to acquire the target probability based on the loss value difference when the loss value difference is greater than zero, wherein the target probability is used to indicate the probability of identifying at least one current attention head group as at least one target attention head group;

[0289] The moving subunit is used to move the second attention head in the third attention head group to the fourth attention head group when the target probability satisfies the condition of no preset probability. At least one current attention head group includes the third attention head group and the fourth attention head group, and the third attention head group includes multiple attention heads.

[0290] The third determining subunit is used to determine at least one current attention head group as at least one target attention head group when the target probability meets the preset probability condition.

[0291] As an optional solution, the conversion unit 510 includes: an eighth acquisition module, used to acquire the shared key weight matrix corresponding to the fifth attention head group based on the elements in the key weight matrix corresponding to each attention head in the fifth attention head group, wherein the first target element in the shared key weight matrix is ​​the average value of the elements at the same position as the first target element in the key weight matrix corresponding to each attention head;

[0292] The first replacement module is used to replace the key weight matrix of all attention heads included in the fifth attention head group with a shared key weight matrix;

[0293] The ninth acquisition module is used to obtain the shared value weight matrix corresponding to the sixth attention head group based on the elements in the value weight matrix corresponding to each attention head in the sixth attention head group. The second target element in the shared value weight matrix is ​​the average value of the elements in the value weight matrix corresponding to each attention head at the same position as the second target element.

[0294] The second replacement module is used to replace the value weight matrix of all attention heads contained in the sixth attention head group with a shared value weight matrix.

[0295] As an optional solution, the device further includes: a fourth acquisition unit, used to acquire the first prompt text, input the prompt text into the target model, and obtain a first result output by the target model, wherein the first result matches the prompt text;

[0296] The fifth acquisition unit is used to acquire image information and second prompt text, and input the image information and the received second prompt text into the target model to obtain a second result output by the target model, wherein the second result matches the image information and the second prompt text.

[0297] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0298] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0299] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0300] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0301] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0302] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0303] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0304] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods in various embodiments of this application.

[0305] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0306] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0307] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A conversion method of a multi-head attention model, the method comprising: The method comprises the following steps: obtaining a target multi-head attention model, wherein the target multi-head attention model comprises at least one attention layer, and the attention layer comprises a plurality of attention heads; traversing at least one of the attention layers, and obtaining at least one target distance based on a first query weight matrix, a first key weight matrix and a first value weight matrix corresponding to the current attention layer, wherein the target distance is used to indicate the similarity between a first attention head and a second attention head, and the plurality of attention heads comprises the first attention head and the second attention head; obtaining an attention weight corresponding to each of the attention heads in the current attention layer based on an output projection matrix corresponding to the current attention layer, wherein the attention weight is used to indicate the importance of the attention head; grouping the plurality of attention heads based on all target distances corresponding to the current attention layer and the attention weight corresponding to each of the attention heads in the current attention layer, to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group comprises at least one attention head; transforming the target multi-head attention model based on at least one target attention head group corresponding to the current attention layer, to obtain a target model; after transforming the target multi-head attention model based on at least one target attention head group corresponding to the current attention layer, to obtain a target model, the method further comprises at least one of the following steps: obtaining a first prompt text, and inputting the prompt text into the target model to obtain a first result output by the target model, wherein the first result matches the prompt text; obtaining image information and a second prompt text, and inputting the image information and the second prompt text into the target model to obtain a second result output by the target model, wherein the second result matches the image information and the second prompt text.

2. The method of claim 1, wherein, obtaining at least one target distance based on a first query weight matrix, a first key weight matrix and a first value weight matrix corresponding to the current attention layer, comprises the following steps: segmenting the first query weight matrix, the first key weight matrix and the first value weight matrix based on the plurality of attention heads, to obtain a plurality of second query weight matrices, a plurality of second key weight matrices and a plurality of second value weight matrices corresponding to the current attention layer; obtaining a query distance based on a second query weight matrix corresponding to the first attention head, wherein the query distance is used to indicate the similarity between the second query weight matrix corresponding to the first attention head and the second query weight matrix corresponding to the second attention head; obtaining a key distance based on a second key weight matrix corresponding to the first attention head, wherein the key distance is used to indicate the similarity between the second key weight matrix corresponding to the first attention head and the second key weight matrix corresponding to the second attention head; obtaining a value distance based on the second value weight matrix corresponding to the first attention head, wherein the value distance is used to indicate the similarity between the second value weight matrix corresponding to the first attention head and the second value weight matrix corresponding to the second attention head; obtaining a target distance between the first attention head and the second attention head based on the query distance, the key distance and the value distance.

3. The method of claim 2, wherein, obtaining a query distance based on the second query weight matrix corresponding to the first attention head, comprising: determining a first element difference value between an element located at the i-th row and the j-th column in the second query weight matrix corresponding to the first attention head and an element located at the i-th row and the j-th column in the second query weight matrix corresponding to the second attention head, wherein i is an integer greater than or equal to 1, and j is an integer greater than or equal to 1; determining the query distance as a value obtained by taking the square root of the sum of squares of all first element difference values.

4. The method of claim 2, wherein, obtaining a key distance based on the second key weight matrix corresponding to the first attention head, comprising: determining a second element difference value between an element located at the i-th row and the j-th column in the second key weight matrix corresponding to the first attention head and an element located at the i-th row and the j-th column in the second key weight matrix corresponding to the second attention head, wherein i is an integer greater than or equal to 1, and j is an integer greater than or equal to 1; determining the key distance as a value obtained by taking the square root of the sum of squares of all second element difference values.

5. The method of claim 2, wherein, obtaining a value distance based on the second value weight matrix corresponding to the first attention head, comprising: determining a third element difference value between an element located at the i-th row and the j-th column in the second value weight matrix corresponding to the first attention head and an element located at the i-th row and the j-th column in the second value weight matrix corresponding to the second attention head, wherein i is an integer greater than or equal to 1, and j is an integer greater than or equal to 1; determining the value distance as a value obtained by taking the square root of the sum of squares of all third element difference values.

6. The method of claim 2, wherein, obtaining a target distance between the first attention head and the second attention head based on the query distance, the key distance and the value distance, comprising: obtaining a first weight coefficient, a second weight coefficient and a third weight coefficient based on the type of the target multi-head attention model, wherein the first weight coefficient is used to indicate the importance of the query distance, the second weight coefficient is used to indicate the importance of the key distance, and the third weight coefficient is used to indicate the importance of the value distance; summing the product of the query distance and the first weight coefficient, the product of the key distance and the second weight coefficient, and the product of the value distance and the third weight coefficient to obtain the target distance between the first attention head and the second attention head.

7. The method of claim 2, wherein, after the first query weight matrix, the first key weight matrix and the first value weight matrix are divided based on a plurality of attention heads to obtain a plurality of second query weight matrices, a plurality of second key weight matrices and a plurality of second value weight matrices corresponding to the current attention layer, further comprising at least one of the following: perform range normalization processing on the plurality of second query weight matrices, the plurality of second key weight matrices, and the plurality of second value weight matrices corresponding to the current attention layer; perform logical function processing on the plurality of second query weight matrices, the plurality of second key weight matrices, and the plurality of second value weight matrices corresponding to the current attention layer; perform clipping and linear normalization processing on the plurality of second query weight matrices, the plurality of second key weight matrices, and the plurality of second value weight matrices corresponding to the current attention layer.

8. The method of claim 1, wherein, Based on the output projection matrix corresponding to the current attention layer, the attention weight corresponding to each of the plurality of attention heads in the current attention layer is obtained, including: Based on the plurality of attention heads, the output projection matrix corresponding to the current attention layer is divided to obtain a plurality of weight sub-matrices corresponding to the current attention layer, wherein the number of the plurality of weight sub-matrices matches the number of the plurality of attention heads. Based on the plurality of weight sub-matrices corresponding to the current attention layer, the attention weight corresponding to each of the plurality of attention heads in the current attention layer is obtained.

9. The method of claim 1, wherein, Based on the plurality of weight sub-matrices corresponding to the current attention layer, the attention weight corresponding to each of the plurality of attention heads in the current attention layer is obtained, including: Iterate through the plurality of weight sub-matrices, and perform square root processing on the sum of squares of all elements in the current weight sub-matrix to obtain the attention weight corresponding to the current attention head, wherein the current attention head matches the current weight sub-matrix.

10. The method of claim 1, wherein, Based on the total target distance corresponding to the current attention layer and the attention weight corresponding to each of the plurality of attention heads in the current attention layer, the plurality of attention heads are grouped to obtain at least one target attention head group corresponding to the current attention layer, including: Obtain at least one initial attention head group corresponding to the current attention layer, wherein the initial attention head group includes at least one attention head. Obtain a target loss value corresponding to at least one attention head group, wherein the target loss value is determined based on at least one of the target distances and the attention weight corresponding to each of the plurality of attention heads. Re-group the plurality of attention heads in the current attention layer until the target loss value reaches a preset convergence condition.

11. The method of claim 10, wherein, Obtaining a target loss value corresponding to at least one attention head group includes: Iterate through each of the initial attention head groups, and perform summation based on at least one of the target distances matching the current initial attention head group to obtain an initial distance sum matching the current initial attention head group. Obtain a first loss value based on all of the initial distance sums, and obtain a second loss value based on all of the initial distance sums and the attention weight corresponding to each of the plurality of attention heads. Obtain a target loss value corresponding to at least one initial attention head group based on the first loss value and the second loss value.

12. The method of claim 11, wherein, Obtaining a target loss value corresponding to at least one initial attention head group based on the first loss value and the second loss value includes: Obtain an adjustment coefficient, wherein the adjustment coefficient is used to control the importance of the second loss value. The sum of the first loss value and the product of the adjustment coefficient and the second loss value is determined as the target loss value.

13. The method of claim 10, wherein, The grouping of the attention heads in the current attention layer is re-performed until the target loss value reaches a preset convergence condition, including: moving a first attention head in a first attention head group to a second attention head group to obtain at least one current attention head group, wherein the at least one initial attention head group includes the first attention head group and the second attention head group, and the first attention head group includes a plurality of attention heads; determining at least one target attention head group based on the at least one current attention head group.

14. The method of claim 13, wherein, determining at least one target attention head group based on the at least one current attention head group, including: obtaining a target loss value corresponding to the at least one current attention head group, and determining a loss value difference between the target loss value corresponding to the at least one current attention head group and a target loss value corresponding to the at least one initial attention head group as the loss value difference; in a case where the loss value difference is less than or equal to zero, determining the at least one current attention head group as the at least one target attention head group; in a case where the loss value difference is greater than zero, obtaining a target probability based on the loss value difference, wherein the target probability is used to indicate a possibility of determining the at least one current attention head group as the at least one target attention head group; in a case where the target probability meets a preset probability condition, moving a second attention head in a third attention head group to a fourth attention head group, wherein the at least one current attention head group includes the third attention head group and the fourth attention head group, and the third attention head group includes a plurality of attention heads; in a case where the target probability meets a preset probability condition, determining the at least one current attention head group as the at least one target attention head group.

15. The method of claim 1, wherein, transforming the target multi-head attention model based on the at least one target attention head group corresponding to the current attention layer to obtain a target model, including: obtaining a shared key weight matrix corresponding to a fifth attention head group based on elements in a key weight matrix corresponding to each attention head in the fifth attention head group, wherein a first target element in the shared key weight matrix is an average value of elements in the same position as the first target element in the key weight matrix corresponding to each attention head; replacing key weight matrices of all attention heads included in the fifth attention head group with the shared key weight matrix; obtaining a shared value weight matrix corresponding to a sixth attention head group based on elements in a value weight matrix corresponding to each attention head in the sixth attention head group, wherein a second target element in the shared value weight matrix is an average value of elements in the same position as the second target element in the value weight matrix corresponding to each attention head; replacing value weight matrices of all attention heads included in the sixth attention head group with the shared value weight matrix.

16. A conversion device of a multi-head attention model, characterized in that, including: The first obtaining unit is configured to obtain a target multi-head attention model, wherein the target multi-head attention model comprises at least one attention layer, and the attention layer comprises a plurality of attention heads; The second obtaining unit is configured to traverse the at least one attention layer, and obtain at least one target distance based on a first query weight matrix, a first key weight matrix and a first value weight matrix corresponding to a current attention layer, wherein the target distance is used to indicate a similarity between a first attention head and a second attention head, and the plurality of attention heads comprise the first attention head and the second attention head; The third obtaining unit is configured to obtain an attention weight corresponding to each of the attention heads in the current attention layer based on an output projection matrix corresponding to the current attention layer, wherein the attention weight is used to indicate an importance of the attention head; The grouping unit is configured to group the plurality of attention heads based on all target distances corresponding to the current attention layer and the attention weight corresponding to each of the attention heads in the current attention layer, to obtain at least one target attention head group corresponding to the current attention layer, wherein the target attention head group comprises at least one attention head. The converting unit is configured to convert the target multi-head attention model based on the at least one target attention head group corresponding to the current attention layer, to obtain a target model. The device is further configured to: After converting the target multi-head attention model based on the at least one target attention head group corresponding to the current attention layer to obtain the target model, the device is further configured to obtain a first prompt text, and input the prompt text into the target model to obtain a first result output by the target model, wherein the first result matches the prompt text; obtain image information and a second prompt text, and input the image information and the second prompt text into the target model to obtain a second result output by the target model, wherein the second result matches the image information and the second prompt text.

17. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when executed by the processor, the computer program implements the steps of the method in any one of claims 1 to 15.

18. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 15.

19. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 15. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 15.

Images