Attention processing method, device, medium and computer program product

Abstract

The present application relates to attention processing method, equipment, medium and computer program product. The method comprises: obtaining a token-independent key normalization calibration factor predetermined based on the root mean square amplitude of the reconstructed key in the calibration sample for the KV head of the attention layer; processing the NoPE key projection of the MLA based on the gain parameter of the key normalization calibration factor and KNorm, so that the token-varying key normalization factor is approximated by the key normalization calibration factor, and the corresponding key normalization scale is combined to the NoPE key projection; when performing autoregressive decoding on the input sequence, the query side transformation is performed on the query of the current token based on the processed NoPE key projection, and the query side transformed query is operated with the potential KV representation in the KV cache to perform attention calculation, wherein the autoregressive decoding does not need to store the token-varying key normalization factor in the KV cache for the historical token.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to attention processing methods, devices, media, and computer program products. Background Technology

[0002] As the size of the Transformer model and the length of the input sequence continue to increase, the model needs to cache the keys and values ​​corresponding to historical words during the autoregressive inference process. The storage space occupied by the KV cache and the memory bandwidth overhead caused by accessing the KV cache have gradually become important factors affecting the inference efficiency of the model.

[0003] Multi-head latent attention (MLA) can use low-dimensional latent key-value representations instead of complete keys and values ​​for caching. By combining partial projection transformations on the key side with the query side, attention computation can be directly applied to the cached latent key-value representations, thereby reducing key-value cache usage and data access overhead during decoding.

[0004] However, some Transformer models also employ query normalization (QNorm) and key normalization (KNorm) for queries and keys, respectively. The normalization factor in KNorm depends on the key of the corresponding term and varies with the term, making it difficult to directly combine the key-side projection with a fixed query-side transformation. If this term-variable normalization is accurately preserved during autoregressive decoding, it may be necessary to explicitly generate reconstructed keys for historical terms, or to additionally store the corresponding normalization factor for each historical term. This increases KV cache usage, storage access overhead, or attention computation costs, hindering the realization of MLA's potential KV cache advantages.

[0005] Therefore, there is a need for an attention processing scheme that can be applied to Transformer models including QNorm and KNorm, and can reduce the caching and data access overhead when using MLA for autoregressive decoding. Summary of the Invention

[0006] The purpose of this invention is to provide attention processing methods, devices, media, and computer program products to solve the problem that when Transformer models including QNorm and KNorm are autoregressively decoded using MLA, the key normalization factor in KNorm, which varies with the lexical, hinders the combination of key-side projections to the query side, thus leading to additional KV caching, storage access, or computational overhead.

[0007] The first aspect of this invention discloses an attention processing method applied to a Transformer model including query normalization (QNorm) and key normalization (KNorm), wherein the attention layer of the Transformer model employs multi-head latent attention (MLA), and the method includes: For the KV head of the attention layer, obtain a word-independent key normalization calibration factor based on the root mean square magnitude of the reconstructed keys in the calibration samples. Based on the key normalization calibration factor and the gain parameter of KNorm, the NoPE key projection of the MLA is processed so that the key normalization factor that varies with the word is approximated by the key normalization calibration factor, and the corresponding key normalization scale is combined with the NoPE key projection. When performing autoregressive decoding on the input sequence, the query of the current word is transformed based on the processed NoPE key projection, and the query transformed by the query side is operated on with the latent KV representation in the KV cache to perform attention calculation.

[0008] According to a first aspect of the invention, the method further includes performing performance recovery processing on the Transformer model, the performance recovery processing comprising: Obtain the pre-transformation model corresponding to the Transformer model, which uses multi-head attention (MHA) or grouped query attention (GQA), as the teacher model; The same input sequence is input into the teacher model and the Transformer model respectively, and the distillation loss is determined based on the difference between the output distribution of the teacher model and the output distribution of the Transformer model. Based on the distillation loss, only the projection parameters corresponding to the query, key, and value in the attention layer, as well as the corresponding QNorm and KNorm parameters, are updated, while the parameters of the attention output projection, word embedding, language model head, and feedforward network remain unchanged.

[0009] According to a first aspect of the invention, the Transformer model further includes a hybrid expert MoE module, the MoE module including a router and multiple experts; The performance recovery process further includes: keeping the parameters of the router and the plurality of experts unchanged to maintain the expert load balancing behavior formed in the pre-training phase, and limiting expert routing changes caused by representation changes in the attention layer based on the distillation loss.

[0010] According to a first aspect of the invention, the performance recovery process further includes: The Transformer model with the first potential KV dimension, after the performance recovery process, is used as the current model; Reduce the latent key-value dimension of the current model to obtain a Transformer model with a second latent key-value dimension, wherein the second latent key-value dimension is smaller than the first latent key-value dimension; Using the model parameters of the current model after reducing the potential KV dimension as the recovery starting point, the performance recovery process is continued to be performed on the Transformer model with the second potential KV dimension. Repeat the process of reducing the potential key-value dimensions and continuing to perform the performance recovery process to obtain the Transformer models corresponding to different potential key-value dimensions.

[0011] According to a first aspect of the present invention, the key normalization calibration factor is determined according to the following steps: for each KV head in each layer of the attention layer, an average value is determined based on the root mean square magnitude of the reconstructed keys corresponding to multiple lexical units in the calibration sample, and the average value is determined as the key normalization calibration factor of the corresponding KV head.

[0012] According to a first aspect of the invention, the attention processing method further includes: before performing rotational position encoding, incorporating the RoPE component of the bond-normalized scale, determined by the bond-normalized calibration factor and the gain parameter of the KNorm, into the RoPE bond projection.

[0013] According to a first aspect of the invention, the MLA is constructed according to the following steps: Based on the contribution of different rotation subspaces to the attention distribution, the rotation subspace used for rotational position encoding RoPE is selected and retained, and its remaining key dimension is used as the NoPE subspace. The key projection and value projection corresponding to the NoPE subspace are combined into a joint KV projection. Based on calibration activation, with the goal of reducing the reconstruction error of the output after calibration activation by the joint KV projection, a low-rank approximation is performed on the joint KV projection to initialize the lower projection, NoPE key projection and value projection of the latent KV representation.

[0014] A second aspect of the present invention discloses an electronic device including a memory storing computer-executable instructions and a processor, wherein when the instructions are executed by the processor, the electronic device implements the attention processing method according to the first aspect of the present invention.

[0015] A third aspect of the present invention discloses a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform an attention processing method according to a first aspect of the present invention.

[0016] A fourth aspect of the present invention discloses a computer program product comprising computer-executable instructions which are executed by a processor to implement the attention processing method according to a first aspect of the present invention.

[0017] The main differences and effects of this invention compared to existing technologies are as follows: In this invention, for the key-value (KV) head of the attention layer, a word-independent key normalization calibration factor, pre-determined based on the root mean square magnitude of the reconstructed keys in the calibration samples, is used to approximate the key normalization factor in KNorm, which originally varied with the word. The corresponding key normalization scale, determined by the key normalization calibration factor and the gain parameter of KNorm, is then combined with the NoPE key projection. This allows the processed NoPE key projection to be combined with the query side, performing a query-side transformation for the current word query during autoregressive decoding, and enabling the transformed query to be directly computed with the latent KV representation in the KV cache. Therefore, it is unnecessary to explicitly generate complete reconstructed keys for each historical word, nor is it necessary to additionally store word-variable KNorm normalization factors in the KV cache for each historical word. Attention calculation can be performed according to the latent KV cache method of MLA, thereby reducing KV cache usage and storage access and memory bandwidth overhead during decoding, and improving autoregressive inference efficiency. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an attention processing method according to an embodiment of this application is shown. Figure 2 A schematic diagram illustrating the QNorm and KNorm adaptation and MLA pre-filling and decoding process for autoregressive decoding according to an embodiment of this application; Figure 3 This diagram illustrates a performance recovery process for a Transformer model converted by MLA according to an embodiment of this application. Figure 4 This is a hardware structure block diagram of an electronic device implementing the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] Example 1

[0021] This embodiment provides an attention processing method applied to a Transformer model including query normalization (QNorm) and key normalization (KNorm), wherein the attention layer of the Transformer model employs multi-head latent attention (MLA). The method includes: S101, for the KV head of the attention layer, obtain a lexical-independent key normalization calibration factor based on the root mean square magnitude of the reconstructed key in the calibration sample. S102, based on the key normalization calibration factor and the gain parameter of the KNorm, the NoPE key projection of the MLA is processed so that the key normalization factor that varies with the word is approximated by the key normalization calibration factor, and the corresponding key normalization scale is combined with the NoPE key projection. S103, during autoregressive decoding of the input sequence, a query-side transformation is performed on the query of the current lexical based on the processed NoPE key-on-key projection, and the query after the query-side transformation is processed in conjunction with the latent KV representation in the KV cache to perform attention calculation. Notably, the autoregressive decoding does not require storing key normalization factors that change with the lexical in the KV cache for historical lexical units.

[0022] The above process will be explained in detail below.

[0023] like Figure 2 As shown, Figure 2 (a) shows the GQA attention structure before the transformation. Figure 2 (b) shows the MLA pre-filling process after applying the attention processing method described in this embodiment. Figure 2 (c) illustrates the MLA autoregressive decoding process after applying the attention processing method described in this embodiment. `full_rope` represents a complete RoPE, and `partial_rope` represents a partial RoPE. The query and key are processed by corresponding query normalization (QNorm) and key normalization (KNorm), respectively, forming the RoPE portion for rotational position encoding and the NoPE portion without rotational position encoding. The NoPE key and value are jointly compressed using latent key-value representation.

[0024] Specifically, in Figure 2 In the pre-filling process shown in (b), a latent KV representation can be obtained based on the input lexical units, and corresponding NoPE keys and values ​​can be generated based on the latent KV representation, while RoPE keys for positional encoding are also obtained. Thus, the latent KV representation and the RoPE keys for subsequent attention calculations can be used as cached content corresponding to historical lexical units. For KNorm, a key normalization calibration factor predetermined based on calibration samples can be used to approximate the key normalization factor that varies with the lexical units, and the corresponding static normalization scale can be combined with the projection onto the NoPE keys and the RoPE keys.

[0025] exist Figure 2In the autoregressive decoding process shown in (c), for the query of the current term, QNorm processing can be maintained, and a query-side transformation is performed on the NoPE query of the current term based on the NoPE key projection combined with the key normalization scale. This allows the transformed NoPE query to be directly operated on with the latent KV representation in the KV cache; the corresponding RoPE query is operated on with the RoPE key in the cache. Thus, attention calculation can be performed according to the latent KV cache method of MLA without explicitly generating complete reconstruction keys for each historical term in the cache, or additionally caching the KNorm normalization factor that changes with the term for each historical term and each KV header.

[0026] In one implementation, MLA uses a joint low-rank representation instead of the complete key and value, and separates positional encoding from potential compression. To simplify the notation, the attention head and grouping index can be omitted, representing the position-independent path as:

[0027] in, Indicates the first The input representation corresponding to each word element This represents the potential key-value representation. This represents the lower projection of the latent KV representation. This indicates projection onto the NoPE key. This represents the projection onto the values. The dimension of the latent key-value representation is less than the total dimension of the original keys and values.

[0028] MLA can also include independent location-aware paths, for example:

[0029] Therefore, the attention logit can be determined jointly by the NoPE path and the RoPE path.

[0030] For the NoPE path, in the absence of key-side normalization hindering matrix absorption, we have: =

[0031] Therefore, the projection onto the NoPE key can be absorbed by the query side, where the current query performs the corresponding query-side transformation and directly operates with the latent KV representation corresponding to the historical tokens. This eliminates the need to explicitly reconstruct the complete NoPE key for historical tokens during autoregressive decoding.

[0032] For Transformer models including QNorm and KNorm, QNorm performs root mean square normalization on the queries of each query header, while KNorm performs root mean square normalization on the keys of each key-value header. For simplicity, the following example uses the corresponding i-th header. For the i-th... An attention point can be represented as:

[0033] in, , This refers to the element-wise gain parameter corresponding to the normalization operation. This represents the root mean square magnitude.

[0034] After completing the Partial-RoPE conversion and KV joint compression, the... The refactoring key of a KV header before RoPE execution can be represented as:

[0035] in, For the NoPE portion of the reconstructed key, This refers to the RoPE portion of the reconstructed key.

[0036] Without considering KNorm, the attention logit for the NoPE path can be rewritten as:

[0037] therefore, It can be combined with the query side calculation.

[0038] However, when KNorm is applied to the key, the root mean square magnitude used by KNorm is:

[0039] The root mean square magnitude depends on the reconstruction key corresponding to the current lexical and varies with the lexical. Although the gain parameter of KNorm... Regardless of the word element, it can be incorporated into the corresponding projection, but the root mean square magnitude cannot be directly incorporated into the fixed query-side projection.

[0040] If the normalization factor that varies with the lexical unit is to be accurately preserved, one approach requires explicitly materializing the complete reconstruction key of each historical lexical unit; another approach requires additionally storing the corresponding normalization factor for each historical lexical unit and each key-value (KV) header, and applying the stored normalization factor during attention calculation. Both approaches increase the storage, data access, or computational overhead in the autoregressive decoding process.

[0041] Unlike KNorm, autoregressive decoding only requires generating a query for the current term. Therefore, QNorm can be directly applied to the current query without needing to save query normalization factors for historical terms. Based on this, this embodiment can maintain the QNorm processing method unchanged while performing an adaptation for KNorm.

[0042] Specifically, in step S101, for each KV header of each layer, a key normalization calibration factor independent of the lexical can be pre-determined based on calibration samples. For the first... Each KV header can be used to denot the key normalization calibration factor as follows: And expressed as:

[0043] in, The key is obtained by reconstructing lexical units from calibration samples.

[0044] Therefore, the bond normalization calibration factor can be used. Similar to the original changes with word elements Thus, the normalized bond can be approximated as:

[0045] because as well as Neither of them changes with the word, so the scales of the two corresponding to the NoPE subspace can be combined and projected onto the NoPE key.

[0046] Specifically, in step S102, the processed projection on the NoPE key can be obtained according to the following formula:

[0047] in, This represents the component in the KNorm gain parameter corresponding to the NoPE subspace.

[0048] After the above processing, in step S103, the attention logit of the NoPE path can be expressed as:

[0049] Therefore, we can first project onto the processed NoPE key. Perform a query-side transformation on the query for the current term, and then directly compare the transformed query with the latent key-value representation in the key-value cache. Perform the calculation.

[0050] For position-aware paths, the corresponding lexical-independent normalization scale can be incorporated into the RoPE key projection before performing rotational position encoding. Thus, during autoregressive decoding, historical lexical units can cache the latent KV representation and the required position-aware representation in an MLA manner, without needing to additionally cache the lexical-dependent normalization factor of KNorm for each historical lexical unit.

[0051] This embodiment approximates the lexical-dependent normalization factor in KNorm by using a pre-determined, lexical-independent key normalization calibration factor based on calibration samples. This allows the static scale of KNorm to be incorporated into the corresponding projection, restoring the matrix absorption of the projection onto the NoPE key towards the query side. Thus, while maintaining QNorm processing and approximately preserving the key scale control effect of KNorm, MLA autoregressive decoding can be performed according to the potential KV cache. This avoids explicitly reconstructing the complete key for each historical lexical or additional caching of the corresponding key normalization factor, thereby reducing KV cache usage and storage access overhead during decoding.

[0052] In service efficiency verification, end-to-end inference tests were performed on a single NVIDIA H200 GPU using vLLM, with BF16 model weights and different latent key-value dimensions and key-value cache precision. In a test set with a prompt length and output length of 16K and a concurrency level of 256, for the Qwen3-30B-A3B model, with a BF16 key-value cache, MLA configurations with latent dimensions of 512, 256, and 128 achieved normalized throughput of 1.45x, 2.37x, and 3.00x respectively compared to the BF16 GQA baseline under the same workload. With a latent dimension of 128 and an FP8 key-value cache, the normalized throughput reached 5.35x. These results demonstrate that reducing key-value cache usage and the corresponding memory bandwidth pressure can translate into improved autoregressive inference throughput.

[0053] Example 2

[0054] This embodiment, based on Embodiment 1, aims to provide further performance recovery processing for Transformer models converted by MLA.

[0055] The performance recovery process includes: obtaining the pre-transformation model corresponding to the Transformer model, which employs Multi-Head Attention (MHA) or Grouped Query Attention (GQA), as the teacher model; inputting the same input sequence into the teacher model and the MLA-transformed Transformer model respectively, and determining the distillation loss based on the difference between the output distribution of the teacher model and the output distribution of the MLA-transformed Transformer model; based on the distillation loss, only updating the projection parameters corresponding to the query, key, and value in the attention layer, as well as the corresponding QNorm and KNorm parameters, while keeping the parameters of the attention output projection, word embedding, language model head, and feedforward network unchanged.

[0056] like Figure 3 As shown, the MHA model or GQA model before transformation can be used as the teacher model, and the Transformer model after MLA transformation can be used as the student model. The same input sequence is input to the teacher model and the student model, and the distillation loss is determined based on the difference between their output distributions. Figure 3 In this context, the "frozen" identifier indicates a module whose parameters remain unchanged during performance recovery, and the "trainable" identifier indicates a module whose parameters are updated based on the distillation loss.

[0057] Specifically, on the student model side, only the attention-side parameters affected by the MLA transformation can be set as updatable parameters. These attention-side parameters include the projection parameters corresponding to the query, key, and value, as well as the corresponding QNorm and KNorm parameters; the parameters of the attention output projection, word embeddings, language model head, and feedforward network remain unchanged. The parameters of the teacher model remain unchanged. Thus, the output distribution of the teacher model can be used to constrain the output changes after the MLA transformation without performing retraining on all model parameters of the student model.

[0058] In one implementation, since the unmodified parameters in both the teacher and student models remain frozen, the corresponding frozen parameters in both models can share parameter storage, thereby further reducing the storage space occupied by model parameters during performance recovery.

[0059] Specifically, the original MHA or GQA model can be used as the teacher model, and the model after MLA transformation can be used as the student model. For the same input sequence, the output distributions corresponding to the teacher model and the student model can be obtained separately, and the output distribution of the teacher model can be used to distill the student model.

[0060] For length of For the input sequence, the distillation loss can be expressed as:

[0061] in, This represents the output distribution of the teacher model. This represents the output distribution of the student model. Indicates the first The sequence of lexical elements preceding each lexical element.

[0062] Compared to training the next word prediction using only a one-hot target, the output distribution of the teacher model can provide distribution information for multiple candidate words, thereby constraining the output bias caused by MLA transformation.

[0063] Regarding parameter updates, only query, key, and value-side parameters affected by MLA transformations are updated. These parameters may include, for example, the downprojection of the latent KV representation, the upprojection of NoPE keys and values, the query projection and key projection corresponding to the RoPE reserved subspace, and the corresponding QNorm and KNorm parameters.

[0064] Accordingly, the parameters of the attention output projection, word embeddings, language model head, and feedforward network remain fixed. Therefore, it is unnecessary to calculate and save gradients and optimizer states for all model parameters.

[0065] For example, for the Qwen3-8B model, only about 0.91B parameters out of 8.19B model parameters can be updated, which corresponds to a reduction of about 88.9% of the trainable parameters; for the Qwen3-30B-A3B model, only about 0.50B parameters out of 30.53B model parameters can be updated, which corresponds to a reduction of about 98.4% of the trainable parameters.

[0066] To verify the impact of distillation target and parameter update range on performance recovery, the next term prediction NTP or distillation can be used as the recovery target, respectively. The parameters were updated using either full parameters or only the QKV side parameters. Example results are shown in Table 1 below.

[0067] Table 1

[0068] As demonstrated by the examples above, under the same recovery budget, updating only the QKV side parameters yields performance close to that of updating all parameters, while significantly reducing the number of parameters involved in the optimization and the corresponding gradient and optimizer state storage overhead. Distillation using the teacher model output distribution can also constrain representation changes caused by MLA transformations, particularly achieving more stable performance recovery in long-context tasks.

[0069] Example 3

[0070] This embodiment applies to Transformer models that include a hybrid expert MoE module. The MoE module may include a router and multiple experts.

[0071] Furthermore, such as Figure 3 As shown, when the feedforward network uses a MoE module, the MoE module can include routers and multiple experts. During performance recovery, the routers and multiple experts can be set as frozen modules, and only the attention-side parameters affected by MLA transformation are updated. In this way, the performance recovery process does not directly change the router and expert parameters formed in the pre-training stage, and limits the changes in attention representation through distillation loss between the teacher model and the student model, thereby reducing the extent to which changes in attention representation further cause changes in expert routing.

[0072] In the performance recovery process described in Example 2, the parameters of the router and multiple experts are kept unchanged, and the attention-side parameters affected by the MLA conversion are updated only based on the distillation loss.

[0073] During pre-training, the MoE model can develop corresponding expert specializations and expert load balancing behaviors. Changes in the attention structure may alter the distribution of hidden states, further influencing the router's selection of different experts.

[0074] Therefore, in this embodiment, the parameters of the router and the expert themselves are not updated to maintain the router and expert parameters already formed during the pre-training phase. Simultaneously, by distilling the output distribution between the teacher and student models, the changes in the attention representation are constrained, thereby limiting the expert routing changes indirectly caused by changes in the attention layer representation.

[0075] Furthermore, since all experts in the feedforward network remain fixed, there is no need to save gradients and optimizer states for each expert during training. This is particularly beneficial for MoE models with a large number of expert parameters, as it reduces storage overhead during the performance recovery phase.

[0076] Taking the Qwen3-30B-A3B model as an example, load balancing violations can be used to evaluate MoE routing changes. Example results are shown in Table 2 below.

[0077] Table 2

[0078] Among them, the maximum violation value represents the maximum value among the maximum violation values ​​of each layer, and the average violation value represents the average value of the corresponding violation values ​​of each layer.

[0079] As can be seen from the above results, under different KV cache compression levels, the average load balancing violation value in this embodiment remains within a range close to the baseline, while directly removing KNorm will cause more obvious load balancing changes.

[0080] Therefore, by freezing the MoE router and experts during the performance recovery phase and utilizing distillation to constrain the representation changes on the attention side, it is possible to reduce the perturbation of the pre-trained expert routing behavior by the MLA architecture transformation while reducing the size of the parameters involved in training.

[0081] Example 4

[0082] Based on Example 2, this embodiment further adopts a progressive approach to obtain Transformer models corresponding to different potential KV dimensions.

[0083] Specifically, the Transformer model with the first potential KV dimension and which has already undergone performance recovery processing is first used as the current model.

[0084] Subsequently, the latent key-value dimension of the current model is reduced to obtain a Transformer model with a second latent key-value dimension, where the second latent key-value dimension is smaller than the first latent key-value dimension.

[0085] The potential key-value dimension can be reduced by pruning some of the potential representation dimensions of the current model. After reducing the potential key-value dimension, the model parameters that have been recovered in the current model are reused, and these model parameters are used as a new starting point for performance recovery. The performance recovery process described in Example 2 is then performed on the Transformer model with the second potential key-value dimension.

[0086] After completing the performance recovery for the second potential key-value dimension, the potential key-value dimension can be reduced again, and subsequent performance recovery can continue based on the recovery results from the previous stage. This process can be repeated to obtain Transformer models corresponding to multiple different potential key-value dimensions.

[0087] The above approach does not require performing a complete recovery independently from the same original model for each candidate potential KV dimension, but can reuse the recovery results already obtained under the previous potential KV dimension, thereby reducing the overhead of repeated recovery required in the process of searching for potential KV dimensions.

[0088] Taking the Qwen2.5-7B model as an example, the results of a progressive compression are shown in Table 3 below.

[0089] Table 3

[0090] Specifically, for the same level of compression, the score difference between independently recovered models and models recovered from a less compressed model can remain within a small range. In the example above, the score difference for the same target dimension remains within 0.5 points.

[0091] Therefore, this embodiment is particularly suitable for finding trade-offs between model performance and key-value cache usage before actual deployment. By progressively reducing the potential key-value dimensions from a lightly compressed, recovered model and continuing the recovery process, the computational overhead of repeatedly performing full performance recovery for different candidate compression levels can be reduced.

[0092] Example 5

[0093] This embodiment further explains the method for determining the bond normalization calibration factor in Embodiment 1 and the method for handling the normalization scale in different subspaces.

[0094] For each attention layer of the Transformer model, a corresponding bond normalization calibration factor can be determined for each key-value (KV) head. Therefore, different layers, as well as different KV heads within the same layer, can each have their own bond normalization calibration factor.

[0095] For a key-value (KV) header with a key-normalized calibration factor to be determined, multiple calibration samples are input into the Transformer model, and the reconstructed keys corresponding to multiple terms in the calibration samples are obtained. For each reconstructed key, its root mean square (RMS) magnitude is determined, and then the average value is determined based on the RMS magnitudes corresponding to multiple terms. This average value is then used as the key-normalized calibration factor for the KV header.

[0096] For the The key normalization calibration factor of a KV header can be expressed as:

[0097] This calibration process can be performed offline. Therefore, in the actual autoregressive decoding process, it is not necessary to calculate and save an independent KNorm denominator for each historical token in real time.

[0098] In one example, the intermediate KV header of the intermediate layer of the Qwen3-8B model was selected, and the bond-normalized calibration factor was estimated using 64 calibration samples at context lengths of 1K, 2K, 4K, 8K, and 16K. The results are shown in Table 4 below.

[0099] Table 4

[0100] In the above example, the estimated values ​​for different context lengths range from 0.8185 to 0.8305, with a coefficient of variation of 0.48%, indicating that the bond normalization calibration factor has high stability under different context lengths.

[0101] After determining the bond normalization calibration factor, the [factor] can be [adjusted / converted / advertised]. and KNorm gain parameters The jointly determined static bond normalization scale is processed according to the NoPE subspace and the RoPE subspace.

[0102] For the NoPE subspace, as described in Example 1, the corresponding scale can be directly projected onto the NoPE key:

[0103] For the RoPE subspace, the RoPE component in the KNorm gain parameter and the corresponding... The scale is incorporated into the RoPE key projection before rotational position encoding is performed. In this way, the keys affected by rotational position encoding already contain the key-normalized scale determined based on the calibration factor, without needing to apply the lexical-related KNorm denominator to each historical lexical after rotational position encoding.

[0104] To verify the effectiveness of using the bond-normalized calibration factor, the method in this embodiment can be compared with the method of directly removing KNorm. Example results are shown in Table 5 below.

[0105] Table 5

[0106] The results above show that, compared to directly removing KNorm to meet the absorption requirements of the MLA matrix, using calibration samples to determine the lexical-independent key normalization calibration factor and combining the corresponding static scale to the NoPE and RoPE key projections can reduce the performance loss caused by architecture transformation, while keeping attention computation compatible with the potential KV cache.

[0107] Example 6

[0108] This embodiment further explains the Partial-RoPE process during MLA construction and the joint low-rank initialization of KV.

[0109] First, based on the contribution of different rotation subspaces to the attention distribution, a portion of the rotation subspaces from the original query and key rotation position encoding subspaces are selected to continue retaining RoPE, while the remaining key dimensions are used as NoPE subspaces.

[0110] In one implementation, the contribution of different rotation subspaces to the attention logit can be evaluated separately, and this contribution can be used as a measure of the contribution of the corresponding rotation subspace to the attention distribution. For example, the rotation subspaces to be retained can be selected based on the head-by-head 2-norm contribution of each rotation subspace to the attention logit.

[0111] The retained rotation subspace forms a position-aware path and continues to employ RoPE; after removing RoPE from the remaining dimensions, the remaining dimensions form a NoPE subspace, which can be further incorporated into potential KV paths.

[0112] Therefore, while retaining some positional encoding capabilities, more key dimensions can participate in subsequent KV joint compression.

[0113] Subsequently, a joint low-rank approximation is performed on the NoPE bond projection and value projection.

[0114] by Indicates NoPE key projection, and compares it with value projection. Combined, we obtain the joint KV projection:

[0115] If we directly perform truncated singular value decomposition in the weight space, the corresponding low-rank approximation objective can be expressed as:

[0116] This approach directly minimizes the reconstruction error of the weight matrix itself, without considering the distribution of the actual input activations.

[0117] This embodiment employs a low-rank approximation of activation-aware calibration activation. Let calibration activation be: Then the following objective can be used to determine the rank not exceeding The joint KV low-rank approximation:

[0118] Therefore, the low-rank approximation directly optimizes the output error after the joint KV projection is applied to the actual calibration activation, rather than optimizing only the weight error of the joint KV projection itself.

[0119] Specifically, we can first perform eigenvalue decomposition on the second-order matrix of calibration activation:

[0120] Then construct:

[0121] For matrix Execution rank is Truncated singular value decomposition:

[0122] The lower projection of the latent KV representation can then be initialized according to the following formula:

[0123] Initialize the NoPE key projection and value projection according to the following formula:

[0124] in, It is possible It is computed on the non-zero feature space.

[0125] The product of the lower projection and the upper projection obtained from the above initialization forms a low-rank approximation that satisfies the target potential KV dimension, and its effect on the KV output reconstruction error after calibration activation is reduced.

[0126] After completing the above initialization, the performance recovery process described in Example 2 can be further used to restore the Transformer model after Partial-RoPE and low-rank initialization.

[0127] To compare the differences between activation-aware low-rank initialization and ordinary SVD initialization, the layer-by-layer average mean square error between the original attention output before training and the attention output after editing can be used as an evaluation metric. Taking the Qwen2.5-7B model as an example, sample results are shown in Table 6 below.

[0128] Table 6

[0129] As can be seen from the above results, under different KV buffer compression levels, the low-rank initialization obtained by minimizing the KV output reconstruction error based on calibration activation can significantly reduce the attention output error before recovery.

[0130] Therefore, this embodiment retains the rotation subspace that contributes significantly to the attention distribution, allowing it to continue to perform positional encoding, and forms a compressible NoPE subspace with the remaining key dimensions. Furthermore, by performing a joint low-rank approximation on the NoPE key projection and value projection with the KV output reconstruction error on the actual calibration activation as the target, the initialized MLA latent projection is made closer to the attention behavior of the original model under the actual input distribution, thereby reducing the initial error after the architecture transformation and facilitating subsequent performance recovery.

[0131] Figure 4 This is a hardware structure block diagram of an electronic device implementing the embodiments of this application.

[0132] like Figure 4As shown, the electronic device 400 may include one or more processors 402, a system motherboard 408 connected to at least one of the processors 402, system memory 404 connected to the system motherboard 408, non-volatile memory (NVM) 406 connected to the system motherboard 408, and a network interface 410 connected to the system motherboard 408.

[0133] Processor 402 may include one or more single-core or multi-core processors. Processor 402 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments of the present invention, processor 402 may be configured to perform one or more embodiments according to various embodiments of this application.

[0134] In some embodiments, system motherboard 408 may include any suitable interface controller to provide any suitable interface to at least one of the processors 402 and / or any suitable device or component communicating with system motherboard 408.

[0135] In some embodiments, system motherboard 408 may include one or more memory controllers to provide an interface to system memory 404. System memory 404 may be used to load and store data and / or instructions. In some embodiments, system memory 404 of electronic device 400 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0136] NVM 406 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 406 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of HDD (Hard Disk Drive), CD (Compact Disc) drive, and DVD (Digital Versatile Disc) drive.

[0137] NVM 406 may include a portion of the storage resources on a device installed on electronic device 400, or it may be accessible by the device, but is not necessarily part of the device. For example, NVM 406 may be accessed over a network via network interface 410.

[0138] Specifically, system memory 404 and NVM 406 may each include a temporary copy and a permanent copy of instruction 420. Instruction 420 may include instructions that, when executed by at least one of processors 402, cause electronic device 400 to perform methods as described in any embodiment of this application. In some embodiments, instruction 420, hardware, firmware, and / or its software components may additionally / alternatively be located in system motherboard 408, network interface 410, and / or processor 402.

[0139] Network interface 410 may include a transceiver for providing a radio interface to electronic device 400, thereby enabling communication with any other suitable device (e.g., front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 410 may be integrated into other components of electronic device 400. For example, network interface 410 may be integrated into at least one of processor 402, system memory 404, NVM 406, and firmware device (not shown) with instructions, wherein when at least one of processor 402 executes the instructions, electronic device 400 implements one or more embodiments of various embodiments of this application.

[0140] Network interface 410 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, network interface 410 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0141] In one embodiment, at least one of the processors 402 may be packaged together with one or more controllers for the system motherboard 408 to form a system-in-package (SiP). In another embodiment, at least one of the processors 402 may be integrated on the same die with one or more controllers for the system motherboard 408 to form a system-on-a-chip (SoC).

[0142] The electronic device 400 may further include: an input / output (I / O) device 412 connected to the system motherboard 408. The I / O device 412 may include a user interface enabling a user to interact with the electronic device 400; the peripheral component interface is designed to allow peripheral components to also interact with the electronic device 400. In some embodiments, the electronic device 400 may also include sensors for determining at least one type of environmental condition and location information related to the electronic device 400.

[0143] In some embodiments, the input / output (I / O) device 412 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash) and a keyboard.

[0144] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0145] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 410 to communicate with components of the positioning network, such as Global Positioning System (GPS) satellites.

[0146] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 400. In other embodiments of this application, the electronic device 400 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0147] Program code can be applied to input instructions to perform the functions described in this invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 402 includes any system having a processor such as a digital signal processor (DSP), microcontroller, application-specific integrated circuit (ASIC), or microprocessor.

[0148] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this invention are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0149] One or more aspects of at least one embodiment can be implemented by instructions stored on a computer-readable storage medium, which, when read and executed by a processor, enable an electronic device to implement the methods of the embodiments described in this invention.

[0150] According to some embodiments of this application, a computer storage medium is disclosed, on which instructions are stored, which, when executed on a computer, cause the computer to perform an attention processing method according to embodiments of this application.

[0151] The method embodiments of this application correspond to this embodiment, and this embodiment can be implemented in conjunction with the method embodiments of this application. The relevant technical details mentioned in the method embodiments of this application are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the method embodiments of this application.

[0152] According to some embodiments of this application, a computer program product is disclosed, including computer-executable instructions that are executed by a processor to implement an attention processing method according to embodiments of this application.

[0153] The method embodiments of this application correspond to this embodiment, and this embodiment can be implemented in conjunction with the method embodiments of this application. The relevant technical details mentioned in the method embodiments of this application are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the method embodiments of this application.

[0154] It is understood that the specific embodiments described herein are merely for illustrative purposes and not for limiting the scope of this application. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to this application, and not all of the structures or processes. It should be noted that similar reference numerals and letters in the drawings denote similar items throughout this application.

[0155] It should be understood that although the terms "first," "second," etc., may be used herein to describe various features, these features should not be limited by these terms. The use of these terms is merely for distinction and should not be construed as indicating or implying relative importance. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0156] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this embodiment based on the specific circumstances.

[0157] The illustrative embodiments of this application include, but are not limited to, attention processing methods, devices, media, and computer program products.

[0158] Various aspects of the illustrative embodiments will be described using terminology commonly employed by those skilled in the art to convey the essence of their work to others skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments will be practiced using the features partially described. Specific figures and configurations are set forth for purposes of explanation in order to provide a more thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without specific details. In some other instances, well-known features have been omitted or simplified herein to avoid obscuring the illustrative embodiments of this application.

[0159] Furthermore, the various operations will be described as multiple separate operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order of description, and many of these operations may be performed in parallel, concurrently, or simultaneously. Moreover, the order of the operations may also be rearranged. The process may be terminated when the described operations are completed, but may also include additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0160] References to "an embodiment," "embodiment," "illustrative embodiment," etc., in this application indicate that the described embodiment may include specific features, structures, or properties; however, each embodiment may or may not necessarily include specific features, structures, or properties. Furthermore, these phrases are not necessarily directed at the same embodiment. Moreover, when specific features are described in conjunction with specific embodiments, the knowledge of those skilled in the art can influence the combination of these features with other embodiments, whether or not those embodiments are explicitly described.

[0161] Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrase “A and / or B” means “(A), (B), or (A and B).”

[0162] As used herein, the term "module" may refer to, as part of, or include: a memory (shared, dedicated, or grouped), an application-specific integrated circuit (ASIC), electronic circuitry and / or a processor (shared, dedicated, or grouped), combinational logic circuitry, and / or other suitable components that provide the said functionality for running one or more software or firmware programs.

[0163] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular drawing does not mean that all embodiments need to include such features; in some embodiments, these features may be omitted or may be combined with other features.

[0164] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions or programs carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or program are run by a machine, the machine may perform the various methods described above. For example, the instructions may be distributed via a network or other computer-readable media. Therefore, machine-readable media may include, but are not limited to, any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, such as floppy disks, optical disks, optical disc read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electronically erasable programmable read-only memories (EEPROMs), magnetic cards or optical cards, or flash memory or tangible machine-readable storage for transmitting network information via electrical, optical, acoustic, or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media includes any form of machine-readable medium suitable for storing or transmitting electronic instructions or machine-readable (e.g., computer-readable) information.

[0165] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, the use of the technical solutions of this application is not limited to the various applications mentioned in the embodiments of this application. Various structures and modifications can be easily implemented with reference to the technical solutions of this application to achieve the various beneficial effects mentioned herein. Within the scope of knowledge possessed by those skilled in the art, all changes made without departing from the spirit of this application should be considered within the scope of this patent application.

Claims

1. An attention processing method applied to a Transformer model including query normalization (QNorm) and key normalization (KNorm), wherein the attention layer of the Transformer model employs multi-head latent attention (MLA), characterized in that, The method includes: For the KV head of the attention layer, obtain a word-independent key normalization calibration factor based on the root mean square magnitude of the reconstructed keys in the calibration samples. Based on the key normalization calibration factor and the gain parameter of KNorm, the NoPE key projection of the MLA is processed so that the key normalization factor that varies with the word is approximated by the key normalization calibration factor, and the corresponding key normalization scale is combined with the NoPE key projection. When performing autoregressive decoding on the input sequence, the query of the current word is transformed based on the processed NoPE key projection, and the query transformed by the query side is operated on with the latent KV representation in the KV cache to perform attention calculation.

2. The attention processing method according to claim 1, characterized in that, It also includes performance recovery processing for the Transformer model, the performance recovery processing including: Obtain the pre-transformation model corresponding to the Transformer model, which uses multi-head attention (MHA) or grouped query attention (GQA), as the teacher model; The same input sequence is input into the teacher model and the Transformer model respectively, and the distillation loss is determined based on the difference between the output distribution of the teacher model and the output distribution of the Transformer model. Based on the distillation loss, only the projection parameters corresponding to the query, key, and value in the attention layer, as well as the corresponding QNorm and KNorm parameters, are updated, while the parameters of the attention output projection, word embedding, language model head, and feedforward network remain unchanged.

3. The attention processing method according to claim 2, characterized in that, The Transformer model also includes a hybrid expert MoE module, which comprises a router and multiple experts; The performance recovery process also includes: Keeping the parameters of the router and the plurality of experts unchanged, and limiting the expert routing changes caused by the representation changes of the attention layer based on the distillation loss.

4. The attention processing method according to claim 2, characterized in that, The performance recovery process also includes: The Transformer model with the first potential KV dimension, after the performance recovery process, is used as the current model; Reduce the latent key-value dimension of the current model to obtain a Transformer model with a second latent key-value dimension, wherein the second latent key-value dimension is smaller than the first latent key-value dimension; Using the model parameters of the current model after reducing the potential KV dimension as the recovery starting point, the performance recovery process is continued to be performed on the Transformer model with the second potential KV dimension. Repeat the process of reducing the potential key-value dimensions and continuing to perform the performance recovery process to obtain the Transformer models corresponding to different potential key-value dimensions.

5. The attention processing method according to claim 1, characterized in that, The bond normalization calibration factor is determined according to the following steps: For each key-value head in each layer of the attention layer, the average value is determined based on the root mean square magnitude of the reconstruction keys corresponding to multiple tokens in the calibration sample; The average value is determined as the bond normalization calibration factor for the corresponding KV head.

6. The attention processing method according to claim 5, further comprising: Before performing rotational position encoding, the RoPE component of the bond-normalized scale, determined by the bond-normalized calibration factor and the gain parameter of the KNorm, is combined with the RoPE bond projection.

7. The attention processing method according to claim 1, characterized in that, The MLA is constructed according to the following steps: Based on the contribution of different rotation subspaces to the attention distribution, the rotation subspace used for rotational position encoding RoPE is selected and retained, and its remaining key dimension is used as the NoPE subspace. The key projection and value projection corresponding to the NoPE subspace are combined into a joint KV projection. Based on calibration activation, with the goal of reducing the reconstruction error of the output after calibration activation by the joint KV projection, a low-rank approximation is performed on the joint KV projection to initialize the lower projection, NoPE key projection and value projection of the latent KV representation.

8. An electronic device, characterized in that, The electronic device includes a memory storing computer-executable instructions and a processor, which, when executed by the processor, cause the electronic device to perform the attention processing method according to any one of claims 1-7.

9. A computer storage medium, characterized in that, Instructions are stored on the computer storage medium, which, when executed on the computer, cause the computer to perform the attention processing method according to any one of claims 1-7.

10. A computer program product, characterized in that, It includes computer-executable instructions that are executed by a processor to implement the attention processing method according to any one of claims 1-7.

Images