Post-normalization rotation quantization normalization layer fusion method and device for large language model

Abstract

The application belongs to the technical field of large language models, and relates to a rotation quantization normalization layer fusion method and device for a post-normalization large language model. By decomposing and retaining scale-invariant shape components of original effective weights of a shared normalization layer in the post-normalization large language model, the original effective weights or equivalent full-scale weights thereof are fused into head weights in a language model head path, the shape components are element-by-element fused with second normalization layer weights in a multi-token prediction branch, and shared normalization layer scaling weights are set to identity, and finally, rotation quantization is performed on a fused post-computation graph. The application eliminates global deterministic scaling bias in chain structure fusion, provides high-precision preprocessing for rotation quantization, reduces fusion error of a multi-token prediction branch while keeping the output of a main path unchanged, and does not introduce additional online inference overhead.

Description

Technical Field

[0001] This invention belongs to the field of large language model technology, specifically relating to a method and apparatus for fusion of rotation quantization normalization layers in post-normalized large language models. Background Technology

[0002] Large Language Models (LLMs) have achieved remarkable success in the field of natural language processing, but their massive number of parameters and computational overhead pose significant challenges to practical deployment. To reduce deployment costs, low-bit quantization techniques are widely used, among which rotation quantization has become an important quantization preprocessing method due to its ability to effectively suppress outliers in activation values ​​and improve quantization resolution. Meanwhile, multi-token prediction (MTP), as an emerging method for improving model architecture, adds an extra prediction branch, enabling the model to predict multiple future tokens simultaneously, effectively improving inference efficiency and generation quality. However, existing rotation quantization methods have compatibility issues with the MTP architecture, preventing their direct integration.

[0003] Existing rotation quantization methods (such as QuaRot and SpinQuant) typically require fusing the normalization layers in the model into an identity transformation during quantization preprocessing to allow orthogonal rotation matrices to freely penetrate the computation graph. This fusion operation can be successfully completed in the traditional normalization layer → linear layer (Norm → Linear) topology. However, in the MTP architecture employing post-normalization, the shared normalization layer (model_norm) at the top of the model serves both the language model head and the MTP branch. Its output not only enters the linear head path but also the downstream normalization layer in the MTP branch, forming a unique normalization layer → normalization layer (Norm → Norm) chain structure. When performing a standard fusion operation on this shared normalization layer, the RMS denominator calculation in the downstream normalization layers of the MTP branch changes, introducing severe systematic scaling errors. Experiments show that on the Qwen3.5-9B model, naive fusion (i.e., the complete fusion described later) causes the Top-1 acceptance rate of the MTP branch to decrease from 0.5993 to 0.5451, a drop of 9.0%. More problematic is that simply choosing not to fuse the shared normalization layer prevents rotation matrix penetration, making rotation quantization completely impossible. Furthermore, different models have different weight conventions for RMSnorm (e.g., the standard γ convention versus the (1+u) convention), further increasing the difficulty of achieving universality for the fusion method.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] One objective of this invention is to address the problems of traditional normalization layer fusion methods, such as the inability to directly fuse chain structures of normalization layers to normalization layers in models containing MTP modules, inconsistent weights across branches, and large rotation quantization errors.

[0006] To achieve the above objectives, this invention provides a method for fusing rotationally quantized normalization layers in a post-normalized large language model. The large language model includes a shared normalization layer and a multi-token prediction branch. One output of the shared normalization layer enters the head of the language model, and the other enters the second scale-invariant normalization layer in the multi-token prediction branch, forming a chain structure from the shared normalization layer to the second normalization layer. The method includes:

[0007] Obtain the original effective weights of the shared normalized layer;

[0008] The original effective weights are decomposed to extract the scale-invariant shape components;

[0009] For the language model head path, the original effective weights or their equivalent full-scale weights are integrated into the language model head weights;

[0010] For the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch, the shape component is fused with the weight of the second normalization layer element by element;

[0011] Set the learnable scaling weights of the shared normalization layer to an constant value;

[0012] Rotation matrix insertion, rotation fusion, and quantization are performed on the merged computation graph.

[0013] Furthermore, the shared normalization layer is located at the final output position of the large language model decoder and adopts a post-normalization architecture. Obtaining the original effective weights of the shared normalization layer includes: locating the corresponding normalization layer instance in the module hierarchy of the large language model based on the identifier of the shared normalization layer; extracting the original weight tensor from the normalization layer instance and creating an independent copy of the original weight tensor to disconnect it from the parameters of the large language model; performing precision conversion and format standardization on the independent copy to generate a standardized weight vector; and extracting the original weight tensor from the normalization layer instance includes: traversing a preset list of candidate parameter names and searching for learnable weight parameters from the normalization layer instance; if the learnable weight parameter adopts an offset parameterization form, converting the offset parameter into an effective weight form; wherein the offset parameterization form follows the (1+u) convention, where u is the learnable offset; converting to an effective weight form includes: calculating the original effective weight γ1=1+u, where 1 is a vector of all 1s with the same dimension as u.

[0014] Further, the step of decomposing the original effective weights and extracting the scale-invariant shape components includes: calculating the root mean square value s of the original effective weights γ1, using the formula s = rms(γ1) = The root mean square value s is used as the global scale component; the original effective weight γ1 is divided by the root mean square value s to generate the shape component. =γ1 / s, wherein the shape component satisfies rms( )=1; After decomposition, the original effective weights are decomposed into two independent components: the global scale component s=rms(γ1), which is a positive scalar representing the overall scaling magnitude of the original effective weights; and the shape component , is a unit RMS vector representing the relative distribution pattern of the original effective weights; wherein, the global scale component is not propagated to the second normalization layer, and the shape component is preserved and used in the second normalization layer path; wherein, the decomposition process is strictly reversible: the original effective weights Able to be precisely decomposed into RMS ( ) It can also be generated by RMS ( )and The reconstruction is accurate, and neither the decomposition nor the reconstruction process produces errors.

[0015] Furthermore, the step of fusing the original effective weights or their equivalent full-scale weights into the language model head weights for the language model head path includes: fusing the original effective weights γ1 along the hidden dimension with the weight matrix W of the language model head. lmPerform row-by-row multiplication: Where d is the dimension of the original effective weights. V represents the size of the vocabulary.

[0016] Further, the step of fusing the shape component with the weights of the second normalization layer element-wise in the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch includes: fusing the shape component with the weights of the second normalization layer element-wise. The weights are multiplied element-wise with the weight parameter γ2 of the second normalized layer to obtain the fused weights. The calculation formula is: If the second normalization layer uses offset parameterization, the weight parameters are converted to the standard form γ2=1+u before fusion, and then converted back to the offset form u′= after fusion. -1, and write back to the second normalization layer; where u is the learnable offset.

[0017] Furthermore, when the stitching projection layer of the multi-token prediction branch has multiple input normalization layers, the shape component is only processed by the input normalization layer that receives the output of the shared normalization layer and is located on the output path of the shared normalization layer. Multiply the weights element by element; for other input normalization layers that do not receive the output of the shared normalization layer, fuse them into the corresponding input dimension range of the stitching projection layer according to their own weights.

[0018] Furthermore, the error introduced by the element-wise weight fusion process is approximately a data-dependent second-order residual error, and the factor of the second-order residual error is... The calculation formula is:

[0019] ;

[0020] in, The input activation values ​​for the second normalized layer. The normalized shape components and RMS ( )=1; ρ is and Pearson correlation coefficient along the channel dimension.

[0021] Further, the step of setting the learnable scaling weights of the shared normalization layer to an identity value includes: confirming that the weight adjustments for the language model head path and the multi-token prediction branch have been completed, and then performing the identity value setting; identifying the type and parameterization form of the shared normalization layer, and determining the configuration method of the identity value; setting the learnable scaling weights of the shared normalization layer to an identity value according to the determined configuration method; and the step of identifying the type and parameterization form of the shared normalization layer includes: determining that the shared normalization layer is a standard root mean square normalization layer or a scaled normalization variant without independent bias and only having learnable channel scaling parameters; for the standard root mean square normalization layer, the identity scaling weights are all-1 vectors; for the variant using the (1+u) offset parameterization form, the offset parameter u is an all-0 vector.

[0022] Furthermore, the step of performing rotation matrix insertion, rotation fusion, and quantization on the fused computation graph includes: constructing an orthogonal rotation matrix and inserting it into the model computation graph; fusing the orthogonal rotation matrix into the linear layer weight matrix of the model; and performing low-bit quantization on the fused weights.

[0023] In other embodiments, a rotation quantization normalization layer fusion device for a post-normalized large language model is provided, capable of executing the rotation quantization normalization layer fusion method for a post-normalized large language model described above; the device includes: a parameter acquisition module configured to acquire the original effective weights of the shared normalization layer; a parameter decomposition module configured to calculate the global root mean square scale of the original effective weights and generate shape components; a main path fusion module configured to fuse the original effective weights or their equivalent full-scale weights into the language model head weights for the language model head path; a branch fusion module configured to fuse the shape components with the weights γ2 of the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch element-wise; an identity configuration module configured to set the learnable scalable weights of the shared normalization layer to an identity value; and a rotation quantization module configured to perform rotation matrix insertion, rotation fusion, and quantization on the fused computation graph.

[0024] Based on the foregoing description, those skilled in the art will understand that this invention decomposes the original effective weights of the shared normalization layer in a post-normalized large language model into global RMS scale and scale-invariant shape components. In the language model head path, the original effective weights or their equivalent full-scale weights are fused into the head weights. In the multi-token prediction branch, the shape components are fused element-wise with the weights of the second normalization layer, and the scaling weights of the shared normalization layer are reset to an identity. Finally, rotation matrix insertion, rotation fusion, and quantization are performed on the fused computation graph. This invention effectively solves the problems of direct fusion of Norm→Norm chain structures, inconsistent weights across branches, and large rotation quantization errors in existing technologies. It achieves high-precision fusion by introducing only second-order residual errors, effectively reducing inference latency and quantization errors. Attached Figure Description

[0025] The accompanying drawings, as part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation of the invention. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0026] Figure 1 This is a flowchart of the rotation quantization normalization layer fusion method for post-normalized large language models in some embodiments of the present invention;

[0027] Figure 2 This is a diagram of key parameters involved in Embodiment 1 of the present invention;

[0028] Figure 3 This is an example diagram of the model adapted in Embodiment 2 of the present invention;

[0029] Figure 4 This is a comparison diagram of the fusion strategy in the synthesis test verification of Embodiment 3 of the present invention;

[0030] Figure 5 This is a comparison chart of the acceptance rates of Qwen3.5-9B MTP in Example 3 of the present invention;

[0031] Figure 6 This is a comparison chart of the acceptance rates of Qwen3.5-27B MTP in Example 3 of the present invention;

[0032] Figure 7 This is a summary chart of cross-model comparisons in Embodiment 3 of the present invention;

[0033] Figure 8 This is a diagram showing the main path verification results in Embodiment 3 of the present invention;

[0034] Figure 9This is a statistical chart of model parameters in Embodiment 3 of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0037] The following reference Figure 1 This document will provide a detailed description of the rotation quantization normalization layer fusion method for post-normalized large language models in some embodiments of the present invention. Figure 1 This is a flowchart of the rotation quantization normalization layer fusion method for post-normalized large language models in some embodiments of the present invention; Figure 2 This is a diagram of key parameters involved in Embodiment 1 of the present invention; Figure 3 This is an example diagram of the model adapted in Embodiment 2 of the present invention; Figure 4 This is a comparison diagram of the fusion strategy in the synthesis test verification of Embodiment 3 of the present invention; Figure 5 This is a comparison chart of the acceptance rates of Qwen3.5-9BMTP in Example 3 of the present invention; Figure 6 This is a comparison chart of the acceptance rates of Qwen3.5-27BMTP in Example 3 of the present invention; Figure 7 This is a summary chart of cross-model comparisons in Embodiment 3 of the present invention; Figure 8 This is a diagram showing the main path verification results in Embodiment 3 of the present invention; Figure 9 This is a statistical chart of model parameters in Embodiment 3 of the present invention.

[0038] In the post-norm MTP architecture, also known as the post-normalized multi-token prediction architecture, the shared normalization layer and the downstream normalization layer form a series. Traditional fusion methods will disrupt the numerical distribution of MTP branches, leading to two unacceptable consequences: either the normalization layer is retained, preventing the rotation matrix from penetrating; or a crude fusion is carried out, introducing a scaling bias of about 114%, which reduces the MTP acceptance rate, i.e., the MTP Top-1 acceptance rate, by 9%.

[0039] This invention presents a rotation quantization normalization layer fusion method for post-normalized large language models. Leveraging the scale invariance of RMSnorm (RootMeanSquareNormalization), it decomposes and selectively fuses shared normalization layer weights. This eliminates the first-order scaling error in the Norm→Norm chain (normalization layer→normalization layer chain structure) of the MTP branch (Multi-Token Prediction branch) without altering the main inference path output. Specifically, the decomposition step breaks down the weights into two independent attributes: scaling magnitude and relative shape. Only the relative shape information is passed to the downstream layer for fusion, while the scaling magnitude is discarded. This makes subsequent identity substitution and rotation quantization possible, effectively solving the architectural problem where rotation quantization cannot be applied due to the Norm→Norm chain. This invention is applicable to post-training low-bit quantization deployment of large language models, particularly models containing multi-token prediction modules (such as the Qwen3.5 series) on resource-constrained devices using INT8 / INT4 quantization.

[0040] like Figure 1 As shown, in some embodiments of the present invention, a method for fusing rotationally quantized normalization layers in a post-normalized large language model is provided. The large language model includes a shared normalization layer and a multi-token prediction branch. One output of the shared normalization layer enters the head of the language model, and the other enters the second scale-invariant normalization layer in the multi-token prediction branch, forming a chain structure from the shared normalization layer to the second normalization layer. The method includes:

[0041] Step S110: Obtain the original effective weights of the shared normalization layer. The shared normalization layer is located at the final output of the large language model decoder, employing a post-normalization architecture. It is situated at the final output of the main branch of the large language model decoder and subsequently branches to the language model head and multi-token prediction branches.

[0042] By starting from the top layer of the model and tracing the connections of normalization layer nodes along the computational path, a shared normalization layer is identified when the output of a normalization layer simultaneously flows to both the language model head and the MTP module. The process then continues downstream along the MTP branch, filtering out irrelevant nodes such as linear layers and activation functions, marking only the next normalization layer as part of the chain structure. This invention can batch process different models, automatically skipping irrelevant nodes such as linear layers and activation functions, focusing on normalization layers, and thus outputting a complete chain path.

[0043] Specifically, step S110 includes: locating the corresponding normalization layer instance in the module hierarchy of the large language model based on the identifier of the shared normalization layer; extracting the original weight tensor from the normalization layer instance and creating an independent copy of the original weight tensor to disconnect it from the parameters of the large language model; performing precision conversion and format normalization on the independent copy to generate a standardized weight vector.

[0044] In some specific embodiments, the step of extracting the original weight tensor from the normalized layer instance includes: traversing a preset list of candidate parameter names and searching for learnable weight parameters from the normalized layer instance; if the learnable weight parameter adopts an offset parameterization form, then converting the offset parameter into an effective weight form. The offset parameterization form follows the (1+u) convention, where u is the learnable offset. Converting to an effective weight form includes: calculating the original effective weight γ1 = 1 + u, where 1 is an all-1 vector with the same dimension as u.

[0045] A topological traversal of the model structure is performed to identify all normalization layer nodes. The output connections of each normalization layer node are analyzed to determine whether it branches to the main inference path and the multi-token prediction branch. For normalization layers determined to be shared, the path from their output to the multi-token prediction branch is traced downstream to detect whether there are downstream normalization layers connected in series with the shared normalization layer. The shared normalization layer is located at the final output position of the large language model decoder, adopts a post-normalization architecture, and is located at the final output position of the main branch of the large language model decoder, after which it branches to the language model head and the multi-token prediction branch.

[0046] In the multi-token prediction branch, the downstream normalization layer connected in series with the shared normalization layer includes at least one of the following: a pre-fc_norm_hidden fully connected normalization layer, located before the feedforward fully connected layer of the multi-token prediction module, which is the shape component. The direct fusion target; the pre-projection fully connected normalization layer, i.e., pre_fc_norm_embedding, is located before the embedded projection layer of the multi-token prediction module, and its own weights are fused into the weights of the corresponding fully connected interval; the multi-token prediction internal output normalization layer, i.e., the MTP internal output normalization layer, is located before the prediction header of the multi-token prediction module, and its own weights are fused into the weights of the multi-token prediction header.

[0047] In some specific examples, the steps of tracing downstream along the output path include: starting from the shared normalization layer node, recursively traversing its successor nodes; during the traversal, skipping the linear transformation layer, activation function layer, and attention mechanism layer, and only retaining the normalization layer node; recording the first normalization layer node encountered in the traversal path as the downstream normalization layer, and forming a normalization layer chain together with the initial shared normalization layer.

[0048] In some specific embodiments, the shared normalization layer and the downstream normalization layer in the multi-token prediction branch are both normalization layers based on root mean square operations, including RMSnorm and its variants; and under the condition of ignoring the numerical stability term ε or mean(x²) being much greater than ε, the forward calculation formula of the variant is: ;

[0049] in, Let be the input activation vector, y be the output activation vector, and rsqrt represent the reciprocal of the square root. To input the mean of the squares of each element, Here, u is the numerical stability constant, u is the learnable offset parameter that becomes a zero vector during the identity transformation, and 1+u is the effective weight. This indicates element-wise multiplication.

[0050] Step S120 involves decomposing the original effective weights to extract the scale-invariant shape component. The decomposed original effective weights include the global scaling magnitude and relative shape pattern. Since RMSNorm is scale-invariant and insensitive to the global scaling of the input, the global scaling component cancels out in the second normalization layer, meaning the global scaling component is not passed to the second normalization layer, thus avoiding any side effects, provided that the numerical stability term ε is ignored or the input mean square is much greater than ε. The shape component carries the relative importance information of the weights in each channel. This information is fully preserved and passed to the downstream layer; that is, the shape component is retained and used in the second normalization layer path. Traditional methods fuse the entire weight vector to the downstream layer, where the global scaling information is repeatedly processed in the downstream RMSNorm, leading to systematic biases in the denominator calculation. This invention, by first decomposing and then selectively passing the weights, eliminates the source of error at its root.

[0051] Specifically, step S120 includes:

[0052] Step S121, calculate the root mean square value of the original effective weight γ1: s = rms(γ1) = , as a global scale component.

[0053] Step S122: Divide the original effective weight γ1 by the root mean square value s to generate the shape component. =γ1 / s, where the shape component satisfies rms( =1.

[0054] In some embodiments, when calculating the global root mean square scale s, a numerical stability term δ can be introduced, and the formula for calculating the shape component is as follows: Where δ is a preset lower limit for a positive real number. Preferably, the range of δ can be set to 10. -8 Up to 10 -6 Among them, the shape component vectors satisfy scale invariance, that is... =1.

[0055] It should be noted that in this embodiment, when introducing the numerical stability term δ, it is necessary to satisfy rms(γ1)≥δ. At this time, rms( )=1, the formula for calculating shape components Valid. In the actual model, RMS ( ) is much larger than δ, such as the measured rms( )=2.14 and δ=10 -8 ,at this time and =γ1 / rms(γ1) are equal, and the shape components satisfy rms( =1.

[0056] Step S123: After decomposition, the original effective weights are decomposed into two independent components: the global scale component s = rms(γ1), which is a positive scalar representing the overall scaling magnitude of the original effective weights; and the shape component... , is a unit RMS vector representing the relative distribution pattern of the original effective weights. The global scale component is not passed to the second normalization layer, while the shape component is preserved and used in the second normalization layer path. Furthermore, subsequent language model head paths still fuse the original effective weights containing the global scale component or their equivalent full-scale weights.

[0057] It should be noted that the decomposition process is strictly reversible: the original effective weights Able to be precisely decomposed into RMS ( ) It can also be generated by RMS ( )and Accurate reconstruction, with no errors generated in either the decomposition or reconstruction process.

[0058] Step S130: For the language model head path, the original effective weights or their equivalent full-scale weights are fused into the language model head weights.

[0059] Specifically, the original effective weights γ1 are interposed along the hidden dimension with the weight matrix W of the language model head. lm Perform row-by-row multiplication:

[0060] ;

[0061] Where d is the dimension of the original effective weights, V represents the size of the vocabulary.

[0062] Step S140: For the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch, the shape component is fused element-wise with the weights of the second normalization layer. Specifically, the shape component is injected into the second normalization layer and multiplied element-wise with the channel scaling weights of that layer, which is equivalent to applying an independent scaling factor to each hidden dimension. This operation eliminates the global deterministic scale offset that is independent of the activation value, and the residual difference after fusion is determined only by the correlation between channels.

[0063] Specifically, step S140 includes: dividing the shape components The weights are multiplied element-wise with the weight parameter γ2 of the second normalized layer to obtain the fused weights. The calculation formula is: .

[0064] If the second normalization layer uses offset parameterization, the weight parameters are converted to the standard form γ2=1+u before fusion, and then converted back to the offset form u′= after fusion. -1, and write it back to the second normalization layer; where u is the learnable weight.

[0065] In some embodiments, when the stitching projection layer of a multi-token prediction branch has multiple input normalization layers, the shape component is only processed for the input normalization layer that receives the output of the shared normalization layer and is located on the output path of the shared normalization layer. The weights are multiplied element-wise; for other input normalization layers that do not receive the output of the shared normalization layer, they are fused into the corresponding input dimension range of the stitching projection layer according to their own weights, thus achieving many-to-one fusion.

[0066] During the element-wise weighted fusion process in step S140, the main residual is approximately the data-dependent channel covariance term. It also includes higher-order Taylor expansion terms, numerically stable terms, and small errors introduced by finite-precision calculations. This embodiment uses the channel covariance term as an example. That is, the introduced error only includes the data-dependent second-order residual error, and the factor of the second-order residual error... The calculation formula is:

[0067] ;

[0068] in, The input activation values ​​for the second normalized layer. For the normalized shape components, and rms( )=1, ensuring that first-order deterministic offsets are precisely eliminated; ρ is and The Pearson correlation coefficient along the channel dimension has a sign that depends on the channel correlation, and its absolute magnitude is usually about 1% or lower. Covariance along the channel dimension; The standard deviation of the square of the shape components; Input the standard deviation of the activation square; The input is the mean of the squared activation values.

[0069] It should be noted that when the weight matrix of the language model head exceeds a preset size threshold, the fusion process is executed in the central processing unit's memory to bypass the graphics processor's memory limitations. The preset size threshold can be determined based on the operator's experience or through multiple trials; no specific limit is specified here.

[0070] It should also be noted that the method of the present invention has strict sequential requirements, that is, after performing fusion processing on the weight parameters of the second normalization layer in the multi-token prediction branch, the operation of setting the shared normalization layer to an identity transformation in step S150 is then performed.

[0071] Step S150: Set the learnable scaling weights of the shared normalization layer to constant values. Specifically, after confirming that the weight adjustments for the language model head path and the multi-token prediction branch have been completed, perform the constant value setting; identify the type and parameterization form of the shared normalization layer, determine the configuration method of the constant values; and set the learnable scaling weights of the shared normalization layer to constant values ​​according to the determined configuration method.

[0072] In some specific embodiments, the step of identifying the type of the shared normalization layer and its parameterization form includes: determining whether the shared normalization layer is a standard root mean square normalization layer (RMSNorm) or a scaled normalization variant that does not contain independent biases and only has learnable channel scaling parameters; for the standard RMSNorm, the identity scaling weights are all-1 vectors; for the variant that adopts the (1+u) offset parameterization form, the offset parameter u is an all-0 vector.

[0073] Specifically, the learnable scaling weights of the shared normalization layer are set to an identical value, making it a normalization operation without learnable channel scaling; for the standard RMSNorm, the identical scaling weights are all 1s; for the (1+u) parameterized form, the offset parameter u is all 0s.

[0074] It is understandable that, after step S150 and before step S160, the method further includes: replacing the normalization layer instance of the identity transformation with an independent copy, so that the activation distribution of the quantization calibration phase is consistent with that of the inference phase.

[0075] Step S160 involves performing rotation matrix insertion, rotation fusion, and quantization on the fused computation graph. Specifically, an orthogonal rotation matrix is ​​constructed and inserted into the model computation graph; the orthogonal rotation matrix is ​​permanently fused into the linear layer weight matrix of the model; and low-bit quantization is performed on the fused weights.

[0076] It should be noted that the method of the present invention, such as steps S110 to S160, is executed sequentially by a unified orchestration function, which ensures a strict causal dependency between each step, avoids accuracy abnormalities caused by incorrect order, and improves the maintainability, portability and automated integration capability of the code.

[0077] In other embodiments of the present invention, a rotation quantization normalization layer fusion device for a post-normalized large language model is provided, capable of executing the rotation quantization normalization layer fusion method for a post-normalized large language model described above. The device includes: a parameter acquisition module configured to acquire the original effective weights of a shared normalization layer; a parameter decomposition module configured to calculate the global root mean square scale of the original effective weights and generate shape components; a main path fusion module configured to fuse the original effective weights or their equivalent full-scale weights into the language model head weights for the language model head path; a branch fusion module configured to fuse the shape components with the weights γ2 of the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch element-wise; an identity configuration module configured to set the learnable scalable weights of the shared normalization layer to an identity value; and a rotation quantization module configured to perform rotation matrix insertion, rotation fusion, and quantization on the fused computation graph.

[0078] Example 1

[0079] MTP Rotational Quantization System

[0080] System Architecture: This embodiment takes a large language model containing a post-norm MTP module as its object. The overall system process is divided into three stages:

[0081] Phase 1: Normalization Layer Fusion (fuse_layernorm): Identify shared normalization layers in the model, perform RMS normalization decomposition, and then fuse them into downstream layers. For the MTP branch, this includes the following sub-steps:

[0082] 1. Obtain a reference to the shared normalization layer model_norm (through the model_norm reference pre-stored during the model loading phase);

[0083] 2. Verify that model_norm has not been merged (by checking for non-identical weights using assertion guards);

[0084] 3. Read And perform RMS normalization to obtain ;

[0085] 4. Write the pre_fc_norm_hidden weights;

[0086] 5. Call the many-to-one fusion function to fuse pre_fc_norm_embedding and pre_fc_norm_hidden into the FC layer;

[0087] 6. Call the one-to-many fusion function to fuse the MTP internal output normalization layer into the MTP lm_head;

[0088] 7. Replace the model_norm reference in MTP with a copy of the identity transformation (to ensure that the activation distribution during the calibration phase is consistent with that during the inference phase).

[0089] Phase 2: Rotation Matrix Insertion (rotate_model): Construct an orthogonal rotation matrix R and insert it into the model calculation graph.

[0090] Phase 3: Fuse_rotate: Permanently fuse the rotation matrix into the weights, completing the preprocessing for rotation quantization.

[0091] The three stages are called sequentially by a unified orchestration function: fuse_layernorm → rotate_model → fuse_rotate.

[0092] The fusion process in Phase One of this embodiment will be described below:

[0093] (1) (1+u)RMSNorm convention adaptation: The RMSNorm of some models (such as Qwen3.5) adopts the (1+u) convention, that is, the forward computation is Under this convention, the identity transformation corresponds to u=0 (not u=1). During fusion, u must first be converted to the standard convention (u...). eff =u+1), after the fusion is complete, the weights will be reset to 0.

[0094] (2) Fusion order constraint: Normalization layer fusion of the MTP branch must be performed before global fusion. The reason is that MTP fusion needs to read the original weights of model_norm. If global fusion is executed first, and model_norm has already been set to the identity transformation, MTP fusion will read meaningless identity weights. This order constraint is ensured in the implementation through assertion guards.

[0095] (3) Identity Transformation Copy Replacement: After fusion, the _model_norm reference in the MTP layer needs to be replaced with an independent identity transformation copy. The reason is: after global fusion is performed, the real model_norm becomes the identity transformation; if the MTP still holds the original reference, the activation distribution in the calibration phase will be inconsistent with that in the inference phase. The replacement operation creates a new normalized layer instance and sets the weights to the identity values.

[0096] (4) CPU Bypass Fusion: For the lm_head and embedding layers, which are at the vocabulary size level, their weight matrices are large. The fusion operation can be performed on the CPU to bypass the GPU memory limit. This is an engineering optimization and does not affect mathematical equivalence. The key parameters in this embodiment are as follows: Figure 2 As shown.

[0097] Example 2

[0098] Multi-model adaptation: The method of this invention has been implemented on the post-norm MTP topology of Qwen3.5MoE and Qwen3.5Dense, such as... Figure 3 As shown.

[0099] Example 3

[0100] Experimental verification

[0101] 1) Synthetic testing and verification: A test model containing the model_norm→pre_fc_norm→fc chain was constructed using synthetic data. The output errors of the three fusion strategies were compared, and the comparison results are as follows: Figure 4 As shown. This embodiment tests cover 6 types. The distribution uses (identity, scalar, random, near_scalar, high_var, realistic) × 3 random seeds. In all test configurations, the relative L2 error of the rms_normed policy does not exceed the smaller of the scheme with full fusion and the scheme with discarded γ1.

[0102] Wherein, the complete fusion described in this invention means that γ1 is completely absorbed into the weights of the second normalized layer ( =γ1⊙γ2), without RMS normalization; discarding γ1 means not passing γ1 to the second normalization layer, only retaining its own weight γ2; rms_normed is the scheme of this invention, meaning that... =γ1 / rms(γ1) and γ2 are fused element-wise.

[0103] 2) End-to-end quantization validation (W8A8INT8 rotation quantization): W8A8INT8 rotation quantization was performed on two model sizes, Qwen3.5-9B and Qwen3.5-27B, and the end-to-end performance of the three MTP normalization layer fusion strategies was compared. The three sets of experiments for each model used the same rotation matrix and quantization parameters, with the only difference being the handling of the model_norm weights in the MTP branch.

[0104] like Figure 5 The image shown is a comparison of the MTP acceptance rates of Qwen3.5-9B (hidden_size=4096, 32 layers); as shown... Figure 6 The image shown is a comparison of the MTP acceptance rates of Qwen3.5-27B (hidden_size=5120, 64 layers); as shown... Figure 7 As shown in the cross-model comparison summary chart. Key findings:

[0105] (1) The strategy of this invention restores MTP accuracy to the FP16 level at both model sizes: the rms_normed Top-1 acceptance rate (0.6011) for 9B and the rms_normed Top-1 acceptance rate (0.6498) for 27B are both on par with their respective FP16 baselines (with differences within ±1% noise), indicating that the RMS normalization decomposition method recovers MTP accuracy to the FP16 level at different model sizes and at different FP16 levels. All values ​​are valid.

[0106] (2) Both erroneous fusion strategies degraded significantly in all configurations: the full fusion strategy resulted in a 7-9% decrease in the MTP Top-1 acceptance rate, and the discard γ1 strategy resulted in a 7-9% decrease in the Top-1 acceptance rate. This degradation occurred consistently across both model sizes, confirming the prevalence of the problem.

[0107] (3) The relative merits of the full fusion strategy and the discarding γ1 strategy are unstable: on 9B, the discarding γ1 strategy (0.5578) is superior to the full fusion strategy (0.5451); however, on 27B, the full fusion strategy (0.6029) is superior to the discarding γ1 strategy (0.5939). This indicates that the relative effectiveness of the two alternative strategies depends on the specific model. The distribution characteristics are not reliable universal solutions. Only the RMS normalization decomposition method of this invention stably recovers FP16 accuracy under all test configurations.

[0108] (4) Different The following methods are all effective: Actual measured original effective weights of the shared RMSNorm layer in Qwen3.5-9B / 27B. The global RMS scales are 2.14 and 1.94, respectively, which deviate significantly from 1. However, after decomposing them into global scale × scale-invariant shape components (rms=1) according to the method of this invention, the shape components are accurately normalized to unit RMS and participate in subsequent Norm→Norm fusion. Regardless of the value of the RMS scale, the fusion only introduces second-order residual error. The method of this invention remains effective and consistent under different model scales.

[0109] 3) Main path verification: Confirm that MTP fusion does not affect the main model. Taking 9B as an example, for instance... Figure 8 As shown, the KLD of the main model path is completely consistent, confirming that the MTP fusion change only affects the MTP branch and does not affect the main generation path.

[0110] The model parameter statistics in this embodiment are as follows: Figure 9 As shown.

[0111] In other embodiments of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the rotation quantization normalization layer fusion method for a post-normalized large language model as described in any of the above descriptions.

[0112] In other embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it can implement the rotation quantization normalization layer fusion method for post-normalized large language models as described above.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, prediction models, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] Those skilled in the art will understand that this invention identifies a shared normalization layer that simultaneously serves the language model head and the MTP branch, obtains its original effective weights, and decomposes them into global scale components and scale-invariant shape components. For the language model head path, the original effective weights or their equivalent full-scale weights are fused into the language model head weights to maintain the main path output unchanged. For the second normalization layer in the MTP branch located on the output path of the shared normalization layer, only the shape components and their weights are fused element-wise. Subsequently, the learnable scaling weights of the shared normalization layer are set to an identity value, making it a normalization operation without learnable channel scaling, and rotation quantization preprocessing is performed on this basis. This invention solves the problem that traditional normalization layer fusion methods cannot correctly perform rotation quantization in models containing MTP modules due to the chain structure from normalization layer to normalization layer, avoiding the approximately 9% MTP accuracy loss introduced by traditional fusion methods. This invention enables, for the first time, rotational quantization to be correctly applied to post-normalized models containing MTP modules, and restores the accuracy of MTP branches to the FP16 baseline level. The core operation requires only one scalar division (such as calculating the root mean square value and normalizing it), which is performed in the offline preprocessing stage. It has no impact on the model inference latency, the main path output remains completely unchanged, and the MTP branch only introduces about 1% of the second-order residual error, which is on the order of the chain structure fusion residual error from the shared normalization layer to the second normalization layer.

[0116] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0117] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0118] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A rotation quantization normalization layer fusion method for a post-normalized large language model, characterized in that, The large language model includes a shared normalization layer and a multi-token prediction branch; The output of the shared normalization layer has one branch entering the language model head and the other branch entering the second scale-invariant normalization layer in the multi-token prediction branch, forming a chain structure from the shared normalization layer to the second normalization layer; the method includes: Obtain the original effective weights of the shared normalized layer; The original effective weights are decomposed to extract the scale-invariant shape components; For the language model head path, the original effective weights or their equivalent full-scale weights are fused into the language model head weights; For the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch, the shape component is fused with the weight of the second normalization layer element by element; Set the learnable scaling weights of the shared normalization layer to an constant value; Rotation matrix insertion, rotation fusion, and quantization are performed on the merged computation graph.

2. The method according to claim 1, characterized in that, The shared normalization layer is located at the final output of the large language model decoder and adopts a post-normalization architecture. The step of obtaining the original effective weights of the shared normalization layer includes: locating the corresponding normalization layer instance in the module hierarchy of the large language model based on the identifier of the shared normalization layer; extracting the original weight tensor from the normalization layer instance and creating an independent copy of the original weight tensor to disconnect it from the parameters of the large language model; performing precision conversion and format normalization on the independent copy to generate a normalized weight vector; and... The step of extracting the original weight tensor from the normalized layer instance includes: traversing a preset list of candidate parameter names and searching for learnable weight parameters from the normalized layer instance; if the learnable weight parameter adopts an offset parameterization form, then converting the offset parameter into an effective weight form; wherein, the offset parameterization form follows the (1+u) convention, where u is the learnable offset; the conversion into an effective weight form includes: calculating the original effective weight γ1=1+u, where 1 is an all-1 vector with the same dimension as u.

3. The method according to claim 1, characterized in that, The step of decomposing the original effective weights and extracting the scale-invariant shape components includes: Calculate the root mean square value s of the original effective weight γ1 using the formula s = rms(γ1) = The root mean square value s is used as a global scale component. Divide the original effective weight γ1 by the root mean square value s to generate the shape component. =γ1 / s, wherein the shape component satisfies rms( )=1; After decomposition, the original effective weights are decomposed into two independent components: a global scale component s = rms(γ1), a positive scalar representing the overall scaling magnitude of the original effective weights; and a shape component. , is a unit RMS vector representing the relative distribution pattern of the original effective weights; wherein, the global scale component is not passed to the second normalization layer, and the shape component is preserved and used in the second normalization layer path; The decomposition process is strictly reversible: the original effective weights Able to be precisely decomposed into RMS ( ) It can also be generated by RMS ( )and The decomposition and reconstruction processes are accurate and do not produce errors.

4. The method according to claim 1, characterized in that, The step of fusing the original effective weights or their equivalent full-scale weights into the language model head weights for the language model head path includes: The original effective weights γ1 are then intersected along the hidden dimension by the weight matrix W of the language model head. lm Perform row-by-row multiplication: ； Where d is the dimension of the original effective weights. V represents the size of the vocabulary.

5. The method according to claim 3, characterized in that, The step of fusing the shape component with the weights of the second normalization layer element-wise in the second normalization layer located on the output path of the shared normalization layer in the multi-token prediction branch includes: The shape components The weights are multiplied element-wise with the weight parameter γ2 of the second normalized layer to obtain the fused weights. The calculation formula is: ; If the second normalization layer uses offset parameterization, the weight parameters are converted to the standard form γ2=1+u before fusion, and then converted back to the offset form u′= before fusion. -1, and write back to the second normalized layer; where u is the learnable offset.

6. The method according to claim 5, characterized in that, When the stitching projection layer of the multi-token prediction branch has multiple input normalization layers, the shape component is only processed by the input normalization layer that receives the output of the shared normalization layer and is located on the output path of the shared normalization layer. Multiply the weights element by element; for other input normalization layers that do not receive the output of the shared normalization layer, fuse them into the corresponding input dimension range of the stitching projection layer according to their own weights.

7. The method according to claim 5, characterized in that, The error introduced by the element-wise weight fusion process is approximately a data-dependent second-order residual error, and the factor of the second-order residual error is... The calculation formula is: ； in, The input activation values ​​for the second normalized layer. The normalized shape components and RMS ( )=1; ρ is and Pearson correlation coefficient along the channel dimension.

8. The method according to claim 1, characterized in that, The step of setting the learnable scalability weights of the shared normalization layer to constant values ​​includes: confirming that the weight adjustments for the language model head path and the multi-token prediction branch have been completed, and then performing the constant value setting; identifying the type and parameterization form of the shared normalization layer, and determining the configuration method for the constant values; and setting the learnable scalability weights of the shared normalization layer to constant values ​​according to the determined configuration method; and... The step of identifying the type and parameterization form of the shared normalization layer includes: determining whether the shared normalization layer is a standard root mean square normalization layer or a scaled normalization variant without independent bias and only having learnable channel scaling parameters; for the standard root mean square normalization layer, the identity scaling weights are all-1 vectors; for the variant using the (1+u) offset parameterization form, the offset parameter u is an all-0 vector.

9. The method according to claim 1, characterized in that, The process of performing rotation matrix insertion, rotation fusion, and quantization on the fused computation graph includes: Construct an orthogonal rotation matrix and insert it into the model computation graph; The orthogonal rotation matrix is ​​then incorporated into the linear layer weight matrix of the model; Low-bit quantization is performed on the merged weights.

10. A rotation quantization normalization layer fusion device for a post-normalized large language model, characterized in that, The apparatus is capable of performing the rotation quantization normalization layer fusion method for post-normalized large language models as described in any one of claims 1 to 9; the apparatus comprises: The parameter acquisition module is configured to acquire the original effective weights of the shared normalization layer. The parameter decomposition module is configured to calculate the global scale component of the original effective weights and use the shape component for the second normalization layer path; The main path fusion module is configured to fuse the original effective weights or their equivalent full-scale weights into the language model head weights for the language model head path. The branch fusion module is configured to fuse the shape component with the weight γ2 of the second normalized layer on an element-wise basis for the second normalized layer located on the output path of the shared normalized layer in the multi-token prediction branch; The identity configuration module is configured to set the learnable scaling weights of the shared normalization layer to an identity value. The rotation quantization module is configured to perform rotation matrix insertion, rotation fusion, and quantization on the fused computation graph.

Images