Three-value low-rank compression initialization method and system for large language model

Abstract

The application relates to the technical field of artificial intelligence, and discloses a ternary low-rank compression initialization method and system for a large language model. The input importance vector and the output importance vector are counted on a calibration set, and a proxy weight matrix is constructed by performing bilateral diagonal scaling on an original weight matrix. Continuous low-rank decomposition is performed on the proxy weight matrix to obtain a continuous low-rank starting point. The update of continuous variables and ternary projection variables is split by using an alternating direction multiplier method, a continuous matrix is represented as the product of row scales, a ternary structure matrix and column scales by TSVID projection, intermediate scales are merged, and inverse scaling is absorbed into deployment scales, and an initialization result composed of the ternary structure matrix and the continuous scaling vector is derived. The method helps to reduce ternary projection mismatch, and does not introduce additional dense matrices in the reasoning stage.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a ternary low-rank compression initialization technique for linear layers of neural network models. Background Technology

[0002] With the widespread deployment of large language models in online services such as intelligent dialogue, code-assisted generation, and document summarization, the main bottleneck for model inference has gradually shifted from insufficient computing power to limited GPU memory capacity, GPU memory bandwidth, and weight memory access efficiency. Taking a dialogue service on a data center GPU cluster as an example, autoregressive decoding requires repeatedly reading a large number of weights from various linear layers such as the attention projection layer and feedforward network to generate each token. The model as a whole exhibits obvious memory-constrained characteristics, and the amount of weight read directly affects decoding latency and service throughput. Furthermore, when deploying large language models or multimodal models on edge devices such as mobile phones and in-vehicle terminals, storage space and memory access bandwidth are even more strained, often requiring the linear layer weights to be compressed to extremely low bits to meet deployment requirements. In addition, visual language models and other models containing large-scale linear layers also face the problem of excessive weight memory access overhead in batch inference scenarios.

[0003] To reduce deployment costs in the aforementioned scenarios, post-training compression, extremely low-bit quantization, and structured low-rank decomposition for linear layers have become important technical directions. However, existing low-rank decomposition or quantization methods typically directly minimize the unweighted reconstruction error in the weight space, failing to explicitly consider the differences in activation energy between different input channels and the differences in sensitivity between different output channels. This makes it difficult to prioritize the protection of task-sensitive regions with limited representational capacity. Existing ternary projection methods often employ a single global scaling factor or simple channel-by-channel scaling, which struggles to adapt to the non-uniformity of row and column amplitudes within the low-rank factor, and projection mismatch easily occurs when continuous low-rank solutions are converted into ternary structures. Furthermore, the search process for optimization starting from a random ternary structure has unstable convergence, while relying solely on a single hard projection easily solidifies erroneous zero values ​​or sign selections. Therefore, a new technical solution is urgently needed to address these issues. Summary of the Invention

[0004] The purpose of this application is to provide a ternary low-rank compression initialization method and system for large language models, so as to alleviate the structural mismatch caused by projecting continuous low-rank solutions onto deployable ternary structures under the dual constraints of limited rank and ternary discrete representation capacity, and reduce the degree to which the reconstruction quality of task-sensitive channels is damaged in the initialization stage, thereby solving the problems mentioned in the background art.

[0005] To achieve the above objectives, embodiments of this application provide a ternary low-rank compression initialization method for large language models, comprising: statistically analyzing the input activation and output-side sensitivity corresponding to the original weight matrix of the linear layer in the large language model on a calibration set to obtain input importance vectors and output importance vectors; performing bilateral diagonal scaling on the original weight matrix to construct a surrogate weight matrix; performing continuous low-rank decomposition on the surrogate weight matrix to obtain a continuous low-rank starting point under a given rank, wherein the continuous low-rank starting point includes a left factor and a right factor; using the continuous low-rank starting point as the initial value and based on the surrogate weight matrix, using the alternating direction multiplier method to split the update of continuous variables from the update of ternary projected variables, and alternately updating the continuous variables. The process continues until a stopping condition is met. Specifically, with the three-valued projected variables fixed on one side, a closed least squares subproblem involving continuous variables on the other side is solved based on the surrogate weight matrix. The sum of the obtained continuous variables and their corresponding dual variables is then projected onto the three-valued projected variables on that side using a three-valued symbolic-valued independent decomposition projection (TSVID). This yields a left factor containing a left three-valued structure matrix and a right factor containing a right three-valued structure matrix. The intermediate scales of the left and right factors are merged into an intermediate-rank scale, and the inverse scaling corresponding to the input and output importance vectors is absorbed into the deployment scale, resulting in a three-valued low-rank initialization result composed of a three-valued structure matrix and a continuous scaling vector.

[0006] The embodiments of this application also provide a ternary low-rank compression initialization system for large language models, comprising: a surrogate weight construction module, used to statistically analyze the input activation and output-side sensitivity corresponding to the original weight matrix of the linear layer in the large language model on a calibration set, to obtain input importance vectors and output importance vectors, and to perform bilateral diagonal scaling on the original weight matrix to construct a surrogate weight matrix; a continuous low-rank initialization module, used to perform continuous low-rank decomposition on the surrogate weight matrix to obtain a continuous low-rank starting point under a given rank, including left and right factors; and a splitting solution module, used to use the continuous low-rank starting point as the initial value and base... For the surrogate weight matrix, the alternating direction multiplier method is used to split the update of continuous variables from the update of ternary projected variables. The continuous variables, ternary projected variables, and dual variables are updated alternately until the stopping condition is met. The ternary sign-value independent decomposition projection is used to obtain the left factor containing the left ternary structure matrix and the right factor containing the right ternary structure matrix. The scale merging and derivation module is used to merge the intermediate scales of the left and right factors into the intermediate rank scale, and absorb the inverse scaling corresponding to the input importance vector and the output importance vector into the deployment scale, and derive the ternary low-rank initialization result composed of the ternary structure matrix and the continuous scaling vector.

[0007] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described ternary low-rank compression initialization method.

[0008] In the above technical solution, by statistically analyzing the input activation and output sensitivity of the linear layer on the calibration set to obtain the input importance vector and output importance vector, and constructing a surrogate weight matrix by diagonal scaling on both sides, the compression target changes from uniform weight error to a weighted target that approximates the non-uniform response of the task loss. The limited rank and ternary discrete representation capacity are no longer simply distributed equally according to the original weight error, but are preferentially allocated to the task-sensitive region, which helps to improve the overall quality of the very low bit compression initialization. After obtaining a continuous low-rank starting point in the surrogate space, the alternating direction multiplier method is used to split the update of continuous variables from the update of ternary projected variables. The continuous variables can freely move towards reducing the reconstruction error. The difference shifts in direction, and the three-valued projection variables always satisfy the discrete constraints required for deployment. The two gradually become consistent through the pull of the dual variables, reducing the risk of solidifying erroneous zero values ​​or erroneous signs in a single hard projection. The continuous matrix is ​​represented as the product of the row scale, the three-valued structure matrix, and the column scale through three-valued sign-value independent decomposition projection, thus decoupling the selection of discrete structure and the fitting of continuous amplitude. The intermediate scales of the left and right factors are then merged into the intermediate rank scale, and the inverse scaling corresponding to the input and output sensitivity is absorbed into the deployment scale. Finally, the initialization result consisting only of the three-valued structure matrix and a small number of continuous scaling vectors is derived, which reduces the weight storage and memory access overhead while maintaining the simplicity of the deployment form.

[0009] Furthermore, when the input importance vector is represented by the square root of the square mean of the input activations and the output importance vector is represented by the square root of the output activation energy or the output gradient energy, the importance statistics can be implemented in scenarios where only forward propagation is performed, and can also be more closely related to the actual response of the task to the output channel in scenarios with task loss. At the same time, the bilateral weighted reconstruction objective is equivalently transformed into a standard least squares problem in the surrogate space. The compression process does not need to explicitly carry weight terms in each iteration, and the compression result can be restored to the original weight space through inverse transformation, thus taking into account both implementation convenience and mathematical equivalence.

[0010] When continuous low-rank decomposition adopts truncated singular value decomposition and absorbs the square roots of the singular values ​​into the left and right factors respectively, the resulting starting point is the optimal continuous low-rank approximation under a given rank, and the numerical magnitudes of the two factors remain balanced, which is beneficial to the stable estimation of the row and column scales in subsequent ternary projections. Subsequent iterations only need to find deployable discrete structures near this low-error starting point, instead of starting the search from random ternary structures, which reduces the difficulty of discrete search and convergence instability.

[0011] During the alternating update process, after fixing the three-valued projection variables on one side, the subproblem of the continuous variables on the other side is a convex quadratic problem with a closed solution. The inverse object is only a small matrix of rank dimension, the update direction is clear and the computational cost is low. The cumulative update of the dual variables continuously records the residual deviation between the continuous variables and the three-valued projection variables into the iteration process, so that the next round of continuous update and three-valued projection both carry this deviation information, which promotes the convergence of the two sets of variables in the iteration.

[0012] In the ternary projection stage, the row and column scales are initialized with the rank-one approximation of the amplitude matrix, so that the initial scale matches the true amplitude structure of the matrix and reduces oscillations in the alternating update. Half of the product of the row and column scales is used as a local threshold to determine the ternary value element by element, so that the zeroing threshold adapts to the amplitude level of the row and column, which can fit the non-uniform distribution of the row and column amplitudes within the low-rank factor more accurately than the global threshold. The closed alternating update of the row and column scales, combined with scale normalization and the absorption of the negative sign into the ternary structure matrix, eliminates the non-uniqueness of the represented scale, so that the row and column scales always represent non-negative amplitudes, and the projection process is more stable.

[0013] When the stopping condition is jointly gated by the global residual and the local residual of the task-sensitive channel, the channels with the highest importance are identified as the set of highly sensitive channels, and their local relative reconstruction residuals are introduced into the stopping decision. If the global residual meets the standard but the local residual does not, the iteration continues. This avoids the local projection mismatch of the highly sensitive channel being masked by the global average error without adding any statistics. Under the same iteration budget, it helps to reduce the reconstruction mismatch on the channels that have a greater impact on the model task, making the initialization results better in terms of task loss.

[0014] In the scheme that includes stabilization processing, the use of historical scale or zeroing processing for near-zero scale anomalies avoids division-by-zero divergence in closed-loop updates. The diagonal stabilization of ill-conditioned inversion ensures the numerical stability of solving continuous subproblems. The mechanism of backtracking to the historical optimum and using the maximum number of iterations as the upper limit when there is no improvement for several consecutive rounds constitutes a convergence closed loop, enabling the iteration process to terminate and derive better results from the historical records. The lower limit protection of the deployment scale avoids the abnormal amplification of the deployment scale by inverse scaling when the importance tends to zero. The above processing works together to help improve the numerical stability and deployment stability of the entire iteration and derivation process.

[0015] When the above scheme is applied to the projection layers and feedforward linear layers of the Transformer structure in large language models, visual language models, or multimodal models, the derived initialization results directly replace the original weight representations. No additional dense matrices are introduced during the inference phase, nor are the connection relationships of the original linear layers changed. This is particularly suitable for engineering deployment scenarios with limited GPU memory, bandwidth, and storage. The corresponding system implements the above processing through a proxy weight construction module, a continuous low-rank initialization module, a splitting solution module, and a scale merging and derivation module. It can be deployed on servers, model compression platforms, model compilation platforms, or other computing devices with processors and memory. Computer-readable storage media storing the corresponding computer program also implement the above processing when the program is executed by a processor, achieving the same technical effects as the method described above.

[0016] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a ternary low-rank compression initialization method for large language models according to an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of a ternary low-rank compression initialization system for large language models according to an embodiment of this application. Detailed Implementation

[0019] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0020] Explanation of some concepts Linear layers refer to network layers in neural networks that implement linear transformations by multiplying the weight matrix with the input vector. Examples include the query projection layer, key projection layer, value projection layer, and output projection layer in the Transformer structure, as well as the upper projection layer, gated projection layer, and lower projection layer in feedforward networks.

[0021] A calibration set is a small collection of samples extracted from the target corpus to statistically analyze the activation or gradient information within the model without updating the model parameters.

[0022] The input importance vector is a vector obtained statistically from the calibration set by input channel, used to characterize the activation energy level of each input channel, denoted as . , its first Each element is obtained by taking the square root of the square mean of the input activation values ​​of that input channel.

[0023] The output importance vector is a vector obtained statistically from the output channels on the calibration set, used to characterize the sensitivity of each output channel, and is denoted as... , its first Each element is obtained by taking the square root of the output activation energy or the output gradient energy of the output channel.

[0024] A proxy weight matrix refers to a matrix that is a proxy weight matrix for the original weight matrix. according to The matrix obtained after bilateral diagonal scaling transforms the weighted reconstruction objective into a standard least squares problem in the surrogate space.

[0025] Rank refers to the intermediate dimension shared by the left and right factors in a low-rank decomposition, denoted as . Its value determines the expressive capacity of the low-rank approximation.

[0026] Truncated singular value decomposition (SVD) is a matrix decomposition method that retains only the largest singular values ​​and their corresponding singular vectors after performing SVD on a matrix. It is used to construct the optimal continuous low-rank approximation for a given rank.

[0027] Alternating Direction Multiplier Method (ADMM) is an optimization method that splits a complex optimization problem into several subproblems that can be solved alternately by introducing auxiliary and dual variables. In this application, it is used to split the update of continuous variables from the update of three-valued projected variables.

[0028] A ternary structure matrix is ​​a matrix whose elements are selected from only a few values. , , A matrix of three discrete values, denoted as , or , used for symbolic and sparse structures that express weights with extremely low bit density.

[0029] Three-valued projected variables refer to low-rank factors that can be deployed on the left or right side after TSVID projection, satisfying the constraints of a three-valued discrete structure. For example... and .

[0030] Dual variables refer to the variables used in ADMM iterations to accumulate the deviation between continuous variables and three-valued projected variables, for example... and .

[0031] Ternary Sign-Value Independent Projection (TSVID) refers to the projection of a continuous matrix onto a given surface. Represented as the product of the row scale vector, the ternary structure matrix, and the column scale vector. The projection operator, where the three values ​​are determined element-wise by a local threshold determined by the product of the row and column scales.

[0032] Scale normalization refers to adjusting the overall factor between row and column scales to eliminate representational distortion. The process of determining the non-uniqueness of the scale and absorbing the sign of the negative scale into the three-valued structure matrix.

[0033] The middle-rank scale refers to the column scale vector of the left factor. Row scaling vector with right factor Vector obtained by element-wise multiplication It is located between the left ternary structure matrix and the right ternary structure matrix.

[0034] Task-sensitive channels refer to input or output channels that are ranked highly in importance based on the input or output importance vectors; they are also known as high-sensitivity channels.

[0035] Local relative reconstruction residual refers to the relative reconstruction error calculated for the row where the high-sensitivity output channel is located or the column where the high-sensitivity input channel is located. It is used together with the global proxy reconstruction residual to gate the stopping of the iteration.

[0036] Deployment scale refers to the continuous scaling vector that absorbs the inverse scaling of the input importance vector and the output importance vector during the derivation stage. It is used to restore the three-valued low-rank initialization result to the original weight space and directly use it for inference deployment.

[0037] Memory-bound refers to a computational process whose time consumption is primarily constrained by memory capacity and memory access bandwidth, rather than by the computing power of the processing unit; autoregressive decoding of large language models typically exhibits this characteristic.

[0038] Autoregressive decoding refers to a decoding method that generates tokens sequentially, with each generation relying on the already generated content and requiring the reading of model weights to perform forward computation.

[0039] The technical concept of this application is explained below with reference to specific application scenarios: In summary, in the specific application scenario of autoregressive decoding of large language models, generating each token requires repeatedly reading a large number of weights from the linear layer, resulting in a significant memory-constrained model. Memory capacity, memory bandwidth, and weight access efficiency constitute deployment bottlenecks. The technical problem this application aims to solve is not simply improving compression performance, but rather, under the dual constraints of limited rank and ternary discrete representation capacity, how to avoid structural mismatch when projecting continuous low-rank optimal solutions onto deployable ternary structures, and how to prevent the reconstruction quality of task-sensitive channels from being compromised during the initialization phase. Technically, this application does not simply sequentially superimpose calibration weighting, low-rank decomposition, and ternary quantization. Instead, it unifies task sensitivity, continuous low-rank approximation, and deployable ternary structure constraints into a single proxy space for collaborative processing (steps 100 to 400 in Figure 1): first, input activation and output-side sensitivity are statistically analyzed on the calibration set to obtain the input importance vector. With output importance vector ,according to By performing a bilateral diagonal scaling on the original weight matrix, the non-uniform response of the task loss to each channel is a priori injected into the weight space, making the optimization of the standard Frobenius error in the surrogate space equivalent to optimizing the task-sensitive weighted objective. Then, a deterministic continuous optimal starting point under a given rank is constructed in the surrogate space using truncated singular value decomposition, so that the discrete search does not have to start from a random three-valued structure. Based on this, the alternating direction multiplier method is used to transform the continuous variables... , Closed-form least squares update and three-valued projected variables , Discrete projection phase splitting, with dual variables , By binding the two sets of variables together without substituting them, the risk of a single hard projection solidifying erroneous zero values ​​or incorrect signs is mitigated; while the TSVID operator, which performs the projection function, represents the matrix to be projected as... Using half of the product of row and column scales as the local threshold for each element precisely matches the non-uniform distribution of row and column amplitudes within the low-rank factor. It is worth emphasizing that there is a deep interdependence and synergy among the above features: the bilateral weighting determines that the principal components retained by the truncated singular value decomposition are biased towards high-sensitivity channels; this continuous starting point defines the search neighborhood of the ADMM iteration; and the row and column scale structure of TSVID merges with the intermediate scale in step 400. And the inverse scaling absorbs the natural connection, so that the sensitivity scaling is ultimately absorbed into the deployment scale, resulting in This result, containing only a three-valued structure matrix and a small number of continuously scaled vectors, does not introduce additional dense matrices during the inference phase and does not change the original linear layer connectivity. The input and output importance vectors are also reused for the iteration termination gating, when the global residual... Local residuals of high-sensitivity channels that meet standards or Even when the target is not met, iteration continues, ensuring that the amplified task-sensitive region in the proxy space remains constrained during the discrete projection and scale merging stages. The aforementioned steps are not arbitrarily interchangeable parallel methods, but rather converge step-by-step along the same technical concept: task-sensitive proxy construction, continuous optimal starting point, discrete split projection, local residual gating, and deployable scale derivation. If examined separately, none of them can independently achieve the effects obtained in this application. It is precisely their organic coupling in a specific way that enables compressed initialization to balance task sensitivity, low reconstruction error, and deployment friendliness under given constraints.

[0040] Furthermore, the inventors of this application, through long-term in-depth research, discovered that the performance of existing compression methods significantly degrades under extremely low bit conditions, not solely due to the reduction in rank. The root cause lies in a systematic deviation between the optimization objective and the actual task loss of the model: directly minimizing the unweighted reconstruction error is equivalent to assuming that all input and output channels have approximately the same impact on the task loss. However, in the Transformer model, the activation energies of different input channels differ significantly, and the impacts of different output channels on the residual flow, attention distribution, and subsequent layer activations are also different. This results in the limited rank and low bit expression capacity being allocated to task-insensitive regions, and the reconstruction quality of highly sensitive channels cannot be guaranteed. After a thorough analysis of the principles of existing ternary projection, the inventors further discovered that continuous low-rank optimal solutions exhibit structural mismatch when projected onto discrete ternary structures. This is because the amplitude distribution of different rows and columns within the low-rank factor is highly non-uniform, and a single global scaling factor or global threshold cannot simultaneously describe the coupling relationship between row scale, column scale, and ternary structure. The reason existing technologies cannot effectively solve this problem is also due to their lack of a mechanism to decouple continuous updates from discrete projection—the search space is too large and convergence is unstable when starting from a random ternary structure, and the erroneous zero value or sign selected by a single hard projection becomes fixed in subsequent stages and cannot be corrected. More subtly, through observation of the iterative convergence process, the inventors also realized that even if the overall reconstruction error has decreased, local mismatches on highly sensitive channels may still be masked by the global average error, making the initialization results unstable in terms of task loss. Based on the above in-depth research, the inventors of this application propose an innovative technical solution: First, a proxy space is constructed by bilaterally weighting the original weights using input-output sensitivity. A deterministic continuous low-rank starting point is obtained in the proxy space. Then, the updates of continuous variables and ternary variables are split using the alternating direction multiplier method. Discretization is completed by simultaneously modeling row-scale, column-scale, and ternary structure projection operators. Local residuals from task-sensitive channels participate in the termination control of the iteration. Finally, the scales are merged, and a directly deployable ternary low-rank initialization result is derived. The implementation process of this application is described in detail below through specific embodiments.

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this application. Unless otherwise specified, the following embodiments and their technical features can be combined with each other. For neural network inference frameworks, matrix operations, numerical implementations of singular value decomposition, storage encoding of ternary matrices, and conventional numerical optimization processes not described in detail herein, those skilled in the art can configure them based on common knowledge and existing engineering implementations.

[0042] Example 1 This embodiment provides a ternary low-rank compression initialization method for large language models, used for compressed initialization of linear layers in a neural network model after training. The neural network model described here can be a large language model, a visual language model, or a multimodal model, containing a large number of linear layers. These linear layers include, for example, the query projection layer, key projection layer, value projection layer, and output projection layer in a Transformer structure, and at least one of the up projection layer, gated projection layer, and down projection layer in a feedforward network. In inference scenarios such as autoregressive decoding, generating each token requires repeatedly reading a large number of weights from these linear layers, resulting in significant memory constraints for the model. GPU memory capacity, GPU memory bandwidth, and weight access efficiency become deployment bottlenecks. The method in this embodiment addresses this technical problem by generating a directly deployable ternary low-rank initialization result for the linear layers, composed of a ternary structure matrix and a small number of continuously scaled vectors, under the condition of limited calibration data. As shown in Figure 1, the method generally includes steps 100 to 400, which are described one by one below.

[0043] Step 100: Construct a task-sensitive proxy weight matrix Let the original weight matrix of the linear layer to be compressed be... ,in This represents the output dimension (i.e., the number of output channels) of the linear layer. Let be the input dimension (i.e., the number of input channels) of this linear layer, and let the compressed weight matrix be denoted as . Traditional low-rank decomposition or quantization methods typically minimize the unweighted weight reconstruction error, i.e.: in, The Frobenius norm of the matrix is ​​represented by the square root of the sum of the squares of all its elements. Equation 1 implicitly assumes that different input channels and different output channels have approximately the same importance to the final task loss. However, in the Transformer model, the activation energies of different input channels differ significantly, and different output channels also have different effects on the residual flow, attention distribution, and subsequent layer activations. If Equation 1 is still used as the compression target, the limited rank and low bit representation capacity are easily allocated to task-insensitive regions, leading to a significant decrease in the performance of the compressed model. Therefore, this embodiment explicitly injects task sensitivity into the weight space through steps 110 to 130.

[0044] Step 110: Statistically analyze the input activations of the linear layer on the calibration set. Specifically, extract a small number of calibration samples from the target corpus, perform forward propagation on the neural network model, and collect the activation data of the linear layer. Input activation of each input channel And calculate the input importance vector accordingly. , its first Each element is determined by the following formula: in, This indicates the averaging operation across the sample, token location, or mini-batch statistical dimension of the calibration set. Input importance vector The The square and square root operations of each element are used to characterize the activation energy level of the input channel. The physical meaning is that the higher the activation energy of the input channel, the more its corresponding weight column is amplified in the forward computation, and the greater the impact of the reconstruction error on the output after propagation through this channel; therefore, it should receive a higher reconstruction priority.

[0045] Step 120: Statistically analyze the output-side sensitivity of the linear layer on the calibration set to obtain the output importance vector. As an aperture that can be implemented with only forward propagation, the output-side sensitivity is determined by the first... Activate output of each output channel Energy characterization: in, To output the importance vector The The element. As an alternative implementation approach, in scenarios where task loss, distillation loss, or verification loss can be obtained, backpropagation can also be performed on the neural network model, by the first element. Output gradient of each output channel Energy characterization of output-side sensitivity: It should be noted that when obtaining labels or performing backpropagation is inconvenient in engineering scenarios, the forward feasible approach shown in Equation 3 can be used; when the task loss is available, the gradient energy approach shown in Equation 4 can make the output importance more closely reflect the actual response of the task loss to the output channel. The higher the output activation energy or output gradient energy, the greater the impact of the reconstruction error on subsequent layers and even the final task loss. Therefore, the above two importance vectors naturally constitute the basis for differentiated protection of different rows and columns of the weight matrix. Furthermore, the statistical approach of the output importance vector is not limited to the above two; it can also be estimated by Fisher information, Hessian diagonal approximation, residual flow sensitivity, or downstream task verification loss changes. These methods can all serve as alternative statistical approaches. Optionally, when the number of calibration samples is small, leading to… , When the statistical variance is too large, it can be adjusted... , After performing stabilization processes such as window smoothing or lower bound truncation, proceed to the next step.

[0046] Step 130, based on the input importance vector and output importance vector The uniform reconstruction objective shown in Formula 1 is rewritten as a two-sided weighted reconstruction objective: in, This represents a diagonal matrix with the vector within the parentheses as its diagonal elements. Formula 5 is equivalent to weighting each row of the reconstruction error matrix according to the output importance and each column according to the input importance, so that the errors on the highly sensitive channels have a greater weight in the objective function. Further, a surrogate weight matrix is ​​defined. Equation 5 is transformed into a standard least squares problem in the surrogate space: in, This represents the reconstruction result in the surrogate space. After this transformation, subsequent low-rank decomposition and ternary projection only need to optimize the standard Frobenius error in the surrogate space, without explicitly carrying weighting terms in each iteration, thus mathematically equivalent to optimizing a task-sensitive weighted objective. After compression, the compressed weights in the original weight space can be recovered through an inverse transformation: in, , These are the inverse matrices of the corresponding diagonal matrices. It should be noted that the two-sided inverse scaling in Equation 7 can be absorbed into the scaling vectors on both the input and output sides in actual deployment (see step 400 for details), therefore no additional dense matrix multiplication is needed during the inference phase. After processing in step 100, regions with high input importance or high output sensitivity are identified in the proxy weight matrix. The region is amplified accordingly, and the subsequent low-rank ternary decomposition will prioritize protecting these regions within the limited representation capacity. This is the specific meaning of the term "task sensitive" in this embodiment.

[0047] Step 200: Perform continuous low-rank decomposition on the surrogate weight matrix to obtain the continuous low-rank starting point. Obtaining the proxy weight matrix This step then constructs a deterministic, continuous starting point for subsequent discrete optimization. First, let's define the objective representation in this embodiment: given the objective rank... (That is, the intermediate dimension), the goal of compression is to... It is approximately decomposed into the product of left and right factors, and further parameterized into a combination of a three-valued structure matrix and a small number of full-precision scaling vectors: in, , These are the deployable low-rank factors after left and right ternary projections, respectively; It is a left-hand ternary structure matrix. For right-hand ternary structure matrices, only the elements of both are taken. , , Three discrete values; , , These are the continuous scaling vectors for the output side, the intermediate rank side, and the input side, respectively. The significance of Equation 8 lies in decoupling the "discrete structure selection" from the "continuous amplitude fitting": the three-valued matrix is ​​responsible for expressing the sign and sparse structure of the weights, requiring only a minimal number of storage bits; the continuous scaling vector is responsible for recovering the amplitudes on different channels, and its storage overhead is negligible compared to the matrix itself. This representation makes the reconstruction under low-bit constraints more stable while maintaining a simple deployment configuration.

[0048] Specifically, in step 210, before introducing the ternary constraints, first... Perform truncated singular value decomposition, retaining the largest. Each singular value and its corresponding singular vector: in, , They are respectively The left and right singular vector matrices, It is a singular value diagonal matrix, with superscript... Indicates matrix transpose; , They are respectively the front A matrix composed of left and right singular vectors. For the front A diagonal matrix composed of singular values. To The diagonal matrix obtained by taking the square root of each singular value in the matrix; , These are the left and right factors, representing consecutive low-rank starting points. Averaging the square roots of the singular values ​​into the two factors helps maintain a balance in their numerical magnitudes, facilitating stable estimation of the row and column scales in subsequent three-valued projections. According to the Eckart-Young theorem, Given rank Down The optimal continuous low-rank approximation is obtained. It is important to emphasize that the role of ADMM in subsequent step 300 is not to find arbitrary low-rank solutions again, but rather to find a deployable structure with low error that satisfies the ternary discrete constraints near this continuous optimal starting point. Compared to optimizing from a random ternary structure, this deterministic strong initialization reduces the discrete search space and helps alleviate convergence instability. Optionally, the continuous low-rank decomposition here is not limited to standard truncated singular value decomposition; stochastic SVD, weighted SVD, incremental SVD, or other matrix decomposition methods that can provide a continuous low-rank starting point can achieve the same technical function.

[0049] Step 300: Update of continuous variables and three-valued projective variables based on ADMM splitting Directly solving low-rank decomposition problems with ternary constraints falls under highly nonconvex discrete optimization. If a first-order hard projection is used, incorrect zero-value or sign selections will be permanently fixed and cannot be corrected in subsequent steps. Therefore, this step employs the Alternating Direction Multiplier Method (ADMM) for variable splitting, while maintaining three sets of variables: continuous variables... , Three-valued projective variables , and dual variables , The continuous variables are not subject to discrete constraints and can freely move in the direction of reducing reconstruction error; the ternary projected variables always satisfy the discrete structural constraints required for deployment; the dual variables record the cumulative deviation between the two, guiding the continuous variables and the ternary projected variables to gradually align during iteration. Steps 310 to 350 below describe a complete iteration and the stopping decision.

[0050] Step 310, variable initialization. This involves initializing the consecutive low-rank starting points obtained in step 200. , Perform TSVID projection once for each variable (see step 330 for the projection process) to obtain the initial three-valued projection variables. , Set continuous variables , dual variables , Preset ADMM penalty coefficient Residual thresholds and maximum number of iterations and the number of stagnant and regressed rounds The above parameters are preset, calibrated, or determined by a calibration set.

[0051] Step 320, closed-form update of continuous variables. Fix the right-hand three-valued projection variable. Under the condition that the continuous variables on the left side are the following convex quadratic problems: Among them, superscript Indicates the first In the iteration, the first term of Equation 10 is the least squares reconstruction term based on the surrogate weight matrix, and the second term is the term that pulls the continuous variable to the neighbor of the "three-valued projected variable minus the dual variable". Control the relative strength between the two terms. This subproblem has a closed-form solution: in, For identity matrix, superscript This represents finding the inversion of a matrix. Since the object being inverted is... The small matrix results in a low computational cost for the closed-form solution. This update shifts the continuous variables on the left-hand side in a direction that reduces the reconstruction error of the surrogate weights, while constraining them from deviating excessively from the current three-valued projected variables through neighboring terms. Here... Depend on First, the TSVID is projected using step 330. This means that in this round, the continuous update and ternary projection on the left side are completed first, followed by the update and projection on the right side. Symmetrically, the ternary projection variable on the left side is fixed. Under the given conditions, the continuous variable on the right side is updated in closed form as follows: As an optional numerical stabilization process, when the number of conditions for finding the inverse term in Formula 11 or Formula 12 exceeds a preset upper limit for the number of conditions. In this case, an extra term can be added on the diagonal of the inverse term. Then solve it again, where To prevent numerical divergence, a small constant is used, thus avoiding ill-conditioned inverse operations.

[0052] Step 330, TSVID ternary projection. , Projecting back to the ternary structure domain respectively, we obtain , The inclusion of the dual variable here imbues the projected target with historical bias information, which is crucial for ADMM to guide the two sets of variables towards consistency. This is achieved using any continuous matrix to be projected. For example, ordinary ternary projection typically uses a single global scaling factor, i.e. The form of scaling is difficult to simultaneously accommodate the magnitude differences between different rows and columns in a matrix; moreover, the non-uniformity of row and column magnitudes within low-rank factors is quite significant, and global scaling will significantly amplify the structural mismatch when projecting continuous solutions onto discrete solutions. Therefore, the Ternary Symbolic Value Independent Projection (TSVID) proposed in this embodiment represents a continuous matrix as the product of the row scale, the ternary structure matrix, and the column scale: in, , These are the row scale vector and column scale vector obtained from the projection, respectively. It is a three-valued structure matrix. , Each is a matrix , The Middle Line 1 The elements of the column. This representation separates amplitude fitting from sign and zero value selection, allowing amplitude non-uniformity across different rows and columns to be expressed using both row and column scales. The specific execution process of TSVID is as follows.

[0053] First, to obtain a stable initialization, for amplitude matrix Perform a rank-one approximation and initialize the row and column scales accordingly: in, This indicates taking the absolute value of each element. Amplitude matrix The maximum singular value, , These are their corresponding left and right singular vectors, respectively. , These are the initial values ​​for the row and column scales, respectively. Since the elements of the magnitude matrix are non-negative, its rank-one approximation precisely captures the dominant distribution pattern of the matrix magnitude in the row and column directions. Therefore, initializing the row and column scales with its decomposition results can match the initial scales with the true magnitude structure of the matrix, avoiding oscillations that may occur when updating alternately from arbitrary initial values.

[0054] Secondly, in a fixed row size Sum and Scale At this point, the three values ​​at each position can be solved independently element by element. Specifically, let the local scale be... Compare the errors caused by setting this element to zero. Error caused by taking a non-zero value The following closed-form local threshold determination rule can be derived: (Formula 15) in, The sign function is defined in Formula 15. It shows that the threshold for determining whether an element is set to zero is not globally uniform, but is determined by the product of the local scales of the row and column in which the element is located: rows or columns with larger amplitudes naturally obtain higher zeroing thresholds, while rows or columns with smaller amplitudes obtain lower thresholds. This achieves local adaptation to the non-uniformity of row and column amplitudes, which is the core difference between TSVID and global threshold three-value projection.

[0055] Furthermore, with a fixed ternary structure matrix At that time, both row and column scales can be updated using one-dimensional least squares closed-form updates: in, , row scale The Element and column scale The One element, To prevent constants with zero denominators, Formulas 15 and 16 are executed alternately several times. This allows the three-valued structure and row and column scales to be adapted to each other and approximate the matrix to be projected.

[0056] Furthermore, due to the representation It has scale non-uniqueness (e.g., using Enlarged the whole by several times, and at the same time (Scaling down by the same factor, with the product remaining unchanged), this embodiment performs gauge normalization after each update round and absorbs the sign of the negative scale into the ternary structure matrix. In the middle, guarantee and It always represents a non-negative amplitude, and the sign information is entirely determined by... To assume responsibility. For example, scale normalization can be performed as follows: in, The scaling factor is the normalization coefficient. , They are vectors , The 2-norm, symbol " "" indicates that the left-hand variable is updated with the right-hand result. This process eliminates redundant degrees of freedom in the solution space, reduces numerical instability caused by arbitrary scale drift between the row and column sides, and also gives the derived scaling vector a clear physical meaning.

[0057] Optionally, this step also includes protection against near-zero scale anomalies: when the row denominator in Formula 16 Or list the denominators Not greater than If the number of non-zero ternary elements in the corresponding row or column is zero, it is determined to be a near-zero scale anomaly. If a valid scale from the previous round exists, it is used and the ternary scale determination is redone; otherwise, the scale of the row or column is set to zero and the anomaly mark is retained. It should be noted that this step does not perform rank redistribution for zero rows and zero columns to maintain the determinism of the valid rank during the iteration process.

[0058] Furthermore, as an optional implementation, the threshold coefficient of half in Formula 15 can also be replaced with a threshold coefficient subject to sparsity constraints. Specifically, the normalized amplitude is calculated first. That is, the first Line 1 The relative magnitude of the product of column elements with respect to the current row and column scale; given a target sparsity on the deployment side. At that time, from Candidate set composed of quantiles Select the threshold coefficient that makes the proportion of zero elements in the three-valued matrix closest to the target sparsity: in, , The matrices to be projected are respectively The number of rows and columns, This is an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. Replacing the fixed threshold coefficient with the generation of a three-valued structure matrix allows the zero-element ratio of the three-valued matrix to correspond to the memory access budget on the deployment side, making it easier to select the structure according to engineering constraints; when the deployment target sparsity is not clearly defined, the fixed local threshold mode shown in Formula 15 can be used.

[0059] Step 340, update the dual variable. (Result) , Then, update the dual variable by accumulating the changes using the following formula: Formula 19 serves to record the residual deviation between the continuous variable and the three-valued projected variable into the dual variable, so that the next round of continuous updates (the neighboring terms in Formulas 11 and 12) and the next round of projection (in step 330) are recorded in the dual variable. , Both variables carry this deviation information, thereby guiding the two sets of variables to gradually converge. Through the alternating execution of steps 320 to 340, the continuous variables continuously move towards the least squares reconstruction direction, and the three-valued projected variables always satisfy the discrete constraints, thus alleviating the problem of erroneous zero values ​​or erroneous signs being fixed in a single hard projection.

[0060] Step 350, Stop Judgment. As an important aspect of this embodiment, the stopping condition in this step is jointly gated by the global residual and the local residuals of the task-sensitive channels, rather than solely determined by the global residual. The technical motivation is that if iteration stops as soon as the residual reconstruction using a global proxy meets the criteria, the local projection mismatch on high-input-importance channels or high-output-sensitivity channels will be masked by the global average error, thereby reducing the initialization quality in terms of task loss; while simply increasing the number of iterations will increase overhead. Therefore, this embodiment reuses the input importance vector already statistically obtained in step 100. With output importance vector (Without adding any new statistics), the "task sensitivity" is extended from the construction phase of the proxy space to the termination control phase of the iteration.

[0061] Specifically, firstly, based on the output importance vector and input importance vector The output and input channels with the highest importance were identified as the set of highly sensitive output channels. and high-sensitivity input channel set For example, channels can be sorted by importance numerically or by quantile, selecting a certain percentage of the top-ranked channels, or a normalized importance threshold can be used; both are equivalent methods. Then, the global proxy reconstruction residual is calculated in each iteration. in, For the first The reconstruction results of the three-valued low-rank factors in the proxy space. Simultaneously, for each highly sensitive output channel... and each high-sensitivity input channel Calculate the local relative reconstruction residual: in, , Represent the first and second parts of the matrix respectively. row and number List, Let L2 represent the norm of a vector. The gating rule is: when the global agent reconstructs the residual... Its threshold has been met. However, it exists make Exceeding the local residual threshold , or exist make Exceeding the local residual threshold If the stopping condition is not met, the next iteration continues. The stopping condition is only met when the global proxy reconstruction residual and the local relative reconstruction residuals of all highly sensitive channels meet their respective thresholds. This avoids the problem of the global average error masking local mismatches in highly sensitive channels, and the iteration process focuses more on channels that have a greater impact on the model task. Optionally, the stopping decision can further introduce ADMM consistency residuals (the relative deviation between continuous variables and three-valued projected variables) and TSVID projection residuals (the relative deviation between the matrix to be projected and the projection result), requiring them to meet their respective thresholds to ensure that continuous variables and discrete variables are sufficiently consistent.

[0062] To ensure the iteration process terminates, this step also includes a convergence loop: if continuous If a smaller global proxy reconstruction residual is not obtained in the next iteration, the regression will be to the three-valued projected variable corresponding to the historical minimum residual. , Stop after that; the number of iterations does not exceed the maximum number of iterations. When the upper limit is reached, the historical best result is derived, and channels with local residuals that do not meet the standard are marked as priority targets for subsequent probabilistic annealing, mixed-rank assignment, or knowledge distillation fine-tuning. Thus, the gating rules, hard upper limit, and historical best backoff work together to avoid prematurely stopping and solidifying local mismatches in highly sensitive channels, while also preventing the iteration from failing to terminate.

[0063] Step 400: Scale Merging and Deployment Export Step 410, intermediate scale merging. After convergence via iteration in step 300, the left factor can be expressed as... The right factor can be expressed as ,in The column scale vector of the left factor. Let be the row scale vector of the right factor. Since the two are adjacent in the product, they can be combined into a middle-rank scale: in, This represents the element-wise multiplication of vectors (Hadamard product). This is the intermediate-rank scale vector. After merging, the target structure shown in Formula 8 is obtained. The output scaling vector With input-side scaling vector The row scale comes from the left factor and the column scale comes from the right factor, respectively.

[0064] Step 420, Inverse Scaling Absorption and Deployment Export. According to Equation 7, the result of the proxy space needs to be inversely scaled on both sides to restore the original weight space. Because... , All are diagonal matrices, and are respectively scaled to the output side in Equation 8. Input-side scaling Adjacent to each other, therefore the inverse scaling can be directly absorbed into the deployment scale, i.e., based on... and The element-wise division result is used as the deployment output scale, with and The element-wise division result is used as the deployment input-side scale. As an optional protective measure, when the importance of some inputs or outputs approaches zero, direct inversion would abnormally amplify the deployment scale. In this case, a lower bound protection is applied to the importance approaching zero: in, This represents element-wise division of a vector. , These are the deployment output-side scale and the deployment input-side scale, respectively. This is the lower limit of importance protection value. This means that the vector containing less than Replace the element with After this step, the final exported ternary low-rank initialization result is: It consists of only two ternary structure matrices and three consecutive scaling vectors. For ease of documentation, the output-side scale will be deployed in the derived results. Deployment of input-side scale They are respectively denoted as , That is, to obtain a structure consistent with the aforementioned target structure. .

[0065] It is important to emphasize the deployment configuration of the results derived in this embodiment: First, the computations during the inference phase can be performed sequentially according to the predetermined order of scaling vectors and three-valued matrices. Diagonal scaling is equivalent to channel-wise vector multiplication, and the multiplication of the three-valued matrix degenerates into addition, subtraction, and selection operations. No additional dense compensation matrices or complex bypass structures are introduced throughout the process. Second, the results directly replace the weight representations of the original linear layers without altering the connection relationships of the linear layers in the neural network model, i.e., without changing the topology of the original model's computational graph. Third, the inverse scaling of input / output sensitivity has been absorbed into the deployment scale, and there are no residual additional matrix multiplications during the deployment phase. These features make this embodiment highly feasible in engineering scenarios with limited GPU memory, bandwidth, and storage. Furthermore, the results derived in this embodiment can be deployed directly or used as input for subsequent probabilistic annealing, mixed-rank allocation, or knowledge distillation fine-tuning. In other words, the output of this embodiment is not a random discrete starting point, but a strongly initialized result that has already incorporated input-output sensitivity, continuous low-rank approximation, ADMM split solution and TSVID row and column scale projection. Subsequent optimization processes only need to perform local corrections in the vicinity of this result, without having to start the search from a random ternary structure.

[0066] Example 2 This embodiment provides a ternary low-rank compression initialization system for large language models, used for compressing and initializing the linear layers of a neural network model. This system corresponds to the method described in Embodiment 1, as shown in Figure 2. It can be deployed on servers, workstations, model compression platforms, model compilation platforms, or other computing devices with processors and memory. It includes a surrogate weight construction module, a continuous low-rank initialization module, a splitting solution module, and a scale merging and derivation module. The surrogate weight construction module is used to statistically analyze the input activation and output-side sensitivity corresponding to the original weight matrix of the linear layer on a calibration set, obtaining input importance vectors and output importance vectors, and then performing bilateral diagonal scaling on the original weight matrix to construct a surrogate weight matrix. The continuous low-rank initialization module is used to perform continuous low-rank decomposition on the surrogate weight matrix to obtain a continuous low-rank starting point under a given rank, including left and right factors. The splitting solution module is used to use the continuous low-rank starting point as the initial value and, based on the surrogate weight matrix, employ the alternating direction multiplier method to update the continuous variables and the ternary projected variables. The update process involves a phase split, alternately updating continuous variables, ternary projected variables, and dual variables until a stopping condition is met. The ternary symbolic-value independent decomposition projection represents the continuous matrix as the product of row scale, ternary structure matrix, and column scale, thus obtaining a left factor containing the left ternary structure matrix and a right factor containing the right ternary structure matrix. The scale merging and derivation module merges the intermediate scales from the left and right factors into an intermediate-rank scale, and absorbs the inverse scaling corresponding to the input and output importance vectors into the deployment scale, deriving a ternary low-rank initialization result composed of the ternary structure matrix and the continuous scaling vector. These modules can be implemented by software programs, hardware logic, or a combination of both. The specific processing procedures of each module are the same as those described in steps 100 to 400 of Example 1, and will not be repeated here to avoid repetition.

[0067] Example 3 This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various steps of the method described in Embodiment 1. The computer-readable storage medium may include a read-only memory, random access memory, a magnetic disk, an optical disk, flash memory, a solid-state drive, or other media capable of storing program instructions. Correspondingly, this application can also be implemented as a computing device, including a processor and a memory. The memory stores program instructions executable by the processor, which, when executed by the processor, implement the method described in Embodiment 1; this will not be elaborated upon here.

[0068] Example 4 This embodiment, in conjunction with Embodiment 1, further illustrates the feasible path for calibration data statistics, sensitive channel determination, iterative parameter values, and deployment implementation. The parameters in this embodiment can be calibrated based on the target model size, the type of linear layer to be compressed, the target rank, the storage budget of the deployment platform, and the stability of the calibration set. Unless otherwise specified, the parameters remain consistent within the same linear layer to be compressed.

[0069] In one implementation, the calibration set can consist of representative text samples, image samples, or multimodal samples from the target model deployment scenario. For example, dozens to hundreds of samples are randomly extracted from a general corpus similar to the target application domain, accumulating tens of thousands to hundreds of thousands of tokens. Without updating the model parameters, the samples are propagated forward in mini-batch, and the mean squared value of input activation and the mean squared value of output activation are accumulated for each linear layer to be compressed. When the output gradient energy is used as the output-side sensitivity, backpropagation is performed on the same calibration samples, and the mean squared value of the gradient of the corresponding output channel is accumulated. The accumulation can be performed according to the sample dimension, the token position dimension, or the mini-batch dimension.

[0070] In one implementation, when determining the set of highly sensitive channels, the importance of output and the importance of input can be ranked first, second, and third, respectively. The channel with the specified ratio is designated as a highly sensitive channel. The calibration can be performed within a range of 10% to 30% based on the model size and the stability of the calibration set; alternatively, channels with normalized importance not less than a preset threshold can be identified as high-sensitivity channels. Either of these two methods can be used, or they can be combined in the same linear layer.

[0071] In one implementation, the penalty coefficient for the alternating direction multiplier method... The surrogate weight matrix can be calibrated based on its spectral norm and the target rank. For example, values ​​that make the magnitudes of continuous terms and neighboring terms comparable in the initial iterations can be chosen, or values ​​that result in smaller surrogate reconstruction residuals and stable iterations can be selected from multiple candidate values ​​on the calibration set. A global residual threshold can also be used. With local residual threshold The location where the residual convergence curve enters the plateau region can be determined by observing it in a small-scale calibration experiment. The local residual threshold can be on the same order of magnitude as or slightly higher than the global residual threshold to avoid iteration failure due to noise in individual channels; the maximum number of iterations. It can take dozens to hundreds of rounds, with the number of rounds that stop and regress. You can take three to ten rounds, until you reach... or continuous When a smaller global proxy reconstruction residual is not obtained in the round, derive the three-valued projection variable with the smaller global proxy reconstruction residual in the history.

[0072] In one implementation, the zero constant is prevented. You can take a fixed small value that matches the calculation precision, such as taking a value in single-precision floating-point calculations. to The magnitude increases accordingly in half-precision or low-precision calculations; the lower limit of importance protection value. The value can be taken to be on the order of one-thousandth to one-hundredth of the largest element of the corresponding importance vector, and the inverse scaling absorption can be performed using... and Element-wise lower bound protection; condition number upper bound. The condition number for finding the inverse term can be preset based on the floating-point precision used and the stability of the matrix solution. When finding the inverse term, add to the diagonal of the inverse term. Then we will try to find the solution.

[0073] In one implementation, an ADMM-TSVID iteration is performed in the following order: first, the right-hand ternary projection variable is fixed. Update the left-hand continuous variable under the condition Secondly The left-side ternary projection variable is obtained by TSVID projection. Once again in a fixed position Update the right-hand continuous variable under the condition Then The right-hand ternary projection variable is obtained by TSVID projection. Finally, the dual variable is updated based on the difference between the continuous variable and the three-valued projected variable. and This gives the iterative process a defined execution path.

[0074] In one implementation, the derived ternary structure matrix and Two bits per element or other distinguishable methods can be used. , , Compact encoding method for storage, continuously scaling vector , , The data can be stored in half-precision floating-point, brain-level floating-point, single-precision floating-point, or fixed-point formats supported by the deployment platform. During inference execution, it can be executed sequentially in an equivalent combination order: input-side deployment scale, right ternary structure matrix, intermediate rank scale, left ternary structure matrix, and output-side deployment scale. Multiplication corresponding to the ternary structure matrix can be achieved through addition, subtraction, and skipping zero elements. Continuously scaling vectors correspond to channel-by-channel vector multiplication. This execution method does not introduce an additional dense compensation matrix or change the connection relationships of the original linear layers. The above numerical ranges are illustrative and can be calibrated by those skilled in the art based on model size, target rank, calibration error stability, and compression time budget.

[0075] Technical effect The above embodiments have the following technical effects: First, by weighting the original weights on both sides using input activation energy and output-side sensitivity to construct a proxy weight matrix, the compression target is made closer to the non-uniform response of the task loss to different channels. The limited rank and ternary discrete representation capacity are preferentially allocated to the task-sensitive region, thereby helping to improve the quality of extremely low bit compression initialization. Second, by truncating singular value decomposition to provide a deterministic continuous optimal starting point, and by using ADMM to decouple continuous least squares updates from ternary discrete projection, the risk of solidifying zero-value errors or incorrect symbols in a single hard projection is reduced. This allows subsequent probabilistic annealing or task distillation stages to only require local correction around the low-error structure, helping to reduce the difficulty of discrete search and training instability. Third, TSVID The projection simultaneously models row-scale, column-scale, and ternary structure matrices, replacing the global threshold with a local adaptive threshold. This allows for a more accurate fitting of the non-uniform distribution of row and column magnitudes within continuous low-rank factors, helping to reduce structural mismatch when projecting continuous low-rank solutions onto the ternary structure. Fourth, the local reconstruction residuals of task-sensitive channels are introduced into the iteration stopping criteria. Without adding any statistics, this avoids the local projection mismatch of highly sensitive channels being masked by the global average error, helping to reduce local reconstruction mismatch of highly sensitive channels under the same iteration budget. Fifth, near-zero scale protection, ill-conditioned inversion stabilization, historical optimal backoff, and deployment scale lower bound protection work together to improve the numerical stability of ADMM-TSVID iteration and the stability of deployment derivation. Sixth, the final derived result contains only a ternary structure matrix and a small number of continuously scaled vectors. No additional dense matrices are introduced during the inference stage, and the connection relationships of the original linear layers are not changed. This results in good deployment friendliness and is suitable for engineering scenarios with limited video memory, bandwidth, and storage.

[0076] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

[0077] It should be noted that in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if a reference is made to performing an action based on an element, it means performing the action at least based on that element, including two cases: performing the action only based on that element, and performing the action based on that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0078] All documents mentioned in this application are considered to be incorporated in their entirety into the disclosure of this application so that they can serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A ternary low-rank compression initialization method for large language models, characterized in that, include: Step 1: Calculate the original weight matrix of the linear layers in the large language model on the calibration set. The corresponding input activation and output sensitivity are used to obtain the input importance vector. and output importance vector , and according to For the original weight matrix Perform bilateral diagonal scaling to construct the surrogate weight matrix. ; Step two, the proxy weight matrix obtained in step one. Perform continuous low-rank decomposition to obtain a given rank. The continuous low-rank starting points, wherein the continuous low-rank starting points include left factors and right factors; Step 3: Using the continuous low-rank starting points obtained in Step 2 as initial values, and based on the proxy weight matrix obtained in Step 1... The alternating direction multiplier method is used to separate the updates of continuous variables from the updates of ternary projected variables, alternately updating the continuous variables, ternary projected variables, and dual variables until the stopping condition is met; wherein, under the condition of fixing the ternary projected variables on one side, based on the surrogate weight matrix Solve the closed least squares subproblem for the continuous variables on the other side, and then project the sum of the obtained continuous variables and their corresponding dual variables onto the three-valued signed-valued independent decomposition projection (TSVID) to obtain the three-valued projected variables on that side, thus obtaining the three-valued structure matrix containing the left side. Left factors and matrix containing right ternary structure The right factor; Step four: Combine the intermediate scales of the left and right factors obtained in step three into an intermediate-rank scale. and the input importance vector obtained in step one and output importance vector The corresponding inverse scaling is absorbed into the deployment scale, deriving a ternary low-rank initialization result composed of a ternary structure matrix and a continuous scaling vector. ; In the formula, This represents a diagonal matrix with the vectors within the brackets as its diagonal elements. , These are the continuous scaling vectors for the output and input sides, respectively.

2. The method according to claim 1, characterized in that, In step one: the input importance vector The element The output importance vector The element or ; The ternary low-rank initialization result is as follows: Restore to the original weight space; In the formula, This represents the averaging operation on the calibration set. For the linear layer Input activation of each input channel For the first Activation of output channels For the first The output gradient of each output channel; This is the compressed weight matrix in the original weight space. This represents the reconstruction result in the proxy space.

3. The method according to claim 1, characterized in that, The continuous low-rank decomposition in step two is as follows: for the surrogate weight matrix Perform truncated singular value decomposition, retaining the largest The left factor of the continuous low-rank starting point is obtained by considering the singular values ​​and their corresponding singular vectors. and right factor This allows the square roots of the singular values ​​to be absorbed into the left and right factors, respectively. In the formula, The proxy weight matrix Singular value decomposition, , They are respectively the front A matrix composed of left and right singular vectors. For the front A diagonal matrix composed of singular values. It is the square root of each element, with superscript. This indicates the matrix transpose.

4. The method according to claim 1, characterized in that, The continuous variables, ternary projective variables, and dual variables mentioned in step three are updated alternately as follows: Fixed right-side ternary projection variables Update the continuous variables on the left side using the closed-form solution: Will The left-side ternary projection variable is obtained through the TSVID projection. ; Fix the left-side ternary projection variable The continuous variables on the right-hand side are updated symmetrically using the closed-form solution: Will The right-side ternary projection variable is obtained through the TSVID projection. ; And by accumulating and updating the dual variables, the continuous variables are gradually made consistent with the three-valued projected variables: In the formula, superscript Indicates the first Round iteration, , These are continuous variables on the left and right sides, respectively. , These are the left and right three-valued projection variables, respectively. , These are the left and right dual variables, respectively. This is the penalty coefficient for the alternating direction multiplier method. For identity matrix, superscript This represents finding the inverse of a matrix.

5. The method according to any one of claims 1 to 4, characterized in that, The TSVID projection will project the continuous matrix to be projected. Represented as row scale ternary structure matrix Column Scale product This includes the continuous matrix. Perform the following operations: right amplitude matrix Perform rank-one approximation And initialize the line scale accordingly. , list of standards ; In fixed row size Sum and Scale At that time, the three-valued structure matrix is ​​determined element-wise using half of the product of the row and column scales as a local threshold. :when Time to take Otherwise take ; In a fixed ternary structure matrix At that time, update the row and column scales using closed-form solutions: Scale normalization is performed after each update to eliminate The non-uniqueness of the scale absorbs the sign of the negative scale into the ternary structure matrix. In the middle, make the scale of the line Sum and Scale All are non-negative; In the formula, , , These are the magnitude matrices. The maximum singular value and its corresponding left and right singular vectors, This indicates taking the absolute value of each element. , Each is a matrix , The Middle Line 1 Column elements, , row scale The Element and column scale The One element, For the sign function, To prevent constants with a denominator of zero.

6. The method according to claim 1, characterized in that, The stopping condition described in step three is jointly gated by the global residual and the local residual of the task-sensitive channel, including: Based on the output importance vector obtained in step one and input importance vector The output and input channels with the highest importance were identified as the set of highly sensitive output channels. and high-sensitivity input channel set ; In each iteration, the global proxy reconstruction residual is calculated, as well as the residual for each highly sensitive output channel. and each high-sensitivity input channel Calculate the local relative reconstruction residual: When the global agent reconstructs the residual Its threshold has been met, but there are make Exceeding the local residual threshold or exist make Exceeding the local residual threshold If the stop condition is not met, the next iteration continues; the stop condition is met only if the global proxy reconstruction residual and the local relative reconstruction residual of all the high-sensitivity channels meet their respective thresholds. In the formula, , The first The three-valued projection variables of the left and right sides of the wheel, , Represent the first and second parts of the matrix respectively. row and number List, The L2 norm of a vector. Denotes the Frobenius norm of a matrix. To prevent constants with a denominator of zero.

7. The method according to claim 5, characterized in that, It also includes at least one of the following stabilization treatments: In the closed-form update of the row-scale or column-scale, when the corresponding row denominator Or list the denominators Not greater than When the number of non-zero ternary elements in the corresponding row or column is zero, it is judged as a near-zero scale anomaly. The effective scale of the previous round is used and the ternary judgment is redone, or the scale of the row or column is set to zero. In the closed-form least squares solution of the continuous variables, when the condition number for finding the inverse term exceeds the upper limit of the condition number... When, add to the diagonal of the inverse term Then we will solve it again; When continuous If a smaller global proxy reconstruction residual is not obtained in a round of iteration, the iteration stops after regressing to the three-valued projected variable corresponding to the historical minimum residual, and the number of iterations does not exceed the maximum number of iterations. When the limit is reached, export the historical best and mark the channels that did not meet the standard as targets for subsequent correction; In step four, when inverse scaling is absorbed to the deployment scale, a lower bound protection is applied to the importance approaching zero, i.e., with , Replace the output importance vector respectively Input importance vector Participate in the absorption calculation of the inverse scaling; In the formula, The upper limit of the condition number, The number of rounds of stagnation and reversal. The maximum number of iterations, This is the lower limit of importance protection value. To prevent constants with a denominator of zero, It is the identity matrix. This means that the vector containing less than Replace the element with .

8. The method according to claim 1, characterized in that, The linear layer includes at least one of the query projection layer, key projection layer, value projection layer, and output projection layer in the Transformer structure, as well as the upper projection layer, gated projection layer, and lower projection layer in the feedforward network; the ternary low-rank initialization result is composed of a ternary structure matrix. , With continuously scaled vectors , , It is composed of elements that do not introduce additional dense matrices during the inference phase and do not change the connection relationships of the linear layers in the large language model.

9. A ternary low-rank compression initialization system for large language models, characterized in that, include: The proxy weight construction module is used to statistically analyze the original weight matrix of the linear layers in the large language model on the calibration set. The corresponding input activation and output sensitivity are used to obtain the input importance vector. and output importance vector , and according to For the original weight matrix Perform bilateral diagonal scaling to construct the surrogate weight matrix. ; A continuous low-rank initialization module is used to initialize the proxy weight matrix obtained by the proxy weight construction module. Perform continuous low-rank decomposition to obtain a given rank. The following are consecutive low-rank starting points, including left and right factors; The splitting solution module is used to solve the problem based on the continuous low-rank starting points and the proxy weight matrix. The alternating direction multiplier method is used to separate the updates of continuous variables from the updates of ternary projected variables, alternately updating the continuous variables, ternary projected variables, and dual variables until the stopping condition is met; wherein, under the condition of fixing the ternary projected variables on one side, based on the surrogate weight matrix Solve the closed least squares subproblem for the continuous variables on the other side, and then project the sum of the obtained continuous variables and their corresponding dual variables onto the three-valued symbolically independent decomposition projection (TSVID) to obtain the three-valued projected variables for that side, thus obtaining the three-valued structure matrix containing the left side. Left factors and matrix containing right ternary structure The right factor; The scale merging and derivation module is used to merge the intermediate scales of the left and right factors obtained by the split solution module into an intermediate-rank scale. and the input importance vector and output importance vector The corresponding inverse scaling is absorbed into the deployment scale, deriving a ternary low-rank initialization result composed of a ternary structure matrix and a continuous scaling vector. ; The meanings of each symbol are the same as those in claim 1.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images