Lightening method and device for shared weight of similar channels of large model

Abstract

The invention discloses a large model similar channel sharing weight lightening method and device, and belongs to the field of artificial intelligence. The method comprises the steps of performing K-Means clustering on a Q / K projector weight matrix of a self-attention layer of a Transform architecture large model according to columns, and generating a cluster representative vector and a cluster label list to replace an original weight matrix to complete compression; restoring the approximate weight matrix based on the representative vector and the cluster label, and constructing an error matrix of the original matrix and the approximate matrix; performing singular value decomposition on the error matrix and retaining a key singular value to obtain two low-dimensional matrixes; during reasoning, multiplying the two low-dimensional matrixes to obtain an approximate error matrix, and adding the approximate error matrix with the approximate weight matrix to obtain a final reasoning weight; the total storage size after compression is calculated through a unified formula, and accurate quantification of storage overhead is achieved. According to the method, the performance degradation under low-bit compression is effectively relieved, the resource demand of model deployment is reduced, and the method is adaptive to various mainstream large models and different-dimension weight matrixes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for lightweighting shared weights for similar channels in large models. Background Technology

[0002] Large Language Models (LLMs), with their rich knowledge reserves and powerful language understanding and generation capabilities, have been widely applied in various industries such as natural language processing and intelligent interaction. Among them, LLMs based on the Transformer architecture (such as the Llama series and Qwen series) have become the mainstream technology direction. As the number of model parameters increases from billions to hundreds of billions and trillions, their storage consumption and computing overhead grow exponentially. This not only places extremely high demands on hardware resources but also limits their deployment on resource-constrained devices (such as terminals and edge devices) and in the environment of domestic operating systems. Therefore, there is an urgent need to reduce resource consumption through model compression technology.

[0003] The current mainstream LLM compression techniques are mainly divided into four core directions. The first is quantization technology, which achieves compression by converting floating-point weights into integer or discrete forms. It is further divided into quantization during training (QAT) and quantization after training (PTQ). The core is to reduce storage overhead through precision conversion, but it relies on quantization-aware training to compensate for precision loss. The second is pruning technology, which simplifies the model by removing redundant weight parameters or structural units (such as channels and layers). It includes unstructured pruning (removing individual parameters) and structured pruning (removing the entire structure), but it requires multi-stage optimization and fine-tuning to restore performance. The third is knowledge distillation technology, which transfers knowledge from a complex "teacher model" to a simple "student model". It is divided into black-box distillation (using only the prediction results) and white-box distillation (using model parameters), but it requires an additional training stage to complete the knowledge transfer. The fourth is low-rank decomposition technology, which decomposes a high-dimensional weight matrix into a low-dimensional matrix product to reduce the parameter dimensionality, but it is sensitive to outliers and easily leads to the loss of key features. In addition, some existing technologies attempt to achieve compression through codebook vector quantization (such as AQLM and GPTVQ), but they rely on beam search, gradient updates or second-order optimization, which has high computational costs and has not formed a unified storage-scale computing system.

[0004] However, existing technologies still have some intractable flaws. First, it relies on complex training processes. Techniques such as quantization, pruning, and distillation all require additional fine-tuning or training steps, consuming a large amount of computing resources and resulting in low deployment efficiency. Second, low-bit compression performance suffers severe degradation. Under extreme compression conditions such as 1 bit, the perplexity (PPL) of existing non-fine-tuning methods (such as RTN and SmoothQuant) can reach thousands or even infinity, making it impossible to measure the accuracy of downstream tasks, and no effective error compensation mechanism has been established. Third, it lacks a precise storage scale calculation scheme and has not formed a unified storage quantization formula covering codebook vectors, cluster labels, and compensation matrices, making it difficult to accurately control the balance between compression ratio and performance. Fourth, it has poor adaptability to non-square matrix weight matrices. The M×N (M≠N) matrices commonly used in Transformer self-attention layers require additional adaptation schemes, and existing technologies do not provide efficient segmentation and splicing logic. Fifth, it suffers from insufficient outlier handling. Key outlier information is easily lost during clustering or quantization, and there is a lack of visualization verification methods to support parameter optimization. Sixth, it suffers from insufficient storage format optimization. It does not fully utilize the advantages of efficient storage formats such as int8, making it difficult to maximize model performance under the same storage overhead.

[0005] These shortcomings make it difficult for existing technologies to meet the deployment requirements of low bit depth, lightweight design, no need for fine-tuning, and compatibility with domestic operating systems, thus limiting the widespread application of LLM. Summary of the Invention

[0006] This invention aims to address the shortcomings of traditional communication management schemes in adapting to the multi-runtime characteristics of specialized operating systems. The technical solution is as follows: On the one hand, a lightweight method for sharing weights across similar channels in large models is provided, targeting domestic operating systems. The method includes: Step 1: Perform channel clustering processing, including performing K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model column by column, and updating the cluster centers iteratively to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. Step 2: Perform representative vector generation and storage compression, including calculating the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generating a cluster number list that records the cluster number to which each channel belongs. The original weight matrix is ​​replaced with the cluster number list and the representative vector to complete parameter compression. Step 3, construct the error matrix, including restoring the approximate weight matrix W' based on the cluster label list and representative vector, and calculating the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'; Step 4: Perform SVD error compensation, which involves performing Singular Value Decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. The first R largest singular values ​​and their corresponding vectors are retained to obtain an M×R dimension matrix. and an R×M dimension matrix ; Step 5, perform reasoning weight repetition, including in the model reasoning stage, the matrix and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; Step 6: Calculate the storage size, including the total storage size M of the compressed matrix, which satisfies M = M1 + M2, where M1 is the storage size of the codebook vector and cluster label vector, M1 = Nc × M × K + N × log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2 = 2 × Rsvd × K × max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

[0007] Optionally, the convergence condition for K-Means clustering in step 1 is that the change in the mean square error of the cluster centers in two consecutive iterations is ≤1e -4 The number of clusters Nc is adjustable. When the target model is Llama-2-7B and Nc=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%. In step 2, the number of clusters CN can be adjusted. When the target model is Llama-2-7B and CN=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%, and the result of log2Nc in the calculation of M1 is rounded up to ensure the storage integrity of the cluster number list.

[0008] Optionally, in step 2, the cluster label list has a dimension of 1×M, the representative vector has a dimension of M×1, and the original M×M weight matrix has a storage size of (CN+1)×M after compression. The rule for retaining the top R largest singular values ​​in step 4 is as follows: after sorting the singular values ​​from largest to smallest, retain the singular values ​​with a cumulative contribution rate ≥ 95% and their corresponding vectors. Let U be a submatrix formed by the column vectors corresponding to the first R singular values ​​in U. It is a submatrix formed by the column vectors corresponding to the first R singular values ​​in V; In step 6, the value of K corresponds to the storage format as follows: when the storage format is int8, K = 1 byte; when the storage format is float16, K = 2 bytes, and the calculation results of M1 and M2 are in bytes.

[0009] Optionally, the method further includes a non-matrix adaptation step, comprising: When the weight matrix is ​​M×N (M≠N), it is divided into multiple min(M,N)×min(M,N) square matrices according to the ratio of max(M,N) / min(M,N). Steps 1-6 are performed on each square matrix and then concatenated in the original order. At this time, the storage size of each sub-square matrix is ​​calculated as M1'=Nc×min(M,N)×K+min(M,N)×log2Nc and M2'=2×Rsvd×K×min(M,N). The total storage is the sum of M1' and M2' of all sub-square matrices.

[0010] Optionally, the non-square matrix can be divided into integers. When the weight matrix is ​​4096×2048, it is divided into two 2048×2048 square matrices. After splicing, the weight distribution and dimensionality of the original matrix are preserved.

[0011] Optionally, the large models that the final inference weights WNew are adapted to in step 5 include Llama-2-7B, Llama-3.1-8B, Qwen-2.5-7B, and Deepseek-R1-Distill-Qwen-7B.

[0012] Optionally, the ratio of the number of bits in the cluster number CN_Bit=log2Nc to the number of rank-preserving bits R_Bit=log2Rsvd can be adapted to different models, specifically including: The Llama series models have a ratio of 0.33-0.6, and the Qwen series models have a ratio of 0.5-1, in order to achieve the optimal perplexity level (PPL).

[0013] Optionally, the method further includes a visualization verification and parameter optimization process, including: A heatmap of the number of vectors within a cluster is generated to verify the uniformity of clustering, and a heatmap of the absolute value of the error matrix is ​​generated to verify the sparse distribution characteristics of the error, providing a basis for the parameter optimization of Nc and Rsvd.

[0014] On the other hand, a lightweight device for sharing weights across similar channels in a large model is also provided, targeting a domestic operating system. The device includes: The channel clustering processing module is used to perform K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model. It iteratively updates the cluster centers to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. The representative vector generation and storage compression module is used to calculate the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generate a cluster number list recording the cluster number to which each channel belongs. The original weight matrix is ​​replaced by the cluster number list and the representative vector to complete the parameter compression. The error matrix construction module is used to reconstruct the approximate weight matrix W' based on the cluster label list and representative vectors, and to calculate the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'. The SVD error compensation module performs singular value decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. It retains the first R largest singular values ​​and their corresponding vectors, resulting in an M×R dimension matrix. and an R×M dimension matrix ; The original module repeats the reasoning weights, which is used during the model reasoning phase to transfer the matrix. and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; The storage size calculation module is used to calculate the total storage size M of the compressed matrix, which satisfies M=M1+M2, where M1 is the storage size of the codebook vector and the cluster label vector, M1=Nc×M×K+N×log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2=2×Rsvd×K×max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

[0015] Compared with the prior art, the present invention has the following significant advantages.

[0016] This invention discloses a lightweight method and apparatus for sharing weights across similar channels in large models, belonging to the field of artificial intelligence. The method includes: performing K-Means clustering on the column-wise weight matrix of the Q / K projector in the self-attention layer of a Transformer architecture large model to generate cluster representative vectors and a list of cluster labels to replace the original weight matrix for compression; reconstructing the approximate weight matrix based on the representative vectors and cluster labels, and constructing an error matrix between the original and approximate matrices; performing singular value decomposition on the error matrix and retaining key singular values ​​to obtain two low-dimensional matrices; multiplying the two low-dimensional matrices during inference to obtain an approximate error matrix, which is then added to the approximate weight matrix to obtain the final inference weights; calculating the total storage size after compression using a unified formula to achieve precise quantification of storage overhead. This invention effectively alleviates performance degradation under low-bit compression, reduces resource requirements for model deployment, and is adaptable to various mainstream large models and weight matrices of different dimensions. Attached Figure Description

[0017] Figure 1 Here is a flowchart of the weight matrix clustering and restoration process; Figure 2 A diagram showing the steps of SVD error compensation; Figure 3 A heatmap of the number of vectors within a cluster; Figure 4 This is a heatmap of the absolute values ​​of the error matrix. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0020] The core technical solution of this invention is a lightweight method for sharing weights of similar channels in large models without fine-tuning. It is applicable to large language models with Transformer architecture and is compatible with mainstream models such as Llama-2-7B, Llama-3.1-8B, Qwen-2.5-7B, and Deepseek-R1-Distill-Qwen-7B.

[0021] Example 1.

[0022] This embodiment takes the Q-projector weight matrix (4096×4096 dimension) of the self-attention layer of the Llama-2-7B model as the processing object, and provides a lightweight method for sharing weights of similar channels in large models for domestic operating systems. The method includes: Step 1: Perform channel clustering processing, including performing K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model column by column, and updating the cluster centers iteratively to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. Specifically, K-Means clustering is performed on the Q-projector weight matrix (4096×4096) of the self-attention layer of the Transformer architecture large language model, column-wise (channel-wise). The cluster centers are iteratively updated to maximize vector similarity within the same cluster and minimize inter-cluster similarity. The iteration terminates when the change in the mean squared error of the cluster centers between two consecutive iterations is ≤1e. -4 The clustering process continues until convergence. In this embodiment, the number of clusters Nc is set to 256, and the clustering process is as follows: Figure 1 As shown on the left, the original weight matrix W is divided into 256 high similarity clusters, laying the foundation for subsequent weight sharing.

[0023] In step 1, the convergence condition for K-Means clustering is that the change in the mean square error of the cluster centers in two consecutive iterations is ≤1e. -4 The number of clusters Nc is adjustable. When the target model is Llama-2-7B and Nc=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%.

[0024] Step 2: Perform representative vector generation and storage compression, including calculating the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generating a cluster number list that records the cluster number to which each channel belongs. The original weight matrix is ​​replaced with the cluster number list and the representative vector to complete parameter compression. Specifically, the mean of all 4096-dimensional vectors within each cluster is calculated, resulting in 256 cluster representative vectors of M×1 dimension (4096×1). Simultaneously, a 1×M dimension (1×4096) cluster label list is generated, recording the cluster number to which each channel belongs. The original 4096×4096 weight matrix is ​​replaced with the cluster label list and the 256 representative vectors to complete parameter compression. After compression, the original weight matrix has a storage size of (CN+1)×M = (256+1)×4096 = 1,052,672 bytes. That is, the cluster label list has a dimension of 1×M, the representative vectors have a dimension of M×1, and the original M×M weight matrix has a storage size of (CN+1)×M after compression. The compression logic is... Figure 1 The process of generating and restoring the representative vectors from the middle to the right is the same.

[0025] In step 2, the number of clusters CN can be adjusted. When the target model is Llama-2-7B and CN=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%, and the result of log2Nc in the calculation of M1 is rounded up to ensure the storage integrity of the cluster number list.

[0026] Step 3, construct the error matrix, including restoring the approximate weight matrix W' based on the cluster label list and representative vector, and calculating the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'; Specifically, based on the cluster label list and representative vector generated in step 2, an approximate weight matrix W' (dimension 4096×4096) is obtained through reverse reconstruction. The error matrix WErr = W - W' between the original weight matrix W and the approximate weight matrix W' is calculated by matrix subtraction. The error matrix has the same dimension as the original matrix, 4096×4096. This process corresponds to... Figure 2 The error matrix generation logic in step 1.

[0027] Step 4: Perform SVD error compensation, which involves performing Singular Value Decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. The first R largest singular values ​​and their corresponding vectors are retained to obtain an M×R dimension matrix. and an R×M dimension matrix ; In step 4, the rule for retaining the top R largest singular values ​​is as follows: after sorting the singular values ​​from largest to smallest, retain the singular values ​​with a cumulative contribution rate ≥ 95% and their corresponding vectors. Let U be a submatrix formed by the column vectors corresponding to the first R singular values ​​in U. It is a submatrix formed by the column vectors corresponding to the first R singular values ​​in V.

[0028] In one example, singular value decomposition (SVD) is performed on the error matrix WErr, yielding a left singular vector matrix U (4096×4096), a singular value diagonal matrix Σ (4096×4096), and a right singular vector matrix V (4096×4096). After sorting the singular values ​​from largest to smallest, the top R (R=64 in this example) largest singular values ​​with a cumulative contribution rate ≥95% and their corresponding vectors are retained. The first 64 column vectors of U are extracted to form an M×R dimension (4096×64) submatrix Ur, and the first 64 column vectors of V are extracted to form the corresponding submatrix Vr. Finally, a 4096×64 dimension matrix UrΣ12 and a 64×4096 dimension matrix Σ12Vr are obtained. This decomposition and dimensionality reduction process is as follows: Figure 2 As shown in steps 2-3.

[0029] Step 5, perform reasoning weight repetition, including in the model reasoning stage, the matrix and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; The large models for which the final inference weights WNew are adapted include Llama-2-7B, Llama-3.1-8B, Qwen-2.5-7B, and Deepseek-R1-Distill-Qwen-7B.

[0030] Specifically, in the model inference stage, the matrices UrΣ12 and Σ12Vr obtained in step 4 are multiplied to obtain the dimension reduction approximation error matrix W'Err (4096×4096); then the approximation weight matrix W' is added to the dimension reduction approximation error matrix W'Err to obtain the final inference weight WNew=W'+W'Err. This weight is obtained by repeating the corresponding steps in the original process. Figure 2 Step 4: WNew is directly used for downstream tasks such as text generation and common sense reasoning.

[0031] Step 6: Calculate the storage size, including the total storage size M of the compressed matrix, which satisfies M = M1 + M2, where M1 is the storage size of the codebook vector and cluster label vector, M1 = Nc × M × K + N × log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2 = 2 × Rsvd × K × max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

[0032] The value of K corresponds to the storage format as follows: when the storage format is int8, K = 1 byte; when the storage format is float16, K = 2 bytes, and the calculation results of M1 and M2 are in bytes.

[0033] Specifically, the total storage size of the compressed matrix is ​​calculated as M = M1 + M2.

[0034] In one example, this embodiment uses the int8 storage format, so K = 1 byte, N is the column dimension of the weight matrix (4096), Rsvd = 64, and max(M,N) = 4096. Wherein, M1 = Nc × M × K + N × log2Nc = 256 × 4096 × 1 + 4096 × log2256 = 1,048,576 + 4096 × 8 = 1,081,344 bytes; M2 = 2 × Rsvd × K × max(M,N) = 2 × 64 × 1 × 4096 = 524,288 bytes; the total storage size M = 1,081,344 + 524,288 = 1,605,632 bytes (approximately 1.53 MB).

[0035] To elaborate further, the ratio of the number of cluster bits CN_Bit=log2Nc to the number of rank-preserving bits R_Bit=log2Rsvd is adapted to different models. Specifically, the ratio is 0.33-0.6 for the Llama series models and 0.5-1 for the Qwen series models to achieve optimal perplexity (PPL).

[0036] The technical advantages of this embodiment are as follows: K-Means clustering enables weight sharing of similar channels, and redundant parameters are converted into cluster labels and representative vectors for storage, significantly reducing model storage overhead; at the same time, SVD decomposition is used to preserve and compensate for key features of the error matrix, making up for the information loss caused by cluster approximation, alleviating performance degradation under low bit compression without additional fine-tuning process, ensuring the stable performance of the model's core language processing and inference functions, and reducing the consumption of computing resources during the deployment phase.

[0037] Example 2.

[0038] This embodiment takes the K-projector weight matrix (dimension 4096×2048) of the self-attention layer of the Qwen-2.5-7B model as the processing object. Based on embodiment 1, a non-square matrix adaptation step is added, and the specific implementation is as follows.

[0039] When the weight matrix is ​​M×N (M≠N), it is divided into multiple min(M,N)×min(M,N) square matrices according to the ratio of max(M,N) / min(M,N). Steps 1-6 are performed on each square matrix and then concatenated in the original order. At this time, the storage size of each sub-square matrix is ​​calculated as M1'=Nc×min(M,N)×K+min(M,N)×log2Nc and M2'=2×Rsvd×K×min(M,N). The total storage is the sum of M1' and M2' of all sub-square matrices.

[0040] Optionally, the non-square matrix can be divided into integers. When the weight matrix is ​​4096×2048, it is divided into two 2048×2048 square matrices. After splicing, the weight distribution and dimensionality of the original matrix are preserved.

[0041] In one example, for each 2048×2048 sub-matrix, steps 1-6 of Example 1 are executed one by one: Step 1 sets the number of clusters Nc=128, and the K-Means clustering convergence condition is that the change in the mean square error of the cluster centers in two consecutive iterations is ≤1e -4Step 2 generates 128 representative vectors of 2048×1 and a 1×2048 cluster label list; Step 3 constructs a 2048×2048 error matrix WErr; Step 4 performs SVD decomposition, retaining the top R=32 singular values ​​with a cumulative contribution rate ≥95%, obtaining a 2048×32 UrΣ12 and a 32×2048 Σ12Vr; Step 5 multiplies them to obtain W'Err and adds it to W' to obtain WNew; Step 6 calculates the storage size of a single submatrix. M1' = Nc × min(M,N) × K + min(M,N) × log2Nc = 128 × 2048 × 1 + 2048 × log2128 = 262,144 + 2048 × 7 = 276,832 bytes, M2' = 2 × Rsvd × K × min(M,N) = 2 × 32 × 1 × 2048 = 131,072 bytes, Total storage of a single sub-matrix = 276,832 + 131,072 = 407,904 bytes.

[0042] Furthermore, the two 2048×2048 sub-matrices processed in steps 1-6 are spliced ​​together in their original order to obtain a complete compressed matrix of 4096×2048, with a total storage of 2×407,904=815,808 bytes (approximately 0.78MB).

[0043] The technical advantages of this embodiment are as follows: For the non-square matrix weight matrix commonly found in the Transformer self-attention layer, the designed split-independent processing-stitching scheme effectively expands the applicable scenarios of the method and avoids compression failure caused by matrix dimension mismatch; the unified storage scale calculation logic clarifies the relationship between the number of clusters, the reserved rank and the storage format, making the balance between storage overhead and performance during compression more controllable, and achieving reasonable compression configuration without relying on experience trial and error, further reducing the difficulty of technology implementation.

[0044] Example 3 This embodiment provides a parameter adaptation and visualization optimization process for the Llama-3.1-8B and Deepseek-R1-Distill-Qwen-7B models, and the specific implementation is as follows.

[0045] Implementation of the Llama-3.1-8B model.

[0046] Perform steps 1-6 of Example 1, setting Nc=96, Rsvd=64, cluster number bits CN_Bit=log296≈6.58, rank-preserving bits R_Bit=log264=6, with a ratio of ≈6.58:6≈1.09, which meets the adaptation requirements of Llama series models of 0.33-0.6; further execute the visualization verification process to generate a heatmap of the number of vectors within a cluster (e.g. Figure 3As shown in the figure, the vectors of each cluster are evenly distributed, verifying the rationality of the clustering; a heatmap of the absolute value of the error matrix is ​​generated (as shown in the figure). Figure 4 As shown in the figure, the sparse distribution of errors is verified, providing a basis for parameter optimization. After final compression, the perplexity of the WikiText-2 dataset is approximately 10.455, and the average accuracy of downstream tasks is approximately 68.18%.

[0047] Implementation of the Deepseek-R1-Distill-Qwen-7B model.

[0048] Following steps 1-6 of Example 1, Nc=192, Rsvd=80, CN_Bit=log2192≈7.58, R_Bit=log280≈6.32, with a ratio of ≈7.58:6.32≈1.20, meeting the 0.5-1 adaptation requirement of the Qwen series models. Further visualization verification was performed, and parameter configuration was optimized. After final compression, the perplexity of the WikiText-2 dataset is approximately 29.358, and the average accuracy of downstream tasks is approximately 60.31%.

[0049] The technical advantages of this embodiment are as follows: by adapting the ratio of cluster number bits to rank-preserving bits, the method can adjust parameter configurations according to the weight characteristics of different series of models, avoiding the adaptation limitations caused by a single parameter setting and improving compatibility with mainstream large models; the visualized heatmap provides an intuitive reference for clustering results and error distribution, allowing the rationality of parameter configurations to be judged without complex analysis, simplifying the parameter optimization process, helping to quickly find the optimal solution that balances compression efficiency and performance, and improving the flexibility of technology application.

[0050] Example 4 Optionally, the above embodiments also include a visualization verification and parameter optimization process, including: A heatmap of the number of vectors within a cluster is generated to verify the uniformity of clustering, and a heatmap of the absolute value of the error matrix is ​​generated to verify the sparse distribution characteristics of the error, providing a basis for the parameter optimization of Nc and Rsvd.

[0051] As an optional extension of the method, two types of heatmaps are generated to intuitively verify the key aspects of compression: the first is a heatmap of the number of vectors within a cluster, which shows the distribution of the number of vectors contained in each cluster to determine whether K-Means clustering has achieved uniform partitioning of vectors and avoids unreasonable clustering results such as excessive concentration of vectors in some clusters or too many isolated clusters; the second is a heatmap of the absolute value of the error matrix, which shows the distribution of the absolute value of the error in each dimension of the error matrix to clarify the overall sparsity of the error and the concentrated areas of outliers.

[0052] Furthermore, the information reflected by the two types of heatmaps directly corresponds to the rationality of the settings of the clustering parameter Nc (number of clusters) and the singular value retention rank Rsvd. When the cluster uniformity is insufficient, the value of Nc can be adjusted, and when the error concentration area exceeds expectations, the retention number of Rsvd can be optimized. This provides a visual basis for parameter adjustment and replaces the traditional parameter optimization method that relies on experience or a lot of trial and error.

[0053] The technical benefits of this process are that it transforms clustering quality and error distribution from abstract data into intuitive images through visualization, avoiding the blind optimization of parameters and reducing the number of trials and errors and time costs. At the same time, based on heatmaps, potential problems in the clustering or error compensation stages can be accurately located, ensuring that the settings of Nc and Rsvd match the characteristics of the model weights. This avoids insufficient compression ratio or information redundancy caused by improper Nc, and also prevents insufficient error compensation caused by unreasonable Rsvd configuration. Thus, while ensuring compression efficiency, it further reduces the risk of performance degradation and improves the adaptability and reliability of the method under different models and compression requirements.

[0054] In summary, the accompanying drawings involved in the embodiments will be described in a unified manner.

[0055] Figure 1 This is a flowchart of the weight matrix clustering and restoration process, used to present the core information flow of "clustering - representative vector generation - restoration".

[0056] The left side shows the original weight matrix W, which is divided into multiple clusters by K-Means clustering in step 1 (an intermediate step) to ensure high similarity of vectors within the same cluster. The middle section shows the representative vector generation process in step 2, where representative vectors are obtained by calculating the mean value within each cluster. The right side shows the restoration process of the approximate weight matrix W', which is formed by matching representative vectors from the cluster label list. For non-square matrices (such as the 4096×2048 matrix in Example 2), this diagram can be expanded into a process of "segmentation-clustering-generation-restoration-segmentation," which intuitively verifies the feasibility of sharing channel clustering weights.

[0057] Figure 2 This is a diagram of the SVD error compensation steps, used to illustrate the error handling and weighted repetition logic in steps 3-5.

[0058] Step 1 corresponds to the generation of the error matrix WErr (the difference between W and W'); Step 2 is the SVD decomposition of WErr to obtain three matrices U, Σ, and V; Step 3 is the dimensionality reduction process after retaining key singular values ​​to obtain UrΣ12 and Σ12Vr; Step 4 is the generation of W'Err (multiplication of the two matrices) and the restoration of the final inference weight WNew (addition of W' and W'Err), which fully presents the closed-loop process of "error generation-decomposition-dimensionality reduction-compensation-restoration", explaining the core reason for the stable performance under low bit compression.

[0059] Figure 3 and 4 This is a heatmap for visual verification, used to support the visual verification process. Figure 3 The heatmap shows the number of vectors within a cluster. The horizontal axis represents the cluster number, and the vertical axis represents the number of vectors. The color intensity reflects the number of vectors. For example, in Example 1, the vectors of the 256 clusters are evenly distributed with no isolated clusters, which verifies the rationality of the clustering in Step 1. Figure 4 The image shows a heatmap of the absolute value of the error matrix. The horizontal and vertical axes represent the matrix dimensions, and the color intensity reflects the magnitude of the error. It can be seen that the overall error is sparse (lighter color), with only a few striped outliers (darker color). This verifies the necessity of retaining key singular values ​​for error compensation in step 4 and provides an intuitive basis for the value of R.

[0060] On the other hand, a lightweight device for sharing weights across similar channels in a large model is also provided, the device comprising: The channel clustering processing module is used to perform K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model. It iteratively updates the cluster centers to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. The representative vector generation and storage compression module is used to calculate the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generate a cluster number list recording the cluster number to which each channel belongs. The original weight matrix is ​​replaced by the cluster number list and the representative vector to complete the parameter compression. The error matrix construction module is used to reconstruct the approximate weight matrix W' based on the cluster label list and representative vectors, and to calculate the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'. The SVD error compensation module performs singular value decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. It retains the first R largest singular values ​​and their corresponding vectors, resulting in an M×R dimension matrix. and an R×M dimension matrix ; The original module repeats the reasoning weights, which is used during the model reasoning phase to transfer the matrix. and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; The storage size calculation module is used to calculate the total storage size M of the compressed matrix, which satisfies M=M1+M2, where M1 is the storage size of the codebook vector and the cluster label vector, M1=Nc×M×K+N×log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2=2×Rsvd×K×max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

[0061] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0062] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above descriptions are merely optional embodiments of this application and are not intended to limit the application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A lightweight method for sharing weights across similar channels in large models, designed for domestic operating systems, characterized in that... The method includes: Step 1: Perform channel clustering processing, including performing K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model column by column, and updating the cluster centers iteratively to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. Step 2: Perform representative vector generation and storage compression, including calculating the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generating a cluster number list that records the cluster number to which each channel belongs. The original weight matrix is ​​replaced with the cluster number list and the representative vector to complete parameter compression. Step 3, construct the error matrix, including restoring the approximate weight matrix W' based on the cluster label list and representative vector, and calculating the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'; Step 4: Perform SVD error compensation, which involves performing Singular Value Decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. The first R largest singular values ​​and their corresponding vectors are retained to obtain an M×R dimension matrix. and an R×M dimension matrix ; Step 5, perform reasoning weight repetition, including in the model reasoning stage, the matrix and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; Step 6: Calculate the storage size, including the total storage size M of the compressed matrix, which satisfies M = M1 + M2, where M1 is the storage size of the codebook vector and cluster label vector, M1 = Nc × M × K + N × log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2 = 2 × Rsvd × K × max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

2. The method according to claim 1, characterized in that, The convergence condition for K-Means clustering in step 1 is that the change in the mean square error of cluster centers in two consecutive iterations is ≤1e -4 The number of clusters Nc is adjustable. When the target model is Llama-2-7B and Nc=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%. In step 2, the number of clusters CN can be adjusted. When the target model is Llama-2-7B and CN=256 and M=4096, the compression rate of a single weight matrix is ​​≥90%, and the result of log2Nc in the calculation of M1 is rounded up to ensure the storage integrity of the cluster number list.

3. The method according to claim 1, characterized in that, In step 2, the cluster label list has a dimension of 1×M, the representative vector has a dimension of M×1, and the original M×M weight matrix has a storage size of (CN+1)×M after compression. The rule for retaining the top R largest singular values ​​in step 4 is as follows: after sorting the singular values ​​from largest to smallest, retain the singular values ​​and their corresponding vectors with a cumulative contribution rate ≥ 95%. Let U be a submatrix formed by the column vectors corresponding to the first R singular values ​​in U. It is a submatrix formed by the column vectors corresponding to the first R singular values ​​in V; In step 6, the value of K corresponds to the storage format as follows: when the storage format is int8, K = 1 byte; when the storage format is float16, K = 2 bytes, and the calculation results of M1 and M2 are in bytes.

4. The method according to claim 1, characterized in that, The method further includes a non-matrix adaptation step, including: When the weight matrix is ​​M×N (M≠N), it is divided into multiple min(M,N)×min(M,N) square matrices according to the ratio of max(M,N) / min(M,N). Steps 1-6 are performed on each square matrix and then concatenated in the original order. At this time, the storage size of each sub-square matrix is ​​calculated as M1'=Nc×min(M,N)×K+min(M,N)×log2Nc and M2'=2×Rsvd×K×min(M,N). The total storage is the sum of M1' and M2' of all sub-square matrices.

5. The method according to claim 4, characterized in that, The non-square matrix is ​​divided into integer proportions. When the weight matrix is ​​4096×2048, it is divided into two 2048×2048 square matrices. After splicing, the weight distribution and dimensionality characteristics of the original matrix are preserved.

6. The method according to claim 1, characterized in that, The large models that the final inference weight WNew is adapted to in step 5 include Llama-2-7B, Llama-3.1-8B, Qwen-2.5-7B, and Deepseek-R1-Distill-Qwen-7B.

7. The method according to claim 1, characterized in that, The ratio of the number of cluster bits CN_Bit=log2Nc to the number of rank-preserving bits R_Bit=log2Rsvd is adapted to different models, specifically including: The Llama series models have a ratio of 0.33-0.6, and the Qwen series models have a ratio of 0.5-1, in order to achieve optimal perplexity level (PPL).

8. The method according to claim 1, characterized in that, The method also includes a visualization verification and parameter optimization process, including: A heatmap of the number of vectors within a cluster is generated to verify the uniformity of clustering, and a heatmap of the absolute value of the error matrix is ​​generated to verify the sparse distribution characteristics of the error, providing a basis for the parameter optimization of Nc and Rsvd.

9. A lightweight device for sharing weights across similar channels in a large model, designed for domestic operating systems, characterized in that... The device includes: The channel clustering processing module is used to perform K-Means clustering on the Q / K projector weight matrix of the self-attention layer of the Transformer architecture large language model. It iteratively updates the cluster centers to maximize the similarity of vectors within the same cluster and minimize the similarity between clusters until convergence. The representative vector generation and storage compression module is used to calculate the mean of all vectors in each cluster to obtain the cluster representative vector, and simultaneously generate a cluster number list recording the cluster number to which each channel belongs. The original weight matrix is ​​replaced by the cluster number list and the representative vector to complete the parameter compression. The error matrix construction module is used to reconstruct the approximate weight matrix W' based on the cluster label list and representative vectors, and to calculate the error matrix WErr=W-W' between the original weight matrix W and the approximate weight matrix W'. The SVD error compensation module performs singular value decomposition (SVD) on the error matrix WErr to obtain the left singular vector matrix U, the singular value diagonal matrix Σ, and the right singular vector matrix V. It retains the first R largest singular values ​​and their corresponding vectors, resulting in an M×R dimension matrix. and an R×M dimension matrix ; The original module repeats the reasoning weights, which is used during the model reasoning phase to transfer the matrix. and Multiply to obtain the dimension-reduced approximation error matrix W'Err, then add the approximation weight matrix W' to W'Err to obtain the final inference weight WNew=W'+W'Err, which is then used for text generation and inference tasks; The storage size calculation module is used to calculate the total storage size M of the compressed matrix, which satisfies M=M1+M2, where M1 is the storage size of the codebook vector and the cluster label vector, M1=Nc×M×K+N×log2Nc (Nc is the number of clusters, K is the storage size of the codebook vector, and N is the column dimension of the weight matrix), and M2 is the storage size of the compensation matrix retained in the SVD stage, M2=2×Rsvd×K×max(M,N) (Rsvd is the singular value rank-preserving function, and max(M,N) is the maximum dimension of the weight matrix).

Images