Sparse large language model weight refinement method, system and device and storage medium

Abstract

The invention discloses a sparse large language model weight refinement method, system and device and a storage medium, which are corresponding schemes, and the scheme realizes training-free and plug-and-play refinement of sparse weight after one-time pruning, significantly reduces the confusion degree, improves the zero sample task performance, and especially has an outstanding effect in a high sparse rate scene. The space is optimized in a unified mode through global soft constraint, heterogeneous Hessian inversion is avoided, pruning prior is effectively utilized in combination with historical momentum, Neumann series extrapolation solution is adopted, iteration time can be remarkably shortened, and expenses of computing resources and memory resources are reduced. The method is oriented to online and operation and maintenance processes of inference services such as text generation, question and answer, classification and retrieval enhanced question and answer and the like, and accesses the current network through hot update or gray release after completing intra-layer weight refinement; the existing word segmentation coding, service gateway and monitoring and rollback mechanisms are reused in the whole process, and it is ensured that the method can be implemented under real service distribution and resource budget.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and deep learning model compression technology, and in particular to a method, system, device and storage medium for refining the weights of a sparse large language model. Background Technology

[0002] Large Language Models (LLMs) have made significant progress in the field of natural language processing, but their ever-increasing parameter size has led to huge overhead in inference computation and memory consumption, limiting their efficient deployment in resource-constrained scenarios (such as edge computing, mobile terminals, and online services). Therefore, model compression techniques (including quantization, distillation, and pruning) have become an important direction for reducing costs and improving deployment efficiency.

[0003] Among these methods, pruning, which removes unimportant weights or structural units to obtain sparse models, is a core technique that balances versatility and inference acceleration potential. Compared to iterative pruning and fine-tuning schemes that require backpropagation and large-scale data, one-shot pruning has gained widespread attention in large language model compression due to its advantages of requiring no training, relying only on a small amount of calibration data, and being computationally efficient. Existing representative methods such as SparseGPT and Wanda have achieved a good trade-off between sparsity and accuracy by utilizing second-order approximation or weight-activation norm heuristics, respectively.

[0004] However, one-time pruning has inherent limitations: it typically employs row-wise or block-wise approximate reconstruction within a layer. As the sparsity increases (e.g., ≥50%), the approximation error accumulates rapidly, leading to a significant increase in language model perplexity and a decrease in zero-shot inference ability. This performance degradation stems primarily from two reasons: first, the pruned sparse weights are coarse-grained approximate solutions, lacking sufficient reconstruction and optimization; second, the Different Row-Hessian Challenge makes row-level Hessian inversion based on linear least squares reconstruction computationally expensive and difficult to scale to large-scale models, hindering high-quality weight reconstruction.

[0005] To alleviate these problems, traditional approaches often introduce fine-tuning or parameter-efficient fine-tuning (such as LoRA) to restore accuracy. However, this reintroduces significant training costs, data dependencies, and engineering complexity, contradicting the original intention of one-time pruning to be concise and efficient. On the other hand, some training-free works focus on mask refinement, improving sparse topologies through a grow-and-prune mechanism, but they pay insufficient attention to the reconstruction and refinement of the pruned weights themselves, resulting in limited final performance.

[0006] In summary, existing technologies have the following pain points and technical bottlenecks: (1) Training cost and data dependence: Recovery schemes that rely on fine-tuning are costly and time-consuming, which is not conducive to plug-and-play deployment.

[0007] (2) Scalability problem caused by row heterogeneous Hessian: Fine reconstruction based on row-level least squares backsolve requires independent matrix inversion for each row. The computation and memory overhead increases sharply with the model dimension, which requires high hardware resources.

[0008] (3) Insufficient utilization of one-time pruning priors: Existing training-free reconstruction methods do not effectively utilize the structural priors contained in the pruning stage, resulting in slow convergence and insufficient stability.

[0009] (4) Iteration overhead and numerical stability: Even with global covariance reconstruction, if dynamic matrix inversion is performed at each step, there is still an O(n) iteration overhead. 3 The computational bottlenecks and numerical risks are at the level of n, where n represents the order of magnitude of time complexity.

[0010] (5) Insufficient compatibility and composability: Some methods rely on specific pruning frameworks and are difficult to be seamlessly combined with mainstream one-time pruning strategies (such as SparseGPT and Wanda) or other training-free techniques (such as adaptive sparse allocation).

[0011] In inference applications such as enterprise knowledge base question answering, customer service dialogues, RAG rewriting, and mobile deployment, the deployment of pruned models requires minute-level fixes, can be rolled back in phases, and can reuse existing service links. Traditional fine-tuning cannot meet the deployment window requirements. Therefore, there is an urgent need for a weight refinement method that is training-free, plug-and-play, and highly scalable. This method should be able to efficiently iteratively optimize the sparse weights generated after a one-time pruning without relying on backpropagation or additional labeled data. It should bypass the bottleneck of row-level Hessian inversion, fully utilize the structural priors of the pruning stage, and significantly reduce the computational and memory overhead of each iteration through low-order approximation techniques. This would allow for stable recovery of model performance across various sparsity rates and LLM architectures. Summary of the Invention

[0012] The purpose of this invention is to provide a method, system, device, and storage medium for refining the weights of a sparse large language model, which can reduce the overhead of computing and memory resources. The refined sparse large language model can achieve comparable task performance to the original large language model in perplexity and zero-shot tasks, while effectively reducing the dependence on hardware resources, thus enabling its application in scenarios where both computing power and storage resources are limited.

[0013] The objective of this invention is achieved through the following technical solution: A method for refining weights in a sparse large language model includes: Obtain sparse large language models, sparse masks, and calibration samples; The text encoding covariance is constructed using calibration samples and combined with dense weights to generate the reference output of the dense model; where the dense model refers to the large language model that has not been pruned, and the dense weights are the weights of the dense model. Initialize soft constraints, introduce a historical momentum strategy, and use the sparse weights of the sparse language model as the initial weights to be refined. Construct an iterative update equation for the weights to be refined, which consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the dense model, and momentum coefficients calculated based on the historical momentum strategy; the second part is the inverse matrix constructed using momentum coefficients and text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initial weights to be refined. After the iterative refinement is completed, the weights to be refined obtained from the last update are structurally merged with the sparse mask to generate the final refined weights. The final refined weights are then loaded back into the sparse large language model layer by layer to complete the update of the sparse large language model. After that, the updated sparse large language model is used to provide text inference services.

[0014] A sparse large language model weight refinement system, used to implement the aforementioned method, includes: The information acquisition unit is used to acquire sparse large language models, sparse masks, and calibration samples. The reference output generation unit of the dense model is used to construct the text encoding covariance using calibration samples and generate the reference output of the dense model by combining the dense weights; where the dense model refers to a large language model that has not been pruned, and the dense weights are the weights of the dense model. The iterative refinement unit is used to initialize soft constraints, introduce historical momentum strategies, use the sparse weights of the sparse language model as the initial weights to be refined, and construct an iterative update equation for the weights to be refined. This iterative update equation for the weights to be refined consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the compact model, and momentum coefficients calculated based on historical momentum strategies; the second part is the inverse matrix constructed using momentum coefficients and text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initialized weights to be refined. The model update and inference unit is used to structurally merge the weights to be refined obtained from the last update with the sparse mask after the iteration refinement is completed, generate the final refined weights, load the final refined weights back into the sparse large language model layer by layer to complete the sparse large language model update, and then use the updated sparse large language model to provide text inference services.

[0015] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] As can be seen from the technical solution provided by the present invention, it achieves training-free, plug-and-play refinement of sparse weights after one-time pruning, significantly reducing perplexity and improving performance on zero-shot tasks, especially in high-sparse-rate scenarios; it unifies the optimization space through global soft constraints and avoids row-heterogeneous Hessian inversion, effectively utilizes pruning priors by combining historical momentum, and further reduces the complexity of each step from O(n) using Neumann series extrapolation. 3 ) decreases to O(n 2This invention significantly shortens iteration time and reduces computational and memory resource consumption while ensuring numerical stability and accuracy. The large language model with refined weights can be deployed in general GPU / CPU environments for model compression and inference acceleration, providing high-performance inference services. This invention addresses the deployment and operation of inference services such as text generation, question answering, classification, and enhanced question answering. It constructs a text encoding matrix and text encoding covariance using unlabeled text from the target business in pre-release or shadow instances. After refining the weights within each layer, it is integrated into the live network via hot updates or canary releases. The entire process reuses existing word segmentation encoding, service gateways, monitoring, and rollback mechanisms to ensure the method is implementable under real business distribution and resource budget constraints. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for refining the weights of a sparse large language model, provided as an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of a sparse large language model weight refinement method provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of a sparse large language model weight refinement system provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0024] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0025] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0026] The following provides a detailed description of a method, system, device, and storage medium for weight refinement of sparse large language models provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0027] Example 1 This invention provides a method for refining the weights of a sparse large language model, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Obtain the sparse large language model, sparse mask, and calibration samples.

[0028] In this embodiment of the invention, the sparse large language model is a pruned large language model.

[0029] Step 2: Construct text encoding covariance using calibration samples and combine it with dense weights to generate a reference output for the dense model.

[0030] In this embodiment of the invention, the calibration samples are text-encoded data used for calibration. The calibration samples are derived from unlabeled text data of the target business (such as FAQs, dialogue logs, and document fragments). After desensitization and cleaning, the existing word segmentation and encoding modules generate a text encoding matrix. To ensure deployment effectiveness, a minimum sample size and length distribution threshold can be set to remove abnormal samples.

[0031] In this embodiment of the invention, the dense model refers to a large language model that has not been pruned, and the dense weight is the weight of the dense model.

[0032] The implementation method for this step is as follows: Construct the input text encoding matrix using calibration samples. ,in, Let N be the symbol for the set of real numbers, and N be the number of calibration samples. Calculate the row dimension for each calibration sample; compute the text encoding covariance. T is the transpose symbol; the reference output for calculating the compact model. ,in, It is a single-layer dense weight. In practical applications, the text encoding covariance can directly reuse the intermediate product generated in a one-time pruning stage (such as SparseGPT), or be calculated offline in the refinement job and cached in the weight refinement job directory for cross-layer reuse and multiple starts.

[0033] Step 3: Iterative refinement of weights.

[0034] In this embodiment of the invention, soft constraints are initialized, a historical momentum strategy is introduced, and the sparse weights of the sparse language model are used as the initial weights to be refined. An iterative update equation for the weights to be refined is constructed. This iterative update equation for the weights to be refined consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the dense model, and the momentum coefficients calculated based on the historical momentum strategy; the second part is the inverse matrix constructed using the momentum coefficients and the text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initialized weights to be refined.

[0035] In this embodiment of the invention, the iterative update equation for the weights to be refined is expressed as: ; in, Indicates the target reconstruction signal, The corresponding inverse matrix, The weights to be refined in the k-th iteration are... B represents the weights to be refined in the (k-1)th iteration; B is the reference output of the compact model. Let be the momentum coefficient for the k-th iteration. Let I be the soft constraint for the k-th iteration, I be the identity matrix, and H be the text encoding covariance.

[0036] In this embodiment of the invention, the calculation of the momentum coefficient according to the historical momentum strategy is expressed as follows: ; in, Let be the momentum coefficient for the k-th iteration. , and is the preset hyperparameter; exp is the natural exponential function.

[0037] In this embodiment of the invention, the engineering approach of using a one-time baseline inverse or Neumann series extrapolation to calculate the inverse matrix includes: The reference inverse is calculated using the text encoding covariance, denoted as: ; If this is the first iteration, then reverse the baseline. It serves as the inverse matrix in the iterative update equation for the weights to be refined; If the current iteration is not the first iteration, then combine the baseline inverse and utilize the momentum coefficient. To leverage the smooth increasing characteristics of Neumann polynomials, an engineering approach using Neumann series extrapolation is employed to calculate the Neumann polynomials. These Neumann polynomials are then used as the inverse matrix in the iterative update equation for the weights to be refined. The Neumann polynomials are expressed as: , k > 0, where k is the current iteration number, meaning the current iteration is not the first iteration. Let be the momentum coefficient for the k-th iteration. Let X be the momentum coefficient for the (k-1)th iteration, X be the extrapolation term of the Neumann series, and the subscript be the index. , All were obtained using inverse calculations based on the baseline.

[0038] In this embodiment of the invention, the soft constraint is a penalty matrix. If the current iteration number k=0, then the soft constraint... For example, soft constraints for initialization If the current iteration number k > 0, then it is calculated using the following formula: ; in, The preset soft constraint strength coefficient, This indicates element-wise multiplication. The weights to be refined in the (k-1)th iteration. is the soft constraint for the (k-1)th iteration; M is the sparse mask.

[0039] Step 4: Refine weight loading and implement business reasoning.

[0040] After the iterative refinement is complete, the weights to be refined obtained from the last update are structurally merged with the sparse mask to generate the final, deployable refined weights, i.e.: ,in, The weights to be refined in the last update, where M is the sparse mask. This indicates element-wise multiplication. To finalize the weights, these finalized weights are loaded layer by layer back into the corresponding weight tensors of the sparse large language model, completing the model version update. Subsequently, the model is enabled in the target application scenario to provide text inference services: the user-input text is segmented and encoded to form a text encoding matrix. This matrix is ​​then used for forward computation through each layer with the finalized weights loaded, outputting the corresponding application results, including but not limited to text generation, question-and-answer responses, classification / matching tags, or structured decision information. To meet engineering deployment requirements, weight loading can be implemented through existing model parameter hot updates or canary release processes. Inference can reuse existing service gateways, batch / streaming decoding, caching, and monitoring components without additional modifications. The application process of this invention completes weight refinement using business corpora as calibration samples before deployment or during off-peak online periods, and seamlessly injects the refined weights into the inference chain. This achieves performance recovery and stable output of the sparse large language model under real business data distribution, balancing low computational overhead and scalable service quality.

[0041] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0042] I. Overall Overview of the Plan

[0043] To address the issues of slow model accuracy recovery, high deployment costs, and disconnect from business data distribution in existing technologies, this invention provides a sparse large language model weight refinement method for text inference services, called Efficient Iterative Weight Refinement (EIWR). EIWR, as an independent post-processing module after one-time pruning, integrates into the existing inference system without modifying the original pruning process or relying on additional labeled data. It directly utilizes the calibration text of the target business to generate its text encoding matrix and text encoding covariance, achieving rapid weight recovery and deployment under real-world application data distribution.

[0044] In the application process, EIWR first receives the pruned sparse large language model and sparse mask in the pre-release / offline environment, imports unlabeled text consistent with the target business as calibration samples, obtains the text encoding matrix through word segmentation and encoding, and statistically forms the text encoding covariance to characterize the business input distribution; then, based on the dense weights before pruning, it generates a dense baseline response (the target response in the covariance domain, which is also the reference output of the dense model mentioned above), which serves as the reference signal for intra-layer alignment. Under the premise of fixed mask constraints, EIWR updates weights by projecting them onto a shared covariance space using a hierarchical matrix approach, avoiding the high latency caused by row-by-row independent inversion. In terms of its iterative strategy, it replaces hard zeroing with cumulative sparse compliance soft constraints, allowing for brief exploration of locations outside the mask in the early stages to expand the feasible solution space and improve reachability. During iterative progress, the compliance penalty intensity is gradually increased, allowing the update results to naturally revert to the target sparse topology, ensuring direct compatibility with the sparse inference engine. Simultaneously, historical momentum is introduced to integrate structural priors with business data-driven directions. The initial sparse weights obtained through pruning are used as prior memory, enabling the update to accelerate convergence while suppressing oscillations, improving stability and robustness under real-world corpus distributions. For inverse mappings that require repeated solutions within a layer, EIWR employs an engineered fast approximation scheme of "one-time baseline inverse + Neumann series extrapolation," achieving gradual approximation with quadratic computational cost under isothermal conditions of layer normalization, balancing numerical accuracy and processing latency to meet the minute-level processing window and single-card GPU memory budget before deployment.

[0045] In the deployment phase, the final refined weights output by EIWR are loaded back into the sparse large language model layer by layer, and integrated into the existing text inference service chain through hot updates or canary releases. In actual business applications, the model uses the refined weights to encode user input text and perform forward inference, serving scenarios such as text generation, question answering, classification / matching, and enhanced question answering. In terms of engineering implementation, EIWR can reuse intermediate statistics generated by pruning processes (such as SparseGPT), adapt to mainstream Transformer architectures such as LLaMA2 / 3 and Qwen, and various sparsity rates, completing full model refinement in a single-GPU environment with minutes of additional time and GB of incremental GPU memory. It can also be used in conjunction with training maskless optimization (such as DsnoT) and adaptive sparse allocation (such as AlphaPruning) to obtain synergistic benefits. Through the above-mentioned application-oriented integrated process of data preparation, in-layer refinement, and online deployment, EIWR significantly improves perplexity and zero-shot task performance while reducing computational and memory overhead, ensuring the availability and scalability of the sparse large language model in real business scenarios.

[0046] II. Detailed introduction of the plan.

[0047] The method provided in this invention addresses the deployment and maintenance scenarios of models after "one-time pruning" in practical text inference services. It provides training-free, plug-and-play in-layer repair and deployment of coarse and sparse weights caused by pruning: In the pre-release / shadow environment, a text encoding matrix and text encoding covariance are first constructed based on the unlabeled text of the target business, and matrix-level weight alignment is performed within this "business input statistics" space; then, global soft constraints allow limited exploration of positions outside the mask in the early stages of iteration and strengthen compliance convergence of the target sparse topology in the later stages of iteration to meet the engineering requirements of the sparse inference engine; at the same time, historical momentum is introduced, and the initial sparse weights obtained by pruning are used as structural priors, adaptively fused with the alignment direction driven by business data, reducing oscillations and accelerating convergence; to meet the latency and memory budget before deployment, a one-time benchmark inversion combined with Neumann series extrapolation is used for fast approximation, completing each update step with quadratic computational overhead without increasing labeled data and backpropagation costs. After refinement, versioned final refined weights are generated and integrated into existing inference services via hot update / canary release, enabling the model to recover and stably output based on real data distribution in text generation, question answering, classification, and other business operations. The following sections explain the core steps mentioned above and their roles in the deployment process.

[0048] 1. Input and display settings.

[0049] Starting with the model obtained after a single pruning, let the dense weight of a certain layer (i.e., the weight before pruning) be denoted as . The initial sparsity weight after pruning is The sparse mask is M (a 1 in the mask indicates that the position should be zeroed), and the calibration sample set is used to construct the input matrix (text encoding) of this layer. (N is the number of calibration samples). Calculate the global input covariance (text-encoded covariance). This can be directly reused as an intermediate result in most one-time pruning processes (such as SparseGPT). To facilitate subsequent matrix-level projection, it is pre-calculated. As the right-hand side of the dense reconstruction (the reference output of the dense model), and with the sparse initial value set to .

[0050] 2. Matrix-level reconstruction of global soft constraints.

[0051] To avoid the large number of row-level covariance inversion calculations caused by performing least-squares back substitution for each row in its independent subspace, this invention performs unified projection (i.e., multiplication of weights by text encoding covariance) and reconstruction in a shared global covariance space: In the k-th iteration, an auxiliary penalty matrix is ​​introduced. This is used to apply cumulative soft constraints to locations that violate the target sparse structure, and the updated relation is: ; and with Gradually increase the penalty for mask position. Among them... This is the soft constraint strength coefficient. This indicates element-wise multiplication. This soft rather than hard sparse constraint allows for brief deviations from the mask in early iterations, expanding the feasible solution space to reduce reconstruction errors. Subsequently, as the soft constraint accumulates, it gradually reverts to the target sparse topology, thereby improving the overall stability and final accuracy of the optimization.

[0052] 3. Inject pruning priors using historical momentum strategies.

[0053] Considering that one-time pruning (such as based on second-order or heuristic importance assessment) provides an initial sparse solution with structural prior meaning, directly discarding this prior will lead to slower optimization convergence and greater instability under high sparsity. This invention adaptively fuses data-driven directions and historical solutions using matrix-level momentum interpolation, updating the solution as follows: ; in, To iterate the relevant momentum coefficients, by increasing the schedule Smooth growth from childhood to adulthood , and The hyperparameters can be set to enhance early exploration and later utilization of prior and historical information. In practice, this historical momentum strategy can significantly reduce iterative oscillations and improve convergence speed and reconstruction quality on high sparsity and large-scale layers.

[0054] 4. Fast approximate inversion of Neumann series extrapolation.

[0055] The updates in Parts 2 and 3 mentioned above both involve modifications to forms such as The dynamic matrix inversion. To avoid O(n) steps... 3 This invention utilizes the direct inversion overhead of ) to reduce the overhead of ) Due to its smooth increasing characteristics, Neumann series extrapolation is used to calculate the reference inverse only once during initialization. And pre-compute several polynomial terms, approximating them with a third-order truncated polynomial in the k-th iteration: , Where X is the extrapolation term of the Neumann series, and the subscript is the ordinal number, thus reducing the complexity of each step to O(n^2). 2 );in, The baseline inverse is calculated using the text-encoded covariance (characterizing the data distribution of the calibration samples), and the remaining three terms X are obtained using the baseline inverse. Under common layer normalization or input normalization conditions, the convergence radius and the upper bound of the residuals can be guaranteed to meet the design requirements, making this approximation numerically stable and reliable.

[0056] Considering that calculating the reference inverse using text encoding covariance and calculating the three terms X using the reference inverse are both conventional techniques, they will not be elaborated here.

[0057] Based on the above introduction, given the maximum number of iterations K or the early stopping criterion based on the reconstructed residuals, execute the loop:

[0058] initialization , ,set up , , and Calculate H, B and and the corresponding Neumann polynomial terms; for k=0,…,K−1, calculated according to the historical momentum strategy. and Construct the right-hand term (i.e., the target reconstruction signal). The Neumann approximation is obtained as follows Update to obtain weights and accumulate soft constraints The iteration is terminated early when the layer output reconstruction error or perplexity surrogate index no longer decreases significantly. After the iteration ends, the mask positions are cleared to obtain the final refined weights. ,in, The weights to be refined in the last update.

[0059] like Figure 2 As shown, this example illustrates the principle of the entire scheme. It should be noted that the specific information such as the size and value of the weight matrix involved in this example is only for illustration and does not constitute a limitation. In practical applications, this specific information can be adjusted according to the actual situation. Figure 2 In the lower right part, Naive represents the original method (e.g., sparsegpt or wanda mentioned later), w.Global represents the first component with the method (i.e. the matrix-level reconstruction / unified projection component with global soft constraints provided by this invention), w.Momentum represents the second component with the method (i.e. the historical momentum strategy / prior fusion provided by this invention), and EIWR represents the complete method of this invention.

[0060] The method provided in this invention can be integrated as an independent post-processing module after a one-time pruning, without changing the original pruning process and mask selection strategy. It can directly reuse H and calibration samples calculated during the pruning stage using methods such as SparseGPT, without requiring additional labeled data or backpropagation. In actual deployment, it can be executed layer-by-layer or module-by-module pipelined to reduce peak memory usage. On mainstream GPUs (such as a single 3090 card), a 7B-scale model with 70% sparsity typically introduces only about a few minutes of additional time and about 1 GB of incremental memory, while a 70B-scale model maintains acceptable linear scalability. After weight refinement, it can be directly used for inference. If used in conjunction with training maskless optimization (e.g., DsnoT) or adaptive sparse allocation (e.g., AlphaPruning), a better sparse topology can be determined before performing the weight refinement of this invention to obtain synergistic benefits.

[0061] The deployment process is as follows: On the data side, recent unlabeled text is extracted from the target business system, and desensitization, cleaning, and segmentation are performed. A text encoding matrix is ​​generated using existing word segmentation and encoding modules, and the text encoding covariance is calculated. On the model side, the pruned sparse model and mask are loaded in the pre-release or shadow copy environment, a dense reference weight snapshot is imported, the in-layer refinement process of this invention is executed, and the reconstruction residual, perplexity proxy, and zero-sample task indicators are monitored in real time. Early stopping occurs when the threshold is met. On the output side, the final refined weights are structurally zeroed out, quantized consistency is verified, and mask alignment is checked to generate a versioned weight package. On the deployment side, the refined weights are injected into low-traffic instances using hot update or canary release strategies, and latency, error rate, and business KPIs are monitored. Aligning metrics with offline metrics, gradually expanding capacity after stabilization, and automatically rolling back in case of anomalies; on the runtime side, the model uses refined weights to provide inference services such as text generation, question answering, and classification, and triggers incremental refinement jobs on a weekly or metric fluctuation threshold to achieve continuous alignment with business distribution; on the governance side, setting upper limits on video memory and duration for job resources, and retaining full-link logs to meet compliance audit and traceability requirements.

[0062] III. Performance Description.

[0063] To verify the effectiveness and efficiency of the present invention (EIWR) after one-time pruning, single-layer reconstruction experiments were conducted on LLaMA2 / 3 and Qwen2.5 (3B–70B) with SparseGPT and Wanda as baselines, sparsity of 60%–90%, and C4 subset of 128×2048 tokens as calibration data, as shown in Table 1.

[0064] Table 1: Perplexity results of different methods under different sparsities and different models

[0065] In addition, comparative experiments were conducted on a chemistry question-answering task, and the results are shown in Table 2. Each column represents one question in the chemistry question-answering task. The first row shows the performance of the unrefined sparse model, and the second row shows the performance of the refined sparse model. The baseline is Wanda in Table 1.

[0066] Table 2: Comparison of results of different methods on the chemical question-answering task

[0067] Single-layer reconstruction experiments show that EIWR reduces the layer output reconstruction error to near the upper limit of row-level least squares back-substitution in less than 1 second, while row-level back-substitution requires hundreds of seconds per layer, demonstrating efficient and high-quality weight reconstruction capabilities. System-wide evaluation shows that at 70% sparsity, EIWR significantly reduces perplexity and improves zero-sample average performance. For example, LLaMA2-7B (SparseGPT baseline) shows a perplexity reduction of approximately 15% on WikiText2, with stable gains on tasks such as PIQA, ARC, BOOLQ, COPA, and MultiRC, and consistent validation on LLaMA3-8B and Qwen2.5-3B. At 80%–90% sparsity... In the 0% extreme sparsity range, EIWR effectively suppresses perplexity degradation and significantly alleviates error accumulation caused by high sparsity. Regarding resource consumption, taking LLaMA2-7B with 70% sparsity as an example, it only increases training time by about 1.8 minutes and requires about 1GB of VRAM. The 70B scale is also within an acceptable range. Compared to the tens to hundreds of GPU hours required for LoRA or full-scale fine-tuning, its training-free and plug-and-play advantages are obvious. Method compatibility and ablation analysis further demonstrate that EIWR can be combined with DsnoT and AlphaPruning to obtain compound benefits. Removing historical momentum or Neumann extrapolation will reduce convergence efficiency and final performance, while the calibration sample size tends to saturate at approximately 128 segments. In summary, EIWR significantly recovers and stably improves sparse LLM performance at extremely low cost without requiring training or additional data. It possesses high sparsity robustness and good engineering feasibility, and can be used as a standard post-processing module for one-time pruning.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0069] Example 2 This invention also provides a sparse large language model weight refinement system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes: The information acquisition unit is used to acquire sparse large language models, sparse masks, and calibration samples. The reference output generation unit of the dense model is used to construct the text encoding covariance using calibration samples and generate the reference output of the dense model by combining the dense weights; where the dense model refers to a large language model that has not been pruned, and the dense weights are the weights of the dense model. The iterative refinement unit is used to initialize soft constraints, introduce historical momentum strategies, use the sparse weights of the sparse language model as the initial weights to be refined, and construct an iterative update equation for the weights to be refined. This iterative update equation for the weights to be refined consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the compact model, and momentum coefficients calculated based on historical momentum strategies; the second part is the inverse matrix constructed using momentum coefficients and text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initialized weights to be refined. The weight loading and inference unit is used to structurally merge the weights to be refined obtained from the last update with the sparse mask after the iteration refinement is completed, and generate the final refined weights; load the final refined weights back into the sparse large language model layer by layer to complete the update of the sparse large language model; and then use the updated sparse large language model to provide text inference services.

[0070] Since the technical details involved in this system have been described in detail in the previous embodiments, they will not be repeated here.

[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0072] Example 3 The present invention also provides a processing device, such as Figure 4As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0073] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0074] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0075] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0076] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0077] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for refining weights in a sparse large language model, characterized in that, include: Obtain sparse large language models, sparse masks, and calibration samples; The text encoding covariance is constructed using calibration samples and combined with dense weights to generate the reference output of the dense model; where the dense model refers to the large language model that has not been pruned, and the dense weights are the weights of the dense model. Initialize soft constraints, introduce a historical momentum strategy, and use the sparse weights of the sparse language model as the initial weights to be refined. Construct an iterative update equation for the weights to be refined, which consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the dense model, and momentum coefficients calculated based on the historical momentum strategy; the second part is the inverse matrix constructed using momentum coefficients and text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initial weights to be refined. After the iterative refinement is completed, the weights to be refined obtained from the last update are structurally merged with the sparse mask to generate the final refined weights. The final refined weights are then loaded back into the sparse large language model layer by layer to complete the update of the sparse large language model. After that, the updated sparse large language model is used to provide text inference services.

2. The method for refining weights in a sparse large language model according to claim 1, characterized in that, The step of constructing text encoding covariance using calibration samples and generating a reference output for the dense model by combining dense weights includes: Construct the input text encoding matrix using calibration samples ,in, Let N be the symbol for the set of real numbers, and N be the number of calibration samples. The row dimension for each calibration sample; Calculate text encoding covariance T is the transpose symbol; Reference output of the computational compact model ,in, It is a single-layer dense weight.

3. The method for refining weights in a sparse large language model according to claim 1, characterized in that, The iterative update equation for the weights to be refined is expressed as: ； in, Indicates the target reconstruction signal, The corresponding inverse matrix, The weights to be refined in the k-th iteration are... B represents the weights to be refined in the (k-1)th iteration; B is the reference output of the compact model. Let be the momentum coefficient for the k-th iteration. Let I be the soft constraint for the k-th iteration, I be the identity matrix, and H be the text encoding covariance.

4. A method for refining weights in a sparse large language model according to claim 1 or 3, characterized in that, The momentum coefficient calculated according to the historical momentum strategy is expressed as follows: ； in, Let be the momentum coefficient for the k-th iteration. , and is the preset hyperparameter; exp is the natural exponential function.

5. A method for refining weights in a sparse large language model according to claim 1 or 3, characterized in that, The engineering methods for calculating the inverse matrix using one-time baseline inverse or Neumann series extrapolation include: The reference inverse is calculated using the text encoding covariance, denoted as: ; If this is the first iteration, then reverse the baseline. It serves as the inverse matrix in the iterative update equation for the weights to be refined; If the current iteration is not the first iteration, then, combining the baseline inverse and utilizing the smooth increasing property of the momentum coefficient, an engineered scheme of Neumann series extrapolation is used to calculate the Neumann polynomial. The Neumann polynomial is then used as the inverse matrix in the iterative update equation for the weights to be refined. The Neumann polynomial is expressed as: , k > 0, where k is the current iteration number. Let be the momentum coefficient for the k-th iteration. Let X be the momentum coefficient for the (k-1)th iteration, X be the extrapolation term of the Neumann series, and the subscript be the index. , All were obtained using inverse calculations based on the baseline.

6. A method for refining weights in a sparse large language model according to claim 1 or 3, characterized in that, The soft constraint is a penalty matrix. If the current iteration number k=0, then the soft constraint... For initialization of soft constraints, if the current iteration number k > 0, then it is calculated using the following formula: ； in, The preset soft constraint strength coefficient, This indicates element-wise multiplication. The weights to be refined in the (k-1)th iteration. is the soft constraint for the (k-1)th iteration; M is the sparse mask.

7. The method for refining weights in a sparse large language model according to claim 1, characterized in that, The final refined weights obtained by combining the last updated refined weights and the sparse mask are expressed as follows: ； in, The weights to be refined in the last update, where M is the sparse mask. This indicates element-wise multiplication. To refine the weights in the final step.

8. A sparse large language model weight refinement system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The information acquisition unit is used to acquire sparse large language models, sparse masks, and calibration samples. The reference output generation unit of the dense model is used to construct the text encoding covariance using calibration samples and generate the reference output of the dense model by combining the dense weights; where the dense model refers to a large language model that has not been pruned, and the dense weights are the weights of the dense model. The iterative refinement unit is used to initialize soft constraints, introduce historical momentum strategies, use the sparse weights of the sparse language model as the initial weights to be refined, and construct an iterative update equation for the weights to be refined. This iterative update equation for the weights to be refined consists of two parts: the first part is the target reconstruction signal constructed using soft constraints, the reference output of the compact model, and momentum coefficients calculated based on historical momentum strategies; the second part is the inverse matrix constructed using momentum coefficients and text encoding covariance. In the current iteration, the target reconstruction signal is calculated by combining the corresponding soft constraints and momentum coefficients. An engineering scheme of one-time benchmark inverse or Neumann series extrapolation is used to calculate the inverse matrix, which is then substituted into the iterative update equation for the weights to be refined to obtain the weights to be refined in the current iteration. The soft constraints are calculated using the weights to be refined updated in the previous iteration. If the current iteration is the first iteration, the weights to be refined updated in the previous iteration are the initialized weights to be refined. The model update and inference unit is used to structurally merge the weights to be refined obtained from the last update with the sparse mask after the iteration refinement is completed, generate the final refined weights, load the final refined weights back into the sparse large language model layer by layer to complete the sparse large language model update, and then use the updated sparse large language model to provide text inference services.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images