Context sparsity large language model efficient optimization system based on input driving

Abstract

The invention discloses a context sparsity large language model efficient optimization system based on input driving. The context sparsity large language model efficient optimization system comprises an optimization reconstruction module, a model level greedy optimization module, a block level greedy preheating module and a block level fine tuning module. A greedy algorithm is used for the large language model to determine sparseness configuration of each Transform Block; then, analyzing a full connection layer in each Block, and under the constraint of target sparsity, preheating sparsity configuration of each full connection layer by utilizing a greedy method and using relatively large compensation so as to prevent from falling into a local minimum value during block reconstruction; and finally, introducing a bridge function to perform sparseness configuration fine tuning, and establishing a relationship between the sparseness and the threshold in combination with a KDE method, thereby realizing fine tuning of the threshold. According to the method, a multi-stage optimization strategy is adopted, and the optimal sparse configuration is obtained through a coarse-to-fine adjustment process. And more effective compression is realized while the model performance is kept.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and deep learning technology of artificial intelligence and computer technology, and in particular to an efficient optimization system for large language models. Background Art

[0002] Deploying multiple large language models (LLMs) is a resource-intensive endeavor. LLMs typically incur higher computational costs, memory access, and memory usage overhead during inference, reducing efficiency in resource-constrained scenarios such as latency, throughput, power consumption, and storage. This poses challenges to their application in edge and cloud scenarios.

[0003] Several technologies have been proposed to address this challenge. At the data level, these include optimizing input prompts (input compression) and better organizing output content (i.e., output compression) to improve efficiency. This optimization does not change the model structure. At the model level, these mainly involve redesigning the model architecture or compressing pre-trained models. The former typically requires expensive pre-training to restore model capabilities, while the latter typically requires a small amount of fine-tuning to preserve model capabilities. System-level optimization typically involves optimizing the inference engine or service system. This type of optimization generally does not require retraining the model and does not degrade model capabilities.

[0004] Model compression typically involves technical directions such as quantization, sparsification, structural optimization, knowledge distillation, and dynamic inference. Here, we focus on model sparsification. Sparsification typically involves weight sparsification, activation sparsification, and attention sparsification. Weight sparsification typically involves pruning the weights of most fully connected layers in a pre-trained model. This typically involves structured pruning (e.g., row-by-row or channel-by-channel), semi-structured pruning (e.g., N:M pruning), and unstructured pruning. Unstructured pruning can achieve higher sparsity while minimizing the impact on model performance. However, due to the random distribution of zeros, specialized encoding methods such as CSR or BCSR are often required to achieve the desired reduction in model size. Regarding model acceleration, most current accelerators are not well-suited to handling unstructured sparsity, so this type of sparsity often fails to improve model inference performance and can even result in performance degradation. Semi-structured sparsity is currently more popular because current GPUs already support this type of sparsity. Compared to structured sparsity, semi-structured sparsity not only reduces model degradation but also offers benefits in inference efficiency. Even so, the degree of sparsity of weights is limited, and it can only bring a certain degree of sparsity.

[0005] In comparison, the sparsity of activations is dynamic and context-dependent, so it can usually achieve relatively high sparsity. Existing technologies such as ReLU^2Wins and ProSparse usually replace the activation function to achieve higher sparsity in the model. However, this usually requires expensive retraining and fine-tuning of the model, which not only places higher demands on the equipment but also on the dataset. When a relatively high sparsity is generated in the model, Dejavu and Powerinfer proposed using predictor technology to predict the sparsity state of the model for acceleration. However, the predictor method can only achieve good prediction results when the model is highly sparse, such as 90%. Therefore, this acceleration method is deeply bound to the ProSparse-type method, that is, only after the model is retrained to achieve high sparsity by replacing the activation function can Dejavu's technical method be used to accelerate the model's inference phase by predictor prediction.

[0006] In intensive computations in LLM (such as fully connected layers), it was found that when the TOPK method was applied to the absolute value of the input features and 50% of the minimum features were sparsified, the model performance was almost unaffected, although this may result in some invalid sparsity. However, directly applying the TOPK method to each input vector is not feasible in practical applications because its time complexity is O(nlogn). To this end, we obtained the threshold by inputting certain calibration data, thereby replacing the TOPK operation with higher time complexity. Subsequently, we counted the inference results of 500,000 tokens from different tasks and found that their corresponding thresholds under a certain sparsity were distributed in a small range. This shows that the sparsity of all tokens can be controlled by a unified threshold.

[0007] Since the LLM model is composed of multiple stacked Transformer Blocks, although the architecture of each Block is the same, their sensitivity to activation sparsity varies. In addition, the sensitivity of the fully connected layers within the Transformer Block to sparsity also varies. In order to optimize the distribution of activation sparsity under the control of global sparsity,

[0008] Problem Description: Current large language models (LLMs), often with billions or even tens of billions of parameters, result in high computational and memory costs, significantly reducing inference efficiency in resource-constrained scenarios. The inference process for large language models is typically divided into a prefill phase and a decoding phase. The prefill phase is primarily computationally bound, while the decoding phase is primarily limited by memory bandwidth. Frequent data movement in the decoding phase significantly increases inference latency.

[0009] However, activation sparsity is usually generated dynamically, and it is not advisable to introduce excessively time-consuming operations. Using the TOPK method to sparsify features in dense computations in LLM (such as fully connected layers) can improve sparsity without losing accuracy, but the high time complexity of the TOPK method requires us to seek more efficient alternatives.

[0010] Furthermore, LLMs are composed of multiple stacked Transformer blocks. Although each block has the same architecture, their sensitivity to activation sparsity varies. The fully connected layers within a Transformer block also vary in their sensitivity to sparsity. Therefore, how to fine-tune the sparsity configuration for different Transformer blocks and fully connected layers to improve computational efficiency under global sparsity constraints remains an urgent problem. Summary of the Invention

[0011] In response to the above-mentioned shortcomings of the existing technology, the present invention proposes an efficient optimization method and system for large language models with context sparsity based on input-driven, which implements an optimized processing solution for accelerating the calculation of the fully connected layer, the main intensive computing unit in the Transformer architecture (i.e., a deep learning model architecture based on the self-attention mechanism).

[0012] The present invention provides an input-driven context-sparse large language model efficient optimization system, which includes a layer-level reconstruction module, a model-level greedy optimization module, a block-level greedy warm-up module, and a block-level fine-tuning module; wherein:

[0013] The hierarchical reconstruction module obtains the optimal sparsity loss threshold of each fully connected layer in each Transformer block of the large language model according to the preset target sparsity;

[0014] The model-level greedy optimization module performs a global sparsity allocation exploration on the large language model by iteratively allocating a sparse budget to each Transformer block. In each round, the fully connected layer within the Transformer block with the smallest sparsity loss is selected to increase the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches a preset target value as an optimization constraint, and performs an initial sparsity optimization configuration.

[0015] The block-level greedy warm-up module performs fine-grained sparsity configuration exploration on the fully connected layer in each Transformer block in the large language model, and performs initial sparsity warm-up configuration with the average sparsity in the Transformer block reaching a preset target value as the warm-up constraint;

[0016] The block-level fine-tuning module uses kernel density estimation to convert the discrete sparsity-threshold relationship into a continuously differentiable form based on the obtained initial sparsity optimization configuration and the initial sparsity warm-up configuration, and implements gradient-based refined optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsification.

[0017] In some embodiments, the model-level greedy optimization module further comprises:

[0018] Assume that the sparsity of the initialized LLM model is zero and enter the main loop iteration process: in each iteration, first check whether the average sparsity of the model is less than the target sparsity; if the condition is met, then traverse each Transformer block in turn, temporarily increase the sparsity of the current block by one step, and calculate the model performance loss after applying the new configuration; by comparing the loss values ​​of all blocks after increasing the sparsity, the algorithm selects the Transformer block with the smallest performance loss, permanently increases its sparsity, and completes this round of iteration; this process is repeated until the average sparsity of the model reaches the target value, and finally outputs the optimized sparsity configuration scheme within the model.

[0019] In some implementations, the block-level greedy preheating module further includes:

[0020] The model-level greedy optimization module outputs the optimized sparsity configuration scheme within the model, obtains the target sparsity value of the level greedy preheating module, and sets it to sp_block1; then, the sparsity of all fully connected layers in the block is initialized to the target sparsity value sp_block1, and the main loop iteration process is entered: in each round of iteration, each fully connected layer in the Transformer block is traversed, the sparsity of each layer is temporarily increased with a fixed step size, forward calculation is performed and the reconstruction loss is evaluated; the reconstruction loss after the sparsity of all layers is increased is compared, the layer with the smallest loss is selected, the sparsity is permanently increased by one step size, and then the gap between the average sparsity within the block and the target is recalculated. If the gap is still greater than the threshold, the next round of iteration is continued, otherwise the final sparsity configuration within the block is returned; this process continues until the average sparsity within the block reaches the target sparsity value or the sparsity loss is less than the preset threshold.

[0021] In some embodiments, the combined sparsity loss for each block is a combination of the sparsity loss and the reconstruction loss in represents the Transformer block sparse loss, Represents the Transformer block reconstruction loss.

[0022] In some implementations, the block-level fine-tuning module introduces a bridge function to establish a relationship between sparsity and a threshold, and combines a kernel density estimation method to implement fine-tuning of the threshold and perform fine-tuning of the sparsity configuration.

[0023] In some embodiments, a hardware Triton operator is used to implement sparse vector multiplication calculations, including in an initialization phase, first receiving input sparse vector parameters, including an output matrix pointer Y, an input weight matrix A, an input vector matrix X, a sparsification threshold threshold, and related dimension parameters, calculating the starting position according to the program block ID, and determining a specific row and column index range for each calculation block, and initializing the corresponding memory pointer.

[0024] Compared with the prior art, the present invention can achieve the following beneficial technical effects:

[0025] 1) Since online sparsity is used, each sparsification is performed in conjunction with contextual information. Therefore, very high sparsity can be achieved without affecting model performance without requiring training or changing the parameters of the original model.

[0026] 2) To determine the corresponding sparsity threshold for each layer given a calibration dataset, a layer-by-layer reconstruction algorithm is designed. This system uses a multi-level optimization strategy to achieve the optimal sparsity configuration through a coarse-to-fine adjustment process.

[0027] 3) Using a multi-level optimization strategy and a progressive optimization process from coarse to fine, this strategy avoids the problem of falling into local optimality in direct fine-grained optimization. It combines model-level, block-level, and layer-level optimization to achieve a more comprehensive sparsification effect. By combining greedy warm-up with gradient descent, it balances the exploration of the search space with optimization efficiency.

[0028] 4) Flexible sparsification processing is implemented, which can adaptively adjust the sparsity based on contextual information and the sensitivity of different Transformer Blocks. It supports differentiated sparsity configuration for different layers of the model, achieving more effective compression while maintaining model performance;

[0029] 5) It is implemented through the Triton programming model, fully leveraging the features of modern hardware, supporting multiple batches of processing, using atomic operations to ensure the correctness of parallel computing, and effectively improving the effectiveness of the model in the decoding phase.

[0030] 6) A magnitude-based pruning strategy is adopted, and thresholding of activation vectors is performed by defining routing, skipping a large number of ineffective calculations. This not only reduces the amount of computation but also avoids unnecessary loading of weight channels into registers, significantly improving inference speed.

[0031] We use kernel density estimation (KDE) to accurately model activation distributions and parameterize thresholds using a sigmoid function, enabling precise optimization based on gradient descent. This differentiable optimization method efficiently adjusts the threshold parameters for each layer, minimizing performance loss while meeting target sparsity, paving the way for subsequent Transformer block-by-block reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a module diagram of the input-driven context-sparse large language model efficient optimization system of the present invention.

[0033] Figure 2 This is the overall framework diagram of the input-driven context-sparse large language model efficient optimization system of the present invention;

[0034] Figure 3 Schematic diagram of the fully connected layer processing process of MOR;

[0035] Figure 4 Schematic diagram of the model-level greedy optimization implementation process;

[0036] Figure 5 Schematic diagram of the block-level greedy warm-up implementation process;

[0037] Figure 6 Schematic diagram of the sparse vector and matrix multiplication operator. DETAILED DESCRIPTION

[0038] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0039] The system uses a greedy algorithm to determine the sparsity configuration for each Transformer block in a large language model. Subsequently, the fully connected layers within each block are analyzed and, subject to the target sparsity, a greedy approach is used to pre-warm the sparsity configuration of each fully connected layer using a large offset to prevent it from falling into a local minimum during block reconstruction. Finally, a bridge function is introduced to fine-tune the sparsity configuration, and the KDE method is used to establish a relationship between sparsity and thresholds, thereby enabling fine-tuning of the thresholds. The model here refers to a large language model composed of fully connected layers based on the Transformer architecture.

[0040] The present invention provides an input-driven, context-sparse large language model efficient optimization system, which optimizes a large language model composed of fully connected layers based on the Transformer architecture. Figure 1 The system is shown to include a layer-level reconstruction module 100 , a model-level greedy optimization module 200 , a block-level greedy warm-up module 300 , and a block-level fine-tuning module 400 .

[0041] The hierarchical reconstruction module 100 obtains the optimal sparsity loss threshold for each fully connected layer in each Transformer block of the large language model according to a preset target sparsity.

[0042] The model-level greedy optimization module 200 performs a global sparsity allocation exploration on the large language model by iteratively allocating a sparsity budget to each Transformer block. In each round, the fully connected layer within the Transformer block with the smallest sparsity loss is selected to increase the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches a preset target value as an optimization constraint, and performs an initial sparsity optimization configuration.

[0043] The block-level greedy warm-up module 300 performs fine-grained sparsity configuration exploration on the fully connected layer within each Transformer block in the large language model, and performs initial sparsity warm-up configuration with the average sparsity within the Transformer block reaching a preset target value as a warm-up constraint;

[0044] The block-level fine-tuning module 400 uses kernel density estimation to convert the discrete sparsity-threshold relationship into a continuously differentiable form based on the obtained initial sparsity optimization configuration and the initial sparsity preheating configuration, thereby implementing gradient-based refined optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsification.

[0045] Specifically, each module is described as follows:

[0046] The hierarchical reconstruction module is the basic component of the system. Its core is to find the optimal threshold parameters for each fully connected layer under a given target sparsity. The module first establishes the difference between the target sparsity and the current sparsity of each fully connected layer as the sparsity loss function, and then introduces a bridge function to convert the discrete sparsity-threshold relationship into a differentiable form. The activation value distribution is accurately modeled by the kernel density estimation (KDE) method, and the threshold is parameterized using the sigmoid function, achieving precise optimization based on gradient descent. This differentiable optimization method can efficiently adjust the threshold parameters of each layer, minimize performance loss while meeting the target sparsity, and lay the foundation for subsequent Transformer block-by-block reconstruction;

[0047] The model-level greedy optimization module explores global sparsity allocation for large language models by iteratively allocating a sparse budget to each Transformer block (same sparsity within the block), selecting the block with the smallest performance loss to increase sparsity in each round until the overall average sparsity of the model reaches a preset target value. The module first establishes a mapping relationship between hierarchical sparsity and threshold parameters, then uses a greedy strategy to traverse each Transformer block, evaluating the impact of increasing sparsity on overall model performance, and ultimately obtaining the optimal model-level sparsity configuration, achieving a reasonable distribution and balance of sparsity across Transformer blocks.

[0048] The block-level greedy warm-up module performs fine-grained sparsity configuration exploration on the fully connected layers within each Transformer block, with the average sparsity within the block reaching a preset target value as the optimization constraint. This module iteratively assigns incremental sparsity to various fully connected layers within the block (such as q, k, v and output of the attention layer, and up and down of the MLP layer), and selects the layer with the smallest reconstruction loss to increase the sparsity in each round until the block-level sparsity target is met. This optimization process effectively narrows the search space, provides a good initial solution for subsequent fine parameter optimization, and achieves a reasonable distribution of sparsity between the fully connected layers within the block, balancing performance loss and computational efficiency.

[0049] The block-level fine-tuning module performs refined parameter adjustments based on the initial sparsity configuration obtained by model-level greedy optimization and block-level greedy warm-up. The module first establishes a mapping relationship between sparsity and threshold parameters for each Transformer block, and then uses kernel density estimation (KDE) technology to convert the discrete sparsity-threshold relationship into a continuous differentiable form to achieve precise gradient-based optimization. By iteratively adjusting the threshold parameters of each layer, the module minimizes the performance loss caused by sparsification while maintaining the block-level sparsity target. This fine-tuning process based on reconstruction loss can achieve more efficient model compression while maintaining the inference performance of large language models, making the sparsity pattern of each fully connected layer more adapted to its specific feature distribution, thereby achieving an optimal balance between computational efficiency and model performance.

[0050] Figure 2 The present invention shows a framework diagram for implementing the efficient optimization system for large language models based on input-driven context sparsity. The large language model involved in this invention uses the Transformer architecture. The primary computationally intensive unit in the Transformer architecture (i.e., a deep learning model architecture based on the self-attention mechanism) is the fully connected layer, and this invention is primarily designed to accelerate computations in these layers.

[0051] Figure 3The figure shows a schematic diagram of the fully connected layer processing of MOR. For each fully connected layer, a sparse representation mask is first obtained based on the activation vector and threshold. The sparse mask determines the sparsity of the activation vector. During the decoding phase, since the input of each layer is a vector with a fixed sparsity level, and the sparsity of the weights is structured, only the weights required for calculation can be loaded based on the sparse locations.

[0052] The specific implementation process of the input-driven context-sparse large language model efficient optimization system proposed in this invention is as follows:

[0053] 1. In the hierarchical reconstruction module, the difference between the target sparsity and the current sparsity of each fully connected layer is first established as the sparsity loss function, and then a bridge function is introduced to convert the discrete sparsity-threshold relationship into a differentiable form. This process performs model inference by inputting a calibration dataset (such as 128 random sentences in the wikitext2 dataset commonly used in large language models) to obtain the hidden state (hidden_state) received by each fully connected layer. Subsequently, a dynamic threshold is applied to convert the hidden state into a sparse representation to generate a sparse vector. The system calculates the difference between the actual sparsity of the sparse vector and the preset target sparsity to form a hierarchical reconstruction sparsity loss. Based on this loss, the learnable parameter scores are fine-tuned through the gradient descent method to ultimately determine the optimal scores value for achieving a specific sparsity target. This precise parameter mapping provides a reliable foundation for the optimization of subsequent layers.

[0054] 2. During the model-level greedy optimization phase, the system uses a greedy algorithm with a large search step size to explore global sparsity configurations for the large language model. This phase focuses on overall model performance, allocating different sparsity budgets to each Transformer block through iterative optimization. Specifically, in each iteration, the algorithm selects the block with the smallest performance loss to increase sparsity, resulting in blocks at different locations in the model receiving differentiated sparsity configurations based on their importance to the task. Typically, shallow and deep blocks maintain lower sparsity to preserve key feature extraction and prediction capabilities, while intermediate blocks adopt higher sparsity. This adaptive allocation strategy ensures that while meeting the target average sparsity, performance loss is minimized, achieving a locally optimal balance between overall performance and computational efficiency. Since the LLM model is composed of multiple stacked Transformer blocks, despite having the same architecture, their sensitivity to activation sparsity varies. Because different Transformer blocks have different sensitivity to activation sparsity, a simple greedy algorithm is used before block-level reconstruction to find the sparsity ratio for each Transformer block within the overall sparsity constraint. In this process, the sparsity of all fully connected layers within each Transformer Block is the same. In order to optimize the distribution of activation sparsity under global sparsity control, a greedy algorithm is first used to determine the sparsity configuration of each TransformerBlock. Subsequently, the fully connected layers within each Block are analyzed, and under the constraint of the target sparsity, a greedy method is used to preheat the sparsity configuration of each fully connected layer using a larger compensation to prevent it from falling into a local minimum during block reconstruction; a bridge function is introduced to establish the relationship between sparsity and threshold, and the kernel density estimation method (KDE) is combined to achieve fine-tuning of the threshold and fine-tune the sparsity configuration (mainly fine-tuning the score of the control threshold: threadhold = sigmoid(scores)). Figure 4The model-level greedy optimization process is shown. The sparsity of the LLM model is initialized to zero, and the main iteration loop begins. In each iteration, the average model sparsity (the sum of the sparsities of all Transformer blocks divided by the number of blocks) is checked to see if it is less than the target sparsity. If so, each Transformer block is traversed in turn, the sparsity of the current block is temporarily increased by a step size, and the model performance loss is calculated after the new configuration is applied. By comparing the loss values ​​after increasing the sparsity of all blocks, the algorithm selects the block with the smallest performance loss and permanently increases its sparsity, completing the iteration. This process is repeated until the average model sparsity reaches the target value, and the optimized sparsity configuration within the model is finally output. Model-level greedy optimization refers to a global sparsity allocation strategy that uses the Transformer block as the basic unit of operation. This level of optimization treats each Transformer block as a whole and assigns it a specific target sparsity, rather than treating each layer within the block separately. In each evaluation round, the algorithm performs end-to-end inference on the complete model, evaluating overall model performance metrics (such as perplexity or task accuracy) to determine the pros and cons of the sparsity allocation scheme. This macro-level optimization ensures that the sparsity allocation across each Transformer block maximizes the overall functionality of the model, providing reasonable constraints for subsequent fine-grained optimization within the block.

[0055] 3. In the block-level greedy warm-up phase, for the specified Transformer block, set as Block1, first obtain the target sparsity value of the block through model-level greedy optimization, set as sp_block1. Subsequently, initialize the sparsity of all fully connected layers in the block to the target value, i.e., sp_block1, and set the initial sparsity loss to infinity. Enter the main loop process, which continues until the average sparsity in the block reaches the target value or the sparsity loss is less than the preset threshold. In each round of iteration, the algorithm traverses each fully connected layer in the block (including q, k, v, out of the attention layer and up, down of the MLP layer, etc.), temporarily increases the sparsity of each layer with a fixed step size, performs forward calculations and evaluates the reconstruction loss. Compare the reconstruction losses after the sparsity of all layers is increased, select the layer with the smallest loss, and permanently increase its sparsity by one step size. Then recalculate the gap between the average sparsity within the block (the average sparsity within the block is the sum of the products of the sparsity and weight of each fully connected layer divided by the sum of the weights of each fully connected layer) and the target. If the gap is still greater than the threshold, continue to the next round of iteration, otherwise return to the final intra-block sparsity configuration (the sparsity of the fully connected layers within the original Transformer Block is sp_block1 to the new intra-block sparsity configuration)

[0056] A discretized greedy search strategy is used to optimize the internal structure of each Transformer block, focusing on the fully connected layers within the block. Since only the score that determines the threshold is updated at this stage, rather than the network parameters, the search space exhibits a highly discrete nature. Therefore, a greedy algorithm with a large step size is used to initially explore the space of possible configurations. Figure 5 The block-level greedy warm-up process is shown. For a specified Transformer block, set as Block1, the target sparsity value of the block is first obtained through model-level greedy optimization, set as sp_block1. Subsequently, the sparsity of all fully connected layers in the block is initialized to the target value, i.e., sp_block1, and the initial sparsity loss is set to infinity. Enter the main loop process, which continues until the average sparsity in the block reaches the target value or the sparsity loss is less than the preset threshold. In each round of iteration, the algorithm traverses each fully connected layer in the block (including q, k, v, out of the attention layer and up, down of the MLP layer, etc.), temporarily increases the sparsity of each layer with a fixed step size, performs forward calculations and evaluates the reconstruction loss. Compare the reconstruction losses after the sparsity of all layers is increased, select the layer with the smallest loss, and permanently increase its sparsity by one step size. Then recalculate the gap between the average sparsity within the block (the average sparsity within the block is the sum of the products of the sparsity and weight of each fully connected layer divided by the sum of the weights of each fully connected layer) and the target. If the gap is still greater than the threshold, continue to the next round of iteration, otherwise return to the final intra-block sparsity configuration (the sparsity of the fully connected layers within the original Transformer Block is sp_block1 to the new intra-block sparsity configuration).

[0057] This iterative optimization process continues until one of the following conditions is met: either the average sparsity reaches the preset target value, at which point the algorithm completes and outputs the final optimization result; or, after all possible sparsity configurations have been tried, the system selects the sparsity configuration that performs best as the output. The entire process, both in the model-level greedy optimization module and the block-level greedy warmup module for the front-end and back-end execution relationships, employs a greedy strategy, selecting the local optimal solution at each step to ultimately achieve the overall optimization goal.

[0058] 4. During the block-level fine-tuning phase, based on the sparsity configuration obtained during the warm-up phase, gradient descent is used to fine-tune the sparsity parameters of each Transformer block. This phase of optimization focuses more on local details, using smaller adjustment steps to precisely optimize the sparsity configuration of each layer, ultimately achieving the goal of achieving efficient compression while maintaining model performance.

[0059] Specifically, layer-by-layer reconstruction involves finding the threshold t corresponding to a layer l in the fully connected layer under the target sparsity, and converting the loss function L of each layer into c As the sparsity loss of layer-by-layer reconstruction, to find the sparsity loss function of each layer Minimum threshold t l To reconstruct the target, as shown below:

[0060]

[0061] The loss function of each layer As shown in the following formula:

[0062]

[0063] in, represents the sparse loss of layer l, represents the target sparsity of the lth layer, r l Indicates the actual sparsity of layer l;

[0064] Define the loss function for each layer The gradient of the threshold of each layer is shown as follows:

[0065]

[0066] Among them, t l represents the threshold of the lth layer;

[0067] Use the bridge function to set the threshold t of the lth layer l and the actual sparsity r l The association is shown in the following formula:

[0068] t l =g(r l )

[0069] Among them, g() represents the bridge function, reflecting the threshold t of the lth layer l and sparsity r l the essential relationship between

[0070] The actual sparsity r of layer l is calculated by kernel density estimation (KDE) l Relative to the threshold t l Gradient The unknown probability density is as follows:

[0071]

[0072] Among them, p(x) represents the unknown probability density, K h is a non-negative kernel function, n is the number of sampling points, x is the point at which the probability density is estimated, x irepresents a sample of observed data points.

[0073] In order to ensure that the threshold is positive and monotonic, the Sigmoid function is used for constraint, as shown in the following formula:

[0074] t l =sigmoid(s l )

[0075] Among them, s l represents a learnable parameter of layer l.

[0076] The block-by-block reconstruction specifically includes:

[0077] Define the sparse loss for each Transformer block As shown in the following formula:

[0078]

[0079] in, N i 、r i , i represents the serial number of the fully connected layer in the block, represents the i-th fully connected layer, r i represents the sparsity of the i-th fully connected layer, N i represents the weight volume of the i-th fully connected layer, Indicates the target sparsity of this Transformer block.

[0080] Reconstruction loss for each block As shown in the following formula:

[0081]

[0082] in Indicates the output corresponding to the Block in the dense model, Indicates the output of the Block after it is sparse;

[0083] The total loss of each block is a combination of the sparsity loss and the reconstruction loss, as shown in the following formula:

[0084]

[0085] Calculate the combined loss pair s l The gradient of is as follows:

[0086]

[0087] In this process, the loss between the sparsity of the sparse vector and the target sparsity is calculated to obtain the reconstruction sparsity loss. The learnable parameter scores is then fine-tuned based on the reconstruction sparsity loss to find the scores under a certain sparsity.

[0088] In summary, the efficient optimization of large language models with contextual sparsity provided by this invention utilizes a multi-level optimization strategy, fully considering the various challenges of model compression. Through a rational hierarchical division and a progressive optimization process, it ensures both effective optimization and improved algorithm convergence efficiency. The entire process forms a complete closed loop, from coarse-grained global optimization to fine-grained local adjustments, ultimately outputting a fully optimized sparse model configuration.

[0089] The block-level greedy algorithm adopted in this invention is a greedy warm-up optimization algorithm process for Transformer Block: a large-step greedy search is used to find the initial sparsity configuration of the fully connected layer within each Transformer Block. The main reason for choosing the greedy algorithm for warm-up is that during the optimization process, only the score that determines the threshold is updated without updating the network parameters, which causes the feasible solution space to be highly discrete. In such a discrete solution space, if continuous optimization methods such as gradient descent are used directly, it is easy to fall into the local optimal solution and it is difficult to reach the global optimal point.

[0090] The unique feature of this warm-up process is its large search step size, which enables the algorithm to quickly explore multiple possible configurations in the discrete solution space, providing a good initial state for subsequent gradient descent fine-tuning. This optimization strategy, from "coarse search" to "fine tuning," effectively balances the algorithm's exploration capabilities and convergence efficiency, ultimately achieving superior compression results. After the warm-up phase, gradient descent is used to fine-tune the sparse parameters of each layer to achieve the final optimization results.

[0091] The model-level greedy algorithm used in this paper is based on the fact that different Transformer Blocks have different sensitivities to activation sparsity. Therefore, before block-level reconstruction, we can use a simple greedy algorithm to find the sparsity ratio of each Transformer Block under the overall sparsity constraint. In this process, the sparsity of all fully connected layers within each TransformerBlock is the same.

[0092] The significant feature of this method is its iterative optimization mechanism, which ensures both the optimization effect and the controllability and convergence of the algorithm through layer-by-layer adjustment and evaluation. This optimization strategy is particularly suitable for sparse optimization of large-scale Transformer models, and can effectively reduce computational overhead while maintaining model performance.

[0093] Furthermore, the present invention uses the Triton programming model (hardware Triton operator) to implement an efficient sparse vector multiplication method with significant performance advantages. The method mainly comprises three core stages: initialization stage, sparse pattern recognition stage and calculation accumulation stage.

[0094] Figure 6 The hardware Triton operator flow is shown. During the initialization phase, the algorithm first receives input parameters, including the output matrix pointer Y, the input weight matrix A, the input vector matrix X, the sparsification threshold threshold, and related dimensional parameters. The system calculates the starting position based on the block ID and determines the specific row and column index range for each computation block. It also initializes the corresponding memory pointers to prepare for subsequent calculations.

[0095] During the sparse pattern recognition phase, the algorithm uses a batch-based approach to traverse the input data. Using a threshold determined by the upper-level algorithm, it identifies and marks elements with significant weights, generating a corresponding computational mask. This mechanism effectively reduces unnecessary computation and improves overall operational efficiency. The algorithm iteratively processes each batch of data, continuously updating and refining the mask information until all batches are processed. This process then determines the union of neurons used across multiple batches.

[0096] In the final accumulation phase, the algorithm first loads the filtered weight matrix according to the mask generated in the previous phase. It then performs matrix multiplication in batches, and accumulates the results into the output matrix using atomic operations. This implementation ensures the correctness of parallel computing while fully utilizing hardware resources, effectively improving computational efficiency.

[0097] The technical innovations of this method are primarily reflected in the following aspects: First, sparsification significantly reduces computational effort and improves algorithmic efficiency; second, a batch processing mechanism enhances data access locality and optimizes memory usage; and third, a block-based computing strategy and the use of atomic operations ensure the correctness and efficiency of parallel computing. Furthermore, this method, implemented on the Triton programming model, fully leverages the characteristics of modern hardware architectures to achieve efficient parallel computing.

Claims

1. An efficient optimization system for large language models based on input-driven context sparsity, characterized by: The system includes a layer-level reconstruction module, a model-level greedy optimization module, a block-level greedy warm-up module, and a block-level fine-tuning module; wherein: The hierarchical reconstruction module obtains the optimal sparsity loss threshold of each fully connected layer in each Transformer block of the large language model according to the preset target sparsity; The model-level greedy optimization module performs a global sparsity allocation exploration on the large language model by iteratively allocating a sparse budget to each Transformer block. In each round, the fully connected layer within the Transformer block with the smallest sparsity loss is selected to increase the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches a preset target value as an optimization constraint, and performs an initial sparsity optimization configuration. The block-level greedy warm-up module performs fine-grained sparsity configuration exploration on the fully connected layer in each Transformer block in the large language model, and performs initial sparsity warm-up configuration with the average sparsity in the Transformer block reaching a preset target value as the warm-up constraint; The block-level fine-tuning module uses kernel density estimation to convert the discrete sparsity-threshold relationship into a continuously differentiable form based on the obtained initial sparsity optimization configuration and the initial sparsity warm-up configuration, and implements gradient-based refined optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsification.

2. The input-driven context-sparse large language model efficient optimization system according to claim 1, characterized in that: The model-level greedy optimization module further includes: Assume that the sparsity of the initialized LLM model is zero and enter the main loop iteration process: in each iteration, first check whether the average sparsity of the model is less than the target sparsity; if the condition is met, then traverse each Transformer block in turn, temporarily increase the sparsity of the current block by one step, and calculate the model performance loss after applying the new configuration; by comparing the loss values ​​of all blocks after increasing the sparsity, the algorithm selects the Transformer block with the smallest performance loss, permanently increases its sparsity, and completes this round of iteration; this process is repeated until the average sparsity of the model reaches the target value, and finally outputs the optimized sparsity configuration scheme within the model.

3. The input-driven context-sparse large language model efficient optimization system according to claim 1, characterized in that: The block-level greedy preheating module further includes: The model-level greedy optimization module outputs the optimized sparsity configuration scheme within the model, obtains the target sparsity value of the level greedy preheating module, and sets it to sp_block1; then, the sparsity of all fully connected layers in the block is initialized to the target sparsity value sp_block1, and the main loop iteration process is entered: in each round of iteration, each fully connected layer in the Transformer block is traversed, the sparsity of each layer is temporarily increased with a fixed step size, forward calculation is performed and the reconstruction loss is evaluated; the reconstruction loss after the sparsity of all layers is increased is compared, the layer with the smallest loss is selected, the sparsity is permanently increased by one step size, and then the gap between the average sparsity within the block and the target is recalculated. If the gap is still greater than the threshold, the next round of iteration is continued, otherwise the final sparsity configuration within the block is returned; this process continues until the average sparsity within the block reaches the target sparsity value or the sparsity loss is less than the preset threshold.

4. The input-driven context-sparse large language model efficient optimization system according to claim 1, characterized in that: The combined sparsity loss of each block is the combination of the sparsity loss and the reconstruction loss in represents the Transformer block sparse loss, Represents the Transformer block reconstruction loss.

5. The input-driven context-sparse large language model efficient optimization system according to claim 1, characterized in that: The model-level greedy optimization module introduces a bridge function to establish the relationship between sparsity and threshold, and combines the kernel density estimation method to achieve fine-tuning of the threshold and perform fine-tuning of the sparsity configuration.

6. The input-driven context-sparse large language model efficient optimization system according to claim 1, characterized in that: The hardware Triton operator is used to implement sparse vector multiplication calculations, including the initialization stage, first receiving the input sparse vector parameters, including the output matrix pointer Y, the input weight matrix A, the input vector matrix X, the sparsification threshold threshold and related dimension parameters, calculating the starting position according to the program block ID, and determining the specific row and column index range for each calculation block, and initializing the corresponding memory pointer.

Images