Input-driven context sparsity-based efficient optimization system for large language models

Abstract

The application discloses an input driving-based context sparsity large language model efficient optimization system, which comprises an optimization reconstruction module, a model level greedy optimization module, a block level greedy preheating module and a block level fine-tuning module; a greedy algorithm is used to determine the sparsity configuration of each Transform Block for the large language model; subsequently, the full connection layers in each Block are analyzed, and under the constraint of the target sparsity, a larger compensation is used to preheat the sparsity configuration of each full connection layer by using the greedy method, so as to prevent falling into a local minimum value during block reconstruction; finally, a bridge function is introduced to fine-tune the sparsity configuration, and the relationship between the sparsity and the threshold is established by combining the KDE method, so that the threshold is fine-tuned. The application adopts a multi-level optimization strategy, and the optimal sparsity configuration is obtained through a coarse-to-fine adjustment process. More efficient compression is realized while the model performance is maintained.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and deep learning technologies in artificial intelligence and computer technology, and in particular to an efficient optimization system for large language models. Background Technology

[0002] Deploying multiple large language models (LLMs) is a very resource-intensive process. LLMs typically require higher computational costs, memory access and memory usage overhead during inference, and their efficiency in resource-constrained scenarios, such as latency, throughput, power consumption and storage, poses a challenge to the application of LLMs in edge and cloud scenarios.

[0003] Several technologies have been proposed to address this challenge. At the data level, optimizations include improving input prompts (input compression) and better organizing output content (output compression) to improve efficiency. This optimization does not change the model structure. At the model level, there are mainly redesigns of the model architecture or compression of pre-trained models. The former usually requires expensive pre-training to recover model capabilities, while the latter mostly requires minimal fine-tuning to preserve model capabilities. System-level optimizations typically involve optimizing the inference engine or service system. These optimizations generally do not require model retraining and do not cause any degradation in model capabilities.

[0004] In model compression, techniques typically include quantization, sparsity, structure optimization, knowledge distillation, and dynamic inference. Here, we focus on model sparsity. Sparsity generally includes weight sparsity, activation sparsity, and attention sparsity. Weight sparsity usually involves pruning the weights of most fully connected layers in the pre-trained model, typically including structured pruning (e.g., row-wise, channel-wise), semi-structured pruning (e.g., N:M pruning), and unstructured pruning. Unstructured pruning can achieve higher sparsity with minimal impact on model performance; however, due to the random distribution of zeros, special encoding methods such as CSR or BCSR are often required to reduce model size. Regarding model acceleration, since most accelerators are not adept at handling unstructured sparsity, this type of sparsity often fails to improve inference performance and may even cause performance degradation. Semi-structured sparsity is currently more favored because current GPUs support such sparse structures. Therefore, semi-structured sparsity not only exhibits less model degradation compared to structured sparsity but also provides benefits in inference efficiency. Even so, the degree of sparsity of the weights is limited; it can only bring about a certain degree of sparsity.

[0005] In contrast, sparsity of activations, being dynamic and context-dependent, typically achieves higher sparsity. Existing techniques such as ReLU^2Wins and ProSparse usually replace activation functions to produce higher sparsity in the model. However, this usually requires costly retraining and fine-tuning of the model, placing higher demands on both equipment and datasets. Once high sparsity is achieved in the model, Dejavu and Powerinfer proposed using predictor techniques to anticipate the model's sparsity state and thus accelerate the process. However, predictor methods often only achieve good prediction results when the model is highly sparsity, such as reaching 90%. Therefore, this acceleration method is deeply tied to ProSparse-like approaches; that is, only after retraining the model through activation function replacement to achieve high sparsity can Dejavu's predictor method be used to accelerate the model's inference phase.

[0006] In computationally intensive LLM layers (such as fully connected layers), it was found that applying the TOPK method to the absolute values ​​of input features, sparsifying the smallest features by 50%, had almost no impact on model performance, although this might introduce some ineffective sparsity. However, directly applying the TOPK method to each input vector is impractical in real-world applications due to its time complexity of O(nlogn). Therefore, we replace the time-complexity TOPK operation by obtaining a threshold through certain calibration data. Subsequently, we analyzed the inference results of 500,000 tokens from different tasks and found that their corresponding thresholds at a certain sparsity were distributed within a small interval. This indicates that the sparsity of all tokens can be controlled by a uniform threshold.

[0007] Since the LLM model is composed of multiple stacked Transformer Blocks, although the architecture of each block is the same, their sensitivities to activation sparsity differ. Furthermore, the fully connected layers within each Transformer Block also vary in their sensitivity to sparsity. To optimize the allocation of activation sparsity under global sparsity control,

[0008] Problem Description: Current Large Language Models (LLMs), typically with billions or even tens of billions of parameters, result in high computational and memory costs, significantly reducing inference efficiency in resource-constrained scenarios. The inference process of large language models is generally divided into a Prefill phase and a Decoding phase. The Prefill phase is primarily computationally intensive (Computing Bound), while the Decoding phase is mainly limited by memory bandwidth (Memory Bound). During the Decoding phase, frequent data movement significantly increases inference latency.

[0009] However, activation sparsity is often dynamically generated, which cannot introduce overly time-consuming operations. Using the TOPK method to sparsify features in intensive computations (such as fully connected layers) in LLMs can improve sparsity without sacrificing accuracy, but the high time complexity of the TOPK method necessitates seeking more efficient alternatives.

[0010] Furthermore, LLM is composed of multiple stacked Transformer Blocks. Although the architecture of each Block is the same, their sensitivities to activation sparsity differ, and the sensitivity of the fully connected layers within the Transformer Blocks to sparsification also varies. Therefore, how to perform fine-grained sparsity configuration for different Transformer Blocks and fully connected layers under global sparsity constraints to improve computational efficiency remains an urgent problem to be solved. Summary of the Invention

[0011] To address the shortcomings of the existing technologies, this invention proposes an efficient optimization method and system for input-driven context-sparse large language models. It implements an optimization scheme to accelerate the computation of the fully connected layer, the main computationally intensive unit in the Transformer architecture (i.e., a deep learning model architecture based on self-attention mechanism).

[0012] This invention discloses an efficient optimization system for large, context-sparse language models based on input-driven principles. The system includes a layer-level reconstruction module, a model-level greedy optimization module, a block-level greedy warm-up module, and a block-level fine-tuning module; wherein:

[0013] The hierarchical reconstruction module obtains the optimal sparsity loss threshold for each fully connected layer within each Transformer block of the large language model based on a preset target sparsity.

[0014] The model-level greedy optimization module explores global sparsity allocation for the large language model. It iteratively allocates a sparsity budget to each Transformer block. In each round, it selects the fully connected layer inside the Transformer block with the smallest sparsity loss and increases the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches the preset target value as an optimization constraint, and performs initial sparsity optimization configuration.

[0015] The block-level greedy preheating module performs fine-grained sparsity configuration exploration on the fully connected layers inside each Transformer block in the large language model, and performs initial sparsity preheating configuration with the average sparsity within the Transformer block reaching a preset target value as a preheating constraint.

[0016] The block-level fine-tuning module uses kernel density estimation to transform the discrete sparsity-threshold relationship into a continuously differentiable form based on the obtained initial sparsity optimization configuration and the initial sparsity preheating configuration, thereby realizing gradient-based fine-tuning of the optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsity.

[0017] In some implementations, the model-level greedy optimization module further includes:

[0018] Assuming the initial sparsity of the LLM model is zero, the main loop iterative process begins: In each iteration, the algorithm first checks if the average sparsity of the model is less than the target sparsity. If the condition is met, each Transformer block is traversed sequentially, and the sparsity of the current block is temporarily increased by one step. After applying the new configuration, the model performance loss is calculated. By comparing the loss values ​​of all blocks after increasing sparsity, the algorithm selects the Transformer block with the smallest performance loss and permanently increases its sparsity, completing the current iteration. This process is repeated until the average sparsity of the model reaches the target value, and finally, the optimized intra-model sparsity configuration scheme is output.

[0019] In some embodiments, the block-level greedy preheating module further includes:

[0020] The optimized intra-model sparsity configuration scheme is output through the model-level greedy optimization module to obtain the target sparsity value of the level-level greedy warm-up module, set as sp_block1. Subsequently, the sparsity of all fully connected layers in this block is initialized to the target sparsity value sp_block1, and the main loop iteration process is entered: In each iteration, each fully connected layer in the Transformer block is traversed, and the sparsity of each layer is temporarily increased by a fixed step size. Forward calculation is performed and the reconstruction loss is evaluated. The reconstruction loss after increasing the sparsity of all layers is compared, and the layer with the smallest loss is selected. The sparsity is permanently increased by one step size, and then the difference between the average sparsity in the block and the target is recalculated. If the difference is still greater than the threshold, the next iteration is continued; otherwise, the final intra-block sparsity configuration is returned. This process continues until the average sparsity in the block reaches the target sparsity value or the sparsity loss is less than the preset threshold.

[0021] In some implementations, the combined sparsity loss for each block is a combination of the sparsity loss and the reconstruction loss. in This represents the sparse loss of the Transformer block. This represents the reconstruction loss of the Transformer block.

[0022] In some implementations, the block-level fine-tuning module introduces a bridge function to establish the relationship between sparsity and a threshold, and combines a kernel density estimation method to fine-tune the threshold and perform sparsity configuration fine-tuning.

[0023] In some implementations, sparse vector multiplication is performed using a hardware Triton operator. This includes receiving input sparse vector parameters during the initialization phase, including the output matrix pointer Y, the input weight matrix A, the input vector matrix X, the sparsification threshold, and related dimension parameters. The starting position is calculated based on the program block ID, and a specific row and column index range is determined for each computation block. At the same time, the corresponding memory pointers are initialized.

[0024] Compared with existing technologies, the present invention can achieve the following beneficial technical effects:

[0025] 1) Because online sparsity is used, each sparsity is combined with context information, so very high sparsity can be achieved without training or changing the parameters of the original model, without affecting the performance of the model.

[0026] 2) To obtain the sparsity threshold for each layer given a calibration dataset, a layer-by-layer reconstruction algorithm was designed. This algorithm employs a multi-level optimization strategy, using a coarse-to-fine adjustment process to obtain the optimal sparsity configuration.

[0027] 3) A multi-level optimization strategy is used, which adopts a gradual optimization process from coarse to fine. This avoids the problem that direct fine-grained optimization may get stuck in local optima. It combines model-level, block-level and hierarchical optimization to achieve a more comprehensive sparsity effect. By combining greedy warm-up and gradient descent, the efficiency of exploring the search space and optimizing is balanced.

[0028] 4) It implements flexible sparsity processing, which can adaptively adjust the sparsity according to the sensitivity of different Transformer Blocks based on context information. It supports differentiated sparsity configuration for different levels of the model, achieving more effective compression while maintaining model performance.

[0029] 5) Implemented through the Triton programming model, it fully utilizes the characteristics of modern hardware, supports processing in multiple batches, uses atomic operations to ensure the correctness of parallel computing, and effectively improves the effectiveness of the model in the decoding stage.

[0030] 6) An amplitude-based pruning strategy is adopted, and activation vectors are thresholded by defining routes to skip a large number of invalid calculations. This not only reduces the amount of computation but also avoids unnecessary weight channels being loaded into registers, thereby significantly improving inference speed.

[0031] Accurate optimization based on gradient descent is achieved by accurately modeling the activation value distribution using the kernel density estimation (KDE) method and parameterizing the threshold using the sigmoid function. This differentiable optimization method can efficiently adjust the threshold parameters of each layer, minimizing performance loss while satisfying the target sparsity, laying the foundation for subsequent block-by-block reconstruction of Transformer layers. Attached Figure Description

[0032] Figure 1 This is a block diagram of the efficient optimization system for input-driven context-sparse large language models according to the present invention.

[0033] Figure 2 This is a diagram illustrating the overall framework of the input-driven, context-sparse, large language model high-efficiency optimization system of the present invention.

[0034] Figure 3 This is a schematic diagram of the fully connected layer processing of MOR;

[0035] Figure 4 This is a schematic diagram illustrating the model-level greedy optimization implementation process.

[0036] Figure 5 A schematic diagram illustrating the block-level greedy preheating process;

[0037] Figure 6 This is a schematic diagram of sparse vector and matrix multiplication operators. Detailed Implementation

[0038] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0039] The system involves using a greedy algorithm to determine the sparsity configuration of each Transformer Block for a large language model. Subsequently, the fully connected layers within each Block are analyzed, and under the constraint of a target sparsity, a greedy method is used to preheat the sparsity configuration of each fully connected layer with a large compensation, thereby preventing getting trapped in local minima during block reconstruction. Finally, a bridge function is introduced for fine-tuning the sparsity configuration, and the KDE method is combined to establish a relationship between sparsity and a threshold, thus achieving fine-tuning of the threshold. The model here refers to a large language model based on the Transformer architecture and composed of fully connected layers.

[0040] This invention provides an efficient optimization system for large language models with context-sparse large language models based on input-driven architecture, which optimizes large language models composed of fully connected layers based on the Transformer architecture. Figure 1 The system is shown to include a layer-level reconstruction module 100, a model-level greedy optimization module 200, a block-level greedy warm-up module 300, and a block-level fine-tuning module 400.

[0041] The hierarchical reconstruction module 100 obtains the optimal sparsity loss threshold for each fully connected layer within each Transformer block of the large language model based on a preset target sparsity.

[0042] The model-level greedy optimization module 200 explores global sparsity allocation for the large language model. It iteratively allocates a sparsity budget to each Transformer block. In each round, it selects the fully connected layer inside the Transformer block with the smallest sparsity loss and increases the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches the preset target value as an optimization constraint, and performs initial sparsity optimization configuration.

[0043] The block-level greedy preheating module 300 performs fine-grained sparsity configuration exploration on the fully connected layers inside each Transformer block in the large language model, and performs initial sparsity preheating configuration with the average sparsity inside the Transformer block reaching a preset target value as a preheating constraint.

[0044] The block-level fine-tuning module 400, based on the obtained initial sparsity optimization configuration and the initial sparsity preheating configuration, uses kernel density estimation to transform the discrete sparsity-threshold relationship into a continuously differentiable form, thereby realizing gradient-based fine-tuning of the optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsity.

[0045] Specifically, each module is described as follows:

[0046] The hierarchical reconstruction module is the fundamental component of this system. Its core function is to find the optimal threshold parameter for each fully connected layer given a target sparsity. This module first establishes the difference between the target sparsity and the current sparsity of each fully connected layer as a sparsity loss function. Then, a bridge function is introduced to transform the discrete sparsity-threshold relationship into a differentiable form. The activation value distribution is accurately modeled using kernel density estimation (KDE), and the threshold is parameterized using the sigmoid function, achieving precise optimization based on gradient descent. This differentiable optimization method can efficiently adjust the threshold parameter of each layer, minimizing performance loss while satisfying the target sparsity, laying the foundation for subsequent block-by-block reconstruction of the Transformer layer.

[0047] The model-level greedy optimization module explores global sparsity allocation for large language models. It iteratively allocates sparsity budgets to each Transformer block (with equal sparsity within each block), selecting the block with the smallest performance loss in each round and increasing its sparsity until the overall average sparsity of the model reaches a preset target value. This module first establishes a mapping relationship between hierarchical sparsity and threshold parameters. Then, it uses a greedy strategy to traverse each Transformer block, evaluating the impact of increasing sparsity on overall model performance, ultimately obtaining the optimal model-level sparsity configuration and achieving a reasonable distribution and balance of sparsity among the Transformer blocks.

[0048] The block-level greedy warm-up module performs fine-grained sparsity configuration exploration on the fully connected layers within each Transformer block, using the average sparsity within the block reaching a preset target value as an optimization constraint. This module iteratively allocates incremental sparsity to various fully connected layers within the block (such as q, k, v, and output of the attention layer, and up and down of the MLP layer), increasing the sparsity of the layer with the smallest reconstruction loss in each round until the block-level sparsity target is met. This optimization process effectively reduces the search space, providing a good initial solution for subsequent fine-grained parameter optimization, while simultaneously achieving a reasonable allocation of sparsity among the fully connected layers within the block, balancing performance loss and computational efficiency.

[0049] The block-level fine-tuning module performs refined parameter adjustments based on the initial sparsity configuration obtained from model-level greedy optimization and block-level greedy warm-up. This module first establishes a mapping relationship between sparsity and threshold parameters for each Transformer block. Then, it uses kernel density estimation (KDE) to transform the discrete sparsity-threshold relationship into a continuously differentiable form, achieving gradient-based precise optimization. By iteratively adjusting the threshold parameters of each layer, this module minimizes the performance loss caused by sparsity while maintaining the block-level sparsity target. This fine-tuning process based on reconstruction loss can achieve more efficient model compression while maintaining the inference performance of large language models, making the sparsity patterns of each fully connected layer more adaptable to its specific feature distribution, thereby achieving the optimal balance between computational efficiency and model performance.

[0050] Figure 2 This paper presents a roadmap for implementing the efficient optimization system of the input-driven context-sparse large language model according to the present invention. The large language model involved in the present invention adopts the Transformer architecture. The main computationally intensive unit in the Transformer architecture (i.e., the deep learning model architecture based on the self-attention mechanism) is the fully connected layer, and the present invention mainly aims to accelerate the computation of the fully connected layer.

[0051] Figure 3The diagram illustrates the fully connected layer processing procedure of MOR. For each fully connected layer, a sparse representation mask is first obtained based on the activation vector and a threshold. The sparse mask determines the sparse state of the activation vector. In the decoding stage, since the input to each layer is a vector, the sparsity is fixed, and the sparsity is structured for the weights. Therefore, only the weights that need to be calculated can be loaded based on the sparse locations.

[0052] The specific implementation process of the input-driven, context-sparse, large language model high-efficiency optimization system proposed in this invention is as follows:

[0053] 1. In the hierarchical reconstruction module, the difference between the target sparsity and the current sparsity of each fully connected layer is first established as the sparsity loss function. Then, a bridge function is introduced to transform the discrete sparsity-threshold relationship into a differentiable form. This process uses a calibration dataset (such as 128 random sentences from the wikitext2 dataset commonly used in large language models) for model inference to obtain the hidden state received by each fully connected layer. Subsequently, a dynamic threshold is applied to convert the hidden state into a sparse representation, generating a sparse vector. The system calculates the difference between the actual sparsity of this sparse vector and the preset target sparsity, forming the hierarchical reconstruction sparsity loss. Based on this loss, the learnable parameters `scores` are fine-tuned using gradient descent to ultimately determine the optimal `scores` value to achieve a specific sparsity target. This precise parameter mapping provides a reliable foundation for the optimization of subsequent layers.

[0054] 2. In the model-level greedy optimization stage, the system employs a greedy algorithm with a large search step size to explore global sparsity configuration for the large language model. This stage primarily focuses on the overall performance of the model, allocating different sparsity budgets to each Transformer Block through iterative optimization. Specifically, in each iteration, the algorithm selects the block with the smallest performance loss and increases its sparsity, allowing blocks at different positions in the model to obtain differentiated sparsity configurations based on their importance to the task—generally, shallow and deep blocks maintain lower sparsity to preserve key feature extraction and prediction capabilities, while intermediate layer blocks bear higher sparsity. This adaptive allocation strategy ensures that while meeting the target average sparsity, performance loss is kept to a minimum, achieving a balance between overall performance and computational efficiency at local optima. Since the LLM model is composed of multiple stacked Transformer Blocks, although the architecture of each Block is the same, their sensitivity to activation sparsity varies. Because different Transformer Blocks have different sensitivities to activation sparsity, a simple greedy algorithm is used before block-level reconstruction to find the sparsity ratio of each Transformer Block under the overall sparsity constraint. In this process, the sparsity of all fully connected layers within each Transformer Block is the same. To optimize the allocation of activation sparsity under global sparsity control, a greedy algorithm is first used to determine the sparsity configuration of each Transformer Block. Subsequently, the fully connected layers within each Block are analyzed, and under the constraint of the target sparsity, a greedy method is used to preheat the sparsity configuration of each fully connected layer with a large compensation to prevent getting trapped in local minima during block reconstruction. A bridge function is introduced to establish the relationship between sparsity and a threshold, and the kernel density estimation method (KDE) is combined to fine-tune the threshold, performing sparsity configuration fine-tuning (mainly fine-tuning the fractional control threshold: threadhold = sigmoid(scores)). Figure 4The model-level greedy optimization process is illustrated. The initial sparsity of the LLM model is set to zero, and the main loop iterates. In each iteration, the algorithm first checks if the average sparsity of the model (the sum of the sparsities of all Transformer Blocks divided by the number of blocks) is less than the target sparsity. If the condition is met, each Transformer Block is iterated sequentially, and the sparsity of the current block is temporarily increased by one step. After applying the new configuration, the model performance loss is calculated. By comparing the loss values ​​of all blocks after increasing sparsity, the algorithm selects the block with the smallest performance loss and permanently increases its sparsity, completing the iteration. This process is repeated until the average sparsity of the model reaches the target value, finally outputting the optimized intra-model sparsity configuration scheme. Model-level greedy optimization refers to a global sparsity allocation strategy based on Transformer blocks as the basic operation unit. This level of optimization treats each Transformer block as a whole, assigning it a specific target sparsity, rather than processing each layer within the block individually. In each round of evaluation, the algorithm performs end-to-end inference on the complete model and determines the merits of the sparsity allocation scheme by evaluating the overall model performance metrics (such as perplexity or task accuracy). This macroscopic optimization ensures that the sparsity allocation among Transformer blocks preserves the overall functionality of the model to the greatest extent possible, providing reasonable constraints for subsequent fine-grained optimization within blocks.

[0055] 3. In the block-level greedy warm-up phase, for a given Transformer block, designated as Block1, the target sparsity value for the block, designated as sp_block1, is first obtained through model-level greedy optimization. Then, the sparsity of all fully connected layers within the block is initialized to the target value, sp_block1, and the initial sparsity loss is set to infinity. The main loop then begins, continuing until the average sparsity within the block reaches the target value or the sparsity loss is less than a preset threshold. In each iteration, the algorithm iterates through each fully connected layer within the block (including q, k, v, and out of the attention layer, and up and down of the MLP layer), temporarily increasing the sparsity of each layer by a fixed step, performing forward computation, and evaluating the reconstruction loss. The reconstruction loss of all layers after increasing sparsity is compared, and the layer with the smallest loss is selected, with its sparsity permanently increased by one step. Then, the difference between the average sparsity within the block (the average sparsity within the block is the sum of the products of the sparsity of each fully connected layer and the weight size, divided by the sum of the weight sizes of each fully connected layer) and the target is recalculated. If the difference is still greater than the threshold, the next iteration continues; otherwise, the final intra-block sparsity configuration is returned (changing from the original sparsity of all fully connected layers within the Transformer Block being sp_block1 to the new intra-block sparsity configuration).

[0056] A discretized greedy search strategy is employed to optimize the internal structure of each Transformer Block, primarily focusing on the fully connected layers within the Block. Since only the score determining the threshold is updated at this stage, without updating the network parameters, the search space exhibits highly discrete characteristics. Therefore, a greedy algorithm with a large stride is used to initially explore the possible configuration space. Figure 5 The block-level greedy warm-up process is illustrated. For a given Transformer block, designated Block1, the target sparsity value, designated sp_block1, is first obtained through model-level greedy optimization. Then, the sparsity of all fully connected layers within the block is initialized to the target value, sp_block1, and the initial sparsity loss is set to infinity. The main loop then continues until the average sparsity within the block reaches the target value or the sparsity loss is less than a preset threshold. In each iteration, the algorithm iterates through each fully connected layer within the block (including q, k, v, and out of the attention layer, and up and down of the MLP layer), temporarily increasing the sparsity of each layer by a fixed step, performing forward computation, and evaluating the reconstruction loss. The reconstruction losses of all layers after increasing sparsity are compared, and the layer with the smallest loss is selected, with its sparsity permanently increased by one step. Then, recalculate the difference between the average sparsity within the block (the average sparsity within the block is the sum of the products of the sparsity of each fully connected layer and the weight size, divided by the sum of the weight sizes of each fully connected layer) and the target. If the difference is still greater than the threshold, continue to the next iteration; otherwise, return to the final intra-block sparsity configuration (changing the original sparsity of all fully connected layers within the Transformer Block to sp_block1, which is now the new intra-block sparsity configuration).

[0057] This iterative optimization process continues until either of the following conditions is met: either the average sparsity reaches the preset target value, at which point the algorithm completes and outputs the final optimization result; or after all possible sparsity configurations have been tried, the system selects the best-performing sparsity configuration as the output result. The entire process of the model-level greedy optimization module and the block-level greedy warm-up module, which are related to the preceding and following execution stages, employs a greedy strategy, selecting a local optimum at each step to ultimately achieve the overall optimization goal.

[0058] 4. In the block-level fine-tuning stage, based on the sparsity configuration obtained in the warm-up stage, gradient descent is used to finely adjust the sparsity parameters of each Transformer Block. This stage of optimization focuses more on local details, using smaller adjustment steps to precisely optimize the sparsity configuration of each layer, ultimately achieving the goal of efficient compression while maintaining model performance.

[0059] Specifically, layer-by-layer reconstruction involves finding the threshold t corresponding to a layer l in a fully connected layer under the target sparsity, and then applying the loss function L of each layer. c As a sparsity loss for layer-by-layer reconstruction, to find the sparsity loss function for each layer. Minimum threshold t l The reconstruction target is as shown in the following formula:

[0060]

[0061] Loss function for each layer As shown in the following formula:

[0062]

[0063] in, This represents the sparse loss of the l-th layer. r represents the target sparsity of the l-th layer. l This represents the actual sparsity of the l-th layer;

[0064] Define the loss function for each layer. The gradient for each threshold is shown in the following equation:

[0065]

[0066] Among them, t l This represents the threshold of the l-th layer;

[0067] Use the bridge function to set the threshold t of the l-th layer l Compared with the actual sparsity r l The association is as follows:

[0068] t l =g(r l )

[0069] Where g() represents the bridge function, reflecting the threshold t of the l-th layer. l With sparsity r l The essential relationship between them;

[0070] The actual sparsity r of the l-th layer is calculated using kernel density estimation (KDE). l Relative to threshold t l gradient The unknown probability density is shown in the following equation:

[0071]

[0072] Where p(x) represents the unknown probability density, K h It is a non-negative kernel function, where n is the number of sampling points, and x is the point where the probability density is estimated. iThis represents the observed data point sample.

[0073] To ensure that the threshold is positive and monotonic, the Sigmoid function is used as a constraint, as shown in the following equation:

[0074] t l =sigmoid(s l )

[0075] Among them, s l This represents a learnable parameter of the l-th layer.

[0076] The block-by-block reconstruction specifically includes:

[0077] Define the sparse loss for each Transformer block. As shown in the following formula:

[0078]

[0079] in, N i r i , i represents the index of the fully connected layer within the block, r represents the i-th fully connected layer, and r represents the i-th fully connected layer. i N represents the sparsity of the i-th fully connected layer. i This represents the weight volume of the i-th fully connected layer. This indicates the target sparsity of the Transformer block.

[0080] Reconstruction loss per block As shown in the following formula:

[0081]

[0082] in This represents the output corresponding to this block within the dense model. This indicates the output corresponding to the Block after it has been sparsed;

[0083] The total loss for each block is a combination of sparsity loss and reconstruction loss, as shown in the following equation:

[0084]

[0085] The combined loss was calculated for s. l The gradient is shown in the following equation:

[0086]

[0087] In this process, the loss between the sparsity of the sparsed vector and the target sparsity is calculated to obtain the reconstruction sparsity loss. Based on the reconstruction sparsity loss, the learnable parameters scores are then fine-tuned to find the scores score under a certain sparsity.

[0088] In summary, the efficient optimization of context-sparse large language models provided by this invention employs a multi-level optimization strategy, fully considering various challenges in the model compression process. Through reasonable hierarchical partitioning and a progressive optimization process, it ensures both the optimization effect and improves the algorithm's convergence efficiency. The entire process forms a complete closed loop, from coarse-grained global optimization to fine-grained local adjustments, ultimately outputting a fully optimized sparse model configuration.

[0089] The block-level greedy algorithm used in this invention, specifically the greedy preheating optimization algorithm for Transformer Blocks, involves a large-step greedy search to find the initial sparsity configuration of the fully connected layers within each Transformer Block. The main reason for choosing a greedy algorithm for preheating is that during optimization, only the score that determines the threshold is updated, not the network parameters. This results in a highly discrete feasible solution space. In such a discrete solution space, directly using continuous optimization methods such as gradient descent can easily lead to getting trapped in local optima and making it difficult to reach the global optimum.

[0090] The unique feature of this warm-up process is the use of a large search step size. This allows the algorithm to quickly explore multiple possible configuration points in the discrete solution space, providing a good initial state for subsequent gradient descent fine-tuning. This optimization strategy, from "coarse search" to "fine adjustment," effectively balances the algorithm's exploration capability and convergence efficiency, ultimately achieving better compression results. After the warm-up phase, gradient descent is used to fine-tune the sparse parameters of each layer, thus obtaining the final optimization result.

[0091] The model-level greedy algorithm used in this invention addresses the issue that different Transformer Blocks have varying sensitivities to activation sparsity. Therefore, before block-level reconstruction, we can use a simple greedy algorithm to find the sparsity ratio of each Transformer Block under the overall sparsity constraint. During this process, the sparsity of all fully connected layers within each Transformer Block is the same.

[0092] The key feature of this method lies in its iterative optimization mechanism. By adjusting and evaluating layer by layer, it ensures both the effectiveness of the optimization and the controllability and convergence of the algorithm. This optimization strategy is particularly suitable for the sparsity optimization of large-scale Transformer models, effectively reducing computational overhead while maintaining model performance.

[0093] Furthermore, this invention utilizes the Triton programming model (hardware Triton operator) to implement an efficient sparse vector multiplication method with significant performance advantages. This method mainly comprises three core stages: an initialization stage, a sparse pattern recognition stage, and a computational accumulation stage.

[0094] Figure 6 The hardware Triton operator flow is illustrated. During the initialization phase, the algorithm first receives input parameters, including the output matrix pointer Y, the input weight matrix A, the input vector matrix X, the sparsity threshold, and relevant dimension parameters. The system calculates the starting position based on the program block ID and determines the specific row and column index range for each computation block, while simultaneously initializing the corresponding memory pointers to prepare for subsequent calculations.

[0095] In the sparse pattern recognition stage, the algorithm uses a batch processing approach to traverse the input data. Using a threshold determined by the upper-layer algorithm, elements with significant weights are identified and labeled, generating corresponding computational masks. This mechanism effectively reduces unnecessary computation and improves overall operational efficiency. The algorithm iteratively processes each batch of data, continuously updating and refining the mask information until all batches are processed, thereby determining the union of neurons used across multiple batches.

[0096] In the final computational accumulation stage, the algorithm first loads the filtered weight matrix based on the mask generated in the previous stage. Then, it performs matrix multiplication in batches and uses atomic operations to accumulate the results into the output matrix. This implementation ensures the correctness of parallel computation while making full use of hardware resources, effectively improving computational efficiency.

[0097] The main technical innovations of this method are reflected in the following aspects: First, sparsity processing significantly reduces the amount of computation and improves algorithm efficiency; second, the batch processing mechanism enhances the locality of data access and optimizes memory usage efficiency; third, the block-based computation strategy and the use of atomic operations ensure the correctness and efficiency of parallel computing. Furthermore, the implementation of this method on the Triton programming model fully utilizes the characteristics of modern hardware architecture, achieving highly efficient parallel computing.

Claims

1. An efficient optimization system for large language models with context-sparse, input-driven processing, characterized in that: The system includes a layer-level reconstruction module, a model-level greedy optimization module, a block-level greedy warm-up module, and a block-level fine-tuning module; among which: The layer-level reconstruction module obtains the optimal sparsity loss threshold for each fully connected layer within each Transformer block of the large language model based on a preset target sparsity. The model-level greedy optimization module explores global sparsity allocation for the large language model. It iteratively allocates a sparsity budget to each Transformer block. In each round, it selects the fully connected layer inside the Transformer block with the smallest sparsity loss and increases the sparsity to the optimal sparsity loss threshold of each fully connected layer until the overall average sparsity of the large language model reaches the preset target value as an optimization constraint, and performs initial sparsity optimization configuration. The block-level greedy warm-up module performs fine-grained sparsity configuration exploration on the fully connected layers within each Transformer block in the large language model. It uses the average sparsity within the Transformer block reaching a preset target value as a warm-up constraint to perform initial sparsity warm-up configuration. The hardware Triton operator is used to implement sparse vector multiplication calculation, including the initialization phase, which first receives the input sparse vector parameters, including the output matrix pointer Y, the input weight matrix A, the input vector matrix X, the sparsification threshold, and the dimension parameter. The starting position is calculated according to the program block ID, and the specific row and column index range is determined for each calculation block. At the same time, the corresponding memory pointers are initialized. The block-level fine-tuning module uses kernel density estimation to transform the discrete sparsity-threshold relationship into a continuously differentiable form based on the obtained initial sparsity optimization configuration and the initial sparsity preheating configuration, thereby realizing gradient-based fine-tuning of the optimal sparsity loss threshold parameter adjustment to minimize the performance loss caused by sparsity.

2. The efficient optimization system for a context-sparse large language model based on input-driven principles according to claim 1, characterized in that, The model-level greedy optimization module further includes: Assuming the initial sparsity of the large language model is zero, the main loop iterative process begins: In each iteration, the algorithm first checks if the average sparsity of the model is less than the target sparsity. If the condition is met, each Transformer block is traversed sequentially, and the sparsity of the current block is temporarily increased by one step. After applying the new configuration, the model performance loss is calculated. By comparing the loss values ​​of all blocks after increasing sparsity, the algorithm selects the Transformer block with the smallest performance loss and permanently increases its sparsity, completing the current iteration. This process is repeated until the average sparsity of the model reaches the target value, and finally, the optimized intra-model sparsity configuration scheme is output.

3. The efficient optimization system for a context-sparse large language model based on input-driven principles according to claim 1, characterized in that, The block-level greedy preheating module further includes: The optimized intra-model sparsity configuration scheme is output through the model-level greedy optimization module to obtain the target sparsity value of the block-level greedy warm-up module, denoted as sp_block1. Subsequently, the sparsity of all fully connected layers in the block is initialized to the target sparsity value sp_block1, and the main loop iteration process is entered: In each iteration, each fully connected layer in the Transformer block is traversed, and the sparsity of each layer is temporarily increased by a fixed step size. Forward calculation is performed and the reconstruction loss is evaluated. The reconstruction loss after increasing the sparsity of all layers is compared, and the layer with the smallest loss is selected. The sparsity is permanently increased by one step size, and then the difference between the average sparsity in the block and the target is recalculated. If the difference is still greater than the threshold, the next iteration is continued; otherwise, the final intra-block sparsity configuration is returned. This process continues until the average sparsity in the block reaches the target sparsity value or the sparsity loss is less than the preset threshold.

4. The efficient optimization system for a context-sparse large language model based on input-driven principles according to claim 1, characterized in that, The combined sparsity loss for each block is a combination of the sparsity loss and the reconstruction loss. ,in This represents the sparse loss of the Transformer block. This represents the reconstruction loss of the Transformer block.

5. The efficient optimization system for a context-sparse large language model based on input-driven principles according to claim 1, characterized in that, The model-level greedy optimization module introduces a bridge function to establish the relationship between sparsity and threshold, and combines the kernel density estimation method to fine-tune the threshold and perform sparsity configuration fine-tuning.

Images