Self-attention driven large model lightweight inference method and system

Abstract

This invention discloses a self-attention-driven lightweight inference method and system for large models, belonging to the field of artificial intelligence large model inference optimization technology. The method involves multi-dimensional basic segmentation and indexing of large model weights, performing offline preprocessing in host memory; extracting the self-attention distribution values ​​of Transformer layers to divide core / non-core weight regions, and dynamically planning differentiated slicing strategies; implementing multi-scale memory scheduling based on access frequency, inter-layer dependencies, and computational overhead, loading and releasing slices on demand, and finally concatenating the inference results. The system implementing this method comprises five functional modules that collaboratively complete lightweight inference. This invention is adaptable to hardware with different memory specifications, reducing memory usage by more than 40%, achieving an inference speedup of ≥1.8 times, and an inference similarity of ≥95%. It requires no modification to the model structure, is compatible with existing frameworks, and has strong engineering practicality.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence large model inference optimization technology, specifically involving a lightweight inference method for large models based on a self-attention mechanism, and an inference system for implementing the method. It can be applied to consumer-grade graphics cards, edge computing terminals, and server hardware with different memory specifications, and is compatible with multi-parameter large-scale model deployments of Transformer, MoE, and dense architectures. Background Technology

[0002] The continuous growth in the parameter size of large models has made their traditional full-load inference method more demanding on hardware memory, which to some extent limits the engineering implementation of large models in diverse hardware environments.

[0003] Existing lightweighting techniques for large models mainly include quantization, pruning, knowledge distillation, model slicing, and rule-based block slicing. Each technique has its limitations: quantization and pruning techniques, while reducing memory usage, can easily introduce irreversible loss of inference accuracy; knowledge distillation relies on high-quality teacher models, has poor generalization, and has high adaptation costs; model slicing techniques are mostly designed for multi-GPU distributed scenarios and are difficult to adapt to single-GPU hardware environments; traditional rule-based block slicing techniques optimize memory by dividing model weights by a fixed ratio, but its mechanical slicing method can easily disrupt the global correlation of the Transformer architecture's self-attention mechanism, potentially leading to a decrease in inference quality. Furthermore, it cannot flexibly adjust the slicing strategy according to hardware memory specifications and model type, making it difficult to achieve an effective balance between memory usage, inference speed, and inference accuracy.

[0004] Currently, there is no lightweight inference solution and supporting implementation system for large models that can adapt to hardware with different memory specifications and flexibly adjust the inference accuracy according to actual needs, without significantly modifying the original structure of the large model or increasing the computational overhead of the inference stage. Summary of the Invention (a) Purpose of the invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a self-attention-driven lightweight inference method and system for large models. It aims to achieve efficient inference of large models on hardware with varying memory specifications by guiding the dynamic planning of slicing strategies through a self-attention mechanism. Simultaneously, it allows for flexible adjustment of slicing-related parameters based on hardware memory specifications, model type, and actual accuracy requirements, striking a balance between memory optimization, inference speed improvement, and inference accuracy assurance. Furthermore, this method and system require minimal modification to the original structure of the large model, do not significantly increase computational overhead during the inference stage, and are compatible with existing mainstream large model inference frameworks. (II) Technical Solution

[0006] This method quantifies the importance of weights through self-attention distribution, realizes dynamic programming of the slicing strategy, and combines offline preprocessing and multi-scale memory scheduling to optimize memory usage while ensuring inference efficiency and accuracy. The specific steps are as follows: S1: Basic Weight Segmentation and Index Construction For the weights of Transformer, MoE, or dense large models, fine-grained basic segmentation is performed according to three dimensions: layer, functional module, and feature channel, generating independently computable slice units; based on the feature transfer logic of the large model, a slice index system is established that includes inter-layer dependencies between slices, access frequency characteristics, and computational overhead characteristics, providing data support for subsequent slice scheduling.

[0007] S2: Offline preprocessing Before inference, the entire preprocessing operation is completed in the host's physical memory, including loading the weights of the large model, standardizing and converting the weight format, and constructing the basic segmentation and slice index in the S1 step. All preprocessing operations do not occupy hardware memory, thus bringing the preprocessing-related overhead forward and avoiding squeezing out the memory and computing resources during the inference stage.

[0008] S3: Self-attention-driven slicing strategy dynamic programming Through 1-2 rounds of lightweight forward computation, the self-attention distribution values ​​of the weights of each layer of the Transformer are extracted to quantify the contribution of each weight unit to the inference result. An attention weight threshold is set, and units with weight values ​​≥ the threshold are divided into core weight regions, while units with weight values ​​< the threshold are divided into non-core weight regions. Fine-grained slicing is used for core weight regions to preserve the global correlation features of self-attention as much as possible, while coarse-grained merging is used for non-core weight regions to optimize memory usage. Finally, a slice scheduling plan table is generated. The attention weight threshold, core weight region slicing granularity, and non-core weight region merging ratio can all be flexibly adjusted according to the hardware memory specifications, model type, and actual inference accuracy requirements.

[0009] S4: On-demand loading and instant release in multi-scale scheduling During the inference phase, a multi-scale priority scheduling strategy based on access frequency, inter-layer dependency, and computational overhead is used to load only the model slices required for the current computation into the hardware memory. At the same time, based on the real-time attention distribution during inference, core weight region slices are dynamically loaded to ensure that core features are not lost. After a single slice is computed, its occupied hardware memory is released in a timely manner to avoid redundant data resident and improve memory utilization.

[0010] S5: Outputting the concatenation of inference results Based on the hierarchical structure of the large model and the feature transfer order of the feature channels, the intermediate inference results of each slice are sequentially spliced ​​and fused to output a complete large model inference result that is basically consistent with the inference format of the full model.

[0011] This system is based on a modular design of functional logic using the methods described above. Each module has independent functions and clear interfaces, working together to achieve the entire process of lightweight inference for large models. It also supports flexible adjustment of slicing-related parameters and adapts to hardware environments with different memory specifications. Specifically, it includes: Model Basic Slicing Module: Used to perform fine-grained basic slicing of large model weights in three dimensions: layers, functional modules, and feature channels. It constructs and updates a slice index system that includes inter-layer dependencies between slices, access frequency, and computational overhead, and outputs standardized slice units and slice index tables. Offline preprocessing module: Used to complete model weight loading, format standardization conversion, basic segmentation and slice index construction in the host physical memory before inference. It is completely isolated from the hardware memory to avoid occupying memory resources during the inference stage. The self-attention slicing strategy planning module is used to extract the self-attention distribution values ​​of the weights of each layer of the Transformer through lightweight forward computation, divide the core / non-core weight regions, dynamically plan the slicing strategy and generate a slice scheduling plan table, and supports flexible configuration and adjustment of attention weight thresholds and the slicing ratio of core / non-core weight regions. Multi-scale memory scheduling module: Used for multi-scale priority scheduling strategies based on access frequency, inter-layer dependency, and computational overhead to realize on-demand loading of model slices, dynamic loading of core weight region slices, and instant release of hardware memory, and manage the hardware memory occupancy status. The inference execution and result assembly module is used to perform inference calculations on model slices, assemble and merge the intermediate inference results of each slice according to the feature transfer order of the large model, and output the complete large model inference result, which is compatible with the output interface of the existing large model inference framework. (III) Beneficial Effects

[0012] The technical solution of this invention has been verified by actual testing and has the following beneficial effects compared with the prior art. Moreover, all of these effects can be adapted to the actual needs of hardware with different video memory specifications by adjusting the relevant slicing parameters: Strong memory adaptability and significant optimization effect: It can be applied to low-memory consumer graphics cards and edge computing terminals with 8GB or less of memory, as well as medium-to-high-memory graphics cards and server hardware with 8GB or more of memory. Through dynamic programming slicing strategy, with a weight retention rate of 60%±5%, the memory usage of large model inference can be reduced by more than 40% compared with the full loading method, effectively improving the adaptability of large models on hardware with different memory specifications. Improved inference efficiency with no additional overhead: Inference speedup can reach 1.8 times or more; all preprocessing and slicing strategy planning operations are completed before inference, and there is basically no additional computational or memory overhead during the inference stage, which does not affect the original computational efficiency of the hardware. The inference accuracy is adjustable to meet engineering needs: By adjusting parameters such as attention weight threshold and slice granularity, the inference accuracy can be flexibly adjusted; under normal parameter configuration, the cosine similarity between the inference result and the full model can reach 95% or more, and the accuracy loss is basically imperceptible in common engineering application scenarios such as natural language processing, GIS spatial analysis, and intelligent question answering; for high-precision scenarios, the inference accuracy can be further improved by increasing the slice granularity. It has good engineering practicality and wide applicability: it basically does not require modification of the original structure of large models, and is compatible with existing mainstream large model inference frameworks such as LLaMA.cpp, vLLM, and TensorRT, and can be directly integrated and deployed; it is compatible with large models of multiple parameters such as 7B and 13B, as well as large models with various architectures such as Transformer, MoE, and dense models. Low deployment cost and good scalability: No customized hardware is required, and it can be implemented on existing consumer-grade, industrial-grade, and server-grade hardware; the core optimization strategy can be flexibly adjusted according to the hardware computing power and memory specifications, and can adapt to the basic needs of large-scale model technology iteration and hardware upgrade.

Claims

1. A self-attention driven large model lightweight inference method, characterized in that, Includes the following steps: s1. Perform fine-grained basic segmentation of the weights of Transformer architecture, MoE architecture or dense large model by layer, functional module and feature channel to generate independent slice units, and establish a slice index system that includes inter-layer dependencies between slices, access frequency characteristics and computational cost characteristics. s2. Before inference, the model loading, weight format conversion, basic segmentation and slice index construction of step 1) are completed in the host physical memory, without occupying hardware video memory; s3. Obtain the self-attention distribution values ​​of the weights of each layer of Transformer through 1-2 rounds of lightweight forward computation, set the attention weight threshold, divide the units with weight values ​​≥ the threshold into core weight regions, divide the units with weight values ​​< the threshold into non-core weight regions, use fine-grained slicing for core weight regions and coarse-grained merging for non-core weight regions, and generate a slice scheduling plan table. The attention weight threshold, core weight region slice granularity, and non-core weight region merging ratio can be flexibly adjusted according to hardware memory specifications, model type, and inference accuracy requirements. s4. In the inference phase, a multi-scale priority scheduling strategy based on access frequency, inter-layer dependency, and computational overhead is used to load only the slices required for the current computation into the hardware memory. Core weight area slices are dynamically loaded based on real-time attention distribution, and the hardware memory occupied by the slices is released in a timely manner after the slice computation is completed. s5. Assemble the inference results of each slice according to the hierarchical structure of the large model and the feature transfer order of the feature channels, and output the complete inference result of the large model.

2. A self-attention-driven lightweight inference system for large models that implements the method of claim 1, characterized in that, include: The model basic slicing module is used to perform fine-grained basic slicing of the weights of large models by layer, functional module, and feature channel, and to establish a slice index system that includes inter-layer dependencies between slices, access frequency features, and computational cost features. The offline preprocessing module is used to complete model loading, weight format conversion, basic segmentation and slice index construction in the host physical memory before inference, without occupying hardware video memory. The self-attention slicing strategy planning module is used to obtain the self-attention distribution values ​​of the weights of each layer of Transformer through lightweight forward computation, divide the core / non-core weight regions and generate a slice scheduling planning table, and supports flexible adjustment of attention weight threshold, core weight region slice granularity and non-core weight region merging ratio. The multi-scale video memory scheduling module is used to implement on-demand loading of slices, dynamic loading of core weight area slices, and instant release of hardware video memory based on multi-scale priority scheduling strategies based on access frequency, inter-layer dependency, and computational overhead. The inference execution and result assembly module is used to perform slice inference calculations, assemble the inference results of each slice according to the hierarchical structure of the large model and the feature transfer order of the feature channels, and output the complete inference result of the large model. The modules work together to enable lightweight inference of large models on hardware with different memory specifications.

3. The self-attention-driven lightweight inference method for large models according to claim 1, characterized in that, The priority of the multi-scale priority scheduling strategy mentioned in step 4) is: high-frequency access slices > inter-layer strong dependency slices > low computational overhead slices.

4. The self-attention-driven lightweight inference method for large models according to claim 1, characterized in that, With a weight retention rate of 60%±5%, the proposed method reduces the memory usage for large model inference by more than 40% compared to the full loading method, and achieves an inference speedup of 1.8 times or more.

5. The self-attention-driven lightweight inference method for large models according to claim 1, characterized in that, When using Cosine similarity as the evaluation standard, the similarity between the inference result and the inference result of the full model can reach 95% or more, and the accuracy loss is basically imperceptible in common engineering application scenarios.

6. The self-attention-driven lightweight inference method for large models according to claim 5, characterized in that, With 8GB of video memory, a large model with 7B parameters based on the Transformer architecture, and an attention weight threshold of 0.7, the cosine similarity between the inference results and the full model inference results is 96.04%.

7. The self-attention-driven lightweight inference method for large models according to claim 1, characterized in that, The proposed method requires virtually no modification to the original structure of the large model and is compatible with existing mainstream large model inference frameworks.

8. The self-attention-driven lightweight inference method for large models according to claim 1, characterized in that, The method is adapted to large models with multiple parameters, such as 7B and 13B, including Transformer, MoE, and dense architectures.

9. The self-attention-driven lightweight inference system for large models according to claim 2, characterized in that, The model basic slicing module, offline preprocessing module, and self-attention slicing strategy planning module run on the host CPU and memory, while the multi-scale memory scheduling module, inference execution and result stitching module run on the hardware memory controller and computing core.

10. The self-attention-driven lightweight inference system for large models according to claim 2, characterized in that, The system can be applied to consumer-grade graphics cards, edge computing terminals, and server hardware, without requiring customized hardware, and can be directly integrated into existing large-scale model inference frameworks.