A method and system for large model compression and end-side deployment based on multi-target driving

Abstract

The application discloses a kind of big model compression and end side deployment method and system based on multi-objective driving, method includes: obtaining baseline big model and vertical field short dialogue data set, complete model initialization and data preprocessing;Using knowledge distillation to baseline big model preliminary compression, obtain first light weight model;Through first multi-objective bayesian optimization, the first light weight model is carried out sparse and vocabulary super parameter optimization, after determining sparse and vocabulary optimal super parameter, the first light weight model is optimized for the first time, and sparse vocabulary optimization model is obtained;Through second multi-objective bayesian optimization, sparse vocabulary optimization model is carried out quantization super parameter optimization, after determining quantization optimal super parameter, sparse vocabulary optimization model is optimized for the second time, and second light weight model is obtained;Second light weight model is compiled to end side chip, and deployment and verification are completed.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and in particular relates to a method and system for large model compression and edge deployment based on multi-objective driven approach. Background Technology

[0002] Large language models, as a core technology in the field of artificial intelligence, have achieved large-scale applications in many areas such as intelligent dialogue, code generation, and intelligent analysis due to their powerful semantic understanding, generation, and reasoning capabilities. Examples include large models such as GPT, Deepseek, Kimi, and Qianwen. These models are generally built on the Transformer architecture, relying on complex matrix storage and computation mechanisms to achieve their core functions, with parameter scales often reaching hundreds of millions or even trillions. This characteristic places extremely high demands on computing and storage chips for the efficient operation of large models, making them a core hardware dependency for their deployment.

[0003] However, edge chips (such as embedded devices and edge chips) are limited by their physical size and hardware architecture, resulting in significant bottlenecks in computing power and storage capacity, making it difficult to support the direct deployment of native large models. Taking the current mainstream small-parameter large language model as an example, even with a basic architecture of 0.6B parameters, its storage volume is still close to 800MB at FP16 precision, far exceeding the storage capacity limit of edge devices. The 20MB-level storage capacity of edge chips is even more insufficient to meet the storage requirements of native large models, which directly limits the penetration of large models into edge scenarios.

[0004] To overcome the hardware limitations of edge deployment and achieve lightweight and edge-based large models, the industry widely adopts sparsity compression techniques such as model pruning and model quantization. Model pruning removes unimportant redundant parameters from the model, constructing a sparse and compact model structure to achieve compression. Model quantization uses low-precision data types (such as INT4 and INT2) to represent the model's activation values ​​and weights, reducing computational complexity and storage overhead. However, practice shows that a single compression method has significant limitations, making it difficult to balance compression ratio and inference accuracy. Either over-compression leads to a significant drop in model accuracy, or accuracy is preserved but the compression requirements for edge storage are not met.

[0005] Furthermore, existing large-model lightweighting technologies mostly employ single-objective optimization strategies, iteratively optimizing only one objective—compression rate or inference accuracy—failing to simultaneously find Pareto optimal solutions, making them highly susceptible to getting trapped in local optima. This makes it impossible to achieve a balanced control between model inference accuracy and storage volume, failing to meet the ultra-lightweight deployment requirements of 20MB-level storage on edge chips. It also leads to severe technical bottlenecks in the large-scale deployment of large models in vertical edge scenarios such as industry and healthcare, hindering the widespread application and industrial penetration of large-model technology from the cloud to the edge.

[0006] Therefore, there is an urgent need for a large model lightweighting technology that can comprehensively balance model compression rate and inference accuracy and adapt to the limitations of edge hardware, so as to achieve efficient deployment of large models on edge chips. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for large model compression and edge deployment based on multi-objective driven approach, so as to solve the above-mentioned problems.

[0008] To achieve the above objectives, in a first aspect of the present invention, a method for large model compression and edge deployment based on multi-objective driven technology is provided, comprising the following steps: Acquire a baseline large model and a short dialogue dataset from a vertical domain, and complete model initialization and data preprocessing; The baseline large model was initially compressed using knowledge distillation to obtain the first lightweight model; Through the first multi-objective Bayesian optimization, the first lightweight model is optimized for sparsity and vocabulary hyperparameters. After determining the optimal hyperparameters for sparsity and vocabulary, the first lightweight model is optimized for the first time to obtain the sparse vocabulary optimized model. Through a second multi-objective Bayesian optimization, the sparse vocabulary optimization model is optimized by quantization hyperparameter search. After determining the optimal quantization hyperparameter, the sparse vocabulary optimization model is optimized a second time to obtain the second lightweight model. The second lightweight model was compiled and adapted to the edge chip to complete the deployment and verification.

[0009] Furthermore, in the aforementioned method for large model compression and edge deployment based on multi-objective driven approach, the baseline large model is an open-source lightweight dialogue large model based on the Transformer architecture, and the data preprocessing includes performing at least one of the following processes on the vertical domain short dialogue dataset: cleaning, desensitization, word segmentation, normalization, or deduplication.

[0010] Furthermore, in the aforementioned method for large model compression and edge deployment based on multi-objective driving, the knowledge distillation method uses a baseline large model with frozen weights as the teacher model to construct a lightweight student model with a parameter count of 0.1B. A multi-dimensional loss function consisting of feature loss, attention loss, and distribution loss is used to complete the core knowledge transfer, thereby compressing the size of the baseline large model to below 200MB.

[0011] Furthermore, in the aforementioned large model compression and edge deployment method based on multi-objective driving, the first multi-objective Bayesian optimization takes model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives, constructs a weighted utility function for iterative optimization, and terminates the optimization and outputs the optimal hyperparameters for sparsity and vocabulary when the number of iterations does not exceed the maximum number of iterations and the weighted utility function meets the convergence condition. The first optimization steps include sparse optimization with weight and activation coordination, adaptive vocabulary optimization, and first LORA fine-tuning.

[0012] Furthermore, in the aforementioned method for large model compression and edge deployment based on multi-objective driving, the optimal hyperparameters of sparsity and vocabulary include sparsity threshold, weight sparsity ratio, activation sparsity ratio, and optimal vocabulary size. The optimal vocabulary size is determined by multi-size gradient sampling of the initial vocabulary and then iteratively optimizing it through multi-objective Bayesian optimization.

[0013] Furthermore, in the aforementioned large model compression and edge deployment method based on multi-objective driving, the second multi-objective Bayesian optimization takes quantization error loss, model storage volume, and accuracy loss as optimization objectives, constructs a weighted utility function for iterative optimization, and adaptively optimizes to determine the optimal quantization step size.

[0014] Furthermore, in the aforementioned multi-objective driven large model compression and edge deployment method, the second optimization step includes hierarchical mixed precision quantization and a second LORA fine-tuning process; the hierarchical mixed precision quantization only performs differentiated bit allocation for model weights, using 4-bit quantization for the weights of the core feature layer and 2-bit quantization for the weights of the redundant feature layer; the core feature layer includes an attention layer and an encoding layer, and the redundant feature layer includes a feedforward layer and an embedding layer.

[0015] Furthermore, in the aforementioned large model compression and edge deployment method based on multi-objective driving, the size of the second lightweight model is strictly controlled within 20MB. The compilation adaptation includes quantization operator adaptation, memory layout optimization, inference engine encapsulation, and static constant solidification. The deployment verification includes verifying storage volume, inference latency, power consumption, domain accuracy, and long-term operational stability.

[0016] In a second aspect of the present invention, a large model compression and edge deployment system based on multi-objective driving is also proposed, including a data processing module, a model initial compression module, a sparse vocabulary optimization module, a quantization optimization module, and an edge deployment verification module. The data processing module is used to acquire the baseline large model and the vertical domain short dialogue dataset, and to complete the model initialization and data preprocessing. The initial model compression module uses knowledge distillation to compress the baseline large model to less than 200MB to obtain the first lightweight model. The sparse word list optimization module optimizes the sparse and word list hyperparameters of the first lightweight model through the first multi-objective Bayesian optimization. After determining the optimal sparse and word list hyperparameters, the first optimization is performed to obtain the sparse word list optimized model. The quantization optimization module optimizes the sparse vocabulary optimization model by performing a second multi-objective Bayesian optimization to find the quantization hyperparameters. After determining the optimal quantization hyperparameters, the second optimization is performed to obtain a second lightweight model with a size controlled within 20MB. The edge deployment verification module is used to compile and adapt the second lightweight model and then deploy it to the edge chip to complete the deployment verification.

[0017] Furthermore, in the aforementioned large model compression and edge deployment system based on multi-objective driving, in the sparse vocabulary optimization module, the first multi-objective Bayesian optimization constructs a weighted utility function for iterative optimization with model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives. A maximum number of iterations is set, and if the weighted utility function value reaches the convergence threshold during the iteration process, the optimization is terminated early. The first optimization includes sparse optimization of the first lightweight model with weight and activation coordination, adaptive vocabulary optimization, and the first LORA fine-tuning. In the quantization optimization module, the second multi-objective Bayesian optimization constructs a weighted utility function with quantization error loss, model storage volume and accuracy loss as optimization objectives to iteratively find the best value. The second optimization includes performing hierarchical mixed precision quantization on the sparse vocabulary optimization model and performing a second LORA fine-tuning process.

[0018] Compared with the prior art, the present invention has at least the following technical effects: This invention abandons the traditional single-objective optimization strategy and adopts a multi-stage progressive lightweight approach. Through knowledge distillation, first optimization, and second optimization, it gradually decomposes the compression pressure, achieving a higher model compression ratio while effectively ensuring the core inference accuracy of the model. Simultaneously, through two multi-objective Bayesian optimizations, it optimizes hyperparameters for sparsity, vocabulary, and quantization, respectively, simultaneously finding Pareto optimal solutions. This effectively avoids local optima and overcomes the predicament of single-objective optimization, which is prone to getting trapped in local optima and failing to balance accuracy and storage volume. It achieves precise and balanced control of model inference accuracy and storage volume. Furthermore, LoRA fine-tuning is performed at different optimization stages to specifically repair accuracy losses at each stage, achieving high-precision preservation of short dialogues in vertical domains with extremely low training costs. This further solves the industry pain point of difficulty in balancing compression ratio and inference accuracy. Through these optimizations, this invention enables the efficient deployment of large models on edge chips, effectively promoting the penetration and large-scale application of large models in vertical edge scenarios such as industry and healthcare. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for large model compression and edge deployment based on multi-objective driven technology in one embodiment of the present invention; Figure 2 This is a flowchart of multi-objective Bayesian optimization in another embodiment of the present invention; Figure 3 This is a flowchart of sparse and vocabulary hyperparameter optimization and model optimization in one embodiment of the present invention; Figure 4 This is a flowchart of quantization hyperparameter optimization and model optimization in one embodiment of the present invention; Figure 5 This is a structural diagram of a large model compression and edge deployment system based on multi-objective driving in one embodiment of the present invention. Detailed Implementation

[0020] The following will describe in more detail a method and system for large model compression and edge deployment based on multi-objective driving according to the present invention, with reference to the schematic diagrams, which illustrate preferred embodiments of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being of broad knowledge to those skilled in the art and is not intended to limit the present invention.

[0021] For clarity, not all features of the actual embodiments are described. In the following description, well-known functions and structures are not detailed in detail, as they would obscure the invention with unnecessary detail. It should be understood that in the development of any actual embodiment, numerous implementation details must be made to achieve the developer's specific objectives, such as changes from one embodiment to another according to limitations related to the system or business. Furthermore, it should be understood that such development work may be complex and time-consuming, but is merely routine work for those skilled in the art.

[0022] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0023] Based on the teachings of this specification, those skilled in the art can form new technical solutions through cross-combination of different implementation methods without creating technical contradictions. Such variations should all be considered to fall within the protection scope of this invention.

[0024] Example 1 This embodiment discloses a large model compression and edge deployment method based on multi-objective driven principles. Designed for short dialogue scenarios in vertical domains, this method solves the problem that single compression techniques struggle to balance compression rate and inference accuracy, achieving extreme lightweighting of large models and efficient edge deployment. (Refer to...) Figure 1 The specific steps of this method are as follows: S1: Obtain the baseline large model and the vertical domain short dialogue dataset, and complete the model initialization and data preprocessing; S2: The baseline large model is initially compressed using knowledge distillation to obtain the first lightweight model; S3: Through the first multi-objective Bayesian optimization, the first lightweight model is optimized for sparsity and vocabulary hyperparameters. After determining the optimal hyperparameters for sparsity and vocabulary, the first lightweight model is optimized for the first time to obtain the sparse vocabulary optimized model. S4: Through the second multi-objective Bayesian optimization, the sparse vocabulary optimization model is optimized by quantization hyperparameter search. After determining the optimal quantization hyperparameter, the sparse vocabulary optimization model is optimized a second time to obtain the second lightweight model. S5: Compile and adapt the second lightweight model to the edge chip to complete deployment verification.

[0025] For step S1, in the baseline large model acquisition and initialization stage, a standardized debugging environment was built based on NVIDIA's regular GPU cluster. The baseline large model selected was an open-source lightweight dialogue large model based on the Transformer architecture, such as Doubao, Deepseek, and Qianwen. Based on the built standardized environment, the entire process of pulling the open-source baseline model, environment adaptation, weight initialization, and inference debugging was completed. The focus was on troubleshooting basic operational issues such as operator incompatibility and weight anomalies to ensure that the baseline large model can run stably and normally.

[0026] The vertical domain short dialogue dataset in this embodiment focuses on vertical domain application scenarios with single-turn or short-turn interactions. It integrates publicly available and compliant domain-related datasets with high-quality manually annotated data to form the dataset. The data preprocessing includes performing at least one of the following processes on the vertical domain short dialogue dataset: cleaning, de-identification, word segmentation, normalization, or deduplication.

[0027] Among them, the cleaning operation is used to remove invalid questions and answers, duplicate sentences, garbled text and other worthless data; the de-identification operation is used to mask private information (such as mobile phone numbers, names, accounts, etc.) in the data to ensure the security and compliance of data use; the word segmentation operation is used to segment the dialogue text into tokens that the model can recognize based on domain-specific word segmentation tools; the normalization operation is used to unify the text format (such as capitalization, punctuation marks and sentence structure); and the deduplication operation is used to further remove duplicate dialogue samples to avoid model overfitting.

[0028] After completing the above data preprocessing, the vertical domain short dialogue dataset is strictly divided into training, validation, and test sets in an 8:1:1 ratio. This reasonable division ensures the consistency of the dataset distribution. The training set is used for model knowledge distillation and LoRA fine-tuning in subsequent steps S2-S4. The validation set is used to verify the performance of the hyperparameter optimization in steps S3 and S4 (multi-objective Bayesian optimization). The test set is used to verify the performance of the second lightweight model on the edge in the final step S5. The standardized dataset also provides a reliable data source for word frequency statistics and initial word screening in subsequent adaptive word optimization.

[0029] Step S1 is a standard pre-process for model compression. It does not involve changes in accuracy or volume, and only completes the preparation of the environment and data to ensure that subsequent experiments are reproducible and the data is unbiased.

[0030] For step S2, this embodiment selects the open-source Qwen-0.6B-Chat model as the baseline large model. The knowledge distillation method adopted is to use the baseline large model with frozen weights as the teacher model and construct a lightweight student model with a parameter size of 0.1B. The core knowledge transfer of the teacher model is completed through a multi-dimensional loss function composed of feature loss, attention loss and distribution loss, and finally the size of the baseline large model is compressed to less than 200MB.

[0031] This embodiment employs a teacher-student model distillation architecture. First, the weights of the teacher model are frozen to ensure that its core knowledge and reasoning capabilities are not compromised. Then, a simplified design based on the Transformer architecture is implemented to construct a lightweight student model with only 0.1B parameters. Specifically, the lightweight student model is built by retaining the core network structure of the teacher model (embedding layer, attention layer, encoding layer, and feedforward layer) and reducing the number of encoder layers, attention heads, and hidden layer dimensions through gradient reduction. This precisely compresses the model's parameter count from 0.6B in the original teacher model to 0.1B, ensuring that the student model's network structure can adapt to the teacher model's knowledge transfer logic while significantly reducing the number of model parameters.

[0032] To achieve efficient transfer of core knowledge from the teacher model, a multi-dimensional loss function consisting of feature loss, attention loss, and distribution loss is employed. Feature loss ensures that the feature vectors output by each layer of the student model remain consistent with those of the teacher model, guaranteeing the student model's core feature extraction capabilities. Attention loss matches the attention weight distribution between the student and teacher models, ensuring the student model possesses consistent contextual association and semantic reasoning capabilities. Distribution loss ensures that the output probability distribution of the student model closely approximates that of the teacher model, guaranteeing the rationality and accuracy of short dialogue generation within the vertical domain. Through the collaborative constraints of this multi-dimensional loss function, the student model comprehensively learns and transfers the domain knowledge and reasoning capabilities of the teacher model.

[0033] Using the industrial fault question-and-answer short dialogue training set preprocessed in step S1 as training data, and the validation set preprocessed in the same batch as the basis for distillation effect verification, knowledge distillation training was performed. During the training process, the parameters of the student model were continuously optimized through backpropagation, and the multidimensional loss function value was continuously minimized until the model loss converged and the domain question-and-answer accuracy on the validation set reached the preset standard, at which point the training was terminated.

[0034] In this embodiment, after actual testing, the storage volume of the student model was reduced from 800MB to 152MB after knowledge distillation and pre-compression, which is far below the compression target of 200MB. Moreover, the domain question-answering accuracy in the industrial fault question-answering scenario is 96.8% preserved, with no obvious semantic loss. The core fault question-answering capabilities of the model for the vertical domain are fully preserved, and the accuracy loss is controlled within a reasonable range acceptable to engineering. The first round of lightweighting and burden reduction of the large model has been successfully completed. The 0.1B student model after training is the first lightweight model.

[0035] For step S3, refer to the process. Figure 3 In this step, the optimization of sparsity and vocabulary hyperparameters employs a multi-objective Bayesian optimization algorithm. This algorithm takes the first lightweight model as input and uses model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives, constructing a weighted utility function for iterative optimization. Specifically, to achieve a synergistic balance among multiple objectives, this embodiment uses a weighted summation method to transform multiple optimization objectives into a solvable weighted utility function, as follows: in, For model accuracy loss, For model sparsity, For runtime memory usage overhead, For vocabulary redundancy, , , , For the weighting coefficients, satisfying The coefficient can be dynamically adjusted according to the actual needs of the scenario, so as to achieve priority control of different optimization goals.

[0036] In this embodiment, the general flowchart of the multi-objective Bayesian optimization algorithm is as follows: Figure 2 As shown, starting with a random initial set of hyperparameters, the relationship between the hyperparameters and the target is fitted using a Gaussian process surrogate model. Candidate parameters are iteratively screened and evaluated. When the number of iterations does not exceed the maximum number of iterations (300 in this embodiment), and if the number of iterations does not exceed the preset maximum number of iterations (e.g., 300), and the change in the weighted utility function value reaches the convergence threshold (≤1×10⁻⁶), the process continues. -3 If the condition is met, the optimization process will be terminated early, and the optimal hyperparameters for the current stage will be output.

[0037] Through the iterative optimization of the first multi-objective Bayesian optimization, the optimal hyperparameters for sparsity and vocabulary are finally determined. These optimal hyperparameters include the sparsity threshold, weight sparsity ratio, activation sparsity ratio, and optimal vocabulary size. Specifically, the optimal vocabulary size is determined through multi-size gradient sampling of the initial vocabulary and iterative optimization using multi-objective Bayesian optimization. First, based on the vertical domain short dialogue dataset, a full-domain word frequency statistical analysis is performed to remove stop words, low-frequency uncommon tokens, and generic meaningless tokens, completing the initial screening of the core vocabulary of the target domain and obtaining the initial vocabulary. Then, multi-size gradient sampling is performed on the initial vocabulary to generate candidate vocabularys of different sizes (in this embodiment, 6000, 8000, 10000, and 12000 are used as examples). The vocabulary size is included as an optimizable hyperparameter in the Bayesian optimization candidate set, and combined with other hyperparameters to participate in iterative optimization, ultimately determining the optimal vocabulary size suitable for the vertical domain short dialogue scenario, achieving precise vocabulary simplification.

[0038] Based on the optimal hyperparameters determined above, the first lightweight model undergoes its first optimization. This first optimization includes weight and activation co-optimization for sparsity, vocabulary adaptive optimization, and a first LORA fine-tuning process. Firstly, weight and activation co-optimization for unstructured sparsity overcomes the limitation of single-weight sparsity, which only compresses model size but cannot reduce inference computation, achieving dual compression of both static parameters and dynamic computation. In this embodiment, the overall sparsity rate is set to 76%, which conforms to the safe range of classic sparse algorithms, effectively preventing a sharp drop in model accuracy. This optimization process simultaneously removes invalid weights and redundant activation neurons from the model. Invalid weights refer to static parameters with values ​​below the sparsity threshold that do not substantially contribute to model feature calculation and inference results, while redundant activation neurons refer to dynamic computation units whose output values ​​are close to 0 during inference and cannot effectively extract features. In practice, the weights and activated neurons are used as hyperparameters for Bayesian optimization. A 0 or 1 mask is set for each weight and each activated neuron (1 indicates retention or activation, and 0 indicates removal or deactivation). The mask distribution is determined by the optimization algorithm to achieve unstructured sparsity. Under the premise of satisfying a sparsity rate of 76%, the core reasoning ability of the model is preserved to the maximum extent.

[0039] Furthermore, while performing weight and activation collaborative sparse optimization, this embodiment performs adaptive vocabulary optimization on the first lightweight model based on the optimal vocabulary size of 8000 determined by Bayesian optimization. Specifically, the optimal vocabulary size is loaded synchronously, redundant tokens in the vocabulary are pruned, and the word embedding layer of the model is compressed with low-rank alignment. The embedding layer parameters are remapped and fine-tuned to ensure that the vocabulary size and embedding layer parameters are accurately matched, avoiding semantic mapping deviations caused by vocabulary simplification, ensuring the model's ability to recognize and understand domain vocabulary, and further reducing the model's storage footprint by compressing the model size. Actual tests show that after weight and activation collaborative sparse optimization and adaptive vocabulary optimization, the model size is reduced to 36MB.

[0040] Furthermore, during the weight activation collaborative sparse optimization and vocabulary adaptive optimization processes, slight model accuracy drift is inevitable. To quickly and efficiently repair the accuracy loss without increasing the model size, LoRA fine-tuning technology is used to perform the first LoRA fine-tuning on the optimized model. During the fine-tuning process, the weights of the model's backbone network are frozen, and low-rank adapters are added only to core feature layers such as the attention layer and the encoding layer. Only a small number of low-rank adaptation parameters are trained. In this embodiment, the low-rank number r=8 and the learning rate=8e-5 are set for this fine-tuning, which significantly reduces the computational and time costs of fine-tuning. The fine-tuning data uses the vertical domain short dialogue training set preprocessed in step S1, and the validation set is used as the basis for effect verification. Fine-tuning is performed until the model accuracy converges.

[0041] Real-world testing shows that after the first LoRA fine-tuning, the model accuracy successfully recovered to 95.2%, with a cumulative accuracy loss of only 4.8%, strictly controlled within 5%. The core inference accuracy was effectively restored, and a sparse vocabulary optimized model that combines lightweight design with high inference accuracy was finally obtained.

[0042] For step S4, the quantization hyperparameter optimization in this step adopts the same multi-objective Bayesian optimization process as in S3, referring to... Figure 4 Taking a sparse vocabulary optimization model as input, and quantization error loss, model storage volume, and accuracy loss as optimization objectives, a weighted utility function is constructed for iterative optimization, as follows: in, To account for quantization error loss, Store the model volume. For the sake of accuracy loss; , , The weighting coefficients are determined through Bayesian adaptive optimization to achieve a dynamic balance of multiple objectives, namely, minimizing the model storage volume while strictly controlling the quantization error loss and accuracy loss to an acceptable range.

[0043] The optimization process focuses on finding the optimal quantization step size. In this embodiment, the search range for the quantization step size is set to 0.005-0.02. After iterative calculation by the algorithm, the optimal quantization step size is determined to be 0.01. This step size not only adapts to the numerical distribution characteristics of each layer of the model but also effectively balances compression efficiency and error control. Through this iterative optimization of multi-objective Bayesian optimization, all optimal quantization hyperparameters are finally determined, including not only the optimal quantization step size but also the quantization bit allocation strategy parameters for each layer, providing parameter basis for the implementation of hierarchical mixed precision quantization.

[0044] Based on the determined optimal hyperparameters, a second optimization is performed on the sparse vocabulary optimization model. This second optimization includes hierarchical mixed precision quantization and a second LORA fine-tuning process. The hierarchical mixed precision quantization in this step is only applied to the model weights. Specifically, during the quantization process, a differentiated bit allocation strategy is implemented based on the importance and redundancy characteristics of different model layers to the core inference accuracy of short dialogues in the vertical domain. The model layers are divided into two categories: core feature layers and redundant feature layers. The core feature layers include attention layers and encoding layers. These layers are the core modules for the model to achieve contextual association, semantic reasoning, and core feature extraction, and are highly sensitive to inference accuracy. Therefore, 4-bit quantization is used to ensure the accuracy of core inference capabilities such as fault assessment and question-and-answer in short dialogue scenarios in the vertical domain. The redundant feature layers include feedforward layers and embedding layers. These layers have high redundancy in short dialogue scenarios and are significantly less sensitive to core inference accuracy than the core feature layers. Therefore, 2-bit quantization is used to achieve extreme compression of the model size.

[0045] It is important to note that the entire quantization process avoids 1-bit quantization schemes, effectively avoiding the hardware adaptation problems and the risk of a sharp drop in accuracy caused by 1-bit quantization.

[0046] Furthermore, layered hybrid precision quantization inevitably introduces a slight loss of accuracy into the model. To compensate for this loss while maintaining the model's extremely lightweight characteristics, this embodiment employs LoRA fine-tuning technology to perform a second LoRA fine-tuning on the quantized model. This fine-tuning strategy remains consistent with the first LoRA fine-tuning, adding low-rank adapters only to core feature layers such as the attention and encoding layers. Only a small number of adapter parameters are trained, without altering the model's main weights, achieving efficient accuracy restoration while minimizing computational and time costs. The fine-tuning process uses the vertical domain short dialogue training set preprocessed in step S1 as training data and the validation set as the basis for performance verification, continuing training until the model accuracy converges, at which point fine-tuning stops.

[0047] Real-world testing shows that after two LoRA fine-tuning steps, the model's accuracy loss was effectively compensated, resulting in the second lightweight model. This model has a solidified size of 19.2MB, strictly controlled within 20MB. In short dialogue scenarios in vertical domains such as industrial fault question answering, the inference accuracy reaches 95.4%, with an accuracy loss of only 4.6% compared to the baseline large model, stably controlled within 5%. Moreover, this accuracy data conforms to the experimental pattern of large model compression in similar vertical domains, achieving the dual goals of extreme lightweighting and high-precision inference.

[0048] For step S5, for the target embedded edge NPU chip, a general edge AI chip architecture, the second lightweight model is compiled and adapted. Specifically, this includes quantization operator adaptation, memory layout optimization, inference engine encapsulation, and static constant solidification. Through standardized compilation operations, the model deployment volume is further compressed and inference latency is reduced, allowing the model to be deeply adapted to the hardware architecture, computing power, storage resources, and software environment of the edge chip.

[0049] Among these features, quantization operator adaptation precisely matches the model's 4-bit or 2-bit layered mixed-precision quantization operators with the native quantization operators of the edge NPU chip, ensuring efficient execution of quantization calculations on the edge chip. Memory layout optimization optimizes the storage distribution of model parameters and intermediate inference calculation results based on the capacity and access speed characteristics of the edge chip's on-chip and off-chip memory, reducing memory access overhead and improving inference efficiency. Inference engine encapsulation encapsulates the model into a lightweight inference engine that can be directly called by the edge chip, simplifying the model's calling process in edge applications and adapting to the application development needs of edge devices. Static constant solidification solidifies static parameters and constants in the model, improving model startup speed and overall inference efficiency.

[0050] Building upon the completion of basic compilation and adaptation, this step also enables dynamic deployment optimization strategies. Addressing common issues such as computing power fluctuations and limited memory resources on edge chips, this approach utilizes real-time computing power scheduling, hot and cold memory separation, high-frequency fault term caching, and low-load offline precision fine-tuning to adapt the model to the complex operating environment on the edge. Under limited computing power and memory constraints, this further compresses the deployment size and reduces inference latency.

[0051] After compilation and adaptation are completed, the model is deployed to the target edge NPU chip device and installed and debugged. Then, the vertical domain short dialogue test set preprocessed in step S1 is used to conduct full-dimensional real-world testing and verification in the short dialogue scenario. Each indicator is tested, including model storage volume, inference latency, power consumption, domain accuracy, and long-term running stability, to ensure that all indicators of the model meet the hard constraints of offline operation on the edge.

[0052] The actual test results show that the optimized second lightweight model in this embodiment occupies 19.2MB of storage on the edge chip, which meets the size requirement of less than 20MB. The peak memory usage is 59MB, the average latency of a single round of inference is 132ms, and the longest latency is 147ms, which meets the latency requirement of ≤150ms for real-time interaction on the edge. At the same time, the model meets the preset standards in terms of power consumption, domain inference accuracy, and stability in processing short dialogue requests continuously for a long time, without any problems such as stuttering, crashes, or distortion of inference results. If all the measured verification indicators of the model meet the preset edge operation requirements, the entire large model compression and edge deployment process is completed; if some indicators do not meet the standards, the corresponding steps are returned for hyperparameter adjustment and model re-optimization until all indicators meet the operation and business requirements of the edge device, ensuring the effective deployment of the large model on the edge.

[0053] In summary, this embodiment abandons the traditional single-objective optimization strategy and adopts a multi-stage progressive lightweight approach. It gradually decomposes the compression pressure through knowledge distillation, collaborative sparse optimization, vocabulary adaptive optimization, and hierarchical hybrid precision quantization, achieving both a higher model compression ratio and effectively ensuring the core inference accuracy of the model. Simultaneously, this embodiment employs two multi-objective Bayesian optimizations to optimize hyperparameters for the sparsity, vocabulary, and quantization stages respectively, simultaneously solving for Pareto optimal solutions. This effectively avoids local optima problems and overcomes the predicament of single-objective optimization, which is prone to getting trapped in local optima and failing to balance accuracy and storage volume. It achieves precise balance between model inference accuracy and storage volume. Furthermore, LoRA fine-tuning is performed at different lightweighting stages to specifically repair accuracy losses at each stage, achieving high-precision preservation of short dialogues in vertical domains with extremely low training costs. This further addresses the industry pain point of difficulty in balancing compression ratio and inference accuracy. Through these optimizations, this embodiment enables efficient deployment of large models on edge chips, effectively promoting the penetration and large-scale application of large models in vertical edge scenarios such as industry and healthcare.

[0054] Example 2 Reference Figure 5 This embodiment discloses a large model compression and edge deployment system based on multi-objective driving. The system can achieve extreme lightweighting of large models and efficient adaptation and deployment of edge chips while ensuring the core inference accuracy of short dialogues in the vertical domain of the model. The system includes a data processing module, a model initial compression module, a sparse vocabulary optimization module, a quantization optimization module, and an edge deployment verification module.

[0055] The data processing module acquires the baseline large-scale model and the short dialogue dataset for the vertical domain, completing model initialization and data preprocessing. This module selects an open-source lightweight dialogue model based on the Transformer architecture as the baseline large-scale model, performing initialization operations such as loading model parameters, verifying network structure, and setting up the inference environment to ensure the baseline large-scale model runs correctly. Simultaneously, it collects the original short dialogue dataset for the target vertical domain, dividing it into training, validation, and test sets in an 8:1:1 ratio. Data cleaning, anonymization, word segmentation, normalization, and deduplication preprocessing operations are then performed sequentially to remove invalid data, protect privacy information, and unify data formats. Finally, a standardized baseline large-scale model and a high-quality short dialogue dataset for the vertical domain are output.

[0056] The initial model compression module employs knowledge distillation to compress the baseline large model to below 200MB, resulting in the first lightweight model. This module uses the baseline large model with frozen weights as the teacher model and constructs a lightweight student model with 0.1B parameters based on the Transformer architecture. Simultaneously, through a multi-dimensional loss function composed of feature loss, attention loss, and distribution loss, and using the vertical domain short dialogue training set output by the data processing module as training data and the validation set as the performance verification basis, the student model parameters are optimized through backpropagation. This achieves a comprehensive transfer of the core dialogue knowledge and reasoning ability of the teacher model to the student model, ultimately resulting in the first lightweight model that retains the core capabilities of the baseline large model while meeting the requirement of being below 200MB in size.

[0057] The sparse vocabulary optimization module performs a first multi-objective Bayesian optimization to optimize the sparsity and vocabulary hyperparameters of the first lightweight model. After determining the optimal sparse and vocabulary hyperparameters, the first optimization is performed to obtain the sparse vocabulary optimized model. The first optimization includes weight and activation co-optimization of the first lightweight model, adaptive vocabulary optimization, and the first LORA fine-tuning. This module uses model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives. It constructs a weighted utility function and performs the first multi-objective Bayesian optimization, setting the maximum number of iterations to 300. If the weighted utility function value reaches the convergence threshold during iteration, the optimization is terminated early. The optimal sparse and vocabulary hyperparameters are determined through iterative optimization. The optimal sparse and vocabulary hyperparameters include the sparsity threshold, weight sparsity ratio, activation sparsity ratio, and optimal vocabulary size. Subsequently, based on the optimal hyperparameters, weight and activation co-unstructured sparse optimization and vocabulary adaptive optimization were simultaneously performed on the first lightweight model. Invalid weights and redundant activation neurons were removed from the model, the vocabulary was compressed, and the embedding layer parameters were remapped. Then, to address the slight accuracy loss during optimization, LoRA fine-tuning was performed for the first time, training only the low-rank adapter parameters of the core layers to quickly repair the model's accuracy. The final output is a lightweight and high-precision sparse vocabulary optimized model.

[0058] The quantization optimization module optimizes the sparse vocabulary optimization model through a second multi-objective Bayesian optimization to find the quantization hyperparameters. After determining the optimal quantization hyperparameters, a second optimization is performed to obtain a second lightweight model with a size controlled within 20MB. The second optimization includes hierarchical mixed precision quantization of the sparse vocabulary optimization model and a second LORA fine-tuning process. This module uses quantization error loss, model storage volume, and accuracy loss as optimization objectives, constructs a weighted utility function to carry out the second multi-objective Bayesian optimization, adaptively optimizes within the value range of 0.005-0.02 to determine the optimal quantization step size, and iteratively obtains the optimal quantization hyperparameters such as the quantization bits of each layer and the quantization calibration method. Based on the optimal hyperparameters, hierarchical mixed precision quantization is performed on the sparse vocabulary optimization model, targeting only the model weights. The weights of core feature layers such as the attention layer and the encoding layer are quantized with 4 bits, while the weights of redundant feature layers such as the feedforward layer and the embedding layer are quantized with 2 bits. This achieves extreme size compression while avoiding hardware adaptation problems and the risk of a sudden drop in accuracy. To address the slight accuracy loss caused by the quantization operation, a second LoRA fine-tuning was performed to repair the accuracy without increasing the model size, ultimately outputting a second lightweight model with a size of less than 20MB.

[0059] The edge deployment verification module is used to compile and adapt the second lightweight model before deploying it to the edge chip, completing deployment verification. This module performs compilation and adaptation operations on the second lightweight model, including quantization operator adaptation, memory layout optimization, inference engine encapsulation, and static constant fixing, tailored to the hardware architecture, computing power, and storage resources of the target edge chip (e.g., an embedded edge NPU chip). Simultaneously, it enables dynamic deployment optimization strategies to adapt to edge computing power and memory constraints. Further, the compiled and adapted model is deployed to the target edge chip device, completing model installation, debugging, and runtime environment configuration. Finally, using the vertical domain short dialogue test set output by the data processing module as verification data, the deployed model is verified in multiple dimensions, with each indicator, including storage volume, inference latency, power consumption, and stability, being tested. If all indicators meet the preset requirements for edge operation, the entire large model compression and edge deployment process is completed. If some indicators fail to meet the requirements, the results are fed back to the corresponding optimization module for hyperparameter adjustment and model re-optimization until all indicators meet the operational and business requirements of the edge device, ensuring the efficient and stable operation of the large model on the edge.

[0060] In summary, this system achieves precise hyperparameter optimization through two multi-objective Bayesian optimizations in the sparse vocabulary optimization module and the quantization optimization module, respectively, ensuring the coordinated balance of multiple objectives during model optimization. A multi-level lightweight strategy gradually compresses the model from the baseline state to an extremely lightweight state of less than 20MB. Two LoRA fine-tuning operations effectively compensate for the accuracy loss in each optimization stage. Finally, the system's edge-side performance is verified through an on-device deployment and validation module. This system solves the problems of large model size, high computational cost, and difficulty in adapting to edge devices associated with traditional large models. It is suitable for the edge-side short dialogue interaction needs of various vertical fields and has good scenario adaptability.

[0061] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for large model compression and edge deployment based on multi-objective driven approach, characterized in that, Includes the following steps: Acquire a baseline large model and a short dialogue dataset from a vertical domain, and complete model initialization and data preprocessing; The baseline large model was initially compressed using knowledge distillation to obtain the first lightweight model; Through the first multi-objective Bayesian optimization, the first lightweight model is optimized for sparsity and vocabulary hyperparameters. After determining the optimal hyperparameters for sparsity and vocabulary, the first lightweight model is optimized for the first time to obtain the sparse vocabulary optimized model. The first optimization includes: setting a 0 or 1 mask for each weight and each activated neuron, determining the mask distribution through multi-objective Bayesian optimization, and performing unstructured sparse optimization in conjunction with weights and activations; constructing a candidate set by multi-size gradient sampling of the initial vocabulary, incorporating the vocabulary size into the first multi-objective Bayesian joint optimization, and after determining the optimal vocabulary size, performing low-rank alignment compression and remapping the embedding layer parameters to complete the vocabulary adaptive optimization; freezing the model backbone weights, adding a low-rank adapter only to the core feature layer, and performing the first LoRA fine-tuning to correct the accuracy drift caused by sparsity and vocabulary optimization; Through a second multi-objective Bayesian optimization, the sparse vocabulary optimization model is optimized by quantization hyperparameter search. After determining the optimal quantization hyperparameter, the sparse vocabulary optimization model is optimized a second time to obtain the second lightweight model. The second optimization includes: performing hierarchical mixed precision quantization, implementing differentiated bit allocation only for model weights, avoiding the 1-bit quantization scheme throughout, using 4-bit quantization for the weights of the core feature layer and 2-bit quantization for the weights of the redundant feature layer; freezing the model backbone weights, adding a low-rank adapter only to the core feature layer, and performing a second LoRA fine-tuning to repair the precision loss caused by quantization; The second lightweight model is compiled and adapted to the edge chip to complete deployment and verification. The size of the second lightweight model is strictly controlled within 20MB. The compilation and adaptation include quantization operator adaptation, memory layout optimization, inference engine encapsulation and static constant solidification. The deployment and verification include verifying storage volume, inference latency, power consumption, domain accuracy and long-term running stability.

2. The method according to claim 1, characterized in that, The baseline large model is an open-source lightweight dialogue large model based on the Transformer architecture. The data preprocessing includes performing at least one of the following processes on the short dialogue dataset in the vertical domain: cleaning, desensitization, word segmentation, normalization, or deduplication.

3. The method according to claim 1, characterized in that, The knowledge distillation method uses a baseline large model with frozen weights as the teacher model to construct a lightweight student model with 0.1B parameters. It uses a multidimensional loss function consisting of feature loss, attention loss, and distribution loss to complete the core knowledge transfer, compressing the size of the baseline large model to less than 200MB.

4. The method according to claim 1, characterized in that, The first multi-objective Bayesian optimization takes model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives. It constructs a weighted utility function for iterative optimization. When the number of iterations does not exceed the maximum number of iterations and the weighted utility function meets the convergence condition, the optimization terminates and the optimal hyperparameters for sparsity and vocabulary are output.

5. The method according to claim 1, characterized in that, The optimal hyperparameters for sparsity and vocabulary include the sparsity threshold, weighted sparsity ratio, activation sparsity ratio, and optimal vocabulary size.

6. The method according to claim 1, characterized in that, The second multi-objective Bayesian optimization takes quantization error loss, model storage volume, and accuracy loss as optimization objectives, constructs a weighted utility function for iterative optimization, and adaptively optimizes to determine the optimal quantization step size.

7. The method according to claim 1, characterized in that, The core feature layer includes an attention layer and an encoding layer, and the redundant feature layer includes a feedforward layer and an embedding layer.

8. A large model compression and edge deployment system based on multi-objective driven architecture, characterized in that, It includes a data processing module, a model initial compression module, a sparse vocabulary optimization module, a quantization optimization module, and an edge deployment and verification module; The data processing module is used to acquire the baseline large model and the vertical domain short dialogue dataset, and to complete the model initialization and data preprocessing. The initial model compression module uses knowledge distillation to initially compress the baseline large model to obtain the first lightweight model; The sparse word list optimization module optimizes the sparse and word list hyperparameters of the first lightweight model through the first multi-objective Bayesian optimization. After determining the optimal sparse and word list hyperparameters, the first optimization is performed to obtain the sparse word list optimized model. The first optimization includes: setting a 0 or 1 mask for each weight and each activated neuron, determining the mask distribution through multi-objective Bayesian optimization, and performing unstructured sparse optimization in conjunction with weights and activations; constructing a candidate set by multi-size gradient sampling of the initial vocabulary, incorporating the vocabulary size into the first multi-objective Bayesian joint optimization, and after determining the optimal vocabulary size, performing low-rank alignment compression and remapping the embedding layer parameters to complete the vocabulary adaptive optimization; freezing the model backbone weights, adding a low-rank adapter only to the core feature layer, and performing the first LoRA fine-tuning to correct the accuracy drift caused by sparsity and vocabulary optimization; The quantization optimization module optimizes the sparse vocabulary optimization model by performing a second multi-objective Bayesian optimization to find the quantization hyperparameters. After determining the optimal quantization hyperparameters, the second optimization is performed to obtain a second lightweight model with a size controlled within 20MB. The second optimization includes: performing hierarchical mixed precision quantization, implementing differentiated bit allocation only for model weights, avoiding the 1-bit quantization scheme throughout, using 4-bit quantization for the weights of the core feature layer and 2-bit quantization for the weights of the redundant feature layer; freezing the model backbone weights, adding a low-rank adapter only to the core feature layer, and performing a second LoRA fine-tuning to repair the precision loss caused by quantization; The edge deployment verification module is used to compile and adapt the second lightweight model and then deploy it to the edge chip to complete the deployment verification. The compilation adaptation includes quantization operator adaptation, memory layout optimization, inference engine encapsulation, and static constant solidification. The deployment verification includes verifying storage volume, inference latency, power consumption, domain accuracy, and long-term operational stability.

9. The system according to claim 8, characterized in that, In the sparse vocabulary optimization module, the first multi-objective Bayesian optimization constructs a weighted utility function for iterative optimization with model accuracy loss, model sparsity, runtime memory consumption, and vocabulary redundancy as optimization objectives. A maximum number of iterations is set. If the weighted utility function value reaches the convergence threshold during the iteration process, the optimization is terminated early. In the quantization optimization module, the second multi-objective Bayesian optimization constructs a weighted utility function with quantization error loss, model storage volume, and accuracy loss as optimization objectives to iteratively find the best solution.

Images