Large model lightweight system and method based on adaptive distillation

Abstract

The invention relates to a large-model lightweight system and method based on adaptive distillation, and belongs to the field of artificial intelligence. The system comprises a feature alignment module, a class-level adaptive distillation module, a sample-level dynamic prompt fusion module, a time sequence stability optimization module and a lightweight training and meta-optimization module. The consistency of feature spaces of the teacher model and the student model is realized through feature alignment; generating a dynamic distillation intensity based on the category level statistical information to adjust the category distillation loss; generating a prompt fusion coefficient through a sample level element network according to sample difference, teacher prediction uncertainty and training progress so as to dynamically weight logits distillation, feature distillation and attention distillation; a time recursion smoothing mechanism based on confidence reduces weight fluctuation and improves training stability. And a double-layer optimization structure is adopted, so that the student model keeps task performance close to that of the teacher model while remarkably reducing the parameter scale and the calculation amount, and the method can be applied to edge intelligent equipment and mobile AI scenes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to a lightweight system and method for large models based on adaptive distillation, and more particularly to a lightweight method for large models based on adaptive distillation for application scenarios of edge intelligence and mobile AI systems. Background Technology

[0002] In recent years, with the continuous development of deep learning technology, large-scale neural network models have made breakthroughs in fields such as computer vision, natural language processing (NLP), and speech recognition. Typical large models, such as Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-trained Transformer (GPT), and Vision Transformer (ViT), have achieved near-human performance on various tasks thanks to their massive parameter count and powerful feature representation capabilities. However, these models typically have hundreds of millions or even billions of parameters, placing extremely high demands on computing resources, GPU memory, and inference time, making them difficult to deploy directly on mobile devices, embedded devices, or edge computing scenarios. How to achieve lightweight and efficient deployment of models while maintaining performance has become a core problem to be solved in the practical application of artificial intelligence.

[0003] Currently, mainstream model compression and acceleration techniques mainly include pruning, quantization, low-rank decomposition, and knowledge distillation. Among them, knowledge distillation (KD) is widely used in various deep network compression tasks because it can significantly reduce model size with less performance loss. Its basic idea is to use a high-performance teacher model to guide the training of a smaller student model, enabling the student model to learn the knowledge representations of the teacher model in the output or intermediate layers, thus simplifying the model structure while maintaining high task accuracy. Traditional distillation methods typically use fixed distillation temperature and weight parameters, using the teacher's output distribution as a soft label to guide the student model's learning. However, this "static distillation" approach ignores the differences between different samples or categories during the learning process and cannot adaptively adjust to dynamic changes during the training phase, resulting in limited knowledge transfer effectiveness.

[0004] In practice, fixed-weight distillation presents two main problems. First, different categories of samples have varying degrees of dependence on knowledge transfer. Some simpler categories can be learned sufficiently early on, while complex categories still require strong teacher guidance later. Using a uniform distillation coefficient can easily lead to overfitting to easy-to-learn samples and underfitting to difficult samples. Second, the knowledge of the teacher model is often distributed across multiple representations, including not only the class probability information of the output layer but also the feature structure and attention relationships of intermediate layers. Traditional distillation methods typically only use information from a single layer, failing to fully exploit the teacher's potential expressive power. This makes it difficult for the student model to fully inherit the representational features of the teacher model, especially exhibiting significant performance degradation in complex tasks and cross-domain scenarios.

[0005] To address these challenges, researchers have begun exploring distillation methods based on adaptive mechanisms, introducing meta-learning to dynamically adjust the distillation process. Meta-learning designs a framework for "learning how to learn," enabling the model to automatically optimize distillation weights or learning rates based on training feedback, thereby enhancing the flexibility and generalization of distillation. For example, one approach introduces an adaptive weight generator at the category level, dynamically allocating distillation intensity based on the training loss and uncertainty of each class, allowing the model to pay more attention to difficult categories. Another approach designs a dynamic weight network at the sample level, adjusting the fusion ratio of different prompts in real time based on sample gradients, feature differences, or teacher-student prediction gaps, thus achieving finer-grained knowledge transfer. These adaptive distillation strategies have achieved significant performance improvements in tasks such as image classification, object detection, and semantic segmentation.

[0006] While existing research has made some progress in dynamic distillation, several shortcomings remain. First, most current methods are limited to learning single-layer distillation weights, lacking a unified modeling of the relationship between the class and sample levels. Second, meta-optimization typically relies on additional validation sets for second-order gradient calculation, increasing computational complexity and hindering applications in large-scale models. Third, distillation weights are susceptible to noise interference during training, leading to unstable distillation signals and affecting the final convergence of the student model. Therefore, there is an urgent need for an adaptive distillation method that guarantees performance while possessing scalability, stability, and efficiency for lightweight training of large models.

[0007] In summary, traditional knowledge distillation methods suffer from problems such as rigid fixed weights, insufficient information utilization, and unstable optimization when faced with large-scale models and complex tasks. Summary of the Invention

[0008] In view of this, the purpose of this invention is to address the problems of fixed distillation weights and insufficient utilization of prompt information in existing knowledge distillation methods, and to propose a lightweight method for large models based on adaptive distillation. By constructing a two-layer meta-network structure, distillation weights are dynamically learned at both the class and sample levels to achieve accurate transfer of teacher knowledge; and a temporal stabilization mechanism is introduced to suppress training instability caused by weight fluctuations during the distillation process. Through the above scheme, the student model can significantly reduce the number of parameters and computational complexity while maintaining accuracy, and has stronger generalization and deployability.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A lightweight system for large models based on adaptive distillation, the system comprising: The feature alignment module performs dimensional alignment on the intermediate layer features of the teacher model and the student model. The feature alignment module performs dimensional transformation on the features output by the teacher model through linear mapping or 1×1 convolution to make them consistent with the feature structure of the student model. The class-level adaptive distillation module dynamically generates class distillation intensity based on class-level statistical information, which includes the average loss and prediction uncertainty of the teacher model and student model in each class. The class-level adaptive distillation module outputs class distillation intensity based on the meta-learning network to adjust the class distillation loss. The sample-level dynamic prompt fusion module dynamically assigns distillation weights for different distillation prompts to a single training sample. Based on the prediction difference between the teacher model and the student model, the teacher prediction entropy, and the current training progress, it dynamically generates prompt fusion coefficients. The distillation prompts include logits layer distillation, feature layer distillation, and attention map distillation. The temporal stability optimization module performs stability correction on class weights and sample-level fusion coefficients based on confidence and through a time-progressive smoothing mechanism, so as to reduce optimization instability caused by weight fluctuations in the early training stage. The lightweight training and meta-optimization module employs a two-layer optimization structure to simultaneously train student model parameters and meta-network parameters. The inner layer optimizes the student model to minimize the overall distillation loss, while the outer layer optimizes the meta-network strategy based on validation set feedback, dynamically adjusting distillation weights and cue allocation.

[0010] Furthermore, the feature alignment module introduces a trainable linear mapping matrix. With bias term The feature dimension transformation is calculated and expressed as:

[0011] Among them, the teacher model in the first The output of the layer is The output of the corresponding layer in the student model is ; Using the normalized mean squared error as the constraint function, it is expressed as:

[0012] in, This is the feature dimension of the student model.

[0013] Furthermore, the class-level adaptive distillation module introduces a meta-learning-driven adaptive weight generation mechanism at the class level, and the class-level meta-network... Using the average loss and uncertainty of the teacher and student models for each category as input, the distillation intensity for that category is dynamically output. , represented as:

[0014] in, , Teachers and students are categorized separately. The average loss on This represents the prediction entropy or variance; Based on the distillation intensity of this category, the category distillation loss is calculated and expressed as:

[0015] in, The distillation temperature. For the training sample set corresponding to category c, This represents the softmax function. These are training samples belonging to category c.

[0016] Furthermore, the sample-level dynamic prompting fusion module, in the first... During the next training iteration, for the samples Meta-network input teacher-student prediction difference Teacher predicts entropy and training progress ratio The output prompts the fusion coefficient, which is represented as:

[0017] The overall sample distillation loss is calculated and expressed as:

[0018] in, Indicates the first Distillation loss as indicated by the class This is the training sample set.

[0019] Furthermore, the time-series stability optimization module will adjust the class-level weights. With sample-level fusion coefficient The update is calculated using the following method:

[0020] The confidence level is expressed as: .

[0021] Furthermore, the lightweight training and meta-optimization module sets the inner-layer optimization objective to the student model parameters. Minimizing the overall distillation loss is expressed as:

[0022] in, For cross-entropy loss, These are the weighting coefficients. For category distillation loss, To account for sample distillation losses, For constraint functions.

[0023] Furthermore, the lightweight training and meta-optimization module sets the outer layer optimization to use meta-network parameters. Using a small validation set as the variable, minimize the meta-loss. We use a first-order gradient approximation method to avoid the extra overhead caused by the second-order gradient.

[0024] Furthermore, the system is used to reduce the size of model parameters and computational complexity while maintaining model accuracy, and supports deployment on edge intelligent devices or mobile AI systems.

[0025] A lightweight method for large models based on adaptive distillation includes the following steps: S1. Acquire training data and perform standardized preprocessing. Input the processed samples into the teacher model and student model respectively. The teacher model is a high-performance model that has been trained, and the student model is a lightweight model to be optimized. S2. Obtain intermediate layer features of the teacher model and student model through the feature alignment module; when the feature dimensions are inconsistent, map the intermediate features of the teacher model to the feature space of the student model through linear mapping or 1×1 convolution to achieve semantic consistency and structural alignment. S3. Perform class-level statistical analysis on the training samples through the class-level adaptive distillation module, generate class distillation weights based on the average loss and prediction uncertainty of the teacher model and student model in each class, and adjust the class-level distillation loss based on the weights. S4. Through the sample-level dynamic prompting fusion module, based on the difference between the prediction results of the teacher model and the student model, the prediction entropy of the teacher model and the training progress, a prompting fusion coefficient is generated for the sample. Based on the fusion coefficient, the logits layer distillation, the feature layer distillation and the attention distillation are weighted and fused to form a sample-level comprehensive distillation loss. S5. The class distillation weights and sample-level fusion coefficients are smoothed by time recursion based on confidence through the time-series stability optimization module to reduce the fluctuations of the distillation signal in the early stage of training and improve training stability. S6. Input the dynamically adjusted distillation weights and student model output features into the lightweight training and meta-optimization module, and use a two-layer optimization structure to update the model parameters: the inner layer is used to optimize the student model to minimize the distillation loss, and the outer layer is used to optimize the meta-network parameters based on the validation set feedback to adaptively adjust the weight generation strategy. S7. Output the trained student model.

[0026] The beneficial effects of this invention are as follows: By introducing a meta-learning-driven class-level and sample-level two-layer meta-network adaptive weight generation mechanism, the distillation intensity is dynamically adjusted to avoid insufficient information transfer caused by fixed coefficients, allowing the student model to fully inherit the knowledge expression of the teacher model. A feature alignment module is proposed, consisting of linear mapping and normalization constraints, to solve the mismatch between the feature dimensions and semantic levels of the intermediate layers of the teacher and student models, improving the stability and effectiveness of the distillation signal and providing a unified feature foundation for multi-layer knowledge transfer. A confidence-based temporal smoothing mechanism is adopted to suppress the instability of weight fluctuations in the early stages of training. By updating the teacher's prediction confidence with weights, the distillation signal evolves smoothly in the time dimension, enhancing training stability and convergence reliability. A lightweight training and meta-optimization module is designed, consisting of a two-layer structure of inner layer updating student parameters and outer layer optimizing meta-network strategies. This reduces the overhead of high-order gradients while ensuring computational efficiency. Experiments verify that it has good scalability and versatility, and can be adapted to different network depths and downstream tasks.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall system structure of a large-scale lightweight method based on adaptive distillation according to an embodiment of the present invention. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0031] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0032] This invention provides a lightweight method for large models based on adaptive distillation. By introducing a bi-layer meta-learning structure, dynamic weight control mechanisms are established at both the class and sample layers. Combined with temporal smoothing and first-order approximation optimization strategies, lightweight and efficient training are achieved while maintaining model accuracy. This method can significantly reduce the computational overhead of large models in practical applications, providing a feasible technical solution for the deployment of edge intelligence and mobile artificial intelligence (AI) systems.

[0033] This method aims to significantly reduce model size and computational complexity while maintaining model performance. It introduces a meta-learning regulation mechanism on the basis of traditional knowledge distillation, and makes the knowledge transfer from the teacher model to the student model more accurate, efficient and stable by dynamically adjusting the distillation intensity and prompt weight.

[0034] Please see Figure 1This is a schematic diagram of the overall system structure of a lightweight large-model method based on adaptive distillation according to an embodiment of the present invention. It mainly includes five functional units: a feature alignment module, a class-level adaptive distillation module, a sample-level dynamic cue fusion module, a temporal stability optimization module, and a lightweight training and meta-optimization module. These modules are interconnected and work together to form a complete lightweight training system. The modules and their data processing flow are as follows: 1. Feature Alignment Module Since teacher and student models typically have different layer depths and parameter sizes, their output feature dimensions and semantic levels often differ. Directly calculating feature distillation loss can lead to unstable gradient propagation and even information mismatch. The goal of this module is to achieve structural unification of the multi-layer feature space, enabling the student model to accurately perceive the representational differences of the teacher model.

[0035] Specifically, the teacher model in the The output of the layer is The output of the corresponding layer in the student model is By introducing a trainable linear mapping matrix. With bias term To achieve feature dimension transformation:

[0036] And the normalized mean squared error is used as the constraint function:

[0037] in This is the feature dimension of the student model.

[0038] In practical implementation, to reduce computational cost, the mapping matrix can be implemented using 1×1 convolution or low-rank decomposition; in multi-layer distillation scenarios, inter-layer pairing strategies can be adopted (such as selecting the teacher's first...). The level corresponds to the student's first (Layer). This module ensures that the feature representations of teachers and students are aligned at the semantic level, providing the necessary conditions for the stability of subsequent distillation.

[0039] 2. Class-level adaptive distillation module Traditional distillation methods assign uniform weights to all classes, ignoring the learning differences between samples at the class level. For example, in multi-class tasks, samples from common classes are easy to fit, while long-tail classes or classes with ambiguous features are more difficult to learn. If a fixed distillation intensity is used, the student model is prone to over-reliance on the mainstream classes, leading to a decline in generalization performance.

[0040] To this end, this invention introduces a meta-learning-driven adaptive weight generation mechanism at the category level. (Class-level meta-network) Using the average loss and uncertainty of the teacher and student models for each category as input, the distillation intensity for that category is dynamically output. :

[0041] in , Teachers and students are categorized separately. The average loss on This represents the prediction entropy or variance, used to measure the model's confidence in that class. Based on the class distillation strength, the class distillation loss is defined as:

[0042] in, The distillation temperature. For the training sample set corresponding to category c, This represents the softmax function. These are training samples belonging to category c.

[0043] This module adaptively amplifies the distillation signal of samples belonging to difficult classes, thereby improving the student model's learning ability on samples with ambiguous class boundaries. Simultaneously, by introducing class-level dynamic adjustment, it reduces the bias propagation of distillation information, enabling the model to achieve a dual improvement in accuracy and robustness without adding additional parameters.

[0044] 3. Sample-level dynamic prompting fusion module: In actual training, the degree of dependence of individual samples on the teacher model varies. Some samples have relatively clear features and can learn independently; however, for high-noise or complex samples, stronger guidance from the teacher is required. The sample-level dynamic prompting fusion module designed in this invention utilizes a meta-network. Each sample is automatically assigned a different distillation weight based on a cue. Cues can include logits layer distillation, feature layer distillation, and attention map distillation, among others.

[0045] In the During the next training iteration, for the samples Meta-network input teacher-student prediction difference Teacher predicts entropy and training progress ratio Output the fusion coefficient:

[0046] The overall sample distillation loss is obtained as follows:

[0047] in Indicates the first Distillation losses can be categorized by factors such as Kullback–Leibler (KL) divergence, mean squared error, or attention map similarity. This is the training sample set.

[0048] This module enables student models to focus on different knowledge levels at different training stages, achieving a progressive learning approach of "structure first, then semantics." Experiments show that dynamic prompting fusion effectively improves distillation stability and convergence speed on public datasets such as ImageNet and CIFAR (Canadian Institute for Advanced Research Dataset).

[0049] 4. Timing stability optimization module In the early training phase, the meta-network has not yet developed a stable understanding of the data distribution, and the generated distillation weights may fluctuate significantly. To suppress this instability, this module employs a confidence-based temporal smoothing mechanism to perform temporal filtering on the distillation weights.

[0050] Class-level weights With sample-level fusion coefficient Update using the following formula:

[0051] The confidence level is defined as:

[0052] The more stable the teacher prediction distribution, the higher the weight update ratio, thus reducing jitter. In actual training, this module can be integrated as an independent optimization layer into the loss calculation stage, significantly improving model stability.

[0053] 5. Lightweight Training and Meta-Optimization Module This module integrates the aforementioned adaptive distillation strategy into an end-to-end optimization system. The entire training process adopts a two-layer optimization structure: the inner layer updates the student model parameters, and the outer layer updates the meta-network parameters, achieving adaptive integration of multi-level distillation signals.

[0054] The inner optimization objective is the student model parameters. Minimize overall distillation losses:

[0055] in For cross-entropy loss, These are the weighting coefficients. Outer layer optimization uses the meta-network parameters. Using a small validation set as the variable, minimize the meta-loss. We also use a first-order gradient approximation method to avoid the extra overhead caused by the second-order gradient.

[0056] In engineering implementation, this module can adopt a distributed training architecture, implementing nested gradient updates via automatic differentiation on PyTorch or TensorFlow platforms. Practical verification shows that this optimized structure, while maintaining model accuracy, can reduce the parameter size to less than 30% of the original model and shorten inference time by approximately 40%, demonstrating high deployment value.

[0057] This embodiment proposes a large-model lightweighting method based on adaptive distillation. It uses a trained teacher model and a student model to be optimized as input, combined with a meta-learning-driven adaptive distillation mechanism, to achieve lightweighting of high-performance models and efficient knowledge transfer. The steps are as follows: 1. After standardizing and preprocessing the training set data, the input samples are simultaneously fed into both the teacher model and the student model. The teacher model is a fully trained high-performance network used to provide high-quality knowledge representation; the student model is a lightweight network with a simplified structure, used as a learning objective for parameter optimization.

[0058] 2. Utilize the feature alignment module to obtain intermediate layer feature representations of the teacher model and the student model. Due to the differences in network structure and feature dimensions between the two, the intermediate layer features of the teacher model are first mapped to the feature space of the student model through linear mapping or 1×1 convolution, ensuring consistency in semantic level and dimension, thereby providing a stable feature foundation for subsequent knowledge distillation.

[0059] 3. In the class-level adaptive distillation stage, statistical analysis is performed on samples of each class through a class-level meta-network. Based on the average loss values ​​and prediction uncertainties of the teacher and student models across each class, class weights are dynamically generated. This weight is used to adjust the degree of influence of different categories in the distillation process, so that complex categories or categories with fewer samples can obtain higher distillation intensity, thereby achieving a balance and improvement in accuracy.

[0060] 4. In the sample-level dynamic suggestion fusion stage, for each input sample The differences between the outputs of the teacher model and the student model, the teacher prediction entropy, and the current training progress are calculated. Dynamic fusion coefficients are generated through a sample-level meta-network. This coefficient is used to balance the influence of various distillation cues (such as logits layer distillation, feature layer distillation, and attention layer distillation), ensuring that the model focuses on different levels of knowledge features at different stages and improving the sufficiency of knowledge transfer.

[0061] 5. During the temporal stabilization optimization phase, to prevent weight fluctuations in the meta-network during the early training stages, a confidence-weighted temporal smoothing mechanism is introduced for class-level and sample-level weights. The confidence level is calculated based on the entropy value predicted by the teacher model; the more stable the prediction, the higher the update ratio. Recursive smoothing effectively suppresses random jitter in the distillation signal, improving the stability and controllability of training convergence.

[0062] 6. The dynamically adjusted distillation weights and feature-aligned student outputs are input together into the lightweight training and meta-optimization module. This module employs a two-layer optimization structure: the inner layer optimizes the student model parameters to minimize the overall distillation loss, while the outer layer optimizes the meta-network parameters to adaptively adjust the weight generation strategy based on validation set feedback. Through this collaborative update mechanism, the student model acquires teacher knowledge while significantly reducing redundant parameters and computational load.

[0063] 7. Finally, the student model obtained after training and optimization maintains the main performance of the teacher model while reducing the number of parameters by about 70% and the inference time by about 40%, achieving a balance between performance and efficiency in both image classification and object detection tasks.

[0064] The large model lightweighting method based on adaptive distillation proposed in this embodiment can be widely applied to the deployment and acceleration of artificial intelligence models. Specific examples are as follows: In mobile intelligent visual recognition scenarios, the method of this invention can be used to distill large visual models (such as ResNet or ViT structures) into lightweight networks (such as MobileNet, EfficientNet, etc.), enabling real-time inference on mobile devices or embedded chips. Through an adaptive distillation mechanism, the model can significantly reduce computational latency while maintaining recognition accuracy, allowing mobile terminals to efficiently perform tasks such as image recognition, object detection, and video analysis with limited computing power. This method not only improves the flexibility of model deployment but also enhances the visual understanding capabilities of intelligent devices in complex environments, providing a feasible technical solution for edge intelligence and low-power AI systems.

[0065] This invention proposes a lightweight method for large models based on adaptive distillation, which can effectively reduce the model size without significantly sacrificing model performance, achieving a balance between high accuracy and high efficiency. Compared with existing technologies, this invention has the following advantages: First, this invention introduces a meta-learning-driven adaptive weight generation mechanism on the basis of the traditional knowledge distillation framework, realizing dynamic control of the distillation process. Through the synergistic effect of class-level and sample-level two-layer meta-networks, the distillation intensity can be adjusted in real time according to the category complexity and sample learning status, thereby avoiding the problem of insufficient information transfer caused by fixed distillation coefficients, and enabling the student model to more fully inherit the knowledge expression of the teacher model.

[0066] Secondly, the feature alignment module proposed in this invention effectively solves the mismatch between the teacher model and the student model in terms of intermediate layer feature dimensions and semantic levels. Through linear mapping and normalization constraints, a comparable relationship is established between the two in the representation space, which improves the stability and effectiveness of the distillation signal from the source and provides a unified feature foundation for multi-layer knowledge transfer.

[0067] Furthermore, this invention introduces a confidence-based temporal smoothing mechanism during distillation training, which can dynamically suppress instability caused by weight fluctuations in the early stages of training. This mechanism updates the prediction confidence of the teacher model output with weights, allowing the distillation signal to evolve smoothly over time, significantly improving the stability of the training process and the reliability of model convergence.

[0068] Furthermore, the lightweight training and meta-optimization module proposed in this invention adopts a two-layer optimization structure: the inner layer updates student model parameters, and the outer layer optimizes the meta-network strategy. This reduces the overhead of high-order gradient calculations while maintaining computational efficiency. Experimental results show that this optimization structure exhibits good scalability and versatility across various tasks and can be easily applied to different types of deep networks and downstream tasks.

[0069] The method of this invention has a simple overall structure and is highly feasible. By dynamically adjusting the distillation weights and integrating multiple layers of knowledge transfer, this invention can reduce the parameter size to about 30% of the original model while maintaining the main performance of the teacher model, and improve the inference speed by about 40%, significantly improving the deployment efficiency of the model. This method can be widely applied to resource-constrained scenarios such as mobile devices, embedded devices, and edge computing, providing an efficient and scalable solution for lightweighting artificial intelligence models.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight system for large-scale models based on adaptive distillation, characterized in that, The system includes: The feature alignment module performs dimensional alignment on the intermediate layer features of the teacher model and the student model. The feature alignment module performs dimensional transformation on the features output by the teacher model through linear mapping or 1×1 convolution to make them consistent with the feature structure of the student model. The class-level adaptive distillation module dynamically generates class distillation intensity based on class-level statistical information, which includes the average loss and prediction uncertainty of the teacher model and student model in each class. The class-level adaptive distillation module outputs class distillation intensity based on the meta-learning network to adjust the class distillation loss. The sample-level dynamic prompt fusion module dynamically assigns distillation weights for different distillation prompts to a single training sample. Based on the prediction difference between the teacher model and the student model, the teacher prediction entropy, and the current training progress, it dynamically generates prompt fusion coefficients. The distillation prompts include logits layer distillation, feature layer distillation, and attention map distillation. The temporal stability optimization module performs stability correction on class weights and sample-level fusion coefficients based on confidence and through a time-progressive smoothing mechanism, so as to reduce optimization instability caused by weight fluctuations in the early training stage. The lightweight training and meta-optimization module employs a two-layer optimization structure to simultaneously train student model parameters and meta-network parameters. The inner layer optimizes the student model to minimize the overall distillation loss, while the outer layer optimizes the meta-network strategy based on validation set feedback, dynamically adjusting distillation weights and cue allocation.

2. The lightweight system for large models based on adaptive distillation according to claim 1, characterized in that: The feature alignment module introduces a trainable linear mapping matrix. With bias term The feature dimension transformation is calculated and expressed as: Among them, the teacher model in the first The output of the layer is The output of the corresponding layer in the student model is ; Using the normalized mean squared error as the constraint function, it is expressed as: in, This is the feature dimension of the student model.

3. The lightweight system for a large model based on adaptive distillation according to claim 1, characterized in that: The class-level adaptive distillation module introduces a meta-learning-driven adaptive weight generation mechanism at the class level, and a class-level meta-network. Using the average loss and uncertainty of the teacher and student models for each category as input, the distillation intensity for that category is dynamically output. , represented as: in, , Teachers and students are categorized separately. The average loss on This represents the prediction entropy or variance; Based on the distillation intensity of this category, the category distillation loss is calculated and expressed as: in, The distillation temperature. For the training sample set corresponding to category c, This represents the softmax function. These are training samples belonging to category c.

4. The lightweight system for a large model based on adaptive distillation according to claim 1, characterized in that: The sample-level dynamic prompting fusion module, in the first During the next training iteration, for the samples Meta-network input teacher-student prediction difference Teacher predicts entropy and training progress ratio The output prompts the fusion coefficient, which is represented as: The overall sample distillation loss is calculated and expressed as: in, Indicates the first Distillation loss as indicated by the class This is the training sample set.

5. A large-scale lightweight system based on adaptive distillation according to claim 1, characterized in that: The time-series stability optimization module will adjust the class-level weights. With sample-level fusion coefficient The update is calculated using the following method: The confidence level is expressed as: 。 6. A lightweight system for large models based on adaptive distillation according to claim 1, characterized in that: The lightweight training and meta-optimization module sets the inner-layer optimization objective to the student model parameters. Minimizing the overall distillation loss is expressed as: in, For cross-entropy loss, These are the weighting coefficients. For category distillation loss, To account for sample distillation losses, For constraint functions.

7. A large-scale lightweight system based on adaptive distillation according to claim 1, characterized in that: The lightweight training and meta-optimization module sets the outer layer optimization based on the meta-network parameters. Using a small validation set as the variable, minimize the meta-loss. We use a first-order gradient approximation method to avoid the extra overhead caused by the second-order gradient.

8. A lightweight system for large models based on adaptive distillation according to claim 1, characterized in that: The system is designed to reduce the size of model parameters and computational complexity while maintaining model accuracy, and supports deployment on edge smart devices or mobile AI systems.

9. A lightweight method for large models based on adaptive distillation, characterized in that, Includes the following steps: S1. Acquire training data and perform standardized preprocessing. Input the processed samples into the teacher model and student model respectively. The teacher model is a high-performance model that has been trained, and the student model is a lightweight model to be optimized. S2. Obtain intermediate layer features of the teacher model and student model through the feature alignment module; when the feature dimensions are inconsistent, map the intermediate features of the teacher model to the feature space of the student model through linear mapping or 1×1 convolution to achieve semantic consistency and structural alignment. S3. Perform class-level statistical analysis on the training samples through the class-level adaptive distillation module, generate class distillation weights based on the average loss and prediction uncertainty of the teacher model and student model in each class, and adjust the class-level distillation loss based on the weights. S4. Through the sample-level dynamic prompting fusion module, based on the difference between the prediction results of the teacher model and the student model, the prediction entropy of the teacher model and the training progress, a prompting fusion coefficient is generated for the sample. Based on the fusion coefficient, the logits layer distillation, the feature layer distillation and the attention distillation are weighted and fused to form a sample-level comprehensive distillation loss. S5. The class distillation weights and sample-level fusion coefficients are smoothed by time recursion based on confidence through the time-series stability optimization module to reduce the fluctuations of the distillation signal in the early stage of training and improve training stability. S6. Input the dynamically adjusted distillation weights and student model output features into the lightweight training and meta-optimization module, and use a two-layer optimization structure to update the model parameters: the inner layer is used to optimize the student model to minimize the distillation loss, and the outer layer is used to optimize the meta-network parameters based on the validation set feedback to adaptively adjust the weight generation strategy. S7. Output the trained student model.

Images