Construction method of lightweight model, computer device, equipment and storage medium

Abstract

The application relates to a lightweight model construction method, a computer device, equipment and a storage medium, and relates to the technical field of model compression. The method comprises the following steps: knowledge distillation is performed based on a student model and multiple teacher models, the student model is pruned and simulated quantization is performed, task loss, distillation loss, pruning loss and quantization loss are calculated respectively; according to a training stage of a current training step, a weight coefficient set is obtained, the various types of losses calculated are weighted and summed to obtain total loss; based on the total loss, the student model parameters and the quantization step are updated, and the next training step is entered; after the training is completed, the pruned channels are removed from the student model, the quantization step is bound with the model weight of the student model, and a lightweight model is output. Through the multi-teacher knowledge distillation and adaptive loss fusion mechanism, the multi-objective collaborative optimization of the lightweight model compression rate, the inference speed and the accuracy is realized under the joint optimization framework of knowledge distillation, pruning and quantization.

Description

Technical Field

[0001] This application relates to the field of model compression technology, and in particular to a method, apparatus, computer device, and computer-readable storage medium for constructing lightweight models. Background Technology

[0002] With the deep application of artificial intelligence technology in the power industry, large-scale visual and linguistic multimodal models based on the Transformer architecture have demonstrated powerful perception and understanding capabilities in scenarios such as transmission line inspection, substation equipment monitoring, and safety operation supervision. These models typically contain hundreds of millions or even billions of parameters, enabling them to accurately identify equipment defects, understand operational behaviors, and assess safety hazards, significantly improving the intelligence level of power operation and maintenance. However, in actual business operations, the deployment of these large models faces significant challenges, including strict resource constraints, sensitivity to inference latency, and high accuracy requirements. Therefore, there is an urgent need to utilize model lightweighting techniques to compress large models into efficient smaller models suitable for edge devices.

[0003] Traditional approaches typically employ one of three methods: knowledge distillation, structured pruning, or quantization compression. However, achieving an optimal balance between model compression ratio, accuracy preservation, and inference speed is challenging. A single knowledge distillation method is limited by the upper limit of the teacher model's capabilities, resulting in a limited compression ratio. A single pruning method experiences a sharp drop in accuracy at high pruning rates. A single quantization method suffers unacceptable accuracy loss at low-bit quantization levels. When distillation, pruning, and quantization are combined sequentially, the loss functions at each stage are independent, with fixed weights at each stage. Fixed-weight strategies cannot adapt to these stage-specific changes, preventing the simultaneous and adequate satisfaction of all optimization objectives. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, and computer-readable storage medium for constructing a lightweight model to address the aforementioned technical problems. Through a multi-teacher knowledge distillation and adaptive loss fusion mechanism, the lightweight model achieves multi-objective synergistic optimization by maintaining compression ratio, inference speed, and accuracy within a joint optimization framework of knowledge distillation, pruning, and quantization.

[0005] Firstly, this application provides a method for constructing a lightweight model, including:

[0006] The training samples of the current training step are input into the student model and multiple teacher models respectively, so that the student model and multiple teacher models can perform forward propagation; the student model is the network model to be trained, and the teacher model is the network model that has been pre-trained and whose parameters are frozen. Each teacher model has a different network architecture or a different distribution of pre-training data.

[0007] The task loss is calculated based on the difference between the predicted results output by the student model and the true labels of the training samples.

[0008] Distillation loss is calculated based on the soft-label output of each teacher model, the soft-label output of each student model, the intermediate layer features of each teacher model, and the intermediate layer features of each student model.

[0009] Based on the weights of each channel in the student model, the pruning loss is calculated, and the student model is pruned according to the importance scores of each channel and the current pruning rate.

[0010] The student model is simulated quantization based on the quantization step size, and the quantization loss is calculated based on the weights of the student model before and after the simulated quantization.

[0011] Based on the training stage corresponding to the current training step, obtain the corresponding set of weight coefficients; the set of weight coefficients includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients;

[0012] Based on the set of weight coefficients, the task loss, distillation loss, pruning loss and quantization loss are weighted and summed to obtain the total loss.

[0013] Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters and quantization step size of the student model, and then the training continues until the preset training completion conditions are met.

[0014] After training is complete, the pruned channels are removed from the trained student model, and the quantization step size obtained during training is bound to the model weights of the trained student model to output a lightweight model.

[0015] In one embodiment, the training phase corresponding to the current training step includes any one of the following: knowledge transfer phase, pruning optimization phase, quantization adaptation phase, and joint fine-tuning phase; the set of weight coefficients corresponding to the knowledge transfer phase includes... , , , ;

[0016] In the set of weight coefficients corresponding to the pruning optimization stage, , , , ;

[0017] In the set of weight coefficients corresponding to the quantization adaptation stage, , , , ;

[0018] In the set of weight coefficients corresponding to the joint fine-tuning phase, , , , ;

[0019] in, This is the task loss weighting coefficient. This is the distillation loss weighting coefficient. This is the pruning loss weighting coefficient. This is the weighting coefficient for quantifying the loss.

[0020] In one embodiment, a set of weight coefficients is obtained based on the training stage corresponding to the current training step, including:

[0021] Get the ratio of the current training step to the total training steps;

[0022] The ratio value is matched with multiple preset ratio ranges; each ratio range corresponds to a different training stage.

[0023] Determine the training stage corresponding to the current training step based on the matching results;

[0024] Obtain the set of weight coefficients corresponding to the determined training stage from multiple preset sets of weight coefficients.

[0025] In one embodiment, before obtaining the total loss by weighted summation of the task loss, distillation loss, pruning loss, and quantization loss based on the set of weight coefficients, the method further includes:

[0026] For each loss term in the task loss, distillation loss, pruning loss, and quantization loss, calculate the gradient variance of the loss term with respect to the corresponding parameter;

[0027] The weight adjustment amount corresponding to the loss term is calculated based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms.

[0028] Based on the weight fine-tuning amount, the weight coefficients corresponding to the loss terms in the weight coefficient set are adjusted to obtain the final weight coefficients corresponding to the loss terms.

[0029] In one embodiment, the student model is pruned based on the importance scores of each channel in the student model and the current pruning rate, including:

[0030] The current pruning rate is determined based on the ratio of the current training steps to the total training steps and the target pruning rate.

[0031] The number of channels to be pruned is determined based on the current pruning rate;

[0032] The channels in the student model are sorted from highest to lowest importance score, and the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be removed are removed.

[0033] In one embodiment, the student model is pruned based on the importance scores of each channel in the student model and the current pruning rate, including:

[0034] The current pruning rate is determined based on the ratio of the current training steps to the total training steps and the target pruning rate.

[0035] Based on the different types of network layers in the student model, assign corresponding pruning ratios to each network layer;

[0036] Based on the current pruning rate and the pruning rate ratio of each network layer, determine the number of channels to be pruned in the current training step for each network layer.

[0037] For each network layer, sort the channels in the network layer from high to low according to their importance scores, and remove the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be removed.

[0038] In one embodiment, the method further includes:

[0039] Obtain the weight parameters of each channel, the gradient information of each channel, and the activation values ​​of the feature maps output by each channel in the student model;

[0040] For each channel in the student model, the importance score of the channel is obtained by weighting and summing the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation values.

[0041] Secondly, this application also provides a lightweight model construction apparatus, comprising:

[0042] The knowledge distillation module is used to input the training samples of the current training step into the student model and multiple teacher models respectively, enabling the student model and multiple teacher models to perform forward propagation. The student model is the network model to be trained, and the teacher models are pre-trained network models with frozen parameters. Each teacher model has a different network architecture or a different distribution of pre-training data. The task loss is calculated based on the difference between the prediction results output by the student model and the true labels of the training samples. The distillation loss is calculated based on the soft label outputs of each teacher model, the soft label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model.

[0043] The structured pruning module is used to calculate the pruning loss based on the weights of each channel in the student model, and to prune the student model according to the importance scores of each channel and the current pruning rate.

[0044] The quantization-aware training module is used to simulate quantization operations on the student model based on the quantization step size, and to calculate the quantization loss based on the weights of the student model before and after the simulated quantization operation.

[0045] The adaptive loss fusion module is used to obtain the corresponding set of weight coefficients according to the training stage corresponding to the current training step. The set of weight coefficients includes the weight coefficients of task loss, distillation loss, pruning loss, and quantization loss. Based on the set of weight coefficients, the task loss, distillation loss, pruning loss, and quantization loss are weighted and summed to obtain the total loss. Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model, and then the model enters the next training step and continues training until the preset training completion conditions are met.

[0046] The lightweight model output module is used to remove the pruned channels from the trained student model after training is completed, and to bind the quantization step size obtained during training to the model weights of the trained student model, thus outputting the lightweight model.

[0047] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0048] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0049] The aforementioned lightweight model construction method, apparatus, computer equipment, and computer-readable storage medium input the training samples of the current training step into the student model and multiple teacher models respectively, enabling forward propagation between the student model and the multiple teacher models. The student model is the network model to be trained, and the teacher models are pre-trained network models with frozen parameters. Each teacher model has a different network architecture or a different distribution of pre-trained data. The task loss is calculated based on the difference between the prediction results output by the student model and the true labels of the training samples. The distillation loss is calculated based on the soft-label outputs of each teacher model, the soft-label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model. The pruning loss is calculated based on the weights of each channel in the student model, and the student model is pruned according to the importance scores of each channel and the current pruning rate. The process involves: 1) Simulating quantization operations on the student model based on the quantization step size, and calculating the quantization loss based on the student model's weights before and after the simulation quantization operation; 2) Obtaining the corresponding set of weight coefficients according to the training stage of the current training step; 3) The set of weight coefficients includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients; 4) Weighted summing of the task loss, distillation loss, pruning loss, and quantization loss based on the set of weight coefficients to obtain the total loss; 5) Backpropagation based on the total loss to update the student model's model parameters, channel sparsity parameters, and quantization step size, then proceeding to the next training step and continuing training until the preset training completion conditions are met; 6) Removing pruned channels from the trained student model and binding the quantization step size obtained during training with the model weights of the trained student model to output a lightweight model. Through this method, a multi-teacher knowledge distillation approach is adopted, utilizing the complementary capabilities of multiple teacher models with different architectures or different pre-training data distributions on different sub-tasks. This integrates and transfers the strengths of each teacher to the student model, effectively compensating for the accuracy loss caused by pruning and quantization, and significantly improving the learning effect and final accuracy of the lightweight model. A joint lightweight framework for knowledge distillation, structured pruning, and quantization compression was constructed. Distillation loss, pruning loss, and quantization loss are weighted and fused in a unified loss function. Through a single backpropagation, the network parameters, channel sparsity parameters, and quantization step size of the student model are simultaneously optimized, overcoming the problems of fragmented objectives and amplified error accumulation in traditional sequential processing. Knowledge distillation provides accuracy supervision for pruning and quantization, maintaining task performance during compression; pruning reduces the computational cost of quantization by decreasing the number of channels; and quantization further compresses the pruned model size through low-precision representation. Multi-objective joint optimization of compression ratio, accuracy preservation, and inference speed is achieved. Based on an adaptive loss fusion mechanism, the problem that fixed-weight strategies cannot achieve optimal configuration at each stage is overcome, significantly improving the convergence efficiency of joint training and the accuracy of the final model. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the construction method of a lightweight model in one embodiment;

[0052] Figure 2 This is a structural block diagram of a lightweight model construction device in one embodiment;

[0053] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0056] In one exemplary embodiment, such as Figure 1 As shown, a method for constructing a lightweight model is provided, including:

[0057] Step 102: Input the training samples of the current training step into the student model and multiple teacher models respectively, so that the student model and multiple teacher models can perform forward propagation; the student model is the network model to be trained, and the teacher model is the network model that has been pre-trained and whose parameters are frozen. Each teacher model has a different network architecture or a different distribution of pre-trained data.

[0058] This involves acquiring multiple pre-trained teacher models and one student model to be trained. The multiple teacher models are large neural networks that have been pre-trained and have their parameters frozen. Each teacher model has a different network architecture or a different distribution of pre-trained data, and therefore has its own strengths and knowledge focuses in different sub-tasks, different modalities, or different data distributions.

[0059] In each training step, the training samples of the current batch are simultaneously input into the student model and all teacher models, and forward propagation is performed separately. Each teacher model, with its parameters frozen, generates its own soft-label output and intermediate-layer feature map for the same input sample. The soft label is the class probability distribution with a temperature coefficient output by the teacher model, containing richer information on inter-class similarity compared to hard labels; the intermediate-layer feature map contains feature representations extracted by the teacher model at different levels of abstraction. The student model also generates its own prediction results and intermediate-layer features. By having each teacher model and student model process the same input simultaneously, a data foundation is provided for subsequent calculations of the KL divergence between each teacher's soft label and the student's soft label, as well as the mean squared error between each teacher's intermediate-layer feature and the corresponding student's layer feature.

[0060] Step 104: Calculate the task loss based on the difference between the prediction results output by the student model and the true labels of the training samples.

[0061] Among them, mission losses The standard supervised loss for the student model on the target task includes the student model's predictions for the current input sample. The true label of the sample Depending on the task type, standard loss functions such as cross-entropy loss, mean squared error loss, or YOLO series detection loss can be used. The formula for calculating the task loss is:

[0062] ;

[0063] in, This represents the standard loss function corresponding to the task.

[0064] Step 106: Calculate the distillation loss based on the soft label output of each teacher model, the soft label output of each student model, the intermediate layer features of each teacher model, and the intermediate layer features of each student model.

[0065] Specifically, the KL divergence between the soft labels output by each teacher model and the soft labels output by each student model is calculated separately, and then weighted and summed to obtain the output distillation loss; the mean squared error between the intermediate layer features of each teacher model and the corresponding layer features of the student model is calculated separately, and then weighted and summed to obtain the feature distillation loss; the initial distillation loss and the feature loss are then integrated to obtain the final distillation loss. The formula for calculating the distillation loss is:

[0066] ;

[0067] in, For the number of teacher models, and These are the output distillation weights and feature distillation weights of the k-th teacher model, respectively. This is the soft-label output of the k-th teacher model, whose input parameter is the teacher model's prediction for the current sample. and temperature parameters ; This is the soft-label output of the student model. Its input parameter is the student model's prediction for the current sample. and temperature parameters ; and These are the intermediate layer features of the teacher model and the student model, respectively; This is the overall weighting coefficient for feature distillation loss, used to control the proportion of feature matching loss in the total distillation loss.

[0068] Step 108: Calculate the pruning loss based on the weights of each channel in the student model, and prune the student model according to the importance scores of each channel and the current pruning rate.

[0069] Among them, pruning losses This is a channel sparsity regularization term, its purpose is to encourage the weights of unimportant channels to approach zero, preparing for subsequent channel pruning. The input parameters for the pruning loss are the weights of each channel. The calculation method involves summing the L2 norms of the weights of each channel, making the overall weights of unimportant channels approach 0.

[0070] ;

[0071] in, The total number of channels. For the first The model uses a weight matrix for each channel, with the input parameters being the current weight parameters of each channel. This regularization term is jointly optimized with the task loss, driving the weights of unimportant channels to converge to zero during training, thereby achieving automatic channel sparsity.

[0072] In one optional implementation, channels are sorted from highest to lowest importance score. The number of channels to be pruned is calculated based on the current pruning rate, and the last-ranked channel, corresponding to the number of channels to be pruned, is removed from the network. In another optional implementation, an importance threshold is set, and all channels with scores below this threshold are considered redundant and removed all at once. Both methods can be flexibly selected based on the specific application scenario and accuracy requirements.

[0073] Optionally, a gradual pruning strategy is adopted. The pruning rate gradually increases with the number of training steps according to a preset growth pattern, rather than reaching the target pruning rate all at once. This approach allows the model sufficient time to adapt to the new network structure through subsequent training after each structural change, restoring the accuracy lost due to pruning and effectively avoiding the accuracy collapse caused by a large-scale pruning all at once. The pruning operation is performed simultaneously with knowledge distillation and quantization-aware training, enabling joint optimization of each stage without the need for an additional separate retraining phase.

[0074] Step 110: Perform simulated quantization on the student model based on the quantization step size, and calculate the quantization loss based on the weights of the student model before and after the simulated quantization.

[0075] This involves quantization-aware training of the model, simulating quantization errors during training. Quantization nodes are inserted during the network's forward propagation, treating the weights as already quantized, thus adapting the model parameters to a lower-precision representation. For example, the quantization operation... It can be represented as:

[0076] ;

[0077] in, For the parameters to be quantized, To quantize the step size, To quantize the number of bits, To round to the nearest integer, This is a truncation operation. Quantization step size. During training, it is updated as a learnable parameter to adaptively adapt to changes in parameter distribution.

[0078] Quantification of loss This is quantized into an error term, used to measure the accuracy loss before and after weight quantization. The input parameter for the quantization loss is the original weights. and quantized weights The calculation method can be the mean square error between the two:

[0079] ;

[0080] in, The total number of parameters to be quantized. For the i-th original parameter, This is the quantized value of the i-th parameter. This loss minimizes the difference in weights before and after quantization, allowing the model parameters to adaptively adjust their distribution during training, thereby reducing the accuracy loss caused by quantization.

[0081] In one optional implementation, different quantization precipitates are set for the visual encoder and language decoder of the multimodal model to adapt to their heterogeneous characteristics. For the visual encoder, due to its high requirement for the accuracy of image detail representation, a first quantization bit count is used for quantization. For example, all or part of the weights of the visual encoder are quantized using INT8 to retain high numerical precision and maintain the accurate representation of fine-grained visual features. For the language decoder, which has a larger number of parameters and a relatively higher tolerance for quantization errors, a second quantization bit count is used for quantization, which is lower than the first quantization bit count. For example, all or part of the weights of the language decoder are quantized using INT4 to achieve higher compression ratio and faster inference speed. Furthermore, within the language decoder, the weights of the self-attention layer are quantized using a third quantization bit count, which is relatively high, to protect the accuracy of semantic relation modeling; the weights of the feedforward network layer are quantized using a fourth quantization bit count, which is relatively low, to pursue higher compression efficiency.

[0082] By setting the differential quantization precision as described above, the visual encoder can still maintain accurate representation of image details under low bit quantization, and the language decoder can achieve greater parameter compression while maintaining semantic understanding capabilities, thus achieving higher precision preservation at the same overall compression rate.

[0083] Step 112: Based on the training stage corresponding to the current training step, obtain the corresponding set of weight coefficients; the set of weight coefficients includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients.

[0084] The entire training process is divided into multiple training phases in chronological order, each with a different core optimization objective. At the current training step, the training phase is determined based on the training progress, and a set of weight coefficients matching that phase is selected, enabling the model to focus on the most important optimization objective at each phase.

[0085] In one optional implementation, the training stage is determined based on the ratio of the current number of training steps to the preset total number of training steps. When this ratio is in a preset first interval, it is determined to be the knowledge transfer stage, and a first set of weight coefficients is obtained, in which the distillation loss weight coefficient has the highest value to focus on transferring teacher knowledge; when this ratio is in a preset second interval, it is determined to be the pruning optimization stage, and a second set of weight coefficients is obtained, in which the pruning loss weight coefficient is increased to focus on promoting channel sparsity; when this ratio is in a preset third interval, it is determined to be the quantization adaptation stage, and a third set of weight coefficients is obtained, in which the quantization loss weight coefficient has the highest value to focus on reducing quantization error; when this ratio is in a preset fourth interval, it is determined to be the joint fine-tuning stage, and a fourth set of weight coefficients is obtained, in which the task loss weight coefficient has the highest value to focus on optimizing the final task accuracy.

[0086] In another alternative implementation, the weight coefficients between stages are smoothly transitioned, meaning that the weight coefficients change continuously with the number of training steps, rather than jumping at the stage boundaries, in order to avoid drastic fluctuations in the loss function during stage switching.

[0087] In another optional implementation, after obtaining the set of basic weight coefficients corresponding to each stage, the weight coefficients are further fine-tuned based on the gradient variance of each loss term, so that the weight configuration is further adapted to the current optimization state within the staged framework. The sum of all weight coefficients in each set of weight coefficients is 1.

[0088] Step 114: Based on the set of weight coefficients, perform a weighted sum of the task loss, distillation loss, pruning loss, and quantization loss to obtain the total loss.

[0089] The distillation loss, pruning loss, quantization loss, and task loss are weighted and fused together, resulting in the following total loss function:

[0090] ;

[0091] in, For mission losses, For distillation losses, Losses due to pruning To quantify the loss, each weight coefficient ω(t) is dynamically adjusted based on the current training step t or the training stage.

[0092] Step 116: Perform backpropagation based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model, then proceed to the next training step and continue training until the preset training completion conditions are met.

[0093] The optimization objective is to use the total loss as the objective, and the gradient of the total loss with respect to the learnable parameters in the student model is calculated using the backpropagation algorithm. Learnable parameters include the network weights and quantization stride of the student model. Based on the calculated gradient, a pre-defined optimizer is used to jointly update the network weights, channel sparsity parameters, and quantization stride of the student model, optimizing the total loss in a decreasing direction. The model parameters of the student model refer to all learnable weights and biases in the student network, including the convolutional kernel weights of convolutional layers, the weight matrix of fully connected layers, and bias terms. These parameters determine the student model's ability to extract features from the input data and the accuracy of the final predicted output. Channel sparsity parameters refer to the weight matrices of each channel involved in the pruning loss calculation. The quantization stride is a key parameter in quantization-aware training, used to control the scaling scale when mapping floating-point weights to low-bit integers. During backpropagation, the parameters of each teacher model remain frozen, not participating in gradient calculation or parameter updates, serving as a fixed source of knowledge to provide supervision signals. After updating the parameters in the current training step, the process moves to the next training step and repeats the aforementioned steps, namely, re-executing forward propagation, calculating each loss, calculating the total loss, and updating the parameters, until the preset training completion conditions are met. These preset training completion conditions include: reaching a preset total number of training steps, or the total loss value decreasing below a preset threshold and remaining stable, or the model's evaluation metrics on the validation set meeting preset requirements. This joint update mechanism allows for the simultaneous optimization of structural pruning and accuracy maintenance during model compression within the same training framework, avoiding the accuracy loss and low training efficiency problems caused by the traditional sequential paradigm of pruning before retraining.

[0094] Step 118: After training is complete, the channels to be pruned are removed from the trained student model, and the quantization step size obtained during training is bound to the model weights of the trained student model to output a lightweight model.

[0095] Specifically, the pruning process physically removes channels marked for pruning during training from the student model's network structure. The rows or columns of the weight matrix corresponding to the pruned channels are directly deleted from the model, retaining no redundant parameters, resulting in a simplified network structure. After this operation, the width of the student model decreases, and the number of network parameters and inference computations are correspondingly reduced.

[0096] For quantization, the quantization step size obtained during training is bound to the model weights. Specifically, the floating-point weights of the student model are converted into low-bit integer representations according to the quantization step size, and the quantization step size is saved in the model file as a fixed parameter during the inference phase. During the inference phase, the quantized integer weights are used directly for integer operations, eliminating the need for further floating-point quantization operations, thereby reducing computational overhead and storage usage during inference.

[0097] After the above processing, a lightweight student model is output. This model possesses both simplified structure and low-precision representation, making it directly adaptable to the computational resource constraints of edge devices. During deployment, only this lightweight model file needs to be loaded; the original large model, teacher model, or auxiliary modules used during training are not required, allowing it to independently complete inference tasks.

[0098] It should be noted that in the embodiments of this application, knowledge distillation, structured pruning, and quantization-aware training are executed in parallel during the same training process, rather than sequentially. In each training step, the student model simultaneously receives supervision signals from all three, which, together with its own task loss, form the total loss for backpropagation, while simultaneously updating the parameters of the student model.

[0099] For example, power AI models need to be deployed in various scenarios, including the cloud, regional centers, and edge devices. Edge devices have limited computing power, small memory capacity, and power consumption, making it difficult to handle the inference computations of large original models. Taking power transmission inspection as an example, edge devices typically have only 2-4GB of memory, while the original large visual language models such as Qwen-VL-Chat have billions of parameters, exceeding the device's capacity just for model loading. Therefore, the lightweight model output by the method in this embodiment can be applied to edge devices.

[0100] For example, scenarios such as safety operation monitoring and real-time monitoring of power equipment require models to complete inference and output results within seconds or even milliseconds. The inference latency of large models at the edge is typically hundreds of milliseconds to several seconds, which is insufficient to meet real-time requirements and limits the application of power AI models in real-time-sensitive scenarios. Therefore, the lightweight model output by the method in this embodiment can significantly reduce the inference latency from hundreds of milliseconds to several seconds of the original large model, meeting the second-level or even millisecond-level real-time requirements of power scenarios while maintaining acceptable task accuracy.

[0101] For example, tasks such as defect identification and fault diagnosis in power scenarios require extremely high model recognition accuracy; both missed detections and false detections can lead to serious safety accidents or economic losses. Traditional lightweighting techniques often incur a significant loss of accuracy while compressing the model. Maintaining high task accuracy while drastically reducing model size is the core challenge in lightweighting large power models. Therefore, the lightweight model output by the method in this embodiment can improve training accuracy.

[0102] The lightweight model construction method described above employs a multi-teacher knowledge distillation approach. This method leverages the complementary capabilities of multiple teacher models with different architectures or pre-trained data distributions across various sub-tasks, integrating and transferring the strengths of each teacher to the student model. This effectively compensates for the accuracy loss caused by pruning and quantization, significantly improving the learning performance and final accuracy of the lightweight model. A joint lightweight framework is constructed, co-optimizing knowledge distillation, structured pruning, and quantization compression. Distillation loss, pruning loss, and quantization loss are weighted and fused in a unified loss function. Through a single backpropagation, the network parameters, channel sparsity parameters, and quantization step size of the student model are simultaneously optimized, overcoming the problems of fragmented objectives and amplified error accumulation in traditional serial processing. Knowledge distillation provides accuracy supervision for pruning and quantization, maintaining task performance during compression. Pruning reduces the computational overhead of quantization by decreasing the number of channels. Quantization further compresses the pruned model size through low-precision representation. This achieves multi-objective joint optimization of compression ratio, accuracy preservation, and inference speed. Based on the adaptive loss fusion mechanism, the problem that the fixed weight strategy cannot achieve optimal configuration at each stage is overcome, which significantly improves the convergence efficiency of joint training and the accuracy of the final model.

[0103] In an exemplary embodiment, the samples of the current batch are input into the student model and the teacher model respectively, and forward propagation is performed:

[0104] Prediction results based on the student model and real labels Calculate task loss ;

[0105] Soft label output based on each teacher model soft label output of student model and the intermediate layer features of each teacher model and student model. and Calculate distillation losses: ;

[0106] Based on the current weights of each channel in the student model Calculate pruning losses ;

[0107] Based on the current original weights of the student model and quantized weights Calculate quantization loss .

[0108] Based on the current number of training steps Total training steps proportion Determine the current training phase and determine the weight coefficients according to the weight configuration of each phase. , , , The value of is determined and adaptively fine-tuned based on the gradient variance.

[0109] Calculate the weighted total loss: .right Perform backpropagation and calculate the gradient. At the same time, all learnable parameters of the student model, such as network weights and quantization step size, are updated.

[0110] After updating the parameters in the current step, proceed to the next training step and repeat the above process until the preset total number of training steps is reached. .

[0111] Through the above joint training process, distillation, pruning, and quantization work together in each training step for the student model. Distillation loss provides a supervisory signal for knowledge transfer, pruning loss drives channel sparsity, and quantization loss guides parameters to adapt to low-precision representation. The three are optimized together in a unified total loss function, and the final output is a lightweight model that has high precision, high sparsity, and quantization friendliness.

[0112] In an exemplary embodiment, the training stage corresponding to the current training step includes any one of the following: knowledge transfer stage, pruning optimization stage, quantization adaptation stage, and joint fine-tuning stage; the set of weight coefficients corresponding to the knowledge transfer stage includes... , , , In the set of weight coefficients corresponding to the pruning optimization stage, , , , In the set of weight coefficients corresponding to the quantization adaptation stage, , , , In the set of weight coefficients corresponding to the joint fine-tuning phase, , , , ;in, This is the task loss weighting coefficient. This is the distillation loss weighting coefficient. This is the pruning loss weighting coefficient. This is the weighting coefficient for quantifying the loss.

[0113] The entire training process is divided into multiple training phases, each with a different optimization focus. Correspondingly, each training phase corresponds to a pre-defined set of weight coefficients to determine the contribution percentage of each loss term in the total loss for that phase. The weight coefficient set includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients. When training enters different phases, the loss is weighted and fused using the weight coefficient set corresponding to that phase, ensuring the model focuses on the most important optimization objective at each phase.

[0114] The first stage, the knowledge transfer stage, focuses on knowledge transfer. Distillation loss plays a dominant role, helping the student model fully absorb the knowledge from the teacher model. In the first set of weight coefficients corresponding to this stage, the task loss weight coefficient is set to a preset base value (e.g., 0.3), the distillation loss weight coefficient is the maximum value in this set, and the pruning loss weight coefficient and quantization loss weight coefficient are set to lower values. In one example, the task loss weight coefficient... Distillation loss weighting coefficient Pruning loss weighting coefficient Quantization loss weight coefficient .

[0115] The second stage, the pruning and optimization stage, marks the initial completion of knowledge transfer in the model, shifting the training focus to channel sparsity. Pruning loss weights are increased to push the weights of less important channels closer to zero, preparing for subsequent pruning. Distillation loss weights are reduced but retained at a certain proportion to maintain knowledge transfer effectiveness. In the second set of weight coefficients corresponding to this stage, the task loss weight coefficients maintain their base values, the pruning loss weight coefficients are increased relative to the first set, and the distillation loss weight coefficients are decreased relative to the first set, to maintain necessary knowledge transfer effectiveness while promoting channel sparsity. In one example... , , , .

[0116] The third stage, the quantization adaptation stage, shifts the training focus to adaptive adjustments to the low-precision representation, primarily minimizing quantization error. Quantization loss weights are increased to ensure the model parameters are fully adapted to the low-precision representation. Pruning loss weights are reduced, and distillation loss is further reduced. In the third set of weight coefficients corresponding to this stage, the task loss weight coefficient maintains its base value, the quantization loss weight coefficient is the maximum value in the third set, and the distillation and pruning loss weight coefficients are set to lower values ​​to avoid interfering with the quantization adaptation process. In one example... , , , .

[0117] The fourth stage, the joint fine-tuning stage, sees the model structure largely finalized. The training focus shifts to optimizing overall task accuracy while maintaining quantization adaptation and knowledge retention, allowing the model to achieve optimal task accuracy with a lightweight design. In this stage's fourth set of weight coefficients, the task loss weight coefficient is greater than the base value and is the maximum value in the set. The distillation loss and quantization loss weight coefficients are appropriately sized, while the pruning loss weight coefficient is the minimum value in the set. The sum of all weight coefficients in each stage's set is 1. In one example... , , , .

[0118] In one optional implementation, the weight coefficients between stages are smoothly switched using linear interpolation. Specifically, a preset transition interval is set before and after the boundary between two adjacent training stages. Within this transition interval, the weight coefficients no longer remain constant but gradually change from the configuration of the previous stage to the configuration of the next stage. This smooth switching method avoids the jumps in total loss and gradient oscillations caused by abrupt changes in weight coefficients at stage boundaries, ensuring that the direction of parameter updates remains continuous and stable during stage switching, effectively guaranteeing the stability of the training process and the convergence accuracy of the final model.

[0119] In this embodiment, the distillation loss is emphasized in the early stage of training to fully transfer teacher knowledge, the pruning loss weight is increased in the middle stage of training to promote channel sparsity, and the quantization loss weight is increased in the later stage of training to reduce quantization error. This overcomes the problem that the fixed weight strategy cannot take into account different optimization objectives at each stage, and significantly improves the effect and convergence efficiency of joint training.

[0120] In an exemplary embodiment, step 112 includes: obtaining the ratio of the current training step to the total training steps; matching the ratio with a plurality of preset ratio ranges; the plurality of ratio ranges correspond to different training stages; determining the training stage corresponding to the current training step based on the matching results; and obtaining a set of weight coefficients corresponding to the determined training stage from a plurality of preset weight coefficient sets.

[0121] The entire training process is divided into four stages, with the stage division based on the current number of training steps. Total training steps proportion .

[0122] For example, Table 1 is an example table of stage division methods. The ratio value is matched with the ratio range of each step shown in Table 1. If the ratio value belongs to 0≤ If the value is less than 0.25, then the training stage corresponding to the current training step is determined to be the knowledge transfer stage.

[0123] Table 1:

[0124]

[0125] In an exemplary embodiment, prior to step 114, the method further includes: calculating the gradient variance of each loss term relative to the corresponding parameter for each loss term among the task loss, distillation loss, pruning loss, and quantization loss; calculating the weight fine-tuning amount corresponding to the loss term based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms; and adjusting the weight coefficients corresponding to the loss terms in the weight coefficient set based on the weight fine-tuning amount to obtain the final weight coefficients corresponding to the loss terms.

[0126] In addition to the phased weight setting, an adaptive fine-tuning mechanism based on gradient variance is further introduced. In each training step, the gradient variance of each loss term relative to its corresponding parameter is calculated. A loss term with a large gradient variance indicates that its optimization is insufficient or its convergence is unstable, requiring an appropriate increase in its weight to strengthen the optimization. The weight fine-tuning amount is:

[0127] ;

[0128] in, This mechanism is designed for fine-tuning the step size hyperparameter. It performs fine-grained adjustments based on staged weights to ensure balanced progress across all optimization objectives.

[0129] In an exemplary embodiment, the student model is pruned based on the importance scores of each channel in the student model and the current pruning rate, including: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; determining the number of channels to be pruned based on the current pruning rate; sorting each channel in the student model from high to low according to its importance score, and pruning the channels that are ranked lower in the sorting result and whose number corresponds to the number of channels to be pruned.

[0130] Pruning is performed gradually during training, employing a progressive pruning strategy. The pruning rate increases linearly or exponentially with the number of training steps, rather than being completed all at once, allowing the model sufficient time to adapt to structural changes. Overall pruning rate... With training steps The changes are as follows:

[0131] ;

[0132] in, To achieve the target pruning rate, This represents the total number of steps in the pruning stage. To control the hyperparameters of pruning speed, This represents the current pruning rate.

[0133] After determining the number of channels to be pruned, the channels in the student model are sorted from highest to lowest importance score. Channel importance score is obtained by comprehensively evaluating the weight magnitude, gradient information, and activation values ​​of the output feature map for each channel; a higher score indicates a more important channel. Based on the number of channels to be pruned in the current step, channels corresponding to that number are selected from the ranking results and marked as redundant channels to be pruned, and then removed from the network structure.

[0134] As the number of training steps increases, the pruning rate gradually increases, and the number of channels to be pruned increases progressively until the target pruning rate is reached. This progressive pruning method allows the model sufficient time to adapt to structural changes after each pruning, avoiding sudden changes in accuracy caused by a large-scale pruning all at once. The pruning operation is performed simultaneously with knowledge distillation and quantization-aware training, and all three share the same loss function and optimization process, eliminating the need for an additional separate retraining phase after pruning.

[0135] In this embodiment, a progressive pruning strategy and a joint training paradigm are adopted. Pruning is carried out gradually during the training process and is optimized in conjunction with distillation and quantization. This eliminates the need for extensive retraining after pruning, effectively controls training costs, and significantly improves the efficiency of building lightweight models.

[0136] In an exemplary embodiment, the student model is pruned based on the importance scores of each channel in the student model and the current pruning rate. This includes: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; assigning a corresponding pruning rate ratio to each network layer according to the type of different network layers in the student model; determining the number of channels to be pruned in the current training step for each network layer based on the current pruning rate and the corresponding pruning rate ratio for each network layer; and for each network layer, sorting the channels in the network layer according to their importance scores from high to low, and pruning the channels that are ranked lower in the sorting result and whose number corresponds to the number of channels to be pruned.

[0137] The pruning rate is determined based on the current training progress and differentiated according to network layer type, with pruning operations performed on each layer separately. Specifically, the current pruning rate is determined using a progressive strategy based on the ratio of the current training steps to the preset total training steps and the preset target pruning rate. The current pruning rate increases non-linearly with the number of training steps, ensuring that the pruning process is completed gradually during training, avoiding accuracy collapse caused by a large-scale pruning all at once. The target pruning rate is the expected total pruning ratio after training, while the current pruning rate is the pruning ratio based on which the actual pruning operation is performed in the current training step.

[0138] After determining the current pruning rate, a corresponding pruning rate ratio is assigned to each network layer according to the type of different network layers in the student model. For example, a relatively low pruning rate ratio is set for the shallow feature extraction layer in the visual encoder to preserve its ability to extract fine-grained features of local textures and edges; a relatively high pruning rate ratio is set for the deep semantic abstraction layer in the visual encoder. Similarly, a relatively low pruning rate ratio is set for the self-attention layer in the language decoder to preserve its ability to model semantic relationships; a relatively high pruning rate ratio is set for the feedforward network layer in the language decoder to compress the number of parameters more significantly. The sum of the pruning rate ratios of all network layers is 1.

[0139] Based on the current pruning rate and the pruning rate ratio corresponding to each network layer, the number of channels to be pruned in the current training step for each network layer is calculated. For each network layer, the channels in that layer are sorted from high to low importance score. Based on the number of channels to be pruned in that layer, the lower-ranked channels corresponding to the number to be pruned are selected as the channels to be pruned and pruned from that network layer. Through this differentiated allocation method, different network layers participate in pruning with different degrees of compression, balancing the different needs of each layer for accuracy and compression efficiency while maintaining the overall pruning rate target.

[0140] Understandably, multimodal power models typically consist of two heterogeneous components: a visual encoder and a language decoder. These components have different computational characteristics and accuracy sensitivities. The visual encoder is responsible for extracting fine-grained visual features from images and is highly sensitive to the local structure of convolutional and attention layers. The language decoder, on the other hand, is responsible for semantic understanding and text generation, requiring strong sequence modeling capabilities. Using a uniform pruning or quantization strategy may result in over-compression of the visual encoder, leading to a loss of fine-grained perceptual capabilities, or under-compression of the language decoder, failing to achieve the expected acceleration.

[0141] Based on this, in one optional implementation, different pruning rates are set for the visual encoder and language decoder of the multimodal model to adapt to their heterogeneous characteristics. For the visual encoder, a relatively mild pruning strategy is adopted to preserve the ability to extract fine-grained visual features. Specifically, the network layers of the visual encoder are divided into shallow feature extraction layers and deep semantic abstraction layers. The shallow feature extraction layer is responsible for extracting fine-grained features such as local texture, edges, and colors of the image, which is crucial for fine tasks such as defect detection. A first pruning rate is set, which is relatively low, for example, 10% to 20%. The deep semantic abstraction layer is responsible for extracting high-level semantic features of the image, which plays a major role in the overall understanding of visual content. A second pruning rate is set, which is relatively higher than the first pruning rate, for example, 20% to 30%. For the language decoder, a moderate pruning strategy is adopted to maintain semantic understanding capabilities. Specifically, the network layers of the language decoder are divided into self-attention layers and feedforward network layers. The self-attention layer is responsible for modeling the semantic relationships between elements in the sequence, playing a decisive role in the quality of language understanding and text generation. A third pruning rate is set, which is relatively low, for example, 15% to 25%. The feedforward network layer contributes significantly to the number of model parameters but has a relatively small impact on the final semantic understanding accuracy. A fourth pruning rate is set, which is relatively higher than the third pruning rate, for example, 30% to 40%.

[0142] By setting the differential pruning rate as described above, the visual encoder retains more channels in the shallow layer to maintain fine-grained perception capabilities, the language decoder retains more attention heads in the self-attention layer to maintain semantic modeling capabilities, and the feedforward network layer, which has a larger number of parameters but a relatively smaller impact on accuracy, undertakes a higher compression ratio, thereby achieving a better balance between overall compression ratio and accuracy maintenance.

[0143] In an exemplary embodiment, the method further includes: obtaining the weight parameters of each channel in the student model, the gradient information of each channel, and the activation values ​​of the feature maps output by each channel; for each channel in the student model, the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation values ​​are weighted and summed to obtain the channel's importance score.

[0144] The importance of each channel is comprehensively evaluated based on indicators such as weight magnitude, gradient information, and feature map activation values, and a channel importance score is calculated. Specifically, the channel importance score... The calculation method is as follows:

[0145] ;

[0146] in, These are the weight parameters for channel c in the student model. This provides the gradient information for channel c. The activation value of the output feature map for channel c. , , These are the weighting coefficients for each indicator. Gradient information is obtained through backpropagation during training, while the activation values ​​of the feature maps are the outputs of each channel during forward propagation.

[0147] In one exemplary embodiment, multiple teacher models are prepared, each with different architectures or different training data distributions. For example, for a power transmission inspection scenario, three teacher models can be prepared: Teacher Model A is a large-scale visual model pre-trained on a general image dataset, which has advantages in general visual feature extraction; Teacher Model B is a model of the same or larger scale fine-tuned on a power equipment defect dataset, which has expertise in defect recognition tasks; and Teacher Model C is a multimodal model that integrates infrared and visible light modalities, which has advantages in multimodal fusion understanding.

[0148] Multi-teacher knowledge distillation is used to integrate and transfer the strengths of each teacher's capabilities to the student model (e.g., lightweight Qwen-VL-Tiny or structurally simplified architectures like MobileNet and ViT-Tiny). Specifically, for each input sample, the soft-label outputs of each teacher's model and the student's model are calculated separately, and the KL divergence for each group is also calculated. To leverage the complementary strengths of multiple teacher models in power defect identification, the contribution of each teacher can be adaptively adjusted using learnable fusion weights.

[0149] A distillation temperature parameter is introduced to control the smoothness of knowledge transfer. The higher the temperature value, the smoother the distribution of soft labels and the richer the inter-category relationship information contained. Optionally, a progressive temperature scheduling strategy is adopted, setting a higher temperature value in the early stage of training to fully transfer inter-category relationship knowledge, and gradually decreasing the temperature value as training progresses to enhance the category discrimination ability.

[0150] Feature maps of each teacher model and student model at a specific intermediate layer are extracted separately, and the mean squared error is calculated as the feature distillation loss to help the student model learn the feature representation ability of the teacher model.

[0151] During the training of the student model, channel sparsity is implemented. A channel sparsity regularization term is added to the loss function to make the weights of unimportant channels approach zero. The importance of each channel is comprehensively evaluated based on indicators such as weight magnitude, gradient information, and feature map activation values, and a channel importance score is calculated.

[0152] For the visual encoder, a relatively mild pruning strategy is adopted to preserve the ability to extract fine-grained visual features. Shallow features in the visual encoder are more important for local texture and edge detection, so a relatively lower pruning rate is set; deep features are more important for semantic abstraction, so a relatively higher pruning rate can be set. For the language decoder, a moderate pruning strategy is adopted to maintain semantic understanding. The self-attention head in the language decoder is crucial for semantic relation modeling, so a relatively low pruning rate is set; feedforward network layers contribute significantly to the number of parameters but have a relatively small impact on accuracy, so a relatively high pruning rate can be set.

[0153] Channels are sorted by importance score, and channels with scores below a preset threshold are pruned. Pruning is performed gradually during training, employing a progressive pruning strategy. The target pruning rate gradually increases from zero to the preset target value during the pruning phase, rather than being completed all at once, allowing the model sufficient time to adapt to structural changes.

[0154] The pruned model is trained with quantization awareness to simulate quantization error during training. Quantization nodes are inserted into the network's forward propagation, treating the weights as already quantized. The quantization step size is updated as a learnable parameter during training to adaptively adapt to the distribution characteristics of parameters in each layer.

[0155] For the visual encoder, a higher precision quantization configuration is used to preserve the accuracy of fine-grained feature extraction, as the visual encoder is relatively sensitive to quantization noise. For the language decoder, the quantization precision configuration is flexibly selected based on accuracy requirements and deployment scenarios. The language decoder has a relatively high tolerance for quantization noise, and a lower precision quantization configuration can be used in scenarios with strict inference latency requirements. The quantization step size of different modules is learned independently, and each module adapts to the distribution characteristics of its own parameters.

[0156] The quantization-aware training and distillation loss are jointly optimized. While performing quantization simulation, the soft labels of the teacher model are used to provide supervision signals to compensate for the accuracy loss caused by quantization.

[0157] An adaptive loss fusion mechanism is designed to weight and fuse distillation loss, pruning loss, quantization loss, and task loss. The weights of the total loss function are dynamically adjusted as the training phase progresses. The phase division is based on the current training step number. Total training steps proportion :

[0158] Phase 1 (Knowledge Transfer Phase, 0≤) <0.25): Knowledge transfer is the primary factor, making distillation loss the dominant factor. The weights are set to... , , , This helps student models fully absorb the knowledge from teacher models.

[0159] Second stage (pruning optimization stage, 0.25≤ <0.5): Primarily focuses on channel sparsity, increasing the weight of pruning loss. The weights are set to... , , , This helps reduce the weight of unimportant channels to near zero, preparing for subsequent pruning.

[0160] Phase 3 (Quantization and Adaptation Phase, 0.5≤ <0.75): Prioritizing minimizing quantization error, increase the weight of quantization loss. The weight is set to... , , , This allows the model parameters to be fully adapted to low-precision representation.

[0161] Phase 4 (Joint Fine-tuning Phase, 0.75≤ ≤1.0): Primarily optimizes task accuracy while maintaining quantitative adaptation. Effect weights are set to... , , , This allows the model to achieve optimal task accuracy while maintaining a lightweight design.

[0162] Optionally, during each training stage, the weights can be adaptively fine-tuned based on the relative magnitude of the gradient variance of each loss term. Loss terms with large gradient variances indicate that their optimization is not yet sufficient, and their weights can be appropriately increased to ensure balanced progress of each optimization objective. The division of each stage and the allocation of weights can be flexibly configured according to the specific task and model size, and the transition between stages can be smoothed using linear interpolation.

[0163] After multi-stage joint training, the low-rank matrix is ​​merged with the original weights, the pruned channels are removed from the network structure, the quantization parameters are bound to the model weights, and a lightweight student model is output.

[0164] For deployment and adaptation, the model supports exporting to multiple formats and is compatible with inference frameworks for various domestically produced chips. The total number of parameters and inference latency of the output model both meet the requirements for real-time inference at the edge.

[0165] Through the above processing, a lightweight model for power scenarios is efficiently constructed, significantly reducing the number of model parameters and inference latency while maintaining high task accuracy.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0167] Based on the same inventive concept, this application also provides a lightweight model construction apparatus for implementing the lightweight model construction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more lightweight model construction apparatus embodiments provided below can be found in the limitations of the lightweight model construction method described above, and will not be repeated here.

[0168] In one exemplary embodiment, such as Figure 2 As shown, a lightweight model construction apparatus is provided, comprising:

[0169] The knowledge distillation module 202 is used to input the training samples of the current training step into the student model and multiple teacher models respectively, enabling the student model and multiple teacher models to perform forward propagation. The student model is the network model to be trained, and the teacher models are pre-trained network models with frozen parameters. Each teacher model has a different network architecture or a different distribution of pre-trained data. The task loss is calculated based on the difference between the prediction results output by the student model and the true labels of the training samples. The distillation loss is calculated based on the soft label outputs of each teacher model, the soft label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model.

[0170] The structured pruning module 204 is used to calculate the pruning loss based on the weights of each channel in the student model, and to prune the student model according to the importance scores of each channel in the student model and the current pruning rate.

[0171] The quantization perception training module 206 is used to perform simulated quantization operations on the student model based on the quantization step size, and to calculate the quantization loss based on the weights of the student model before and after the simulated quantization operation.

[0172] The adaptive loss fusion module 208 is used to obtain the corresponding set of weight coefficients according to the training stage corresponding to the current training step. The set of weight coefficients includes the weight coefficients of task loss, distillation loss, pruning loss, and quantization loss. Based on the set of weight coefficients, the task loss, distillation loss, pruning loss, and quantization loss are weighted and summed to obtain the total loss. Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model, and the model enters the next training step and continues training until the preset training completion conditions are met.

[0173] The lightweight model output module 210 is used to remove the pruned channels from the trained student model after training is completed, and bind the quantization step size obtained during training to the model weights of the trained student model to output the lightweight model.

[0174] The lightweight model construction device described above employs a multi-teacher knowledge distillation method. Leveraging the complementary capabilities of multiple teacher models with different architectures or pre-training data distributions across various sub-tasks, the advantages of each teacher are integrated and transferred to the student model. This effectively compensates for the accuracy loss caused by pruning and quantization, significantly improving the learning performance and final accuracy of the lightweight model. A joint lightweight framework is constructed, co-optimizing knowledge distillation, structured pruning, and quantization compression. Distillation loss, pruning loss, and quantization loss are weighted and fused in a unified loss function. Through a single backpropagation, the network parameters, channel sparsity parameters, and quantization step size of the student model are simultaneously optimized, overcoming the problems of fragmented objectives and amplified error accumulation in traditional serial processing. Knowledge distillation provides accuracy supervision for pruning and quantization, maintaining task performance during compression. Pruning reduces the computational overhead of quantization by decreasing the number of channels. Quantization further compresses the pruned model size through low-precision representation. This achieves multi-objective joint optimization of compression ratio, accuracy preservation, and inference speed. Based on the adaptive loss fusion mechanism, the problem that the fixed weight strategy cannot achieve optimal configuration at each stage is overcome, which significantly improves the convergence efficiency of joint training and the accuracy of the final model.

[0175] In an exemplary embodiment, the training stage corresponding to the current training step includes any one of the following: knowledge transfer stage, pruning optimization stage, quantization adaptation stage, and joint fine-tuning stage; the set of weight coefficients corresponding to the knowledge transfer stage includes... , , , In the set of weight coefficients corresponding to the pruning optimization stage, , , , In the set of weight coefficients corresponding to the quantization adaptation stage, , , , In the set of weight coefficients corresponding to the joint fine-tuning phase, , , , ;in, This is the task loss weighting coefficient. This is the distillation loss weighting coefficient. This is the pruning loss weighting coefficient. This is the weighting coefficient for quantifying the loss.

[0176] In an exemplary embodiment, the adaptive loss fusion module 208 is further configured to obtain the ratio of the current training step to the total training steps; match the ratio with a plurality of preset ratio ranges; the plurality of ratio ranges correspond to different training stages; determine the training stage corresponding to the current training step based on the matching result; and obtain the set of weight coefficients corresponding to the determined training stage from a plurality of preset weight coefficient sets.

[0177] In an exemplary embodiment, the adaptive loss fusion module 208 is further configured to calculate the gradient variance of the loss term relative to the corresponding parameter for each loss term among the task loss, distillation loss, pruning loss, and quantization loss; calculate the weight fine-tuning amount corresponding to the loss term based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms; and adjust the weight coefficients corresponding to the loss terms in the weight coefficient set based on the weight fine-tuning amount to obtain the final weight coefficients corresponding to the loss terms.

[0178] In an exemplary embodiment, the structured pruning module 204 is further configured to determine the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; determine the number of channels to be pruned based on the current pruning rate; sort the channels in the student model from high to low according to their importance scores; and prune the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0179] In an exemplary embodiment, the structured pruning module 204 is further configured to: determine the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; assign a corresponding pruning rate ratio to each network layer according to the type of different network layers in the student model; determine the number of channels to be pruned in each network layer in the current training step based on the current pruning rate and the corresponding pruning rate ratio of each network layer; and for each network layer, sort the channels in the network layer according to their importance scores from high to low, and prune the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0180] In an exemplary embodiment, the structured pruning module 204 is further configured to obtain the weight parameters of each channel in the student model, the gradient information of each channel, and the activation values ​​of the feature maps output by each channel; for each channel in the student model, the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation values ​​are weighted and summed to obtain the channel's importance score.

[0181] Each module in the aforementioned lightweight model construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0182] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for constructing a lightweight model.

[0183] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: inputting training samples of the current training step into a student model and multiple teacher models respectively, causing the student model and the multiple teacher models to perform forward propagation; the student model is a network model to be trained, and the teacher models are pre-trained network models with frozen parameters, each teacher model having a different network architecture or a different pre-training data distribution; calculating a task loss based on the difference between the prediction results output by the student model and the true labels of the training samples; calculating a distillation loss based on the soft-label outputs of each teacher model, the soft-label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model; calculating a pruning loss based on the weights of each channel in the student model, and based on the importance scores of each channel in the student model, and... The current pruning rate is used to prune the student model. Simulated quantization is performed on the student model based on the quantization step size, and the quantization loss is calculated based on the weights of the student model before and after the simulated quantization. A set of weight coefficients is obtained according to the training stage corresponding to the current training step. This set includes weight coefficients for task loss, distillation loss, pruning loss, and quantization loss. The task loss, distillation loss, pruning loss, and quantization loss are weighted and summed based on the weight coefficient set to obtain the total loss. Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model. The next training step is then initiated and training continues until the preset training completion conditions are met. After training, the pruned channels are removed from the trained student model, and the quantization step size obtained during training is bound to the model weights of the trained student model to output a lightweight model.

[0185] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the ratio of the current training step to the total training steps; matching the ratio with a plurality of preset ratio ranges; the plurality of ratio ranges correspond to different training stages; determining the training stage corresponding to the current training step based on the matching results; and obtaining the set of weight coefficients corresponding to the determined training stage from a plurality of preset weight coefficient sets.

[0186] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each loss term among the task loss, distillation loss, pruning loss, and quantization loss, calculate the gradient variance of the loss term relative to the corresponding parameter; calculate the weight fine-tuning amount corresponding to the loss term based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms; and adjust the weight coefficients corresponding to the loss terms in the weight coefficient set based on the weight fine-tuning amount to obtain the final weight coefficients corresponding to the loss terms.

[0187] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; determining the number of channels to be pruned based on the current pruning rate; sorting the channels in the student model from high to low according to their importance scores, and pruning the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0188] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; assigning a corresponding pruning rate ratio to each network layer according to the type of different network layers in the student model; determining the number of channels to be pruned in the current training step for each network layer based on the current pruning rate and the corresponding pruning rate ratio for each network layer; and for each network layer, sorting the channels in the network layer according to their importance scores from high to low, and pruning the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0189] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the weight parameters of each channel in the student model, the gradient information of each channel, and the activation value of the feature map output of each channel; for each channel in the student model, the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation value are weighted and summed to obtain the channel's importance score.

[0190] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: inputting training samples of the current training step into a student model and multiple teacher models respectively, causing the student model and multiple teacher models to perform forward propagation; the student model is a network model to be trained, and the teacher models are pre-trained network models with frozen parameters, each teacher model having a different network architecture or a different distribution of pre-trained data; calculating the task loss based on the difference between the prediction results output by the student model and the true labels of the training samples; calculating the distillation loss based on the soft-label outputs of each teacher model, the soft-label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model; calculating the pruning loss based on the weights of each channel in the student model, and based on the importance scores of each channel in the student model and the current pruning rate. The student model is pruned; a simulated quantization operation is performed on the student model based on the quantization step size, and the quantization loss is calculated based on the weights of the student model before and after the simulated quantization operation; according to the training stage corresponding to the current training step, the corresponding set of weight coefficients is obtained; the set of weight coefficients includes the task loss weight coefficient, distillation loss weight coefficient, pruning loss weight coefficient, and quantization loss weight coefficient; based on the set of weight coefficients, the task loss, distillation loss, pruning loss, and quantization loss are weighted and summed to obtain the total loss; backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model, and the next training step is entered to continue training until the preset training completion conditions are met; after training is completed, the pruned channels are removed from the trained student model, and the quantization step size obtained from the training is bound to the model weights of the trained student model to output a lightweight model.

[0191] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the ratio of the current training step to the total training steps; matching the ratio with a plurality of preset ratio ranges; the plurality of ratio ranges correspond to different training stages; determining the training stage corresponding to the current training step based on the matching results; and obtaining a set of weight coefficients corresponding to the determined training stage from a plurality of preset weight coefficient sets.

[0192] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each loss term among the task loss, distillation loss, pruning loss, and quantization loss, calculate the gradient variance of the loss term relative to the corresponding parameter; calculate the weight fine-tuning amount corresponding to the loss term based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms; and adjust the weight coefficients corresponding to the loss terms in the weight coefficient set based on the weight fine-tuning amount to obtain the final weight coefficients corresponding to the loss terms.

[0193] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; determining the number of channels to be pruned based on the current pruning rate; sorting the channels in the student model from high to low according to their importance scores, and pruning the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0194] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the current pruning rate based on the ratio of the current training step to the total training steps and the target pruning rate; assigning a corresponding pruning rate ratio to each network layer according to the type of different network layers in the student model; determining the number of channels to be pruned in the current training step for each network layer based on the current pruning rate and the corresponding pruning rate ratio for each network layer; and for each network layer, sorting the channels in the network layer according to their importance scores from high to low, and pruning the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be pruned.

[0195] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the weight parameters of each channel in the student model, the gradient information of each channel, and the activation value of the feature map output of each channel; for each channel in the student model, the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation value are weighted and summed to obtain the channel's importance score.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0199] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for constructing a lightweight model, characterized in that, The method includes: The training samples of the current training step are input into the student model and multiple teacher models respectively, so that the student model and the multiple teacher models perform forward propagation; the student model is a network model to be trained, and the teacher model is a network model that has been pre-trained and whose parameters are frozen. Each teacher model has a different network architecture or a different distribution of pre-training data. The task loss is calculated based on the difference between the prediction results output by the student model and the true labels of the training samples. Distillation loss is calculated based on the soft-label output of each teacher model, the soft-label output of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model. Based on the weights of each channel in the student model, the pruning loss is calculated, and the student model is pruned according to the importance scores of each channel in the student model and the current pruning rate. The student model is subjected to simulated quantization based on the quantization step size, and the quantization loss is calculated based on the weights of the student model before and after the simulated quantization operation. Based on the training stage corresponding to the current training step, obtain the corresponding set of weight coefficients; the set of weight coefficients includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients; Based on the set of weight coefficients, the task loss, the distillation loss, the pruning loss, and the quantization loss are weighted and summed to obtain the total loss; Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters and quantization step size of the student model, and then the training continues until the preset training completion conditions are met. After training is complete, the pruned channels are removed from the trained student model, and the quantization step size obtained during training is bound to the model weights of the trained student model to output a lightweight model.

2. The method according to claim 1, characterized in that, The training phase corresponding to the current training step includes any one of the following: knowledge transfer phase, pruning and optimization phase, quantization adaptation phase, and joint fine-tuning phase. The set of weight coefficients corresponding to the knowledge transfer stage , , , ; The set of weight coefficients corresponding to the pruning optimization stage. , , , ; The set of weight coefficients corresponding to the quantization adaptation stage , , , ; The set of weight coefficients corresponding to the joint fine-tuning stage , , , ; in, This is the task loss weighting coefficient. This is the distillation loss weighting coefficient. This is the pruning loss weighting coefficient. This is the weighting coefficient for quantifying the loss.

3. The method according to claim 2, characterized in that, The step of obtaining the corresponding set of weight coefficients based on the training stage corresponding to the current training step includes: Obtain the ratio of the current training step to the total training steps; The ratio value is matched with multiple preset ratio ranges; each ratio range corresponds to a different training stage. Determine the training stage corresponding to the current training step based on the matching results; Obtain the set of weight coefficients corresponding to the determined training stage from multiple preset sets of weight coefficients.

4. The method according to claim 1, characterized in that, Before obtaining the total loss by weighted summation of the task loss, distillation loss, pruning loss, and quantization loss based on the set of weight coefficients, the method further includes: For each of the task loss, distillation loss, pruning loss, and quantization loss, calculate the gradient variance of the loss term relative to the corresponding parameter; The weight adjustment amount corresponding to the loss term is calculated based on the ratio of the gradient variance of the loss term to the sum of the gradient variances of all loss terms. Based on the weight fine-tuning amount, the weight coefficients corresponding to the loss item in the weight coefficient set are adjusted to obtain the final weight coefficients corresponding to the loss item.

5. The method according to any one of claims 1 to 4, characterized in that, The step of pruning the student model based on the importance scores of each channel and the current pruning rate includes: The current pruning rate is determined based on the ratio of the current training step to the total training steps and the target pruning rate. The number of channels to be pruned is determined based on the current pruning rate; The channels in the student model are sorted from high to low according to their importance scores, and the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be removed are removed.

6. The method according to any one of claims 1 to 4, characterized in that, The step of pruning the student model based on the importance scores of each channel and the current pruning rate includes: The current pruning rate is determined based on the ratio of the current training step to the total training steps and the target pruning rate. Based on the different types of network layers in the student model, assign corresponding pruning ratios to each network layer; Based on the current pruning rate and the pruning rate ratio corresponding to each network layer, determine the number of channels to be pruned in the current training step for each network layer. For each network layer, the channels in the network layer are sorted from high to low according to their importance scores, and the channels that are ranked lower in the sorting results and whose number corresponds to the number of channels to be removed are removed.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the weight parameters of each channel, the gradient information of each channel, and the activation values ​​of the feature maps output by each channel in the student model; For each channel in the student model, the importance score of the channel is obtained by weighted summing of the L2 norm of the channel's weight parameters, the L2 norm of the channel's gradient information, and the L2 norm of the channel's feature map activation values.

8. A lightweight model construction apparatus, characterized in that, The device includes: The knowledge distillation module is used to input the training samples of the current training step into the student model and multiple teacher models respectively, enabling the student model and the multiple teacher models to perform forward propagation. The student model is a network model to be trained, and the teacher models are pre-trained network models with frozen parameters. Each teacher model has a different network architecture or a different distribution of pre-trained data. The task loss is calculated based on the difference between the prediction results output by the student model and the true labels of the training samples. The distillation loss is calculated based on the soft label outputs of each teacher model, the soft label outputs of the student model, the intermediate layer features of each teacher model, and the intermediate layer features of the student model. The structured pruning module is used to calculate the pruning loss based on the weights of each channel in the student model, and to prune the student model according to the importance scores of each channel and the current pruning rate. The quantization perception training module is used to perform simulated quantization operations on the student model based on the quantization step size, and to calculate the quantization loss based on the weights of the student model before and after the simulated quantization operation. An adaptive loss fusion module is used to obtain a set of weight coefficients according to the training stage corresponding to the current training step. The set of weight coefficients includes task loss weight coefficients, distillation loss weight coefficients, pruning loss weight coefficients, and quantization loss weight coefficients. Based on the set of weight coefficients, the task loss, distillation loss, pruning loss, and quantization loss are weighted and summed to obtain the total loss. Backpropagation is performed based on the total loss to update the model parameters, channel sparsity parameters, and quantization step size of the student model, and the model enters the next training step and continues training until the preset training completion conditions are met. The lightweight model output module is used to remove the pruned channels from the trained student model after training is completed, and bind the quantization step size obtained during training to the model weights of the trained student model to output the lightweight model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images