Model compression method and system for adaptively adjusting knowledge distillation temperature, and medium

Abstract

The application belongs to the technical field of model compression, and discloses a model compression method and system for adaptively adjusting knowledge distillation temperature, and a medium. Image samples are collected, and data cleaning is performed on abnormal data and missing data in the image samples, including directly discarding the abnormal data and restoring the missing data by using a nearest neighbor interpolation or a bilinear interpolation method. A Mosaic-4 data enhancement method is used to randomly perform flipping, scaling, cropping and other operations on the image samples, and four transformed images are randomly selected for splicing, so as to enhance the image samples. A minimum adaptive distillation loss function is established, the knowledge of a teacher model is transferred to a student model, and knowledge distillation is realized. The student model is trained by using a training set, an optimal total loss function is determined, and model compression is realized. The application better balances the transfer of the knowledge of the teacher model to the student model, and a lighter student model is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model compression, and particularly relates to a model compression method and system for adaptively adjusting knowledge distillation temperature and a medium. BACKGROUND

[0002] At present, with the increasing complexity of deep learning models, the parameter quantity and the calculation quantity thereof also increase, resulting in problems such as too large volume, slow calculation speed, high storage cost and the like in deploying deep learning models. Therefore, deep learning model compression technology becomes more and more important. Knowledge distillation is an effective deep learning model compression technology, which constructs a framework of guiding a student model by a teacher model, that is, knowledge of a complex teacher model is transferred to a lightweight student model in a knowledge distillation manner to realize model compression. The technology can reduce the model parameter quantity and the calculation quantity while maintaining high accuracy.

[0003] In order to better transfer the knowledge of the teacher model to the student model, the common method is to manually set the distillation temperature parameter. However, the setting of the distillation temperature parameter often requires rich experience and a large amount of debugging, which is not conducive to the wide application of knowledge distillation.

[0004] Through the above analysis, the problems and defects of the prior art are that the setting of the distillation temperature parameter often requires rich experience and a large amount of debugging, which is not conducive to the wide application of knowledge distillation. SUMMARY

[0005] In view of the problems of the prior art, the application provides a model compression method and system for adaptively adjusting knowledge distillation temperature and a medium.

[0006] The application is implemented as follows. A model compression method for adaptively adjusting knowledge distillation temperature comprises the following steps:

[0007] Firstly, image samples are collected, and abnormal data and missing data in the image samples are cleaned, including directly discarding the abnormal data and restoring the missing data by using a nearest neighbor interpolation or a bilinear interpolation method;

[0008] Secondly, a Mosaic-4 data enhancement method is used to randomly perform operations such as flipping, scaling and cropping on the image samples, and four randomly selected images after transformation are spliced to enhance the image samples;

[0009] Thirdly, a minimum adaptive distillation loss function is established to complete the transfer of knowledge of the teacher model to the student model, realize knowledge distillation, train the student model by using a training set, determine an optimal total loss function, and realize model compression.

[0010] Furthermore, after the second step, all image samples need to be labeled and divided into training and test sets in a 7:3 ratio.

[0011] Furthermore, after dividing the training and test sets, a ResNet101 model needs to be selected as the teacher model, and trained using the training set, initialized with a distillation temperature T. T The activation function Softmax is used to determine the soft label L1 of the teacher model;

[0012] The ResNet18 model was selected as the student model and trained using the training set. The model was adaptively adjusted to accommodate the distillation temperature T. S The activation function Softmax is used to determine the soft label L2 of the student model.

[0013] Further includes:

[0014] (1) Using sharpness S T This indicates the degree of dispersion of the i-th type of output results from the teacher model;

[0015]

[0016] Among them, TP i This represents the output of the i-th class of the teacher model;

[0017] (2) Using sharpness S S This indicates the degree of dispersion of the i-th type of output results in the student model;

[0018]

[0019] Among them, SP i This represents the output of the i-th class of the student model;

[0020] (3) Using G T-S The Taylor expansion of the expression that measures the difference between the outputs of the teacher model and the student model is:

[0021]

[0022] Where k represents the number of image samples of the i-th class;

[0023] (4) Assumption G T-S It can be represented as:

[0024]

[0025] in, It is the variance of the teacher model output, used express; It is the variance of the student model output, used express;

[0026] (5)G T-S This can be further expressed as:

[0027]

[0028] (6) Let G T-S =0, determine the distillation temperature T of the teacher model. T and the student model distillation temperature T S The relationship between std T / T T =std S / T S .

[0029] Furthermore, the third step includes: minimizing the adaptive distillation loss function L. AD This minimizes the divergence between L1 and L2, enabling the transfer of knowledge from the teacher model to the student model and achieving knowledge distillation.

[0030] The student model was trained using the training set, and the activation function Softmax was selected to determine the hard label L3 of the student model.

[0031] Minimize the loss function L of the student model S This minimizes the divergence between L3 and the true label;

[0032] According to L AD and L S Determine the optimal total loss function L. total This enables model compression.

[0033] Further includes:

[0034] 1) In the knowledge distillation process, utilizing a distillation temperature T T The Softmax activation function is used to evaluate the output TP of the j-th sample in the i-th class of the teacher model. ij Perform the activation operation and determine T. ij ;

[0035]

[0036] 2) Utilizing a distillation temperature T S The output SP of the Softmax activation function for the j-th sample of the i-th class in the student model. ij Perform the activation operation and determine S ij ;

[0037]

[0038] 3) Adaptive distillation loss function L ADRepresented as:

[0039]

[0040] 4) Loss function L of the student model S Represented as:

[0041]

[0042] Among them, y ij This represents the true label value of the j-th sample in the i-th class;

[0043] 5) Total loss function L total Represented as:

[0044] L total =λL AD +(1-λ)L S

[0045] Where λ represents the weighting factor;

[0046] 6) Initialize the distillation temperature T of the teacher model. T According to std T / T T =std S / T S Relationship and total loss function L total The optimal principle is used to adaptively adjust the distillation temperature T of the student model. S ;

[0047] 7) When L total When the loss factor α is less than the preset value, determine the optimal distillation temperature T for the student model. S =t, thereby completing the transfer of knowledge from the teacher model to the student model, and finally realizing model compression that adaptively adjusts the knowledge distillation temperature.

[0048] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the model compression method for adaptively adjusting the knowledge distillation temperature.

[0049] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the model compression method for adaptively adjusting the knowledge distillation temperature.

[0050] Another objective of this invention is to provide an information data processing terminal for implementing the model compression method of adaptively adjusting the knowledge distillation temperature.

[0051] Another object of the present invention is to provide a model compression system for adaptively adjusting the knowledge distillation temperature, which implements the aforementioned model compression method for adaptively adjusting the knowledge distillation temperature. The model compression system for adaptively adjusting the knowledge distillation temperature includes:

[0052] The data cleaning module is used to collect image samples and clean abnormal and missing data in the image samples, including directly discarding abnormal data and recovering missing data using nearest neighbor interpolation or bilinear interpolation.

[0053] The sample processing module is used to randomly perform operations such as flipping, scaling, and cropping on image samples using the Mosaic-4 data augmentation method, and then randomly select four transformed images to stitch together, thereby enhancing the image samples.

[0054] The model compression module is used to establish a loss function that minimizes the adaptive distillation function, completes the knowledge transfer from the teacher model to the student model, and realizes knowledge distillation; it uses the training set to train the student model, determines the optimal total loss function, and realizes model compression.

[0055] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0056] First, this invention uses a model compression method that adaptively adjusts the knowledge distillation temperature to better balance the transfer of knowledge from the teacher model to the student model, thereby obtaining a lighter student model. Compared with traditional knowledge distillation methods, this method can reduce the dependence on the structure of the teacher model, improve the generalization ability of the student model, and thus enhance robustness. This method can dynamically adjust the distillation temperature to adapt to the differences between the teacher model and the student model, thereby improving the accuracy of the student model.

[0057] Secondly, this invention proposes a model compression method that adaptively adjusts the knowledge distillation temperature, which can automatically adjust the distillation temperature parameters, avoiding the trouble of manually adjusting the distillation temperature parameters, and enabling the lightweight student model to maintain high accuracy.

[0058] Third, does the technical solution of this invention overcome technical bias?

[0059] Adaptability: The adaptive knowledge distillation temperature adjustment method can automatically learn the most suitable distillation temperature for the current task and data without manual setting. It can dynamically adjust the distillation temperature according to the characteristics of the model and data, thereby better adapting to different scenarios and task requirements.

[0060] Improving robustness: The adaptive knowledge distillation temperature method can improve the robustness of the model. Since different models and data may have different characteristics, manually set distillation temperatures may not be suitable for all situations. The adaptive temperature method, however, can dynamically adjust according to the actual situation, thereby improving the model's adaptability and robustness.

[0061] Improved performance: The adaptive knowledge distillation temperature method can help optimize model training. By automatically adjusting the distillation temperature based on the characteristics of the model and data, it can better balance knowledge transfer between the teacher and student models, thereby improving the performance and generalization ability of the student model.

[0062] Reduced parameter tuning work: Manually setting the distillation temperature requires a certain amount of parameter tuning work, extensive experience, and a lot of adjustments. Adaptive temperature methods, on the other hand, can automatically learn the appropriate temperature, reducing the workload and uncertainty of manual parameter tuning. Attached Figure Description

[0063] Figure 1 This is a flowchart of the adaptive adjustment of knowledge distillation temperature model compression method provided in this embodiment of the invention;

[0064] Figure 2 This is a block diagram of the adaptive adjustment of knowledge distillation temperature model compression method provided in the embodiments of the present invention;

[0065] Figure 3 This is a schematic diagram of the model compression method for adaptively adjusting the knowledge distillation temperature provided in this embodiment of the invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] like Figure 1 As shown, the adaptive adjustment of knowledge distillation temperature model compression method provided in this embodiment of the invention includes the following steps:

[0068] S101: Collect image samples and perform data cleaning on abnormal and missing data in the image samples, including directly discarding abnormal data and recovering missing data using nearest neighbor interpolation or bilinear interpolation.

[0069] S102: The Mosaic-4 data augmentation method is used to randomly flip, scale, and crop the image samples, and four transformed images are randomly selected and stitched together to enhance the image samples.

[0070] S103: Establish a loss function that minimizes the adaptive distillation function to transfer knowledge from the teacher model to the student model, thus achieving knowledge distillation; train the student model using the training set to determine the optimal total loss function and achieve model compression.

[0071] The steps of the model compression method provided in this embodiment of the invention are as follows:

[0072] 1) Data cleaning: After collecting image samples, abnormal and missing data are cleaned. Abnormal data is discarded directly, and missing data is recovered using nearest neighbor interpolation or bilinear interpolation to ensure the quality and integrity of the samples.

[0073] 2) Data augmentation: The Mosaic-4 data augmentation method is used to randomly flip, scale, and crop the image samples, and four transformed images are randomly selected and stitched together to increase the number and diversity of samples and improve the generalization ability of the model.

[0074] 3) Knowledge Distillation: An adaptive distillation loss function is established to transfer knowledge from the teacher model to the student model, achieving knowledge distillation. Specifically, the output of the teacher model is used as the target label, and the difference between the output of the student model and the target label is used as the loss function. The parameters of the student model are optimized through backpropagation to better fit the output of the teacher model.

[0075] 4) Training the student model: The student model is trained using a three-year set of relational words to determine the optimal total loss function and achieve model compression. Specifically, the knowledge distillation loss function and other loss functions (such as the classification loss function) are weighted and summed, and the parameters of the student model are optimized through the backpropagation algorithm, so that it has a smaller model size and faster inference speed while maintaining high accuracy.

[0076] The adaptive knowledge distillation temperature model compression system provided in this embodiment of the invention includes:

[0077] The data cleaning module is used to collect image samples and clean abnormal and missing data in the image samples, including directly discarding abnormal data and recovering missing data using nearest neighbor interpolation or bilinear interpolation.

[0078] The sample processing module is used to randomly perform operations such as flipping, scaling, and cropping on image samples using the Mosaic-4 data augmentation method, and then randomly select four transformed images to stitch together, thereby enhancing the image samples.

[0079] The model compression module is used to establish a loss function that minimizes the adaptive distillation function, completes the knowledge transfer from the teacher model to the student model, and realizes knowledge distillation; it uses the training set to train the student model, determines the optimal total loss function, and realizes model compression.

[0080] The adaptive knowledge distillation temperature model compression system provided in this embodiment of the invention includes the following modules:

[0081] 1) Data cleaning module: After collecting image samples, it cleans the abnormal and missing data, discards the abnormal data directly, and uses nearest neighbor interpolation or bilinear interpolation to recover the missing data, so as to ensure the quality and integrity of the samples.

[0082] 2) Sample processing module: This module uses the Mosaic-4 data augmentation method to randomly flip, scale, and crop image samples, and then randomly selects four transformed images to stitch together, thereby increasing the number and diversity of samples and improving the model's generalization ability.

[0083] 3) Model Compression Module: This module is used to establish a loss function that minimizes the adaptive distillation process, enabling knowledge transfer from the teacher model to the student model, thus achieving knowledge distillation. Specifically, the teacher model's output is used as the target label, and the difference between the student model's output and the target label is used as the loss function. The backpropagation algorithm is used to optimize the student model's parameters, allowing it to better fit the teacher model's output. Furthermore, to adaptively adjust the distillation temperature, this module can dynamically adjust the distillation temperature based on the performance differences between the student and teacher models, making knowledge transfer more efficient.

[0084] The working principle of the adaptive knowledge distillation temperature model compression system provided in this embodiment of the invention is as follows: First, the collected image samples are cleaned and augmented to obtain high-quality and diverse training samples; then, knowledge distillation technology is used to transfer the knowledge of the teacher model to the student model to achieve model compression and acceleration; finally, by dynamically adjusting the distillation temperature, the knowledge transfer is made more efficient and the model performance is improved.

[0085] The adaptive adjustment knowledge distillation temperature model compression method provided in this invention has the following steps:

[0086] S1. Collect image samples and clean the abnormal and missing data in the image samples, including directly discarding abnormal data and using nearest neighbor interpolation or bilinear interpolation to recover missing data.

[0087] S2. The Mosaic-4 data augmentation method is used to randomly flip, scale, and crop the image samples, and four transformed images are randomly selected and stitched together to enhance the image samples.

[0088] S3. Label all image samples and divide them into training and test sets in a 7:3 ratio;

[0089] S4. Select the ResNet101 model as the teacher model and train it using the training set, initializing it with a distillation temperature T. T The activation function Softmax is used to determine the soft label L1 of the teacher model;

[0090] S5. Select the ResNet18 model as the student model and train it using the training set, adaptively adjusting the model based on the distillation temperature T. S The activation function Softmax is used to determine the soft label L2 of the student model;

[0091] S6. Minimize the adaptive distillation loss function L AD This minimizes the divergence between L1 and L2, enabling the transfer of knowledge from the teacher model to the student model and achieving knowledge distillation.

[0092] S7. Train the student model using the training set, select the activation function Softmax, and determine the hard label L3 of the student model.

[0093] S8. Minimize the loss function L of the student model. S This minimizes the divergence between L3 and the true label;

[0094] S9, according to L AD and L S Determine the optimal total loss function L. total To achieve model compression;

[0095] Steps S4 and S5 include the following steps:

[0096] S10, using sharpness S T This indicates the degree of dispersion of the i-th type of output results from the teacher model;

[0097]

[0098] Among them, TP i This represents the output of the i-th class of the teacher model;

[0099] S11, using sharpness S S This indicates the degree of dispersion of the i-th type of output results in the student model;

[0100]

[0101] Among them, SP i This represents the output of the i-th class of the student model;

[0102] S12, using G T-S The Taylor expansion of the expression that measures the difference between the outputs of the teacher model and the student model is:

[0103]

[0104] Where k represents the number of image samples of the i-th class;

[0105]

[0106] in, It is the variance of the teacher model output, used express; It is the variance of the student model output, used express;

[0107] S14, G T-S This can be further expressed as:

[0108]

[0109] S15, Order G T-S =0, determine the distillation temperature T of the teacher model. T and the student model distillation temperature T S The relationship between std T / T T =std S / T S ;

[0110] Steps S6-S9 include the following steps:

[0111] S16. In the knowledge distillation process, utilizing the distillation temperature T T The Softmax activation function is used to evaluate the output TP of the j-th sample in the i-th class of the teacher model. ij Perform the activation operation and determine T. ij ;

[0112]

[0113] S17, utilizing a distillation temperature T S The output SP of the Softmax activation function for the j-th sample of the i-th class in the student model. ij Perform the activation operation and determine S ij ;

[0114]

[0115] S18, Adaptive distillation loss function L AD It can be represented as:

[0116]

[0117] S19, Loss function L of the student model S It can be represented as

[0118]

[0119] Among them, y ij This represents the true label value of the j-th sample in the i-th class;

[0120] S20, Total Loss Function L total It can be represented as

[0121] L total =λL AD +(1-λ)L S

[0122] Where λ represents the weighting factor;

[0123] S21. Initialize the distillation temperature T of the teacher model. T According to std T / T T =std S / T S Relationship and total loss function L total The optimal principle is used to adaptively adjust the distillation temperature T of the student model. S ;

[0124] S22, when L total When the loss factor α is less than the preset value, determine the optimal distillation temperature T for the student model. S =t, thereby completing the transfer of knowledge from the teacher model to the student model, and finally realizing model compression that adaptively adjusts the knowledge distillation temperature.

[0125] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0126] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes them in conjunction with the data, charts and other information of the experimental process.

[0127] This invention uses sharpness to represent the dispersion of a certain type of output results in both the teacher and student models, and obtains the difference between the output results of the teacher and student models by subtracting the two. Then, using Taylor expansion and theoretical assumptions, the distillation temperature T of the teacher model is determined. T and the student model distillation temperature T S The relationship between std T / T T =std S / T S From this, we can deduce the relationship and obtain the distillation temperature T containing the teacher model. T Distillation temperature T of the student model S Therefore, when using knowledge distillation for model compression, it is only necessary to initialize the distillation temperature T of a teacher model. T Based on the relationship between the distillation temperatures of the teacher model and the student model, the distillation temperature T of the teacher model can be adaptively and dynamically adjusted. T and the student model distillation temperature T S This is the process of transferring knowledge from the teacher's model to the student's model.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A model compression method for adaptively adjusting the knowledge distillation temperature, characterized in that, Includes the following steps: The first step is to collect image samples and clean the abnormal and missing data in the image samples, including directly discarding abnormal data and using nearest neighbor interpolation or bilinear interpolation to recover missing data. The second step involves using the Mosaic-4 data augmentation method to randomly flip, scale, and crop the image samples, and then randomly selecting four transformed images to stitch together, thereby enhancing the image samples. The third step is to establish a loss function that minimizes the adaptive distillation function to complete the transfer of knowledge from the teacher model to the student model, thereby achieving knowledge distillation. The student model is trained using a training set to determine the optimal total loss function and achieve model compression. The third step includes: minimizing the adaptive distillation loss function L. AD This minimizes the divergence between L1 and L2, completing the transfer of knowledge from the teacher model to the student model and achieving knowledge distillation; where the teacher model has a soft label L1 and the student model has a soft label L2. The student model was trained using the training set, and the activation function Softmax was selected to determine the hard label L3 of the student model. Minimize the loss function L of the student model S This minimizes the divergence between L3 and the true label; According to L AD and L S Determine the optimal total loss function L. total To achieve model compression; Further includes: 1) In the knowledge distillation process, utilizing a distillation temperature T T The Softmax activation function is used to evaluate the output TP of the j-th sample in the i-th class of the teacher model. ij Perform the activation operation and determine T. ij ; 2) Utilizing a distillation temperature T S The output SP of the Softmax activation function for the j-th sample of the i-th class in the student model. ij Perform the activation operation and determine S ij ; 3) Adaptive distillation loss function L AD Represented as: 4) Loss function L of the student model S Represented as: Among them, y ij This represents the true label value of the j-th sample in the i-th class; 5) Total loss function L total Represented as: THE total =λL AD +(1-λ)L S Where λ represents the weighting factor; 6) Initialize the distillation temperature T of the teacher model. T According to std T / T T =std S / T S Relationship and total loss function L total The optimal principle is used to adaptively adjust the distillation temperature T of the student model. S ; 7) When L total When the loss factor α is less than the preset value, determine the optimal distillation temperature T for the student model. S =t, thereby completing the transfer of knowledge from the teacher model to the student model, and finally realizing model compression that adaptively adjusts the knowledge distillation temperature.

2. The model compression method for adaptively adjusting the knowledge distillation temperature as described in claim 1, characterized in that, After the second step, all image samples need to be labeled and divided into training and test sets in a 7:3 ratio.

3. The model compression method for adaptively adjusting the knowledge distillation temperature as described in claim 2, characterized in that, After dividing the training and test sets, a ResNet101 model needs to be selected as the teacher model, and it should be trained using the training set, initialized with a distillation temperature T. T The activation function Softmax is used to determine the soft label L1 of the teacher model; The ResNet18 model was selected as the student model and trained using the training set. The model was adaptively adjusted to accommodate the distillation temperature T. S The activation function Softmax is used to determine the soft label L2 of the student model.

4. The model compression method for adaptively adjusting the knowledge distillation temperature as described in claim 3, characterized in that, Further includes: (1) Using sharpness S T This indicates the degree of dispersion of the i-th type of output results from the teacher model; Among them, TP i This represents the output of the i-th class of the teacher model; (2) Using sharpness S S This indicates the degree of dispersion of the i-th type of output results in the student model; Among them, SP i This represents the output of the i-th class of the student model; (3) Using G T-S The Taylor expansion of the expression that measures the difference between the outputs of the teacher model and the student model is: Where k represents the number of image samples of the i-th class; in, It is the variance of the teacher model output, used express; It is the variance of the student model output, used express; (5)G T-S This can be further expressed as: (6) Let G T-S =0, determine the distillation temperature T of the teacher model. T and the student model distillation temperature T S The relationship between std T / T T =std S / T S .

5. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the model compression method for adaptively adjusting the knowledge distillation temperature as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the model compression method for adaptively adjusting the knowledge distillation temperature according to any one of claims 1 to 4.

7. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the model compression method for adaptively adjusting the knowledge distillation temperature as described in any one of claims 1 to 4.

8. A model compression system for adaptively adjusting the knowledge distillation temperature, implementing the model compression method for adaptively adjusting the knowledge distillation temperature according to any one of claims 1 to 4, characterized in that, The adaptive knowledge distillation temperature adjustment model compression system includes: The data cleaning module is used to collect image samples and clean abnormal and missing data in the image samples, including directly discarding abnormal data and recovering missing data using nearest neighbor interpolation or bilinear interpolation. The sample processing module is used to randomly flip, scale, and crop image samples using the Mosaic-4 data augmentation method, and then randomly select four transformed images to stitch together, thereby enhancing the image samples. The model compression module is used to establish a loss function that minimizes the adaptive distillation function, completes the knowledge transfer from the teacher model to the student model, and realizes knowledge distillation; it uses the training set to train the student model, determines the optimal total loss function, and realizes model compression.

Images