Self-distillation methods, computer equipment, and storage media based on tolerance-labeled specifications.

Abstract

This invention discloses a self-distillation method, computer device, and storage medium based on tolerant labels. The method specifically includes the following steps: acquiring an image dataset for training a deep neural network model; training a predefined deep neural network using the image dataset, with the optimization objectives being cross-entropy optimization loss and self-distillation loss, the soft labels required for self-distillation loss being provided through an adaptive memory structure; and outputting the prediction results for an image classification task using the trained model. This method is applicable to supervised image recognition or detection tasks. Addressing overfitting issues during optimization, it performs self-distillation without requiring a large pre-trained teacher model, allowing the model to converge at a better point and achieve more robust results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a self-distillation method, computer device, and storage medium based on tolerance labels. Background Technology

[0002] In recent years, thanks to powerful computing power, massive amounts of data, and efficient neural network models, deep learning has developed rapidly, demonstrating strong performance in computer vision and natural language processing. In image recognition, deep network model training typically relies on one-hot encoding of labels from the original images and cross-entropy loss for optimization. However, one-hot encoding can cause the network to overfit the training set's image distribution, failing to achieve good recognition results on the test image distribution and affecting the model's robustness. This overfitting problem caused by one-hot encoding and cross-entropy training can be mitigated through data augmentation and soft label supervision. Data augmentation methods enhance model robustness by adding different perturbations such as spatial and color transformations to the original data. Soft label supervision methods typically include label smoothing and knowledge distillation. Label smoothing generates soft labels by assigning the original labels and uniformly distributed weights to supervise the network; knowledge distillation uses a pre-trained large teacher model to acquire soft labels to guide and transfer knowledge to smaller student models. Knowledge distillation uses a distillation loss to distill the smaller model from the larger model, improving the smaller model's performance and achieving a certain degree of model compression. The traditional distillation loss is:

[0003]

[0004] Here, p is the output of the teacher model, and q is the output of the student model. However, in real-world applications, obtaining a pre-trained teacher model is difficult or requires significant resources for training. This is where self-distillation algorithms come in. The teacher model's output p can come from the model's own knowledge, such as past predictions or the output of a Siamese network. In this case, a pre-trained teacher model is unnecessary; the model can perform its own distillation.

[0005] Furthermore, in existing soft-label supervision methods, the acquisition of soft labels is usually consistent with the real labels. That is, the category label corresponding to the soft label is the same as the category to which the image sample belongs. Under this setting, even if the soft label can carry additional inter-category information, it will still lead to overfitting of the model to some extent. Summary of the Invention

[0006] To address the aforementioned problems, this invention aims to provide a self-distillation method, computer device, and storage medium based on tolerance-based labels to provide adaptive soft-label supervision for different images, reducing overfitting during model training. The soft-label supervision method involves mutual supervision between correctly and incorrectly predicted samples, narrowing the distribution of samples within a class and enabling the model to converge at a better point. For the soft label acquisition method, this invention uses an adaptive memory structure to store past prediction results during model training, providing soft-label supervision for the current iteration, thus improving the method's flexibility. Furthermore, this invention can be well integrated with data augmentation methods and various existing data augmentation strategies, effectively improving the model's recognition performance and enhancing its robustness.

[0007] To achieve the above objectives, the first technical solution of the present invention is as follows:

[0008] A self-distillation method based on tolerance labels includes the following steps:

[0009] S1. Obtain the image dataset used to train the deep neural network model;

[0010] S2. Train a predefined deep neural network model using the image dataset. The optimization objectives are cross-entropy optimization loss and self-distillation loss. The soft labels required for the self-distillation loss are provided through an adaptive memory structure.

[0011] S3. Use the trained deep neural network model to output the prediction result of the classification of the given image.

[0012] Furthermore, in step S1, the image dataset is {(x1,y1),(x2,y2),…,(x...} N ,y N )}, where N is the total number of images in the dataset, x i and y i Let y represent the i-th image and its corresponding class label, respectively. i ∈{1,…,C}, where C represents the total number of categories, and each category contains multiple images.

[0013] Furthermore, step S2 specifically includes:

[0014] S21. Predefine a memory structure M related to the category and training prediction results. correct M wrong The size of each is C*K*C, where K is the preset memory size;

[0015] S22. When training a deep neural network, a batch processing method is used. First, a batch of image samples is randomly sampled from the image dataset. Where the batch size N bs Pre-given;

[0016] S23. Input the image from B into a deep neural network to obtain the corresponding prediction output. z i This is the model's output prediction vector, with a size of 1*C. After obtaining the output prediction result, it is retrieved from the memory structure M according to the MemoryRead operation. correct and M wrong get

[0017] S24. Based on the MemoryWrite operation, adaptively select the image prediction results in B to update the memory result M. correct M wrong ;

[0018] S25. Use adaptive soft labels to construct the self-distillation learning loss function L. SKD Simultaneously, all predictions in a batch are used to construct the cross-entropy optimized loss function L. cls ;

[0019] S26. Based on the cross-entropy optimization loss and self-distillation loss, the total loss function is obtained as follows:

[0020] L = L cls +η*L SKD

[0021] Where η is a pre-set weighting parameter used to balance the contributions of the two loss functions;

[0022] S27. Based on the obtained total loss function, train the deep neural network model using the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the backpropagation algorithm.

[0023] S28. Repeat steps S22 to S27 until the model converges.

[0024] Furthermore, in step S25, the self-distillation learning loss function L SKD for:

[0025]

[0026] in, Where τ is the temperature coefficient.

[0027] Furthermore, in step S25, the cross-entropy optimization loss function L cls for:

[0028] Where y ic p is the true label for the image. ic Output the prediction results for the model.

[0029] Furthermore, in step S23, the MemeryRead operation is as follows: for image x i The model outputs a prediction result of p. i When the prediction is correct, i.e., argmax(p) i )=y i From M wrong Randomly read one As x i The soft label, when the prediction is wrong, i.e., argmax(p i )≠y i From M correct Randomly read one As x i The soft labels enable mutual supervision between samples that are predicted incorrectly and those that are predicted correctly.

[0030] Furthermore, in step S24, the MemoryWrite operation is as follows: for image x i The model outputs a prediction result of p. i When the prediction is correct, i.e., argmax(p) i )=y i Calculate the entropy of the prediction result:

[0031] H(p i )=-∑p i log p i

[0032] By comparing H(p) i ) and M correct Entropy H(p′) among all samples i The smallest sample, if H(p) i )>H(p′ i ), then in M correct Update p′ i For p i Keep M correct The samples in the dataset all have relatively high entropy; when a prediction error occurs, M is replaced according to the first-in, first-out principle. wrong The first sample p′ i , using p i Replace p′ i ;

[0033] Furthermore, in step S3, the given image x is calculated. i The method for classification prediction results is as follows: xi Inputting a trained deep neural network M, we obtain the predicted output p. i The predicted label is:

[0034] y pred =argmax(p i )

[0035] That is, y pred This represents the prediction result for the image.

[0036] The second technical solution of the present invention is a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the self-distillation method based on a tolerance label described in the first solution above.

[0037] The third technical solution of the present invention is a computer-readable storage medium comprising a stored program, wherein the computer-readable storage medium stores computer instructions for causing a computer to execute the steps of the self-distillation method based on a tolerance label described in the first solution above.

[0038] The self-distillation method, computer device, and storage medium based on tolerance tags of the present invention have the following advantages compared with existing knowledge distillation methods:

[0039] First, this invention flexibly designs a memory structure to store and provide soft labels, so that the model does not need a pre-trained teacher model for supervision, which can improve model performance without increasing additional computing resources.

[0040] Secondly, this invention adaptively assigns a tolerant label to each sample, allowing correctly predicted samples and incorrectly predicted samples to supervise each other. This reduces overfitting during training and improves the robustness of the model while bringing the intra-class distribution closer together.

[0041] The self-distillation method based on tolerance labels of this invention has significant application value in computer vision recognition and few-shot scene training, and can also be extended to other computer vision and natural language processing tasks. For example, in object detection, the self-distillation method of this invention can provide self-supervised information for different categories of detection, alleviate overfitting in the object detection process, and improve the object detection effect. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the self-distillation method based on tolerance labels according to the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0044] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0045] refer to Figure 1 In a preferred embodiment of the present invention, the self-distillation method based on a tolerance label includes the following steps:

[0046] First, obtain the image dataset used to train the deep neural network model. The image dataset is {(x1,y1),(x2,y2),…,(x...} N ,y N )}, where N is the total number of images in the dataset, x i and y i Let y represent the i-th image and its corresponding class label, respectively. i ∈{1,…,C}, where C represents the total number of categories, and each category contains multiple images.

[0047] Secondly, a predefined deep neural network model is trained using an image dataset, with the optimization objectives being cross-entropy optimization loss and self-distillation loss. The deep neural network model is a classification network model, such as ResNet or DenseNet. Specifically, the steps include:

[0048] The first step is to predefine a memory structure M associated with the category and the training prediction results. correct M wrong Both are C*K*C, where K is the preset memory size.

[0049] The second step involves using batch processing when training the deep neural network. First, a batch of image samples is randomly sampled from the image dataset. Where the batch size N bs Given in advance.

[0050] The third step is to input the image from B into a deep neural network to obtain the corresponding prediction output. z iThis is the model's output prediction vector, with a size of 1*C. After obtaining the output prediction result, it is retrieved from the memory structure M according to the MemoryRead operation. correct and M wrong get

[0051] Specifically, the MemeryRead operation is as follows: for image x i The model outputs a prediction result of p. i When the prediction is correct, i.e., argmax(p) i )=y i From M wrong Randomly read one As x i The soft label, when the prediction is wrong, i.e., argmax(p i )≠y i From M correct Randomly read one As x i The soft labels enable mutual supervision between samples that are predicted incorrectly and those that are predicted correctly.

[0052] Fourth, based on the MemoryWrite operation, the image prediction results in B are adaptively selected to update the memory result M. correct M wrong .

[0053] Specifically, the MemeryWrite operation is: for image x i The model outputs a prediction result of p. i When the prediction is correct, i.e., argmax(p) i )=y i Calculate the entropy of the prediction result:

[0054] H(p i )=-∑p i log p i

[0055] By comparing H(p) i ) and M correct Entropy H(p′) among all samples i The smallest sample, if H(p) i )>H(p′ i ), then in M correct Update p′ i For p i Keep M correct The samples in the dataset all have relatively high entropy; when a prediction error occurs, M is replaced according to the first-in, first-out principle. wrong The first sample p′i , using p i Replace p′ i .

[0056] Fifth step: Use the adaptive soft labels to construct the self-distillation learning loss function L. SKD Meanwhile, all predictions in a single batch are used to construct the cross-entropy optimization loss:

[0057]

[0058] Where y ic p is the true label for the image. ic Output the prediction results for the model;

[0059] Self-distillation loss function L SKD for:

[0060]

[0061] in, Where τ is the temperature coefficient.

[0062] Step 6: Based on the aforementioned cross-entropy optimization loss and self-distillation loss, the total loss function is obtained as follows:

[0063] L = L cls +η*L SKD

[0064] Where η is a pre-set weighting parameter used to balance the contributions of the two loss functions.

[0065] Step 7: Based on the obtained loss function, train the deep neural network using the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the backpropagation algorithm.

[0066] Step 8: Repeat steps 2 through 7 until the model converges.

[0067] Finally, the trained deep neural network model outputs a prediction result for classifying a given image. Specifically, it calculates the classification result for a given image x. i The method for classification prediction results is as follows: x i Inputting a trained deep neural network M, we obtain the predicted output p. i The predicted label is:

[0068] y pred =argmax(p i )

[0069] That is, y pred This represents the prediction result for the image.

[0070] Through the above technical solutions, this invention provides a model self-distillation method based on tolerant labels. This invention introduces a memory structure to adaptively provide tolerant soft labels during model training, while simultaneously constraining the distribution between correctly and incorrectly predicted samples. By optimizing the intra-class distribution structure, the model can converge at better points, mitigating overfitting during training and improving image recognition performance.

[0071] Embodiments of the present invention also provide a computer, including a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the steps of the self-distillation method based on tolerance labels shown in the above embodiments.

[0072] Embodiments of the present invention also provide a computer-readable storage medium comprising a stored program, wherein the computer-readable storage medium stores computer instructions for causing a computer to execute the steps of the self-distillation method based on a tolerance label as described in the above embodiments.

[0073] It will be apparent to those skilled in the art that the modules or steps of this invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this invention is not limited to any particular hardware and software combination.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A self-distillation method based on tolerance labels, characterized in that, Includes the following steps: S1. Obtain the image dataset used to train the deep neural network model, wherein the image dataset is... ,in The total number of images in the dataset. and They represent the first Images and their corresponding class labels , This indicates the total number of categories, with each category containing multiple images; S2. Train a predefined deep neural network model using the image dataset. The optimization objectives are cross-entropy optimization loss and self-distillation loss. The soft labels required for the self-distillation loss are provided through an adaptive memory structure, specifically including: S21. Predefine a memory structure related to a category and training prediction results. , Both are of the following sizes K is the preset memory size; S22. When training a deep neural network, a batch processing method is used. First, a batch of image samples is randomly sampled from the image dataset. Batch size Pre-given; S23, will The image is input into a deep neural network to obtain the corresponding prediction output. , This is the model's output prediction vector result, with a size of After obtaining the output prediction results, according to Operations from memory structure and get ; S24, according to Operation will Image prediction results are adaptively selected to update memory results. , ; S25. Use adaptive soft labels to construct a self-distillation learning loss function. Simultaneously, all predictions in a batch are used to construct the cross-entropy optimization loss function. ; S26. Based on the cross-entropy optimization loss and self-distillation loss, the total loss function is obtained as follows: in These are pre-set weighting parameters used to balance the contributions of the two loss functions; S27. Based on the obtained total loss function, train the deep neural network model using the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the backpropagation algorithm. S28. Repeat steps S22 to S27 until the model converges; S3. Use the trained deep neural network model to output the prediction result of the classification of the given image.

2. The self-distillation method based on tolerance labels as described in claim 1, characterized in that, In step S25, the self-distillation learning loss function for: in, ,in It is the temperature coefficient.

3. The self-distillation method based on tolerance labels as described in claim 2, characterized in that, In step S25, the cross-entropy optimization loss function for: in For the true label of the image, Output the prediction results for the model.

4. The self-distillation method based on tolerance labels as described in claim 3, characterized in that, In step S23, the aforementioned The operation is as follows: for the image The model outputs the prediction result as follows: When the prediction is correct, that is ,from Randomly read one As The soft label, when the prediction is incorrect, i.e. ,from Randomly read one As The soft labels enable mutual supervision between samples that are predicted incorrectly and those that are predicted correctly.

5. The self-distillation method based on tolerance labels as described in claim 3, characterized in that, In step S24, the The operation is as follows: for the image The model outputs the prediction result as follows: When the prediction is correct, that is Calculate the entropy of the prediction result: By comparison and Entropy in all samples The smallest sample, if So in Update for ,Keep The samples in the dataset all have relatively high entropy; when a prediction is incorrect, the data is replaced according to the first-in, first-out principle. The first sample ,use replace .

6. The self-distillation method based on tolerance labels as described in any one of claims 1-5, characterized in that, In step S3, a given image is calculated. The method for classification prediction results is as follows: Input the trained deep neural network The predicted output results are obtained. The predicted label is: Right now This represents the prediction result for the image.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the self-distillation method based on tolerance labels as described in any one of claims 1 to 6.

8. A computer-readable storage medium comprising a stored program, wherein, The computer-readable storage medium stores computer instructions for causing a computer to perform the steps of the self-distillation method based on a tolerance label, as described in any one of claims 1 to 6.

Images