Robust knowledge distillation method based on ambiguity-guided mutual label updating

Abstract

The present invention discloses a robust knowledge distillation method based on ambiguity-guided mutual label updating, including: A. constructing a training data set; B. constructing a teacher network and a student network; C. training the teacher network through a robust learning method; D. performing ambiguity-aware weight estimation and weight allocation on each sample in the data set; E. re-labeling the samples in the data set according to a small loss standard, updating the labels by a label propagation algorithm combined with the characteristics of the teacher network, and then calculating the loss and updating the network parameters; F. performing a mutual label propagation algorithm between the student network and the teacher network, and updating the sample labels, calculating the loss and updating the network parameters; G. importing the test image data into the student network, thereby obtaining a prediction result and using it for image classification; this scheme can effectively improve the robustness of knowledge distillation to noisy labels, thereby obtaining a high-performance lightweight network in a noisy label environment, which is more suitable for actual situations.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a robust knowledge distillation method based on ambiguity-guided mutual label updating. Background Art

[0002] In recent years, many model compression methods have been proposed to reduce the number of parameters of convolutional neural networks to achieve the purpose of model acceleration. Among these methods, knowledge distillation plays an important role. Knowledge distillation usually includes a teacher network and a student network. The student network has significantly enhanced its generalization ability by learning the "dark knowledge" contained in the output of the teacher network. However, in actual situations, the training data set often contains a large amount of label noise, and overfitting of this label noise will significantly affect the performance of knowledge distillation. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a robust knowledge distillation method based on ambiguity-guided mutual label updating. The scheme is based on a label update strategy and dynamically updates labels during the knowledge distillation process, thereby greatly reducing the impact of noisy labels on knowledge distillation and improving the robustness of the model.

[0004] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0005] A robust knowledge distillation method based on ambiguity-guided mutual label updating, comprising:

[0006] A. Construct a training data set and preprocess it according to preset conditions;

[0007] B. Build teacher network and student network;

[0008] C. Import the training data set into the teacher network for training through a robust learning method to obtain a teacher model with preset performance;

[0009] D. Import the training dataset into the student network, and then perform ambiguity-aware weight estimation and weight assignment for each sample in the training dataset;

[0010] E. The student network relabels the samples in the training data set according to the preset conditions and the small loss standard, and then updates the labels with the label propagation algorithm based on the characteristics of the teacher network, and then calculates the loss and updates the network parameters;

[0011] F. Perform mutual label propagation algorithm between the student network and the teacher network, update the sample labels, calculate the loss, and update the network parameters;

[0012] G. Import the test image data into the student network, forward propagate the student model of the student network to obtain the prediction result and use it for image classification.

[0013] As a possible implementation, further, in step A, the training data set includes a noisy data set with noise label data and / or a noise-free data set with noise-free label data; wherein the noise-free data set with noise-free label data is processed by injecting noise to generate a synthetic noise data set.

[0014] As one of the better noise-free data set processing methods, preferably, the noise-free data set is divided into two equal parts, one of which is used for student network training after being injected with noise, and the other is not processed and used for teacher network training. The noise is set to CN noise, and the teacher model is trained using standard cross entropy loss.

[0015] As another preferred method for processing a noise-free data set, preferably, the method for injecting noise is to add symmetrical and / or asymmetrical noise to the noise-free data set.

[0016] As a preferred implementation mode, preferably, the noise data set includes one or more of the ANIMAL-10N data set and the Clothing1M data set, and the noise-free data set includes the CIFAR-100 data set.

[0017] As a preferred implementation method, preferably, in step B, the teacher network and the student network are one of the following:

[0018] (1) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 16 layers and a width coefficient of 2;

[0019] (2) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 40 layers and a width coefficient of 1;

[0020] (3) The teacher network is a residual network with 56 layers, and the student network is a residual network with 20 layers.

[0021] As a preferred implementation mode, preferably, in step D, an ambiguity-aware weight estimation module is used to perform ambiguity-aware weight estimation and weight assignment on each sample in the training data set. The ambiguity-aware weight estimation module includes two fully connected layers, and a PRelu layer is further provided between the two fully connected layers. Step D specifically includes:

[0022] Import all samples in the training data set into the student network, obtain their features, and then calculate the prototype features of each category. The formula is as follows:

[0023]

[0024] in, is the number of training data, N c Indicates the number of samples of category c;

[0025] The feature distribution score of each sample is calculated as follows:

[0026]

[0027] The ambiguous feature vector is obtained by concatenating the labels and feature distribution scores in round t, and the formula is as follows:

[0028]

[0029] in, As the ambiguous feature vector, it is sent into a two-layer perceptron network to obtain the final sample weight, and the formula is as follows:

[0030]

[0031] in are two fully connected layers, σ represents the PreLU operation;

[0032] The weights are written in matrix form as follows:

[0033]

[0034] As a preferred implementation mode, preferably, step E specifically includes:

[0035] Use the features of the teacher network to construct a k-nn graph G = <V, E>, where V and E represent the vertex set and edge set respectively, and the similarity matrix between vertices is described as follows:

[0036]

[0037] in Represents sample x i Features under the teacher network, NN k (x i ) represents the sample x i k nearest neighbors, then, we can get a symmetric adjacency matrix Then normalize W t get Among them, D is the diagonal matrix; at the same time, according to the small loss standard, the original annotation of the training data set will be re-annotated according to the student network small loss standard, and the formula is as follows:

[0038]

[0039] in, Represents the student network for sample x i predictions, is the original label of the sample, is the indicator function, represents the set of clean samples selected by the student network;

[0040] Combine the k-nn graph G and update the label matrix Z and sample weight matrix For label propagation, the formula is as follows

[0041]

[0042] Among them, LP can be defined as In the formula, ⊙ represents the multiplication between corresponding elements, and β is a hyperparameter used to balance the loss; the updated label Y is calculated (t) After that, the loss is calculated according to the following formula:

[0043]

[0044] in, Defined as:

[0045]

[0046] The mixup algorithm is used to obtain mixed sample data, where the mixed sample data is a virtual sample, and the formula is as follows:

[0047]

[0048]

[0049] Construct the distillation loss under the above virtual samples The formula is as follows:

[0050]

[0051] in,

[0052] τ is the temperature parameter;

[0053] Similarly, define the classification loss of mixed samples The formula is as follows:

[0054]

[0055] Finally, the following loss is defined to make the student network imitate the similarity between samples of the teacher network. The formula is as follows:

[0056] .

[0057] As a preferred embodiment, preferably, step F includes:

[0058] First, the ambiguity-aware weight estimation module in step D is used to estimate the weight for each sample and obtain the weight matrix And update the label as follows:

[0059]

[0060]

[0061] Finally, the loss is calculated using formula (4) to update the network parameters.

[0062] Based on the above scheme, the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned robust knowledge distillation method based on ambiguity-guided mutual label updating.

[0063] By adopting the above-mentioned technical scheme, the present invention has the following beneficial effects compared with the prior art: this scheme is based on the teacher-student network, and proposes a two-stage label update method including label propagation with small loss selection and mutual label propagation. The label update strategy based on this design can effectively improve the robustness of knowledge distillation to noisy labels, thereby obtaining a high-performance lightweight network in a noisy label environment. Compared with the traditional knowledge distillation method, this scheme takes into account the more practical noise label problem, making the knowledge distillation algorithm more applicable to actual situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a schematic diagram of a brief implementation process of the method of the present invention.

[0066] Figure 2 It is a brief principle flow diagram of the method of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0068] like Figure 1 or Figure 2 As shown, this scheme is a robust knowledge distillation method based on ambiguity-guided mutual label updating, which includes:

[0069] A. Construct a training data set and preprocess it according to preset conditions;

[0070] In this step, the training datasets used are three common image classification datasets, namely the CIFAR-100 dataset, the ANIMAL-10N dataset, and the Clothing1M dataset. The CIFAR-100 dataset is a noise-free dataset that does not contain noise labels. It can be made into a synthetic noise dataset by adding symmetric and asymmetric noise. In addition, this solution also provides a CN noise, which is to divide the noise-free dataset into two parts, one of which is injected with noise (symmetric and asymmetric noise) for student network training, and the other is not processed for teacher network training. The standard cross entropy loss of the CN noise is used to train the teacher network. In addition, the ANIMAL-10N dataset and the Clothing1M dataset are datasets in real scenes, which contain about 8% and 38% noise label data respectively. In addition, during the training process, image rotation, flipping and other methods can also be used for data enhancement. To simplify the description, the following uses Represents the dataset used to train the student network.

[0071] B. Build teacher network and student network;

[0072] In this step, the teacher network generally has a more complex model structure than the student network. This solution can adopt the following three pairs of network structures commonly used in knowledge distillation, which are:

[0073] (1) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 16 layers and a width coefficient of 2, i.e., WRN_40_2-WRN_16_2;

[0074] (2) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 40 layers and a width coefficient of 1, that is, WRN_40_2-WRN_40_1;

[0075] (3) The teacher network is a residual network with 56 layers, and the student network is a residual network with 20 layers, i.e., resent56-resnet20.

[0076] To simplify the description, the following uses and Represents the student model and the teacher model.

[0077] C. Import the training data set into the teacher network for training through a robust learning method to obtain a teacher model with preset performance;

[0078] In this step, when the imported training dataset is CN noise, the standard cross entropy loss of the CN noise is used to train the teacher network. For the other noise types (symmetric, asymmetric, real noise), this scheme uses the classic robust learning algorithm DivideMix (J. Li, R. Socher, and SCHoi, "Dividemix: Learning with noisy labels as semi-supervised learning," in Int. Conf. Learn. Represent., 2019.) to pre-train a teacher model.

[0079] D. Import the training dataset into the student network, and then perform ambiguity-aware weight estimation and weight assignment for each sample in the training dataset;

[0080] In this step, an ambiguity-aware weight estimation module is used to estimate and assign ambiguity-aware weights to each sample in the training data set. The ambiguity-aware weight estimation module includes two fully connected layers, and a PRelu layer is provided between the two fully connected layers. This step specifically includes:

[0081] Import all samples in the training data set into the student network, obtain their features, and then calculate the prototype features of each category. The formula is as follows:

[0082]

[0083] in, is the number of training data, N c Indicates the number of samples of category c;

[0084] Next, the feature distribution score of each sample is calculated as follows:

[0085]

[0086] Then, the labels and feature distribution scores in round t are concatenated to obtain the ambiguous feature vector, which is formulated as follows:

[0087]

[0088] in, As the ambiguous feature vector, it is sent into a two-layer perceptron network to obtain the final sample weight, and the formula is as follows:

[0089]

[0090] in, are two fully connected layers, σ represents the PreLU operation, and the two-layer perceptron network is the aforementioned ambiguous perception weight estimation module, which contains two fully connected layers and outputs a scalar (weight). Specifically, this scheme converts the label and similarity score Splice together to get ambiguous features Then it is sent to the ambiguity-aware weight estimation module to calculate the weight of the sample;

[0091] The weights are written in matrix form as follows:

[0092]

[0093] E. The student network relabels the samples in the training data set according to the preset conditions and the small loss standard, and then updates the labels with the label propagation algorithm based on the characteristics of the teacher network, and then calculates the loss.

[0094] Update network parameters;

[0095] This step specifically includes:

[0096] First, the features of the teacher network are used to construct a k-nn graph G = <V, E>, where V and E represent the vertex set and edge set respectively, and the similarity matrix between vertices is described as follows:

[0097]

[0098] in Represents sample x i Features under the teacher network, NN k (x i ) represents the sample x i k nearest neighbors, then, we can get a symmetric adjacency matrix Then normalize W t get Among them, D is the diagonal matrix; at the same time, according to the small loss standard, the original annotation of the training data set will be re-annotated according to the student network small loss standard, and the formula is as follows:

[0099]

[0100] in, Represents the student network for sample x i predictions, is the original label of the sample, is the indicator function, represents the set of clean samples selected by the student network;

[0101] Combine the k-nn graph G and update the label matrix Z and sample weight matrix The label propagation formula is as follows:

[0102]

[0103] Among them, LP can be defined as In the formula, ⊙ represents the multiplication between corresponding elements, and β is a hyperparameter used to balance the loss; the updated label Y is calculated (t) After that, the loss is calculated according to the following formula:

[0104]

[0105] in, Defined as:

[0106]

[0107] In order to further improve the robustness of distillation, the mixup algorithm (Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In ICLR, 2018) is used to obtain mixed sample data. The mixed sample data is a virtual sample, and its formula is as follows:

[0108]

[0109]

[0110] Construct the distillation loss under the above virtual samples The formula is as follows:

[0111]

[0112] in,

[0113] τ is the temperature parameter;

[0114] Similarly, define the classification loss of mixed samples The formula is as follows:

[0115]

[0116] Finally, in order to make the student network have better feature representation ability, the following loss is defined to make the student network imitate the similarity between samples of the teacher network. The formula is as follows:

[0117] .

[0118] F. Perform mutual label propagation algorithm between the student network and the teacher network, update the sample labels, calculate the loss, and update the network parameters;

[0119] This step specifically includes:

[0120] First, the ambiguity-aware weight estimation module in step D is used to estimate the weight for each sample and obtain the weight matrix And update the label as follows:

[0121]

[0122]

[0123] Finally, the loss is calculated using formula (4) to update the network parameters.

[0124] G. Import the test image data into the student network, forward propagate the student model of the student network to obtain the prediction result and use it for image classification.

[0125] This step is the reasoning stage of this scheme. It only uses the trained student network. The test image is sent to the student network, and the student network obtains the output to complete the reasoning.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0128] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A robust knowledge distillation method based on ambiguity-guided mutual label updating, characterized in that: It includes: A. Construct a training data set and preprocess it according to preset conditions; B. Build teacher network and student network; C. Import the training data set into the teacher network for training through a robust learning method to obtain a teacher model with preset performance; D. Import the training dataset into the student network, and then perform ambiguity-aware weight estimation and weight assignment for each sample in the training dataset; E. The student network relabels the samples in the training data set according to the preset conditions and the small loss standard, and then updates the labels with the label propagation algorithm based on the characteristics of the teacher network, and then calculates the loss and updates the network parameters; F. Perform mutual label propagation algorithm between the student network and the teacher network, update the sample labels, calculate the loss, and update the network parameters; G. Import the test image data into the student network, forward propagate the student model of the student network to obtain the prediction result and use it for image classification; Wherein, step E specifically includes: Use the features of the teacher network to construct the k-nn graph G =<V,E> , where V and E represent the vertex set and edge set respectively, and the similarity matrix between vertices is described as follows: in, Represents sample x i Features under the teacher network, NN k (x i ) represents the sample x i k nearest neighbors, then, we can get a symmetric adjacency matrix Then normalize W t get Among them, D is the diagonal matrix; at the same time, according to the small loss standard, the original annotation of the training data set will be re-annotated according to the student network small loss standard, and the formula is as follows: in, Represents the student network for sample x i predictions, is the original label of the sample, is the indicator function, represents the set of clean samples selected by the student network; Combine the k-nn graph G and update the label matrix Z and sample weight matrix The label propagation formula is as follows: Among them, LP can be defined as In the formula, ⊙ represents the multiplication between corresponding elements, and β is a hyperparameter used to balance the loss; the updated label Y is calculated (t) After that, the loss is calculated according to the following formula: in, Defined as: The mixup algorithm is used to obtain mixed sample data, where the mixed sample data is a virtual sample, and the formula is as follows: Construct the distillation loss under the above virtual samples The formula is as follows: in, τ is the temperature parameter; Similarly, define the classification loss of mixed samples The formula is as follows: Finally, the following loss is defined to make the student network imitate the similarity between samples of the teacher network. The formula is as follows: Step F comprises: First, the ambiguity-aware weight estimation module in step D is used to estimate the weight for each sample and obtain the weight matrix And update the label as follows: Finally, the loss is calculated using formula (4) to update the network parameters.

2. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 1, characterized in that: In step A, the training data set includes a noisy data set with noise label data and / or a noise-free data set with noise-free label data; wherein the noise-free data set with noise-free label data is injected into the noise-free data set to generate a synthetic noise data set.

3. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 2, characterized in that: The noise-free dataset is divided into two equal parts, one of which is used for student network training after being injected with noise, and the other is not processed and is used for teacher network training. The noise is set to CN noise, and the teacher model is trained using standard cross entropy loss.

4. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 2, characterized in that: The method of injecting noise is to add symmetrical and / or asymmetrical noise into the noise-free data set.

5. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 4, characterized in that: The noise data set includes one or more of the ANIMAL-10N data set and the Clothing1M data set, and the noise-free data set includes the CIFAR-100 data set.

6. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 1, characterized in that: In step B, the teacher network and the student network are one of the following: (1) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 16 layers and a width coefficient of 2; (2) The teacher network is a wide residual network with 40 layers and a width coefficient of 2, and the student network is a wide residual network with 40 layers and a width coefficient of 1; (3) The teacher network is a residual network with 56 layers, and the student network is a residual network with 20 layers.

7. The robust knowledge distillation method based on ambiguity-guided mutual label updating according to claim 1, characterized in that: In step D, an ambiguity-aware weight estimation module is used to perform ambiguity-aware weight estimation and weight assignment on each sample in the training data set. The ambiguity-aware weight estimation module includes two fully connected layers, and a PRelu layer is also provided between the two fully connected layers. Step D specifically includes: Import all samples in the training data set into the student network, obtain their features, and then calculate the prototype features of each category. The formula is as follows: in, is the number of training data, N c Indicates the number of samples of category c; The feature distribution score of each sample is calculated as follows: The ambiguous feature vector is obtained by concatenating the labels and feature distribution scores in round t, and the formula is as follows: in, As the ambiguous feature vector, it is sent into a two-layer perceptron network to obtain the final sample weight, and the formula is as follows: in, are two fully connected layers, σ represents the PreLU operation; The weights are written in matrix form as follows:

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the robust knowledge distillation method based on ambiguity-guided mutual label updating as described in one of claims 1 to 7.

Images