Multi-stage contrastive knowledge distillation method

Abstract

The application discloses a multi-stage contrast knowledge distillation method. The method comprises the following steps: acquiring a training sample image and a theoretical classification label corresponding to the training sample image, and performing data enhancement on the training sample image to obtain at least one to-be-processed sample image; based on the at least one to-be-processed sample image and the theoretical classification label, a training sample is constructed; the training sample is processed based on a pre-trained teacher network to obtain a first smoothed probability distribution and a first model output corresponding to a plurality of teacher sub-models; the training sample is input into a to-be-trained student network to obtain an actual output result and a second model output corresponding to a plurality of student sub-models; a target model loss is determined, and model parameter adjustment is performed on the to-be-trained student model based on the target model loss to obtain an image classification model. The technical scheme realizes the effect that the performance of the trained student network is further close to or even exceeds the performance of the teacher network.

Description

Technical Field

[0001] This invention relates to the field of knowledge distillation technology, and more particularly to a multi-stage comparative knowledge distillation method. Background Technology

[0002] Knowledge distillation is a mainstream model compression method. Essentially, it extracts the probability distribution of training data from a high-performance complex network (i.e., the teacher network) and passes it to a low-performance simple network (i.e., the student network) to guide the training process of the student model, thereby improving the performance of the student model.

[0003] In related technologies, traditional knowledge distillation methods typically use the probability distribution of the teacher network output to train the student network. This training method may fail to fully transfer knowledge to the student network, making it difficult to fully utilize the knowledge contained in the teacher network to improve the representation ability of the lightweight network, thus limiting the knowledge transfer effect and resulting in low accuracy of neural network model compression. Summary of the Invention

[0004] This invention provides a multi-stage comparative knowledge distillation method to achieve the effect of transferring richer knowledge from the teacher network to the student network, so that the performance of the student network can further approach or even exceed the performance of the teacher network, thereby achieving the goal of high performance of a simple model.

[0005] According to one aspect of the present invention, a multi-stage comparative knowledge distillation method is provided, the method comprising:

[0006] Obtain training sample images and theoretical classification labels corresponding to the training sample images, and perform data augmentation on the training sample images to obtain at least one sample image to be processed. Based on the at least one sample image to be processed and the theoretical classification labels, construct training samples.

[0007] The training samples are processed based on the pre-trained teacher network to obtain a first smoothed probability distribution and a first model output corresponding to multiple teacher sub-models;

[0008] The training samples are input into the student network to be trained to obtain the actual output results and the second model output corresponding to multiple student sub-models. The actual output results include the actual classification results and the second smoothed probability distribution.

[0009] Based on the first smoothed probability distribution, multiple first model outputs, the actual output results, multiple second model outputs, and the theoretical classification labels, the target model loss is determined, and the model parameters of the student model to be trained are adjusted based on the target model loss to obtain an image classification model.

[0010] The technical solution of this invention obtains training sample images and corresponding theoretical classification labels, and performs data augmentation on the training sample images to obtain at least one image to be processed. Based on the at least one image to be processed and the theoretical classification labels, training samples are constructed. Then, the training samples are processed based on a pre-trained teacher network to obtain a first smoothed probability distribution and first model outputs corresponding to multiple teacher sub-models. Further, the training samples are input into a student network to be trained to obtain actual output results and second model outputs corresponding to multiple student sub-models. Finally, based on the first smoothed probability distribution, multiple first model outputs, actual output results, multiple second model outputs, and theoretical classification labels, the target model loss is determined, and the model parameters of the student model to be trained are adjusted based on the target model loss to obtain an image classification model. This solves the problems in related technologies, such as the inability to fully transfer knowledge to the student network, the difficulty in fully utilizing the knowledge contained in the teacher network to improve the representation ability of lightweight networks, the limited knowledge transfer effect, and the low accuracy of neural network model compression. It achieves the effect of transferring richer knowledge from the teacher network to the student network, allowing the performance of the student network to approach or even exceed the performance of the teacher network, thereby achieving the goal of high performance from a simple model.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a multi-stage comparative knowledge distillation method provided in Embodiment 1 of the present invention;

[0014] Figure 2 This is a flowchart of the online training process for teachers to be trained according to Embodiment 1 of the present invention;

[0015] Figure 3 This is a flowchart of a multi-stage comparative knowledge distillation method provided in Embodiment 1 of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Example 1

[0019] Figure 1 This is a flowchart of a multi-stage comparative knowledge distillation method provided in Embodiment 1 of the present invention. This embodiment is applicable to training student networks based on pre-trained teacher networks. The method can be executed by a multi-stage comparative knowledge distillation device, which can be implemented in hardware and / or software and can be configured in a terminal and / or server. Figure 1 As shown, the method includes:

[0020] S110. Obtain training sample images and theoretical classification labels corresponding to the training sample images, and perform data augmentation on the training sample images to obtain at least one sample image to be processed. Based on at least one sample image to be processed and the theoretical classification label, construct training samples.

[0021] The training sample images can be images captured by a camera device; images reconstructed by an image reconstruction model; or images pre-stored in storage space. In this embodiment, the training sample images can be images obtained from open-source image classification datasets in different fields. For example, the image classification dataset can be the CIFAR-100 dataset or the ImageNet dataset, etc. The theoretical classification label can be the true category to which the training sample image belongs.

[0022] In this embodiment, data augmentation refers to a method of increasing the amount of data by adding minor modifications to existing data or by recreating synthetic data based on existing data. Optionally, data augmentation may include methods such as image rotation, random cropping, image flipping, color dithering, and Gaussian noise.

[0023] In practical applications, multiple training samples can be constructed before training the student network. Then, the model can be trained based on these constructed training samples. Specifically, first, training sample images can be acquired and processed to obtain theoretical classification labels corresponding to the training sample images. Further, data augmentation processing can be performed on the acquired training sample images using a preset data augmentation method to obtain at least one sample image to be processed. Then, the at least one sample image to be processed, along with the theoretical classification labels of the corresponding training sample images, can be used as a set of training samples, thereby constructing a rich set of training samples based on the above method.

[0024] It should be noted that, in order to improve the accuracy of the model, as many and rich training samples as possible can be obtained.

[0025] It should also be noted that the benefits of data augmentation on the acquired training sample images and constructing training samples based on the data-augmented images and theoretical classification labels are as follows: by expanding the data domain to generate new supervision signals, the teacher network generates feature maps that can contain more hidden information, thereby improving the performance of the teacher network, i.e., classification accuracy, and also increasing the learnable upper limit of the student network performance.

[0026] S120. Based on the pre-trained teacher network, the training samples are processed to obtain the first smoothed probability distribution and the first model output corresponding to multiple teacher sub-models.

[0027] The teacher network can be a pre-trained, high-performance, complex network model that performs the same task as the student network. The teacher network can be used to assist in training the corresponding student network. The teacher network can be a neural network model of any structure; optionally, it can be a deep convolutional neural network model. In this embodiment, the teacher network can include multiple teacher sub-models, fully connected layers, and a classifier. Each teacher sub-model can include a teacher backbone network module and a teacher auxiliary network module. The teacher backbone network module can be a convolutional neural network model including multiple convolutional modules and multiple pooling layers. The teacher auxiliary network module can be an auxiliary branch network connected to the teacher backbone network module. For example, the teacher auxiliary network module can be a convolutional neural network including one convolutional layer. In practical applications, the teacher backbone neural network module in each teacher sub-model is sequentially connected, and then connected to the fully connected layer and the classifier; simultaneously, for each teacher sub-model, the teacher backbone network module is connected to the teacher auxiliary network module. That is, the model output of the teacher backbone neural network can be used as the model input of the corresponding teacher auxiliary network module, and thus, the model output of the teacher auxiliary network module can be used as the first model output corresponding to that teacher sub-model.

[0028] It should be noted that the number of teacher sub-models included in the teacher network can be any value, and optionally, it can be up to 3.

[0029] In this embodiment, the first smoothed probability distribution can be a smoothed classification probability distribution. In practical applications, a smoothing algorithm can be pre-deployed in the classifier. Then, after obtaining the model output of the fully connected layer, the model output can be smoothed according to the smoothing algorithm in the classifier, thereby obtaining a smoothed probability distribution. Those skilled in the art will understand that in the process of performing a classification task based on a neural network model, generally, after processing by the classifier, at least one probability value corresponding to a category can be output. Then, the category with the highest probability value can be output as the classification result. However, in the knowledge distillation process, it is necessary to extract the model output from the teacher network to train the student network. Even though the probability value corresponding to any category is negligible, it can still convey the knowledge learned by the teacher network. Therefore, in order to apply the probability values ​​corresponding to all categories output by the classifier to the training process of the student network, making the trained student network have stronger generalization ability, the probability values ​​can be smoothed to change the magnitude of the probability values ​​without changing the original probability distribution, making them comparable. After smoothing, the higher the temperature coefficient involved in the smoothing process, the smoother the smoothed probability distribution of the classifier output becomes, the greater the entropy of its distribution, the more the information carried by the negative label will be amplified, and the model training will pay more attention to the negative label.

[0030] In practical applications, after obtaining the training samples, the training samples can be input into the pre-trained teacher network. Then, the teacher network can process the sample images to be processed in the training samples to obtain the first smoothed probability distribution and the first model output corresponding to multiple teacher sub-models.

[0031] Optionally, the training samples are processed based on the pre-trained teacher network to obtain a first smoothed probability distribution and a first model output corresponding to multiple teacher sub-models. This includes: processing the training samples sequentially based on multiple teacher backbone network modules, fully connected layers, and classifiers in the teacher network to obtain the first smoothed probability distribution; for each teacher sub-model, the model output of the teacher backbone network module in the teacher sub-model is used as the model input of the teacher auxiliary network module of the teacher sub-model, and the model output of the teacher auxiliary network module is used as the first model output corresponding to the teacher sub-model.

[0032] In practical applications, after obtaining the training samples, they can be input into the teacher network. First, based on the teacher backbone network module in the first teacher sub-model, features are extracted from the image of the sample to be processed in the training samples to obtain the first teacher feature. Then, based on the teacher backbone network module in the second teacher sub-model, the first teacher feature is processed to obtain the second teacher feature. Then, based on the teacher backbone network module in the third teacher sub-model, the second teacher feature is processed to obtain the third teacher feature, and so on. After processing by multiple teacher backbone network modules included in the teacher network, the teacher feature to be processed can be obtained. Further, the teacher feature to be processed can be processed sequentially based on fully connected layers and a classifier to obtain the first smoothed probability distribution. Simultaneously, for each teacher sub-model included in the teacher network, after obtaining the model output of the teacher backbone network module in the teacher sub-model, this model output can be input into the teacher auxiliary network module connected to that teacher backbone network module for processing, thereby obtaining the first model output corresponding to the teacher sub-model. Specifically, taking the first teacher sub-model as an example, when the teacher backbone network module in the first teacher sub-model extracts features from the sample image to be processed, obtaining the first teacher features, these features can be input into both the teacher backbone network module in the next teacher sub-model and the teacher auxiliary network module in the first teacher sub-model. Then, the teacher auxiliary network module can process the first teacher features to obtain the first model output corresponding to the first teacher sub-model. It should be noted that the method for determining the first model output for each teacher sub-model included in the teacher network is the same, and will not be elaborated further in this embodiment.

[0033] It should be noted that before applying the teacher network provided in this embodiment, the teacher network to be trained can be trained first. The training process of the teacher network to be trained is described below: 1. Obtain training sample images and theoretical classification labels corresponding to the training sample images, and perform data augmentation on the training sample images to obtain at least one sample image to be processed. Based on the at least one sample image to be processed and the theoretical classification label, construct training samples; 2. Input the training samples into the teacher network to be trained to obtain the actual classification results and model outputs corresponding to multiple teacher sub-models; 3. Determine the first model loss based on the actual classification results and theoretical classification labels; 4. Determine the second model loss based on the model outputs corresponding to multiple teacher sub-models; 5. Determine the target model loss based on the first model loss and the second model loss, and adjust the model parameters of the teacher network to be trained based on the target model loss to obtain the trained teacher network.

[0034] For example, such as Figure 2The diagram shows the training process of the teacher network to be trained: 1. Data augmentation of training sample images and construction of training samples; 2. Inputting the training samples into the teacher network to be trained, and extracting features from the training samples through the backbone convolutional neural network and auxiliary networks in the teacher network to be trained; 3. Outputting the classification probability distribution and the model outputs corresponding to multiple auxiliary networks; 4. Determining the target model loss and updating the model parameters according to the gradient descent algorithm to obtain the high-performance teacher network after training.

[0035] S130. Input the training samples into the student network to be trained to obtain the actual output results and the second model output corresponding to multiple student sub-models.

[0036] In this embodiment, the model parameters in the student network to be trained can be default values. The model parameters in the student network to be trained are corrected using training samples and a pre-trained teacher network to obtain the trained student network. The student network can be a low-performance neural network model with a simple structure. The student network to be trained can be a neural network model with the same network structure as the teacher network. If the teacher network is a deep convolutional neural network model, the student network to be trained is also a deep convolutional neural network model. The student network to be trained can include multiple student sub-models, fully connected layers, and a classifier. Student sub-models can include a student backbone network module and a student auxiliary network module. The student backbone network module can be a convolutional neural network model including multiple convolutional modules and multiple pooling layers. The student auxiliary network module can be an auxiliary branch network connected to the backbone network module. For example, the student auxiliary network module can be a convolutional neural network including one convolutional layer. In practical applications, the student backbone network modules in each student sub-model are connected sequentially, and then connected to the fully connected layer and the classifier. At the same time, for each student sub-model, the student backbone network module is connected to the student auxiliary network module. That is to say, the model output of the student backbone neural network can be used as the model input of the corresponding student auxiliary network module, and thus, the model output of the student auxiliary network module can be used as the second model output of the corresponding student sub-model.

[0037] In this embodiment, the actual output includes the actual classification result and the second smoothed probability distribution. The actual classification result is the image classification category output after the training samples are input into the student network to be trained. The second smoothed probability distribution can be the classification probability distribution output after the training samples are input into the student network to be trained, after being smoothed.

[0038] In practical applications, after obtaining the training samples, the training samples can be input into the student network to be trained. Then, the training sample images in the training samples can be processed based on the student network to obtain the actual output results and the second model output corresponding to each student sub-model.

[0039] Optionally, the training samples are input into the student network to be trained to obtain the actual output results and the second model outputs corresponding to multiple student sub-models. This includes: processing the training samples based on multiple student backbone network modules, fully connected layers, and classifiers in the student network to be trained to obtain the actual output results; for each student sub-model, the model output of the student backbone network module in the student sub-model is used as the model input of the student auxiliary network module of the student sub-model, and the model output of the student auxiliary network module is used as the second model output corresponding to the student sub-model.

[0040] In practical applications, after obtaining the training samples, they can be input into the student network to be trained for processing. First, feature extraction is performed on the sample images to be processed in the training samples using the student backbone network module in the first student sub-model, yielding the first student feature. Then, the first student feature is processed using the student backbone network module in the second student sub-model, yielding the second student feature. This process continues, with the second student feature processed by the student backbone network module in the third student sub-model, yielding the third student feature. This process is repeated until the student features are obtained. Further, fully connected layers and a classifier are used to process these student features sequentially, resulting in the actual output. Simultaneously, for each student sub-model in the student network to be trained, after obtaining the model output from the student backbone network module, this output can be input into the student auxiliary network module connected to that student backbone network module for processing. This yields the second model output corresponding to the student sub-model. Specifically, taking the first student sub-model as an example, when the student backbone network module in the first student sub-model extracts features from the sample image to be processed, obtaining the first student features, these features can be input into both the student backbone network module in the next student sub-model and the student auxiliary network module in the first student sub-model. Then, the student auxiliary network module can process the first student features to obtain the second model output corresponding to the first student sub-model. It should be noted that the method for determining the corresponding second model output is the same for each student sub-model included in the student network to be trained, and will not be elaborated further in this embodiment.

[0041] S140. Based on the first smoothed probability distribution, multiple first model outputs, actual output results, multiple second model outputs, and theoretical classification labels, determine the target model loss, and adjust the model parameters of the student model to be trained based on the target model loss to obtain the image classification model.

[0042] In this embodiment, after obtaining the first smoothed probability distribution, the first model output corresponding to each teacher sub-model, the actual output result, and the second model output corresponding to each student sub-model, the target model loss can be determined based on the first smoothed probability distribution, the first model output corresponding to each teacher sub-model, the actual output result, the second model output corresponding to each student sub-model, and the theoretical classification labels in the training samples. The target model loss can be understood as the loss value used by the student network to be trained when adjusting its model parameters.

[0043] In practical applications, the actual output includes the actual classification result and the second smoothed probability distribution. When training the student network based on a pre-trained teacher network, the classification loss of the student network can be determined based on the difference between the actual classification result and the theoretical classification label; the contrastive loss between the teacher network and the student network for the classification prediction result can be determined based on the difference between the first and second smoothed probability distributions; the contrastive loss between the teacher network and the student network for the output of the auxiliary network module can be determined based on the difference between multiple first model outputs and multiple second model outputs; and the additional loss between the output of the student network's auxiliary network module and the self-supervised label can be determined based on multiple second model outputs. Furthermore, the target model loss can be determined based on these four types of losses.

[0044] Optionally, the target model loss is determined based on the first smoothed probability distribution, multiple first model outputs, actual output results, multiple second model outputs, and theoretical classification labels. This includes: determining the first model loss based on the actual classification results and theoretical classification labels in the actual output results; determining the second model loss based on the multiple second model outputs; determining the third model loss based on the first smoothed probability distribution and the second smoothed probability distribution in the actual output results; determining the fourth model loss based on the multiple first model outputs and multiple second model outputs; and obtaining the target model loss based on the first model loss, second model loss, third model loss, and fourth model loss.

[0045] In this embodiment, the first model loss can be the difference between the actual classification result and the theoretical classification label. The second model loss can be the difference between the second model outputs of multiple auxiliary network modules in the student network to be trained and the self-supervised labels. The third model loss can be the difference between the first smoothed probability distribution of the teacher network output and the second smoothed probability distribution of the student network output. The fourth model loss can be the difference between the first model output of the teacher network auxiliary network module and the second model output of the student network auxiliary network module.

[0046] Optionally, the first model loss is determined based on the actual classification result and theoretical classification label in the actual output result, including: performing loss processing on the actual classification result and theoretical classification label according to the pre-set first loss function to obtain the first model loss.

[0047] In this embodiment, the first loss function can be any loss function, and optionally, it can be the cross-entropy loss function.

[0048] In practical applications, given a known actual classification result, a loss function can be applied to both the actual classification result and the theoretical classification labels in the training samples. This loss value can then be used as the first model loss.

[0049] For example, assuming the first loss function can be the cross-entropy loss function, the first model loss can be determined according to the following formula:

[0050]

[0051] in, This represents the loss of the first model; x represents the training sample image with a temperature coefficient of 1. i The probability distribution of y (i.e., the actual classification result); i Indicates theoretical classification labels.

[0052] Optionally, the second model loss can be determined based on multiple second model outputs, including: performing loss processing on the multiple second model outputs according to the first loss function to obtain the second model loss.

[0053] In practical applications, to improve the prediction results of fine-grained classification tasks, the student network to be trained can be self-distilled based on the output of the auxiliary network modules. Therefore, given the second model outputs of each auxiliary network module in the student network to be trained, the loss values ​​can be obtained by applying a first loss function to the multiple second model outputs and the pre-determined self-supervised labels, and these loss values ​​can be used as the second model loss.

[0054] For example, assuming the first loss function is the cross-entropy loss function, the second model loss can be determined according to the following formula:

[0055]

[0056] in, The second model loss is represented by U; U represents the loss of the training sample image x. i The number of transformations for data augmentation; l represents the nth auxiliary network module; trans(·) represents any data augmentation method; τ represents the temperature coefficient for smoothing the output of the second model; c j Labels indicating the relationship between the data-augmented images and the training sample images.

[0057] Optionally, the third model loss is determined based on the first smoothed probability distribution and the second smoothed probability distribution in the actual output result, including: performing loss processing on the first smoothed probability distribution and the second smoothed probability distribution according to a pre-set second loss function to obtain the third model loss.

[0058] In this embodiment, the second loss function can be any loss function, optionally, it can be the KL divergence loss function. The KL divergence loss function can represent the degree of difference between one probability distribution and another.

[0059] In practical applications, given the first smoothed probability distribution of the teacher network output and the second smoothed probability distribution of the student network output to be trained, the first and second smoothed probability distributions can be processed by the second loss function to obtain the loss value, which can be used as the third model loss.

[0060] For example, assuming the second loss function is the KL divergence loss function, the third model loss can be determined according to the following formula:

[0061]

[0062] Among them, L KL_trans The third model loss is represented by U; U represents the loss of the training sample image x. i The number of transformations for data augmentation; τ represents the temperature coefficient of the smoothing operation; D KL (·) represents the KL divergence; p T (trans j (x); τ) represents the first smoothed probability distribution; p S (trans j (x);τ) denotes the second smoothed probability distribution.

[0063] Optionally, the fourth model loss is determined based on multiple first model outputs and multiple second model outputs, including: performing loss processing on the multiple first model outputs and multiple second model outputs according to the second loss function to obtain the fourth model loss.

[0064] In practical applications, given the first model output of each auxiliary network module in the teacher network and the second model output of each auxiliary network module in the student network to be trained, the loss can be processed on the multiple first model outputs and multiple second model outputs according to the second loss function to obtain the loss value, and this loss value is used as the fourth model loss.

[0065] For example, assuming the second loss function is the KL divergence loss function, the fourth model loss can be determined according to the following formula:

[0066]

[0067] Among them, L KL_con The fourth model loss is represented by U; U represents the loss of the training sample image x. i The number of transformations for data augmentation; τ represents the temperature coefficient of the smoothing operation; D KL (·) represents the KL divergence; l represents the nth auxiliary network module; This represents the output of the first model; This represents the output of the second model.

[0068] In practical applications, after determining the first model loss, the second model loss, the third model loss, and the fourth model loss, the target model loss can be determined based on the first model loss and its corresponding weights, the second model loss and its corresponding weights, the third model loss and its corresponding weights, and the fourth model loss and its corresponding weights.

[0069] Optionally, the target model loss is obtained based on the first model loss, the second model loss, the third model loss, and the fourth model loss, including: determining the first weight value corresponding to the first model loss, the second weight value corresponding to the second model loss, the third weight value corresponding to the third model loss, and the fourth weight value corresponding to the fourth model loss; multiplying the first model loss and the first weight value to obtain the first unprocessed loss value; multiplying the second model loss and the second weight value to obtain the second unprocessed loss value; multiplying the third model loss and the third weight value to obtain the third unprocessed loss value; and multiplying the fourth model loss and the fourth weight value to obtain the fourth unprocessed loss value; and adding the first unprocessed loss value, the second unprocessed loss value, the third unprocessed loss value, and the fourth unprocessed loss value to obtain the target model loss.

[0070] In this embodiment, the first weight value can be a numerical value representing the proportion of the first model loss in the target model loss. The second weight value can be a numerical value representing the proportion of the second model loss in the target model loss. The third weight value can be a numerical value representing the proportion of the third model loss in the target model loss. The fourth weight value can be a numerical value representing the proportion of the fourth model loss in the target model loss.

[0071] In practical applications, the first weight value corresponding to the first model loss, the second weight value corresponding to the second model loss, the third weight value corresponding to the third model loss, and the fourth weight value corresponding to the fourth model loss are determined separately. Further, the first model loss is multiplied by the first weight value to obtain a single value, which can be used as the first unprocessed loss value; the second model loss is multiplied by the second weight value to obtain a single value, which can be used as the second unprocessed loss value; the third model loss is multiplied by the third weight value to obtain a single value, which can be used as the third unprocessed loss value; and the fourth model loss is multiplied by the fourth weight value to obtain a single value, which can be used as the fourth unprocessed loss value. Finally, the first, second, third, and fourth unprocessed loss values ​​are added together to obtain the target model loss.

[0072] For example, the target model loss can be determined based on the following formula:

[0073]

[0074] Where L represents the target model loss; α1 represents the first weight value; α represents the first model loss; α2 represents the second weight value; α represents the second model loss; α3 represents the third weight value; L KL_trans α4 represents the third model loss; L represents the fourth weight value; KL_con This represents the loss of the fourth model.

[0075] Furthermore, the model parameters of the student network to be trained can be adjusted according to the target model loss, thereby obtaining an image classification model.

[0076] In practice, the convergence of the loss function corresponding to the target model loss can be used as a training objective. For example, whether the training error is less than a preset error; whether the error change tends to stabilize; or whether the current number of iterations equals a preset number. If convergence conditions are met, such as the training error of the loss function corresponding to the target model loss being less than the preset error; or the error change trend tending to stabilize, it indicates that the training of the student network is complete, and iterative training can be stopped. If convergence conditions are not met, other training samples can be obtained to continue training the student network until the training error of the target model loss is within a preset range. When the training error of the target model loss converges, the trained student network can be used as an image classification model. That is, at this point, inputting the image to be classified into the image classification model will accurately obtain the category corresponding to the image.

[0077] For example, such as Figure 3 The diagram shown is a schematic of the multi-stage comparative knowledge distillation method.

[0078] The technical solution of this invention obtains training sample images and corresponding theoretical classification labels, and performs data augmentation on the training sample images to obtain at least one image to be processed. Based on the at least one image to be processed and the theoretical classification labels, training samples are constructed. Then, the training samples are processed based on a pre-trained teacher network to obtain a first smoothed probability distribution and first model outputs corresponding to multiple teacher sub-models. Further, the training samples are input into a student network to be trained to obtain actual output results and second model outputs corresponding to multiple student sub-models. Finally, based on the first smoothed probability distribution, multiple first model outputs, actual output results, multiple second model outputs, and theoretical classification labels, the target model loss is determined, and the model parameters of the student model to be trained are adjusted based on the target model loss to obtain an image classification model. This solves the problems in related technologies, such as the inability to fully transfer knowledge to the student network, the difficulty in fully utilizing the knowledge contained in the teacher network to improve the representation ability of lightweight networks, the limited knowledge transfer effect, and the low accuracy of neural network model compression. It achieves the effect of transferring richer knowledge from the teacher network to the student network, allowing the performance of the student network to approach or even exceed the performance of the teacher network, thereby achieving the goal of high performance from a simple model.

[0079] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-stage contrastive knowledge distillation method, characterized in that, include: Obtain training sample images and theoretical classification labels corresponding to the training sample images, and perform data augmentation on the training sample images to obtain at least one sample image to be processed. Based on the at least one sample image to be processed and the theoretical classification labels, construct training samples. The training samples are processed based on the pre-trained teacher network to obtain a first smoothed probability distribution and a first model output corresponding to multiple teacher sub-models; The training samples are input into the student network to be trained to obtain the actual output results and the second model output corresponding to multiple student sub-models. The actual output results include the actual classification results and the second smoothed probability distribution. Based on the first smoothed probability distribution, multiple first model outputs, the actual output results, multiple second model outputs, and the theoretical classification labels, the target model loss is determined, and the model parameters of the student network to be trained are adjusted based on the target model loss to obtain an image classification model. The step of determining the target model loss based on the first smoothed probability distribution, multiple first model outputs, the actual output results, multiple second model outputs, and the theoretical classification label includes: determining a first model loss based on the actual classification results in the actual output results and the theoretical classification label; determining a second model loss based on multiple second model outputs; determining a third model loss based on the first smoothed probability distribution and the second smoothed probability distribution in the actual output results; determining a fourth model loss based on multiple first model outputs and multiple second model outputs; and obtaining the target model loss based on the first model loss, the second model loss, the third model loss, and the fourth model loss.

2. The method according to claim 1, characterized in that, The teacher network includes multiple teacher sub-models, fully connected layers, and a classifier. Each teacher sub-model includes a teacher backbone network module and a teacher auxiliary network module. The training samples are processed based on the pre-trained teacher network to obtain a first smoothed probability distribution and a first model output corresponding to the multiple teacher sub-models, including: The training samples are processed sequentially based on the multiple teacher backbone network modules, fully connected layers, and classifiers in the teacher network to obtain the first smoothed probability distribution; For each teacher sub-model, the model output of the teacher backbone network module in the teacher sub-model is used as the model input of the teacher auxiliary network module in the teacher sub-model, and the model output of the teacher auxiliary network module is used as the first model output corresponding to the teacher sub-model.

3. The method according to claim 1, characterized in that, The student network to be trained includes multiple student sub-models, fully connected layers, and a classifier. Each student sub-model includes a student backbone network module and a student auxiliary network module. The process of inputting the training samples into the student network to obtain the actual output results and the second model outputs corresponding to the multiple student sub-models includes: The training samples are processed based on the multiple student backbone network modules, fully connected layers, and classifiers in the student network to be trained to obtain the actual output results; For each student sub-model, the model output of the student backbone network module in the student sub-model is used as the model input of the student auxiliary network module in the student sub-model, and the model output of the student auxiliary network module is used as the second model output corresponding to the student sub-model.

4. The method of claim 1, wherein, The step of determining the first model loss based on the actual classification result in the actual output and the theoretical classification label includes: The first model loss is obtained by applying a loss function to the actual classification result and the theoretical classification label.

5. The method according to claim 4, characterized in that, The step of determining the second model loss based on multiple second model outputs includes: The loss of the second model is obtained by applying the first loss function to the outputs of multiple second models.

6. The method of claim 1, wherein, The step of determining the third model loss based on the first smoothed probability distribution and the second smoothed probability distribution in the actual output result includes: The first smoothed probability distribution and the second smoothed probability distribution are subjected to loss processing according to the pre-set second loss function to obtain the third model loss.

7. The method of claim 6, wherein, The step of determining the fourth model loss based on multiple first model outputs and multiple second model outputs includes: The loss of the fourth model is obtained by applying the loss function to the multiple first model outputs and multiple second model outputs.

8. The method of claim 1, wherein, The step of obtaining the target model loss based on the first model loss, the second model loss, the third model loss, and the fourth model loss includes: A first weight value corresponding to the first model loss, a second weight value corresponding to the second model loss, a third weight value corresponding to the third model loss, and a fourth weight value corresponding to the fourth model loss are determined respectively; Multiply the first model loss and the first weight value to obtain the first unprocessed loss value; multiply the second model loss and the second weight value to obtain the second unprocessed loss value; multiply the third model loss and the third weight value to obtain the third unprocessed loss value; and multiply the fourth model loss and the fourth weight value to obtain the fourth unprocessed loss value. The target model loss is obtained by adding the first unprocessed loss value, the second unprocessed loss value, the third unprocessed loss value, and the fourth unprocessed loss value.

Images