An image classification method based on online knowledge distillation

Abstract

The application relates to the technical field of image classification, and discloses an image classification method based on online knowledge distillation, which comprises the following steps: inputting an image to be classified, calibrating a portrait in image data, constructing a training image set, extracting global features of the image, superimposing local features in the global features, constructing a plurality of equal network frameworks, constructing an integrated teacher model, aligning the probability distribution between the integrated teacher model and the equal networks, introducing a feature embedding space, obtaining a total contrast loss function under M equal networks, introducing a weight factor to obtain a total loss function of the classification model, training the data set into a training set to obtain an image classification result, calibrating a local feature region separately by extracting the local feature region, separating the local feature region from other regions of the image, obtaining an image classification model in a multi-feature space, and fusing the global features of the image and the local features to obtain a more generalized image classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and particularly relates to an image classification method based on online knowledge distillation. BACKGROUND

[0002] Online knowledge distillation is a knowledge transfer model of mutual cooperation between multiple student models trained from scratch, and is a knowledge distillation method of a teacher-free model. Existing online knowledge distillation methods focus on transferring instance-level class probability distributions between various student models, and ignore the representation knowledge with a larger amount of information for online transfer. Research has found that a multi-view contrastive learning method for online knowledge distillation can extract phenotypic features in images, mainly by capturing the correlation of the encoding representations of data instances between multiple peer networks, so as to maximize the consistency of the representations of the same input instances from different views, and simultaneously spread the representations of input instances with different labels from different views. In the framework of multiple peer networks, a fully connected layer is used to linearly transform the representations to a contrastive embedding layer, and an integrated teacher model is constructed in all peer networks. The knowledge distillation probability of the integrated teacher model is transmitted to a specific student network for the final image classification. However, the model does not consider the multi-target nature of the image itself, especially when it comes to multi-person feature images. Due to the phenomenon of mutual covering of the person targets in the image itself, the knowledge distillation probability in the integrated teacher model deviates from the actual situation, and the introduction of the knowledge distillation probability with a large error to the student network directly leads to a decrease in the correct probability of the final image classification result. Therefore, in view of the complex characteristics of the multi-target object of the person image, how to improve the accuracy of feature extraction of the image person is of great significance to the accuracy of the image classification result. SUMMARY

[0003] The present application aims to provide an image classification method based on online knowledge distillation, which can effectively solve the problems in the background art by extracting local feature regions, individually labeling the local feature regions, distinguishing the local feature regions from other regions of the image, obtaining an image classification model in a multi-feature space, and fusing the global features and the local features of the image to obtain a more generalized image classification model.

[0004] To achieve the above object, the present application provides the following technical scheme:

[0005] An image classification method based on online knowledge distillation comprises the following steps:

[0006] Step 1: input the image to be classified, label the portraits in the image data, and divide the image into separate target regions (Ω1), person overlapping regions (Ω2) and background regions (Ω3) based on the number and overlap degree of the recognition boxes in the image data;

[0007] Step 2: Construct the training graph set, including the local training graph set and the global training graph set;

[0008] Step 3: Extract global features from the image, and overlay local features onto the global features. Use the global segmentation results from the image dataset as supervision information to guide the parameter updates of the local feature extraction module. Define the global feature vector as F. For individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3), feature vectors F for different regions can be obtained respectively. i An embedding layer is added to project the data into an i-dimensional feature vector, where i represents the number of sample classes in the dataset. The resulting feature vector is considered as the prediction vector of the image training model for individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3). The knowledge distillation loss L between the global feature extraction model and the local feature extraction model is defined. i =-Flog(σ(F) i ))-(1-F)log(1-σ(F i ), where σ represents the sigmoid activation function;

[0009] Step 4: Construct multiple identical network frameworks, each trained using cross-entropy loss between predicted probability distributions and hard labels;

[0010] Step 5: Construct an ensemble teacher model. By performing Naive Bayes ensemble classification on the predicted probability distributions of all equivalent networks, the soft clustering probability distribution of the m-th network is calculated.

[0011] Step 6: Transfer the output probability distribution of the ensemble teacher model to the equivalent network framework, aligning the probability distribution between the ensemble teacher model and the equivalent network;

[0012] Step 7: For a given training set D contains multiple N instances with class C. It learns the feature relationships between different networks, introduces a feature embedding space, and establishes a fully connected interaction model through multi-view comparative learning to obtain the total contrastive loss function under M equivalent networks.

[0013] Step 8: Introduce weighting factors to obtain the total loss function of the classification model;

[0014] Step 9: Based on the total loss function of the image classification model, incorporate the dataset into the training set for training and obtain the image classification results.

[0015] As a further scheme of the present application: the knowledge distillation mode between the global feature extraction model and the local feature extraction model in step three is a hybrid knowledge distillation, which can remove the interference of noise signals in the global feature model and ensure that the samples of the training graph set can be fully trained.

[0016] As a further scheme of the present application: the cross-entropy loss in step four is represented as wherein p m (y|x) is a normalized probability value by using a softmax activation function, and represents a normalized function value, and the cross-entropy loss of M equal networks is

[0017] As a further scheme of the present application: the soft clustering probability distribution of the mth network in step five is obtained by introducing a softening factor T to soften the data, and represents the soft clustering probability of the cth class, and the soft clustering probability distribution of the mth network is

[0018] As a further scheme of the present application: step six is to align the probability distribution between the integrated teacher model and the equal network by using KL divergence, that is,

[0019] As a further scheme of the present application: the total contrastive loss function of M equal networks in step seven is obtained by calculating the two model view contrastive loss, and the total contrastive loss function L of M equal networks is c =∑ 1≤a＜b＜M L(f a ,f b )。

[0020] As a further scheme of the present application: the two model view contrastive loss in step seven is obtained by defining the integrated teacher network model as fa and a certain equal network model as fb, and the embedded features generated in the training set D are and The positive sample data pair is defined as At this time, i=j, if i≠j, is a negative sample data pair, the multi-classification at the instance level is converted into binary classification of discriminating positive samples and negative samples, and for a given corner point and the probability of matching is the probability of positive samples wherein τ represents a characteristic factor, Zi represents a normalization constant, and the probability of is uniformly distributed, matching is the probability of being a positive sample In addition, define that the sampling frequency exists in every k negative samples and 1 positive sample, and k << N, then the posterior probability of can be obtained from the actual distribution of the positive samples, that is Define D = 1, combined with the maximum likelihood estimation method, the comparative loss of the integrated teacher network model fa to the peer network model fb is obtained The symmetry can obtain the comparative loss Lcb of the peer network model fb to the integrated teacher network model fa, based on the two model views, it is expanded to multiple model views, a fully connected interaction model is established, and the total comparative loss function L of the M peer networks is obtained c = ∑ 1≤a＜b＜M L(f a , f b ).

[0021] As a further scheme of the application: the total loss function L of the classification model in step eight is L ce + T 2 · L kl + β· L c , T 2 represents the balance weight between soft and hard labels, and β is a constant factor for scaling the comparative loss between models, and the value range is 0-1. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Fig. 1 It is a structural schematic diagram of an image classification method based on online knowledge distillation.

[0024] Fig. 2 It is a flowchart of an image classification method based on online knowledge distillation. DETAILED DESCRIPTION

[0025] In order to make the technical problems to be solved by the application, the technical solutions and the beneficial effects more clearly, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0026] Example 1:

[0027] Please see Figs. 1-2 In this embodiment of the invention, an image classification method based on online knowledge distillation includes the following steps:

[0028] Step 1: Input the image to be classified, label the human figures in the image data, and divide the image into separate target regions (Ω1), human figure overlapping regions (Ω2), and background regions (Ω3) based on the number and overlap of recognition boxes in the image data.

[0029] Step 2: Construct the training graph set, including the local training graph set and the global training graph set;

[0030] Step 3: Extract global features from the image and overlay local features onto them. To remove noise interference from the global feature model and ensure sufficient training of the training dataset, a hybrid knowledge distillation method is used between the global and local feature extraction models. This method uses the global segmentation results from the image dataset as supervision information to guide the parameter updates of the local feature extraction module. The global feature vector is defined as F. For individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3), feature vectors F1 for each region can be obtained. i An embedding layer is added to project the data into an i-dimensional feature vector, where i represents the number of sample classes in the dataset. The resulting feature vector is considered as the prediction vector of the image training model for individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3). The knowledge distillation loss L between the global feature extraction model and the local feature extraction model is defined. i =-Flog(σ(F) i ))-(1-F)log(1-σ(F i ), where σ represents the sigmoid activation function;

[0031] Step 4: Construct multiple identical network frameworks. Each network is trained using the cross-entropy loss between the predicted probability distribution and hard labels. Given an instance x with label y, the cross-entropy loss of the m-th network is expressed as:

[0032] in p m (y|x) is the normalized probability value obtained by using the softmax activation function, and Let represent the normalized function value, then the cross-entropy loss for M identical networks.

[0033] Step five: build the ensemble teacher model, through the prediction probability distribution of all peer networks, introduce the softening factor T to soften the data, get The soft clustering probability of the mth network is

[0034] Step six: transfer the output probability distribution of the ensemble teacher model to the peer network framework, and align the probability distribution between the ensemble teacher model and the peer network by using the KL divergence, that is

[0035] Step seven: for a given training set D contains multiple N instances with C classes, learns the feature relationship between different networks, introduces the feature embedding space, and establishes a fully connected interaction model through multi-view contrastive learning to get the total contrastive loss function of M peer networks;

[0036] First, calculate the view contrastive loss of two models, define the ensemble teacher network model as fa, a peer network model as fb, and the embedding features generated in the training set D as and Define the positive sample data pair as At this time, i = j, if i ≠ j, is the negative sample data pair, which converts the multi-classification at the instance level to the binary classification of distinguishing positive samples and negative samples. For a given corner point and The probability of matching is the probability of positive samples Where τ represents the feature factor, and Zi represents the normalization constant, and The probability of matching is the probability of positive samples In addition, define that the sampling frequency exists in every k negative samples and 1 positive sample, and k << N, then The posterior probability of can be obtained from the actual distribution of positive samples, that is Define D = 1, and combine the maximum likelihood estimation method to get the contrastive loss of the ensemble teacher network model fa to the peer network model fb The symmetry can get the contrastive loss Lcb of the peer network model fb to the ensemble teacher network model fa;

[0037] Second, based on the expansion of two model views to multi-model views, a fully connected interaction model is established to get the total contrastive loss function Lc of M peer networks 1≤a＜b＜M L(f af b );

[0038] Step 8: Introduce weighting factors to obtain the total loss function L = L for the classification model. ce +T 2 ·L kl +β·L c T 2 This represents the balance weight between the soft and hard labels. β is a constant factor used to scale the comparative loss between models, and its value ranges from 0 to 1.

[0039] Step 9: Based on the total loss function of the image classification model, incorporate the dataset into the training set for training and obtain the image classification results.

[0040] A classification evaluation metric is introduced to assess the accuracy of the image classification model. Recall (Re), precision (Pr), F1 score, and accuracy (Ac) are selected as evaluation indicators. TP, TN, FP, and FN represent the relationship between the prediction accuracy of the classification results and the actual scenario. TP indicates that the predicted sample is positive and the actual sample is positive; TN indicates that the predicted sample is negative and the actual sample is negative; FP indicates that the predicted classification result is positive and the actual sample is negative; and FN indicates that the predicted classification result is negative and the actual sample is positive. As can be seen from the calculation results, F1 is the harmonic mean of TP, TN, FP, and FN. The higher the F1, the stronger the generalization ability of the classification model.

[0041] By adopting the above technical solution: by extracting local feature regions and labeling them separately, the local feature regions are separated from other regions of the image, resulting in a multi-feature space image classification model. By fusing global and local features of the image, a more generalized image classification model can be obtained.

[0042] Example 2:

[0043] Please see Figs. 1-2 In this embodiment of the invention, an image classification method based on online knowledge distillation includes the following steps:

[0044] Step 1: Input the image to be classified. First, use super-resolution technology to process the acquired low-resolution image to improve the image resolution and reduce the impact of low image resolution on image feature labeling. Second, label the human figures in the image data. Based on the number and overlap of the recognition boxes in the image data, the image is divided into separate target regions (Ω1), human figure overlap regions (Ω2), and background regions (Ω3).

[0045] Step 2: Construct the training graph set, including the local training graph set and the global training graph set;

[0046] Step three: extract the global features of the image, superimpose the local features in the global features, use the global segmentation results of the image dataset as supervision information to guide the parameter update of the local feature extraction module, define the global feature vector as F, and obtain the feature vectors of different regions F i , add an embedding layer to project it into an i-dimensional feature vector, i represents the number of sample categories in the dataset, and the final obtained feature vector is regarded as the prediction vector of the image training model for the single target region (Ω1), the person overlapping region (Ω2) and the background region (Ω3), define the knowledge distillation loss L i between the global feature extraction model and the local feature extraction model i = -F log(σ(F i ))-(1-F)log(1-σ(F m )), where σ represents the sigmoid activation function;

[0047] Step four: construct multiple equivalent network frameworks, each network is trained through the cross-entropy loss between the prediction probability distribution and the hard label, and through a given instance x with a label y, the cross-entropy loss of the mth network is represented as where p m (y|x) is the normalized probability value by using the softmax activation function, and represents the normalized function value, then for the cross-entropy loss of M equivalent networks

[0048] Step five: construct an integrated teacher model, perform naive Bayesian ensemble classification on the prediction probability distribution of all equivalent networks, introduce a softening factor T to soften the data, and obtain represents the soft clustering probability of the cth class, and the soft clustering probability distribution of the mth network is further calculated as

[0049] Step six: transfer the output probability distribution of the integrated teacher model to the equivalent network framework, align the probability distribution between the integrated teacher model and the equivalent network, that is

[0050] Step seven: for a given training set D contains multiple N instances with C classes, learns the feature relationship between different networks, introduces a feature embedding space, and establishes a fully connected interaction model through multi-view contrastive learning to obtain the total contrastive loss function of M equivalent networks;

[0051] First, calculate the two model view contrast loss, by defining the integrated teacher network model as fa, a certain peer network model as fb, and the generated embedding features in the training set D as and Define the positive sample data pair as At this time, i=j, if i≠j, is a negative sample data pair, and the instance-level multi-classification is converted into a binary classification of discriminating positive samples and negative samples. For a given corner point and The probability of matching is the probability of a positive sample Where τ represents the characteristic factor, and Zi represents the normalization constant, and The probability of matching is the probability of a positive sample In addition, define the sampling frequency to exist in every k negative samples and 1 positive sample, and k<<N, then The posterior probability of is obtained from the actual distribution of positive samples, that is Define D=1, and combine the maximum likelihood estimation method to obtain the contrast loss of the integrated teacher network model fa to the peer network model fb The symmetry can obtain the contrast loss Lcb of the peer network model fb to the integrated teacher network model fa;

[0052] Second, based on the expansion of two model views to multiple model views, a fully connected interaction model is established, and the total contrast loss function L of M peer networks is obtained c =∑ 1≤a＜b＜M L(f a ,f b );

[0053] Step eight: introduce the weight factor to obtain the total loss function of the classification model, L=L ce +T 2 ·L kl +β·L c , T 2 represents the balance weight between soft and hard labels, and β is a constant factor for scaling the contrast loss between models, with a value range of 0-1;

[0054] Step nine: according to the total loss function of the image classification model, the data set is included in the training set to train, and the image classification result is obtained.

[0055] The above method adds extra local region feature learning, which may cause image overfitting due to excessive parameters. Therefore, the formula to reduce the correlation between feature maps, which can prevent overfitting of the feature maps, is an input value, is an output value, (x, y) is the coordinate position of the target image pixel, i, j is the feature map channel serial number, N is the total channel number, k, alpha, beta are self-defined hyperparameters, the prediction result is verified by using visual analysis theory, the prediction distribution change before and after the transition region feature analysis is added can be more intuitively seen, with the increase of the number of images in the training set, the error rate or accuracy of the image training result will change to a certain extent, therefore, in the image person training process, the positive sample data set and the negative sample data set should be balanced as much as possible, so that the phenomenon that the classification performance of image training is unstable can be avoided.

[0056] The working principle of the application is that: a multi-feature fusion model is established by introducing local features into global features to fuse and classify the image data set, and the feature learning parameters are processed by dimension reduction, so that the phenomenon of overfitting of the feature map does not occur when the image data set is complex, the operation ability of data classification processing and detection recognition is guaranteed, and the efficiency of image classification is improved.

[0057] The above description is only a preferred embodiment of the application, but the protection scope of the application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the application within the technical range disclosed by the application, which should be covered in the protection scope of the application.

Claims

1. An image classification method based on online knowledge distillation, characterized in that, Includes the following steps: Step 1: Input the image to be classified, and label the human figures in the image data. Based on the number and overlap of the recognition boxes in the image data, the image is divided into a separate target region (Ω1), a human figure overlap region (Ω2), and a background region (Ω3). Step 2: Construct the training graph set, including the local training graph set and the global training graph set; Step 3: Extract global features from the image, and overlay local features onto the global features. Use the global segmentation results from the image dataset as supervision information to guide the parameter updates of the local feature extraction module. Define the global feature vector as F. For individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3), feature vectors F for different regions can be obtained respectively. i An embedding layer is added to project the data into an i-dimensional feature vector, where i represents the number of sample categories in the dataset. The resulting feature vector is considered as the prediction vector of the image training model for individual target regions (Ω1), overlapping areas of people (Ω2), and background regions (Ω3). The knowledge distillation loss between the global feature extraction model and the local feature extraction model is defined. , where σ represents the sigmoid activation function; Step 4: Construct multiple identical network frameworks. Each network is trained using the cross-entropy loss between the predicted probability distribution and the hard labels. The cross-entropy loss is obtained by taking an instance x with label y and expressing it as follows: ,in , It is achieved by utilizing the normalized probability values ​​of the softmax activation function, and Let represent the normalized function value, then the cross-entropy loss for M identical networks. ; Step 5: Construct an ensemble teacher model. Calculate the soft clustering probability distribution of the m-th network by performing Naive Bayes ensemble classification on the predicted probability distributions of all identical networks. The calculation of the soft clustering probability distribution of the m-th network is achieved by introducing a softening factor T to soften the data. c∈{1,2,3,...,C} Let represent the soft clustering probability of the c-th class. Then the soft clustering probability distribution of the m-th network is: ； Step Six: Transfer the output probability distribution of the ensemble teacher model to the peer network framework, aligning the probability distributions between the ensemble teacher model and the peer network; Step Six aligns the probability distributions between the ensemble teacher model and the peer network by utilizing KL divergence, i.e. ; Step 7: For a given training set D contains multiple N instances of class C. We learn the feature relationships between different networks, introduce a feature embedding space, and establish a fully connected interaction model through multi-view comparative learning to obtain the total contrastive loss function under M equivalent networks. Step 8: Introduce weighting factors to obtain the total loss function of the classification model; Step 9: Based on the total loss function of the image classification model, incorporate the dataset into the training set for training and obtain the image classification results.

2. The image classification method based on online knowledge distillation according to claim 1, characterized in that, The knowledge distillation method between the global feature extraction model and the local feature extraction model in step three is a hybrid knowledge distillation, which can remove the interference of noise signals in the global feature model and ensure that the samples of the training set can be fully trained.

3. The image classification method based on online knowledge distillation according to claim 1, characterized in that, In step seven, the total contrastive loss function for M equivalent networks is calculated by extending the contrastive loss between two model views to obtain the total contrastive loss function for M equivalent networks. .

4. The image classification method based on online knowledge distillation according to claim 3, characterized in that, In step seven, the loss between the two model views is calculated by defining the ensemble teacher network model as fa and a certain equivalent network model as fb, and the embedded features generated in the training set D are... and Define positive sample data pairs as At this point, i = j. If i ≠ j, For negative sample data pairs, the instance-level multi-class classification is transformed into a binary classification that distinguishes between positive and negative samples. For a given corner point... and The probability of matching The probability of being a positive sample ,in τ represents the characteristic factor, Zi represents the normalization constant, and at the same time, we define... The probability of matching is uniformly distributed. The probability of being a positive sample Furthermore, the sampling frequency is defined to exist in every k negative samples and 1 positive sample, and Then it is possible to obtain from the actual distribution of positive samples The posterior probability, i.e. Defining D=1, and using the maximum likelihood estimation method, we obtain the contrastive loss between the ensemble teacher network model fa and the equivalent network model fb. Symmetrically, we can obtain the contrastive loss Lcb from the equivalent network model fb to the ensemble teacher network model fa. Based on the two-model view, we extend to a multi-model view, establish a fully connected interaction model, and obtain the total contrastive loss function under M equivalent networks. 。 5. The image classification method based on online knowledge distillation according to claim 1, characterized in that, The total loss function of the classification model in step eight , This represents the balance weight between the soft and hard labels. β is a constant factor used to scale the contrast loss between models, with a value ranging from 0 to 1.

Images