Semi-supervised multi-label image classification method based on inter-class relationship with adaptive weights

Abstract

The present invention discloses a semi-supervised multi-label image classification method based on inter-class relations with adaptive weights; normalization operation is performed on pre-acquired images to construct a semi-supervised classification data set including supervised data and unsupervised data; a weight calculation and prediction fusion model including an original classifier, a class-related classifier, and a weight calculation module is constructed; and the classifier and the weight calculation module are trained; and the image classification result is predicted using the trained weight calculation and prediction fusion model. The present invention can train the weight calculation and prediction fusion model and predict the classification result of the target image within an effective time, and can be migrated to a variety of mainstream multi-label classification models based on inter-class relations at the same time. It is simple to implement, suitable for actual application environments, and has considerable use value.

Description

Technical Field

[0001] The invention belongs to the field of image classification, and in particular relates to a semi-supervised multi-label image classification method based on inter-class relations with adaptive weights. Background Art

[0002] The development of deep learning has important implications for the performance improvement of feature learning and classification tasks. However, training deep networks requires a large number of labeled samples. In practical applications, most of the available samples lack category labels, which makes classification training particularly difficult. In the face of this difficulty, the usual approach is to train an effective classifier from a small number of labeled samples and add reasonable auxiliary information to the unlabeled samples. Semi-supervised learning is one of the most promising research topics in the current field of computer vision. In recent years, there has been an increasing amount of research on semi-supervised image classification, which can be roughly divided into the following two categories from the perspective of optimization methods: (1) entropy minimization method; (2) consistency regularization method. The entropy minimization method is obtained by minimizing the entropy predicted by the model. The model gradually selects unsupervised data with higher confidence from the supervised data set, labels them falsely, and adds them to the supervised training set, thereby realizing the utilization of unsupervised data and improving the generalization performance of the model; this type of method is based on the assumption that the distribution of unsupervised data, supervised data and test sets are similar, and that simple unsupervised samples can be correctly labeled by the model trained on supervised data, thereby improving generalization by increasing the number of samples. This type of method can utilize unsupervised data, but how to utilize a sufficient amount of unsupervised data while ensuring the high accuracy and confidence of these pseudo labels is a problem that requires continuous exploration and experimentation. The consistency regularization method improves the representation ability of the model by utilizing data invariance. In recent years, the representation ability of deep learning has been enhanced. Many researchers have used the consistency of the same image between multiple models, or multiple data-enhanced versions of the same image in a model to constrain the model, hoping that the model can predict the same results as the original image for multiple enhanced versions of the same data or images with noise, thereby improving the representation ability of the model to be suitable for the classification of the test set. This type of method has had obvious effects in computer vision applications in recent years, but for different computer vision tasks, how to more fully mine consistency information requires more exploration.

[0003] Different from the single-label multi-classification problem, in the real-world multi-label classification problem of natural images, objects of multiple classes often appear in the same image at the same time, which increases the difficulty of learning entropy minimization for single-label semi-supervised learning and limits the design of consistency constraints. At the same time, in recent years, multi-label image classification methods based on inter-class relations have dominated. If there is not enough supervised data, the knowledge of inter-class relations will become unreliable, which will damage the generalization of the model and the accuracy during testing. At the same time, the inter-class relations of current multi-label classification methods mostly have the problem of long-tail distribution, that is, for images with atypical co-occurrence relations, the inter-class relations will destroy the original predictions and cause wrong predictions.

[0004] In view of the shortcomings of the above two types of semi-supervised methods and multi-label classification methods based on inter-class relations, the semi-supervised multi-label image classification method based on inter-class relations with adaptive weights is proposed in the present invention. It considers learning a more adaptive inter-class relationship weight, and adds consistency constraints between the original and inter-class relationship-based classification prediction results based on the inter-class relationship with adaptive weights to help each other learn. It uses the inter-class relationship prediction, the original classification prediction and the weights to define more robust pseudo-labels and consistency constraints for unsupervised data, thereby increasing the generalization of the model and improving the image classification accuracy. Summary of the invention

[0005] Purpose of the invention: In order to solve the problem of semi-supervised learning in multi-label image classification, the present invention proposes a semi-supervised multi-label image classification method based on inter-class relations with adaptive weights, which effectively utilizes supervised and unsupervised samples in the training set, combines inter-class relationship prediction with original prediction, realizes the classification of target images and improves the classification accuracy of the model on the test set.

[0006] Technical solution: The semi-supervised multi-label image classification method based on inter-class relationship with adaptive weights described in the present invention specifically includes the following steps:

[0007] (1) Image preprocessing: Normalize the pre-acquired images and construct a semi-supervised classification dataset including supervised and unsupervised data;

[0008] (2) constructing a weight calculation and prediction fusion model; the model includes an original classifier, a class-related classifier, and a weight calculation module, adopting the original classifier and the class-related classifier in the mainstream multi-label classification model based on the co-occurrence relationship between classes, and the weight calculation module is implemented by a multi-layer perceptron, and each constraint loss function is added to the classifier and the weight calculation module;

[0009] (3) Training the classifier and weight calculation module: Use the supervised data of the preprocessed image data set in step 1 to train the two image classifiers and the weight calculation module in step 2 respectively, and then make the predictions of the two classifiers supervise each other according to the value of the weight calculation module, and perform semi-supervised training on the unsupervised data and the supervised data based on the weight calculation;

[0010] (4) The trained weight calculation and prediction fusion model predicts the image classification result: Use the model trained in step 3 to test the target image to obtain the final classification result.

[0011] Furthermore, the semi-supervised classification dataset in step (1) is:

[0012] In the semi-supervised classification task, given a training set with supervised image samples Unsupervised image samples in is the ith supervised image, is the classification label of the corresponding image, is the i-th unsupervised image without a label, n l is the number of supervised image samples, n u is the number of supervised image samples; given an unlabeled test set image sample in is the i-th image in the test set, n t is the number of test set image samples, and the processing task is to predict the labels of the test set images

[0013] Furthermore, the implementation process of step (2) is as follows:

[0014] (21) A mainstream multi-label image classifier related to inter-class relations and a weight calculation module implemented by a multi-layer perceptron are used. The output of the weight calculation unit is as follows:

[0015] α i (x i )=A(C i (F(x i )), C r (F(x i )), F(x i ))

[0016] Among them, F is the feature extractor, C i is the original classifier, C r is the class-related classifier, A is the weight calculation module, α i (x i ) is the image x iBased on the weight calculation results of the weight calculation module; based on this weight, the final classifier prediction is as follows:

[0017] C(x i )=α i (x i )*C r (F(x i ))+(1-α i (x i ))C i (F(x i ))

[0018] Among them, C(x i ) means that after fusing the two classifiers, the model can i The final classification prediction result; for supervised data The loss between the predictions of the trained classifier and the true values ​​is shown below:

[0019]

[0020] Among them, l ml-sm It is a multi-label soft margin loss function (MultiLabelSooftMarginLoss), where two classifiers and the weight prediction module participate in the training together;

[0021] (22) The loss based on the inter-class relationship weight is:

[0022]

[0023] Among them, H is a Boolean function that determines whether the formula in the brackets is valid, τ is the threshold for calculating the loss, which is usually set to 0.5, and l MSE Represents the mean square error to constrain the original classifier and class-related classifier for sample x i Consistency of forecasts;

[0024] (23) Self-learning pseudo-labels based on inter-class relationship weights and consistency constraints are used to add supervision information to unsupervised data and improve the generalization of the model learned only based on supervised data. The pseudo-label of , the weighted pseudo-label is defined as:

[0025]

[0026] Among them, EMA means taking the weighted sliding average of multiple predictions of the same sample at different times. The pseudo-label calculation selects one of the two predictions according to the tendency of the weight. The formula for pseudo-label loss is:

[0027]

[0028] Among them, τ ps The threshold for participating in the calculation of pseudo-label loss is only involved in the loss calculation when it is above a certain confidence level;

[0029] The consistency loss is made between the predictions of the two classifiers and the fusion weights. The formula is:

[0030]

[0031] in, and For sample Two data augmentation data.

[0032] Furthermore, the implementation process of the multi-label image classifier related to the inter-class relationship described in step (21) is as follows:

[0033] Select a feature extractor that can extract image features and a feature-based original classifier; based on the statistical information of the dataset and class-related features, use a graph convolutional network to construct features and classifiers based on inter-class relationships.

[0034] Furthermore, the weight calculation module based on the multi-layer perceptron described in step (21) is implemented as follows:

[0035] The prediction results of the original classifier, the prediction results of the classifier based on inter-class relations, and the features related to the known classes are selected as inputs, and the number of neurons in the output layer is defined according to the number of categories; the number of neurons in the hidden layer is selected according to the dimension of the input, and the number of neurons in the hidden layer is proportional to the size of the input.

[0036] Furthermore, the implementation process of step (3) is as follows:

[0037] (31) Adjust the number of neural network layers in the weight calculation module, set the initial learning rate, number of training cycles and loss function weight;

[0038] (32) Use the trained feature extractor to train the original classifier and class-related classifier with initial fixed weights, observe the prediction accuracy of the supervised image and the changes of each loss function in the process, and modify the relevant parameters according to the changes;

[0039] (33) After the initial training of the classifier is completed, freeze the parameters of the two trained classifiers, train the weight calculation module, observe the changes in weights and the accuracy of supervised images, and modify relevant parameters according to the changes;

[0040] (34) Alternately debug the classifier and weight calculation module, label the unsupervised data and calculate the consistency loss, and observe the changes in each loss function until the weight calculation and prediction fusion model converge;

[0041] (35) After sufficient training and parameter adjustment, the final trained weight calculation and prediction fusion model is obtained, saved and used for testing.

[0042] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: 1. The weight calculation and prediction fusion module proposed in the present invention combines the original prediction with the prediction based on inter-class relations, reduces the loss of classifier prediction accuracy caused by the long-tail distribution problem of inter-class relations, and enables the prediction results of the two classifiers to help each other in training through adaptive weights; 2. The present invention is based on adaptive inter-class relationship weight calculation to add more robust pseudo-labels and consistency constraints to unsupervised data; 3. The present invention can train the weight calculation and prediction fusion model model within an effective time and predict the classification results of the target image, and can be migrated to a variety of mainstream multi-label classification models based on inter-class relations. It is simple to implement, suitable for actual application environments, and has considerable use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the present invention;

[0044] Figure 2 A schematic diagram of the structure of the weight calculation and prediction fusion model constructed by the present invention;

[0045] Figure 3 This is a flow chart of the steps of training the overall model and using the model to predict image classification results in the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, the present invention proposes a semi-supervised multi-label image classification method based on inter-class relations with adaptive weights, constructs a weight calculation and prediction fusion model, combines the original prediction with the prediction based on inter-class relations, reduces the prediction accuracy loss caused by the long-tail distribution problem of inter-class relations, and uses adaptive weights to enable the two prediction results to help each other train, adding more robust pseudo-labels and consistency constraints to unsupervised data at multiple levels of features, predictions, and inter-class relations, thereby achieving efficient unsupervised data utilization and model training. Specifically, the following steps are included:

[0048] Step 1: Image preprocessing: Normalize the pre-acquired images and construct a semi-supervised classification dataset including supervised and unsupervised data.

[0049] Given a training set with supervised image samples Unsupervised image samples in is the ith supervised image, is the classification label of the corresponding image, is the i-th unsupervised image without a label, n l is the number of supervised image samples, n u is the number of supervised image samples; given an unlabeled test set image sample in is the i-th image in the test set, n t is the number of test set image samples, and the processing task is to predict the labels of the test set images

[0050] Step 2: Construct a weight calculation and prediction fusion model; the model includes an original classifier, a class-related classifier and a weight calculation module. The original classifier and class-related classifier in the mainstream multi-label classification model based on inter-class co-occurrence relationship are adopted. The weight calculation module is implemented with a multi-layer perceptron, and various constraint loss functions are added to the classifier and weight calculation module.

[0051] like Figure 2 The weight calculation and prediction fusion model structure is shown as follows:

[0052] To predict the classification labels of the test set images, a mainstream multi-label image classifier with inter-class relationship correlation and a weight calculation module implemented by a multi-layer perceptron are used. The output of the weight calculation unit is as follows:

[0053] α i (x i )=A(C i (F(x i )), C r (F(x i )), F(x i ))

[0054] Among them, F is the feature extractor, C i is the original classifier, C r is the class-related classifier, A is the weight calculation module, α i (x i ) is the image x i Based on the weight calculation results of the weight calculation module; based on this weight, the final classifier prediction is as follows:

[0055] C(x i )=α i (x i )*C r (F(x i ))+(1-α i (x i ))C i (F(x i ))

[0056] Among them, C(x i ) indicates that after fusing two classifiers, the weight calculation and prediction fusion model is used for sample x i The final classification prediction result; for supervised data The loss between the predictions of the trained classifier and the true values ​​is shown below:

[0057]

[0058] Among them, l ml-sm It is a multi-label soft margin loss function (MultiLabelSoftMarginLoss), in which two classifiers and weight prediction modules participate in the training together.

[0059] In order to make the results of the original classifier and the class-related classifier influence each other and provide supervision information for unsupervised data without true values, a loss based on the weight of the relationship between classes is designed here. The formula is as follows:

[0060]

[0061] Where n is the total number of samples in the training set including supervised and unsupervised data, x i is the i-th sample, H is a Boolean function that determines whether the formula in the brackets is valid, τ is the threshold for calculating the loss, which is usually set to 0.5, and l MSE Represents the mean square error to constrain the original classifier and class-related classifier for sample x i Prediction consistency. This loss is used to optimize the prediction of the original classifier with the prediction of class-related relationships. It provides constraint information for both supervised and unsupervised samples to improve the accuracy of the classifier. It is only used to optimize the original classifier and does not participate in the gradient update of the class-related classifier.

[0062] In order to utilize unsupervised data without artificial labels to reduce the classification error rate, self-learning pseudo-labels based on inter-class relationship weights and consistency constraints are used to add supervision information to unsupervised data and improve the generalization of the model learned only based on supervised data. The pseudo-label of , the weight-based pseudo-label is defined as follows:

[0063]

[0064] Among them, EMA means taking the weighted moving average of multiple predictions of the same sample at different times, and the pseudo-label calculation selects one of the two predictions according to the tendency of the weight. The formula for defining the pseudo-label loss is as follows:

[0065]

[0066] τ ps To participate in the calculation of the threshold of pseudo-label loss, it participates in the loss calculation only when it is higher than a certain confidence level (confident that a certain category exists or convinced that a certain category does not exist); at the same time, in order to make more use of the information of unsupervised data, a consistency loss is made between the predictions and fusion weights of the two classifiers. The formula is as follows:

[0067]

[0068] in, and For sample Two data that have undergone different data enhancement operations.

[0069] Step 3: Training classifiers and weight calculation modules: Use the supervised data of the preprocessed image dataset in step 1 to train the two image classifiers and weight calculation modules in step 2 respectively. Then, the predictions of the two classifiers are supervised by each other according to the value of the weight calculation module, and semi-supervised training is performed on unsupervised data and supervised data based on the weight calculation. Figure 3 As shown:

[0070] 1) Adjust the number of neural network layers in the weight calculation module, set the initial learning rate, number of training cycles and loss function weight.

[0071] 2) Use the trained feature extractor to train the original classifier and class-related classifier with initial fixed weights, observe the prediction accuracy of supervised images and the changes of various loss functions in the process, and modify the relevant parameters according to the changes.

[0072] 3) After the initial training of the classifier is completed, freeze the parameters of the two trained classifiers, train the weight calculation module, observe the changes in weights and the accuracy of supervised images, and modify the relevant parameters according to the changes;

[0073] 4) Alternately debug the classifier and weight calculation modules, pseudo-label the unsupervised data and calculate the consistency loss, and observe the changes in each loss function until the weight calculation and prediction fusion model converge.

[0074] 5) After sufficient training and parameter adjustment, the final trained weight calculation and prediction fusion model is obtained, saved and used for testing.

[0075] Step 4: Calculate the trained weights and predict the fusion model to predict the image classification results: Use the model trained in step 3 to test the target image to be classified and obtain the final classification result.

[0076] The quality of the image has a direct impact on the implementation of the algorithm and the classification effect. Normalizing the image is a way to simplify the calculation and is of great significance to improving the classification accuracy. Given an image sample X, according to the formula img = X-mean / std, where mean and std are 0.5, the image pixel values ​​are normalized to numbers in [-1, 1].

[0077] The above is a detailed introduction to the semi-supervised multi-label image classification method based on inter-class relations with adaptive weights provided by the present invention. It is worth noting that there are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention, which is only used to help understand the method and core idea of ​​the present invention; at the same time, for those skilled in the art, modifications and adjustments made on the basis of the core idea of ​​the present invention will be regarded as the scope of protection of the present invention. In summary, the content of this specification should not be understood as a limitation on the present invention, and the scope of protection of the present invention should be defined by the attached claims.

Claims

1. A semi-supervised multi-label image classification method based on inter-class relations with adaptive weights, characterized in that: The following steps are involved: (1) Image preprocessing: Normalize the pre-acquired images and construct a semi-supervised classification dataset including supervised and unsupervised data; (2) constructing a weight calculation and prediction fusion model; the model includes an original classifier, a class-related classifier, and a weight calculation module, adopting the original classifier and the class-related classifier in the mainstream multi-label classification model based on the co-occurrence relationship between classes, and the weight calculation module is implemented by a multi-layer perceptron, and each constraint loss function is added to the classifier and the weight calculation module; (3) Training the classifier and weight calculation module: Use the supervised data of the preprocessed image data set in step 1 to train the two image classifiers and the weight calculation module in step 2 respectively, and then make the predictions of the two classifiers supervise each other according to the value of the weight calculation module, and perform semi-supervised training on the unsupervised data and the supervised data based on the weight calculation; (4) The trained weight calculation and prediction fusion model predicts the image classification result: Use the model trained in step 3 to test the target image to be classified and obtain the final classification result; The semi-supervised classification dataset in step (1) is: In the semi-supervised classification task, given a training set with supervised image samples Unsupervised image samples in is the ith supervised image, is the classification label of the corresponding image, is the i-th unsupervised image without a label, n l is the number of supervised image samples, n u is the number of supervised image samples; Given an unlabeled test set image sample in is the i-th image in the test set, n t is the number of test set image samples, and the processing task is to predict the labels of the test set images The implementation process of step (2) is as follows: (21) A mainstream multi-label image classifier related to inter-class relations and a weight calculation module implemented by a multi-layer perceptron are used. The output of the weight calculation unit is as follows: α i (xi)=A(C i (F(xi)),C r (F(xi)),F(xi)) Among them, F is the feature extractor, C i is the original classifier, C r is the class-related classifier, A is the weight calculation module, α i (x i ) is the image x i Based on the weight calculation results of the weight calculation module; based on this weight, the final classifier prediction is as follows: C(x i )=α i (x i )*C r (F(x i ))+(1-α i (x i ))C i (F(x i ) Among them, C(x i ) means that after fusing the two classifiers, the model can i The final classification prediction result; for supervised data The loss between the predictions of the trained classifier and the true values ​​is shown below: Among them, l ml-sm It is a multi-label soft margin loss function (MultiLabelSoftMarginLoss), where two classifiers and the weight prediction module participate in the training together; (22) The loss based on the inter-class relationship weight is: Among them, H is a Boolean function that determines whether the formula in the brackets is valid, τ is the threshold for calculating the loss, which is usually set to 0.5, and l MSE Represents the mean square error to constrain the original classifier and class-related classifier for sample x i Consistency of forecasts; (23) Self-learning pseudo-labels based on inter-class relationship weights and consistency constraints are used to add supervision information to unsupervised data and improve the generalization of the model learned only based on supervised data. The pseudo-label of , the weighted pseudo-label is defined as: Among them, EMA means taking the weighted sliding average of multiple predictions of the same sample at different times. The pseudo-label calculation selects one of the two predictions according to the tendency of the weight. The formula for pseudo-label loss is: Among them, τ ps The threshold for participating in the calculation of pseudo-label loss is only involved in the loss calculation when it is above a certain confidence level; The consistency loss is made between the predictions of the two classifiers and the fusion weights. The formula is: in, and For sample Two data augmentation data; The implementation process of step (3) is as follows: (31) Adjust the number of neural network layers in the weight calculation module, set the initial learning rate, number of training cycles and loss function weight; (32) Use the trained feature extractor to train the original classifier and class-related classifier with initial fixed weights, observe the prediction accuracy of the supervised image and the changes of each loss function in the process, and modify the relevant parameters according to the changes; (33) After the initial training of the classifier is completed, freeze the parameters of the two trained classifiers, train the weight calculation module, observe the changes in weights and the accuracy of supervised images, and modify relevant parameters according to the changes; (34) Alternately debug the classifier and weight calculation module, label the unsupervised data and calculate the consistency loss, and observe the changes in each loss function until the weight calculation and prediction fusion model converge; (35) After sufficient training and parameter adjustment, the final trained weight calculation and prediction fusion model is obtained, saved and used for testing.

2. The method for semi-supervised multi-label image classification based on inter-class relations with adaptive weights according to claim 1, characterized in that: The implementation process of the multi-label image classifier related to the inter-class relationship described in step (21) is as follows: Select a feature extractor that can extract image features and a feature-based original classifier; based on the statistical information of the dataset and class-related features, use a graph convolutional network to construct features and classifiers based on inter-class relationships.

3. The method for semi-supervised multi-label image classification based on inter-class relations with adaptive weights according to claim 1, characterized in that: The implementation process of the weight calculation module based on the multi-layer perceptron described in step (21) is as follows: The prediction results of the original classifier, the prediction results of the classifier based on inter-class relations, and the features related to the known classes are selected as inputs, and the number of neurons in the output layer is defined according to the number of categories; the number of neurons in the hidden layer is selected according to the dimension of the input, and the number of neurons in the hidden layer is proportional to the size of the input.

Images