A semi-supervised multi-label image classification method based on reliable pseudo labels

Abstract

The application relates to the technical field of computer vision and image processing, and provides a semi-supervised multi-label image classification method based on reliable pseudo labels, steps of which comprise the following: an initial model is constructed, and high-value samples are actively screened for manual labeling based on label correlation and prediction uncertainty. A K-NN sample graph is constructed, information propagation is carried out by using a graph convolution network to enhance prediction. A dynamic threshold is independently determined for each category, and a tolerance mechanism is combined to generate positive and negative pseudo labels with high confidence, so that the interference of ambiguous samples is avoided. Finally, model parameters are iteratively updated by combining a supervision loss and a pseudo label loss, and the graph propagation and the pseudo label screening are jointly optimized to continuously improve the multi-label classification accuracy. Through label correlation uncertainty evaluation, the application can preferentially select samples with high uncertainty in related categories for manual labeling, iteratively and jointly optimizes adaptive graph propagation and high-confidence pseudo label screening, and then improves the prediction result.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and more specifically, to a semi-supervised multi-label image classification method based on reliable pseudo-labels. Background Technology

[0002] In multi-label image classification tasks, it is necessary to simultaneously identify multiple object or scene categories present in an image. To reduce the reliance on large-scale manual annotation, semi-supervised learning methods aim to improve model performance by utilizing a small number of labeled samples and a large number of unlabeled samples. Among existing technologies, the three closest implementation schemes to this invention are as follows: (1) Generating pseudo-labels based on a fixed threshold: This method trains an initial classification model using labeled samples, and then uses the model to predict unlabeled samples to obtain the predicted probabilities of each category. When the predicted probability of a certain category is higher than the globally preset threshold, that category is assigned as a pseudo-label to the unlabeled sample, and it is added to the next round of training.

[0003] (2). Sample selection based on active learning: This method evaluates the information of unlabeled samples through the model, using common metrics such as prediction entropy, confidence, or uncertainty. The system prioritizes the samples with the least uncertainty in the model, which are then manually labeled, and the newly labeled samples are added to the training set.

[0004] (3) Semi-supervised multi-label classification based on graph structure: This method treats each image sample as a node in a graph, establishes edge connections based on the feature similarity between samples, and constructs a graph network. Through graph convolution and other propagation algorithms, the label information of labeled nodes is propagated along the edges to adjacent unlabeled nodes, thereby generating predictions for unlabeled nodes.

[0005] However, all of the aforementioned existing technologies have significant drawbacks. First, Scheme One uses the same fixed threshold for all categories, ignoring the varying learning difficulty and prediction distribution differences among different categories in multi-label tasks. This leads to the loss of many reliable pseudo-labels for easily learned categories, while difficult-to-learn categories may be incorrectly labeled, impacting model accuracy. Scheme Two, the active learning method, relies on uncertain sample selection, typically calculating a single score for a single category or the entire image. It fails to adequately consider the co-occurrence relationships and semantic relevance between multi-label categories, resulting in unrepresentative selected samples and low efficiency in improving the overall multi-label performance of the model. Finally, Scheme Three's graph propagation depends on the quality of the initial pseudo-labels. Once an initial prediction error occurs, it propagates rapidly and amplifies through the edges of the graph network, severely affecting the stability of model training and the final classification result.

[0006] In summary, a semi-supervised multi-label image classification method needs to be proposed to ensure training stability and continuously improve the classification accuracy of multi-label images under limited manual annotation conditions. Summary of the Invention

[0007] In view of this, the present invention proposes a semi-supervised multi-label image classification method based on reliable pseudo-labels. This method can fully consider the learning difficulty and prediction distribution differences of different categories in multi-label tasks. In the active learning sample selection, it integrates the co-occurrence relationship and semantic relevance between categories and selects representative samples for labeling. It also adopts a category-specific dynamic threshold and tolerance mechanism to screen high-confidence pseudo-labels. At the same time, it suppresses the propagation and accumulation of errors along the graph structure through iterative joint optimization of adaptive graph propagation and pseudo-label screening.

[0008] To achieve the above objectives, the present invention proposes the following technical solution: A semi-supervised multi-label image classification method based on reliable pseudo-labels includes: Obtain a multi-label image dataset and divide it into a labeled sample set and an unlabeled sample set; Construct an initial neural network model, and train the initial neural network model using a labeled sample set to obtain the initial model; The label correlation matrix between categories is calculated based on the labeled sample set; the unlabeled samples are input into the initial model to obtain the predicted probability of each unlabeled sample in each category, and the prediction uncertainty of each category is calculated; the correlation uncertainty score of each unlabeled sample is calculated by combining the label correlation and the prediction uncertainty; the unlabeled samples are sorted according to the correlation uncertainty score, the sample with the highest score is selected for manual labeling, and the newly labeled sample is added to the labeled sample set; Extract all sample features, construct a K-NN sample graph based on feature similarity, apply a graph convolutional network to the K-NN sample graph for information propagation, update the representation of each node, and obtain a prediction matrix enhanced by graph structure. For each category, the distribution of predicted scores for that category is analyzed, a dynamic threshold specific to that category is determined through a search algorithm, and pseudo-label generation rules are formulated based on the dynamic threshold and tolerance range; all predictions that meet the conditions are collected to construct a pseudo-label set. Construct a total loss function, update the model using the backpropagation algorithm, recalculate the features and prediction matrix using the updated model, and repeat the graph propagation, pseudo-label generation and model optimization process until the model converges or reaches the preset number of iterations.

[0009] Furthermore, the initial neural network model includes a feature extractor and a classifier. The feature extractor is used to extract deep features of the input image, and the classifier is used to output the predicted probability of the image in each category. The initial training process uses images of samples from the labeled sample set as input and the corresponding manually labeled ground truth as supervision signals. The initial model parameters are obtained by minimizing the multi-label classification loss.

[0010] Furthermore, the prediction uncertainty is calculated using the following formula: U jc =1-max( jc ,1- jc ) in, jc For the initial model on unlabeled samples x j The predicted probability of belonging to category c; the closer the predicted probability is to 0.5, the higher the uncertainty; the closer the predicted probability is to 0 or 1, the lower the uncertainty.

[0011] Furthermore, the relevant uncertainty score is calculated using the following formula: CU j = [ U jc + l | r cr | | U jc - U jr | ] in, U jc For the sample x j Prediction uncertainty in category c, r cr Let λ represent the correlation between category c and category r, λ be the weighting coefficient of label correlation, and C be the total number of categories.

[0012] Furthermore, the K-NN sample graph is constructed as follows: the K-nearest neighbor algorithm is used to extract the feature vector of each sample, calculate the feature similarity between samples, and use nodes to represent samples and edges to represent the similarity relationship between samples.

[0013] Furthermore, the pseudo-tag generation rules include: Based on dynamic threshold t * cand the preset tolerance range d If the predicted score is higher than t * c +δ, indicating a positive pseudo-label of 1, if the predicted score is lower than... t * c -d If the predicted score falls within the fuzzy range, the prediction for that category of the sample is ignored and no pseudo-label is generated.

[0014] Furthermore, the total loss function is shown in the following equation: L = L sup + αL pseudo in, L sup For supervised loss of labeled samples, L pseudo For pseudo-label loss of unlabeled samples, α The hyperparameters for balancing the weights.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention, through label-related uncertainty assessment, prioritizes samples with high uncertainty in relevant categories for manual annotation. These samples can more efficiently help the model interpret easily confused category boundaries, achieving higher model performance improvements with the same annotation budget compared to traditional single-metric methods.

[0016] This invention employs a dynamic threshold and tolerance mechanism for each category, tailoring screening criteria to each category. This effectively avoids wasting easily classified pseudo-labels and introducing noise from difficult-to-classify categories. The tolerance mechanism eliminates interference from ambiguous samples, ensuring that all pseudo-labels used for training have high confidence, thereby stabilizing the training process and preventing the model from being misled by incorrect labels.

[0017] This invention iteratively and jointly optimizes adaptive graph propagation and high-confidence pseudo-label selection. This improves prediction results, while the strict pseudo-label selection mechanism filters out errors that may occur during propagation, interrupting the accumulation of errors. This allows the model to continuously and stably learn from unlabeled data, ultimately significantly improving classification accuracy. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the semi-supervised multi-label image classification method based on reliable pseudo-labels of the present invention; Figure 2 This is a diagram illustrating the overall workflow of the semi-supervised multi-label image classification method based on reliable pseudo-labels of this invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] This embodiment proposes a semi-supervised multi-label image classification method based on reliable pseudo-labels, such as... Figure 1 As shown, it includes the following steps: 1. Data preparation and initial model building Obtain a multi-label image dataset and divide it into a labeled sample set containing manually annotated data. D l and unlabeled sample set D u Construct an initial neural network model that includes a feature extractor and a classifier. Utilize a labeled sample set. D l The model is initially trained to obtain initial model parameters.

[0021] 2. Active learning sample selection for relevant uncertainties Step 2 aims to obtain data from the unlabeled sample set. D u Samples that significantly improved model performance were selected and manually labeled. This was based on a labeled sample set. D l The correlation between categories was calculated using the Pearson correlation coefficient to obtain the label correlation matrix. r Take the values ​​corresponding to category c and category r in the label matrix respectively, calculate the change of the label value with respect to their respective average values, and normalize them to obtain the result. r cr Elements in the matrix r crThis represents the correlation between category c and category r. A larger absolute value indicates a stronger correlation between the two categories, and vice versa. Subsequently, unlabeled samples are sequentially input into the initial model trained in step 1 to obtain the predicted probability of each unlabeled sample for each category. Let the initial model predict the probability of unlabeled samples for each category. x j The predicted probability of belonging to category c is The prediction uncertainty of this sample in category c is... U jc Calculated according to the following formula: U jc =1-max( jc ,1- jc The closer the predicted probability is to 0.5, the more difficult it is for the model to determine whether the category exists, and the higher the corresponding uncertainty. The closer the predicted probability is to 0 or 1, the clearer the model's judgment, and the lower the corresponding uncertainty. After obtaining the predicted uncertainty for each category, the overall uncertainty of the same sample across all categories is considered simultaneously with the label correlation to calculate the correlation uncertainty score for the unlabeled sample: CU j = [ U jc + l | r cr | | U jc - U jr | ] The first term of the formula represents the overall predictive uncertainty of the sample across all categories, while the second term measures the difference in uncertainty between related categories. l The weights represent the label relevance. Unlabeled samples are ranked according to their relevance uncertainty scores, and the top-Q samples with the highest scores are manually labeled. The newly labeled samples are then moved into the labeled sample set. D l At the same time, it is removed from the unlabeled set.

[0022] 3. Image-based communication Features are extracted from all samples. Based on feature similarity, a K-NN sample graph is constructed using the K-Nearest Neighbors algorithm, where nodes represent samples and edges represent similarity relationships between samples. A two-layer graph convolutional network is applied to the constructed sample graph for information propagation. By aggregating information from neighboring nodes, the representation of each node is updated, thereby obtaining a prediction matrix enhanced by the graph structure. ZThis step effectively utilizes information from labeled samples to correct predictions for unlabeled samples.

[0023] 4. Dynamic pseudo-tag optimization Step 4 determines the pseudo-label threshold based on the predicted probability distribution of different categories. A threshold set consisting of multiple values ​​is then established. T * =[ t * 1 , t * 2 ...... t * c For example, candidate thresholds can be set at intervals of 0.05 from 0.50 to 1.00, and a dynamic threshold specific to each category can be determined using a search algorithm. t * c This avoids the class imbalance problem caused by using a globally uniform threshold. After obtaining the optimal dynamic threshold for each class, a tolerance δ is set on both sides of the threshold. A pseudo-label generation rule is defined: if the predicted score is higher than... t * c +δ, indicating a positive pseudo-label of 1, if the predicted score is lower than... t * c - δ is used to determine if a sample is a negative class and the pseudo-label is 0. If the predicted score falls within the fuzzy interval, the prediction for that class is ignored, and no pseudo-label is generated. All high-confidence predictions that meet the conditions form a reliable pseudo-label set P. After each round of model parameter updates, the predicted probabilities of the threshold calibration set and unlabeled samples are recalculated using the updated model. The dynamic thresholds for each class are searched again, and a new reliable pseudo-label set P is generated.

[0024] 5. Joint Optimization and Model Update Construct the total loss function L = L sup + αL pseudo It consists of two parts. The supervised loss for labeled samples. L sup Pseudo-label loss for unlabeled samples L pseudo ,in αTo balance the hyperparameters of the weights, the parameters of the feature extractor, classifier, and graph convolutional network are updated using the backpropagation algorithm. After the model is updated, the next iteration begins. The process returns to the graph propagation step, recalculating the features and prediction matrix using the updated model, and repeating the graph propagation, pseudo-label generation, and model optimization processes until the model converges or reaches the preset number of iterations.

[0025] like Figure 2 As shown, the entire workflow of this embodiment can be divided into five modules. The first module is the initial dataset preparation. The second module is active learning sample selection, which is responsible for selecting the most informative unlabeled samples for manual annotation. The third module utilizes the structural information between samples to optimize the feature representation and prediction results of the unlabeled samples. The fourth module is responsible for generating high-quality pseudo-labels for the unlabeled samples for subsequent semi-supervised training. The fifth module is responsible for integrating all information for the final model training and parameter iteration.

[0026] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A semi-supervised multi-label image classification method based on reliable pseudo-labels, characterized in that, include: Obtain a multi-label image dataset and divide it into a labeled sample set and an unlabeled sample set; Construct an initial neural network model, and train the initial neural network model using a labeled sample set to obtain the initial model; The label correlation matrix between categories is calculated based on the labeled sample set; the unlabeled samples are input into the initial model to obtain the predicted probability of each unlabeled sample in each category, and the prediction uncertainty of each category is calculated; the correlation uncertainty score of each unlabeled sample is calculated by combining the label correlation and the prediction uncertainty; the unlabeled samples are sorted according to the correlation uncertainty score, the sample with the highest score is selected for manual labeling, and the newly labeled sample is added to the labeled sample set; Extract all sample features, construct a K-NN sample graph based on feature similarity, apply a graph convolutional network to the K-NN sample graph for information propagation, update the representation of each node, and obtain a prediction matrix enhanced by graph structure. For each category, the distribution of predicted scores for that category is analyzed, a dynamic threshold specific to that category is determined through a search algorithm, and pseudo-label generation rules are formulated based on the dynamic threshold and tolerance range; all predictions that meet the conditions are collected to construct a pseudo-label set. Construct a total loss function, update the model using the backpropagation algorithm, recalculate the features and prediction matrix using the updated model, and repeat the graph propagation, pseudo-label generation and model optimization process until the model converges or reaches the preset number of iterations.

2. The method according to claim 1, characterized in that, The initial neural network model includes a feature extractor and a classifier. The feature extractor is used to extract deep features of the input image, and the classifier is used to output the predicted probability of the image in each category. The initial training process uses images of samples from the labeled sample set as input and the corresponding manually labeled ground truth values ​​as supervision signals. The initial model parameters are obtained by minimizing the multi-label classification loss.

3. The method according to claim 1, characterized in that, The prediction uncertainty is calculated using the following formula: U jc =1-max( jc ,1- jc ), in, jc For the initial model on unlabeled samples x j The predicted probability of belonging to category c; the closer the predicted probability is to 0.5, the higher the uncertainty; the closer the predicted probability is to 0 or 1, the lower the uncertainty.

4. The method according to claim 1, characterized in that, The relevant uncertainty score is calculated using the following formula: CU j = [ U jc + λ | ρ cr | | U jc - U jr | ]， in, U jc For the sample x j Prediction uncertainty in category c, ρ cr Let λ represent the correlation between category c and category r, λ be the weighting coefficient of label correlation, and C be the total number of categories.

5. The method according to claim 1, characterized in that, The K-NN sample graph is constructed as follows: the K-nearest neighbor algorithm is used to extract the feature vector of each sample, calculate the feature similarity between pairs of samples, and use nodes to represent samples and edges to represent the similarity relationship between samples.

6. The method according to claim 1, characterized in that, The pseudo-tag generation rules include: Based on dynamic threshold t * c and the preset tolerance range δ If the predicted score is higher than t * c +δ, indicating a positive pseudo-label of 1, if the predicted score is lower than... t * c -δ If the predicted score falls within the fuzzy range, the prediction for that category of the sample is ignored and no pseudo-label is generated.

7. The method according to claim 1, characterized in that, The total loss function is shown in the following formula: L = L sup + αL pseudo， in, L sup For supervised loss of labeled samples, L pseudo For pseudo-label loss of unlabeled samples, α The hyperparameters for balancing the weights.

Images