Multi-label image classification method based on anti-bias confidence weighted loss
By constructing a class symbiotic relationship graph and a dynamic graph convolutional network, and combining it with an anti-bias confidence weighted loss function, the model bias problem caused by missing class labels in multi-label image classification is solved, achieving efficient multi-label image classification and reducing annotation costs.
Patent Information
- Application Number
- CN202611393385.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-09
- Publication Date
- 2026-11-10
AI Technical Summary
Existing multi-label image classification methods are prone to misleading the model when faced with data containing missing class labels, leading to a decrease in recall and overall performance. Furthermore, the cost of labeling training data is high, and existing methods struggle to dynamically adapt to labeling biases and capture high-order semantic relationships between labels in images.
We adopt a multi-label image classification method based on anti-biased confidence weighted loss. By constructing a class co-occurrence graph and a dynamic graph convolutional network, combined with the anti-biased confidence weighted loss function, we dynamically adjust the supervision strategy for missing labels. We also use a multilayer perceptron and graph convolutional network to mine the deep interaction structure between image regions and labels, thereby reducing the labeling cost.
It effectively alleviates model bias caused by missing labels, significantly reduces the labeling cost of training data, and improves the recall and overall performance of multi-label image classification.
Smart Images

Figure CN123071735A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision multi-label image classification technology, specifically involving a multi-label image classification method based on anti-bias confidence weighted loss. Background Technology
[0002] Multi-label image classification refers to assigning multiple class labels to each image from a predefined set of class labels. Multi-label image classification has wide-ranging applications, including multi-object recognition and detection, album classification, image semantic annotation, and image retrieval. Supervised deep learning methods based on big data have achieved state-of-the-art performance in computer vision applications such as multi-label image classification. However, constructing large-scale labeled training sample sets for multi-label images is very time-consuming, expensive, and often requires specialized knowledge. In real-world scenarios, the multi-label image annotations used for model training are often incomplete; this type of data is called missing class label data.
[0003] Missing class label data refers to data where the class labeling is incomplete, with missing component values in the class label vector. Specifically, when labeling sample class labels, only the class labels that the sample may contain are evaluated and labeled (the state labels for contained and uncontained class labels are 1 and 0, respectively), while the remaining potentially contained class labels have no corresponding state labels. If the state labels at certain component positions in a sample's class label vector are missing (neither 0 nor 1), then this sample is called a missing class label sample, and the class label corresponding to the missing state label position is the missing class label of this sample.
[0004] To address the aforementioned label missing problem, most existing mainstream multi-label classification methods are designed based on the assumption of complete labeling. Their loss functions can mislead the model, severely weakening the classifier's recall and overall performance. To mitigate this issue, researchers have attempted to introduce strategies such as confidence weighting, lesson learning, or pseudo-label correction. However, these methods typically rely on static thresholds or independent estimation of the missing probability of each label, making it difficult to capture high-order semantic relationships between labels in the image and unable to dynamically adapt to the labeling bias distribution of different samples. Meanwhile, although graph neural networks are used to model label co-occurrence relationships, conventional graph convolutional structures are mostly based on fixed statistical priors, lacking the ability to adaptively adjust to changes in feature representations during training. Furthermore, their message passing mechanism is prone to amplifying error propagation when faced with noisy labels. Therefore, there is an urgent need to design a robust learning framework that can effectively mine the deep interaction structure between image regions and labels while resisting the influence of spurious negative bias, in order to fully exploit the potential of multi-label classification models under real-world conditions of incomplete labeling. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-label image classification method based on anti-bias confidence weighted loss. This method can effectively alleviate model bias caused by missing labels and significantly reduce the labeling cost of training data.
[0006] The technical solution adopted in this invention is a multi-label image classification method based on bias-resistance confidence weighted loss, specifically: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0007] The invention is further characterized in that: The specific implementation method of step 1 is as follows: Assume the training sample set of multi-labeled images containing missing class labels is , It is the set of all class labels in the training sample set. Represents the first in the training sample set There are category labels, among which , This represents the total number of class labels in the training sample set. This represents the total number of training samples. Represents the first in the training sample set Image samples, Represents training sample images The category label vector, express The Each component, if ,express contain ,like ,express Does not contain ,like , indicating uncertainty Does it contain The corresponding tag at this time That is, the sample image Missing class labels.
[0008] The specific implementation method of step 2 is as follows: Construct a completely undirected graph with the number of nodes. The nodes are respectively That is, one node corresponds to one category label; Based on the training sample set Constructing the observation label matrix ,make The Line number The elements of the column are ,Right now From the training sample set Observation label matrix Extract all positive label positions, i.e., label values. For positions where 1 is used, calculate the category label. The conditional probability relationship between them is as follows: First, let the binarized label matrix be... , The Line number The elements of the column are If and only if hour Calculate the co-occurrence matrix ,in for The transpose of the given value yields the result. , It is the set of real numbers; Then, calculate the conditional probability matrix. , The Line number Column elements ,in Co-occurrence matrix No. Line number The elements of the column, i.e. Indicates the first training sample set The first tag and the first The number of times each tag appears together , For the first The total frequency of occurrence of each label, where ε is a small constant to prevent the denominator from being zero. Indicates when the first When the first tag appears The probability of each label appearing, and then the conditional probability matrix. After symmetry processing, the corresponding expression is: Then, for the symmetric conditional probability matrix Each row in the graph is normalized using the L1 norm so that the sum of the weights in each row is 1, resulting in the adjacency matrix of the symbiotic relationship graph. .
[0009] In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
[0010] The specific implementation method for step 3 is as follows: Step 3.1: Build the backbone network: Select a deep convolutional neural network. The last layer of the deep convolutional neural network is the basic classification layer. Set the number of neurons in the basic classification layer to [value missing]. The activation function of the basic classification layer is the Sigmoid activation function. The penultimate layer of the deep convolutional neural network is regarded as the feature layer. Several convolutional layers precede the feature layer. The output of the feature layer is called the global feature. The output of the basic classification layer without the Sigmoid activation function is called the preliminary prediction. After the input multi-label image extracts the global features and obtains the preliminary prediction through the backbone network, the obtained global features and the preliminary prediction are concatenated. Then, the concatenated result is sent to the dynamic graph convolutional network module for refined prediction based on the class symbiotic relationship graph. Step 3.2: Constructing a Dynamic Graph Convolutional Network: The dynamic graph convolutional network consists of three parts: a fusion graph generation module, a graph convolutional network, and a classification layer; the dynamic graph convolutional network module is responsible for label-based classification. The symbiotic relationship diagram between them refines the preliminary prediction; Step 3.2 specifically involves: Step 3.2.1: Constructing the Fusion Graph Generation Module: The fusion graph generation module includes two parallel branches: a multilayer perceptron dynamic graph generation branch and a fusion gating branch; assuming the global features of the image obtained after processing by the backbone network are... Preliminary forecast is ,in For the set of real numbers, For batch processing numbers, To determine the dimension of the global features, first... and By concatenating the vectors, we obtain the joint vector. Then the joint vector The input is fed into a multilayer perceptron (MLP), which is used to map the joint vector dimension to... Do it later Dimensional reshaping and row-wise normalization using Softmax ensures that the sum of each row is 1, yielding the adjacency matrix of the dynamic graph. The structure of the dynamic graph is the same as that of the symbiotic relationship graph, the difference being the adjacency matrix; joint vector It will also be sent into a structure as The fusion gate branch is used to output the sample-by-sample fusion weights. Finally, for the class symbiotic relationship graph constructed in step 2. Add batch dimension And broadcast along that dimension To make it consistent with the dynamic image According to weight The fusion process is performed to obtain the adjacency matrix of the fused graph. , , Depend on indivual matrix Composition, in which , Indicates the first The adjacency matrix of the fusion graph corresponding to each training sample; the fusion graph is then input into a graph convolutional network; Step 3.2.2: Constructing a Graph Convolutional Network: A three-layer graph convolutional network is adopted. The structure of each layer of the graph convolutional network is the same as that of the fused graph, and the adjacency matrix is also the same. The difference lies in the node values of each layer of the graph convolutional network. The input of the first layer of the graph convolutional network is the initial node features. After the graph convolution operation of the first layer, the output of the first layer is the input of the second layer, and the output of the second layer is the input of the third layer. After the three layers of graph convolution operations, the value of each node in the output graph is a vector. Specifically: Assume there is an identity matrix The number of training samples per batch is ,for Add batch dimension And broadcast along that dimension And for the expanded The last dimension Perform a linear projection so that it passes through the linear projection layer from Dimensional encoding mapping to In the semantic space, the initial node features are obtained. , For a custom semantic space dimension, Depend on indivual matrix composition, Indicates the first The initial node features corresponding to each training sample ; The following symbols are defined: For graph convolutional networks The output of the layer, For graph convolutional networks Linear transformation of the layer, Here, is a non-linear activation function, where ; The output of the first layer of the convolutional network is... , , , For residual projection layer; The output of the second layer of the graph convolutional network is , , ; The output of the third layer of the graph convolutional network is , , .
[0011] Step 3.2.3: Construct the classification layer: The classification layer is a single-layer fully connected layer, and the number of neurons in the classification layer is set to... Its input is the output of the third layer of the graph convolutional network. The activation function of the neurons in the classification layer is the Sigmoid function. The output of the classification layer is a probability vector of a multi-label image belonging to different class labels. The output of the classification layer without the Sigmoid activation function is denoted as... ,Will This is called refined forecasting.
[0012] The specific implementation method for step 4 is as follows: Step 4.1: Construct the label loss function: For labels that are explicitly labeled as positive, ... The value is 1 or negative. For label locations with a value of 0, asymmetric focusing loss is used for supervision, specifically: First, assume the observation label matrix of the current batch of sample data. ,make The Line number The elements of the column are ,Right now , Indicates the first The training sample at the th ... The label value for each category label position, let ,in , , For manually specified smoothing parameters; Indicates to The smoothing value is used to adjust the size of the label loss function below; Then, let the prediction probability matrix be... , This is the matrix formed by the predicted vectors of the current batch of samples. For the Sigmoid function, a marker mask is used. The marked position is 1, otherwise it is 0; For the For each training sample, the asymmetric focusing loss is... ,in, Indicates when Time The training sample at the th ... Loss for each category label position, Indicates when Time The training sample at the th ... Loss for each category label position, , , It is the numerical stability constant. Represents the prediction probability matrix The Line number The elements of the column; the asymmetric focusing loss over all training samples in the current batch. The label loss function is obtained by summing and averaging the values. for: .
[0013] Step 4.2: Construct the unlabeled loss function, specifically by defining the set of unlabeled label indices as follows: ,in Indicates the first The training sample at the th ... The label values for each category label position are defined as the set of positive label indices. ,in Indicates the first The training sample at the th ... The label value for each category label position; for each Calculate the confidence level of the correlation. , Indicates the first Calculate the prediction confidence based on the fusion graph corresponding to each training sample. ,in For the Sigmoid function, For bandwidth parameters; use correlation confidence. and prediction confidence The final confidence level is obtained by fusing the data using a weighted average method and then performing nonlinear enhancement. Based on final confidence level Unlabeled loss function for: ; Step 4.3: Construct the positive sample mining loss function, specifically: the positive sample mining loss function is... ,in This represents the set of highly reliable missing labels that have been discovered. Step 4.4: Combine the label loss function obtained in steps 4.1, 4.2, and 4.3 Unlabeled loss function and positive sample mining loss function The bias-resistant confidence-weighted loss function is obtained. for: , in , , Custom weight parameters.
[0014] The specific implementation method for step 5 is as follows: The initial prediction loss function is determined as follows: The refined prediction loss function is ,in, This is a preliminary prediction obtained from the backbone network. To refine the predictions obtained through the dynamic graph convolutional network, the consistency constraint loss function is constructed as follows: , in , , , For the Sigmoid function; The overall objective loss function is then... ,in , , These are the weighting coefficients.
[0015] Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0016] The beneficial effects of this invention are: (1) The present invention proposes a multi-label image classification method based on anti-bias confidence weighted loss. The anti-bias confidence weighted loss function proposed in the present invention can apply different strategies to supervise the label positions of different states in the sample, thereby effectively mitigating the model bias caused by missing labels.
[0017] (2) The method of the present invention can significantly reduce the labeling cost of training data. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the multi-label image classification network model based on anti-bias confidence weighted loss built in the method of this invention. Detailed Implementation
[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0020] This invention provides a multi-label image classification method based on bias-resistance confidence-weighted loss, specifically as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; The specific implementation method of step 1 is as follows: Assume the training sample set of multi-labeled images containing missing class labels is , It is the set of all class labels in the training sample set. Represents the first in the training sample set There are category labels, among which , This represents the total number of class labels in the training sample set. This represents the total number of training samples. Represents the first in the training sample set Image samples, Represents training sample images The category label vector, express The Each component, if ,express contain ,like ,express Does not contain ,like , indicating uncertainty Does it contain The corresponding tag at this time That is, the sample image Missing class labels.
[0021] Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; The specific implementation method of step 2 is as follows: Construct a completely undirected graph with the number of nodes. The nodes are respectively That is, one node corresponds to one category label; Based on the training sample set Constructing the observation label matrix ,make The Line number The elements of the column are ,Right now From the training sample set Observation label matrix Extract all positive label positions, i.e., label values. For positions where 1 is used, calculate the category label. The conditional probability relationship between them is as follows: First, let the binarized label matrix be... , The Line number The elements of the column are If and only if hour Calculate the co-occurrence matrix ,in for The transpose of the given value yields the result. , It is the set of real numbers; Then, calculate the conditional probability matrix. , The Line number Column elements ,in Co-occurrence matrix No. Line number The elements of the column, i.e. Indicates the first training sample set The first tag and the first The number of times each tag appears together , For the first The total frequency of occurrence of each label, where ε is a small constant to prevent the denominator from being zero. Indicates when the first When the first tag appears The probability of each label appearing, and then the conditional probability matrix. After symmetry processing, the corresponding expression is: Then, for the symmetric conditional probability matrix Each row in the graph is normalized using the L1 norm so that the sum of the weights in each row is 1, resulting in the adjacency matrix of the symbiotic relationship graph. .
[0022] Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss, such as... Figure 1 As shown; In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
[0023] The specific implementation method for step 3 is as follows: Step 3.1: Build the backbone network: Select a deep convolutional neural network (e.g., ResNet18, ResNet34, ResNet101, ResNet152, AlexNet, VGG, etc.). The last layer of the deep convolutional neural network is the basic classification layer. Set the number of neurons in the basic classification layer to [value missing]. The activation function of the basic classification layer is the Sigmoid activation function. The penultimate layer of the deep convolutional neural network is regarded as the feature layer. Several convolutional layers precede the feature layer. The output of the feature layer is called the global feature. The output of the basic classification layer without the Sigmoid activation function is called the preliminary prediction. After the input multi-label image extracts the global features and obtains the preliminary prediction through the backbone network, the obtained global features and the preliminary prediction are concatenated. Then, the concatenated result is sent to the dynamic graph convolutional network module for refined prediction based on the class symbiotic relationship graph. Step 3.2: Constructing a Dynamic Graph Convolutional Network: The dynamic graph convolutional network consists of three parts: a fusion graph generation module, a graph convolutional network, and a classification layer; the dynamic graph convolutional network module is responsible for label-based classification. The symbiotic relationship diagram between them refines the preliminary predictions, specifically as follows: Step 3.2.1: Constructing the Fusion Graph Generation Module: The fusion graph generation module includes two parallel branches: a multilayer perceptron dynamic graph generation branch and a fusion gating branch. The purpose of generating the fusion graph is to generate an adaptive class relationship graph for each input sample. It is assumed that the global features of the image obtained after processing by the backbone network are... Preliminary forecast is ,in For the set of real numbers, For batch processing numbers, To determine the dimension of the global features, first... and By concatenating the vectors, we obtain the joint vector. Then the joint vector The input is fed into a multilayer perceptron (MLP), which is used to map the joint vector dimension to... Do it later Dimensional reshaping and row-wise normalization using Softmax ensures that the sum of each row is 1, yielding the adjacency matrix of the dynamic graph. The structure of the dynamic graph is the same as that of the symbiotic relationship graph, the difference being the adjacency matrix; joint vector It will also be sent into a structure as The fusion gate branch is used to output the sample-by-sample fusion weights. Finally, for the class symbiotic relationship graph constructed in step 2. Add batch dimension And broadcast along that dimension To make it consistent with the dynamic image According to weight The fusion process is performed to obtain the adjacency matrix of the fused graph. , The structure of the fusion graph is the same as that of the symbiotic relationship graph, the difference being the adjacency matrix. Depend on indivual matrix Composition, in which , Indicates the first The adjacency matrix of the fusion graph corresponding to each training sample; the fusion graph is then input into a graph convolutional network; Step 3.2.2: Constructing a Graph Convolutional Network: A three-layer graph convolutional network is adopted. The structure of each layer of the graph convolutional network is the same as that of the fused graph, and the adjacency matrix is also the same. The difference lies in the node values of each layer of the graph convolutional network. The input of the first layer of the graph convolutional network is the initial node features. After the graph convolution operation of the first layer, the output of the first layer is the input of the second layer, and the output of the second layer is the input of the third layer. After the three layers of graph convolution operations, the value of each node in the output graph is a vector. Specifically: Assume there is an identity matrix The number of training samples per batch is ,for Add batch dimension And broadcast along that dimension And for the expanded The last dimension Perform a linear projection so that it passes through the linear projection layer from Dimensional encoding mapping to In the semantic space, the initial node features are obtained. , For a custom semantic space dimension, actually Depend on indivual matrix composition, Indicates the first The initial node features corresponding to each training sample; The following symbols are defined: For graph convolutional networks The output of the layer, For graph convolutional networks Linear transformation of the layer, Here, is a non-linear activation function, where ; The output of the first layer of the convolutional network is... , , , For residual projection layers, residual connections maintain node identity information, preventing node features from being completely assimilated by neighbors during message passing, while also mitigating gradient vanishing. The output of the second layer of the graph convolutional network is , , This layer performs deeper semantic combination, and each node can perceive the information of its second-order neighbors; The output of the third layer of the graph convolutional network is , , .
[0024] The choice of the number of layers in a three-layer graph convolutional network is based on the following considerations: a single layer can only utilize first-order neighbor information, resulting in insufficient graph reasoning ability; three layers can utilize third-order neighbor information, covering most of the actually meaningful label dependency paths; more layers can easily lead to oversmoothing problems.
[0025] Step 3.2.3: Construct the classification layer: The classification layer is a single-layer fully connected layer, and the number of neurons in the classification layer is set to... Its input is the output of the third layer of the graph convolutional network. The activation function of the neurons in the classification layer is the Sigmoid function. The output of the classification layer is a probability vector of a multi-label image belonging to different class labels. The output of the classification layer without the Sigmoid activation function is denoted as... ,Will This is called refined forecasting.
[0026] Step 4: Construct the bias-resistant confidence-weighted loss function; The specific implementation method for step 4 is as follows: The bias-resistant confidence-weighted loss function is designed for the missing label problem. Its core idea is to apply different strategies of supervision signals to the label bits in different states (labeled positive, labeled negative, unlabeled).
[0027] Step 4.1: Construct the label loss function: For labels that are explicitly labeled as positive, ... The value is 1 or negative. For label locations with a value of 0, asymmetric focusing loss (ASL) is used for supervision, specifically: First, assume the observation label matrix of the current batch of sample data. ,make The Line number The elements of the column are ,Right now , Indicates the first The training sample at the th ... The label value for each category label position, let ,in , , For manually specified smoothing parameters; Indicates to The smoothing value is used to adjust the size of the label loss function below; Then, let the prediction probability matrix be... , This is the matrix formed by the predicted vectors of the current batch of samples. For the Sigmoid function, a marker mask is used. The marked position is 1, otherwise it is 0; For the For each training sample, the asymmetric focusing loss is... ,in, Indicates when Time The training sample at the th ... Loss for each category label position, Indicates when Time The training sample at the th ... Loss for each category label position, , , It is the numerical stability constant. Represents the prediction probability matrix The Line number The elements of the column; the asymmetric focusing loss over all training samples in the current batch. The label loss function is obtained by summing and averaging the values. for: .
[0028] Step 4.2: Construct the unlabeled loss function, specifically by defining the set of unlabeled label indices as follows: ,in Indicates the first The training sample at the th ... The label values for each category label position are defined as the set of positive label indices. ,in Indicates the first The training sample at the th ... The label value for each category label position; for each Calculate the confidence level of the correlation. , Indicates the first Calculate the prediction confidence based on the fusion graph corresponding to each training sample. ,in For the Sigmoid function, For bandwidth parameters; use correlation confidence. and prediction confidence The final confidence level is obtained by fusing the data using a weighted average method and then performing nonlinear enhancement. Among them, The purpose of the squaring operation is to use non-linear enhancement to widen the gap between high and low correlations; based on the final confidence level... Unlabeled loss function for: ; Step 4.3: Construct the positive sample mining loss function, specifically: the positive sample mining loss function is... ,in This represents the set of highly reliable missing labels that have been discovered. Step 4.4: Combine the label loss function obtained in steps 4.1, 4.2, and 4.3 Unlabeled loss function and positive sample mining loss function The bias-resistant confidence-weighted loss function is obtained. for: , in , , Custom weight parameters.
[0029] Step 5: Construct the overall objective loss function; The specific implementation method of step 5 is as follows: The overall objective loss function applies the anti-biased confidence weighted loss function from step 4) to both the preliminary and refined predictions, incorporating consistency constraints. The preliminary prediction loss function is determined as follows: The refined prediction loss function is ,in, This is a preliminary prediction obtained from the backbone network. To refine the predictions obtained through the dynamic graph convolutional network, the consistency constraint loss function is constructed as follows: , in , , , For the Sigmoid function; The overall objective loss function is then... ,in , , These are the weighting coefficients.
[0030] Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0031] Example 1 The ResNet101 network was selected as the backbone network, and the proposed method was experimentally verified on two commonly used image classification datasets (MS COCO and Pascal VOC2007).
[0032] The Pascal VOC2007 dataset contains 9963 images across 20 categories, with 5011 images in the training set and 4952 images in the test set. MS COCO, on the other hand, is a large image dataset developed and maintained by Microsoft, covering 80 categories. It contains 118287 images in the training set, 5000 images in the validation set, and 40670 images in the test set. Since the test set annotations are not publicly available, the validation set is used as the test set in actual evaluations.
[0033] In the experiment, to construct a multi-label image training sample set containing missing class labels, the original dataset was first preprocessed: specifically, the class label vectors of all training samples in the original training set were used to form a class label matrix, where each row corresponds to a training sample, and the matrix elements have values of 0 or 1. Then, for each training sample's class label vector, 50% of its label information was retained, and the remaining unretained label values were uniformly set to... , indicating that the corresponding category label is missing, meaning it is unknown whether the sample contains the corresponding category. This constructs a multi-label image training sample set containing missing category labels, with a label retention rate of 50%.
[0034] The experiment used mean average precision (mAP) as the evaluation metric. mAP measures the multi-label classification performance of the network model, and its value ranges from 0 to 1. The higher the mAP value, the better the overall classification performance of the network model.
[0035] Table 1 presents a comparison of the mean accuracy (mAP) of the method of this invention with existing methods (Curriculum labeling, GCN-ML, SSGRL) on the Pascal VOC 2007 and MS COCO datasets. As can be seen from Table 1, the mAP values of the method of this invention are significantly higher than those of the existing methods compared to the present invention on both datasets. The anti-biased confidence weighted loss function proposed in this invention can apply different strategies of supervision to the label positions of different states in the samples, thereby effectively mitigating model bias caused by missing labels, fully demonstrating the effectiveness of the method of this invention.
[0036] Table 1
[0037] Example 2 A multi-label image classification method based on bias-resistance confidence-weighted loss is as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; The specific implementation method of step 1 is as follows: Assume the training sample set of multi-labeled images containing missing class labels is , It is the set of all class labels in the training sample set. Represents the first in the training sample set There are category labels, among which , This represents the total number of class labels in the training sample set. This represents the total number of training samples. Represents the first in the training sample set Image samples, Represents training sample images The category label vector, express The Each component, if ,express contain ,like ,express Does not contain ,like , indicating uncertainty Does it contain The corresponding tag at this time That is, the sample image Missing class labels.
[0038] Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0039] Example 3 A multi-label image classification method based on bias-resistance confidence-weighted loss is as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; The specific implementation method of step 2 is as follows: Construct a completely undirected graph with the number of nodes. The nodes are respectively That is, one node corresponds to one category label; Based on the training sample set Constructing the observation label matrix ,make The Line 1 The elements of the column are ,Right now From the training sample set Observation label matrix Extract all positive label positions, i.e., label values. For positions where 1 is used, calculate the category label. The conditional probability relationship between them is as follows: First, let the binarized label matrix be... , The Line 1 The elements of the column are If and only if hour Calculate the co-occurrence matrix ,in for The transpose of the given value yields the result. , It is the set of real numbers; Then, calculate the conditional probability matrix. , The Line 1 Column elements ,in Co-occurrence matrix No. Line 1 The elements of the column, i.e. Indicates the first training sample set The first tag and the first The number of times each tag appears together , For the first The total frequency of occurrence of each label, where ε is a small constant to prevent the denominator from being zero. Indicates when the first When the first tag appears The probability of each label appearing, and then the conditional probability matrix. After symmetry processing, the corresponding expression is: Then, for the symmetric conditional probability matrix Each row in the graph is normalized using the L1 norm so that the sum of the weights in each row is 1, resulting in the adjacency matrix of the symbiotic relationship graph. .
[0040] Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0041] Example 4 A multi-label image classification method based on bias-resistance confidence-weighted loss is as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
[0042] Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0043] Example 5 A multi-label image classification method based on bias-resistance confidence-weighted loss is as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
[0044] The specific implementation method for step 3 is as follows: Step 3.1: Build the backbone network: Select a deep convolutional neural network. The last layer of the deep convolutional neural network is the basic classification layer. Set the number of neurons in the basic classification layer to [value missing]. The activation function of the basic classification layer is the Sigmoid activation function. The penultimate layer of the deep convolutional neural network is regarded as the feature layer. Several convolutional layers precede the feature layer. The output of the feature layer is called the global feature. The output of the basic classification layer without the Sigmoid activation function is called the preliminary prediction. After the input multi-label image extracts the global features and obtains the preliminary prediction through the backbone network, the obtained global features and the preliminary prediction are concatenated. Then, the concatenated result is sent to the dynamic graph convolutional network module for refined prediction based on the class symbiotic relationship graph. Step 3.2: Constructing the Dynamic Graph Convolutional Network: The Dynamic Graph Convolutional Network consists of three parts: a fused graph generation module, a graph convolutional network, and a classification layer; the dynamic graph convolutional network module is responsible for label-based classification. The symbiotic relationship diagram between them refines the preliminary predictions.
[0045] Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
[0046] Example 6 A multi-label image classification method based on bias-resistance confidence-weighted loss is as follows: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
[0047] The specific implementation method for step 3 is as follows: Step 3.1: Build the backbone network: Select a deep convolutional neural network. The last layer of the deep convolutional neural network is the basic classification layer. Set the number of neurons in the basic classification layer to [value missing]. The activation function of the basic classification layer is the Sigmoid activation function. The penultimate layer of the deep convolutional neural network is regarded as the feature layer. Several convolutional layers precede the feature layer. The output of the feature layer is called the global feature. The output of the basic classification layer without the Sigmoid activation function is called the preliminary prediction. After the input multi-label image extracts the global features and obtains the preliminary prediction through the backbone network, the obtained global features and the preliminary prediction are concatenated. Then, the concatenated result is sent to the dynamic graph convolutional network module for refined prediction based on the class symbiotic relationship graph. In step 3.1, the selected deep convolutional neural network is ResNet18, ResNet34, ResNet101, ResNet152, AlexNet, or VGG deep convolutional neural network.
[0048] Step 3.2: Constructing the Dynamic Graph Convolutional Network: The Dynamic Graph Convolutional Network consists of three parts: a fused graph generation module, a graph convolutional network, and a classification layer; the dynamic graph convolutional network module is responsible for label-based classification. The symbiotic relationship diagram between them refines the preliminary predictions.
[0049] Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
Claims
1. A multi-label image classification method based on bias-resistance confidence-weighted loss, characterized in that, Specifically: Step 1: Prepare a multi-label image training sample set containing missing class labels; Step 2: Construct a class co-occurrence relationship graph based on a multi-label image training sample set containing missing class labels; Step 3: Build a multi-label image classification network model based on bias-resistant confidence-weighted loss; Step 4: Construct the bias-resistant confidence-weighted loss function; Step 5: Construct the overall objective loss function; Step 6: Train the network model built in Step 3 on the training sample set based on the total target loss function constructed in Step 5; Step 7: Input the image to be classified into the network model trained in Step 6, and obtain the predicted category of the image in the classification layer of the dynamic graph convolutional network.
2. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 1, characterized in that, The specific implementation method of step 1 is as follows: Assume the training sample set of multi-labeled images containing missing class labels is , It is the set of all class labels in the training sample set. Represents the first in the training sample set There are category labels, among which , This represents the total number of class labels in the training sample set. This represents the total number of training samples. Represents the first in the training sample set Image samples, Represents training sample images The category label vector, express The Each component, if ,express contain ,like ,express Does not contain ,like , indicating uncertainty Does it contain The corresponding tag at this time That is, the sample image Missing class labels.
3. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 2, characterized in that, The specific implementation method of step 2 is as follows: Construct a completely undirected graph with the number of nodes. The nodes are respectively That is, one node corresponds to one category label; Based on the training sample set Constructing the observation label matrix ,make The Line number The elements of the column are ,Right now From the training sample set Observation label matrix Extract all positive label positions, i.e., label values. For positions where 1 is used, calculate the category label. The conditional probability relationship between them is as follows: First, let the binarized label matrix be... , The Line number The elements of the column are If and only if hour Calculate the co-occurrence matrix ,in for The transpose of the given value yields the result. , It is the set of real numbers; Then, calculate the conditional probability matrix. , The Line number Column elements ,in Co-occurrence matrix No. Line number The elements of the column, i.e. Indicates the first training sample set The first tag and the first The number of times each tag appears together , For the first The total frequency of occurrence of each label, where ε is a small constant to prevent the denominator from being zero. Indicates when the first When the first tag appears The probability of each label appearing, and then the conditional probability matrix. After symmetry transformation, the corresponding expression is: Then, for the symmetric conditional probability matrix Each row in the graph is normalized using the L1 norm so that the sum of the weights in each row is 1, resulting in the adjacency matrix of the symbiotic relationship graph. .
4. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 3, characterized in that, In step 3, the multi-label image classification network model based on bias-resistance confidence weighted loss consists of two modules: a backbone network and a dynamic graph convolutional network. The dynamic graph convolutional network includes a fusion graph generation module, a graph convolutional network, and a classification layer.
5. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 4, characterized in that, The specific implementation method of step 3 is as follows: Step 3.1: Build the backbone network: Select a deep convolutional neural network. The last layer of the deep convolutional neural network is the basic classification layer. Set the number of neurons in the basic classification layer to [value missing]. The activation function of the basic classification layer is the Sigmoid activation function. The penultimate layer of the deep convolutional neural network is regarded as the feature layer. Several convolutional layers precede the feature layer. The output of the feature layer is called the global feature. The output of the basic classification layer without the Sigmoid activation function is called the preliminary prediction. After the input multi-label image extracts the global features and obtains the preliminary prediction through the backbone network, the obtained global features and the preliminary prediction are concatenated. Then, the concatenated result is sent to the dynamic graph convolutional network module for refined prediction based on the class symbiotic relationship graph. Step 3.2: Constructing a Dynamic Graph Convolutional Network: The dynamic graph convolutional network consists of three parts: a fusion graph generation module, a graph convolutional network, and a classification layer; the dynamic graph convolutional network module is responsible for label-based classification. The symbiotic relationship diagram between them refines the preliminary predictions.
6. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 5, characterized in that, In step 3.1, the selected deep convolutional neural network is ResNet18, ResNet34, ResNet101, ResNet152, AlexNet, or VGG deep convolutional neural network.
7. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 6, characterized in that, Step 3.2 specifically involves: Step 3.2.1: Constructing the Fusion Graph Generation Module: The fusion graph generation module includes two parallel branches: a multilayer perceptron dynamic graph generation branch and a fusion gating branch; assuming the global features of the image obtained after processing by the backbone network are... Preliminary forecast is ,in For the set of real numbers, For batch processing numbers, To determine the dimension of the global features, first... and By concatenating the vectors, we obtain the joint vector. Then the joint vector The data is fed into a multilayer perceptron, which is used to map the joint vector dimension to... Do it later Dimensional reshaping and row-wise normalization using Softmax ensures that the sum of each row is 1, yielding the adjacency matrix of the dynamic graph. The structure of the dynamic graph is the same as that of the symbiotic relationship graph, the difference being the adjacency matrix; joint vector It will also be sent into a structure as The fusion gate branch is used to output the sample-by-sample fusion weights. Finally, for the class symbiotic relationship graph constructed in step 2. Add batch dimension And broadcast along that dimension To make it consistent with the dynamic image According to weight The fusion process is performed to obtain the adjacency matrix of the fused graph. , , Depend on indivual matrix Composition, in which , Indicates the first The adjacency matrix of the fusion graph corresponding to each training sample; the fusion graph is then input into a graph convolutional network; Step 3.2.2: Constructing a Graph Convolutional Network: A three-layer graph convolutional network is adopted. The structure of each layer of the graph convolutional network is the same as that of the fused graph, and the adjacency matrix is also the same. The difference lies in the node values of each layer of the graph convolutional network. The input of the first layer of the graph convolutional network is the initial node features. After the graph convolution operation of the first layer, the output of the first layer is the input of the second layer, and the output of the second layer is the input of the third layer. After the three layers of graph convolution operations, the value of each node in the output graph is a vector. Specifically: Assume there is an identity matrix The number of training samples per batch is ,for Add batch dimension And broadcast along that dimension And for the expanded The last dimension Perform a linear projection so that it passes through the linear projection layer from Dimensional encoding mapping to In the semantic space, the initial node features are obtained. , For a custom semantic space dimension, Depend on indivual matrix composition, Indicates the first The initial node features corresponding to each training sample ; The following symbols are defined as follows: For graph convolutional networks The output of the layer, For graph convolutional networks Linear transformation of the layer, Here, is a non-linear activation function, where ; The output of the first layer of the convolutional network is... , , , For residual projection layer; The output of the second layer of the graph convolutional network is , , ; The output of the third layer of the graph convolutional network is , , ; Step 3.2.3: Construct the classification layer: The classification layer is a single-layer fully connected layer, and the number of neurons in the classification layer is set to... Its input is the output of the third layer of the graph convolutional network. The activation function of the neurons in the classification layer is the Sigmoid function. The output of the classification layer is a probability vector of a multi-label image belonging to different class labels. The output of the classification layer without the Sigmoid activation function is denoted as... ,Will This is called refined forecasting.
8. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 7, characterized in that, The specific implementation method for step 4 is as follows: Step 4.1: Construct the label loss function: For labels that are explicitly labeled as positive, ... The value is 1 or negative. For label locations with a value of 0, asymmetric focusing loss is used for supervision, specifically: First, assume the observation label matrix of the current batch of sample data. ,make The Line number The elements of the column are ,Right now , Indicates the first The training sample at the th ... The label value for each category label position, let ,in , , For manually specified smoothing parameters; Indicates to The smoothed value; Then, let the prediction probability matrix be... , This is the matrix formed by the predicted vectors of the current batch of samples. For the Sigmoid function, a marker mask is used. The marked position is 1, otherwise it is 0; For the For each training sample, the asymmetric focusing loss is... ,in, Indicates when Time The training sample at the th ... Loss for each category label position, Indicates when Time The training sample at the th ... Loss for each category label position, , , It is the numerical stability constant. Represents the prediction probability matrix The Line number The elements of the column; the asymmetric focusing loss over all training samples in the current batch. The label loss function is obtained by summing and averaging the values. for: ; Step 4.2: Construct the unlabeled loss function, specifically by defining the set of unlabeled label indices as follows: ,in Indicates the first The training sample at the th ... The label values for each category label position are defined as the set of positive label indices. ,in Indicates the first The training sample at the th ... The label value for each category label position; for each Calculate the confidence level of the correlation. , Indicates the first Calculate the prediction confidence based on the fusion graph corresponding to each training sample. ,in For the Sigmoid function, For bandwidth parameters; use correlation confidence. and prediction confidence The final confidence level is obtained by fusing the data using a weighted average method and then performing nonlinear enhancement. Based on final confidence level Unlabeled loss function for: ; Step 4.3: Construct the positive sample mining loss function, specifically: the positive sample mining loss function is... ,in This represents the set of highly reliable missing labels that have been discovered. Step 4.4: Combine the label loss function obtained in steps 4.1, 4.2, and 4.3 Unlabeled loss function and positive sample mining loss function The bias-resistant confidence-weighted loss function is obtained. for: , in , , Custom weight parameters.
9. The multi-label image classification method based on anti-bias confidence weighted loss according to claim 8, characterized in that, The specific implementation method of step 5 is as follows: The initial prediction loss function is determined as follows: The refined prediction loss function is ,in, This is a preliminary prediction obtained from the backbone network. To refine the predictions obtained through the dynamic graph convolutional network, the consistency constraint loss function is constructed as follows: , in , , , For the Sigmoid function; The overall objective loss function is then... ,in , , These are the weighting coefficients.
