Image multi-label classification method based on graph convolutional neural network and class activation mapping
By constructing a label relationship graph and a graph convolutional neural network, combined with class activation mapping, the difficult problems of building label dependencies and obtaining semantic information in multi-label image classification are solved, achieving more efficient image multi-label classification.
Patent Information
- Application Number
- CN202211190736.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing multi-label image classification methods have difficulties in constructing label dependencies and obtaining image semantic information, resulting in insufficient classification efficiency and accuracy.
By constructing a label relationship graph and a graph convolutional neural network, combined with class activation mapping, the correlation between labels and image semantic information is captured, an image multi-label classification model is constructed, and classification is performed using image feature extraction and back propagation.
It improves the accuracy and efficiency of multi-label image classification, effectively solves the problems of difficulty in building label dependencies and incomplete image semantics acquisition, and improves classification performance.
Smart Images
Figure CN115457332B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image classification, and more specifically, relates to an image multi-label classification method based on graph convolutional neural network and class activation mapping. Background Art
[0002] Multi-label image classification (MLIC) in computer vision aims to assign a set of labels to each image. It is commonly used in fields such as scene understanding, human attribute recognition, and protein subcellular classification. With the development of CNNs, MLIC has achieved significant breakthroughs in performance. However, the efficiency of traditional classification methods has quickly reached a bottleneck. Due to the complexity of reality, the diversity of labels, and the inconsistency of image quality, further development of MLIC remains a significant challenge.
[0003] Traditional multi-label image classification methods rely on two main approaches: algorithm adaptation and problem transformation. Improvements to classification algorithms primarily focus on modifying existing algorithms based on the application of multi-label image classification, leading to the development of a series of multi-label classification algorithms, such as ML-KNN and Rank-SVM. Common problem transformation approaches primarily transform the multi-label image classification task into multiple independent binary classification tasks. However, these methods completely ignore the inherent nature of the multi-label image classification task, namely, label co-occurrence dependencies. Therefore, all candidate regions in the image are first extracted, then single-label recognition is performed on each candidate region, and finally, all results are maximized and merged to obtain the final multi-label prediction result. Alternatively, researchers have proposed classifier chaining (CC), which connects all binary classifiers through a chain based on the BR algorithm. This approach considers pairwise label relationships, which improves the model's generalization ability. However, it also has a significant drawback: it is difficult to find a suitable label dependency construction criterion.
[0004] With the rapid development of CNNs, the efficiency of multi-label image classification has made rapid progress. However, researchers have gradually discovered the fundamental flaw of these methods - labels are likely not independent, and most labels have complex dependencies. For example, when an image is labeled "Tennis," it is likely to also have the labels "People," "Shoe," and "Ball." The possibility of the label "Computer" is very small. Therefore, people have begun to focus on exploring label correlations. To express the abstract correlation between labels, ML-GCN and A-GCN boldly attempted to combine label co-occurrence information with graph neural networks (GCNs) to establish label relationships. As research deepened, people believed that the goal of multi-label image classification is to assign the correct set of labels to each image, rather than to locate all objects, so as not to waste computer resources on unnecessary areas. How to achieve the localization of the target area is also a major challenge.
[0005] In summary, the existing methods do not achieve a perfect balance in extracting image semantic information and building label relationships, which has a great impact on image multi-label classification and needs further improvement. Summary of the Invention
[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an image multi-label classification method based on graph convolutional neural network and class activation mapping. By capturing the correlation between labels and image semantic information, it solves the problems of difficulty in constructing label dependencies and incomplete image semantics in multi-label image problems, thereby improving the performance of image multi-label classification.
[0007] In order to achieve the above-mentioned object of the invention, the image multi-label classification method based on graph convolutional neural network and class activation mapping of the present invention includes the following steps:
[0008] S1: Obtain N image samples according to actual needs and normalize them to the preset size, and obtain the label vector Y of each image sample n =[y n,1 ,y n,2 ,…,y n,M ], y n,m Used to indicate whether the nth image sample belongs to the category corresponding to the mth label, y m =1 means belongs to, y m =0 means it does not belong, m=1,2,…,M, where M represents the number of labels;
[0009] S2: Generate a label relationship graph using the following method:
[0010] S2.1: Based on the label vectors of N image samples, for each label m, statistics y n,m =1 number of image samples am , then the M labels are grouped into label pairs, and for each label pair (m, m′) statistics y n,m =1 and y n,m′ =1 number of image samples b m,m′ , where m′=1,2,…,M and m′≠m; then the co-occurrence probability P(m′|m) of label m and label m′ is calculated using the following formula:
[0011]
[0012] S2.2: For each tag, use the word embedding algorithm to obtain the corresponding text vector;
[0013] S2.3: Treat the text vector of each tag as a node in the tag relationship graph. If the co-occurrence probability P(m′|m) is greater than a preset threshold, add a directed edge from tag m to tag m′; otherwise, do not add the edge, thus obtaining the tag relationship graph. The adjacency matrix of the tag relationship graph is denoted by A, and the node feature matrix consisting of each tag text vector as a row vector is denoted by H.
[0014] S3: Build an image multi-label classification model, including image feature extraction module, global pooling layer, convolution layer, sigmoid layer, local feature screening module, back propagation module, upsampling module, graph convolutional neural network, classification module, maximum pooling layer and softmax layer, where:
[0015] The image feature extraction module is used to extract the K local feature images I obtained by back propagation from the input image I0. k Perform image feature extraction, k = 1, 2, ..., K, and obtain a feature map of size H × W × D. Send the feature map F0 of the input image I0 to the global pooling layer and the back propagation module, and convert the K local feature images I k The feature map F k Send to the global pooling layer;
[0016] The global pooling layer is used to receive the feature map F k′ Perform global pooling respectively, k′=0,1,2,…,K, and obtain the D-dimensional feature vector G k′ And send it to the classification module, and send the feature vector G0 to the convolution layer;
[0017] The convolution layer is used to perform a convolution operation on the feature vector G0 using a 1×1×M convolution kernel to obtain an M-dimensional feature vector C0 and send it to the sigmoid layer;
[0018] The Sigmod layer is used to process the received feature vector C0 using the sigmod function, and convert each element value in the feature vector C0 to the range [0,1] to obtain the predicted probability vector in Represents the predicted probability that the input image I0 belongs to the mth class, and then the predicted probability vector Send to the local feature screening module;
[0019] The local feature screening module is used to predict the probability vector Sort the M prediction probabilities from large to small and select the top K prediction probabilities m k Indicates the category corresponding to the filtered k-th prediction probability, and then the K prediction probabilities Send to the back propagation network;
[0020] The back propagation network is used to receive the K predicted probabilities And feature map F0, respectively according to category m k The predicted probability Perform back propagation to obtain category m k Class activation map A k And sent to the upsampling module, the back propagation formula is as follows:
[0021]
[0022] Among them, D represents the number of channels of the feature map F0, ReLU() represents the relu function, Represents the feature data of the dth channel in the feature map F0, Indicates feature data The weight is calculated as follows:
[0023]
[0024] H and W represent the height and width of the feature map F0 respectively. Represents the feature data of the d-th channel of the feature map F0 at the coordinate (h, w);
[0025] The upsampling module is used to receive the K class activation maps A k Upsample to the size of the input image I0 to obtain the local feature image I k And send it to the image feature extraction module;
[0026] The graph convolutional neural network is used to perform feature learning on the label relationship graph to obtain the embedded representation vector of each label. The input of the graph convolutional neural network is the node feature matrix H, and the adjacency matrix A of the label relationship graph is used as the initial value of the correlation coefficient matrix of the graph convolutional neural network. The embedded representation vector of each label is used as a column vector to construct a label embedding representation matrix E of size D×M, where D represents the dimension of the embedding representation vector. The label embedding representation matrix E is output as the classifier to the classification module.
[0027] The classification module is used to receive K+1 feature vectors G k′ , use classifier E to classify each feature vector G k′ Perform classification and obtain the feature vector G k′ The corresponding estimated probability vector The expression for classification is as follows:
[0028]
[0029] in, According to the eigenvector G k′ The estimated probability of the input image I0 belonging to the mth label category is obtained, and the K+1 estimated probability vectors are obtained Send to the max pooling layer.
[0030] The maximum pooling layer is used to estimate the K+1 probability vectors received Perform maximum pooling to obtain the estimated probability vector And send it to the softmax layer;
[0031] The softmax layer is used to estimate the probability vector using the softmax function Processing is performed to obtain the estimated label vector in Represents the estimated probability that the input image I0 belongs to the mth label category;
[0032] S4: Take each image sample in step S1 as input, and the corresponding label vector Y n =[y n,1 ,y n,2 ,…,y n,M ] as the expected output, train the image multi-label classification model constructed in step S3;
[0033] S5: For the image to be classified, normalize it to a preset size, then input it into the trained image multi-label classification model, and filter out the labels with a probability greater than a preset threshold from the obtained estimated label vector as the label of the image to be classified.
[0034] The present invention is an image multi-label classification method based on graph convolutional neural networks and class activation mapping. A label relationship graph is generated according to the label vector set of image samples, and the embedded representation of each label is obtained through the graph convolutional neural network. A classifier is constructed based on this. In the image multi-label classification model, the global features of the input image are back-propagated to obtain local features. The classifier is used to estimate the global features and local features respectively to obtain the estimated probability vector of the input image, and finally the final estimated probability vector of the input image is obtained by fusion.
[0035] The present invention has the following beneficial effects:
[0036] 1) This invention solves the problem of difficulty in constructing label dependency relationships in multi-label image problems by constructing a label relationship graph to obtain label dependency relationships;
[0037] 2) The present invention extracts image features from both global and local aspects when extracting image features, effectively solving the problem of incomplete image semantics.
[0038] 3) The classifier constructed based on the label relationship graph in the present invention can retain a weaker semantic structure in the word embedding space. Using this classifier to classify image features can capture the correlation between labels and image semantic information, thereby improving the accuracy of multi-label classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a flowchart of a specific implementation of the image multi-label classification method based on graph convolutional neural network and class activation mapping of the present invention;
[0040] Figure 2 It is a flow chart of constructing a classifier based on a label relationship graph in the present invention;
[0041] Figure 3 It is a structural diagram of the image multi-label classification model in the present invention. DETAILED DESCRIPTION
[0042] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.
[0043] Example
[0044] Figure 1 This is a flowchart of a specific implementation of the image multi-label classification method based on graph convolutional neural network and class activation mapping of the present invention. Figure 1 As shown, the specific steps of the image multi-label classification method based on graph convolutional neural network and class activation mapping of the present invention include:
[0045] S101: Get multi-label image samples:
[0046] Obtain N image samples according to actual needs and normalize them to the preset size, and obtain the label vector Y of each image sample n =[y n,1 ,y n,2 ,…,y n,M ], y n,m Used to indicate whether the nth image sample belongs to the category corresponding to the mth label, y m =1 means belongs to, y m =0 means it does not belong, m=1,2,…,M, where M represents the number of labels.
[0047] S102: Generate a label relationship diagram:
[0048] In order to obtain and utilize label dependencies, the present invention learns a set of interdependent classifiers from previously counted label co-occurrence information and label word embeddings based on a graph convolutional neural network (GCN). Therefore, it is necessary to first generate a label relationship graph. Figure 2 This is a flow chart of generating a label relationship diagram in the present invention. Figure 2 As shown, the specific steps of generating a label relationship graph in the present invention include:
[0049] S201: Calculate the co-occurrence probability:
[0050] According to the label vector of N image samples, for each label m, statistics y n,m =1 number of image samples a m , then the M labels are grouped into label pairs, and for each label pair (m, m′) statistics y n,m =1 and y n,m′ =1 number of image samples b m,m′ , where m′=1,2,…,M and m′≠m. Then the co-occurrence probability P(m′|m) of label m and label m′ is calculated using the following formula:
[0051]
[0052] S202: Get label word embedding:
[0053] For each tag, we use a word embedding algorithm to obtain the corresponding text vector. Word embedding is a commonly used text processing method, and its specific method will not be described in detail here.
[0054] S203: Constructing a label relationship diagram:
[0055] The text vector of each tag is used as a node in the tag relationship graph. If the co-occurrence probability P(m′|m) is greater than a preset threshold, a directed edge is added from tag m to tag m′; otherwise, no edge is added, thus forming a tag relationship graph. The adjacency matrix of the tag relationship graph is denoted by A, and the node feature matrix, consisting of each tag text vector as a row vector, is denoted by H. Obviously, the size of the adjacency matrix A is M×M, and the size of the node feature matrix is M×Q, where Q represents the dimension of the text vector.
[0056] S103: Build an image multi-label classification model:
[0057] In order to better realize image multi-label classification, the present invention constructs an image multi-label classification model based on graph convolutional neural network and class activation mapping. Figure 3 This is the structural diagram of the image multi-label classification model in the present invention. Figure 3 As shown in the figure, the image multi-label classification model in the present invention includes an image feature extraction module, a global pooling layer, a convolution layer, a sigmoid layer, a local feature screening module, a backpropagation module, an upsampling module, a graph convolutional neural network, a classification module, a maximum pooling layer, and a softmax layer. Each module is described in detail below.
[0058] The image feature extraction module is used to extract the K local feature images I obtained by back propagation from the input image I0. k Perform image feature extraction, k = 1, 2, ..., K, and obtain a feature map of size H × W × D. Send the feature map F0 of the input image I0 to the global pooling layer and the back propagation module, and convert the K local feature images I k The feature map F k The image feature extraction module in this embodiment adopts a convolutional neural network (CNN), and the specific structure can be set as needed.
[0059] The global pooling layer is used to receive the feature map F k′ Perform global pooling respectively, k′=0,1,2,…,K, and obtain the D-dimensional feature vector G k′ And sent to the classification module, and the feature vector G0 is sent to the convolution layer.
[0060] The convolution layer is used to perform a convolution operation on the feature vector G0 using a 1×1×M convolution kernel to obtain an M-dimensional feature vector C0 and send it to the sigmoid layer.
[0061] The Sigmod layer is used to process the received feature vector C0 using the sigmod function, and convert each element value in the feature vector C0 to the range [0,1] to obtain the predicted probability vector in Represents the predicted probability that the input image I0 belongs to the mth class, and then the predicted probability vector Send to the local feature screening module.
[0062] The local feature screening module is used to predict the probability vector Sort the M prediction probabilities from large to small and select the top K prediction probabilities m k Indicates the category corresponding to the filtered k-th prediction probability, and then the K prediction probabilities Send to the back propagation network.
[0063] The back propagation network is used to receive the K predicted probabilities And feature map F0, respectively according to category m k The predicted probability Perform back propagation to obtain category m k Class activation map A k And sent to the upsampling module, the back propagation formula is as follows:
[0064]
[0065] Among them, D represents the number of channels of the feature map F0, ReLU() represents the relu function, Represents the feature data of the dth channel in the feature map F0, Indicates feature data The weight is calculated as follows:
[0066]
[0067] H and W represent the height and width of the feature map F0 respectively. Represents the feature data of the d-th channel of the feature map F0 at the coordinate (h, w).
[0068] It can be seen that the back propagation network can pass the category m k The predicted probability Perform back propagation, and then use the gradient information back propagated to the feature map F0 to calculate the importance of each channel in the feature map F0 Then through the weight The data of each channel of the feature map F0 is weighted and summed, and finally the class activation map is obtained through the ReLU activation function.
[0069] The upsampling module is used to receive the K class activation maps A k Upsample to the size of the input image I0 to obtain the local feature image I k And sent to the image feature extraction module.
[0070] A graph convolutional neural network (GCNN) is used to learn features from the label relationship graph and obtain embedding representation vectors for each label. The GCNN input is the node feature matrix H, and the adjacency matrix A of the label relationship graph serves as the initial value of the GCNN's correlation coefficient matrix. The embedding representation vector of each label is used as a column vector to construct a label embedding representation matrix E of size D × M, where D represents the dimension of the embedding representation vector. The label embedding representation matrix E is used as the classifier output to the classification module.
[0071] Graph convolutional neural networks (GCNs) perform convolution operations on graphs, similar to convolution operations on images. The basic idea is to update node representations by propagating information between nodes. Through multiple iterations, features between nodes are aggregated to produce a final embedded representation. In this embodiment, the GCN uses a node feature matrix H, and the output of the last layer is the classifier E.
[0072] The classification module is used to receive K+1 feature vectors G k′ , use classifier E to classify each feature vector G k′ Perform classification and obtain the feature vector G k′ The corresponding estimated probability vector The essence of classification is to transform the classifier feature vector G k′ Perform matrix multiplication with the classifier E, that is, the classification expression is as follows:
[0073]
[0074] in, According to the eigenvector G k′ The estimated probability of the input image I0 belonging to the mth label category is obtained, and the K+1 estimated probability vectors are obtained Send to the max pooling layer.
[0075] The maximum pooling layer is used to estimate the K+1 probability vectors received Perform maximum pooling to obtain the estimated probability vector And sent to the softmax layer.
[0076] The softmax layer is used to estimate the probability vector using the softmax function Processing is performed to obtain the estimated label vector in Represents the estimated probability that the input image I0 belongs to the mth label category.
[0077] S104: Training image multi-label classification model:
[0078] Take each image sample in step S101 as input, and the corresponding label vector Yn =[y n,1 ,y n,2 ,…,y n,M ] as the expected output, the image multi-label classification model constructed in step S103 is trained.
[0079] In order to improve the training effect, the calculation method of the loss function during the training process in this embodiment is as follows:
[0080] For the input image I0 and K local feature images I output by the classification module k The eigenvector G k′ The corresponding estimated probability vector Use the softmax function to estimate each probability vector Processing to obtain estimated label vector Then calculate K+1 estimated label vectors respectively The loss between the true label vector of the input image I0 k′ Loss k′ The specific calculation method can be selected according to actual needs. In this embodiment, a soft margin loss function (SoftMargin Loss) is used.
[0081] Obviously, loss0 is the global loss of input image I0, where K loss k It is the local loss of the input image I0. Then the K+1 losses are weighted averaged. k Fusion, get the final loss L, the calculation formula is as follows:
[0082]
[0083] Among them, λ k′ Indicates the preset loss loss k′ The corresponding weight. Weight λ k′ According to actual needs, the present invention selects the features corresponding to the top K largest prediction probabilities when screening local features. In order to adapt to this feature, the weight λ k′ The preferred setting is λ0>λ1>…>λ K , that is, the weight of the global loss is the largest, and the local loss decreases in the order of the prediction probability of its local features.
[0084] S105: Image Classification:
[0085] For the image to be classified, it is normalized to a preset size, and then input into the trained image multi-label classification model. The labels with a probability greater than a preset threshold (0.5 in this embodiment) are screened out from the obtained estimated label vector as the labels of the image to be classified.
[0086] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention, and a more advanced image multi-label classification method in the prior art is used for comparison. The comparison method used in this embodiment adopts models including: CNN-RNN, RNN-attention, Order FreeRNN, ML-ZSL, SRN, partial-BCE, SSNP, and uses global average precision (mAP), average per-class precision (CP), per-class recall (CR), F1 value (CF1), global average precision (OP), global recall (OR), and global F1 value (OF1) as evaluation indicators to compare the image multi-label classification effects of the present invention and the comparison method. In order to facilitate calculation and comparison, the labels with the top three highest scores are used when calculating the evaluation indicators in this embodiment.
[0087] Table 1 is a comparison table of evaluation indicators of the present invention and the comparative method in this embodiment.
[0088]
[0089]
[0090] Table 1
[0091] As shown in Table 1, the three evaluation indicators of global average precision (mAP), F1 value (CF1), and global F1 value (OF1) of the present invention have achieved the best results among all methods, and other indicators are also in the forefront. It can be seen that the present invention has great advantages in image multi-label classification.
[0092] In multi-label image classification, labeling small objects (such as bottles, chairs, tables, sofas, and plants) is challenging. Specifically, when small objects are present in an image, it is difficult to extract their features and assign labels. Therefore, this example specifically analyzes the performance of the present invention and the comparative method on small objects. Table 2 compares the average precision and global average precision of the present invention and the comparative method on small objects in this example.
[0093]
[0094]
[0095] Table 2
[0096] As shown in Table 2, the present invention has achieved the best classification accuracy for most small-sized objects, and is also at the forefront in other categories. It can be seen that the present invention also performs well in feature extraction and label classification of small-sized objects.
[0097] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.
Claims
1. A multi-label image classification method based on graph convolutional neural network and class activation mapping, characterized in that: The following steps are involved: S1: Obtain N image samples according to actual needs and normalize them to the preset size, and obtain the label vector Y of each image sample n =[y n,1 ,y n,2 ,…,y n,M ], y n,m Used to indicate whether the nth image sample belongs to the category corresponding to the mth label, y m =1 means belongs to, y m =0 means it does not belong, m=1,2,…,M, where M represents the number of labels; S2: Generate a label relationship graph using the following method: S2.1: Based on the label vectors of N image samples, for each label m, statistics y n,m =1 number of image samples a m , then the M labels are grouped into label pairs, and for each label pair (m, m′) statistics y n,m =1 and y n,m′ =1 number of image samples b m,m′ , where m′=1,2,…,M and m′≠m; then the co-occurrence probability P(m′|m) of label m and label m′ is calculated using the following formula: S2.2: For each tag, use the word embedding algorithm to obtain the corresponding text vector; S2.3: Treat the text vector of each tag as a node in the tag relationship graph. If the co-occurrence probability P(m′|m) is greater than a preset threshold, add a directed edge from tag m to tag m′; otherwise, do not add the edge, thus obtaining the tag relationship graph. The adjacency matrix of the tag relationship graph is denoted by A, and the node feature matrix consisting of each tag text vector as a row vector is denoted by H. S3: Build an image multi-label classification model, including image feature extraction module, global pooling layer, convolution layer, sigmoid layer, local feature screening module, back propagation module, upsampling module, graph convolutional neural network, classification module, maximum pooling layer and softmax layer, in: The image feature extraction module is used to extract the K local feature images I obtained by back propagation from the input image I0. k Perform image feature extraction, k = 1, 2, ..., K, and obtain a feature map of size H × W × D. Send the feature map F0 of the input image I0 to the global pooling layer and the back propagation module, and convert the K local feature images I k The feature map F k Send to the global pooling layer; The global pooling layer is used to receive the feature map F k′ Perform global pooling respectively, k′=0,1,2,…,K, and obtain the D-dimensional feature vector G k′ And send it to the classification module, and send the feature vector G0 to the convolution layer; The convolution layer is used to perform a convolution operation on the feature vector G0 using a 1×1×M convolution kernel to obtain an M-dimensional feature vector C0 and send it to the sigmoid layer; The Sigmod layer is used to process the received feature vector C0 using the sigmod function, and convert each element value in the feature vector C0 to the range [0,1] to obtain the predicted probability vector in Represents the predicted probability that the input image I0 belongs to the mth class, and then the predicted probability vector Send to the local feature screening module; The local feature screening module is used to predict the probability vector Sort the M prediction probabilities from large to small and select the top K prediction probabilities m k Indicates the category corresponding to the filtered k-th prediction probability, and then the K prediction probabilities Send to the back propagation network; The back propagation network is used to receive the K predicted probabilities And feature map F0, respectively according to category m k The predicted probability Perform back propagation to obtain category m k Class activation map A k And sent to the upsampling module, the back propagation formula is as follows: Among them, D represents the number of channels of the feature map F0, ReLU() represents the relu function, Represents the feature data of the dth channel in the feature map F0, Indicates feature data The weight is calculated as follows: H and W represent the height and width of the feature map F0 respectively. Represents the feature data of the d-th channel of the feature map F0 at the coordinate (h, w); The upsampling module is used to receive the K class activation maps A k Upsample to the size of the input image I0 to obtain the local feature image I k And send it to the image feature extraction module; The graph convolutional neural network is used to perform feature learning on the label relationship graph to obtain the embedded representation vector of each label. The input of the graph convolutional neural network is the node feature matrix H, and the adjacency matrix A of the label relationship graph is used as the initial value of the correlation coefficient matrix of the graph convolutional neural network. The embedded representation vector of each label is used as a column vector to construct a label embedding representation matrix E of size D×M, where D represents the dimension of the embedding representation vector. The label embedding representation matrix E is output as the classifier to the classification module. The classification module is used to receive K+1 feature vectors G k′ , use classifier E to classify each feature vector G k′ Perform classification and obtain the feature vector G k′ The corresponding estimated probability vector The expression for classification is as follows: in, According to the eigenvector G k′ The estimated probability of the input image I0 belonging to the mth label category is obtained, and the K+1 estimated probability vectors are obtained Send to the maximum pooling layer; The maximum pooling layer is used to estimate the K+1 probability vectors received Perform maximum pooling to obtain the estimated probability vector And send it to the softmax layer; The softmax layer is used to estimate the probability vector using the softmax function Processing is performed to obtain the estimated label vector in Represents the estimated probability that the input image I0 belongs to the mth label category; S4: Take each image sample in step S1 as input, and the corresponding label vector Y n =[y n,1 ,y n,2 ,…,y n,M ] as the expected output, train the image multi-label classification model constructed in step S3; S5: For the image to be classified, normalize it to a preset size, then input it into the trained image multi-label classification model, and filter out the labels with a probability greater than a preset threshold from the obtained estimated label vector as the label of the image to be classified.
2. The image multi-label classification method according to claim 1, characterized in that: The loss function during the training of the image multi-label classification model in step S4 is calculated using the following method: For the input image I0 and K local feature images I output by the classification module k The eigenvector G k′ The corresponding estimated probability vector Use the softmax function to estimate each probability vector Processing to obtain estimated label vector Then calculate K+1 estimated label vectors respectively The loss between the true label vector of the input image I0 k′ ; The final loss L is calculated using the following formula: Among them, λ k′ Indicates the preset loss loss k′ The corresponding weight.
3. The image multi-label classification method according to claim 2, characterized in that: The step weight λ k′ Satisfying λ0>λ1>…>λ K .