Improved multi-scale double dynamic graph convolution multi-label image classification method

Abstract

The invention provides an improved multi-label image classification method based on multi-scale double dynamic graph convolution. The invention discloses the multi-label image classification method based on the improved multi-scale double dynamic graph convolution. The invention provides an improved multi-scale double dynamic graph convolution multi-label image classification method for solving the problems that traditional class activation mapping contains noise and the convolution effect becomes poor due to the high expansion rate of cavity convolution. The method comprises the following steps: firstly, introducing depth separable convolution to replace cavity convolution, and redesigning a combination mode of different scale features to obtain improved multi-scale features; secondly, a weight suppression mechanism is introduced to weaken class activation of irrelevant classes, and the class activation is multiplied by the improved multi-scale features to generate a new content awareness vector; and finally, inputting the content perception vector into a double-graph convolution fusion embedded network for final classification. Experimental results show that the precision of the provided model is improved on MS-COCO and VOC2007 data sets, and it is proved that the improved model has better performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, in particular to an improved multi-scale dual dynamic graph convolution multi-label image classification method. BACKGROUND

[0002] Early multi-label classification research mainly relies on convolutional neural network (CNN) for feature extraction and classification. These CNN-based multi-label classification methods mostly only make different degrees of improvement on the existing structure or combine with other advanced models, and fail to fundamentally solve the inherent defects such as the easy loss of key information by pooling operation and the inability to effectively model the correlation between labels.

[0003] In order to overcome these problems, scholars try to use recurrent neural network (RNN), which has strong short-term memory ability and is widely used in processing time series data such as video, audio and text. Some researchers use RNN for multi-label image classification. Compared with CNN, RNN can capture semantic information at different levels and improve classification effect. However, in the case of complex label information, the learning effect of RNN is often limited and cannot achieve the expected performance.

[0004] In recent years, the rapid development of graph structure provides a new research perspective for multi-label classification. Graph structure is very suitable for modeling the relationship between labels. Each node of the graph can be regarded as a label, and the edge between the nodes represents the relationship between the labels. By increasing the number of graph neural network layers, the information of nodes and edges is updated during the training process, and finally the label prediction is performed through the classifier. The content-aware vector obtained by mapping the image features through class activation and combining with the image features can be used as a graph node. However, the traditional class activation mapping is prone to noise and inaccurate positioning. Related research establishes a framework to locate the instance region, eliminates the influence of irrelevant categories, or constructs a content-aware loss function to enhance the accuracy of instance region selection, so as to solve this problem.

[0005] However, real scene images generally have multi-scale target distribution characteristics. If only the single-scale features at the top of the network are relied on for prediction, the spatial resolution of small-scale target information will be attenuated and the semantic representation will be mismatched, which will further cause the problem of incomplete identification of some categories and precision decline in multi-label classification tasks. SUMMARY

[0006] The application aims to solve the problems of traditional class activation map containing noise and high dilatation rate of atrous convolution leading to poor convolution effect, and provides an improved multi-scale double dynamic graph convolution multi-label image classification method, which can fuse deep semantic features and shallow detail features through the construction of a multi-scale feature extraction architecture, form multi-level and multi-granularity composite feature expression, enhance the spatial perception ability and semantic decoupling ability of the model to heterogeneous targets in the image, and finally realize systematic optimization of multi-label classification accuracy and improve multi-label image classification ability.

[0007] The technical scheme is an improved multi-scale double dynamic graph convolution multi-label image classification method, comprising the following steps:

[0008] Step 1: feature extraction is performed on the multi-label image to be classified to obtain image features F;

[0009] Step 2: the image features F are subjected to first deep separable convolution to obtain output features out1; the image features F and the output features out1 are spliced and input into second deep separable convolution to generate output features out2; the image features F, the output features out1 and the output features out2 are spliced and input into third deep separable convolution to obtain output features out3; the output features out1 and the output features out2 are spliced to form intermediate features f1, and the intermediate features f1 and the output features out3 are spliced to obtain features f2; at the same time, a standard 1x1 convolution is applied to the image features F to generate output features out, and in another parallel branch, pooling and up-sampling operations are performed on the image features F to extract global features pool; finally, the output features out1, the output features out2, the output features out3, the intermediate features f1, the features f2, the output features out and the global features pool are spliced to obtain multi-scale features F1;

[0010] Step 3: the image features F are converted into class activation map through convolution layer, and the class activation map is regularized by using sigmoid function to obtain original class activation; the original class activation is subjected to noise suppression by using inhibition weight to obtain noise-suppressed class activation map; the noise-suppressed class activation map and the multi-scale features F1 are aggregated to obtain content perception vector;

[0011] Step 4: the content perception vector is input into static graph convolution network to obtain updated node vector, the updated node vector is input into dynamic graph convolution network to obtain dynamic adjacency matrix, and the updated node vector and the dynamic adjacency matrix are input into graph convolution network to obtain label discrimination vector;

[0012] Step 5: Global spatial pooling is performed on the noise-suppressed class activation map to obtain a set of label prediction scores, and another set of label prediction scores is obtained by passing the label discrimination vector through a binary classifier, and the average of the two sets of label prediction scores is taken as the final label prediction score;

[0013] Step 6: According to the final label prediction score, the classification result of the multi-label image to be classified is obtained.

[0014] Further, in step 1, the feature extraction of the multi-label image to be classified to obtain the image feature F includes:

[0015] The residual network is trained on the ImageNet dataset to obtain a pre-trained residual network; and the pre-trained residual network is used as a feature extraction backbone network;

[0016] The pre-trained residual network is used to extract features of the multi-label image to be classified to obtain the image feature F;

[0017] The residual network is composed of a convolutional pooling layer, four residual blocks, a global average pooling layer, and a fully connected layer.

[0018] Further, the dilation rates of the first, second, and third depth separable convolutions are different.

[0019] Further, in step 3, the original class activation is represented as:

[0020] m c =Sigmoid(Conv(F))

[0021] In the formula, Conv(·) represents a convolutional layer, Sigmoid(·) represents an activation function, and m c represents the original class activation.

[0022] Further, in step 3, the original class activation is suppressed by using a suppression weight to obtain a noise-suppressed class activation map, which includes:

[0023] The suppression weight W c is represented as:

[0024]

[0025] In the formula, st represents a set threshold value, and avg(m c ) represents the average activation value of the class c.

[0026] The suppression weight W c is selectively aggregated with the original class activation m c, by weighting the activation value of each category to suppress the region below the threshold, denoted as:

[0027] M c = W c ⊙ m c

[0028] wherein ⊙ represents an element-wise multiplication operation;

[0029] The class activation map after noise suppression is aggregated with the multi-scale feature F1 to obtain a content-aware vector, denoted as:

[0030]

[0031] In the formula, V represents the content-aware vector.

[0032] Further, the step 4 specifically includes:

[0033] First, the content-aware vector V is input into a static graph convolutional network, denoted as:

[0034] H = LRELU(A s VW s )

[0035] wherein H represents the updated node vector of the content-aware vector V, LRELU(·) represents an activation function, A s and W s respectively represent an adjacency matrix and a weight matrix;

[0036] Second, the updated node vector H is input into a dynamic graph convolutional network to obtain a dynamic adjacency matrix A d , denoted as:

[0037] h g = conv(GAP(H))

[0038] H' = concat(H, h g )

[0039] A d = conv(H')

[0040] Finally, the updated node vector H and the dynamic adjacency matrix A d are input into a graph convolutional network to obtain a label discriminant vector Z:

[0041] Z = LRELU(A d HW).

[0042] Further, the step 4 further includes:

[0043] The self-attention mechanism and the cross-attention mechanism are adopted to reconstruct the label discriminant vector, denoted as:

[0044]

[0045] wherein Q1=ZW q1 , K1=VW k1 , V1=ZW v1 , W q1 , W k1 , W v1 is a projection matrix for responsible linear transformation based on label discriminative vector Z, d1 is the dimension size of K1; k

[0046]

[0047] wherein Q2=Z1W q2 , K2=V2=VW k2 , W q2 , W k2 is a projection matrix for responsible linear transformation based on Z1 and content-aware vector V respectively, d2 is the dimension size of K2;

[0048] Taking Z2 as the input of dynamic graph convolution network, repeating step 4, the final label discriminative vector Z3 is obtained:

[0049] Z3=DGCN(Z2).

[0050] Further, in step 5, the global spatial pooling is performed on the noise-suppressed class activation map to obtain a set of label prediction scores, denoted as:

[0051] s1=mean(topk(Conv 2×2 (M c ),dim))

[0052] wherein dim=-1, k=1;

[0053] The label discriminative vector is input into a binary classifier to obtain another set of label prediction scores, denoted as:

[0054] s2=BC(Z3)

[0055] wherein BC(·) represents the binary classifier.

[0056] ​Beneficial effects: In view of the problems that the traditional class activation map contains noise and the high dilated rate of the dilated convolution leads to poor convolution effect, the improved multi-scale feature is obtained by first introducing the depth separable convolution to replace the dilated convolution and redesigning the combination mode of different scale features; secondly, the weight suppression mechanism is introduced to weaken the class activation of irrelevant categories and multiply the improved multi-scale feature to generate a new content-aware vector; finally, the content-aware vector is input into the double graph convolution fusion embedding network for final classification; the experimental results show that the precision of the application is improved on the MS-COCO and VOC2007 data sets, proving that the method has better performance. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A structure diagram of an improved multi-scale double dynamic graph convolution multi-label image classification method is provided.

[0058] Figure 2 The related class instance activation map is visualized. DETAILED DESCRIPTION

[0059] The technical scheme of the embodiment will be further described in combination with the drawings and examples.

[0060] As shown in the drawings, Figure 1 An improved multi-scale double dynamic graph convolution multi-label image classification method is provided, which mainly includes the following steps:

[0061] Step 1: training a residual network on an ImageNet dataset to obtain a pre-trained residual network, taking the pre-trained residual network as a feature extraction backbone network, and performing feature extraction on the original image through the pre-trained residual network; the residual network used in the embodiment is composed of an initial convolution pooling layer, four residual blocks, a global average pooling layer and a fully connected layer. After feature extraction by the pre-trained residual network, an image feature F with a size of h x omega = 14 x 14 and a channel number d = 2048 can be obtained.

[0062] Step 2: inputting the extracted image feature F into an improved multi-scale feature conversion module to obtain each scale feature F1 and improve the model perception ability; the specific operation includes:

[0063] First, the image feature F is processed by a depth separable convolution with a dilated rate r = 1 to obtain an output feature out1.

[0064] Subsequently, the image feature F and the output feature out1 are spliced and input into a depth separable convolution with a dilated rate r = 3 to generate an output feature out2.

[0065] After that, the image feature F, the output feature out1, and the output feature out2 are spliced and input into a depth separable convolution with a hole rate r = 5 to obtain an output feature out3.

[0066] In subsequent processing, the output feature out1 and the output feature out2 are spliced to form an intermediate feature f1, and the intermediate feature f1 and the output feature out3 are spliced to obtain a feature f2. At the same time, in order to continuously retain the receptive field information of the original input, a standard 1x1 convolution is applied to the image feature F to generate an output feature out.

[0067] In another parallel branch, the global feature pool of the image feature F is extracted by performing a pooling and upsampling operation on the image feature F.

[0068] Finally, the output feature out1, the output feature out2, the output feature out3, the intermediate feature f1, the feature f2, the output feature out, and the global feature pool are spliced to obtain a multi-scale feature F1 as Concat(out1, out2, out3, f1, f2, out, pool).

[0069] Step 3: input the image feature F into the non-relevant instance suppression module to obtain a class activation map after suppressing noise, multiply the multi-scale feature and the class activation map to obtain a content-aware vector; the specific operation includes:

[0070] First, the image feature map is extracted by image feature learning Where D, H, and W represent the number of channels, height, and width of the feature map, respectively. The image feature F is converted into a class activation map by a convolutional layer and is regularized by a sigmoid function to obtain the original class activation m c as shown in the following formula:

[0071] m c = Sigmoid(Conv(F))

[0072] Where Conv(·) is a convolutional layer and Sigmoid(·) is an activation function.

[0073] Then, a threshold st is set as a reference value. In the training process, when there are some weakly activated non-relevant classes, their activation values are compared with the threshold to determine which class instances should be retained and which should be suppressed to generate a suppression weight W c to highlight the instance region. The calculation method of W c is as follows:

[0074]

[0075] Where avg(mc ) represents the average activation value of the category c. When the average activation value of the category is greater than the threshold value, the weight 1.0 is assigned, indicating that the influence of the category should remain unchanged; when the category activation value is weak, the weight 0.1 is assigned to suppress the influence of the category instance while retaining a small amount of information to prevent weak category instances from having an unnecessary impact on the final result. The threshold value st is set to 0.3 for the best effect.

[0076] The inhibition weight W c The original category activation m c is selectively aggregated, and the area below the threshold is inhibited by weighting the activation value of each category, represented as:

[0077] M c = W c ⊙m c

[0078] where ⊙ represents the element-wise multiplication operation. This process effectively reduces noise while highlighting discriminative instance areas.

[0079] Finally, the category activation map M c is aggregated with the multi-scale feature F1, and the aggregated content-aware vector V (V = v1, v2, … v c ) is obtained, represented as:

[0080]

[0081] Step 4: Input the content-aware vector V into the dual graph convolution fusion embedding network to reconstruct the graph nodes and obtain the label discriminative vector Z; the specific operation includes:

[0082] First, input the content-aware vector V into the static graph convolution network, represented as:

[0083] H = LRELU(A s VW s )

[0084] where H represents the updated node vector of the content-aware vector V, LRELU(·) represents the activation function, A s and W s represent the adjacency matrix and weight matrix respectively.

[0085] Second, the updated node vector H is input into the dynamic graph convolution network to obtain the dynamic adjacency matrix A d , represented as:

[0086] h g = conv(GAP(H))

[0087] H' = concat(H, h g )

[0088] A d =conv(H')

[0089] Finally, the updated node vector H is input into the dynamic adjacency matrix A d to obtain the label discriminative vector Z, that is:

[0090] Z = LRELU(A d HW)

[0091] In order to improve the accuracy of label prediction, self-attention mechanism and cross-attention mechanism are used to reconstruct the label discriminative vector, which is represented as:

[0092]

[0093] wherein Q1 = ZW q1 , K1 = ZW k1 , V1 = ZW v1 , W q1 , W k1 , W v1 are projection matrices for linear transformation based on the label discriminative vector Z, d1 k is the dimension size of K1;

[0094]

[0095] wherein Q2 = Z1W q2 , K2 = V2 = VW k2 , W q2 , W k2 are projection matrices for linear transformation based on Z1 and content-aware vector V respectively, d2 is the dimension size of K2;

[0096] Finally, Z2 is taken as the input of the dynamic graph convolution network, and the above process is repeated, so as to obtain the final label discriminative vector Z3:

[0097] Z3 = DGCN(Z2)

[0098] Step 5: global spatial pooling is performed on the class activation map M c to obtain a set of label prediction scores s1, and the label discriminative vector Z is input into the binary classifier BC to obtain another set of label prediction scores s2, and the average value of the two sets of prediction scores is taken as the final label prediction score s. The specific representation is:

[0099] s1 = mean(topk(Conv 2×2 (M c ), dim))

[0100] s2 = BC(Z3)

[0101]

[0102] where dim=-1, k=1.

[0103] To verify the performance of the improved multi-scale dual dynamic graph convolution multi-label image classification method proposed in this embodiment, the following experiments and result analysis are performed:

[0104] The comparative experiments are performed on two public multi-label image recognition datasets, COCO2014 and VOC2007, based on the PyTorch framework, and the training process is completed using an NVIDIA GeForce RTX 4060Ti graphics card. In the specific implementation, the method of this embodiment uses a deep residual network pre-trained on the ImageNet dataset for feature extraction. In the training process, random cropping and adjusting the image size to 448x448 are used, and random horizontal flipping is used for data augmentation. The optimizer selects SGD, the momentum is set to 0.9, the weight decay is 0.001, and the batch size of the GPU is 16. In the dual dynamic graph convolution network classification, the initial learning rate is set to 0.5, and the initial learning rate of the deep residual network is set to 0.025. The entire training process is performed for 50 rounds, and the learning rate is reduced by 10 times at 30 rounds. These training strategies help improve the training effect of the model and ensure better generalization performance.

[0105] Figure 2 The activation map of the related class of the method of the present application. ResNet101 and MFE2DGCN have a fuzzy problem in positioning, and are prone to produce noise. In contrast, the method of this embodiment performs best in positioning the instance region, and can effectively suppress some class regions with weak activation values. For example, in the bus class, ResNet101 locates the attention to the car next to it, while ADDGCN can accurately locate the bus, but also produces noise. Relatively speaking, the method of this embodiment can accurately locate the bus region, showing stronger positioning ability and suppression effect.

[0106] To compare fairly with other existing methods, overall / class precision (OP / CP), overall / class recall (OR / CR), overall / class F1 score (OF1 / CF1), and mean average precision (mAP) are used as evaluation indicators.

[0107] Table 1 is the comparative experimental results of the method of the present application on COCO2014. Five indexes of mAP, OF1, CF1, OP and CP are used for comprehensive evaluation. Under the standard test resolution of 448*448, mAP reaches 86.6%, which is 1 percentage point higher than MFE2DGCN, OF1 and CF1 increase by 1.1% and 0.1% respectively, which shows the advantage of the method in class boundary division; OP and CP increase by 0.9% and 1.0% respectively, which shows the improvement of the model prediction reliability; OR increases by 0.7% and surpasses the performance of other methods. Under the standard test resolution of 576*576, mAP reaches 87.7%, which is 1.3 percentage points higher than MFE2DGCN, OF1 and CF1 increase by 0.7% and 0.2% respectively, which shows the advantage of the method in class boundary division, and OR and CR double indexes lead and surpass the performance of other methods.

[0108] Table 1 is the comparative experimental results of the method of the present application on COCO2014. Five indexes of mAP, OF1, CF1, OP and CP are used for comprehensive evaluation. Under the standard test resolution of 448*448, mAP reaches 86.6%, which is 1 percentage point higher than MFE2DGCN, OF1 and CF1 increase by 1.1% and 0.1% respectively, which shows the advantage of the method in class boundary division; OP and CP increase by 0.9% and 1.0% respectively, which shows the improvement of the model prediction reliability; OR increases by 0.7% and surpasses the performance of other methods. Under the standard test resolution of 576*576, mAP reaches 87.7%, which is 1.3 percentage points higher than MFE2DGCN, OF1 and CF1 increase by 0.7% and 0.2% respectively, which shows the advantage of the method in class boundary division, and OR and CR double indexes lead and surpass the performance of other methods.

[0109] Algorithm Test resolution OP OR OF1 CP CR CF1 mAP ResNet-101 448*448 86.4 71.8 78.4 82.7 67.4 74.3 79.7 ML-GCN 448*448 85.8 75.4 80.3 85.1 72.0 78.0 83.0 ADD-GCN 448*448 86.2 76.6 81.1 85.1 73.6 79.0 84.2 SALGL 448*448 87.8 77.6 82.4 87.2 74.5 80.4 85.5 MSFGAT 448*448 85.8 76.3 80.8 84.1 73.4 78.4 83.6 PCMT 448*448 85.8 67.3 81.5 81.8 74.5 80.0 83.3 MSFA 448*448 89.7 72.7 80.3 87.0 70.2 77.7 83.0 MFE2DGCN 448*448 88.4 76.0 81.7 87.5 73.7 80.6 85.6 Ours 448*448 89.3 76.7 82.8 88.5 73.7 80.7 86.6 ADD-GCN 576*576 85.7 78.3 81.8 89.1 75.1 79.8 84.9 2S-DGCN 576*576 86.8 78.0 82.2 84.9 75.7 80.0 85.6 C-Tran 576*576 87.7 76.5 81.7 86.3 74.3 79.9 85.1 ISLC 576*576 88.9 75.6 81.8 88.3 72.5 79.7 85.3 LVPL 576*576 87.9 77.0 82.1 86.6 74.9 80.3 86.0 SIG-MLIC 576*576 87.7 77.5 82.3 86.4 75.1 80.3 85.6 MFE2DGCN 576*576 89.5 76.4 82.4 88.8 73.7 81.5 86.4 Ours 576*576 86.4 78.4 83.1 86.3 76.8 81.7 87.7

[0110] Table 2 is the comparative experimental results of the method of the present application on VOC2007. The results show that the method of the present application achieves higher accuracy in most categories, and the mAP is 0.9% higher than MFE2DGCN, reaching the highest value among all methods. This further proves the superiority of the method of the present application in multi-label image recognition tasks.

[0111] Table 2 is the comparative experimental results of the method of the present application on VOC2007. The results show that the method of the present application achieves higher accuracy in most categories, and the mAP is 0.9% higher than MFE2DGCN, reaching the highest value among all methods. This further proves the superiority of the method of the present application in multi-label image recognition tasks.

[0112] Method aero bike bird boat bottle bus car cat chair cow table dog horse motor person plant sheep sofa train tv mAP ResNet-101 99.5 97.7 97.8 96.4 65.7 91.8 96.1 97.6 74.2 80.9 85.0 98.4 96.5 95.9 98.4 70.1 88.3 80.2 98.9 89.2 89.9 ML-GCN 99.6 98.3 97.9 97.6 78.2 92.3 97.4 97.4 79.2 94.4 86.5 97.4 97.9 97.1 98.7 84.6 95.3 83.0 98.6 90.4 93.1 TSGCN 98.9 98.5 96.8 97.3 87.5 94.2 97.4 97.7 84.1 92.6 89.3 98.4 98.0 96.1 98.7 84.9 96.6 87.2 98.4 93.7 94.3 ADD-GCN 99.7 98.5 97.6 98.4 80.6 94.1 96.6 98.1 80.4 94.9 85.7 97.9 97.9 96.4 99.0 80.2 97.3 85.3 98.9 94.1 93.6 2S-DGCN 99.6 98.0 98.9 98.2 83.1 96.0 96.8 98.0 82.4 96.8 89.9 98.7 98.6 97.5 98.9 84.9 98.9 81.1 99.2 94.3 94.5 DSDL 99.8 98.7 98.4 97.9 81.9 95.4 97.6 98.3 83.3 95.0 88.6 98.0 97.9 95.8 99.0 86.6 95.9 86.4 98.6 94.4 94.4 SST 99.8 98.6 98.5 98.4 85.5 94.7 97.9 98.6 83.0 96.8 85.7 98.8 98.9 95.7 99.1 85.4 96.2 84.3 99.1 95.0 94.5 ML-SGM 99.9 98.8 98.5 98.6 86.3 96.0 98.0 99.2 84.5 97.6 87.7 99.2 98.9 97.2 99.3 86.0 98.3 87.2 99.1 95.3 95.2 PCMT 99.7 98.7 98.5 98.2 82.2 95.7 97.4 97.8 81.7 95.3 85.3 98.2 98.3 96.4 99.0 84.9 96.4 85.5 98.8 95.1 94.2 MSFA 99.8 98.6 98.2 99.1 84.1 95.5 98.0 98.4 84.3 98.4 88.3 98.4 98.6 97.3 99.0 86.7 98.3 88.5 99.2 95.2 94.9 LVPL 99.9 98.8 98.6 98.6 84.8 98.5 97.6 98.5 83.6 97.2 88.8 99.3 99.0 98.7 99.1 87.3 99.2 87.8 99.6 97.0 95.1 MFE2DGCN 99.7 98.4 98.7 98.1 84.8 98.0 98.4 98.4 83.4 97.5 91.6 99.3 98.9 98.7 99.3 88.1 98.5 83.0 99.4 96.1 95.4 Ours 99.7 99.2 99.3 98.4 85.5 98.5 97.5 98.4 83.7 97.9 92.0 99.3 98.9 98.9 99.2 88.3 98.4 87.7 98.9 96.1 96.5

[0113] Table 3 is the ablation experiment of the method of the present application. In order to verify the effectiveness of the used modules, the improved multi-scale feature conversion module (MFT) and the weight suppression module (IS) are added to MFE2DGCN respectively. From the experimental results, on the COCO dataset, after adding the weight suppression module, mAP increases to 86.6%, increasing by 0.2%; after adding the improved multi-scale conversion module, mAP is 86.7%, increasing by 0.3%; and when both modules are added, mAP reaches 87.7%, increasing by 2.1%. Similar results are also obtained on the VOC2007 dataset. These experimental results fully prove the positive contribution of each module to the model performance.

[0114] Table 3 is the ablation experiment of the method of the present application on COCO2014 and VOC2007 datasets.

[0115] Method mAP(COCO) mAP(VOC) MFE2DGCN 86.4 95.4 MFE2DGCN+MFT 86.7 95.9 MFE2DGCN+IS 86.6 95.7 Ours 87.7 96.5

Claims

1. An improved multi-scale dual dynamic graph convolution multi-label image classification method, characterized in that: The method comprises the following steps: Step 1: feature extraction is performed on the multi-label image to be classified to obtain image features F; Step 2: the image features F are subjected to first deep separable convolution to obtain output features out1; the image features F and the output features out1 are spliced and sent to second deep separable convolution to generate output features out2; the image features F, the output features out1 and the output features out2 are spliced and input to third deep separable convolution to obtain output features out3; the output features out1 and the output features out2 are spliced to form intermediate features f1, and the intermediate features f1 and the output features out3 are spliced to obtain features f2; at the same time, a standard 1x1 convolution is applied to the image features F to generate output features out, and in another parallel branch, a pooling and up-sampling operation is performed on the image features F to extract global features pool; finally, the output features out1, the output features out2, the output features out3, the intermediate features f1, the features f2, the output features out and the global features pool are spliced to obtain multi-scale features F1; Step 3: the image features F are converted into class activation mapping through a convolution layer, and the class activation mapping is regularized using a sigmoid function to obtain original class activation; noise suppression is performed on the original class activation using a suppression weight to obtain noise-suppressed class activation mapping; The noise-suppressed class activation mapping and the multi-scale features F1 are aggregated to obtain a content-aware vector; Step 4: the content-aware vector is input into a static graph convolution network to obtain an updated node vector, the updated node vector is input into a dynamic graph convolution network to obtain a dynamic adjacency matrix, and the updated node vector and the dynamic adjacency matrix are input into a graph convolution network to obtain a label discrimination vector; Step 5: global spatial pooling is performed on the noise-suppressed class activation mapping to obtain a group of label prediction scores, and the label discrimination vector is input into a binary classifier to obtain another group of label prediction scores, and the average value of the two groups of label prediction scores is taken as the final label prediction score; Step 6: a classification result of the multi-label image to be classified is obtained according to the final label prediction score.

2. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 1, characterized in that: In step 1, the feature extraction on the multi-label image to be classified to obtain the image features F comprises the following steps: A residual network is trained on an ImageNet dataset to obtain a pre-trained residual network; the pre-trained residual network is used as a feature extraction backbone network; The pre-trained residual network is used to extract features of the multi-label image to be classified to obtain the image features F; The residual network comprises a convolutional pooling layer, four residual blocks, a global average pooling layer and a fully connected layer.

3. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 1, characterized in that: The dilation rates of the first deep separable convolution, the second deep separable convolution and the third deep separable convolution are different.

4. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 1, characterized in that: In step 3, the original class activation is represented as: m c = Sigmoid(Conv(F)) where Conv(·) represents a convolutional layer, Sigmoid(·) represents an activation function, m c represents the original class activation.

5. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 4, characterized in that: In step 3, the noise-suppressed class activation mapping is obtained by performing noise suppression on the original class activation using a suppression weight, and the specific operation comprises the following steps: The inhibition weight W c is expressed as: where st denotes a set threshold value; avg(m c ) denotes an average activation value of the class c; The inhibition weight W c The original class activation map m c The region below the threshold is inhibited by weighting the activation value of each class, denoted as: M c = W c ⊙ m c Wherein, represents an element-wise multiplication operation. The noise-suppressed class activation map is aggregated with the multi-scale feature F1 to obtain a content-aware vector, denoted as: In the formula, V represents the content-aware vector.

6. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 5, characterized in that: The specific operation of step 4 includes: First, the content-aware vector V is input into a static graph convolution network, denoted as: H = LRELU(A s VW s ) wherein H represents a node vector after updating the content-aware vector V, LRELU(·) represents an activation function, A s and W s represent an adjacency matrix and a weight matrix, respectively; Secondly, the updated node vector H is input into a dynamic graph convolution network to obtain a dynamic adjacency matrix A d , which is expressed as: h g = conv(GAP(H)) H' = concat(H, h g ) A d = conv(H') Finally, the updated node vector H is inputted into the dynamic adjacency matrix A d to obtain the label discriminant vector Z: Z = LRELU(A d HW).

7. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 6, characterized in that: Step 4 further includes: The self-attention mechanism and the cross-attention mechanism are used to reconstruct a label discrimination vector, denoted as: wherein Q1 = ZW q1 , K1 = ZW k1 , V1 = ZW v1 , W q1 , W k1 , W v1 is a projection matrix for responsible linear transformation based on the label discrimination vector Z, d k is the dimension size of K1; wherein Q2=Z1W q2 , K2=V2=VW k2 , W q2 , W k2 are projection matrices for responsible linear transformation based on Z1and content-aware vector V respectively, and d2is the dimension size of K2. Z2 is taken as the input of the dynamic graph convolution network, and step 4 is repeated to obtain a final label discrimination vector Z3: Z3=DGCN(Z2).

8. The improved multi-scale dual dynamic graph convolution multi-label image classification method according to claim 7, characterized in that: In step 5, the noise-suppressed class activation map is globally spatially pooled to obtain a group of label prediction scores, denoted as: s1 = mean(topk(Conv 2×2 (M c ), dim)) In the formula, dim=-1, and k=1. The label discrimination vector is input into a binary classifier to obtain another group of label prediction scores, denoted as: s2=BC(Z3) In the formula, BC(·) represents the binary classifier.

Images