A multi-label image classification method based on deep learning

Abstract

The present invention provides a multi-label image classification method based on deep learning, comprising: obtaining an image to be classified with multiple labels; inputting the image to be classified into a trained multi-label image classification model to obtain a classification result; the multi-label image classification model comprises a feature extraction network and a Transformer structure connected to the feature extraction network; the Transformer structure consists of an encoder and a decoder; the encoder is used to extract features and integrate output dimensions; the decoder is used to cross-calculate the input label category with the corresponding features. The present invention can improve the accuracy of the multi-target image classification algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a multi-label image classification method based on deep learning. Background Art

[0002] Image classification is a basic task in computer vision. This task replaces human visual judgment, extracts features from input images, and assigns known category labels to them, providing important support for tasks such as object detection and semantic segmentation. In general, image classification can be divided into two types according to the number of labels in the image: single-label image classification and multi-label image classification (MLIC). In actual situations, some images have problems such as occluded targets, complex scenes, and multiple subjects corresponding to labels appear in the same image. From the perspective of development history, MLIC is an extension of the single-label image classification problem, but the characteristics of multi-label images make the MLIC task more difficult and complex than single-label classification. Directly applying the single-label classification model to solve the MLIC problem often does not produce ideal results. In addition, there is a certain correlation between different labels. If this information is ignored, the output space will grow exponentially with the increase in the number of labels, and the number of corresponding labels in each combination will be unequal, which further increases the difficulty of learning. Multi-label image classification is in great demand in real life. It has a wide range of application scenarios, from network image classification, album management to disease prediction, autonomous driving, and other fields. These scenarios have high requirements for classification accuracy. How to improve the accuracy of multi-target image classification algorithms is of great research value. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a multi-label image classification method based on deep learning, which can improve the accuracy of the multi-target image classification algorithm.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A multi-label image classification method based on deep learning, comprising:

[0006] Get an image to be classified with multiple labels;

[0007] The image to be classified is input into a trained multi-label image classification model to obtain a classification result; the multi-label image classification model includes a feature extraction network and a Transformer structure connected to the feature extraction network; the Transformer structure consists of an encoder and a decoder; the encoder is used to extract features and integrate output dimensions; the decoder is used to cross-calculate the input label category with the corresponding features.

[0008] Preferably, the feature extraction step of the feature extraction network comprises:

[0009] Initialize the structure and parameters of the feature extraction network; the feature extraction network is a Resnet-101 model;

[0010] Five convolutional layers are used to calculate the image to be classified to obtain the global features of the image to be classified; wherein the bottleneck convolution block of the fifth convolutional layer is replaced by three parallel dilated convolution structures, and the dilation rates are 1, 2, and 3 respectively.

[0011] Preferably, the parameters are weight data obtained by training the Resnet-101 model on the MS-COCO large-scale visual dataset.

[0012] Preferably, the steps of extracting features and integrating output dimensions of the encoder are as follows:

[0013] Flatten the feature matrix formed by the global features into a one-dimensional vector, denoted as F;

[0014] Initialize the weight matrix W Q , W E , W V ;

[0015] The feature matrix F and the weight matrix W Q , W E , W V Multiply them and we get Q F , K F 、V F Matrix, the specific calculation method is:

[0016] Preferably, the decoder performs the following steps to cross-calculate the input label category and the corresponding feature:

[0017] The label vector of the image is denoted as E; the feature matrix corresponding to the label vector is Q E , K E 、V E matrix;

[0018] The label vector is calculated using the standard Multihead() method of the PyTorch library, and the result T1 is obtained through the multi-head attention module. The calculation method of the result T1 is expressed as: T1 = MulitHead(Q E ,K E ,V E );

[0019] T1 and K F , K F The matrix is ​​calculated through the cross attention module to obtain the result T2. The calculation method of the result T2 is expressed as: T2 = MulitHead (T1, K F ,V F ).

[0020] Preferably, the multi-label image classification model further includes a fully connected layer and a sigmoid activation function; the fully connected layer is used to calculate the result T2, and then passes through the sigmoid activation function to finally generate a vector for each image with a dimension equal to the total number of categories to be classified, and the value of each dimension of the vector represents the probability p that the image to be detected belongs to each corresponding category n ; probability p n The calculation formula is: n =Sigmoid(wT2+b); where w is the weight coefficient and b is the intercept.

[0021] Preferably, the loss function of the multi-label image classification model is an asymmetric loss function.

[0022] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0023] The present invention provides a multi-label image classification method based on deep learning, comprising: obtaining an image to be classified with multiple labels; inputting the image to be classified into a trained multi-label image classification model to obtain a classification result; the multi-label image classification model comprises a feature extraction network and a Transformer structure connected to the feature extraction network; the Transformer structure consists of an encoder and a decoder; the encoder is used to extract features and integrate output dimensions; the decoder is used to cross-calculate the input label category with the corresponding features. The present invention can improve the accuracy of the multi-target image classification algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0026] Figure 2 A model framework diagram provided for an embodiment of the present invention;

[0027] Figure 3 The receptive field of the dilated convolution at different dilation rates provided by the embodiment of the present invention;

[0028] Figure 4 A partial schematic diagram of a Trident Block provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of an improved Trident Block provided in an embodiment of the present invention;

[0030] Figure 6 A structure diagram of a decoder provided in an embodiment of the present invention;

[0031] Figure 7 Data distribution of the PASCAL VOC 2007 dataset provided by the embodiment of the present invention;

[0032] Figure 8 A multi-label image classification model diagram based on improved residual network provided in an embodiment of the present invention;

[0033] Fig. 9 A flow chart of a multi-label image classification method based on an improved residual network provided in an embodiment of the present invention;

[0034] Fig.10 A test image provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a multi-label image classification method based on deep learning, comprising:

[0038] Step 100: Obtain an image to be classified with multiple labels;

[0039] Step 200: Input the image to be classified into a trained multi-label image classification model to obtain a classification result; the multi-label image classification model includes a feature extraction network and a Transformer structure connected to the feature extraction network; the Transformer structure consists of an encoder and a decoder; the encoder is used to extract features and integrate output dimensions; the decoder is used to cross-calculate the input label category with the corresponding features.

[0040] like Figure 2 As shown in the figure, combining the design ideas of TridentNet and DETR network, this embodiment proposes a multi-label image classification model based on improved ResNet and Transformer. The whole model consists of two modules. The first stage is the feature extraction network, which uses improved ResNet. The second stage consists of encoder and decoder architecture.

[0041] Specifically, in the field of target detection, the scale of the target to be detected is often very uncertain. Some targets that are too large or too small pose great challenges to the detection task. There are three mainstream solutions to the scale change problem:

[0042] 1) Image pyramid: The network model directly extracts features from multiple scales of the same image. This method makes full use of the information of the image. Large objects correspond to images with lower resolution, and small objects correspond to images with higher resolution. This method is simple and effective, such as the SNIP network, but it requires a lot of computation.

[0043] 2) Feature Pyramid: Proposed in FPN, this method makes full use of image information and obtains features of multiple scales of the image by fusing feature maps of different depths. This network structure combines the spatial information of the shallow network and the abstract semantic features of the deep network, but sacrifices the consistency of features at different scales.

[0044] TridentNet proposes a new solution, arguing that under the same receptive field, the performance of the network will be different when detecting objects of different sizes. First, we will introduce the concepts of dilated convolution and Trident block.

[0045] Furthermore, in the traditional CNN network, the model completes the feature extraction of the image through multiple layers of convolution and improves the receptive field of each pixel through pooling. However, such operation will lead to the loss of image information. In order to avoid the problem of insufficient model detection performance caused by information loss caused by multiple pooling, the prior art proposes a dilation convolution method and introduces the concept of dilation rate. In the convolution kernel, a value of 0 is used to represent a hole, and the specific number of holes is determined by the dilation rate.

[0046] Figure 3 The receptive field of dilated convolution under different dilation rates is shown. When the dilation rate is 1, it is equivalent to ordinary convolution calculation without hole filling; when the dilation rate is 2, the receptive field expands to 7*7; when the dilation rate is 3, the receptive field is 13*13. Compared with traditional convolution operations, the receptive field of dilated convolution grows exponentially with the increase of the convolution kernel. The advantage of dilated convolution is that it increases the receptive field of feature extraction without increasing the amount of calculation, overcoming the problem of information loss in the traditional convolution feature extraction process.

[0047] Alternatively, the main idea of ​​TridentNet is to use branches with different receptive fields to detect targets of different scales, so the Trident Block is proposed. This module contains several parallel branches, each of which has a dilated convolution block. These branches have the same size of convolution kernel, and the only difference is the dilation rate of the dilated convolution. In this way, the model has multiple different receptive fields. Take the residual block in ResNet as an example, the detailed structure is as follows Figure 4 As shown in the figure. In TridentBlock, three dilated convolutions with different dilation rates are designed. During the training process, each branch is optimized separately. Small targets go through the first branch, medium targets go through the second branch, and large targets go through the third branch. This allows targeted training. During prediction, each branch will produce a detection result, and the result of each branch will be processed by NMS (Non-Maximum Suppression). At the same time, the three convolution kernels share parameters, which not only allows the model to maintain the same number of parameters as the original structure and avoid the risk of overfitting caused by the increase in parameters, but also allows the features at different scales to be fused, so that the same target has the same feature expression ability at different scales.

[0048] This embodiment draws on the idea of ​​TridentNet and improves the original Trident module to make it more suitable for completing classification-related tasks. Figure 5 As shown in Figure 1, its main structure is still three parallel dilated convolution branches and shared parameters, but there are two differences from the structure in the original object detection network. First, the calculation results of the three branches are added and fused. Second, each branch is no longer designed as a residual structure, but the entire module is regarded as a residual block, and the input information is calculated with the result of the fusion of the three branches. Finally, this module is added to the fifth stage of ResNet-101, as shown in Table 1.

[0049] Table 1 Improved ResNet-101 structure

[0050]

[0051] Furthermore, Transformer was first applied to natural language processing tasks and mainly consists of an encoder and a decoder. The encoder uses a multi-head attention mechanism to extract global features, and then uses a feedforward neural network to complete the transformation of the output dimension; in the decoder, it is mainly composed of a multi-head attention module and a cross-attention module, and the final output result is integrated through a feedforward neural network. The reason why the Transformer structure is used to solve the multi-label classification problem is that there are often multiple targets in an image, and these targets also have some potential dependencies. This requires the model to adaptively extract features for different targets and capture the semantic and spatial relationships between different labels. The cross-attention mechanism in the Transformer decoder structure can cross-calculate the input label category with the corresponding features, effectively extract local features, and learn the relationship between different labels [q2l,53]; secondly, the multi-head attention module in the Transformer can learn the features of a target from different angles. When the target is occluded or the perspective changes, the model can have better detection capabilities.

[0052] Specifically, given an input image x, assuming there are N categories, this embodiment represents the label corresponding to the input x as y=s[y1,…,y n ], where y n ∈{0,1}, if y n =1, indicating that the input x has the label of this category, otherwise y n = 0. The model predicts the probability of each category appearing p = [p1,…,p n ], where p n∈[0,1]. First, x is input into the TridentNet network for feature extraction, and the encoder is used to further learn the image feature F. Then, the label embedding is used as Q0 to input into the decoder. Through the cross attention module, the features related to the category are learned from the image features. Each decoder layer i updates T from the output of the previous layer. i-1 , specifically as follows:

[0053]

[0054] The wavy lines represent vectors modified by adding positional encodings. and are two intermediate variables. The multi-head attention module and the cross-attention module use the same structure. The main difference is the source of key and value. In the multi-head attention module, query, key, and value all come from label embedding, while in the cross-attention module, key and value all come from extracted image features. Each layer of Decoder input T i-1 , by inputting it into the cross-attention module for calculation together with the extracted feature F, each label embedding learns the features related to the category. After that, the label embedding will be updated layer by layer, and the relevant information from the image will be gradually learned through the cross-attention module.

[0055] like Figure 6 As shown, assuming that the model has L layers of decoders, in the last layer, the feature vector Q is obtained L , through linear projection and Sigmoid function, the features of each category can be projected into a probability matrix p n :

[0056] p n =Sigmoid(wQ L +b) (2)

[0057] Optionally, a key feature of multi-label classification is that when the total number of labels is large, there will be an imbalance between positive and negative samples, resulting in inaccurate classification. Most of the images in the dataset only contain a small part of the label class, which means that no matter which class, the number of positive samples is much smaller than the number of negative samples. Multi-label classification often uses a binary cross entropy function as a loss function, but this cannot solve the problem of imbalance between positive and negative samples. The traditional binary cross entropy loss function is as follows:

[0058]

[0059] Among them, y iRepresents the input image x i The real label, Represents the model's response to the input image x i The calculated predicted value. In order to solve the problem of imbalance between positive and negative samples, a simplified asymmetric loss function is adopted, which is a variant of Focal Loss and is defined as:

[0060]

[0061] in, Represents the predicted probability, γ represents the correction value. When γ is 0, it is equivalent to the binary cross entropy loss function. In order to further emphasize the importance of positive samples in the loss function, different γ values ​​are set for positive and negative samples to represent different correction degrees. Usually, γ is set to - >γ + , and further get formula (5):

[0062]

[0063] Substituting formula (5) into formula (3), we get the asymmetric loss function:

[0064]

[0065] Through the asymmetric loss function, the imbalance of positive and negative samples in the data set can be balanced as much as possible, so that the model can better learn the characteristics of positive samples in the data.

[0066] The MLIC task is more complex than the single-label image classification task. Different labels are not independent but have certain correlations. Therefore, more indicators are needed to evaluate the performance of the model.

[0067] The first is precision and recall. Precision refers to the proportion of real positive data among all data predicted as positive samples; recall refers to the proportion of data accurately predicted as positive samples by the model among all real positive data. The calculation formula is as follows:

[0068]

[0069] Among them, TP refers to the number of true positive samples predicted as positive samples, FP refers to the number of true negative samples predicted as positive samples, FN refers to the number of true samples predicted as negative samples, and TN refers to the number of true negative samples predicted as negative samples.

[0070] In addition, the average precision (AP) of each category and the average precision (mAP) of all categories are used for evaluation. AP means that for a single category, with Precision as the ordinate and Recall as the ordinate, a PR curve can be obtained, and the area enclosed by the PR curve is the AP of the category. The calculation formula is as follows:

[0071]

[0072] The mAP is the average value of each AP in the classification (meanAverage Precision), which can reflect the overall performance of the model. The calculation formula is as follows:

[0073]

[0074] N is the total number of label categories in the dataset, and AP(n) is the AP value of the nth category label.

[0075] In addition, the overall precision (OP), overall recall (OR), overall F1 value (OF1) and per-class precision (CP), per-class recall (CR), per-class F1 value (CF1) are also calculated. The larger the values ​​of these indicators, the better the performance of the model. c Represents the number of images predicted correctly for the nth category, n c Indicates the number of images predicted by the model as the nth category, n c Indicates the number of images with the true label n in the data. The specific calculation formula is as follows:

[0076]

[0077] For each image, if the probability predicted by the model is greater than the threshold, it is regarded as a positive sample, otherwise it is a negative sample. OF1 and CF1 are the main indicators because they can reflect the precision and query rate at the same time, which can more comprehensively reflect the performance of the model.

[0078] Specifically, the experimental details of this embodiment are as follows:

[0079] All experiments in this embodiment are implemented under Linux system through PyTorch design, and equipped with Nvidia A100 GPU for training and testing. According to the settings in DETR, sine and cosine encoding is used to represent the position information of the input, and the Adam optimizer is used to train 120 epochs, (β1, β2) = (0.9, 0.9999), the maximum learning rate is set to 1e-4, and a cutout factor of 0.5 and a weight decay of 1e-2 are used for regularization to suppress overfitting of the model. The image input is uniformly set to 448*448, and the ImageNet pre-trained model is used as the backbone to train on multi-label image classification related data sets.

[0080] MS-COCO dataset experiments:

[0081] As shown in Table 2, MS-COCO is the first large-scale dataset built for object detection and segmentation, and has been widely used to evaluate multi-label image classification in recent years. It contains 122,585 images and covers 80 common categories. On average, each image has 2.9 labels, of which 82,081 images are used for training and 40,504 images are used for testing.

[0082] Table 2 Experimental results on the MS-COCO dataset

[0083]

[0084]

[0085] The model proposed in this embodiment is compared with some mainstream methods. All methods use ResNet-101 as the backbone network for extracting image features. As can be seen from Table 2, the method proposed in this embodiment is improved by 6.6% compared with ResNet, 1.9% compared with ML-GCN, and 1.1% compared with MS-CMA, MCAR, SSGRL and other methods. In addition, key indicators such as CF1 and OF1 have also been improved, which is the best among all the compared methods.

[0086] PASCALVOC 2007 dataset experiments:

[0087] PASCALVOC 2007 is a widely used multi-label image classification dataset. This dataset can also be used for tasks such as object detection and semantic segmentation. Figure 7As shown in Figure 1, the PASCAL VOC 2007 dataset consists of 9963 images, including 5011 images as training sets and 4952 images as validation sets. All images are composed of 20 categories, which can be divided into four categories: "vehicle", "household", "animal", and "person". Each image contains at least one label.

[0088] To further verify the performance of the model proposed in this embodiment, it is compared with multiple methods on the PASCAL VOC 2007 dataset. All the compared methods use convolutional neural networks to extract features and combine other methods to model the correlation between labels. It can be seen from Table 3 that the model proposed in this embodiment achieves the highest mAP value, especially in some larger categories such as aeroplane, bus, and train in the image. Good results have been achieved, which fully demonstrates that the improved Resnet network in this embodiment can effectively improve the ability to learn and extract sample features by improving the receptive field of convolution, and verifies the effectiveness of the classification method.

[0089] Table 3 Experimental results on PASCAL VOC 2007 dataset

[0090]

[0091]

[0092] In the original Transformer architecture, the author used an architecture with multiple encoders and decoders stacked together. Considering that the feature extraction work is completed by the backbone part, the encoder plays a role in better feature fusion. The relationship between different backbones and the number of decoders and model performance was mainly experimentally explored on the MS-COCO dataset.

[0093] Table 4 Impact of the number of decoders on model performance

[0094]

[0095]

[0096] From Table 4, we can see that with the same number of encoders and decoders, when TridentNet is used as the backbone, the mAP of the model is higher than that of the architecture with ResNet-101 as the backbone. This result shows that the improvement of the convolution part of ResNet-101 in this embodiment effectively improves the model's ability to extract features. Secondly, the stacking of different numbers of decoders also has a significant impact on the final classification effect. Under the architecture with 3 decoders, the optimal mAP result is achieved.

[0097] Loss Experiment

[0098] A comparative experiment was conducted on the standard binary cross entropy loss function and the asymmetric loss function. As shown in Table 5, the results show that compared with the traditional cross entropy loss function, the improved asymmetric loss function improves the accuracy of the model by nearly 4%, and can effectively suppress the impact of a large number of negative samples.

[0099] Table 5 Impact of different losses on model performance

[0100]

[0101]

[0102] Optionally, how to efficiently extract image features and fully explore the potential relationship between different categories is a key research direction of multi-label image classification tasks. This embodiment proposes a multi-label image classification method based on the fusion of improved ResNet and Transformer encoder-decoder architecture. This method uses the ResNet-101 model to extract features from images and uses the cross-attention mechanism of the Transformer decoder to learn the relationship between different images and labels. Through experimental comparison with multiple methods on the MS-COCO and PASCAL VOC 2007 datasets, the method proposed in this embodiment has achieved good results in multiple indicators, verifying the effectiveness of the method.

[0103] As an optional implementation, Figure 8 and Fig. 9 As shown, the present invention aims to protect a multi-label image classification method based on an improved residual network, the main steps of which are as follows:

[0104] Step 1: Get the image to be classified

[0105] Step 2: Build a multi-label image classification model, which consists of a Resnet-based feature extraction network and a Transformer structure.

[0106] (1) Image global feature extraction: Use deep convolutional neural network to extract the global features of the image:

[0107] Initialize the structure and parameters of the Resnet-101 model. The parameters here refer to the weight data obtained by training the Resnet-101 model on the MS-COCO large-scale visual dataset. Send the image to be classified into the model, and after the five convolutional layers Conv1, Con2_x, Con3_x, Con4_x, and Con5_x are calculated, the global features of the image are obtained. Among them, the Con5_x layer is improved, and the bottleneck convolution block is replaced by 3 parallel dilated convolution structures, with dilation rates of 1, 2, and 3 respectively, so that features of different scales can be extracted in a targeted manner.

[0108] (2) Use the encoder to further extract features based on the multi-head attention module and integrate the output dimensions as follows:

[0109] ① Flatten the global feature matrix of the image obtained in step 2 above into a one-dimensional vector, denoted as F;

[0110] ② Initialize the weight matrix W Q , W E , W V ;

[0111] ③ Combine the feature matrix F described in ① and the weight matrix W described in ② Q , W E , W V Multiply them together and you get Q F , K F 、V F Matrix, the specific calculation method is:

[0112]

[0113] (3) Using the decoder, based on the multi-head attention and cross-attention modules, the input label category is cross-calculated with the corresponding features as follows:

[0114] ①The label vector of the image is recorded as E, and its corresponding feature matrix is ​​Q E , K E 、V E matrix;

[0115] ②Use the label vector of the image to calculate using the standard Multihead() method of the PyTorch library, and obtain the result T1 through the multi-head attention module. The calculation method can be expressed as:

[0116] T1=MultiHead(Q E ,K E ,VE )

[0117] ③T1 and K F , K F The matrix is ​​calculated through the cross attention module to obtain the result T2. The calculation method can be expressed as:

[0118] T2=MultiHead(T1,K F ,V F )

[0119] (4) The feature values ​​after cross calculation are calculated through the fully connected layer and the sigmoid activation function to obtain the final classification result. The method is as follows:

[0120] The T2 matrix is ​​input into a fully connected network for calculation, and then passes through the Sigmoid activation function to finally generate a vector for each image with a dimension equal to the total number of categories to be classified. The value of each dimension of the vector represents the probability that the image to be detected belongs to each corresponding category. The specific calculation method is defined as follows:

[0121] p n =Sigmoid(wT2+b)

[0122] Step 3: Train the model in step 2 using the data in step 1

[0123] Classification result description: This classification method can theoretically classify any type of image. The classification result depends on how many categories are defined in the training dataset. Take the commonly used MS-COCO dataset as an example. This dataset defines 80 categories, including [people, motorcycles, cars, ..., hair dryers, toothbrushes]. After the above model is trained with the data from this dataset, you can input pictures for prediction.

[0124] Enter a picture like Fig.10 For the image shown in the figure, after calculation, the output result should be pn=[0.98, 0.75, 0.42, …, 0.02, 0.01]. Each value represents the probability of each category appearing in the image. Usually 0.5 is set as the threshold. If the result is greater than 0.5, it is considered that the image has the features of the category. By mapping the calculated pn with the corresponding category, we can get the classification result ["person", "motorcycle"].

[0125] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0126] In this embodiment, specific examples are used to illustrate the principle and implementation of the present invention. The above description of the embodiment is only used to help understand the method and core idea of ​​the present invention; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-label image classification method based on deep learning, characterized in that: include: Get an image to be classified with multiple labels; Inputting the image to be classified into a trained multi-label image classification model to obtain a classification result; The multi-label image classification model includes a feature extraction network and a Transformer structure connected to the feature extraction network; the Transformer structure consists of an encoder and a decoder; the encoder is used to extract features and integrate output dimensions; The decoder is used to perform cross calculation on the input label category and the corresponding feature; The feature extraction step of the feature extraction network includes: Initializing the structure and parameters of the feature extraction network; the feature extraction network is an improved Resnet-101 model; the improved Resnet-101 model uses the Trident module as a residual block, adds and fuses the calculation results of the three branches in the Trident module with the input information of the Trident module, and the obtained result is used as the output information of the Trident module; The image to be classified is calculated using five convolutional layers to obtain the global features of the image to be classified; wherein the bottleneck convolution block of the fifth convolutional layer is replaced by three parallel dilated convolution structures, and the dilation rates are 1, 2, and 3 respectively; the convolution kernels of the three parallel dilated convolution structures are: The loss function of the multi-label image classification model is an asymmetric loss function; the asymmetric loss function is: Where L is the asymmetric loss; N is the number of input images; L i is the asymmetric loss of the i-th input image; y i is the true label of the input image; Represents the predicted value calculated by the model for the input image; γ + is the degree of correction of the positive sample; - is the degree of correction of negative samples.

2. The multi-label image classification method based on deep learning according to claim 1, characterized in that: The parameters are weight data obtained by training the Resnet-101 model on the MS-COCO large-scale visual dataset.

3. The multi-label image classification method based on deep learning according to claim 1, characterized in that: The steps of extracting features and integrating output dimensions of the encoder are as follows: Flatten the feature matrix formed by the global features into a one-dimensional vector, denoted as F; Initialize the weight matrix W Q , W E , W V ; The feature matrix F and the weight matrix W Q , W E , W V Multiply them and we get Q F , K F 、V F Matrix, the specific calculation method is:

4. The multi-label image classification method based on deep learning according to claim 3 is characterized in that: The decoder performs cross calculations on the input label category and the corresponding features as follows: The label vector of the image is denoted as E; the feature matrix corresponding to the label vector is Q E , K E 、V E matrix; The label vector is calculated using the standard Multihead() method of the PyTorch library, and the result T1 is obtained through the multi-head attention module. The calculation method of the result T1 is expressed as: T1 = MulitHead(Q E ,K E ,V E ); T1 and K F 、V F The matrix is ​​calculated through the cross attention module to obtain the result T2. The calculation method of the result T2 is expressed as: T2 = MulitHead (T1, K F ,V F ).

5. The multi-label image classification method based on deep learning according to claim 1, characterized in that: The multi-label image classification model also includes a fully connected layer and a sigmoid activation function; the fully connected layer is used to calculate the result T2, and then pass through the sigmoid activation function to finally generate a vector with a dimension equal to the total number of categories to be classified for each image, and the value of each dimension of the vector represents the probability p that the image to be detected belongs to each corresponding category n ; probability p n The calculation formula is: n =Sigmoid(wT2+b); where w is the weight coefficient and b is the intercept.

Images