A multi-label image classification method based on two-dimensional dependency

Abstract

The application relates to a multi-label image classification method based on two-dimensional dependency, which comprises the following steps: receiving an image to be classified; inputting the image to be classified into a two-dimensional dependency model to obtain an image classification result; wherein the two-dimensional dependency model comprises a spatial feature dependency module and a label semantic dependency module; the spatial feature dependency module comprises a feature extraction part, a feature fusion part, a feature enhancement part and an output part; and the label semantic dependency module comprises a GCN network part and a global relationship enhancement part. The application solves the problems of limited fixed convolution kernel receptive field and insufficient label relationship extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-label image classification, and particularly relates to a multi-label image classification method based on two-dimensional dependency. BACKGROUND

[0002] The multi-label image classification problem is a task of classifying images into multiple related labels or categories. It is different from the traditional single-label image classification task, which only needs to assign a main label or category to an image. The multi-label image classification task can be used in social media management, product identification and recommendation, medical image analysis, automatic driving and other fields. Due to the fundamentality and importance of the multi-label image classification task, research on it is also increasing.

[0003] Since the objects in the image always appear at the same time, a key research direction of the multi-label image classification problem is to establish the dependency between the objects. This dependency includes the feature dependency in the image space and the dependency in the label semantics. The algorithm needs to establish the dependency in these two dimensions respectively, and combine the dependency in these two dimensions to more efficiently complete the classification task.

[0004] Regarding the establishment of spatial dependency between objects, the most commonly used is the CNN-based backbone network. However, due to the limitation of the size of the convolution kernel, the perception range of the CNN network is limited. The limited receptive field will cause the establishment of spatial dependency to be significantly affected by the pixel distance between objects. Therefore, it is difficult to establish spatial dependency between objects that are far apart. Regarding the establishment of label semantic dependency, GCN is the most commonly used network. However, the use of first-order neighborhood aggregation in the standard GCN formula leads to limited expressive ability of GCN, lacking of global information integration. Similar to the problem of limited CNN receptive field, GCN is difficult to extract global information between label semantics. For example, according to human common sense, the probability of the simultaneous appearance of "dog" and "fork" in an image is very low, but when there is "person", the probability of the simultaneous appearance of "dog" and "fork" will be greatly increased. However, if the algorithm cannot capture the global dependency relationship between label semantics well, the algorithm will still consider that the probability of the simultaneous appearance of "dog" and "fork" is very low even if there is "person".

[0005] Therefore, the existing methods have certain limitations in modeling label semantic dependency. In summary, the existing algorithms have the following two problems:

[0006] (1) Due to the limitation of the size of the convolution kernel, the receptive field of the CNN network is limited. It is difficult to establish effective long-distance spatial dependency.

[0007] (2) GCN uses first-order neighborhood aggregation, which results in limited expression ability of GCN and lack of global information integration. It is difficult to establish global label semantic dependency. SUMMARY

[0008] The application provides a multi-label image classification method based on two-dimensional dependency, which can solve the problems of limited receptive field of fixed convolution kernel and insufficient label relationship extraction.

[0009] The technical scheme adopted by the application to solve the technical problems is to provide a multi-label image classification method based on two-dimensional dependency, comprising the following steps:

[0010] receiving an image to be classified;

[0011] inputting the image to be classified into a two-dimensional dependency model to obtain an image classification result;

[0012] The two-dimensional dependency model comprises a spatial feature dependency module and a label semantic dependency module.

[0013] The spatial feature dependency module comprises:

[0014] a feature extraction part for extracting feature information based on a ResNet network;

[0015] a feature fusion part for fusing feature information output by different layers of the ResNet network to obtain a fusion result;

[0016] a feature enhancement part for performing feature enhancement on the output of the last layer of the ResNet network to obtain an enhanced result;

[0017] an output part for merging and outputting the fusion result and the enhanced result;

[0018] The label semantic dependency module comprises:

[0019] a GCN network part for extracting relationships between labels;

[0020] a global relationship enhancement part for enhancing the extracted relationships between labels.

[0021] The feature fusion part takes the feature tensors extracted by the 3rd layer and the 4th layer of the ResNet network as input, and comprises:

[0022] a convolution unit for performing one layer of convolution on the feature tensor extracted by the 3rd layer of the ResNet network to obtain u1, and performing two layers of convolution on the feature tensor extracted by the 4th layer of the ResNet network to obtain u2 and u3;

[0023] A first interpolation operation unit is configured to perform bilinear interpolation operation on u1, u2 and u3 to obtain u1', u2' and u3';

[0024] An information fusion unit is configured to add u1', u2' and u3' to obtain a fusion factor, and add the fusion factor to u1', u2' and u3' to obtain new u1', u2' and u3';

[0025] An information compensation unit is configured to multiply new u1', u2' and u3' to obtain a compensation factor, and multiply the compensation factor by new u3' to complete information compensation to obtain u3' after completing information compensation;

[0026] A second interpolation operation unit is configured to perform interpolation operation on new u1' and u2', and u3' after completing information compensation to restore the fused feature tensor to the original dimension to obtain U1, U2 and U3;

[0027] A max-pooling unit is configured to perform global max-pooling operation on U1, U2 and U3 respectively;

[0028] A concatenation unit is configured to concatenate U1, U2 and U3 after max-pooling operation to obtain a concatenated feature tensor;

[0029] A full connection unit is configured to perform full connection layer operation on the concatenated feature tensor.

[0030] The output of the feature enhancement part is represented as f FEM =(f 1 ,f 2 ,...,f C ), wherein C represents the number of required output label categories, f i represents the global enhancement attention of the i-th category, represents the global average attention, A i represents the feature attention, represents the attention weight coefficient of the i-th category and the j-th category, C i () is the classifier of the i-th category, T is a control proportion coefficient, x n represents the output of the last layer of the ResNet network, ω i represents the fusion attention, α and β are proportion weight coefficients.

[0031] The GCN network part constructs a correlation matrix in the form of conditional probability, which is specifically:

[0032] Count the occurrence of label pairs in the training set to obtain a matrix D;

[0033] Derive a conditional probability matrix based on the matrix D, wherein P ij = D ij / N j , D ij represents the number of times that label i and label j appear simultaneously, N j represents the total number of times that label j appears in the training set, P ij represents the probability that label i appears when label j exists;

[0034] Filter out noise using a threshold τ on the conditional probability matrix to obtain a correlation matrix, denoted as

[0035] Perform weighted processing on the obtained correlation matrix to obtain p is the assigned weight.

[0036] The global relationship enhancement part includes a plurality of maximum attention heads, and the operation results of each maximum attention head are spliced, wherein a Query tensor, a Key tensor and a Value tensor are generated in each maximum attention head, a maximum pooling operation is performed on the Query tensor, and then the original dimension is restored through a copy operation; the processed tensor is added to a standard self-attention mechanism.

[0037] Advantages

[0038] Compared with the prior art, the present application has the following advantages and positive effects: by using the ResNet network to extract feature maps in layers, and through cross-layer fusion and feature enhancement, the present application can emphasize high-dimensional abstract information while fusing the feature information of targets in different spatial positions, thereby effectively establishing long-distance dependencies between targets and alleviating the problem of limited receptive field of fixed convolution kernels. Based on the multi-head maximum attention mechanism, the present application performs a maximum pooling operation on the Query tensor based on the self-attention mechanism, thereby improving the global enhancement effect and solving the problem of insufficient label relationship extraction, and establishing the global semantic dependency relationship between labels. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a structural schematic diagram of a two-dimensional dependency model in an embodiment of the present application;

[0040] Figure 2 is a structural schematic diagram of a feature fusion part in an embodiment of the present application;

[0041] Figure 3 is a structural schematic diagram of a global relationship enhancement part in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The application will be further described below in connection with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. Furthermore, it should be understood that after reading the content of the application, those skilled in the art can make various modifications or changes to the application, and these equivalent forms also fall within the scope defined by the appended claims.

[0043] Embodiments of the application relate to a multi-label image classification method based on two-dimensional dependency, comprising the following steps: receiving an image to be classified; inputting the image to be classified into a two-dimensional dependency model to obtain an image classification result. As shown in the figure, the two-dimensional dependency model proposed in the embodiment adopts a double-branch network structure, and the model comprises a spatial feature dependency module (SFDM) and a label semantic dependency module (LSDM), which are respectively used to establish spatial feature dependency and global label semantic dependency. Figure 1

[0044] The SFDM uses a ResNet network as a backbone network, and cooperates with a proposed feature fusion module (FFM) and a feature enhancement module (FEM) to realize the establishment of spatial feature dependency. Specifically as follows:

[0045] ResNet is used as a backbone network in the SFDM, and it is assumed that the resolution of the input image I is 448x448. Feature maps X3∈R 1024×28×28 and X4∈R 2048×14×14 are obtained from Layer 3 and Layer 4 of the ResNet network respectively.

[0046] X3 passes through a convolutional layer to generate a tensor u1. X4 passes through two convolutional layers to generate tensors u2 and u3 respectively. u1, u2 and u3 are input into the FFM, and the output X4 of Layer 4 in the ResNet network is input into the FEM. The fusion output of the FEM and the FFM is taken as the output of the SFDM. After the above operation, the output feature tensor of the SFDM contains long-distance spatial dependency. The output of the SFDM is represented as follows:

[0047] X SFDM =f FFM (f L3 (I,θ cnn ),f L4 (I,θ cnn ))+f FEM (f L4 (I,θ cnn )) (1)

[0048] Where θ CNN is a CNN parameter, and f L3 ​() represents the output of Layer 3, f L4 () represents the output of Layer 4. f FFM () represents the output of FFM. f FEM () represents the output of FEM.

[0049] The role of FFM is to perform cross-layer feature fusion to establish long-distance spatial dependencies. The FFM structure is shown in Figure 2 . Since the shallow layers of ResNet contain less abstract information and often have redundant features, the output feature tensors X3 and X4 are extracted from Layer 3 and Layer 4 as inputs for fusion. X3 is generated through a convolutional layer to generate u1∈R 512×28×28 , and X4 is generated through two convolutional layers to generate u2∈R 512×14×14 and u3∈R 512×7×7 , respectively. The purpose of using two convolutional layers is to introduce scale transformation on the highly abstract feature maps of the last layer to achieve better cross-scale fusion effect and establish more effective dependencies. u1, u2, and u3 are unified in dimension through bilinear interpolation operation to obtain u1', u2', and u3'. Then these tensors are added as feature fusion factors, denoted as Bf. Subsequently, Bf is added to u1', u2', and u3', respectively, to achieve information fusion between them, thereby obtaining new u1', u2', and u3'. For the highly abstract new u3', an information compensation step is needed. The compensation method is to multiply the new u1', u2', and u3', and then multiply the result by a compensation coefficient, denoted as Cf. Finally, the compensation factor Cf is multiplied by the new u3' to complete the information compensation, thus completing the cross-layer information fusion. Through interpolation, the new u1', u2', and compensated u3' after fusing information with each other are restored to the original dimension, denoted as U1, U2, and U3, respectively. Subsequently, global max-pooling operations are performed on each tensor to convert their dimensions to [B, 512], where B represents the batch size. Then the three feature tensors are concatenated to obtain a feature tensor with dimension [B, 3x512]. Finally, the feature tensor passes through a fully connected layer to obtain a feature tensor with dimension [B, C], where C represents the number of desired output classes. The output of FFM can be represented as follows:

[0050] f FFM = ξ GCL (Ψ(f L3 (I, θ cnn ), f L4 (I, θ cnn ))) (2)

[0051] where Ψ() represents the convolution, interpolation, and feature fusion operations on X3 and X4. ξ GCLrepresents the global maximum pooling, concatenation operation and fully connected layer operation of the feature vector.

[0052] The information of Layer4 output in ResNet is highly abstract. In order to enhance these feature representations, FEM is designed. FEM is used to enhance the highly abstract image information of the output of the last layer of ResNet, and the calculation process of the module includes calculating attention weight coefficient, global average attention, feature attention, fusion attention and global enhancement attention.

[0053] The output of Layer4 is X4∈R 2048×14×14 , which can be represented as x1, x2, …, x 196 (x i ∈R 2048 ). First, the calculation formula of the attention weight coefficient of the i-th and j-th categories is defined as follows:

[0054]

[0055] Where C i () is the i-th classifier, and T is the control proportion coefficient.

[0056] For general features, the definition of global average attention is as follows:

[0057]

[0058] In order to enhance the abstract features, the calculation of feature attention is defined as follows (which can be regarded as a kind of attention mechanism for specific categories, which focuses on the classification scores of different positions in different categories):

[0059]

[0060] In order to better enhance the features, the calculation of fusion attention is defined as follows:

[0061]

[0062] So far, the global enhancement attention of the i-th category has been obtained:

[0063]

[0064] Where, α and β are proportion weight coefficients.

[0065] The final output of FEM is as follows:

[0066] f FEM =(f 1 ,f 2 ,…,f C ) (8)

[0067] where C denotes the number of desired output label categories.

[0068] The LSDM in this embodiment uses a GCN network as a feature extraction network, and combines a global relationship enhancement module (GREM) to enhance the global label semantic dependency.

[0069] GCN networks are a research direction that has developed rapidly in recent years. CNNs can only handle problems with Euclidean data, while GCN networks are suitable for more general data structures. By considering the directional relationship between nodes, GCN networks can effectively capture complex data structures. Messages can be propagated between nodes, and node representations will be updated after message passing. The goal of a GCN network is to learn a function f() on a graph structure, which can update the representation of each node. The function f() takes the feature description F l ∈R n×d and the adjacency matrix A ∈ R n×n as input, where n denotes the number of nodes and d denotes the dimension of the node features. Each layer of nodes can be represented as follows:

[0070] F l+1 = f(F l , A) (9)

[0071] After applying the convolution operation, it can be represented as follows:

[0072]

[0073] where is a normalized version of the adjacency matrix A, F l denotes the node information state of the l-th layer, W l ∈R d×d' denotes a learnable transformation parameter, and h() denotes an activation function. By stacking multiple GCN layers using the above equation, complex relationships between nodes can be learned.

[0074] The correlation matrix describes the correlation between nodes and guides the information propagation between nodes in the GCN. Typically, the correlation matrix is predefined. In this embodiment, a data-driven method is used to construct the correlation matrix, i.e., the correlation matrix is constructed in the form of conditional probability.

[0075] First, the occurrence of label pairs in the training set is calculated to obtain a matrix D ∈ R C×C , where C denotes the number of label categories, and D ij denotes the number of times that label i and label j appear together. Based on the matrix D, the conditional probability matrix P can be derived as follows:

[0076] P ij = D ij / Nj (11)

[0077] where N j denotes the total number of occurrences of label j in the training set, P ij = P(L i |L j ) denotes the probability of label i occurring when label j exists.

[0078] The above method of obtaining the correlation matrix can encounter two problems. First, the data can exhibit a long-tail distribution. Second, the correlation matrix calculated from the training set can have overfitting problems, affecting its generalization ability. To solve these problems, consider binarizing the correlation P and filtering out noise using a threshold. Therefore, the correlation matrix can be represented as follows:

[0079]

[0080] However, binarization can cause over-smoothing problems. To alleviate this situation, a re-weighting scheme can be adopted, represented as follows:

[0081]

[0082] where A' denotes the re-weighted correlation matrix, and p denotes the assigned weight set. Through the re-weighting operation, the re-weighted correlation matrix retains both the information of individual nodes and the information from neighboring nodes, effectively solving the over-smoothing problem.

[0083] The information representing the graph structure can be propagated between nodes in the GCN. Since GCN uses first-order neighborhood aggregation, the expressive power of GCN is limited, lacking global information integration, which leads to difficulties in modeling global label semantic dependency relationships. To solve this problem, the present embodiment proposes GREM, as shown in Figure 3 GREM consists of multiple max attention heads. The information processed by GCN will be divided into multiple heads, and within each max attention head, Query, Key, and Value tensors will be generated from the feature tensor. Specifically, a max pooling operation is performed on the Query tensor, and then the original dimension is restored through the Repeat operation. The processed tensor is added to the standard self-attention mechanism. Finally, the results of each head operation are spliced to restore the input dimension. Using the attention mechanism can enhance the global relationship between GCN nodes, and by performing max pooling and Repeat operations on the Query tensor, global features can be enhanced. Therefore, through GREM, global dependencies between label semantics can be established. The calculation within each head of GREM can be represented by the following formula:

[0084]

[0085] wherein, Rp() represents a repeated operation, and MP() represents a max pooling operation. The outputs of each attention head are spliced and then fused using a Wo matrix to obtain the output of the GREM as follows:

[0086] f GREM =Concet(Max_Atten1,…,Max_Attenh)W o (15)

[0087] The overall representation of the LRDM can be represented as follows:

[0088] X LRDM =f GCN (N,θ gcn )×f L4 (I,θ cnn )+f GREM (f GCN (N,θ gcn ))×f L4 (I,θ cnn ) (16)

[0089] To verify the effectiveness of the present embodiment, a model is built based on the Pytorch platform and experiments are conducted on the PASCAL VOC 2007 dataset.

[0090] Experimental data: The VOC2007 dataset is one of the widely used benchmark datasets in computer vision and object recognition research. The dataset is composed of images from 20 different classes, including people, cars, animals, and common objects, etc. Each image is labeled with a class label of the object, and the dataset consists of a training set (5011 images) and a test set (4952 images), including a total of 9963 images across 20 classes. The VOC2007 dataset provides a standard benchmark for researchers to train and evaluate object detection and image classification algorithms.

[0091] Experimental details: In the SFDM, ResNet-101 is used as the backbone network, which is pre-trained on the ImageNet dataset. For the LRDM, the GCN consists of two layers with output dimensions of 1024 and 2048, respectively. During training, the input images are randomly cropped and adjusted to 448x448, and then randomly horizontally flipped. In terms of network optimization, SGD is used as the optimizer with a momentum of 0.9 and a weight decay of 10 -4 . The initial learning rate is set to 0.01, and the network is trained for a total of 100 epochs. In addition, a warm-up scheduler is used to train the model, which completes the warm-up phase within 2 epochs.

[0092] Experimental results:

[0093] The experimental results are shown in Table 1, and the present embodiment is compared with the prior art (based on the PASCAL VOC 2007 dataset).

[0094]

[0095]

[0096] Table 1

[0097] It can be found that, by using the ResNet network to hierarchically extract feature maps, and by cross-layer fusion and feature enhancement, the present application can emphasize high-dimensional abstract information while fusing the feature information of the target in different spatial positions, thereby effectively establishing long-distance dependencies between targets and relieving the problem of limited receptive field of fixed convolution kernels. Based on the multi-head maximum attention mechanism, the present application performs maximum pooling operation on the Query tensor on the basis of the self-attention mechanism, thereby improving the global enhancement effect and solving the problem of insufficient label relationship extraction, and establishing global semantic dependency relationships between labels.

Claims

1. A multi-label image classification method based on two-dimensional dependency, characterized by, The method comprises the following steps: receiving an image to be classified; inputting the image to be classified into a two-dimensional dependency model to obtain an image classification result; wherein the two-dimensional dependency model comprises a spatial feature dependency module and a label semantic dependency module; the spatial feature dependency module comprises: a feature extraction part for extracting feature information based on a ResNet network; a feature fusion part for fusing feature information output by different layers of the ResNet network to obtain a fusion result; a feature enhancement part for performing feature enhancement on the output of the last layer of the ResNet network to obtain an enhanced result; an output part for merging and outputting the fusion result and the enhanced result; the label semantic dependency module comprises: a GCN network part for extracting relationships between labels; a global relationship enhancement part for enhancing the extracted relationships between labels; the global relationship enhancement part comprises a plurality of maximum attention heads, and the operation results of each maximum attention head are spliced, wherein a Query tensor, a Key tensor and a Value tensor are generated in each maximum attention head, and a maximum pooling operation is performed on the Query tensor, and then the original dimension is restored through a copying operation; the processed tensors are added to a standard self-attention mechanism. 2.The multi-label image classification method based on two-dimensional dependency according to claim 1, wherein, The feature fusion part takes the feature tensors extracted by the 3rd layer and the 4th layer of the ResNet network as input, and comprises: a convolution unit for performing one-layer convolution processing on the feature tensor extracted by the 3rd layer of the ResNet network to obtain u1, and performing two-layer convolution processing on the feature tensor extracted by the 4th layer of the ResNet network to obtain u2 and u3; a first interpolation operation unit for performing bilinear interpolation operation on u1, u2 and u3 to obtain u1', u2' and u3'; an information fusion unit for adding u1', u2' and u3' to obtain a fusion factor, and adding the fusion factor to u1', u2' and u3' respectively to obtain new u1', u2' and u3'; an information compensation unit for multiplying new u1', u2' and u3' and then multiplying by a compensation coefficient to obtain a compensation factor, and multiplying the compensation factor by new u3' to complete information compensation to obtain u3' after completing information compensation; a second interpolation operation unit for performing interpolation operation on new u1' and u2', and u3' after completing information compensation, restoring the fused feature tensor to the original dimension to obtain U1, U2 and U3; a maximum pooling unit for performing global maximum pooling operation on U1, U2 and U3 respectively; a splicing unit for splicing U1, U2 and U3 after maximum pooling operation to obtain a spliced feature tensor; a fully connected unit for performing fully connected layer operation on the spliced feature tensor. 3.The multi-label image classification method based on two-dimensional dependency according to claim 1, wherein, The output of the feature enhancement part is represented as wherein, represents the number of required output label categories, , represents the global enhancement attention of the first class, represents the global average attention, , represents the feature attention, , represents the attention weight coefficient of the first class and the first class, , is a classifier of the first class, is a control proportion coefficient, represents the output of the last layer of the ResNet network, represents the fusion attention, , and are proportion weight coefficients. 4.The multi-label image classification method based on two-dimensional dependency according to claim 1, wherein, The GCN network part constructs a correlation matrix in the form of conditional probability, specifically: The occurrence of the pair of labels in the training set is calculated to obtain a matrix ; Based on matrix The conditional probability matrix is derived, where, , denotes the label and the label occurs simultaneously, denotes the label the total number of occurrences in the training set, denotes the probability that the label occurs when the label is present; using a threshold value on the conditional probability matrix filtering out noise, resulting in a correlation matrix represented as ; The correlation matrix obtained is weighted to obtain , is the assigned weight.

Images