Multi-label image classification method based on hypergraph Transform

Abstract

The invention discloses a multi-label image classification method based on a hypergraph Transform, and compared with the prior art, the multi-label image classification method based on the hypergraph Transform has the following beneficial effects: (1) a multi-label image classification model based on the hypergraph Transform is constructed, and the model shows stronger anti-noise capability and generalization when a complex relationship between vertexes is mined, so that the classification accuracy is improved; (2) a network model of the adaptive hypergraph Transform is provided, the model defines a hypergraph incidence matrix of image content perception, and a hypergraph incidence relation between vertexes is constructed through an attention mechanism, so that the robustness of the model is enhanced;

Description

Technical Field

[0001] The present invention belongs to the field of image classification in computer vision, and in particular to multi-label prediction of two-dimensional images by a hypergraph Transformer in a multi-label scenario. Background Art

[0002] Multi-label image classification is an important task in computer vision, involving many applications such as scene recognition and automatic image annotation. Its purpose is to extract features from a two-dimensional image containing multiple categories of targets and predict the category label corresponding to the image. Compared with single-label image classification, multi-label image classification is more difficult. Not only does it require the simultaneous recognition of multiple targets, but it also faces possible problems such as inconsistent sizes between targets and occlusion of targets. To address these problems, existing methods mainly consider enhancing the model's feature extraction capabilities, or improving recognition accuracy by mining the associations between targets.

[0003] Convolutional Neural Network (CNN) is widely used in multi-label image classification tasks due to its powerful feature extraction capabilities. For example, the residual network (ResNet) solves the problem of gradient vanishing in deep neural networks through residual connections, and is still one of the most commonly used feature extraction networks. However, due to the limitation of the local receptive field, ResNet cannot directly establish long-distance dependencies between pixels in the image, and its ability to capture global contextual information in the image is poor. In recent years, Vision Transformer and Swin Transformer have migrated the attention mechanism in the field of natural language processing (NLP) to image classification. As a result, the deep learning model can establish long-distance dependencies in the image under the influence of the attention mechanism, complete the focus on the local important areas of the image, and thus improve the recognition accuracy. Although the above method can improve the feature extraction ability of the deep learning model, it ignores the potential connection between multiple targets in the image.

[0004] Graph Neural Network (GNN) is mainly used to learn the label co-occurrence relationship or semantic association between category labels in multi-label image classification. By expressing the relationship between labels as a graph structure, the model's ability to capture target associations is improved. However, since the graph structure can only construct a one-to-one connection between vertices, this connection does not conform to the complex relationship between real objects in the real world. Therefore, hypergraphs that can simultaneously connect multiple vertices through hyperedges are introduced into the field of multi-label image classification. For example, the adaptive hypergraph neural network (AdaHGNN) learns the high-order correlation between data by constructing an adaptive association matrix. Compared with the general hypergraph that uses prior knowledge to initialize the association matrix, the learnable association matrix in AdaHGNN can be dynamically updated during the training process of the model, and information transfer is more flexible. Most of the above methods require extracting features corresponding to all the labels to be predicted from the image, and using the features of each type of label as the vertices of the graph structure or hypergraph. But in fact, the labels carried by a single image are limited. If the features corresponding to all labels are directly extracted from the image, then some of these label features must be noise features. Therefore, even if the hypergraph meets the learning requirements of multi-label image classification for high-order correlations between targets, it is still necessary to further consider the importance of different vertex features in the hypergraph to reduce the impact of noise features. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a multi-label image classification method based on a hypergraph Transformer to solve the above technical problems.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides the following technical solution: a multi-label image classification method based on a hypergraph Transformer, comprising the following steps:

[0009] S1: Extract image features using feature extraction network;

[0010] S2: Decouple the extracted image features using the semantic decoupling model to obtain the semantic features of all possible classes;

[0011] S3: Build a hypergraph Transformer model to jointly optimize the decoupled semantic features;

[0012] S4: Perform multi-label classification on the image based on the optimized features.

[0013] Preferably, the image feature extraction module is a convolutional neural network, and step S1 specifically utilizes the residual network of the convolutional neural network to extract features from the image.

[0014] Preferably, the semantic feature decoupling module is a semantic feature interaction network, and step S2 specifically utilizes a low-rank bilinear pooling operation to decouple the target semantic features of the image.

[0015] Preferably, the semantic feature optimization module is a hypergraph Transformer network, and step S3 is to use the hypergraph Transformer to construct the co-occurrence relationship between tags to optimize the target semantic features obtained in step S2.

[0016] Preferably, step S3 includes the following sub-steps:

[0017] S31: Combining the target semantic features obtained in step S2 with the attention mechanism, weighting the association matrix of the hypergraph to calculate the hyperedge features;

[0018] S32: Combining the hyperedge features calculated in step S31 with the attention mechanism, weighting the association matrix of the hypergraph to update the vertex features.

[0019] Preferably, step S4 is to classify the target semantic features optimized by the hypergraph Transformer network into image labels through a linear layer.

[0020] Preferably, a cross entropy loss function is used as a constraint when training the entire model for multi-label image classification based on the hypergraph Transformer.

[0021] (III) Beneficial effects

[0022] Compared with the prior art, the present invention provides a multi-label image classification method based on a hypergraph Transformer, which has the following beneficial effects: (1) A multi-label image classification model based on a hypergraph Transformer is constructed, which exhibits stronger noise resistance and generalization when mining complex relationships between vertices, thereby improving the classification accuracy; (2) An adaptive hypergraph Transformer network model is proposed, which defines an image content-aware hypergraph association matrix and constructs hypergraph association relationships between vertices through an attention mechanism, thereby enhancing the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of the steps of the multi-label image classification method based on the hypergraph Transformer of the present invention;

[0024] Figure 2 This is a sub-step flowchart of the present invention using the hypergraph Transformer to optimize semantic features. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The present invention provides a multi-label image classification method based on a hypergraph Transformer, comprising the following steps:

[0027] S1: Data preprocessing

[0028] Preferably, during the training phase of the model, we first resize the input image to 640×640; then in random cropping, randomly select a value from {640,576,512,384,320} as the width and height of the cropped image; finally, resize the cropped image to 576×576. During the testing phase of the model, we only resize the input image to 640×640 for verification and do nothing else.

[0029] S2 feature extraction and semantic decoupling

[0030] Preferably, the feature extraction module of the present method adopts the ResNet-101 model pre-trained on the ImageNet dataset. After the image I after data preprocessing is input into the feature extraction module, the output f of the fourth stage of ResNet-101 is used. image ∈w×h×C1 as the feature map, where w and h are the width and height of the feature map respectively. C1 is the number of channels of the feature map, which is 2048. Then for each category label c, the pre-trained GloVe model is used to obtain d s dimensional word embedding vector f word , as shown in formula (1). c is the semantic word of category label c, f g (·) is the pre-trained GloVe model.

[0031] f word =f g (w c ) (1)

[0032] In the semantic decoupling module, for each position (w, h) in the feature map, the corresponding image feature f is fused through a low-rank bilinear pooling operation image With the word embedding vector f word , as shown in formula (2). Then at f wordUnder the guidance of , an attention coefficient is calculated, as shown in formula (3), which represents the importance of the position (w, h) in the feature map. After calculating the attention coefficients for all positions in the feature map, the softmax function is used to normalize all positions, as shown in formula (4).

[0033]

[0034] Among them, tanh(·) is the hyperbolic tangent function, f a (·) is the attention function implemented by the fully connected layer. U, V, P, b are learnable parameters, and ⊙ represents the dot product operation.

[0035] Finally, weighted average pooling is performed on all positions of the feature map to obtain the semantic feature vector f that encodes information about the category label c sd ={f0,f1,...,f C-1}, as shown in formula (5).

[0036] f sd =∑ w,h a c,wh f c,wh (5)

[0037] S3 feature optimization

[0038] Preferably, given the features of all vertices And the incidence matrix of the hypergraph A l-layer hypergraph Transformer can capture high-order relationships between vertices and update vertex features. Specifically, we first use a weight matrix Vertex feature F sd Mapping is performed and the activated vertex features are obtained through the LeakyReLU function The second step is to transform the vertex feature u k With the attention matrix Multiplying together gives the adjacency matrix between category vertices Use A class The association matrix H is weighted and normalized by the softmax function; finally, the weighted association matrix is ​​multiplied by the mapped vertex features to obtain the representation of the hyperedge The above process is as follows Figure 2 , expressed as:

[0039]

[0040] Among them, the superscript l represents the number of layers of the model, h jk It is a super edge j Midpoint v k The adjacency matrix Aclass The weighted correlation coefficient. Similarly, the same steps are used to update the vertex features. First, a weight matrix Features of hyperedges Mapping is performed and the activated hyper-edge features are obtained through the LeakyReLU function The second step is to use the hyperedge feature z k With the attention matrix After multiplication, we get the adjacency matrix between hyperedges A edge The association matrix is ​​weighted and normalized with the softmax function; finally, the weighted association matrix is ​​multiplied with the mapped hyperedge features to update the vertex features. The above process is expressed as follows:

[0041]

[0042] Among them, h' ki Represents the v connecting the vertices i The super edge e k Through the adjacency matrix A edge Weighted correlation coefficient.

[0043] At this point, the above operation completes a vertex-hyperedge-vertex information transfer and updates the vertex features. In order to improve the robustness of the representation learned by the model, this method uses a feedforward layer and multi-head attention in the hypergraph Transformer layer. Finally, the output of this layer's hypergraph Transformer is obtained, and the process is expressed by the formula:

[0044] head i =HGA i (F l-1 ,H) (12)

[0045] MH-HGA(F l-1 ,H)=concat(head1,...,head h ) (13)

[0046] F′ (l) =LN(MH-HGA(F l-1 ,H)+F l-1 ) (14)

[0047] F l =LN(FFN(F′ (l) )+F′ (l) ) (15)

[0048] Among them, HGA(·) represents the hypergraph neural network constructed by this method, FFN(·) represents a feed-forward layer, concat(·) is a tensor concatenation operation, and LN(·) represents a layer normalization operation. Finally, F l The model predicts the label of the input image through a classification head.

[0049] S4 Optimization Goals

[0050] In the multi-label image classification task, assuming that the training set is X = {I i ,y i}, where i = 1, 2, ..., T, T represents the number of images in the training set. i With y i Represents the i-th image and its corresponding label respectively, then for N targets there are y i ={y i1 ,...,y ij ,...,y in}. When the true label includes label j, y ij is equal to 1, otherwise y ij is 0. The optimization function used in this method is the cross entropy loss function, that is

[0051]

[0052] Among them, p ij Represents the probability obtained by the classifier after a tanh function.

[0053] Compared with the prior art, the present invention provides a multi-label image classification method based on a hypergraph Transformer, which has the following beneficial effects: (1) A multi-label image classification model based on a hypergraph Transformer is constructed, which exhibits stronger noise resistance and generalization when mining complex relationships between vertices, thereby improving the classification accuracy; (2) An adaptive hypergraph Transformer network model is proposed, which defines an image content-aware hypergraph association matrix and constructs hypergraph association relationships between vertices through an attention mechanism, thereby enhancing the robustness of the model.

[0054] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-label image classification method based on hypergraph Transformer, characterized in that: The following steps are involved: S1: Extract image features using feature extraction network; S2: Decouple the extracted image features using the semantic decoupling model to obtain the semantic features of all possible classes; S3: Build a hypergraph Transformer model to jointly optimize the decoupled semantic features; S4: Perform multi-label classification on the image based on the optimized features.

2. The multi-label image classification method based on hypergraph Transformer according to claim 1, characterized in that: The image feature extraction module is a convolutional neural network, and specifically uses the residual network of the convolutional neural network to extract features from the image.

3. The multi-label image classification method based on hypergraph Transformer according to claim 1, characterized in that: The semantic feature decoupling module is a semantic feature interaction network, which specifically utilizes the low-rank bilinear pooling operation to decouple the target semantic features of the image.

4. The multi-label image classification method based on hypergraph Transformer according to claim 1, characterized in that: The semantic feature optimization module is a hypergraph Transformer, and the sub-step S3 is to optimize the target semantic features obtained in the sub-step S2 using the hypergraph Transformer network model.

5. The multi-label image classification method based on hypergraph Transformer according to claim 4 is characterized in that: The step S3 comprises the following sub-steps: S31: Combining the target semantic features obtained in step S2 with the attention mechanism, weighting the association matrix of the hypergraph to calculate the hyperedge features; S32: Combining the hyperedge features calculated in step S31 with the attention mechanism, weighting the association matrix of the hypergraph to update the vertex features.

6. The multi-label image classification method based on hypergraph Transformer according to claim 1, characterized in that: The step S4 is to classify the target semantic features optimized by the hypergraph Transformer model into image labels through a linear layer.

Images