A multi-label image recognition method based on hidden context label relationship modeling

Abstract

The application provides a multi-label image recognition method based on hidden context label relationship modeling, comprising: obtaining an image to be recognized, performing feature extraction based on a feature extraction network to obtain hierarchical features, feeding the hierarchical features into a Squeeze-and-Excitation module to perform context information extraction, and performing information fusion to obtain features with multi-layer context information; using a hidden context information embedding module to embed the features with multi-layer context information to obtain hidden context information embedding features, and using a class decoupling module to decouple the features with multi-layer context information to obtain class-related features; and performing feature fusion on the hidden context information embedding features and the class-related features to obtain fused features; and inputting the fused features into a label relationship modeling module to perform classification and prediction to obtain a final result. The application can solve the problem that the co-occurrence relationship of labels cannot accurately represent the relevance of labels when a multi-label image recognition method in the prior art is used.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multi-label image recognition method based on latent context label relationship modeling. Background Technology

[0002] With the rapid development of computing hardware and computer vision, numerous researchers have dedicated themselves to designing efficient network architectures to overcome the performance bottleneck of single-label image recognition, achieving performance levels comparable to humans in some areas. However, in real-world scenarios, multi-label images are far more common than single-label images, leading more and more researchers to shift their focus to multi-label recognition tasks. The goal of multi-label image recognition is to enable the network to output all label information in an image at once. However, because each image contains multiple labels, and the number of labels varies from image to image, it presents a greater challenge than single-label recognition.

[0003] Currently, the mainstream research direction for multi-label image recognition tasks is to improve recognition performance by reducing the search space through constructing correlations between labels. A common approach is to construct a graph structure from the label information, then use the graph structure to capture the relationships between labels, and model the label relationships using the co-occurrence relationship as the label correlation. However, the co-occurrence relationship of labels cannot accurately represent the label correlation, leading to incorrect recognition results.

[0004] Therefore, there is an urgent need for a new multi-label image recognition method that can solve the problem that the co-occurrence relationship of labels cannot accurately represent the relevance of labels when processing multi-label image recognition in existing technologies. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a multi-label image recognition method based on implicit context label relationship modeling, which can solve the problem that the co-occurrence relationship of labels cannot accurately represent the relevance of labels when processing multi-label image recognition in the prior art.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a multi-label image recognition method based on implicit context label relationship modeling, the method comprising the following steps:

[0007] The image to be identified is acquired, and features are extracted based on a preset feature extraction network to obtain features at each level. The obtained features at each level are then fed into a preset Squeeze-and-Excitation module for context information extraction. The extracted context information at each level is then fused to obtain features with multi-level context information.

[0008] Using a pre-defined latent context information embedding module, the obtained features with multi-layered context information are embedded into the embedding vectors of each category to obtain latent context information embedded features. Using a pre-defined category decoupling module, the obtained features with multi-layered context information are decoupled to obtain category-related features. Furthermore, the obtained latent context information embedded features and category-related features are fused to obtain fused features.

[0009] In the preset label relationship modeling module, the obtained fused features are input, and the final result is predicted by classification.

[0010] The specific steps of acquiring the image to be identified, extracting features based on a preset feature extraction network to obtain features at each level, and then feeding the obtained features at each level into a preset Squeeze-and-Excitation module for context information extraction, and further fusing the extracted context information at each level to obtain features with multi-layer context information, include:

[0011] Using the formula X3, X4, X5 = f backbone (I) Feature extraction is performed on the image to be identified to obtain features at each level; where, f backbone (·) represents the feature extraction network, X3, X4 and X5 are the outputs of the last three layers of the feature extraction network, and I represents the image to be identified;

[0012] Through formula f se (X)=X×f sigmoid (f fc (f pool (X))), extract context information from these three outputs to obtain three context information items X′3, X′4, X′5; where X represents the input, f se (·) indicates the Squeeze-and-Excitation module, f fc (·) indicates a two-layer fully connected operation, f sigmoid (·) represents the sigmoid function, f pool (·) indicates a pooling operation;

[0013] Through formula X ml =[f inter (X′3): f inter (X′4): f inter [X′5] Matches and links the sizes of the three contextual information X′3, X′4, and X′5 to obtain a feature X with multi-layered contextual information. ml : where f inter (·) indicates an interpolation operation, and [:] indicates a linking operation.

[0014] The hidden context information embedding module is composed of a self-attention network and a forward propagation network.

[0015] The specific steps of using a preset latent context information embedding module to embed the obtained features with multi-layered context information into each category embedding vector to obtain latent context information embedded features, and using a preset category decoupling module to decouple the obtained features with multi-layered context information to obtain category-related features, and further fusing the obtained latent context information embedded features with category-related features to obtain fused features include:

[0016] Define a set of learnable class embedding vectors L eb ;

[0017] The resulting feature and category embedding vector L, which possesses multi-layered contextual information, is then used. eb Input the cross-attention network to embed contextual information into the category embedding vector L. eb Furthermore, a forward propagation network is used to map the dimension back to the input size in order to obtain the latent context information embedding feature L. ca Among them, L ca =f ffn (L ca′ ), f Q (·), f K (·), f V (·) represents three linear mapping functions; Q, K, and V represent the learnable parameters corresponding to these three functions, respectively; d k L represents the scaling factor; ca′ Indicates intermediate output variables; f softmax (·) represents the softmax function;

[0018] Through formula L sp =f poject (X ml The features with multi-layered contextual information are decoupled to obtain category-related features L. sp : where f project (·) represents the decoupling function. Here, a 1x1 convolutional layer is used as the decoupling function for decoupling.

[0019] Using the formula L=[L ca :L sp ], embedding latent contextual information into feature L ca Category-related features L sp Feature fusion is performed to obtain the fused feature L.

[0020] The label relationship modeling module is a Transformer encoder; wherein,

[0021] The label relationship modeling module predicts the final result by performing the following steps:

[0022] Through formula L co =f encoder (L), the fused feature L in the Transformer encoder yields the label relationship feature L. co ; where f encoder (·) indicates a Transformer encoder;

[0023] Through formula Label relationship feature L co Mapped to prediction confidence Among them, f fc (·) indicates a fully connected layer;

[0024] Through formula and formula Class-related features L sp and implicit contextual information embedding features L ca Mapped to prediction confidence respectively and Among them, f pool (·) represents the pooling function;

[0025] Through formula Prediction confidence and The data is then fused to obtain the final confidence level. And it classifies and predicts the final result.

[0026] Wherein, the final confidence level The loss function is:

[0027]

[0028] Where C represents the number of categories, y represents the category label, and σ(·) represents the sigmoid function.

[0029] Implementing the embodiments of the present invention has the following beneficial effects:

[0030] This invention not only proposes a context information extraction module that can effectively extract multi-layered latent context information, and a latent context information embedding module that embeds the latent context information into the category embedding vector, thereby containing direct label relationships, but also proposes a label relationship modeling module that utilizes the long-distance information capture capability of the Transformer encoder to efficiently construct label relationships. This solves the problem that existing technologies cannot accurately represent the correlation of labels when processing multi-label image recognition due to the inability of the co-occurrence relationship of labels. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0032] Figure 1 A flowchart illustrating a multi-label image recognition method based on latent context label relationship modeling provided in this embodiment of the invention;

[0033] Figure 2 A logic diagram of the latent context information embedding module in a multi-label image recognition method based on latent context label relationship modeling provided in an embodiment of the present invention;

[0034] Figure 3 To adopt Figure 2 The visualization result of the implicit context information embedding module embedding implicit context information into features;

[0035] Figure 4 A logic diagram of the label relationship modeling module in a multi-label image recognition method based on latent context label relationship modeling provided in an embodiment of the present invention;

[0036] Figure 5 An overall logic diagram of a multi-label image recognition method based on latent context label relationship modeling provided in an embodiment of the present invention;

[0037] Figure 6 This diagram illustrates a comparison between a multi-label image recognition method based on implicit context label relationship modeling provided in this embodiment of the invention and traditional image recognition methods. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] like Figure 1As shown in the figure, a multi-label image recognition method based on latent context label relationship modeling is provided in an embodiment of the present invention. The method includes the following steps:

[0040] Step S1: Obtain the image to be recognized and extract features based on the preset feature extraction network to obtain features at each level. Then, send the obtained features at each level into the preset Squeeze-and-Excitation module to extract context information. Further, fuse the extracted context information at each level to obtain features with multi-level context information.

[0041] The specific process is as follows: First, obtain the image I to be recognized.

[0042] Secondly, a predefined feature extraction network extracts features from the image I to be recognized, obtaining features at each level. For example, using the formula X3, X4, X5 = f backbone (I) Feature extraction is performed on the image to be identified to obtain features at each level; where, f backbone (·) represents the feature extraction network, where X3, X4, and X5 are the outputs of the last three layers of the feature extraction network.

[0043] Next, a predefined Squeeze-and-Excitation module extracts contextual information from features at each level, obtaining contextual information for each level. For example, through formula f se (X)=X×f sigmoid (f fc (f pool (X))), extract context information from these three outputs to obtain three context information items X′3, X′4, X′5; where X represents the input, f se (·) indicates the Squeeze-and-Excitation module, f fc (·) indicates a two-layer fully connected operation, f sigmoid (·) represents the sigmoid function, f pool (·) indicates a pooling operation.

[0044] Finally, the contextual information from each level is fused to obtain features with multi-layered contextual information. For example, through formula X... ml =[f inter (X′3): f inter (X′4): f inter [X′5] Matches and links the sizes of the three contextual information X′3, X′4, and X′5 to obtain a feature X with multi-layered contextual information. ml : where f inter (·) indicates an interpolation operation, and [:] indicates a linking operation.

[0045] Step S2: Using a preset latent context information embedding module, the obtained features with multi-layered context information are embedded into the embedding vectors of each category to obtain latent context information embedded features. The obtained features with multi-layered context information are decoupled using a preset category decoupling module to obtain category-related features. Furthermore, the obtained latent context information embedded features and category-related features are fused to obtain fused features.

[0046] The specific process is as follows: First, a latent context information embedding module is predefined, which consists of a cross-attention network and a forward propagation network, such as... Figure 2 As shown.

[0047] Secondly, define a set of learnable class embedding vectors L. eb .

[0048] Next, the resulting feature and category embedding vector L, which contains multi-layered contextual information, is... eb Input the cross-attention network to embed contextual information into the category embedding vector L. eb Furthermore, a forward propagation network is used to map the dimension back to the input size in order to obtain the latent context information embedding feature L. ca Among them, L ca =f ffn (L ca′ ), f Q (·), f K (·), f V (·) represents three linear mapping functions; Q, K, and V represent the learnable parameters corresponding to these three functions, respectively; d k L represents the scaling factor; ca′ Indicates intermediate output variables; f softmax (·) represents the softmax function;

[0049] Then, through formula L sp =f porject (X ml The features with multi-layered contextual information are decoupled to obtain category-related features L. sp : where f project (·) represents the decoupling function. Here, a 1x1 convolutional layer is used as the decoupling function for decoupling.

[0050] Finally, using the formula L = [L ca :L sp ], embedding latent contextual information into feature L ca Category-related features L sp Feature fusion is performed to obtain the fused feature L.

[0051] In one example, such as Figure 3 The image shown is a visualization of the embedded features of hidden context information. Figure 3 As can be seen, this feature has a direct relationship after embedding implicit contextual information.

[0052] Step S3: Input the obtained fused features into the preset label relationship modeling module, and classify and predict the final result.

[0053] The specific process is as follows: First, a predefined label relationship modeling module is defined, which is a Transformer encoder, and its structure is as follows: Figure 4 As shown.

[0054] At this point, the label relationship modeling module predicts the final result by performing the following steps:

[0055] Through formula L co =f encoder (L), the fused feature L in the Transformer encoder yields the label relationship feature L. co ; where f encoder (·) indicates a Transformer encoder;

[0056] Through formula Label relationship feature L co Mapped to prediction confidence Among them, f fc (·) indicates a fully connected layer;

[0057] Through formula and formula Class-related features L sp and implicit contextual information embedding features L ca Mapped to prediction confidence respectively and Among them, f pool (·) represents the pooling function;

[0058] Through formula Prediction confidence and The data is then fused to obtain the final confidence level. And it classifies and predicts the final result. Among them, the final confidence level The loss function is:

[0059]

[0060] Where C represents the number of categories, y represents the category label, and σ(·) represents the sigmoid function.

[0061] At this point, the overall logic diagram of a multi-label image recognition method based on latent context label relationship modeling in this embodiment of the invention is as follows: Figure 5 As shown. In Figure 6 The present invention is illustrated in comparison with traditional methods. Traditional methods use tag co-occurrence as the tag relationship, but we believe that there is no direct relationship between tags, and a relationship can only be generated through contextual information.

[0062] Implementing the embodiments of the present invention has the following beneficial effects:

[0063] This invention not only proposes a context information extraction module that can effectively extract multi-layered latent context information, and a latent context information embedding module that embeds the latent context information into the category embedding vector, thereby containing direct label relationships, but also proposes a label relationship modeling module that utilizes the long-distance information capture capability of the Transformer encoder to efficiently construct label relationships. This solves the problem that existing technologies cannot accurately represent the correlation of labels when processing multi-label image recognition due to the inability of the co-occurrence relationship of labels.

[0064] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.

[0065] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1.A multi-label image recognition method based on hidden context label relationship modeling, characterized in that, The method comprises the following steps: Obtaining an image to be identified, and performing feature extraction based on a preset feature extraction network to obtain hierarchical features, and sending the obtained hierarchical features into a preset Squeeze-and-Excitation module for context information extraction, further performing information fusion on the extracted hierarchical context information to obtain features with multi-layer context information, and the specific steps comprise: By formula X3, X4, X5 = f backbone (I), feature extraction is performed on the image to be identified to obtain hierarchical features; wherein, f backbone (·) represents a feature extraction network, X3, X4, and X5 are the last three layers of the output of the feature extraction network, and I represents the image to be identified. By formula f se (X) = X x f sigmoid (f fc (f pool (X))) respectively, and context information extraction is performed on the three outputs to obtain three context information X'3, X'4 and X'5; wherein, X represents input, f se (·) represents a Squeeze-and-Excitation module, f fc (·) represents two-layer full connection operation, f sigmoid (·) represents a sigmoid function, f pool (·) represents a pooling operation; By formula X ml = [f inter (X'3): f inter (X'4): f inter (X'5)] match and link the sizes of the three context information X'3, X'4, X'5 to obtain a feature X with multi-layer context information ml : wherein f inter (.) represents an interpolation operation, and [ : ] represents a linking operation Using a preset hidden context information embedding module to embed the obtained features with multi-layer context information into various category embedding vectors to obtain hidden context information embedding features, and using a preset category decoupling module to decouple the obtained features with multi-layer context information to obtain category-related features, and further performing feature fusion on the obtained hidden context information embedding features and category-related features to obtain fused features, and the hidden context information embedding module is composed of a self-attention network and a forward propagation network, and the specific steps comprise: defining a set of learnable class embedding vectors L eb ; The obtained feature and category embedding vector L with multi-layer context information eb The input cross-attention network is used to realize the embedding of the context information into the category embedding vector L eb In the middle, and further utilize the forward propagation network to map the dimension back to the input size to obtain the hidden context information embedding feature L ca ; wherein, L ca = f ffn (L ca′ ), f Q (·), f K (·), f V (·) represent three linear mapping functions; Q, K, V represent the learnable parameters corresponding to the three functions respectively; d k represents a scaling factor; L ca′ represents an intermediate output variable; f softmax (·) represents a softmax function; By formula L sp = f project (X ml ), the obtained features with multi-layer context information are decoupled to obtain class-related features L sp : wherein f project (·) represents a decoupling function, and here a 1x1 convolutional layer is used as the decoupling function for decoupling; By formula L = [L ca : L sp ], the implicit context information is embedded into the feature L cu and the category-related feature L sp is fused to obtain a fused feature L; in a preset label relationship modeling module, the obtained fused feature is input, and a final result is predicted by classification. 2.The multi-label image recognition method based on hidden context label relationship modeling of claim 1, wherein, The label relationship modeling module is a Transformer encoder; wherein The label relationship modeling module predicts the final result by performing the following steps, specifically comprising: By formula L co = f encoder (L), the fusion feature L is obtained in the Transformer encoder, and the label relationship feature L co is obtained; wherein f encoder (·) represents a Transformer encoder; By the formula The label relation feature L co is mapped to the prediction confidence where f fc (·) denotes a fully connected layer; By formula and formula The category-related feature L sp and the hidden context information embedding feature L ca are respectively mapped to the prediction confidence and where f pool (·) represents a pooling function; The prediction confidence is fused with the prediction confidence and the prediction confidence to obtain the final confidence and the final result is classified. 3.The multi-label image recognition method based on the hidden context label relationship modeling of claim 2, wherein, the final confidence The loss function is: Wherein, C represents the number of categories, y represents the category label, and σ(·) represents the sigmoid function.

Images