A multi-label image recognition method based on label inference

Abstract

The application relates to the technical field of digital information transmission, and discloses a multi-label image recognition method based on label reasoning, a scale feature extraction module, a label feature decoupling module, a self-attention module, a label reasoning module and a loss function construction module; the multi-scale feature extraction module extracts features of the picture by using a convolutional neural network; the label feature decoupling module decouples categories corresponding to each label by defining label embedding features; the self-attention module decouples label features on the label feature decoupling module and makes information between each feature intercommunicate; and the label reasoning module reasons labels based on the self-attention module; the application decouples labels by using a currently popular self-attention module and reasons the hidden labels, thereby effectively constructing the correlation between the labels and finally improving the performance of multi-label image recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a multi-label image recognition method based on label inference. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning, the performance of various tasks in the field of computer vision has also been greatly improved. Among them, image label recognition task is one of the most basic and important tasks in the field of computer vision. This task aims to input an image, extract image features through a specific algorithm, and finally use a classifier to correctly identify it. Traditional image recognition algorithms often extract image features through manually set rules. These features are usually only suitable for certain specific scenarios, so their versatility is limited. With the development of deep learning in recent years, end-to-end training can enable the network to adaptively learn reasonable features, thereby greatly improving the generalization of the algorithm. Current image label recognition work focuses on single-label image recognition. However, in real-world scenarios, images often contain multiple labels. Therefore, studying multi-label image recognition is more practical and has gradually become the focus of current research. However, multi-label images contain multiple labels, and the number of labels contained in each image is not fixed. Therefore, compared with single-label image recognition tasks, multi-label image recognition tasks are more challenging. In addition, multi-label image recognition tasks have more practical applications, such as product recognition in intelligent retail, global context feature extraction in target detection and semantic segmentation, count target recognition in counting tasks, and medical image recognition, etc.

[0003] The traditional approach to handling multi-label images is to decouple the task into multiple single-label recognition tasks. However, since multi-label images have multiple labels, the number of label combinations increases geometrically with the number of labels. Therefore, this method has limited performance improvement. Later researchers found that although images contain multiple labels, there is a correlation between the labels. For example, "sky" and "white cloud" are strongly correlated label combinations. Pictures with "white cloud" labels often also have "sky". However, "penguin" and "desert" are not related, and these two labels will not appear at the same time. Therefore, if we can design an algorithm to fully utilize the above rules, we can effectively filter label combinations and reduce the search space. Through this phenomenon, current methods use deep learning networks to construct label relationships to reduce the search space of label combinations and improve the performance of the final classification.

[0004] The defects of the prior art exist in the following two points:

[0005] 1. Since multi-label images contain multiple labels and the number of labels contained in each image is not fixed, it is difficult to reasonably use neural networks to construct label relationships.

[0006] 2、Feature extraction is insufficient, hyperparameters are designed in advance and cannot be self-adaptive, can only be used for specific scene effect, and universality is not strong, and strong prior information is needed to assist training. SUMMARY

[0007] Technical problems to be solved

[0008] In view of the deficiencies of the prior art, the present application provides a multi-label image recognition method based on label reasoning, a multi-label image recognition algorithm based on label reasoning, and a self-attention module that is currently popular is used to decouple labels and reason about the masked labels, thereby effectively establishing the correlation between the labels and ultimately improving the performance of multi-label image recognition.

[0009] Technical scheme

[0010] To achieve the above object, the present application provides the following technical scheme: a multi-label image recognition method based on label reasoning, comprising a multi-scale feature extraction module, a label feature decoupling module, a self-attention module, a label reasoning module, and a loss function construction module, the multi-scale feature extraction module uses a convolutional neural network to extract features from the picture; the label feature decoupling module decouples the categories corresponding to each label by defining label embedding features.

[0011] Further, the self-attention module decouples the label features based on the label feature decoupling module and allows information between each feature to propagate; the label reasoning module reasons about the labels based on the self-attention module.

[0012] Further, the loss function construction module is composed of two loss functions, which are mapped to the predicted label confidence by a fully connected layer, and then the final loss result is calculated with the annotation information.

[0013] 4. The multi-label image recognition method based on label reasoning according to any one of claims 1-3, comprising the following steps:

[0014] S1, first input an image I, and use a multi-scale feature extraction module to extract features from the picture by a convolutional neural network;

[0015] S2, after obtaining the global features, use the label embedding features in the label feature decoupling module to decouple the categories corresponding to each label;

[0016] S3, use the self-attention module to reason about the labels after the label features have been decoupled;

[0017] S4, calculate the final loss result by the loss function construction module for the decoupled features.

[0018] Further, the higher the layer output of the neural network, the richer the semantic information contained in the feature, but the smaller the output feature, the easier to miss small object information, and the lower the layer, just the opposite, so the high-level convolutional layer output feature has rich semantic information, but due to the problem of too small feature size, small object information will be missed, so the high-level feature and low-level feature are fused to process the problem of small object loss, so as to obtain more robust features, and the last three layers of output features are recorded as 、 、 The fused feature is recorded as The specific algorithm is as follows:

[0019] .

[0020] Among them, it needs to be specifically explained that and represent down-sampling and up-sampling operations respectively, C represents the number of channels, and H and W represent the length and width of the feature respectively.

[0021] Further, the definition of the label embedding feature N represents the number of label categories in the data set. The label embedding feature can adaptively update the parameters with the learning of the network, and finally learn the features corresponding to each label. Then, the global feature is flattened and transposed, and the algorithm is as follows: .

[0022] Among them, it needs to be specifically explained that represents the feature after flattening and transposing, and represent flattening and transposing operations respectively. Then, the label embedding feature and are spliced to obtain the input feature of the self-attention module, and the algorithm is as follows:

[0023] .

[0024] Among them, it needs to be specifically explained that represents the splicing operation. After obtaining the input feature of the self-attention module, the label feature is decoupled by using the self-attention module. The self-attention module can effectively make the information between each feature propagate to each other, so as to learn the semantic information specific to each category, thereby realizing the decoupling operation of the label feature. The input feature is projected into three matrices: Query, Key, Value, respectively recorded as Q, K, V, by using three learnable matrices. Then, the self-attention operation is performed on the three matrices to obtain the output feature , and the specific algorithm is as follows:

[0025] .

[0026] Among them, it needs to be specified that represents a softmax function, represents a scaling coefficient, D represents the dimension of the feature, and finally, the feature is mapped to the original dimension by using the normalization layer and the full connection layer to obtain the final output , the self-attention module can better decouple the label information according to the inductive bias assumption multi-layer self-attention module.

[0027] Further, a learnable label reasoning embedding feature is defined in the self-attention module as The function of the feature is to reason the masked label, then randomly mask a label feature, and concatenate the remaining label feature and the reasoning embedding feature, and send it to the self-attention module for reasoning. The label reasoning embedding feature can adaptively learn the remaining label feature information in the learning process, so as to reason the masked feature label. In order to improve the reasoning efficiency, the following rules are defined for the random masking process: all positive example labels are masked once, and 4 negative example labels are randomly sampled for masking.

[0028] Further, the loss function construction module is composed of two loss functions. First, the decoupled feature is mapped to the predicted label confidence by using the full connection layer, and then the final loss result is calculated with the annotation information, as follows:

[0029] ;

[0030] .

[0031] Among them, it needs to be specified that represents a selection function, -N represents selecting N features from the back to the front, y represents the annotation information, represents a sigmoid function, and then the loss function is calculated for the predicted result of reasoning, as follows: .

[0032] Finally, the two loss functions are added to obtain the final loss function: .

[0033] Further, the multi-scale feature extraction module can be replaced by a feature pyramid module. The feature pyramid module utilizes the high resolution of low-level features and the high semantic information of high-level features, and achieves the prediction effect by fusing these features of different layers. And the prediction is carried out separately on each fused feature layer, that is, the high-level features with low resolution and high semantic information and the low-level features with high resolution and low semantic information are connected from top to bottom.

[0034] Advantages

[0035] The application provides a multi-label image recognition method based on label inference.

[0036] The multi-label image recognition method based on label inference effectively constructs the correlation between labels by decoupling and inferring the masked labels using the currently popular self-attention module, thereby improving the performance of multi-label image recognition. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The overall flowchart of the application.

[0038] Figure 2 The test index table on the COCO data set of the application.

[0039] Figure 3 The test index table on the VOC data set of the application. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0041] It should be understood that the sizes of the various portions shown in the drawings are not necessarily shown to scale.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0043] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the specification.

[0044] It should be noted that like numbers and letters refer to like items throughout the drawings, and once an item is defined in one drawing, it should not require further discussion in subsequent drawings.

[0045] The embodiments of the present application can be applied to a computer system / server, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, and the like.

[0046] The computer system / server can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform particular tasks or implement particular abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including memory storage devices.

[0047] Embodiment 1

[0048] The present application provides a technical solution: a multi-label image recognition method based on label reasoning, comprising a multi-scale feature extraction module, a label feature decoupling module, a self-attention module, a label reasoning module, and a loss function construction module. The multi-scale feature extraction module extracts features from the picture using a convolutional neural network. The label feature decoupling module decouples the categories corresponding to each label by defining label embedding features. The self-attention module decouples the label features based on the label feature decoupling module and allows information between each feature to be transmitted to each other. The label reasoning module reasons about the labels based on the self-attention module. The loss function construction module is composed of two loss functions. The loss function construction module maps the predicted label confidence using a fully connected layer, and then calculates the final loss result with the annotation information.

[0049] A management method for a multi-label image recognition method based on label reasoning, specifically as follows:

[0050] 101、First, input an image I, and extract features from the picture using a convolutional neural network through a multi-scale feature extraction module;

[0051] In this embodiment, it needs to be specifically pointed out that the higher the layer output, the richer the semantic information contained in the feature, but the smaller the output feature, the easier to miss small object information, and the lower the layer, just the opposite, therefore, although the high-level convolutional layer output feature has rich semantic information, due to the problem of too small feature size, small object information will be missed, so the high-level feature and low-level feature are fused to process the problem of small object loss, so as to obtain more robust features, the features output by the last three layers are recorded as 、 、 The fused feature is recorded as The specific algorithm is as follows:

[0052] .

[0053] Among them, it needs to be specifically pointed out that and represent down-sampling and up-sampling operations respectively, C represents the number of channels, H and W represent the length and width of the feature respectively, and the embodiment is not specifically limited.

[0054] 102, after obtaining the global feature, the class corresponding to each label is decoupled by using the defined label embedding feature in the label feature decoupling module;

[0055] In this embodiment, it needs to be specifically pointed out that the defined label embedding feature N represents the number of label categories in the data set, and the label embedding feature can update the parameters adaptively with the learning of the network, and finally learn the features corresponding to each label. Then, the global feature is flattened and transposed, and the algorithm is as follows: .

[0056] Among them, it needs to be specifically pointed out that represents the feature after flattening and transposition, and represent flattening and transposition operations, then the label embedding feature and are spliced to obtain the input feature of the self-attention module, and the algorithm is as follows:

[0057] .

[0058] Among them, it needs to be specifically pointed out that denotes a concatenation operation, after the input features are obtained from the self-attention module, the label features are decoupled by using the self-attention module, the self-attention module can effectively make the information between each feature propagate to each other, so as to learn the semantic information specific to each category, thereby realizing the decoupling operation of the label features, the specific algorithm is as follows: the input features are projected into three matrices: Query, Key, Value, respectively denoted as Q, K, V, and then the self-attention operation is performed on the three matrices to obtain the output features :

[0059] .

[0060] Among the specific descriptions denotes a softmax function, denotes a scaling coefficient, D denotes the dimension of the feature, finally, the feature is mapped to the original dimension by using the normalization layer and the full connection layer to obtain the final output , the self-attention module can better decouple the label information according to the inductive bias assumption of the multi-layer self-attention module.

[0061] 103, the decoupled label features are obtained, and the self-attention module is used to reason the label;

[0062] In this embodiment, it needs to be specifically explained that a learnable label reasoning embedding feature is defined in the self-attention module , the function of the feature is to reason the masked label, then a label feature is randomly masked, and the remaining label features and the reasoning embedding feature are concatenated and sent to the self-attention module for reasoning, then the label reasoning embedding feature can adaptively learn the information of the remaining label features in the learning process, thereby reasoning the masked feature label, in order to improve the reasoning efficiency, the following rules are defined for the random masking process: all positive example labels are masked once, and 4 negative example labels are randomly sampled for masking, this embodiment is not specifically limited.

[0063] 104, the decoupled features are calculated by the loss function construction module to obtain the final loss result;

[0064] In this embodiment, it needs to be specifically explained that the loss function construction module is composed of two loss functions, first, the decoupled features are mapped to the predicted label confidence by using the full connection layer, and then the final loss result is calculated with the annotation information, the process is as follows:

[0065] ;

[0066] .

[0067] Wherein, it needs to be specifically pointed out that wherein Indicates a selection function, -N indicates selecting N features from back to front, y indicates label information, Indicates a sigmoid function, and then a loss function is calculated for the predicted result of reasoning, and the process is as follows: .

[0068] Finally, two loss functions are added to obtain a final loss function: .

[0069] Embodiment 2

[0070] The application provides a technical scheme: a multi-label image recognition method based on label reasoning, comprising a feature pyramid module, a label feature decoupling module, a label reasoning module, a loss function construction module and a self-attention module.

[0071] A management method of a multi-label image recognition method based on label reasoning, specifically as follows:

[0072] 101, first input an image I, and extract features of the image by using a convolutional neural network through the feature pyramid module;

[0073] In this embodiment, it needs to be specifically pointed out that the feature pyramid module uses low-layer features with high resolution and high-layer features with high semantic information, and achieves the prediction effect by fusing features of different layers, and the prediction is separately performed on each fused feature layer, that is, the high-layer features with low resolution and high semantic information and the low-layer features with high resolution and low semantic information are connected from top to bottom, so that all features at different scales have rich semantic information, and the embodiment is not limited in particular.

[0074] 102, after obtaining the global features, the categories corresponding to each label are decoupled by using the defined label embedding features in the label feature decoupling module;

[0075] In this embodiment, it needs to be specifically pointed out that the label embedding features N represents the number of label categories in the data set, and the label embedding features can adaptively update parameters along with the learning of the network, and finally learn the features corresponding to each label, and then perform flattening and transposition operations on the global features, and the algorithm is as follows: .

[0076] Wherein, it needs to be specifically pointed out that Indicates the features after flattening and transposition, And Respectively represent the flattening and transposition operations, and then the label embedding features and are spliced to obtain the input features of the self-attention module , and the algorithm is as follows:

[0077] .

[0078] wherein, it needs to be specifically pointed out that indicates a splicing operation, after the input feature obtained from the self-attention module, the self-attention module is used to decouple the label feature, the self-attention module can effectively make the information between each feature propagate to each other, so as to learn the semantic information specific to each category, thereby realizing the decoupling operation of the label feature, using three learnable matrices to project the input feature into three matrices: Query, Key, Value, respectively denoted as Q, K, V, and then performing self-attention operation on the three matrices to obtain the output feature , the specific algorithm is as follows:

[0079] .

[0080] wherein, it needs to be specifically pointed out that indicates a softmax function, indicates a scaling coefficient, D indicates the dimension of the feature, and finally, the feature is mapped to the original dimension by using a normalization layer and a fully connected layer to obtain the final output The self-attention module can better decouple the label information according to the inductive bias assumption.

[0081] 103, the label feature that has been decoupled is used to infer the label by using the self-attention module;

[0082] In this embodiment, it needs to be specifically pointed out that a learnable label inference embedding feature is defined as The function of the feature is to infer the masked label, then a label feature is randomly masked, and the remaining label feature and the inference embedding feature are spliced and sent to the self-attention module for inference, so that the label inference embedding feature can adaptively learn the remaining label feature information in the learning process, thereby inferring the masked feature label. In order to improve the inference efficiency, the following rules are defined for the random masking process: all positive example labels are masked once, and 4 negative example labels are randomly sampled for masking. This embodiment is not specifically limited.

[0083] 104, the decoupled feature is calculated by a loss function construction module to obtain the final loss result;

[0084] In this embodiment, it needs to be specifically pointed out that the loss function construction module is composed of two loss functions. First, the decoupled feature is mapped to the predicted label confidence by using a fully connected layer, and then the final loss result is calculated with the annotation information, the process is as follows:

[0085] ;

[0086] .

[0087] wherein, it needs to be specifically explained that wherein represents a selection function, -N represents selecting N features from the back to the front, y represents the label information, represents a sigmoid function, then the loss function is calculated for the predicted result of the inference, the process is as follows: .

[0088] Finally, the two loss functions are added to obtain the final loss function: .

[0089] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-label image recognition method based on label reasoning, characterized in that: It includes a multi-scale feature extraction module, a label feature decoupling module, a self-attention module, a label inference module, and a loss function construction module. The multi-scale feature extraction module uses a convolutional neural network to extract features from images. The label feature decoupling module decouples the categories corresponding to each label by defining label embedding features. The self-attention module defines a learnable label inference embedding feature as follows: The function of this feature is to infer the masked label, then randomly mask one label feature, and concatenate the remaining label features with the inference embedding feature, and then feed it into the self-attention module for inference. In this way, the label inference embedding feature can adaptively learn the remaining label feature information during the learning process, thereby inferring the masked feature label. In order to improve the inference efficiency, the following rules are defined for the random masking process: all positive label is masked once, and four negative label samples are randomly selected for masking. The defined tag embedding feature N represents the number of label categories in the dataset. The embedded features of these labels can adaptively update their parameters as the network learns, ultimately learning the features corresponding to each label. Then, the global features are flattened and transposed. The algorithm is as follows: , What needs to be specifically explained is This indicates the features after flattening and transposition. and These represent flattening and transpose operations, respectively; C represents the number of channels; and H and W represent the length and width of the feature, respectively. Then, the labels are embedded into the features and... Concatenation is performed to obtain the input features of the self-attention module. The algorithm is as follows: ， What needs to be specifically explained is This represents the concatenation operation. After obtaining the input features of the self-attention module, the self-attention module is used to decouple the label features. The self-attention module can effectively allow information to propagate between the various features, thereby learning the semantic information specific to each category, thus achieving the decoupling operation of the label features. The input features are projected into three matrices: Query, Key, and Value, denoted as Q, K, and V, respectively, using three learnable matrices. Then, the self-attention operation is performed on these three matrices to obtain the output features. The specific algorithm is as follows: ， What needs to be specifically explained is This represents the softmax function. The scaling factor is represented by , and D represents the dimension of the feature. Finally, a normalization layer and a fully connected layer are used to map the feature back to its original dimension to obtain the final output. According to the inductive bias hypothesis, the self-attention module can better decouple the label information. The loss function construction module consists of two loss functions. First, the decoupled features are mapped to predicted label confidence using a fully connected layer. Then, the final loss result is calculated together with the labeled information. The process is as follows: ； ， Among them, it is necessary to explain in detail that This represents the selection function, -N indicates selecting N features from the end to the beginning, and y represents the annotation information. Let represent the sigmoid function, and then calculate the loss function based on the inference prediction results, as follows: , Finally, the two loss functions are added together to obtain the final loss function: .

2. The multi-label image recognition method based on label reasoning according to claim 1, characterized in that: The self-attention module decouples the label features based on the label feature decoupling module and allows information to propagate between the various features; the label inference module infers the labels based on the self-attention module.

3. The multi-label image recognition method based on label reasoning according to claim 1, characterized in that: The loss function construction module consists of two loss functions, which are mapped to the predicted label confidence using a fully connected layer, and then the final loss result is calculated together with the annotation information.

4. A multi-label image recognition method based on label reasoning according to any one of claims 1-3, comprising the following steps: S1. First, input an image I, and use a convolutional neural network to extract features from the image through the multi-scale feature extraction module. S2. After obtaining the global features, use the defined label embedding features in the label feature decoupling module to decouple the categories corresponding to each label. S3. Use the self-attention module to infer the labels after obtaining the decoupled label features; S4. Calculate the final loss result for the decoupled features using the loss function construction module.

5. The multi-label image recognition method based on label reasoning according to claim 1, characterized in that: The higher the layer of the neural network, the richer the semantic information contained in the output features. However, the smaller the output features, the easier it is to miss information about small objects. Conversely, lower layers have the opposite effect. Therefore, although the features output by high-level convolutional layers have rich semantic information, they may miss information about small objects due to their small size. To address this issue, high-level and low-level features are fused to obtain more robust features. The features output by the last three layers are denoted as... , , The fused features are denoted as The specific algorithm is as follows: , What needs to be explained in detail is and These represent downsampling and upsampling operations, respectively. C represents the number of channels, and H and W represent the length and width of the feature, respectively.

6. The multi-label image recognition method based on label reasoning according to claim 1, characterized in that: The multi-scale feature extraction module can be replaced by a feature pyramid module. The feature pyramid module utilizes the high resolution of low-level features and the high semantic information of high-level features to achieve the prediction effect by fusing these features from different layers. The prediction is performed separately on each fused feature layer, that is, the high-level features with low resolution and high semantic information and the low-level features with high resolution and low semantic information are connected from top to bottom.

Images