A multi-modal based weakly supervised target localization method

Abstract

The application discloses a kind of weak supervision target positioning methods based on multi-modal, comprising the following steps: obtaining image dataset and is divided into training dataset and test dataset, the training dataset is composed of image and classification label;Adjust the width and height of image data in image dataset, and do normalization to image;Classification label corresponding to image data is generated text data related to category by template;Conformer classification network is constructed;CLIP Text Encoder text encoder and Liner linear layer are constructed;The classification label of image is converted into text data, with the aid of the powerful image-text matching capability of CLIP, the image-text matching target added in the training process, additional supervision information is introduced for the training of network model, to alleviate the problem of class activation mapping.In addition, the application uses Conformer as the classification network of training, and uses the byproduct attention matrix of network model itself to refine class activation mapping, further improves the positioning performance of network model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image target positioning, and particularly relates to a weakly supervised target positioning method based on multi-modal. BACKGROUND

[0002] In a convolutional neural network (CNN), a convolution operation is good at extracting local features, but it is difficult to capture global representations. Inspired by the natural language processing field, ViT introduced the transformer structure into the field of computer vision, and the self-attention module in the transformer can effectively extract global information. However, due to the lack of some inductive bias of CNN, the local feature extraction capability of visual transformer is weaker than that of CNN. Later, a hybrid network structure Conformer with CNN-Transformer double branches was proposed to combine the capabilities of CNN and Transformer to improve the learning ability of the network model. The feature coupling unit (FCU) in the middle establishes a connection between the two branches, and fuses the local features from the CNN branch and the global features from the Transformer branch.

[0003] CLIP is a multi-modal model based on contrastive learning. Through training on a large number of image-text pairs, CLIP learns powerful image-text matching capability. Due to its strong generalization ability, it has been widely used in various downstream tasks.

[0004] Weakly supervised object localization aims to train the network model using only image-level labels of the training data, so as to determine the position of the object in the image. Although the fully supervised object localization method has made great progress, a large amount of high-quality labeled data needs to be collected before training the network model, and the cost of data collection is very high. The weakly supervised object localization method only needs to obtain the image-level label of the training data, and a large amount of training data can be obtained by searching on a search engine using keywords. Efficient training data acquisition makes weakly supervised object localization widely concerned in recent years.

[0005] The core method of weakly supervised object localization is class activation mapping, which extracts the class activation mapping of the trained classification model to generate the detection box of the target in the image. However, the class activation mapping usually only activates the most discriminative regions in the image, which makes it difficult to cover the complete target object. When the image scene is complex, the class activation mapping will be disturbed by the background information, and the irrelevant background content will be activated incorrectly. In recent years, methods such as adversarial erasing, divergent activation, and long-range capability of Transformer have been proposed to alleviate the above problems, but the improvement of these methods is still limited, and these methods do not consider improving the weakly supervised localization performance through additional supervision information. SUMMARY

[0006] The present application aims to provide a multi-modal based weakly supervised target positioning method to solve the problems raised in the background art.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] A multi-modal based weakly supervised target positioning method, comprising the following steps:

[0009] S1, obtaining an image data set and dividing it into a training data set and a test data set; the training data set is composed of images and classification labels;

[0010] S2, adjusting the width and height of the images in the image data set and performing standardization processing on the images; the classification labels corresponding to the image data are used to generate text data related to the categories through a template;

[0011] S3, constructing a Conformer classification network; the Conformer classification network is a CNN-Transformer double-branch structure, and there are L layers in total;

[0012] S4, constructing a CLIP Text Encoder text encoder and a Linear linear layer;

[0013] S5, obtaining an image representation I through the Conformer classification network for the image data;

[0014] S6, obtaining a text representation T through the CLIP Text Encoder text encoder and the Linear linear layer for the text data;

[0015] S7, calculating the cross-entropy loss of the image representation I and the corresponding classification label y;

[0016] S8, calculating the contrastive loss after L2 Norm of the image representation I and the text representation T respectively;

[0017] S9, updating the network model parameters using Adam;

[0018] S10, repeating steps S5-S9 until convergence;

[0019] S11, in the reasoning process, the test image is obtained after passing through the Conformer classification network, class activation map (CAM), accumulated L layer attention matrix and classification result, the attention matrix of L layer is fused and the attention map and affinity map are extracted from it, in order to further optimize the CAM, the CAM is first multiplied with the affinity map, then the output is Hadamard product with the attention map to obtain the final optimized CAM, finally, the threshold is set, the value greater than the threshold in the CAM is set to 255, and the value less than the threshold is set to 0, to generate a binary graph, the maximum pixel connected region is obtained according to the binary graph, to form the positioning frame of the final target.

[0020] Preferably, the width and height of the image data set image in step S2 are both adjusted to 224, and the image is standardized. The classification label corresponding to the image is filled into the template "a photo of a [classification label]", to obtain the class-related text data.

[0021] Preferably, the Conformer classification network in step S3 adopts Conformer-S version, and has 12 layers. The down sampling ratio of the convolution block in the last layer of the CNN branch is adjusted from 2 to 1.

[0022] Preferably, the CLIP Text Encoder in step S4 adopts ViT-B / 16 version, and is set as non-updatable in the training process.

[0023] Preferably, the threshold in step S11 is set to 0.1, and the binary graph is generated by comparing the class activation map with the threshold;

[0024] The technical effect and advantage of the present application: the weakly supervised target positioning method based on multi-modal proposed by the present application has the following advantages compared with the prior art: it can solve the weakly supervised target positioning problem under the condition of only containing image attribute classification label. Compared with the previous weakly supervised target positioning method, the method content of the present application is novel, and additional supervision information is introduced in the training process. In particular, the classification label is used to generate class-related text data through a template, and the CLIP text encoder is used to extract text representation rich in semantics and representative. In the process of matching the text representation and the image representation, the problem of local activation in the class activation map is alleviated. In addition, the attention matrix obtained by the Conformer classification network itself is separated into the attention map and the affinity map to further optimize the class activation map, so that the final target positioning performance is better. BRIEF DESCRIPTION OF DRAWINGS

[0025] Fig. 1 The flowchart of the present application;

[0026] Fig. 2 is a structural schematic diagram of the present application;

[0027] Fig. 3 is a result schematic diagram of the present application; DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. The specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0029] The present application provides a multi-modal based weakly supervised target localization method as shown in Figs. 1-3 The method comprises the following steps:

[0030] S1, acquiring an image dataset and dividing it into a training dataset and a test dataset, wherein the training dataset is composed of images and classification labels;

[0031] S2, adjusting the width and height of the image data in the image dataset, and performing normalization processing on the images; the classification labels corresponding to the image data are converted into text data;

[0032] S3, constructing a Conformer classification network; the Conformer classification network adopts a Conformer-S version with a total of 12 layers (L=12), wherein the down-sampling ratio of the Conv Block in the last layer is adjusted from 2 to 1 in this implementation;

[0033] S4, constructing a CLIP Text Encoder text encoder and a Linear linear layer; the CLIP Text Encoder adopts a ViT-B / 16 version;

[0034] S5, the image data passes through the Conformer classification network to obtain image representations I of the CNN branch and the Transformer branch, respectively;

[0035] S6, the text data passes through the CLIP Text Encoder text encoder and the Linear linear layer to obtain text representations T;

[0036] S7, calculating a cross-entropy loss between the image representations I and the corresponding classification labels y;

[0037] S8, calculating a contrastive loss between the image representations I and the text representations T after L2 Norm is performed on the image representations I and the text representations T, respectively

[0038] S9, updating the network model parameters with Adam;

[0039] S10, repeating steps S5-S9 until convergence;

[0040] S11, in the inference process, after the test image passes through the Conformer classification network, the class activation map (CAM), the accumulated L-layer attention matrix and the classification result are obtained, the attention matrix of the L layer is fused and the attention map and affinity map are extracted therefrom, in order to further optimize the CAM, the CAM is first multiplied with the affinity map, then the output thereof is Hadamard producted with the attention map to obtain the final optimized CAM, finally, a threshold is set, the values greater than the threshold in the CAM are set to 255 and the values less than the threshold are set to 0 to generate a binary graph, the maximum pixel connected region is obtained according to the binary graph to form a positioning frame of the final target.

[0041] Specifically, the CUB-200-2011 bird dataset is used in step S1, the dataset has a total of 11788 bird images, including 200 bird sub-classes, wherein the training dataset has 5994 images and the test set has 5794 images, and each image provides a class label of the image.

[0042] Specifically, in step S2, the width and height of the image in the image dataset are adjusted and uniformly set to 224, and the image is standardized. The classification label corresponding to the image data is filled into the template "a photo of a [classification label]" to obtain the text data corresponding to the image data.

[0043] Specifically, the Conformer in the step S3 is composed of one stem module, one CNN branch, one Transformer branch, and L-1 FCU modules, and the two branches respectively contain L transformer blocks and convolution blocks. The stem module is used to preliminarily extract local features. In the Transformer branch, the output of the stem module is generated into image blocks (patch tokens) through convolution, and then a class block (class token) is spliced in front of the image blocks as the input of the first transformer block. The input of the subsequent transformer blocks is the sum of the output of the FCU and the output of the previous transformer block. The FCU is used to establish the connection between the two branches and fuse the local features of the CNN branch and the global features of the Transformer branch. The Transformer head is a linear layer that extracts the class block in the output of the last transformer block, and the Transformer head maps the channel number of the class block to the class number. In the CNN branch, the overall design follows the classic pyramid scheme. With the increase of the number of layers, the size of the feature map gradually decreases, and the number of channels of the feature map gradually increases. In particular, the down-sampling ratio of the last convolution block is adjusted from 2 to 1. The Convolution head is a convolution layer with a convolution kernel size of 3x3, and the output of the last convolution block is mapped to the class number through the Convolution head, followed by a global average pooling layer.

[0044] Specifically, the CLIP Text Encoder in the step S4 includes a tokenizer and a 12-layer transformer encoder. The tokenizer can convert the input text data into encoded information, and the transformer encoder is used to extract text features. The CLIP Text Encoder is set to be non-updatable during the training process. The Linear linear layer converts the channel number of the input features.

[0045] Specifically, in the step S5, for an RGB image with a resolution of HxW D X The output obtained after the stem module are input into two branches respectively. In the transformer branch, F stem image blocks are generated through convolution where P is the width and height of the image patch. A class patch is concatenated in front of the image patch to get the input of the first transformer block After L-layer encoding, the outputs of the two branches are and where N = h x w. From the class patch is extracted The channel number of the class patch is mapped to the class number through the transformer head to get the image representation of the transformer branch where c is the class number. After the convolution head, the class activation map is obtained M is obtained after the global average pooling layer, and the image representation of the CNN branch is obtained The image representations of the two branches are added to obtain the final image representation I = I Trans + I Conv .

[0046] Specifically, the step S6 converts the text data into encoded information after passing through the word segmenter, and then inputs the encoded information into the 12-layer transformer encoder to extract features. The output is converted into the final text representation T after the Linear linear layer converts the channel number into the class number. More specifically, the text data D T After the CLIP Text Encoder, the output is obtained F Text After inputting the Linear linear layer, the final text representation is obtained

[0047] Specifically, the step S7 calculates the cross-entropy loss between the image representation I and the corresponding classification label y The specific calculation formula is as follows:

[0048]

[0049] Specifically, in the step S8, the image representation I and the text representation T are respectively subjected to L2-norm, and I' = L2-norm(I) and T' = L2-norm(T) are obtained. The contrastive loss is calculated between I' and T' The specific calculation formula is as follows, where τ is a learnable temperature parameter:

[0050]

[0051] Specifically, in step S9, the network model parameters are updated using Adam, with an initial learning rate of 0.00005. After 30 training epochs, the learning rate is adjusted to 0.00001, and a total of 60 training epochs are performed. The final loss function of the network model is... During training, the parameters of CLIP TextEncoder are not updated.

[0052] Specifically, in step S11, during the inference process, the text encoding branch is discarded, and the test image is processed by the Conformer classification network to obtain the category activation mapping. The accumulated attention matrix W of layer L and the classification results. The attention matrix is ​​an intermediate product of multi-head attention operations; for the attention matrix in the transformer block of layer l... W l The calculation formula is as follows: Where D = d trans G is the number of heads. The average of the attention matrices over all heads is then used to obtain... After summing the accumulated L-layer attention matrices, we obtain from Separate attention map Harmony and affinity Where Γ(.) is the transformation operation, To further optimize the category activation map M, the category activation map M is first compared with the affinity map M... Aff Perform matrix multiplication, then its output is compared with the attention map M. Attn Performing the Hadamard product operation yields the final optimized class activation map. The optimized calculation process is as follows:

[0053]

[0054] Finally, Normalization is performed, and the threshold λ = 0.1 is set. Values ​​greater than the threshold are set to 255, and values ​​less than the threshold are set to 0 to generate a binary image. The largest connected pixel region is obtained from the binary image to form the final target's bounding box.

[0055] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A weakly supervised target localization method based on multimodal methods, characterized in that, Includes the following steps: S1. Obtain the image dataset and divide it into a training dataset and a test dataset; the training dataset consists of images and classification labels; S2. Adjust the width and height of the images in the image dataset and perform standardization on the images; generate category-related text data for the classification labels corresponding to the image data using templates; S3. Construct a Conformer classification network; the Conformer classification network is a CNN-Transformer dual-branch structure with a total of L layers; S4. Construct the CLIP Text Encoder and the Liner layer; S5. Image data is used to obtain image representation I through the Conformer classification network; S6. Text data is processed by CLIP Text Encoder and Linear layer to obtain text representation T; S7. Calculate the cross-entropy loss between the image representation I and the corresponding classification label y; S8, image representation I and text representation T are respectively processed by L2 Norm to calculate contrast loss. S9. Update the network model parameters using Adam; S10. Repeat steps S5 to S9 until convergence. S11. During the inference process, the test image passes through the Conformer classification network to obtain the Class Activation Map (CAM), the accumulated L-layer attention matrix, and the classification result. The L-layer attention matrix is ​​fused and the attention map and affinity map are extracted from it. In order to further optimize the CAM, the CAM is first multiplied by the affinity map, and then its output is multiplied by the attention map to obtain the final optimized CAM. Finally, a threshold is set, and the values ​​in the CAM that are greater than the threshold are set to 255, and the values ​​that are less than the threshold are set to 0 to generate a binary map. The largest pixel connected region is obtained from the binary map to form the final target localization box.

2. The weakly supervised target localization method based on multimodal methods according to claim 1, characterized in that: In step S2, the width and height of the images in the image dataset are adjusted to 224, and the images are standardized. The corresponding classification labels of the images are filled into the template "a photo of a [classification label]" to obtain category-related text data.

3. The weakly supervised target localization method based on multimodal methods according to claim 1, characterized in that: In step S3, the Conformer classification network uses the Conformer-S version, which has 12 layers. The downsampling ratio of the convolution block in the last layer of the CNN branch is adjusted from 2 to 1.

4. The weakly supervised target localization method based on multimodal methods according to claim 1, characterized in that: In step S4, CLIP Text Encoder uses version ViT-B / 16 and is set to be non-updatable during training.

5. The weakly supervised target localization method based on multimodal methods according to claim 1, characterized in that: In step S11, the threshold is set to 0.1, and a binary map is generated by comparing the category activation map with the threshold.

Images