Weak supervision target positioning method based on semantic-driven Transform model

Abstract

The invention discloses a weak supervision target positioning method based on a semantic-driven Transform model, and the method comprises the following steps: S1, carrying out the preprocessing of a to-be-detected picture, and converting the to-be-detected picture into a format suitable for neural network processing; s2, sending the preprocessed to-be-detected picture and the label of the corresponding picture level into a neural network; the neural network comprises a Transform backbone network, a semantic kernel integration module SKI and a semantic enhancement adaptation module SBA; s3, the neural network processes the picture, and outputs a probability value of the to-be-detected picture corresponding to each category in a training process; s4, outputting the coordinate, category and score of the frame predicted by the to-be-detected picture in the test process; according to the method, the attention graph is more focused on the foreground region, and more accurate semantic perception target positioning is realized in a weak supervision environment, so that the explanation precision and correlation of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a weakly supervised target positioning method based on a semantically driven Transformer model. Background Art

[0002] In recent years, with the rapid improvement of computer performance and the vigorous development of big data technology, visual information data has shown an explosive growth trend. Various multimedia data types, including static images, dynamic images, video files, and audio files, are spreading on social media and the Internet at an unprecedented rate. As one of the core issues in the field of computer vision, object detection technology has been widely used, covering many fields such as target tracking, behavior recognition, human-computer interaction, autonomous driving, and face recognition. This research direction has attracted the attention and in-depth exploration of a large number of scholars since the beginning of the 20th century. In particular, technologies such as face recognition, video surveillance, object detection, image content review, and biometric recognition have gradually become hot topics in computer vision research and are widely used in practical scenarios such as medical health, intelligent transportation, smart city management, and public safety, such as medical imaging diagnosis, posture estimation, station security, autonomous driving assistance systems, and video surveillance behavior analysis.

[0003] As an important research topic in the field of computer vision and machine learning, target localization integrates cutting-edge technologies from multiple disciplines such as image processing, pattern recognition, deep learning, and automatic control. Traditional target localization tasks usually involve identifying and marking the location of the main target in an image, which is usually located in a prominent area or center of the image. With the rapid development of social networks and video platforms, the multimedia resources people are exposed to have increased dramatically, and the application scenarios of target localization technology have gradually expanded, covering areas such as face detection in social platforms, pedestrian detection in images and video sequences, vehicle recognition in traffic monitoring, and providing image content assistance for visually impaired groups.

[0004] Current research on target localization mainly focuses on the application of convolutional neural networks (CNNs). Traditional target detectors usually rely on massive data with bounding box annotations during training. However, the collection process of bounding box annotations is not only time-consuming and labor-intensive, but also requires a lot of manpower costs, which greatly limits the practical application scope of the detector. In contrast, image-level label acquisition is relatively simple. For example, by using tools such as image search engines or photo sharing websites, images containing specific targets can be quickly filtered out from massive images. Therefore, the task of weakly supervised object localization (WSOL) based on image-level annotations for training has gradually become a research hotspot.

[0005] Existing WSOL methods usually rely on CNN classifiers to generate localization heatmaps, and predict the bounding box of the target object through the localization head to cover the high response areas in the image. However, WSOL faces two major challenges, and there is a significant performance gap compared with fully supervised object detection methods. First, CNN classifiers tend to over-rely on salient regions when processing tasks and ignore less salient features in the image, making it difficult for the model to learn comprehensive target localization information. Since the backbone network is usually pre-trained through image classification tasks, the learning process of the localization head is complex and difficult. Second, the bounding box generated by the localization head usually only captures the highly discriminative parts of the target object, ignoring the integrity of the target. For example, in a bird image, even if the features of the bird's head or feathers are sufficient for classification, the bounding box generated based only on these regions cannot correctly locate the entire bird object.

[0006] To address this problem, most existing WSOL methods rely on Class Activation Mapping (CAM) to activate feature regions related to the target category through classification losses. However, CAM can only capture the most discriminative regions of the target object, resulting in the generated bounding box being smaller than the actual target area. To overcome this limitation, researchers have proposed a variety of technical improvements to enhance the performance of CAM, such as graph propagation technology, data augmentation strategies, adversarial erasure, and spatial relationship activation. Although these improvements have improved the positioning effect of WSOL to a certain extent, due to the inherent local feature extraction limitations of convolutional neural networks, global feature relationships cannot be effectively modeled, resulting in limited performance in complete target positioning.

[0007] The basic idea of ​​the existing technology is to generate an attention map based on the class label and combine it with the feature map of a specific category to generate a localization map. However, this mechanism may produce ambiguous attention maps, that is, the generated attention maps are not clear in the category area. The reason why these attention maps show semantic agnostic characteristics is mainly attributed to two types of noise: 1) Basic noise, which comes from the model's prior knowledge of currently irrelevant categories. For example, in an image labeled "stingray" ( Figure 2 2) Novel noise refers to background clusters that are completely beyond the model’s prior knowledge. This type of noise activation weakens the model’s focus and causes it to allocate attention to irrelevant background regions. For example, in an image labeled “bird” ( Figure 2 The bottom example in the first column shows that the attention map not only highlights the bird, but also extends to background clusters such as wood. These unlearned, new background clusters induce scattered activation patterns, weakening the model's focus and mistakenly directing attention to irrelevant background areas. As a result, existing weakly supervised object localization methods have poor positioning accuracy and are difficult to meet practical usage needs. Summary of the invention

[0008] To solve the above problems, the present invention proposes a weakly supervised target localization method based on a semantically driven Transformer model. This method combines the advantages of weakly supervised learning to achieve lower-cost image annotation and less label information, obtain richer and more efficient feature expression, and thus improve the positioning performance and accuracy of the model.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A weakly supervised target localization method based on a semantically driven Transformer model includes the following steps:

[0011] S1, preprocessing the image to be detected, for converting the image to be detected into a format suitable for neural network processing;

[0012] S2, sending the pre-processed image to be detected and its corresponding image-level label to the neural network; the neural network includes a Transformer backbone network, a semantic kernel integration module SKI and a semantic enhancement adaptation module SBA; wherein the Transformer backbone network is used for feature extraction of the image; the semantic kernel integration module SKI is used to construct semantic convolution kernels in all Transformer self-attention feature maps; the semantic enhancement adaptation module SBA integrates and compares the language-image pre-training model, and provides more refined semantic guidance in the form of category text and feature pairs;

[0013] S3, the neural network processes the image and outputs the probability value of each category of the image to be detected during the training process;

[0014] The training process of the neural network in step S3 includes the following specific processes:

[0015] S31. Given a dataset set with image-level labels, divide the dataset set into a training image sample set and a test image sample set;

[0016] S32, randomly select an image I from the training image sample set, input the image I and its corresponding image-level label y into the Transformer backbone network of the neural network, and obtain the feature map output by the network The features enter the classifier through the global pooling layer and obtain the network classification loss

[0017] S33, the feature map X generated by the category-aware stimulus module csm and feature maps generated by the object constraint module Send it to the semantic kernel integration module SKI to generate a semantic kernel K ski After that, the semantic kernel K ski By convolution from the feature map Extract semantic clues from the feature map X ski , the new feature map X ski After the global pooling layer, it enters the classifier to generate category probability distribution Get the semantic core loss The calculation formula is:

[0018]

[0019] in, represents the semantic core loss; C represents the number of categories; i represents the cyclic summation subscript, and the value of i ranges from 0 to C-1; y i Represents the true label of the i-th category; e represents a natural constant; j represents a cyclic summation subscript, and the value of j ranges from 0 to C-1; represents the predicted probability of the i-th category; represents the predicted probability of the jth category;

[0020] S34, image I and feature map generated by object constraint module and its complement Multiply and input into the image encoder of the contrastive language-image pre-trained model In the foreground vector v f and background vector v b ; Input the image-level label t as text to the text encoder of the contrastive language-image pre-trained model In the text vector v t ; Finally, the text vector v t , foreground vector v f and background vector v b Input the semantic enhancement adaptation module SBA to get the semantic enhancement loss The calculation formula is:

[0021]

[0022]

[0023]

[0024] s f =sim(v t , v f ), s b =sim(v t , v b ),

[0025] Among them, v f represents the foreground vector; represents image encoder; I represents image; represents the feature map generated by the object constraint module; v b represents the background vector; v t represents a text vector; Represents a text encoder; represents semantic lifting loss; s f Indicates v t and v f The similarity between two vectors; s b Indicates v t and v b The similarity between two vectors; sim() represents the cosine similarity function;

[0026] S35. The total loss of the neural network is calculated as:

[0027]

[0028] in, Represents the total loss of the neural network; and are the category perception stimulus loss function, the object background constraint loss function, and the object foreground constraint loss function respectively; λ1, λ2, λ3, λ4, and λ5 are all loss weights, all initialized to 1; α is a hyperparameter;

[0029] S4. During the test, the coordinates, category and score of the predicted box of the image to be detected are output.

[0030] Preferably, the preprocessing in step S1 includes the following specific processes:

[0031] S11, applying a standardization operation to normalize pixel values ​​of the image;

[0032] S12, scaling the image after normalizing pixel values ​​to a fixed size of 256×256;

[0033] S13. Randomly crop the scaled image into sub-images of 224×224.

[0034] Preferably, in step S4, in the test phase, according to the category of the predicted box, the feature map obtained in step S32 In the above example, take out the corresponding feature map M out The feature map X generated by the category-aware stimulation module obtained in step S33 csm In the above example, take out the corresponding feature map M csm ;

[0035] Positioning heat map M fuse The calculation formula is:

[0036]

[0037] Among them, M fuse Represents the positioning heat map; Represents the feature map generated by the object constraint module; Indicates the corresponding element multiplication operation; M out Representation feature map The feature map extracted from M csm Represents the feature map X csm Finally, the class activation map is used to generate a bounding box to achieve target positioning.

[0038] After adopting the above technical scheme, the present invention has the following beneficial effects: the present invention firstly enhances the semantic-driven learning within the SDTR framework by collaboratively fusing the semantic information of the category-aware stimulus module and the object constraint module through the semantic kernel integration module SKI; and then introduces more powerful prior knowledge of image-text pairs through the semantic enhancement adaptation module SBA, further enhancing the semantic guidance in target positioning, aiming to isolate and highlight the features related to the target object more accurately by optimizing the attention mechanism. Specifically, the present invention achieves more accurate and semantically-aware target positioning in a weakly supervised environment by making the attention map more focused on the foreground area, thereby improving the interpretation accuracy and relevance of the model. Extensive experiments conducted on the CUB-200-2011 dataset and the ILSVRC dataset show that compared with other competitive methods, the weakly supervised target positioning method (abbreviated as SDTR) based on the semantically driven Transformer model proposed in the present invention can achieve significant performance improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 A schematic diagram of the present invention exploring the attention mechanism diagram in the Transformer model;

[0041] Figure 3 It is a schematic diagram of the network structure of the neural network of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] like Figures 1 to 3 As shown in FIG. 1 , a weakly supervised target localization method based on a semantically driven Transformer model, referred to as SDTR, includes the following steps:

[0044] S1, preprocessing the image to be detected, for converting the image to be detected into a format suitable for neural network processing;

[0045] The pre-processing in step S1 includes the following specific processes:

[0046] S11, applying a standardization operation to normalize pixel values ​​of the image;

[0047] S12, scaling the image after normalizing pixel values ​​to a fixed size of 256×256;

[0048] S13, randomly cropping the scaled image into sub-images of 224×224;

[0049] S2, sending the pre-processed image to be detected and its corresponding image-level label to the neural network; the neural network includes a Transformer backbone network, a semantic kernel integration module SKI and a semantic enhancement adaptation module SBA; wherein the Transformer backbone network is used for feature extraction of the image; the semantic kernel integration module SKI is used to construct semantic convolution kernels in all Transformer self-attention feature maps; the semantic enhancement adaptation module SBA integrates and compares the language-image pre-training model, and provides more refined semantic guidance in the form of category text and feature pairs;

[0050] S3, the neural network processes the image and outputs the probability value of each category of the image to be detected during the training process;

[0051] The training process of the neural network in step S3 includes the following specific processes:

[0052] S31. Given a dataset set with image-level labels, divide the dataset set into a training image sample set and a test image sample set;

[0053] S32, randomly select an image I from the training image sample set, input the image I and its corresponding image-level label y into the Transformer backbone network of the neural network, and obtain the feature map output by the network The features enter the classifier through the global pooling layer and obtain the network classification loss

[0054] S33, the feature map X generated by the category-aware stimulus module csm and feature maps generated by the object constraint module Send it to the semantic kernel integration module SKI to generate a semantic kernel K ski After that, the semantic kernel K ski By convolution from the feature map Extract semantic clues from the feature map X ski , the new feature map X ski After the global pooling layer, it enters the classifier to generate category probability distribution Get the semantic core loss The calculation formula is:

[0055]

[0056] in, represents the semantic core loss; C represents the number of categories; i represents the cyclic summation subscript, and the value of i ranges from 0 to C-1; y i Represents the true label of the i-th category; e represents a natural constant; j represents a cyclic summation subscript, and the value of j ranges from 0 to C-1; represents the predicted probability of the i-th category; represents the predicted probability of the jth category;

[0057] S34, image I and feature map generated by object constraint module and its complement Multiply and input into the image encoder of the contrastive language-image pre-trained model In the foreground vector v f and background vector v b ; Input the image-level label t as text to the text encoder of the contrastive language-image pre-trained model In the text vector v t ; Finally, the text vector v t , foreground vector v f and background vector v b Input the semantic enhancement adaptation module SBA to get the semantic enhancement loss The calculation formula is:

[0058]

[0059]

[0060]

[0061] s f =sim(v t , v f ), s b =sim(v t , v b ),

[0062] Among them, v f represents the foreground vector; represents image encoder; I represents image; represents the feature map generated by the object constraint module; v b represents the background vector; v t represents a text vector; Represents a text encoder; represents semantic lifting loss; s f Indicates v t and v f The similarity between two vectors; s b Indicates v t and v bThe similarity between two vectors; sim() represents the cosine similarity function;

[0063] S35. The total loss of the neural network is calculated as:

[0064]

[0065] in, Represents the total loss of the neural network; and are the category perception stimulus loss function, the object background constraint loss function, and the object foreground constraint loss function respectively; λ1, λ2, λ3, λ4, and λ5 are all loss weights, all initialized to 1; α is a hyperparameter;

[0066] S4. Output the coordinates, category and score of the box predicted by the image to be tested during the test process;

[0067] In step S4, during the test phase, the feature map obtained in step S32 is In the above example, take out the corresponding feature map M out The feature map X generated by the category-aware stimulation module obtained in step S33 csm In the above example, take out the corresponding feature map M csm ;

[0068] Positioning heat map M fuse The calculation formula is:

[0069]

[0070] Among them, M fuse Represents the positioning heat map; Represents the feature map generated by the object constraint module; Indicates the corresponding element multiplication operation; M out Representation feature map The feature map extracted from M csm Represents the feature map X csm Finally, the class activation map is used to generate a bounding box to achieve target positioning.

[0071] The effect of the present invention is further illustrated by the following simulation experiment.

[0072] 1) Simulation conditions

[0073] The present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python.

[0074] 2) Simulation content

[0075] Take the CUB-200-2011 dataset and the ILSVRC dataset, train the network according to the steps of the weakly supervised target localization method based on the semantically driven Transformer model of the present invention, and use the test set for testing. Table 1 and Table 2 are the localization and classification results of the present invention and other methods on the two datasets, respectively.

[0076] Table 1: Comparison of positioning performance of various methods

[0077]

[0078] Table 2: Comparison of classification performance of each method

[0079]

[0080] It can be found from Table 1 and Table 2 that compared with other methods, the present invention has the best effect. Among them, SDTR (Ours) is the result of the present invention, the evaluation index Loc.Acc. represents the Top-1 / 5 and GT-known positioning accuracy, and Cls.Acc. represents the classification accuracy. The weakly supervised target positioning method (SDTR) based on the semantic-driven Transformer model of the present invention has a classification and positioning performance of 83.87% and 81.33% on the CUB-200-2011 dataset, and a classification and positioning performance of 77.66% and 58.20% on the ILSVRC dataset. Compared with other methods, the positioning accuracy is higher, which proves the better classification and positioning effects of the present invention.

[0081] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A weakly supervised target localization method based on a semantically driven Transformer model, characterized in that: The following steps are involved: S1, preprocessing the image to be detected, for converting the image to be detected into a format suitable for neural network processing; S2, sending the pre-processed image to be detected and its corresponding image-level label to the neural network; the neural network includes a Transformer backbone network, a semantic kernel integration module SKI and a semantic enhancement adaptation module SBA; wherein the Transformer backbone network is used for feature extraction of the image; the semantic kernel integration module SKI is used to construct semantic convolution kernels in all Transformer self-attention feature maps; the semantic enhancement adaptation module SBA integrates and compares the language-image pre-training model, and provides more refined semantic guidance in the form of category text and feature pairs; S3, the neural network processes the image and outputs the probability value of each category of the image to be detected during the training process; The training process of the neural network in step S3 includes the following specific processes: S31. Given a dataset set with image-level labels, divide the dataset set into a training image sample set and a test image sample set; S32, randomly select an image I from the training image sample set, input the image I and its corresponding image-level label y into the Transformer backbone network of the neural network, and obtain the feature map output by the network The features enter the classifier through the global pooling layer and obtain the network classification loss S33, the feature map X generated by the category-aware stimulus module csm and feature maps generated by the object constraint module Send it to the semantic kernel integration module SKI to generate a semantic kernel K ski After that, the semantic kernel K ski By convolution from the feature map Extract semantic clues from the feature map X ski , the new feature map X ski After the global pooling layer, it enters the classifier to generate category probability distribution Get the semantic core loss The calculation formula is: in, represents the semantic core loss; C represents the number of categories; i represents the cyclic summation subscript, and the value of i ranges from 0 to C-1; y i represents the true label of the i-th category; e represents a natural constant; j represents a cyclic summation subscript, and the value of j ranges from 0 to C-1; represents the predicted probability of the i-th category; represents the predicted probability of the jth category; S34, image I and feature map generated by object constraint module and its complement Multiply and input into the image encoder of the contrastive language-image pre-trained model In the foreground vector v f and background vector v b ; Input the image-level label t as text to the text encoder of the contrastive language-image pre-trained model In the text vector v t ; Finally, the text vector v t , foreground vector v f and background vector v b Input the semantic enhancement adaptation module SBA to get the semantic enhancement loss The calculation formula is: s f =sim(v t ,v f ),s b =sim(v t ,v b ), Among them, v f represents the foreground vector; represents image encoder; I represents image; represents the feature map generated by the object constraint module; v b represents the background vector; v t Represents a text vector; Represents a text encoder; represents semantic lifting loss; s f Indicates v t and v f The similarity between two vectors; s b Indicates v t and v b The similarity between two vectors; sim() represents the cosine similarity function; S35. The total loss of the neural network is calculated as: in, Represents the total loss of the neural network; and are the category perception stimulus loss function, the object background constraint loss function, and the object foreground constraint loss function respectively; λ1, λ2, λ3, λ4, and λ5 are all loss weights, all initialized to 1; α is a hyperparameter; S4. During the test, the coordinates, category and score of the predicted box of the image to be detected are output.

2. A weakly supervised target localization method based on a semantically driven Transformer model as claimed in claim 1, characterized in that: The pre-processing in step S1 includes the following specific processes: S11, applying a standardization operation to normalize pixel values ​​of the image; S12, scaling the image after normalizing pixel values ​​to a fixed size of 256×256; S13. Randomly crop the scaled image into sub-images of 224×224.

3. The weakly supervised target localization method based on the semantically driven Transformer model according to claim 1, characterized in that: In step S4, during the test phase, the feature map obtained in step S32 is In the above example, take out the corresponding feature map M out The feature map X generated by the category-aware stimulation module obtained in step S33 csm In the above example, take out the corresponding feature map M csm ; Positioning heat map M fuse The calculation formula is: Among them, M fuse Represents the positioning heat map; Represents the feature map generated by the object constraint module; Indicates the corresponding element multiplication operation; M out Representation feature map The feature map extracted from M csm Represents the feature map X csm Finally, the class activation map is used to generate a bounding box to achieve target positioning.

Images