Visual positioning method for supervising image-text feature fusion based on text mask

By using the graphic feature fusion method of text mask supervision in visual positioning, using ViT and BERT to extract features and generate mask supervision heat maps, the problem that visual positioning methods in the prior art cannot accurately capture the detailed features of the target object. Achieve more accurate visual positioning effect.

CN119027641APending Publication Date: 2024-11-26CHANGCHUN UNIV OF SCI & TECH
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411131844.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing visual positioning methods cannot accurately capture the detailed characteristics of the target object, resulting in ambiguity during positioning, especially when given a given descriptive expression is blurred and there are too many similar objects in the matching picture.

Method used

Using the graphic feature fusion method based on text mask supervision, multiple mask supervision heat maps are generated by constructing mask text and combining them with images, as additional inputs of the decoder for regression prediction, enhancing the model's positioning ability of the target object . The method includes a visual encoder, a text encoder, a visual language alignment module, a text enhancement module and a cross-modal decoder, extract features using ViT and BERT, and achieves the final target positioning through iterative regression prediction.

Benefits of technology

It effectively solves the positioning ambiguity problem caused by the inability to accurately capture the detailed characteristics of the target object in the visual positioning method, and improves the accuracy and accuracy of visual positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027641A_ABST
    Figure CN119027641A_ABST
Patent Text Reader

Abstract

The invention discloses a visual positioning method for supervising image-text feature fusion based on a text mask, belongs to the technical field of computer vision and image processing, and uses ViT and BERT as a visual encoder and a text encoder. Firstly, an input text is subjected to two times of mask processing to obtain two masked texts, and an expanded text obtained through mask processing can cooperate with a full text to generate a finer mask supervision heat map through a visual language alignment module. Secondly, designing a text enhancement module, and weighting and suppressing different words in a text by using Bi-GRU with phrase attention, so that the model pays more attention to text information related to a positioning target, and meanwhile, the influence of background information unrelated to the positioning target is reduced; and finally, the cross-modal decoder receives the enhanced text features and performs iterative regression prediction on the original image together with the mask supervision heat map, so that the problem of positioning ambiguity caused by incapability of accurately capturing detail features of the target object in an existing visual positioning method is effectively solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and image processing technology, and in particular, relates to a visual positioning method based on text mask supervision and image-text feature fusion. Background Art

[0002] Visual Grounding (VG) aims to use natural language and images as input to output objects described in natural language in images, usually combining object detection technology and natural language processing technology. Early research on visual grounding mainly includes two-stage methods and single-stage methods. The main difference between them is whether a region proposal network is generated, while the processing methods of other parts are basically the same. Convolutional neural networks (CNN) are used to extract image features, long short-term memory neural networks (LSTM) are used to extract text features, and then efforts are made to design better feature fusion modules for feature alignment. Finally, the length, width and center point coordinates of the positioning box and the corresponding confidence are regressed, and the most appropriate positioning box is selected using the confidence. In recent years, with the emergence of Transformer and its series of variant models, inspired by its powerful encoding ability and compatibility with multiple modal data in the form of input with token sequences as input, new visual grounding methods based on Transformer have emerged.

[0003] Transformer was first proposed in the field of Natural Language Processing (NLP) as a model specifically designed to solve natural language translation problems. With the help of its internal attention mechanism's ability to perceive global features, subsequent research designed the Vision Transformer (ViT) model for image feature extraction and the BERT model for text feature extraction based on Transformer. Since ViT does not rely on CNN and directly uses Transformer for image feature sequence prediction, while BERT uses stacked Transformer encoders to generate deep bidirectional language representations through a pre-training-fine-tuning paradigm, given the unique advantages of these models, Transformer has gradually become the basic paradigm for visual positioning research today.

[0004] Existing visual positioning methods often use these network architectures in combination, using ViT or CNN to extract image features, using BERT or LSTM to extract text features, and then designing a feature fusion module with an encoder-decoder architecture or an attention module for image-text matching to fuse image and text features, and finally treating it as a regression problem to regress the positioning box. However, this method often fails to accurately capture the detailed features of the target object, which may lead to deviations in positioning. For example, when a given descriptive expression is vague, but there are too many similar objects in the matching picture, the model may find it difficult to distinguish between these objects with similar levels of detail, which can easily lead to positioning ambiguity.

[0005] In order to solve the problem of positioning ambiguity caused by the inability of existing visual positioning methods to accurately capture the detailed features of the target object, some studies have been devoted to designing better feature fusion modules or using text for recursive step-by-step reasoning. However, due to the limited input text information, the performance improvement is not significant. Considering the importance of text information for positioning the target object, it is necessary to design a visual positioning method that fully utilizes text supervision to fuse image and text features. Summary of the invention

[0006] In view of this, the purpose of the present invention is to propose a visual positioning method based on text mask supervision and image-text feature fusion, following the basic paradigm of research in the field of visual positioning, and designing a visual positioning model in which text and image are first encoded, then feature aligned, and finally decoded for regression prediction of positioning frames. Mask text is constructed based on the input text and combined with the input image to generate multiple mask supervision heat maps, which are used as additional inputs of the decoder for the final regression prediction, so as to solve the problem of positioning ambiguity caused by the inability of existing visual positioning methods to accurately capture the detailed features of the target object.

[0007] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a visual positioning method based on text mask supervision image and text feature fusion, comprising the following steps:

[0008] Step 1: Construct a dataset of image-text pairs (I, T); the image-text pairs (I, T) consist of an image and a text describing the image content in the image;

[0009] Step 2: Construct a visual localization model for text mask supervision and text feature fusion; the visual localization model for text mask supervision and text feature fusion includes a visual encoder, a text encoder, a visual language alignment module, a text enhancement module and a cross-modal decoder; the visual encoder is used to extract image features through ViT; the text encoder is used to extract text features through BERT; the visual language alignment module is used to generate a mask supervision heat map according to the text features extracted by the text encoder and the image features extracted by the visual encoder, and the visual language alignment module includes two branches, the first branch includes a multi-head self-attention layer, a multi-head cross-modal attention layer and a convolutional layer, the second branch includes two multi-head attention layers, and the second branch takes the output of the first branch as input; the text enhancement module has a built-in bidirectional gated recurrent unit (Bidirectional Gated Recurrent Unit, Bi-GRU) with phrase attention, and the text enhancement module is used to enhance the key information of the input full text; the cross-modal decoder contains two multi-head attention layers and two FFN layers, which are used to receive the enhanced text features and mask supervision heat map, and perform iterative regression prediction on the original image together to achieve the final target positioning;

[0010] Step 3: Input the image-text pair (I, T) as the text mask supervised image-text feature fusion visual localization model, and perform two mask operations on the input text according to the set mask rules before input to form two masked texts. and Finally, we get an unmasked full text T and two masked texts. and After masking, the text encoder uses BERT to extract text features and obtains an unmasked full text feature F t And two masked text features and The visual encoder uses ViT to extract the image feature map F of image I i ;

[0011] Step 4: Combine the three text features F t , and And the image feature map F i Into the visual language alignment module, each text feature is respectively aligned with the image feature map F i Concatenate the input image-text feature pairs to obtain a full-text image-text feature pair (F t ,F i ) and two masked text-image feature pairs and The first branch in the visual language alignment module receives the image-text feature pairs and outputs the supervision heat map F of all target objects involved in the text. O The second branch uses the output of the first branch as the query vector Q and the value vector V, respectively, and combines them with the image feature map F i Calculate the attention weights and generate the mask supervision heat map F M ;

[0012] Step 5: Extract the unmasked full text feature F in step 3 separately t , and send it to the text enhancement module to enhance the key text and suppress the interference text, and finally obtain the enhanced text features

[0013] Step 6: Transform the image feature map F i , Mask supervision heat map F M And the enhanced text features are sent together to the cross-modal decoder to perform iterative regression prediction to achieve the final target positioning. Specifically, the first multi-head attention layer in the cross-modal decoder receives the initial learnable parameters And the enhanced text features Use attention calculation to convert the initial learnable parameters Add to the text information, and then store it with an intermediate parameter t1; the second multi-head attention layer receives the intermediate parameter t1 and the mask supervision heat map F M And the image feature map F i , calculate the correlation scores of the three, and obtain the visual feature t2 of the target object for text positioning; the visual feature t2 passes through the first FFN layer, and uses the MLP with ReLU activation function inside it to perform feature update, and a pre-set benchmark IOU value is compared with the feature confidence T output by the first FFN layer. i For comparison, when T i When <IOU, use new learnable parameters Store the current visual feature t2 and replace the initial learnable parameters Execute the next round of iterative prediction; when T i When ≥IOU, the updated visual features are input into the second FFN layer to enhance the feature expression and obtain the coordinates of the positioning center point (x i ,y i ) and length and width (w i ,h i ), and finally, draw a positioning frame based on the coordinate information to achieve visual positioning of text to image;

[0014] At this point, the visual positioning method based on text mask supervision and image and text feature fusion is completed.

[0015] Among them, in step 3, the process of performing two masking operations on the input text according to the set masking rules is as follows: use the NLTK library to extract the lexical attributes of the input text, and mask it twice continuously according to the attributes of each word in the input text. Each masking only masks one word on the basis of the previous masking.

[0016] Among them, the process of using BERT to extract text features by the text encoder in step 3 is as follows:

[0017] First, T, and As three input texts, T, and Each word ID in is represented as a one-hot vector, and all the one-hot vectors in a single text constitute the text encoding of the text; then, a [CLS] tag and a [SEP] tag are attached to the beginning and end of the text encoding respectively, and the learnable position embedding is added to the text encoding, where the [CLS] tag represents the category of the input text and [SEP] is used to divide different texts; finally, the text encoding with the additional tags is used as the input of BERT, and the text features F of the three input texts are obtained in turn. t , and

[0018] Furthermore, in step 4, the multi-head self-attention layer of the first branch in the visual language alignment module receives the image feature map F processed by the normalization layer. i , extract F i The correlation and dependency between each image block and other image blocks is used as the query vector Q i The value of the input text feature is used as a key-value pair (K t ,V t ) and the query vector Q i Then, the following formula is used to calculate the attention weight score of each pixel block in the text to the image, denoted as A T→I ,Right now:

[0019]

[0020] In the formula, Q i represents the i-th query vector, K t and V t Represents a key-value pair. It's K t The transpose of , d represents the dimension of the feature vector;

[0021] According to the attention weight score A T→I ={A1,A2,...,...,A n}, select the image pixel blocks corresponding to the first 10% attention weights and weight them to obtain the weighted features corresponding to the input text, then the weighted features are processed by the normalization layer and sent to the convolution layer to extract the weighted feature map and refine the features (feature refinement: use the convolution kernel in the convolution layer to slide on the feature map, each slide only focuses on a small area of ​​the image, extract the high-dimensional features of each small image block through convolution calculation, and then use the pooling operation to reduce the dimension, and finally retain the most significant features and filter out unnecessary information to achieve feature refinement. This is a prior art and the specific operation process will not be described in detail). In the image feature map F i The corresponding pixel blocks are labeled according to the weight size to generate a preliminary supervision heat map F that contains all target objects this time. O , this branch will be repeated N times, where N ≥ 3, to ensure that the three text features F t , and All of them participated in at least one operation, and each operation was based on the supervision heat map F obtained in the previous operation. O Do merge updates based on , that is:

[0022]

[0023] in, represents the supervision heat map obtained by the Nth operation, represents the supervised heat map obtained by the N+1th operation, where N refers to the number of image blocks into which the input image I is divided. Indicates the merging of the i-th image block of the N+1-th operation and the i-th image block of the N-th operation;

[0024] Then we enter the second branch of the visual language alignment module, F O After being fed into the second branch of the visual-language alignment module, F O First, it is input into the first multi-head attention layer as a key-value pair vector (K, V), and then compared with the full text F t The query vector Q represented by calculates the attention weight once and obtains Then, As the query vector and key vector (Q, K) are fed into the second multi-head attention layer, and F as the value vector V O Calculate the attention weight again and get Finally, F O , and The fusion is performed through the following formula:

[0025]

[0026] Among them, f(·) is the fusion function, which performs feature fusion through a maximum pooling operation and outputs the mask supervision heat map F M .

[0027] Further, the specific process of step 5 is as follows:

[0028] F t Represents bidirectional input and Then use the following formula to further adjust the text features:

[0029]

[0030] Among them, F t ′ represents the text feature of the bidirectional output of the bidirectional gated recurrent unit with phrase attention, E t and θ t is the encoder and parameters of a bidirectional gated recurrent unit with phrase attention; Represents a text sequence of length L and dimension C.

[0031] Then, phrase attention is used to adjust the text feature F t ′Assign weights, and the weight assignment is shown as follows:

[0032] φ=soft max(FC(F t ′))

[0033] Among them, FC(·) represents the fully connected layer. After obtaining the attention weight, in order to strengthen the weight of the text information related to the positioning target in the text and reduce the weight of irrelevant background information, the following formula is used to weight it to the adjusted text feature F t Above ':

[0034]

[0035] In the formula, Represents the enhanced text features, which enhances the distinction between key description text and other text by assigning different attention weights to different words.

[0036] Further, the specific process of step 6 is as follows:

[0037] In the first iteration, a learnable parameter is predefined is the initial representation of iterative regression, as the query Q tq Input to the first multi-head attention layer and the enhanced text features Represents a key-value pair vector Calculate text attention once and generate text representation This allows subsequent attention calculations of text and images to include learnable parameters To facilitate the next iterative calculation, the specific formula is as follows:

[0038]

[0039] in, and Represents a key-value pair consisting of enhanced text features. yes The transpose of , d represents the feature vector dimension, Represents a text sequence of length 1 and dimension C.

[0040] Then, the second multi-head attention layer receives the text representation t1 and the mask supervision heatmap F M And the image feature map F i , calculate the correlation score, using t1 and F M In F i Extract the target area of ​​interest and obtain the visual features of the positioning target Finally, the first FFN layer is used to update the features and refine the query positioning features to make them more focused on the target object and obtain the positioning confidence T i , and use t2 to update the initial learnable parameters The iterative update value t′ is obtained, and the calculation formula is as follows:

[0041]

[0042] Among them, LN represents the layer normalization operation;

[0043] The first FFN layer refines the query positioning features for secondary prediction, but before the prediction, a benchmark value is set in advance to compare the positioning confidence Ti with the benchmark IOU. i When <IOU, use new learnable parameters Store the current visual feature t2 and replace the initial learnable parameters Perform the next round of iterative prediction, the formula is as follows:

[0044]

[0045] Where LN represents the layer normalization operation and t′ is the iterative update value.

[0046] When T i When ≥IOU, it is sent to the second FFN layer for processing, and the internal MLP with ReLU activation function is used for prediction to obtain the coordinates of the positioning center point (x i ,y i ) and length and width (w i ,hi ), calculate the coordinates of the four vertices of the positioning box and draw the positioning box to achieve visual positioning of text to image. The vertex calculation formula is as follows:

[0047]

[0048] Where (x1, y1) represents the coordinates of the upper left corner of the positioning box, (x1, y2) represents the coordinates of the lower left corner of the positioning box, (x2, y1) represents the coordinates of the upper right corner of the positioning box, and (x2, y2) represents the coordinates of the lower right corner of the positioning box.

[0049] Since each iteration is based on the previous one, the weighted loss of L1 loss function and GIOU loss function is used as the training loss function to extract the coordinates of the positioning center point and the length and width b pr ={x i ,y i ,w i ,h i}, the real center point coordinates and length and width b tr ={x,y,w,h}, the overall training loss function is set as follows:

[0050]

[0051] in, represents the L1 loss function, represents the GIOU loss function, λ1 and λ2 are used to weight the two loss functions so that Keep between 0 and 1, and the calculation formulas are as follows:

[0052]

[0053] Among them, b min Indicates the minimum closed rectangular box area that can contain the predicted box and the real box, b pr and b tr They represent the predicted rectangular area and the real rectangular area respectively. The above loss function is used for training to achieve visual positioning of text to image.

[0054] Through the above-mentioned design scheme, the present invention can bring the following beneficial effects: the visual positioning method based on text mask supervision and image-text feature fusion proposed in the present invention uses ViT and BERT as the visual encoder and text encoder respectively. First, in order to give full play to the supervisory role of text in locating the target object in visual positioning, the input text is masked twice to obtain two masked texts. The masked expanded text can be used together with the full text to generate a more refined text supervision heat map using the visual language alignment module. Secondly, a text enhancement module is designed, which uses Bi-GRU with phrase attention to weight and suppress different words in the text, so that the model pays more attention to the text information related to the positioning target, while reducing the influence of background information irrelevant to the positioning target. Finally, the cross-modal decoder receives the enhanced text features and text supervision heat map. Figure 1 It performs iterative regression prediction on the image feature map, which effectively solves the problem of positioning ambiguity caused by the inability of existing visual positioning methods to accurately capture the detailed features of the target object, and can achieve better visual positioning effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to understand the present invention and do not constitute improper limitations of the present invention. In the drawings:

[0056] Figure 1 This is a framework diagram of the visual localization method based on text mask supervision and image-text feature fusion;

[0057] Figure 2 This is a schematic diagram of the visual language alignment module structure;

[0058] Figure 3 Schematic diagram of the cross-modal decoder structure. DETAILED DESCRIPTION

[0059] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the present invention is clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the present invention is not limited to the following embodiments, and the specific implementation method can be determined according to the technical scheme of the present invention and the actual situation. In order to avoid confusing the essence of the present invention, the known methods, processes and flows are not described in detail.

[0060] The visual positioning method based on text mask supervision and image-text feature fusion proposed in the present invention follows the basic paradigm of research in the field of visual positioning, and designs a visual positioning model in which text and image are first encoded, then feature aligned, and finally decoded and regressed to predict the positioning frame. First, the input text is masked, and two masked texts are constructed in sequence. The three texts are sent to BERT to extract their respective text features, and the image-text pairs are constructed one by one with the image features extracted by ViT, and sent to the visual language alignment module to generate the text supervision heat map of the key target. Secondly, the unmasked full text is extracted separately and sent to the text enhancement module, and the Bi-GRU with text attention is used to assign different weights to each word to obtain enhanced text features. Finally, the cross-modal decoder receives the enhanced text features and text supervision heat map, and performs iterative regression prediction on the original image together, and selects the target object referred to by the positioning text according to the coordinates and confidence obtained by regression. Through the synergy of the above three modules, the problem of positioning ambiguity caused by the inability to accurately capture the detailed features of the target object in the existing visual positioning method can be effectively solved, and a better visual positioning effect can be obtained.

[0061] Specifically, the visual positioning method based on text mask supervision and image-text feature fusion proposed in the present invention comprises the following steps:

[0062] Step 1: Visual localization research aims to find objects described in text in images based on descriptive text, so it involves both image and text. Collect datasets related to visual localization (such as RefCOCO, RefCOCO+, and ReferItGame, which are commonly used datasets of VG), which contain a large number of image-text pairs (I, T), such as a pasture photo with white cows and brown cows and an English text "A white cow in a pasture with brown cows".

[0063] Step 2: Design a visual positioning model architecture for text-mask supervision and image-text feature fusion, the model includes a visual encoder, a text encoder, a visual language alignment module, a text enhancement module and a cross-modal decoder; the visual encoder is used to extract image features through ViT; the text encoder is used to extract text features through BERT; the visual language alignment module is used to generate a mask supervision heat map according to the text features extracted by the text encoder and the image features extracted by the visual encoder, the visual language alignment module includes two branches, the first branch includes a multi-head self-attention layer, a multi-head cross-modal attention layer and a convolutional layer, the second branch includes two multi-head attention layers, and the second branch takes the output of the first branch as input; the text enhancement module has a built-in bidirectional gated recurrent unit with phrase attention, and the text enhancement module is used to enhance the key information of the input full text; the cross-modal decoder contains two multi-head attention layers and two FFN layers, which are used to receive the enhanced text features and text supervision heat maps, and perform iterative regression prediction on the original image together to achieve the final target positioning.

[0064] Step 3: Use the image-text pair (I, T) obtained in step 1 as the input of the model, and send I and T to the visual encoder and text encoder in step 2 respectively to extract image features and text features.

[0065] Given an input image in Represents an image with height H, width W and number of channels C, where C = 3 represents 3-channel color RGB. ViT is used as the backbone network of the visual encoder to obtain the image features of the input image Specifically, for the input image First, the linear projection is flattened into N non-overlapping image blocks to obtain a flattened image block sequence. in represents the number of image blocks, H and W represent the width and height of the input image I, respectively, and P represents the image resolution. Each flattened image block belongs to an image with height H, width W and number of channels C, where C = 3 represents 3-channel color RGB. The Transformer encoder in ViT receives a sequence of flattened image blocks. Perform global context reasoning and output an image feature map of the same size as the input

[0066] Given an input text T, use the NLTK library to extract the lexical attributes of the input text, judge the influence of each word on the sentence components according to the pre-set rules, select some words with less influence according to the rules, and perform two mask operations (the criterion for measuring the influence is that if the meaning of the original text is not affected by removing a word, then this word is a word with less influence), and generate two masked texts and The specific rules are set as follows.

[0067] In the composition of the entire sentence, prepositions, linking words and attributive words usually have little impact on the actual meaning of the sentence, and they are given priority when masking. If the sentence does not contain these types of words, auxiliary words, pronouns and numerals are secondary choices, because these types of words only play a role in reference assistance and modification, and proper masking does not affect the main meaning of the sentence. It should be noted that since adjectives and verbs have a great influence on the meaning of the sentence, they are never masked. If the sentence only contains these two types of words, no masking operation is performed. At this time, the masking operation actually obtains three full texts, namely

[0068] After masking, BERT is used to extract the respective text features. Specifically, the two masked texts and each word ID in the full text are first represented as a one-hot vector, and all the one-hot vectors in each text constitute the text encoding of the text. Then a [CLS] tag and a [SEP] tag are attached to the beginning and end of the text encoding, respectively, and the learnable position embedding is added to the text encoding. The [CLS] tag represents the category of the input text, and [SEP] is used to divide different texts. Finally, the text encoding with the additional tags is used as the input of BERT, and the text features of the three input texts are obtained in turn. and Where N t is the number of text tokens, C t It is the text feature dimension.

[0069] Step 4: Combine the three text features F t , and And the image feature map F i Each text feature will be concatenated with the image feature to create an input text-image feature pair, and a full text-image feature pair (F t ,F i ) and two masked text-image feature pairs and Next, we will use the full text and picture to compare (F t ,F i) as an example to introduce the processing process of the visual language alignment module. At this time, the spliced ​​text image feature pair F ti The formula is as follows:

[0070]

[0071] in, is the embedding representation of each word in the input text, It is the embedded representation of each image block in the input image feature map. The label [SEP] is used to separate each pair of input image-text feature pairs, which are then fed into the visual language alignment module.

[0072] The visual language alignment module consists of two branches. The first branch first receives a pair of image-text feature pairs (F t ,F i ), outputs the supervision heat map of all target objects involved in the text. The second branch uses the output of the first branch as the query vector Q and the value vector V and the image feature map F respectively. i Calculate the attention weights and generate an attention heatmap for only the key target objects. Specifically, the multi-head self-attention layer of the first branch receives the image feature map F processed by the normalization layer. i , extract F i The correlation and dependency between each image block and other image blocks is used as the query vector Q i The input text features are used as key vector and value vector pairs (K t ,V t ) and the query vector Q i Then, the following formula is used to calculate the attention weight score of each pixel block in the text to the image, denoted as A T→I ,Right now:

[0073]

[0074] In the formula, Q i represents the i-th query vector, K t and V t Represents a key-value pair, K t T It's K t The transpose of , d represents the dimension of the feature vector;

[0075] According to the attention weight score A T→I ={A1,A2,......,A n}, select the image pixel blocks corresponding to the first 10% attention weights and weight them to obtain the weighted features corresponding to the input text. Then, the weighted features are processed by the normalization layer and sent to the convolution layer to extract the weighted feature map and refine the features. i The corresponding pixel blocks are labeled according to the weight size to generate a preliminary supervision heat map F that contains all target objects this time. O , this branch will be repeated N times, where N ≥ 3, to ensure that the three text features F t , and All of them participated in at least one operation, and each operation was based on the supervision heat map F obtained in the previous operation. O Do merge updates based on , that is:

[0076]

[0077] in, represents the supervision heat map obtained by the Nth operation, represents the supervised heat map obtained by the N+1th operation, where N refers to the number of image blocks into which the input image I is divided. It represents the merging of the i-th image block of the N+1-th operation and the i-th image block of the N-th operation.

[0078] Then we enter the second branch and use two consecutive multi-head attention layers to improve F O , generate mask supervision heat map F M Specifically, F O After being sent to the second branch, it is first input into the first multi-head attention layer as a key-value pair vector (K, V) and combined with the full text feature F t The query vector Q represented by calculates the attention weight once and obtains Then, As the query vector and key vector (Q, K) are fed into the second multi-head attention layer, and F as the value vector V O Calculate the attention weight again and get Finally, F O , and The fusion is performed through the following formula:

[0079]

[0080] Among them, f(·) is the fusion function, which performs feature fusion through a maximum pooling operation and outputs the generated mask supervision heat map F M .

[0081] Step 5: Extract the unmasked full text feature F in step 3 separately t, and send it to the text enhancement module, and further enhance it using the bidirectional gated recurrent unit (Bi-GRU) with phrase attention. Since Bi-GRU is a bidirectional network, F t Represents bidirectional input and Then use the following formula to further adjust the text features:

[0082]

[0083] Among them, F t ′ represents the text feature of the bidirectional output of the bidirectional gated recurrent unit with phrase attention, E t and θ t is the encoder and parameters of a bidirectional gated recurrent unit with phrase attention; Represents a text sequence of length L and dimension C.

[0084] Then, phrase attention is used to adjust the text feature F t ′Assign weights, and the weight assignment is shown as follows:

[0085] φ=softmax(FC(F t ′))

[0086] Among them, FC(·) represents the fully connected layer. After obtaining the attention weight, in order to strengthen the weight of the text information related to the positioning target in the text and reduce the weight of irrelevant background information, the following formula is used to weight it to the adjusted text feature F t Above ':

[0087]

[0088] In the formula, Represents the enhanced text features, which enhances the distinction between key description text and other text by assigning different attention weights to different words.

[0089] Step 6: Transform the image feature map F i , Mask supervision heat map F M And the enhanced text features are sent together to the cross-modal decoder to perform iterative regression prediction to achieve the final target positioning. Specifically, in the first iteration, a learnable parameter is predefined is the initial representation of iterative regression, as the query Q tq Input to the first multi-head attention layer and the enhanced text features Represents a key-value pair vector Calculate text attention once and generate text representation This allows subsequent attention calculations of text and images to include learnable parameters To facilitate the next iterative calculation, the specific formula is as follows:

[0090]

[0091] in and Represents a key-value pair consisting of enhanced text features. yes The transpose of , d represents the feature vector dimension, represents a text sequence of length 1 and dimension C; then, the second multi-head attention layer receives the text representation t1 and the mask supervision heat map F M And the image feature map F i , calculate the correlation score, using t1 and F M In F i Extract the target area of ​​interest and obtain the visual features of the positioning target Finally, the first FFN layer is used to update the features and refine the query positioning features to make them more focused on the target object and obtain the positioning confidence T i , and use t2 to update the initial learnable parameters The iterative update value t′ is obtained, and the calculation formula is as follows:

[0092]

[0093] Among them, LN represents the layer normalization operation.

[0094] The first FFN layer refines the query positioning features for secondary prediction. Before prediction, a benchmark IOU value is pre-set and the positioning confidence T i Compared with the baseline IOU, when T i When <IOU, use new learnable parameters Store the current visual feature t2 and replace the initial learnable parameters Perform the next round of iterative prediction, the formula is as follows:

[0095]

[0096] Where LN represents the layer normalization operation, and t′ is the iterative update value;

[0097] When T i When ≥IOU, it is sent to the second FFN layer for processing, and the internal MLP with ReLU activation function is used for prediction to obtain the coordinates of the positioning center point (x i ,y i ) and length and width (wi ,h i ), calculate the coordinates of the four vertices of the positioning box and draw the positioning box to achieve visual positioning of text to image. The vertex calculation formula is as follows:

[0098]

[0099] Where (x1, y1) represents the coordinates of the upper left corner of the positioning box, (x1, y2) represents the coordinates of the lower left corner of the positioning box, (x2, y1) represents the coordinates of the upper right corner of the positioning box, and (x2, y2) represents the coordinates of the lower right corner of the positioning box.

[0100] Since each iteration is based on the previous one, the weighted loss of L1 loss function and GIOU loss function is used as the training loss function to extract the coordinates of the positioning center point and the length and width b pr ={x i ,y i ,w i ,h i}, the real center point coordinates and length and width b tr ={x,y,w,h}, the overall training loss function is set as follows:

[0101]

[0102] in, represents the L1 loss function, represents the GIOU loss function, λ1 and λ2 are used to weight the two loss functions so that Keep between 0 and 1, and the calculation formulas are as follows:

[0103]

[0104] Among them, b min Indicates the minimum closed rectangular box area that can contain the predicted box and the real box, b pr and b tr They represent the predicted rectangular area and the real rectangular area respectively. The above loss function is used for training to achieve visual positioning of text to image.

Claims

1. A visual positioning method based on text mask supervision and image-text feature fusion, characterized in that: The following steps are involved: Step 1: Construct a dataset of image-text pairs (I, T); the image-text pairs (I, T) consist of an image and a text describing the image content in the image; Step 2: construct a visual localization model for text mask-supervised image-text feature fusion; the text mask-supervised image-text feature fusion visual localization model includes a visual encoder, a text encoder, a visual language alignment module, a text enhancement module and a cross-modal decoder; the visual encoder is used to extract image features through ViT; the text encoder is used to extract text features through BERT; the visual language alignment module is used to generate a mask supervision heat map according to the text features extracted by the text encoder and the image features extracted by the visual encoder, and the visual language alignment module includes two branches, the first branch includes a multi-head self-attention layer, a multi-head cross-modal attention layer and a convolutional layer, the second branch includes two multi-head attention layers, and the second branch takes the output of the first branch as input; The text enhancement module has a built-in bidirectional gated recurrent unit with phrase attention, and the text enhancement module is used to enhance the key information of the input full text; the cross-modal decoder contains two multi-head attention layers and two FFN layers, and the cross-modal decoder is used to receive the enhanced text features and mask supervision heat map, and perform iterative regression prediction on the original image together to achieve the final target positioning; Step 3: Input the image-text pair (I, T) as the text mask supervised image-text feature fusion visual localization model, and perform two mask operations on the input text according to the set mask rules before input to form two masked texts. and Finally, we get an unmasked full text T and two masked texts. and After masking, the text encoder uses BERT to extract text features and obtains an unmasked full text feature F t And two masked text features and The visual encoder uses ViT to extract the image feature map F of image I i ; Step 4: Combine the three text features F t , and And the image feature map F i Into the visual language alignment module, each text feature is respectively aligned with the image feature map F i Concatenate the input image-text feature pairs to obtain a full-text image-text feature pair (F t ,F i ) and two masked text-image feature pairs and The first branch in the visual language alignment module receives the image-text feature pairs and outputs the supervision heat map F of all target objects involved in the text. O The second branch uses the output of the first branch as the query vector Q and the value vector V, respectively, and combines them with the image feature map F i Calculate the attention weights and generate the mask supervision heat map F M ; Step 5: Extract the unmasked full text feature F in step 3 separately t , and send it to the text enhancement module to enhance the key text and suppress the interference text, and finally obtain the enhanced text features Step 6: Transform the image feature map F i , Mask supervision heat map F M And the enhanced text features are sent together to the cross-modal decoder to perform iterative regression prediction to achieve the final target positioning. Specifically, the first multi-head attention layer in the cross-modal decoder receives the initial learnable parameters And the enhanced text features Use attention calculation to convert the initial learnable parameters Add to the text information, and then store it with an intermediate parameter t1; the second multi-head attention layer receives the intermediate parameter t1 and the mask supervision heat map F M And the image feature map F i , calculate the correlation scores of the three, and obtain the visual feature t2 of the target object for text positioning; the visual feature t2 passes through the first FFN layer, and uses the MLP with ReLU activation function inside it to perform feature update, and a pre-set benchmark IOU value is compared with the feature confidence T output by the first FFN layer. i For comparison, when T i When <IOU, use new learnable parameters Store the current visual feature t2 and replace the initial learnable parameters Execute the next round of iterative prediction; when T i When ≥IOU, the updated visual features are input into the second FFN layer to enhance the feature expression and obtain the coordinates of the positioning center point (x i ,y i ) and length and width (w i ,h i ), and finally, draw a positioning frame based on the coordinate information to achieve visual positioning of text to image; At this point, the visual positioning method based on text mask supervision and image and text feature fusion is completed.

2. The visual positioning method based on text mask supervision and image-text feature fusion according to claim 1 is characterized in that: In step 3, the process of performing two masking operations on the input text according to the set masking rules is as follows: use the NLTK library to extract the lexical attributes of the input text, and mask it twice continuously according to the attributes of each word in the input text. Each masking only masks one word on the basis of the previous masking.

3. The visual positioning method based on text mask supervision and image-text feature fusion according to claim 1 is characterized in that: The process of using BERT to extract text features in the text encoder in step 3 is as follows: First, T, and As three input texts, T, and Each word ID in is represented as a one-hot vector. All the one-hot vectors in a single text constitute the text encoding of the text. Then, a [CLS] tag and a [SEP] tag are attached to the beginning and end of the text encoding, respectively, and the learnable position embedding is added to the text encoding, where the [CLS] tag represents the category of the input text and [SEP] is used to divide different texts. Finally, the text encoding with the additional tags is used as the input of BERT to obtain the text features F of the three input texts in turn. t , and 4. The visual positioning method based on text mask supervision and image-text feature fusion according to claim 1 is characterized in that: In step 4, the multi-head self-attention layer of the first branch in the visual language alignment module receives the image feature map F after being processed by the normalization layer. i , extract F i The correlation and dependency between each image block and other image blocks is used as the query vector Q i The value of the input text feature is used as a key-value pair (K t ,V t ) and the query vector Q i Then, the following formula is used to calculate the attention weight score of each pixel block in the text to the image, denoted as A T→I ,Right now: In the formula, Q i represents the i-th query vector, K t and V t Represents a key-value pair. It's K t The transpose of , d represents the dimension of the feature vector; According to the attention weight score A T→I ={A1,A2,......,A n }, A n It represents the nth attention weight score. The image pixel blocks corresponding to the first 10% attention weights are selected and weighted with the weights to obtain the weighted features corresponding to the input text. Then, the weighted features are processed by the normalization layer and sent to the convolution layer to extract the weighted feature map and refine the features. i The corresponding pixel blocks are labeled according to the weight size to generate a preliminary supervision heat map F that contains all target objects this time. O , this branch will be repeated N times, where N ≥ 3, to ensure that the three text features F t , and All of them participated in at least one operation, and each operation was based on the supervision heat map F obtained in the previous operation. O Do merge updates based on , that is: in, represents the supervision heat map obtained by the Nth operation, represents the supervised heat map obtained by the N+1th operation, where N refers to the number of image blocks into which the input image I is divided. Indicates the merging of the i-th image block of the N+1-th operation and the i-th image block of the N-th operation; Then we enter the second branch of the visual language alignment module, F O After being fed into the second branch of the visual-language alignment module, F O First, it is input into the first multi-head attention layer as a key-value pair vector (K, V) and combined with the full text feature F t The query vector Q represented by calculates the attention weight once and obtains Then, As the query vector and key vector (Q, K) are fed into the second multi-head attention layer, and F as the value vector V O Calculate the attention weight again and get Finally, F O , and The fusion is performed through the following formula: Among them, f(·) is the fusion function, which performs feature fusion through a maximum pooling operation and outputs the mask supervision heat map F M .

5. The visual positioning method based on text mask supervision and image-text feature fusion according to claim 1 is characterized in that: Step 5 specifically includes: F t Represents bidirectional input and Then use the following formula to further adjust the text features: Among them, F t ′ represents the text feature of the bidirectional output of the bidirectional gated recurrent unit with phrase attention, E t and θ t is the encoder and parameters of a bidirectional gated recurrent unit with phrase attention; Represents a text sequence of length L and dimension C; Then, phrase attention is used to adjust the text feature F t ′Assign weights, and the weight assignment is shown as follows: φ=softmax(FC(F t ′)) Among them, FC(·) represents the fully connected layer. After obtaining the attention weight, in order to strengthen the weight of the text information related to the positioning target in the text and reduce the weight of irrelevant background information, the following formula is used to weight it to the adjusted text feature F t Above ': In the formula, Represents the enhanced text features, which enhances the distinction between key description text and other text by assigning different attention weights to different words.

6. The visual positioning method based on text mask supervision and image-text feature fusion according to claim 1 is characterized in that: Step 6 specifically includes: In the first iteration, a learnable parameter is predefined is the initial representation of iterative regression, as the query Q tq Input to the first multi-head attention layer and the enhanced text features Represents a key-value pair vector Calculate text attention once and generate text representation This allows subsequent attention calculations of text and images to include learnable parameters To facilitate the next iterative calculation, the specific formula is as follows: in and Represents a key-value pair consisting of enhanced text features. yes The transpose of , d represents the feature vector dimension, represents a text sequence of length 1 and dimension C; then, the second multi-head attention layer receives the text representation t1 and the mask supervision heat map F M And the image feature map F i , calculate the correlation score, using t1 and F M In F i Extract the target area of ​​interest and obtain the visual features of the positioning target Finally, the first FFN layer is used to update the features and refine the query positioning features to make them more focused on the target object and obtain the positioning confidence T i , and use t2 to update the initial learnable parameters The iterative update value t′ is obtained, and the calculation formula is as follows: Among them, LN represents the layer normalization operation; The first FFN layer refines the query positioning features for secondary prediction. Before prediction, a benchmark IOU value is pre-set and the positioning confidence T i Compared with the baseline IOU, when T i When <IOU, use new learnable parameters Store the current visual feature t2 and replace the initial learnable parameters Perform the next round of iterative prediction, the formula is as follows: Where LN represents the layer normalization operation, and t′ is the iterative update value; When T i When ≥IOU, it is sent to the second FFN layer for processing, and the internal MLP with ReLU activation function is used for prediction to obtain the coordinates of the positioning center point (x i ,y i ) and length and width (w i ,h i ), calculate the coordinates of the four vertices of the positioning box and draw the positioning box to achieve visual positioning of text to image. The vertex calculation formula is as follows: Where (x1, y1) represents the coordinates of the upper left corner of the positioning box, (x1, y2) represents the coordinates of the lower left corner of the positioning box, (x2, y1) represents the coordinates of the upper right corner of the positioning box, and (x2, y2) represents the coordinates of the lower right corner of the positioning box. Since each iteration is based on the previous one, the weighted loss of L1 loss function and GIOU loss function is used as the training loss function to extract the coordinates of the positioning center point and the length and width b pr ={x i ,y i ,w i ,h i }, the real center point coordinates and length and width b tr ={x,y,w,h}, the overall training loss function is set as follows: in, represents the L1 loss function, represents the GIOU loss function, λ1 and λ2 are used to weight the two loss functions so that Keep between 0 and 1, and the calculation formulas are as follows: Among them, b min Indicates the minimum enclosed rectangular box area that can contain the predicted box and the real box, b pr and b tr They represent the predicted rectangular area and the real rectangular area respectively. The above loss function is used for training to achieve visual positioning of text to image.

Citation Information

Cited By

  • Apple fruit detection method in complex scene based on multi-modal fusion technology

    CN119851264A

  • User intention-driven controllable image segmentation and semantic capture method

    CN119904868A

  • Scene character recognition pre-training method and system based on language perception

    CN120544176A

  • Visual positioning method, visual positioning device and computer storage medium

    CN121600062A

  • Video generation method and device based on multi-stream cross attention, equipment and medium

    CN122640609A