A one-stage visual positioning model construction method based on multi-step reasoning

Abstract

The application discloses a one-stage visual positioning model construction method based on multi-step reasoning, and the constructed model is composed of a text content coding module, an image coding module, an image-text interaction module and a positioning module. The text content coding module is used for coding the text; the image coding module is used for coding the input image; the image-text interaction module is used for interacting the image and the text coding, and finally, the positioning module is used for predicting the image content bounding box corresponding to the text from the multi-modal features. The one-stage visual positioning model constructed by the application introduces a new image-text interaction strategy, greatly reduces the redundant information in the data reduction with the help of the mutual guidance of the text and the image, and improves the efficiency and accuracy of the visual positioning.

Description

Technical Field

[0001] This invention relates to the field of visual positioning, specifically a method for constructing a one-stage visual positioning model based on multi-step reasoning. The constructed model significantly improves the positioning accuracy in multimodal image and text scenarios, while also possessing real-time performance and robustness. It can typically be applied in scenarios such as question answering models, autonomous driving, and multimodal retrieval. Background Technology

[0002] Image-text understanding is a key technology in the multimodal domain. Existing image-text modeling methods cannot achieve true image-text understanding without fine-grained alignment between images and text. Visual localization is a crucial step in establishing fine-grained alignment between image and text data. Given an image and a piece of text associated with it, it can accurately locate the instance referred to by the text within the image. Visual localization methods can be flexibly integrated with downstream tasks, either serving as a pre-training foundation directly connected to downstream tasks or providing additional fine-grained alignment data to downstream models. This technology can also support multimodal retrieval and visual natural language navigation, making it an important foundation for multimodal artificial intelligence. Visual localization involves high complexity in handling image-text alignment, requiring both accuracy and robustness / real-time performance. Most existing two-stage methods differ in the scoring function, network architecture, multi-task learning, and training algorithms in the second step. However, the complex fusion module cannot be jointly learned with the detector backbone, which may limit their capabilities in multimodal inference. One-stage methods focus on directly locating reference objects without generating candidate objects. Therefore, constructing a one-stage method model that incorporates an image-text encoding interaction strategy to significantly improve the accuracy of visual localization has high research value. Summary of the Invention

[0003] The purpose of this invention is to address the problem that existing visual localization models neglect the interaction between images and text. It proposes a one-stage visual localization model construction method based on multi-step reasoning, which enhances the accuracy of the model in generating localization prediction boxes by using text-guided image training and learning in the model.

[0004] The specific technical solution for achieving the objective of this invention is as follows:

[0005] A method for constructing a one-stage visual localization model based on multi-step reasoning, which uses a dataset containing the following during the construction process:

[0006] Image database I = I,...,I; and its query phrase or reference expression Q = {p} ,..., The bounding box of the image content corresponding to the phrase or quotation. Wherein is the number of phrases or reference expressions for a given image;

[0007] The constructed model consists of a text content encoding module, an image encoding module, an image-text interaction module, and a localization module. The text content encoding module is used to encode the text; the image encoding module is used to encode the input image; the image-text interaction module is used to interact with the image and text encoding; and finally, the localization module is used to predict the bounding box of the image content corresponding to the text based on the multimodal features after the interaction.

[0008] The construction process is as follows:

[0009] 1) Encode the text and construct a text content encoding module.

[0010] First, the input phrase or quotation expression is segmented into discrete word sequences by a tokenizer. Then, the BERT model is used to transform the discrete sequences into continuous text features with contextual semantics, which is formally defined as follows:

[0011] T = {t, t, ..., t}

[0012] Each element represents a vector representation of a word, where t cs It is a special tag that contains the entire sentence context representation, where N represents the length of the sequence;

[0013] 2) Encode the input image to construct an image encoding module.

[0014] The input image is encoded using Darknet-53, and the resulting image features are as follows:

[0015]

[0016] Where D v The dimensions of a visual feature are represented by H and W, which are the height and width of the visual feature.

[0017] 3) Image-text interaction module

[0018] Low-level interaction:

[0019] For a given intermediate visual feature map and text representation The module will t cs The text-conditional visual feature map is output as a conditional feature. The module calculates visual channel correction with text as a condition to optimize the semantic representation of the visual features. Then, visual spatial correction with text as a condition is performed to focus more on the region where the object related to the query sentence is located.

[0020] Specifically, firstly, By applying 2D average pooling, its spatial dimension is reduced, and its semantic representation is obtained. f and t csBy concatenating these signals, activating them with a sigmoid function, and then inputting them into a multilayer perceptron, the channel attention distribution can be obtained. Along the channel dimension, in and Element-wise multiplication is performed between them; the specific formula is as follows:

[0021]

[0022] A = σMLP[f,t]

[0023]

[0024] Where F′ represents the visual feature map after channel optimization, σ is the sigmoid activation, and ⊙ is the element-wise multiplication;

[0025] The text representation t is broadcast to each spatial location of the visual feature map and concatenated with the visual features; then, a linear layer is used to reduce the dimensionality of the feature channels to 1, and sigmoid activation is applied to obtain the attention matrix; the spatial correction of the text condition is expressed as:

[0026]

[0027]

[0028] in, For spatial attention diagrams, A spatially optimized visual feature map;

[0029] Advanced Interaction:

[0030] For a given visual feature map And the text feature sequence T, first Two-dimensional average pooling is applied, and then projected onto the text feature space through a linear transformation:

[0031]

[0032] in The merged visual features are represented; f is used as a feature of the visual marker and inserted into the text feature sequence; based on this, the multi-head self-attention (MHSA) module is used to model the visual-guided language representation:

[0033]

[0034]

[0035] in, This represents the feature sequence after multi-head self-attention; the text features here will be further fed into the next step of inference, while f uses the attention mechanism to query and aggregate useful text information from the text features for visual inference in the current inference step; the aggregated features are represented as f, and further fused back into the visual features;

[0036] Deep convolution fuses the connected features together; furthermore, a standard CBAM module is used to enhance feature representation capabilities; the fusion process is described as follows:

[0037]

[0038]

[0039] in The fused and modulated visual features are represented and fed into the next inference step;

[0040] 4) Positioning module

[0041] The prediction head is in anchor-free form. For a visualized feature map with 32×32=1024 locations, a 4D vector t is regressed. ,j =l ,j ,t ,j ,r ,j ,h ,j , representing the offset from the left, top, right, and bottom boundaries to the center of each location; and predicting a confidence score s for each predicted bounding box. ,j The box with the highest score will be the final prediction:

[0042] During training, the central 5×5 region is designated as the positive sample; a smoothed L1 loss is used to train bounding box regression; the smoothed L1 loss reduces the gradient and improves convergence while maintaining accurate predictions; to improve the model's generalization ability, the widely used GloU loss, which is unaffected by scale, is employed; the formulas for the two regression loss functions are as follows:

[0043]

[0044]

[0045] Where N p For the positive sample set, t p The result of the model prediction; This represents the offset of the truth box; to train confidence predictions, the model introduces a binary cross-entropy loss:

[0046]

[0047] Here N nRepresents the negative sample set; s p Indicates the confidence level of positive samples; s n This represents the confidence level of negative samples; the training objective of the model is:

[0048]

[0049] In test mode, the 4D offset vector t predicted by the model will be used. i,j =(l i,j ,t i,j ,r i,j ,h i,j Restore the bounding box position on the original image, that is, (x l ,y t ,x r ,y b ) pred , representing the coordinates of the top left and bottom right corners of the predicted bounding box.

[0050] This invention employs a one-stage visual localization model comprised of four parts: a text content encoding module, an image encoding module, an image-text interaction module, and an image-text generation module. By using different model structures to accomplish different tasks, the results of each part are made more accurate, avoiding the impact of differences between tasks on overall performance, ultimately making the entire visual localization model more accurate and efficient. The model utilizes three different loss functions. and and This allows for full utilization of the dataset during the optimization process, accelerating model optimization and reducing computational resource costs. The optimization process also automatically finds a suitable balance point, resulting in more accurate final results. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention;

[0052] Figure 2 Flowchart of an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Except for the content specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the field, and the present invention does not have any particular limitations.

[0054] See Figure 1 A method for constructing a one-stage visual localization model based on multi-step reasoning, which uses a dataset containing the following during the construction process:

[0055] Image database I = I1,...,IN ; and its query phrase / reference expression Q p ={p i} i＝1,...,M The bounding box of the image content corresponding to the phrase / quote. Wherein is the number of phrase / reference expressions for a given image;

[0056] The constructed model consists of a text content encoding module, an image encoding module, an image-text interaction module, and a localization module. The text content encoding module is used to encode the text; the image encoding module is used to encode the input image; the image-text interaction module is used to interact with the image and text encoding; and finally, the localization module is used to predict the bounding box of the image content corresponding to the text based on the multimodal features after the interaction.

[0057] The construction process is as follows:

[0058] 1) Encode the text and construct a text content encoding module.

[0059] First, the input phrase or quotation expression is segmented into discrete word sequences by a tokenizer. Then, the BERT model is used to transform the discrete sequences into continuous text features with contextual semantics, which is formally defined as follows:

[0060]

[0061] Each element represents a vector representation of a word, where t cls N is a special tag that contains a representation of the entire sentence's context. t Indicates the length of the sequence;

[0062] 2) Encode the input image to construct an image encoding module.

[0063] The input image is encoded using Darknet-53, and the resulting image features are as follows:

[0064]

[0065] Where D v The dimensions of a visual feature are represented by H and W, which are the height and width of the visual feature.

[0066] 3) Image-text interaction module

[0067] Low-level interaction:

[0068] For a given intermediate visual feature map and text representation The module will t clsThe text-conditional visual feature map is output as a conditional feature. The module calculates visual channel correction with text as a condition to optimize the semantic representation of the visual features. Then, visual spatial correction with text as a condition is performed to focus more on the region where the object related to the query sentence is located.

[0069] Specifically, firstly, By applying 2D average pooling, its spatial dimension is reduced, and its semantic representation is obtained. f and t cls By concatenating these signals, activating them with a sigmoid function, and then inputting them into a multilayer perceptron, the channel attention distribution can be obtained. Along the channel dimension, in and Element-wise multiplication is performed between them; the specific formula is as follows:

[0070]

[0071] A c =σ(MLP([f,t) cls ]))

[0072]

[0073] Where F′ represents the visual feature map after channel optimization, σ is the sigmoid activation, and ⊙ is the element-wise multiplication;

[0074] Represent the text as t cls Broadcast to each spatial location of the visual feature map and concatenate it with the visual features; then, a linear layer is used to reduce the dimensionality of the feature channels to 1, and sigmoid activation is applied to obtain the attention matrix; the text-conditional spatial correction is expressed as:

[0075]

[0076]

[0077] in, For spatial attention diagrams, A spatially optimized visual feature map;

[0078] Advanced Interaction:

[0079] For a given visual feature map And the text feature sequence T, first Two-dimensional average pooling is applied, and then projected onto the text feature space through a linear transformation:

[0080]

[0081] in The merged visual features are represented; f is used as a feature of the visual marker and inserted into the text feature sequence; based on this, the multi-head self-attention (MHSA) module is used to model the visual-guided language representation:

[0082]

[0083]

[0084] in, This represents the feature sequence after multi-head self-attention; the text features here will be further fed into the next step of inference, while f uses the attention mechanism to query and aggregate useful text information from the text features for visual inference in the current inference step; the aggregated features are represented as f, and further fused back into the visual features;

[0085] Deep convolution fuses the connected features together; furthermore, a standard CBAM module is used to enhance feature representation capabilities; the fusion process is described as follows:

[0086]

[0087]

[0088] in The visual features represented by the fused modulation are fed into the next inference step; DWConv obtains new feature maps by convolution layer by layer according to depth (number of channels), and uses 1×1 convolution to recombine these new feature maps, and the spatial information is re-acquired by the inverse BP neural network.

[0089] 4) Positioning module

[0090] The prediction head is in anchor-free form. For a visualized feature map with 32×32=1024 locations, a 4D vector t is regressed. ,j =l ,j ,t ,j ,r ,j ,h ,j , representing the offset from the left, top, right, and bottom boundaries to the center of each location; and predicting a confidence score s for each predicted bounding box. ,j The box with the highest score will be the final prediction:

[0091] During training, the central 5×5 region is designated as the positive sample; a smoothed L1 loss is used to train bounding box regression; the smoothed L1 loss reduces the gradient and improves convergence while ensuring accurate prediction; to improve the model's generalization ability, the widely used GIoU loss, which is unaffected by scale, is employed; the formulas for the two regression loss functions are as follows:

[0092]

[0093]

[0094] Where N p For the positive sample set, t p The result of the model prediction; This represents the offset of the truth box; to train confidence predictions, the model introduces a binary cross-entropy loss:

[0095]

[0096] Here N n Represents the negative sample set; s p Indicates the confidence level of positive samples; s n This represents the confidence level of negative samples; the training objective of the model is:

[0097]

[0098] Example

[0099] Given the reference expression: {"umbrellas over two people's heads"}, see the image below. Figure 2 .

[0100] First, the input text is encoded by encoding each word into a 768D vector, which is then input into the text encoding model BERT to obtain the encoded information T.

[0101] The input image was resized to 3×256×256, and a pre-trained Darknet-53 on COCO object detection was used as the visual encoder. Here, the visual features from the 102nd convolutional layer, with a size of 32×32×256, were employed. The original visual feature map was then fed into a visual input with a 1×1 convolutional layer to output the visual features. Visual feature maps are obtained by performing low-level interactions between visual and textual features. And text feature sequence T.

[0102] In high-order interactions, the interaction between text and images is treated as a multi-round inference process. In each round, the subquery learner references textual conditional visual features and constructs a subquery that gradually reduces reference ambiguity. Textual features guide visual features in each round. Figure 2 In the first round, image encoding identifies multiple object bounding boxes; in the second round, the word "umbrella" in the sentence filters the anchors for image recognition; in the third round, the word "two people" in the sentence further filters out bounding boxes relevant to the task; in the final round, the word "above" in the sentence locates the bounding box on the umbrella above the two people, completing the visual localization task of text information in the image.

Claims

1.A method for constructing a one-stage visual localization model based on multi-step reasoning, characterized in that, The method uses a dataset containing the following in the construction process: Image database I = I1,..., In N ; And its query phrase or quote expression Q p = {p i} i＝1,...,M ; image content bounding box corresponding to the phrase or quote expression where n is the number of phrase or quote expressions for a given image; The constructed model is composed of a text content encoding module, an image encoding module, an image-text interaction module, and a positioning module; the text content encoding module is used to encode the text; the image encoding module is used to encode the input image; the image-text interaction module is used to interact the image and the text encoding, and finally use the positioning module to predict the image content bounding box corresponding to the text according to the multi-modal features after interaction; The construction process is as follows: 1) encode the text to construct the text content encoding module First, the input phrase or reference expression is divided into discrete word sequences by a tokenizer, and then the discrete sequence is converted into continuous text features with context semantics by using the BERT model, which is defined as follows: Each element represents a vector representation of a word, where t cls is a special token that contains the entire sentence context representation, N t represents the length of the sequence; 2) encode the input image to construct the image encoding module The input image is encoded using Darknet-53, and the obtained image features are as follows: where D v represents the dimension of the visual feature, H, W are the height and width of the visual feature; 3) image-text interaction module Low-order interaction: For a given intermediate visual feature map and text representation The module will t cls as a conditional feature, output the text-conditioned visual feature map; the module calculates the text-conditioned visual channel correction to optimize the semantic representation of the visual feature; and then the text-conditioned visual spatial correction to pay more attention to the area where the object related to the query sentence is located; Specifically, first, we apply 2-D average pooling to reduce its spatial dimension and get its semantic representation Apply 2-D average pooling to reduce its spatial dimension and get its semantic representation f and t cls Concatenate them and input them into a multi-layer perceptron after sigmoid activation to get the channel attention distribution Along the channel dimension, perform element-wise multiplication between and The specific formula is as follows: A c = σ(MLP([f, t cls l)) Where F' represents the visual feature map after channel optimization, sigma is the sigmoid activation, and is the element multiplication; Broadcast the text representation t to each spatial position of the visual feature map, and connect it with the visual feature; then use a linear layer to reduce the feature channel to 1, and apply sigmoid activation to get the attention matrix; the spatial correction expression of the text condition is: wherein, a spatial attention map, a spatially optimized visual feature map; High-order interaction: For a given visual feature map and text feature sequence T, first apply two-dimensional average pooling and project it to the text feature space by a linear transformation: wherein denotes the merged visual features; f is taken as one feature of the visual marker and inserted into the sequence of text features; on this basis, the multi-head self-attention (MHSA) module is adopted to model the language representation under the guidance of the vision: wherein, denotes the multi-head self-attended feature sequence; the text features here are further sent to the next reasoning step, and f queries and aggregates useful text information from the text features by means of the attention mechanism, for visual reasoning in the current reasoning step; the aggregated features are denoted as f, and are further fused back into the visual features; Deep convolution fuses the connected features together; in addition, a standard CBAM module is used to enhance the feature representation capability; the fusion process is expressed as: wherein represents the fused modulated visual features and is fed to the next inference step; 4) positioning module The prediction head is in the form of Anchor-free, for a 32x32=1024 locations visualized feature map, regress a 4D vector t i,j =(l i,j ,t i,j ,r i,j ,h i,j ), which represents the offset from the left, top, right and bottom boundary to the center of each location; and predict a confidence score s ,j for each predicted bounding box, the highest score box will be the final prediction: During the training process, the center 5*5 region is designated as the positive sample; in order to train the bounding box regression, the smooth L1 loss is used; the smooth L1 loss reduces the gradient in the case of accurate prediction, and improves the convergence; in order to improve the generalization ability of the model, the widely used GIoU loss which is not affected by the scale is used; the formulas of the two regression loss functions are as follows: where N is the positive sample set, and t is the result predicted by the model. is the offset of the true value box; in order to train the confidence prediction, the model introduces a binary cross-entropy loss: Here N n represents the negative sample set; s represents the confidence of the positive sample; s n represents the confidence of the negative sample; the training target of the model is:

Images