A visual positioning method based on multi-modal feature alignment

Abstract

The application provides a visual positioning method based on multi-modal feature alignment. The method comprises the following steps: inputting a picture and a corresponding referential expression into a feature extraction module to extract multi-scale grid features and text features respectively, and generating region features by using a Deformable DETR model and the grid features; inputting the text features and the region features into a Transformer encoder after connection to generate learnable tokens; obtaining multi-modal features by scaling dot product attention operation of the text features and the grid features; and performing target segmentation mask prediction processing according to the multi-modal features and the learnable tokens to obtain a predicted mask and taking the predicted mask as a visual positioning result of input data. The method solves the problem of lack of object-level information of grid features, enhances feature representation, and effectively improves segmentation performance in complex query or complex picture scenes. Global and local features are fused by using a hybrid operation, multi-modal features are enriched, and semantic alignment is effectively realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual localization method based on multimodal feature alignment. Background Technology

[0002] Visual localization is a typical research task in the visual-linguistic domain, aiming to locate the position of the referred object in an image based on the referential representation. Depending on the granularity of localization, visual localization tasks can be divided into referential representation understanding and referential image segmentation, which locate the target using bounding boxes and binary masks, respectively.

[0003] Referential representation refers to the natural language expression used to describe specific objects in a scene. This expression typically includes descriptions of the target, such as color, shape, and location, to accurately locate or describe regions of interest in an image.

[0004] With the rapid development of big data technology, the internet, and social media, multimodal data such as images, text, and videos are ubiquitous, and single-modal text or image information can no longer meet people's needs. Therefore, interdisciplinary research between computer vision and natural language processing, two major fields of deep learning, has become a hot topic. Unlike tasks that only involve a single modality, multimodal tasks require more granular information extraction from data of two or more different modalities. At the same time, they also need to analyze and reason about the relationships between different modalities. That is, multimodal models must not only fully understand the content of a single modality, but also bridge the semantic gap between different modalities, effectively modeling and interacting with multimodal information.

[0005] Since 2015, combining vision and language has become a hot research topic, with continuous development in research connecting computer vision and natural language processing. Many typical research tasks have emerged, such as visual question answering, image captioning, and visual language navigation. One representative task is visual localization. Based on the granularity of localization, visual localization tasks can be divided into referential expression understanding and referential image segmentation. Although recent research on visual localization has made some progress, the semantic gap between different modalities still presents significant challenges. Models need to perform intra-modal and inter-modal learning; that is, they must not only fully understand the content of the image and the content of the referential expression, but also associate the information provided by different modalities and align the same semantic information to ultimately locate the correct referent in the image. Therefore, the core problem of visual localization is multimodal fusion and alignment.

[0006] Currently, existing visual localization methods employ candidate region features generated by object detectors or grid features generated by backbone networks as visual features for images. While region features provide object-level information, they may lack contextual information, hindering the extraction of fine-grained image details. Grid features, while enriching contextual information, lack object-level information, making it difficult to perceive and distinguish multiple objects within an image. Therefore, a key challenge is to find a way to leverage both the object-level information inherent in region features for object relationship reasoning and the fine-grained contextual semantic information provided by grid features to offer richer visual feature representations.

[0007] In feature fusion, existing methods suffer from insufficient multimodal feature fusion. Early methods mapped visual and textual features to the same feature space and fused them using simple concatenation and addition operations. With the widespread application of attention mechanisms in natural language processing and computer vision, some researchers introduced them into multimodal feature fusion, achieving some performance improvements compared to earlier work. More recently, some researchers have proposed earlier fusion methods, performing multimodal fusion during image encoding, which further improves performance. However, most methods still employ a single convolution or attention mechanism for feature fusion. Because convolution and attention operations have different characteristics—attention focuses on global low-frequency features, while convolution focuses on local high-frequency features—this single-feature fusion approach may lead to insufficient feature fusion. Therefore, designing a fusion method that simultaneously considers global low-frequency features and local high-frequency features remains an unexplored problem.

[0008] In existing denotation image segmentation methods, most network models employ an encoder-decoder structure. This involves first encoding visual and textual features separately, and then using a visual-language cross-modal decoder to fuse the features to predict the mask. Since cross-modal interaction only occurs after feature encoding, and the cross-modal decoder is only responsible for aligning visual and textual features, it may not be able to effectively utilize the rich Transformer layers in the encoder to mine useful multimodal context.

[0009] The existing LAVT model is a language-aware Transformer model that encodes visual and textual features together by performing early fusion of cross-modal features during the visual feature encoding stage. It leverages the Transformer's rich intermediate layers and powerful relational modeling capabilities to extract effective multimodal contextual information. Multi-scale multimodal feature maps are aggregated in a top-down manner, and finally, a lightweight mask predictor is used to obtain accurate segmentation results.

[0010] The disadvantages of the existing visual localization methods mentioned above include: the existing methods use grid features generated by backbone networks as visual features, which enriches the contextual information of the image, but lacks object-level information. Especially when the text description is more complex and it is necessary to perceive and reason about the relationship between multiple objects in the image, the grid features cannot effectively extract the relationship information between objects.

[0011] Meanwhile, existing visual localization methods mainly focus on the fusion of multimodal features, aiming to achieve semantic alignment between two modalities by designing different feature fusion methods. However, most methods process visual and linguistic information independently, employing only a single convolution or attention mechanism for feature fusion in the final stage. Due to the different characteristics of convolution and attention operations—attention operations focus more on global low-frequency features, while convolution operations focus more on local high-frequency features—a single fusion method may lead to insufficient feature fusion. Summary of the Invention

[0012] This invention provides a visual localization method based on multimodal feature alignment to improve the generalization ability of models in complex image scenes or language descriptions.

[0013] To achieve the above objectives, the present invention adopts the following technical solution.

[0014] A visual localization method based on multimodal feature alignment includes:

[0015] Obtain the input data to be visually localized, which includes an image and its corresponding denotation;

[0016] The input data is fed into the feature extraction module, which uses the BERT model to extract text features from the denotative expression, uses the Swin Transformer model to extract multi-scale grid features from the image, uses the Deformable DETR model and the grid features to generate region features, and then concatenates the text features and region features and inputs them into the Transformer encoder to generate a learnable token.

[0017] The text features and the multi-scale grid features are combined using a scaling dot product attention operation to obtain multimodal features;

[0018] Based on the multimodal features and the learnable token, a target segmentation mask is predicted to obtain a predicted mask, which is then used as the visual localization result of the input data.

[0019] Preferably, the input data is fed into a feature extraction module, which uses a BERT model to extract text features from the denotative expression, uses a Swin Transformer model to extract multi-scale grid features from the image, and uses a Deformable DETR model and grid features to generate region features, including:

[0020] The feature extraction module includes a visual feature extraction network and a text feature extraction network. The image is input into the visual feature extraction network in the feature extraction module. The visual feature extraction network uses a pre-trained Swing Transformer model to extract multi-scale grid features of the image. The grid features are combined with randomly initialized feature vectors and input into a pre-trained Deformable DETR model to obtain object queries and use them as region features. The multi-scale grid features and region features are used as visual features, which are multi-channel tensors.

[0021] The denotation corresponding to the image is input into the text feature extraction network in the feature extraction module. The text feature extraction network uses a pre-trained BERT model to obtain the hidden layer representation of the text and extract the text features of the denotation.

[0022] Preferably, the step of concatenating the text features and region features and then inputting them into the Transformer encoder to generate a learnable token includes:

[0023] The object-aware module concatenates the text features and region features and inputs them into a Transformer encoder to extract the object-level information of the query perception, generate a learnable token, and use the learnable token as the target query for subsequent mask prediction.

[0024] Preferably, the step of obtaining multimodal features by scaling dot product attention operation from the text features and the multi-scale grid features includes:

[0025] The hybrid fusion module includes a convolutional modulation branch, an attention fusion branch, and a hybrid fusion operation branch. The input to the convolutional modulation branch is visual features and text features after pooling. The visual features and text features are multiplied by a dot and then fed into the network constructed by the convolutional operation to obtain the fused multimodal features. The input to the attention fusion branch is visual features q and text features k and v after pooling. Q, K, and V are obtained through linear transformation. Attention weights are calculated by applying a dot product and Softmax normalization to Q and K. For each position or channel, the attention weight determines its contribution to the fusion result. The attention weights are weighted and summed with respect to V to obtain the fused multimodal features. The hybrid fusion operation branch mixes and fuses the two fused multimodal features through dynamic convolution and linear transformation operations to obtain the final multimodal features.

[0026] Preferably, the step of predicting a target segmentation mask based on the multimodal features and the learnable token to obtain a predicted mask, and using the predicted mask as the visual localization result of the input data, includes:

[0027] The final multimodal features and the learnable token are input into the mask prediction module. The multimodal features are concatenated along the channel dimension using bilinear interpolation upsampling. The concatenated feature vector is then multiplied by the learnable token and activated with sigmoid. The prediction result is binarized to obtain the prediction mask based on a threshold of 0.5. The prediction mask is used as the visual localization result. The loss function is a linear combination of BCE loss and Dice loss. The prediction mask contains 0s and 1s, where a 1 indicates that the pixel represents the target object. The prediction mask is then combined with the original image, and each pixel of the original image is adjusted according to the corresponding mask value to obtain the final segmentation result.

[0028] As can be seen from the technical solutions provided by the embodiments of the present invention above, the embodiments of the present invention propose a deep learning network model for reference expression segmentation based on multimodal feature hybrid fusion with enhanced object perception. An object perception module is introduced on the basis of grid features. This module takes the object query in the DETR-based target detector as the region feature and introduces a learnable token as the target object query. Then, the region feature, the learnable token and the text feature are input into the Transformer encoder to extract the object-level information of the target perception.

[0029] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A structural diagram of a deep learning network model framework for denotation representation segmentation provided in an embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating a visual localization method based on multimodal feature alignment, provided in an embodiment of the present invention. Detailed Implementation

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0034] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0035] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0036] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0037] This invention addresses the problems of insufficient information contained in grid feature representations and inadequate multimodal feature fusion by constructing a deep learning network model for pointer representation segmentation based on object-aware enhancement and multimodal feature hybrid fusion. The structure of this pointer representation segmentation deep learning network model framework is as follows: Figure 1 As shown, it mainly includes: a feature extraction module, an object perception module, a hybrid fusion module, and a mask prediction module. The feature extraction module includes a visual feature extraction module and a text feature extraction module.

[0038] 1. Object-aware module

[0039] The object-aware module includes a pre-trained DETR-based object detector for generating object queries as region features, a learnable token for aggregating target object information and using it as a target query in subsequent segmentation operations, and a Transformer encoder whose input is a concatenation of the learnable token, region features, and text features for extracting language-aware object-level information.

[0040] 2. Hybrid Fusion Module

[0041] This module is responsible for enhancing multimodal feature fusion to achieve semantic alignment. Located at various stages of the visual encoder, this module employs an early fusion strategy, fusing multimodal features through convolutional modulation and attention fusion branches respectively. The outputs of the two branches are concatenated along the channel dimension and then further fused using a hybrid operation composed of dynamic convolutions.

[0042] The hybrid fusion module includes a convolutional modulation branch, an attention fusion branch, and a hybrid fusion operation. The convolutional modulation branch takes visual features and pooled text features as input. The visual and text features are multiplied and then fed into the network constructed by the convolutional operation to obtain the fused multimodal features. The attention branch takes visual and text features as input, using the visual features as q and the text features as k and v, respectively, to obtain the fused multimodal features. Next, the multimodal features are hybridized and fused through dynamic convolution and linear transformation operations to enhance semantic alignment. Finally, combining the multimodal features generated by the hybrid fusion module and the learnable token updated by the object awareness module, the token is treated as a target query, and a segmentation mask is predicted using mask classification.

[0043] 3. The feature extraction module consists of two parts: a visual feature extraction network and a text feature extraction network. The visual feature extraction network uses a pre-trained Swin Transformer model to extract visual features from images. The specific processing includes: loading the pre-trained Swin Transformer model; converting the format of the input image data to ensure it meets the expectations of the pre-trained model, typically a three-channel tensor; and inputting the converted image data into the Swin Transformer model to obtain visual feature representations, typically multi-channel tensors. The text feature extraction network uses a pre-trained BERT model to extract text features representing referential expressions. The specific processing includes: loading the BERT model and its corresponding tokenizer; using the BERT tokenizer to convert the text into a format acceptable to the BERT model, including the index sequence `input_ids` for each word after word segmentation, the binary mask `attention_mask` to identify valid tokens in the input sequence, and the token sequence `token_type_ids` to identify which sentence a token belongs to; and inputting the converted text data into the BERT model to obtain the hidden layer representation of the text, i.e., the text feature representation.

[0044] 4. Mask prediction module

[0045] The mask prediction module can be viewed as a segmentation head, responsible for generating prediction results, and is located in the last layer of the entire model. It receives multi-scale, multimodal features generated by the hybrid fusion module and learnable tokens updated by the object awareness module, generating prediction results through mask classification. The specific processing includes: concatenating multi-scale features along the channel dimension using bilinear interpolation upsampling; performing a dot product operation between the concatenated feature vector and the learnable token followed by sigmoid activation; and finally binarizing the prediction results with a threshold of 0.5. Therefore, the output prediction result is a binarized tensor.

[0046] Based on the aforementioned deep learning network model for referential representation segmentation, this invention provides a visual localization method based on multimodal feature alignment. The method's processing mainly consists of three stages. In the feature extraction stage, an early fusion strategy is employed, utilizing an object-aware module to extract object-level information related to the target, thereby enhancing feature representation. In the feature fusion stage, considering the different characteristics of convolutional and attention operations, feature fusion is performed using convolutional and attention branches respectively, followed by further fusion using a hybrid operation to enrich the multimodal feature fusion.

[0047] The processing flow of a visual localization method based on multimodal feature alignment provided in this embodiment of the invention is as follows: Figure 2 As shown, the processing steps include the following:

[0048] Step S10: Feature extraction.

[0049] The input image data and corresponding denotations come from commonly used standard datasets, such as RefCOCO, RefCOCO+, and RefCOCOg. The datasets provide images and their corresponding denotations, along with labeled segmentation masks for training and evaluating the model.

[0050] Referential expressions are natural language expressions that describe specific objects in a scene.

[0051] The image and its corresponding denotation are input into the feature extraction module. For the denotation, it is input into the BERT model to extract text features; for the image, it is input into the Swin Transformer to extract multi-scale grid features. Simultaneously, the grid features, combined with randomly initialized features, are input into a Deformable DETR model pre-trained on an object detection task to obtain object queries, which are then used as region features.

[0052] The text and region features described above are concatenated and input into a Transformer encoder to extract query-aware object-level information, generating a learnable token. Finally, this learnable token is used as the target query for subsequent mask prediction.

[0053] Step S20: Feature fusion.

[0054] The aforementioned multi-scale grid features are used as visual features with different spatial resolutions. After obtaining the text features and multi-scale grid features in the feature extraction stage, the pooled text features and visual features are input into the convolution modulation branch. After performing a dot product operation on the text features and visual features, the input is fed into the bottleneck structure formed by the 1*1 convolution operation to obtain multimodal features rich in local high-frequency information.

[0055] Visual features are represented as q, and textual features as k and v. A scaling dot product attention operation is used to obtain multimodal features rich in global low-frequency information. First, attention scores are calculated using q and k through a dot product operation. Then, the scores are normalized to attention weights using a softmax function. For each position or channel, the attention weight determines its contribution to the fusion result. Finally, the attention weights are weighted and summed over v. The two fused multimodal features are then mixed and fused through dynamic convolution and linear transformation operations to obtain the final multimodal features.

[0056] Step S30: Mask prediction.

[0057] The target segmentation mask is predicted based on the learnable tokens generated in the feature extraction stage and the multimodal features generated in the feature fusion stage. First, the multimodal features at multiple scales are combined in a top-down manner through operations such as upsampling and concatenation. Then, a dot product operation is performed with the learnable tokens and sigmoid activation is applied. Finally, the prediction result is binarized according to a threshold of 0.5 to obtain the prediction mask. The loss function is a linear combination of BCE loss and Dice loss.

[0058] The input data for the mask prediction module includes multi-scale, multi-modal features generated by the fusion module and learnable tokens updated by the object perception module. The output data is a binary tensor, i.e., the predicted mask. The predicted mask is the visual localization result of the input data. Since the predicted mask only contains 0s and 1s, a position of 1 indicates that the pixel represents the target object. The result of visual localization is the predicted binary mask. This mask is combined with the original image, and each pixel of the original image is adjusted according to the corresponding mask value to obtain the final segmentation result.

[0059] In the feature extraction stage, the BERT model used to extract text features and the Deformable DETR model used to obtain object queries are initialized using pre-trained parameters, and the parameter initialization process only needs to be performed once.

[0060] To extract object-level information for target perception, it is necessary to first introduce regional features containing object-level information, and then extract relevant object information for target perception based on the denotation expression. Therefore, the object perception module of this invention first generates object queries as regional features using the DETR model, then extracts information related to the target from text features and regional features using the Transformer encoder, and aggregates relevant object information using learnable tokens.

[0061] Because of the different characteristics of convolution and attention operations, a hybrid fusion module is proposed. First, multimodal features are fused using convolutional modulation and attention branches to obtain multimodal features rich in local high-frequency information and multimodal features rich in global low-frequency information, respectively. Then, dynamic convolution and linear transformation operations are used to fuse the two to obtain the final multimodal features, achieving better semantic alignment.

[0062] In summary, the method of this invention addresses the problem of insufficient object-level information in grid features, enhances feature representation, and effectively improves segmentation performance in complex query or complex image scenarios. To address the issue of insufficient feature fusion, a hybrid fusion module is designed, utilizing convolutional and attention branches for feature fusion respectively, and employing a hybrid operation to fuse global and local features, enriching multimodal features and effectively achieving semantic alignment.

[0063] This invention designs a deep learning network model for reference representation segmentation based on object-aware enhancement and multimodal feature fusion. An object-aware module is introduced on top of grid features, utilizing both the object-level information inherent in region features and the fine-grained contextual information provided by grid features, thus solving the problem of insufficient information from single features. The Transformer is used to extract object-level information for target perception, enhancing feature representation and effectively improving segmentation performance in complex query or image scenarios.

[0064] Most existing methods employ a single convolutional or attention mechanism for feature fusion. Due to the different characteristics of convolutional and attention operations—attention operations focus more on global low-frequency features, while convolutional operations focus more on local high-frequency features—a single fusion approach may result in insufficient feature fusion. Therefore, this invention designs a hybrid fusion module that uses convolutional and attention branches for feature fusion separately, and then utilizes a hybrid operation to fuse features rich in local high-frequency information and global low-frequency information, effectively achieving semantic alignment.

[0065] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0066] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0067] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0068] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual localization method based on multimodal feature alignment, characterized in that, include: Obtain the input data to be visually localized, which includes an image and its corresponding denotation; The input data is fed into the feature extraction module, which uses the BERT model to extract text features from the denotative expression, uses the Swin Transformer model to extract multi-scale grid features from the image, uses the Deformable DETR model and the grid features to generate region features, and then concatenates the text features and region features and inputs them into the Transformer encoder to generate a learnable token. The text features and the multi-scale grid features are combined using a scaling dot product attention operation to obtain multimodal features; Based on the multimodal features and the learnable token, a target segmentation mask is predicted to obtain a predicted mask, which is then used as the visual localization result of the input data. The step of concatenating the text features and region features and then inputting them into the Transformer encoder to generate a learnable token includes: The object-aware module concatenates the text features and region features and inputs them into a Transformer encoder to extract the object-level information of the query perception, generate a learnable token, and use the learnable token as the target query for subsequent mask prediction. The step of predicting a target segmentation mask based on the multimodal features and the learnable token to obtain a predicted mask, and using the predicted mask as the visual localization result of the input data, includes: The final multimodal features and the learnable token are input into the mask prediction module. Multimodal features at multiple scales are concatenated along the channel dimension using bilinear interpolation upsampling. The concatenated feature vector is then multiplied by the learnable token and activated with sigmoid. The prediction result is binarized to obtain the prediction mask based on a threshold of 0.

5. This prediction mask is used as the visual localization result. The loss function is a linear combination of BCE loss and Dice loss. The prediction mask contains 0s and 1s; a 1 indicates that the pixel represents the target object. The prediction mask is then combined with the original image, and each pixel of the original image is adjusted according to the corresponding mask value to obtain the final segmentation result.

2. The method according to claim 1, characterized in that, The input data is fed into a feature extraction module, which uses a BERT model to extract text features from the denotative expression, uses a SwinTransformer model to extract multi-scale grid features from the image, and uses a Deformable DETR model and grid features to generate region features, including: The feature extraction module includes a visual feature extraction network and a text feature extraction network. The image is input into the visual feature extraction network in the feature extraction module. The visual feature extraction network uses a pre-trained Swing Transformer model to extract multi-scale grid features of the image. The grid features are combined with randomly initialized feature vectors and input into a pre-trained Deformable DETR model to obtain object queries and use them as region features. The multi-scale grid features and region features are used as visual features, which are multi-channel tensors. The denotation corresponding to the image is input into the text feature extraction network in the feature extraction module. The text feature extraction network uses a pre-trained BERT model to obtain the hidden layer representation of the text and extract the text features of the denotation.

3. The method according to claim 1, characterized in that, The method of obtaining multimodal features by scaling dot product attention operation from the text features and the multi-scale grid features includes: The hybrid fusion module includes a convolutional modulation branch, an attention fusion branch, and a hybrid fusion operation branch. The input to the convolutional modulation branch is visual features and text features after pooling. The visual features and text features are multiplied by a dot and then fed into the network constructed by the convolutional operation to obtain the fused multimodal features. The input to the attention fusion branch is visual features q and text features k and v after pooling. After linear transformation, Q, K, and V are obtained. Q and K are used to calculate attention weights through dot product and Softmax normalization. For each position or channel, the attention weight determines its contribution to the fusion result. The attention weights are weighted and summed with V to obtain the fused multimodal features. The hybrid fusion operation branch mixes and fuses the two fused multimodal features through dynamic convolution and linear transformation operations to obtain the final multimodal features.

Images