Cross-modal vision-text feature enhancement method based on open vocabularies

Abstract

The invention discloses a cross-modal vision-text feature enhancement method based on open vocabularies, and belongs to the cross technical field of computer vision and natural language processing. The invention aims to provide a cross-modal vision-text feature enhancement method based on open vocabularies, and aims to enhance vision-text features and reduce feature differences between vision-text bimodals of the open vocabularies. Open vocabulary visual text enhancement is researched in a unified manner through a feature extraction and enhancement module, a text-guided query selection module, a cross-modal decoder module and a clause-level text feature module. According to the method, the features of different modes can be aligned, the interference between open vocabulary category names is reduced, and the most relevant specified target in the image is guided and selected by enhancing the text. The method has a wide application prospect, can be applied to the fields of visual text fusion, remote sensing image processing and the like, and provides an effective solution for open vocabulary visual-text cross-modal feature enhancement.

Description

Technical Field

[0001] This invention discloses a cross-modal visual-text feature enhancement method based on open vocabulary, belonging to the interdisciplinary field of computer vision and natural language processing. The purpose of this invention is to provide a cross-modal visual-text feature enhancement method based on open vocabulary, aiming to enhance visual-text features and reduce feature differences between the open vocabulary visual-text bimodalities. It unifies the study of open vocabulary visual-text enhancement through a feature extraction and enhancement module, a text-guided query selection module, a cross-modal decoder module, and a clause-level text feature module. This invention can align features from different modalities, reduce interference between open vocabulary category names, and guide the selection of the most relevant specified target in the image through enhanced text. The method of this invention has broad application prospects and can be applied to fields such as visual-text fusion and remote sensing image processing, providing an effective solution for open vocabulary visual-text cross-modal feature enhancement. Background Technology

[0002] Image recognition, a crucial research area in computer vision, has been extensively studied in both natural and remote sensing imagery. Traditional image recognition can identify all targets in an image, but it cannot understand specific targets described in text. Combining natural language understanding with image recognition to select specific targets within an image has become a hot research topic in computer vision. Furthermore, most existing image recognition methods achieve limited target recognition through closed-set training, while open-vocabulary techniques, through joint training of images and text, can recognize categories not included in the training set. Cross-modal visual-text recognition aims to enable a detector to understand the meaning of textual descriptions given an image and a natural language expression, thereby achieving the recognition of specific targets. Open-vocabulary visual-text recognition faces three major challenges: significant differences in visual and textual feature modalities, insufficient representation of visual and textual features, and difficulty in recognizing unknown categories. Therefore, open-vocabulary visual-text recognition has consistently lacked relevant research. Thus, proposing an effective open-vocabulary visual-text cross-modal feature enhancement method is of great significance for fields such as visual-text fusion and remote sensing image processing.

[0003] The purpose of this invention is to provide an open-vocabulary cross-modal visual-text feature enhancement method. This method utilizes a feature enhancer to enhance visual text features and a text-guided query selection module to enhance the association between text and a specified target. Clause-level text features are used to eliminate interference between different category names while preserving the features of each word. Finally, a cross-modal decoder module is used to achieve better cross-modal enhancement capabilities.

[0004] Compared to existing methods, the open-vocabulary cross-modal visual-text feature enhancement method of this invention has the following advantages. First, the feature enhancement method employed can deeply enhance cross-modal features and improve the alignment capability of visual-text features. Second, the introduction of text-guided query selection can effectively enhance the association between text and the specified target. Simultaneously, the designed clause-level text features can minimize inter-text interference. Finally, better cross-modal enhancement is achieved through a cross-modal decoder module. The method of this invention has broad application prospects in fields such as visual-text fusion and remote sensing image processing, providing an effective visual-text feature enhancement solution for related fields.

[0005] In summary, the open-vocabulary visual-text cross-modal feature enhancement method proposed in this invention is of great significance in addressing the problem of fusing and enhancing visual and textual features. The application of this method will effectively enhance the understanding of visual-textual features, providing strong support for the identification of specific targets in remote sensing image processing. Summary of the Invention

[0006] This invention addresses the visual-text recognition task within the interdisciplinary field of computer vision and natural language processing, proposing an open-vocabulary visual-text cross-modal feature enhancement method. The technical solution of this invention is as follows:

[0007] In a first aspect, the present invention provides an open-vocabulary visual-text cross-modal feature enhancement method, comprising the following steps:

[0008] Step 1: Input Image: Process all images to the same size 640*640*3. The input image is defined as I. v Input text: The input text is defined as I. l ;

[0009] Step 2: Extract text I from S1 l In the input text feature extractor, BERT is used to generate text embeddings P for the text. i ;

[0010] Step 3: Transfer image I from S1 v The input is fed into the visual feature extractor to obtain visual features O. i ;

[0011] Step 4: Extract the text features P from S2. i And the visual features O output by S3 i The input is fed into the feature enhancement module, which contains multiple feature enhancement layers. It enhances image features through deformable self-attention and enhances text features using a self-attention mechanism, thus achieving visual-text feature enhancement processing.

[0012] The specific steps are as follows:

[0013] (1) The feature enhancement layer deeply fuses the image and text encoders, and the process can be represented as follows:

[0014]

[0015] Where MHA represents the multi-head attention module, O i P represents the visual features from the i-th layer of the visual backbone. i This represents the text features from the i-th layer of the text backbone, and information fusion of the view and speech modalities is achieved through a cross-modal multi-head attention module.

[0016] (2) Subsequently, the enhanced image features and text features are deeply fused, which can be represented as:

[0017]

[0018] Where L represents the number of detectors in the multi-head attention mechanism, and BERTLayer is a BERT layer added on top of the pre-trained BERT. The enhanced text features... and visual features Output the results.

[0019] Step 5: Extract the text features F from S2. l and visual features F v The input to the feature encoder mainly consists of multi-head attention and uses the visual feature map F. v ∈R C×H×W As a query, use text embedding F l ∈R C×L As the key and value, multi-head cross-attention can integrate textual information into the visual feature map, obtaining F. c ∈R C×H×W On the second floor, F c As a query and key, F v As a value. This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F. vc ∈R C×H×W Its output can be represented as:

[0020]

[0021] Among them, W Q W k W V For the linear projection weights of the query, key, and value, d kThis represents the projection channel dimension. Finally, visual-text cross-modal enhancement is achieved through a text-guided query selection module. This module first calculates the cross-modal validation score, which can be expressed as:

[0022]

[0023] Here, α and σ are learnable parameters. By calculating validation scores, the semantic relevance of each visual feature to the textual representation is modeled, creating a more salient feature map for the reference object. Based on the cross-modal validation scores, the image region most relevant to the text is selected.

[0024] Step Six: Retrieve relevant features through a cross-modal decoder for target localization. Each cross-modal query is sequentially fed into a self-attention layer, an image cross-attention layer (for combining image features), a text cross-attention layer (for combining text features), and a feedforward neural network layer in each cross-modal decoder layer. In the designed cross-modal decoder, each decoder layer contains a text cross-modal self-attention layer and an image cross-modal self-attention layer, which can better inject text and image information into the query to achieve better modality alignment.

[0025] In a second aspect, the present invention provides an electronic device, including a memory and a processor;

[0026] Memory, used to store computer programs;

[0027] The processor is used to execute computer programs stored in memory, and during execution, it implements the aforementioned open-vocabulary visual-text cross-modal feature enhancement method.

[0028] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the aforementioned open vocabulary visual-text cross-modal feature enhancement method.

[0029] This invention presents an open-vocabulary visual-text cross-modal feature enhancement method, aiming to enhance image and text features and improve visual-text cross-modal correlation. It unifies the study of open-vocabulary visual-text enhancement through a feature enhancement module, a text-guided query selection module, and a feature decoder module. This invention can align features from different modalities, reduce interference between open-vocabulary category names, and guide the selection of the most relevant specified target in the image through enhanced text. The method has broad application prospects and can be applied to fields such as visual-text fusion and remote sensing image processing, providing an effective solution for open-vocabulary visual-text cross-modal feature enhancement.

[0030] In summary, the open-vocabulary visual-text cross-modal feature enhancement method provided by this invention has good performance, and its main beneficial effects are as follows:

[0031] 1. This method can effectively solve the problem of cross-modal enhancement of visual and text features, improve the alignment ability of image features and text embeddings, and demonstrate its effectiveness in improving the reliability of visual-text cross-modal attention representation;

[0032] 2. This method can make full use of text information and achieve more accurate target recognition through in-depth text understanding and cross-modal association. The method uses text-guided query selection and associates target features with text information to achieve accurate target extraction.

[0033] 3. This method employs a feature decoder module, which, through cross-modal self-attention of text and images, better injects text and image information into the query, effectively improving the visual-text cross-modal attention representation capability. Attached Figure Description

[0034] Figure 1 A flowchart of an open-vocabulary visual-text cross-modal feature enhancement method provided by the present invention;

[0035] Figure 2 A flowchart illustrating the structure of the feature enhancement module provided by this invention;

[0036] Figure 3 The flowchart shows the structure of the cross-modal decoder module provided by this invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, this invention provides an open-vocabulary visual-text cross-modal feature enhancement method, comprising the following steps:

[0039] S1. Input Image: Process all images to the same size 640*640*3. The input image is defined as I. v Input text: The input text is defined as I. l ;

[0040] S2. Transfer the text I from S1. l In the input text feature extractor, BERT is used to generate text embeddings P for the text. i ;

[0041] S3. Transfer image I from S1v The input is fed into the visual feature extractor to obtain visual features O. i ;

[0042] S4. The text features P output from S2 i And the visual features O output by S3 i The input is fed into the feature enhancement module, which contains multiple feature enhancement layers. It enhances image features through deformable self-attention and enhances text features using a self-attention mechanism, thus achieving visual-text feature enhancement processing.

[0043] S5. The feature enhancement layer deeply fuses the image and text encoders, and the process can be represented as follows:

[0044]

[0045] Where MHA represents the multi-head attention module, O i P represents the visual features from the i-th layer of the visual backbone. i This represents the text features from the i-th layer of the text backbone, and information fusion of the view and speech modalities is achieved through a cross-modal multi-head attention module.

[0046] S6. Subsequently, the enhanced image features and text features are deeply fused, which can be represented as:

[0047]

[0048] Where L represents the number of detectors in the multi-head attention mechanism, and BERTLayer is a BERT layer added on top of the pre-trained BERT. The enhanced text features... and visual features Output the results.

[0049] S7. Convert the text features F output by S2 l and visual features F v The input to the feature encoder mainly consists of multi-head attention and uses the visual feature map F. v ∈R C×H×W As a query, use text embedding F l ∈R C×L As the key and value, multi-head cross-attention can integrate textual information into the visual feature map, obtaining F. c ∈R C×H×W On the second floor, F c As a query and key, F v As a value. This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F. vc ∈R C×H×W Its output can be represented as:

[0050]

[0051] Among them, W Q W K W V For the linear projection weights of the query, key, and value, d k This represents the projection channel dimension. Finally, visual-text cross-modal enhancement is achieved through a text-guided query selection module. This module first calculates the cross-modal validation score, which can be expressed as:

[0052]

[0053] Here, α and σ are learnable parameters. By calculating validation scores, the semantic relevance of each visual feature to the textual representation is modeled, creating a more salient feature map for the reference object. Based on the cross-modal validation scores, the image region most relevant to the text is selected.

[0054] S8. Relevant features are retrieved through a cross-modal decoder for target localization. Each cross-modal query is sequentially fed into a self-attention layer, an image cross-attention layer (for combining image features), a text cross-attention layer (for combining text features), and a feedforward neural network layer in each cross-modal decoder layer. In the designed cross-modal decoder, each decoder layer contains a text cross-modal self-attention layer and an image cross-modal self-attention layer, which can better inject text and image information into the query to achieve better modality alignment.

[0055] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (e.g., a computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed, it implements the method provided in the aforementioned embodiment of the present invention.

[0056] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (e.g., ROM / RAM, disk, optical disk), which stores a computer program. When the computer program is executed by a computer, it implements the method provided in the aforementioned embodiment of the present invention.

[0057] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A cross-modal visual-text feature enhancement method based on open vocabulary, characterized in that, Specifically, it includes the following steps: Step 1: Input the corresponding image and the corresponding original text; Step 2: Input the input text into the text feature extractor and use BERT to generate text embeddings for the text; Step 3: Input the input image into the visual feature extractor to obtain visual features; Step 4: Combine the output text features and image features through the feature enhancement module to perform visual-text feature enhancement processing; Step 5: Input the output text features and visual features into the text-guided query selection module to achieve cross-modal enhancement processing; Step 6: Use a cross-modal decoder to retrieve relevant features from the output visual-text fusion features for object localization.

2. The iteratively enhanced visual-text cross-modal attention representation method according to claim 1, characterized in that: In step one, all images are processed to the same size 640*640*3, and the input image is defined as I. v Input text: The input text is defined as I. l。 3. The iteratively enhanced visual-text cross-modal attention representation method according to claim 2, characterized in that: Text I l In the input text feature extractor, BERT is used to generate text embeddings P for the text. i。 4. The iteratively enhanced visual-text cross-modal attention representation method according to claim 3, characterized in that: Image I v The input is fed into the visual feature extractor to obtain visual features O. i。 5. The iteratively enhanced visual-text cross-modal attention representation method according to claim 4, characterized in that: The output text features P i And the visual features O output by S3 i The input is fed into the feature enhancement module, which contains multiple feature enhancement layers. It enhances image features through deformable self-attention and enhances text features using a self-attention mechanism, thus achieving visual-text feature enhancement processing. The specific steps are as follows: The specific steps are as follows: (1) The feature enhancement layer deeply fuses the image and text encoders, and the process can be represented as follows: Where MHA represents the multi-head attention module, O i P represents the visual features from the i-th layer of the visual backbone. i This represents the text features from the i-th layer of the text backbone, and information fusion of the view and speech modalities is achieved through a cross-modal multi-head attention module. (2) Subsequently, the enhanced image features and text features are deeply fused, which can be represented as: Where L represents the number of detectors in the multi-head attention mechanism, and BERTLayer is a BERT layer added on top of the pre-trained BERT. The enhanced text features... and visual features Output the results.

6. The iteratively enhanced visual-text cross-modal attention representation method according to claim 5, characterized in that: The text features F output by S2 l and visual features F v The input to the feature encoder mainly consists of multi-head attention and uses the visual feature map F. v ∈R C×H×W As a query, use text embedding F l ∈R C×L As the key and value, multi-head cross-attention can integrate textual information into the visual feature map, obtaining F. c ∈R C×H×W On the second floor, F c As a query and key, F v As a value. This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F. vc ∈R C×H×W Its output can be represented as: Among them, W Q W K W V For the linear projection weights of the query, key, and value, d k This represents the projection channel dimension. Finally, visual-text cross-modal enhancement is achieved through a text-guided query selection module. This module first calculates the cross-modal validation score, which can be expressed as: Here, α and σ are learnable parameters. By calculating validation scores, the semantic relevance of each visual feature to the textual representation is modeled, creating a more salient feature map for the reference object. Based on the cross-modal validation scores, the image region most relevant to the text is selected.

7. The iteratively enhanced visual-text cross-modal attention representation method according to claim 6, characterized in that: Relevant features are retrieved through a cross-modal decoder for target localization. Each cross-modal query is sequentially fed into a self-attention layer, an image cross-attention layer (for combining image features), a text cross-attention layer (for combining text features), and a feedforward neural network layer within each cross-modal decoder layer. In the designed cross-modal decoder, each decoder layer includes a text cross-modal self-attention layer and an image cross-modal self-attention layer, which allows for better incorporation of text and image information into the query, resulting in improved modality alignment.

Images