Cyclic enhanced vision-text cross-modal attention characterization method

Abstract

The invention provides a cyclic enhanced vision-text cross-modal attention characterization method and a cyclic enhanced vision-text cross-modal attention characterization system. The invention aims to provide a vision-text cross-modal attention characterization method, and aims to enhance vision-text association and reduce feature difference between vision-text bimodals. Visual text fusion is researched in a unified manner through an iterative multi-stage vision-text fusion module, a mask reference-based central point supervision module and a cyclic enhancement decoder module. According to the method, the text information can be fully utilized, more comprehensive and more detailed text embedding is obtained, the context understanding of the image is enhanced, and the visual target most relevant to the text features is fused through circular enhancement. The method has a wide application prospect, can be applied to the fields of visual text fusion, remote sensing image processing and the like, and provides an effective solution for visual-text cross-modal feature representation.

Description

Technical Field

[0001] The present invention discloses a cyclically enhanced visual-textual cross-modal attention representation method, which belongs to the cross-technical field of computer vision and natural language processing. The purpose of the present invention is to provide a visual-textual cross-modal attention representation method, aiming to strengthen the visual-textual association and reduce the feature differences between the visual-textual bimodalities. Visual-text fusion is studied uniformly through iterative multi-level visual-text fusion modules, mask reference-based center point supervision modules and cyclically enhanced decoder modules. The present invention can make full use of text information, obtain more comprehensive and detailed text embedding, and enhance the contextual understanding of the image, and fused the visual targets most related to the text features through cyclic enhancement. The method of the present invention has broad application prospects, can be applied to fields such as visual-textual fusion and remote sensing image processing, and provides an effective solution for visual-textual cross-modal feature representation. Background Art

[0002] Image recognition is an important research field in computer vision, and there are a lot of studies in the fields of natural scene and remote sensing scene images. Traditional image recognition can identify all targets in an image, but cannot identify specific targets. Understanding and identifying the contextual information contained in natural language by computer, and making guided selections of targets in the image have become a hot topic in image recognition research. Visual-text recognition is a basic visual-language cross-modal fusion task. Given an image and a natural language expression, the goal is to make the detector recognize the target described by a specific natural language. In the field of remote sensing scene images, visual-text recognition has three major difficulties: large differences in visual language feature modalities, insufficient representation of visual language text features, and easy confusion of fine-grained target features. Therefore, there is always a lack of corresponding research on visual-text recognition. Therefore, proposing an effective visual-text cross-modal feature representation method is of great significance to fields such as visual-text fusion and remote sensing image processing.

[0003] The purpose of the present invention is to provide a cyclically enhanced visual-textual cross-modal attention representation method. The method uses a mask-referenced center point supervision module to enhance the contextual understanding of the entire image, and improves the reliable representation of different key descriptions in the text expression through a multi-level text feature enhancement module. In addition, the multi-level iterative visual-text fusion module improves the fusion ability of visual-text features through cyclic enhancement. In addition, the present invention also introduces a cyclically enhanced decoder module to obtain more complete and reliable features to improve visual language comprehension capabilities.

[0004] Compared with existing methods, the cyclically enhanced visual-textual cross-modal attention representation method of the present invention has the following advantages. First, the center point supervision method with mask reference can deeply extract textual expressions and improve text comprehension ability. Secondly, the introduction of iterative multi-level visual-text fusion can effectively improve the fusion ability of visual-textual cross-modal attention. Finally, by cyclically enhancing the decoder module, the representation ability of visual-text features is enhanced. The method of the present invention has broad application prospects in the fields of visual-text fusion and remote sensing image processing, and provides an effective visual-textual attention representation solution for related fields.

[0005] In summary, the proposed cyclic enhanced visual-textual cross-modal attention representation method is of great significance in dealing with the fusion representation of visual features and textual features. The application of this method will effectively enhance the understanding ability of visual-textual features and provide strong support for the recognition of specific targets in remote sensing image processing. Summary of the invention

[0006] The present invention proposes a cyclic enhanced visual-text cross-modal attention representation method for visual-text recognition tasks in the cross-technical field of computer vision and natural language processing. The technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for cyclically enhancing visual-textual cross-modal attention representation, comprising the following steps:

[0008] Step 1: Input image: All images are processed into the same size of 640*640*3. The input image is defined as I v ; Input text: The input text is defined as I l ;

[0009] Step 2: Convert the text in S1 to I l The input is sent to the mask enhancement module, which masks random words in the text according to carefully designed rules, and repeats this process up to three times to obtain a full text and three randomly masked texts. Finally, BERT is used to generate different text embeddings for these sentences.

[0010] Step 3: Image I in S1 v Input into the visual feature extractor to obtain the visual feature F v ;

[0011] Step 4: Output text features of S2 and the visual features F output by S3 vThe input is sent to the visual-text alignment module, which contains two multi-head attention mechanisms to enhance the feature map by collecting relevant semantic information from the corresponding textual expression, and secondly to process the visual-text fusion representation through center point supervision.

[0012] The specific steps are as follows:

[0013] (1) Visual-text cross-modal attention representation F output by the visual-text alignment module vc (i,j) can be expressed as:

[0014]

[0015] Among them, W Q , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension, and R(·) is the sinusoidal position encoding of the relative position.

[0016] (2) The loss value L of the center point supervision output key It can be expressed as:

[0017]

[0018] Among them, CELoss is the cross entropy loss, To predict the center point, y is the center point truth obtained from the center of each ground-truth box.

[0019] Step 5: Output text features of S2 and the visual features F output by S3 v The input is fed into the iterative multi-level visual-text fusion module, which consists of four iterative stages. The first multi-head attention layer uses the visual feature map F v ∈R C×H×W As Query, use the text embedding F in the multi-level text feature enhancement module l ∈R C×L As Key and Value. Multi-head cross attention can integrate text information into the visual feature map to obtain F c ∈R C×H×W On the second floor, F c As query and key, and F v This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F vc ∈R C×H×W In the i-th iteration, the first multi-head attention The output can be expressed as:

[0020]

[0021] Among them, W Q , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension. First, according to and Get the verification score value through the pixel-level verification module:

[0022]

[0023] Among them, α and σ are learnable parameters. The verification score models the semantic relevance of each visual feature with the textual expression. Therefore, a more salient feature map can be built for the reference object by pixel-wise modulation of the visual features and the verification score. and Enter the second multi-head attention:

[0024]

[0025] In each iteration, the i-th visual feature map Become the initial feature map (i.e. ). After four iterations and using the element-by-element maximization strategy, the final fusion feature is obtained:

[0026]

[0027] Step 6: Retrieve relevant features for object localization through a cyclically enhanced decoder. The decoder consists of N stages, each consisting of the same network architecture (with non-shared weights) for iterative cross-modal reasoning. In the first stage, a learnable target query is used As the initial representation of the target object, the target query is input into the decoder, visual features are extracted based on the textual expression, and its feature representation is iteratively updated in the subsequent stage to Specifically, in stage i, the target query As the query input to the first multi-head attention module, the semantic description information of the target object is collected from the text embedding to generate a text representation t l ∈R C×1 Then, in the second multi-head attention module, the text representation t l As a query, calculate its discriminant feature F established previously v The correlation is then obtained from the visual feature map F m Collect the features of interest in the l , obtain the visual features of the target object t v ∈RC×1 Afterwards, the collected features t v To update the target query

[0028]

[0029] Here, LN(·) denotes layer normalization and FFN(·) is a feed-forward network consisting of two linear projection layers with ReLU activation.

[0030] In a second aspect, the present invention provides an electronic device, including a processor and a processor;

[0031] Memory, used to store computer programs;

[0032] The processor is used to execute the computer program stored in the memory, and implements the aforementioned cyclic enhanced visual-text cross-modal attention representation method during execution.

[0033] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the aforementioned method of cyclically enhanced visual-textual cross-modal attention representation is implemented.

[0034] The visual-text cross-modal attention representation method aims to strengthen the visual-text association and reduce the feature differences between the visual-text bimodalities. The visual-text fusion is studied in a unified manner through an iterative multi-level visual-text fusion module, a center point supervision module based on mask reference, and a cyclic enhancement decoder module. The present invention can make full use of text information, obtain more comprehensive and detailed text embeddings, and enhance the contextual understanding of the image, and fuse the visual targets most relevant to the text features through cyclic enhancement. The method of the present invention has broad application prospects and can be applied to fields such as visual-text fusion and remote sensing image processing, providing an effective solution for visual-text cross-modal feature representation.

[0035] In summary, the above-mentioned cyclic enhanced visual-text cross-modal attention representation method provided by the present invention has good effects, and the main beneficial effects are specifically as follows:

[0036] 1. This method can effectively solve the problem of cross-modal fusion of visual and language features, improve the alignment ability of image features and text embedding, and demonstrate its effectiveness in improving the reliability of visual-text cross-modal attention representation;

[0037] 2. This method can make full use of text information to obtain more comprehensive and detailed text embedding. The method uses the center point supervision of the mask reference to improve text enhancement by using diversified text information, thereby enriching the extraction of text information;

[0038] 3. This method adopts a recurrent enhanced decoder module, which gradually refines the visual-textual attention representation by recurrently fusing multi-scale text embeddings and image features to obtain more complete and reliable features, effectively improving the visual-textual cross-modal attention representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a cyclically enhanced visual-textual cross-modal attention representation method provided by the present invention;

[0040] Figure 2 A structural flow chart of the iterative multi-level vision-text fusion module provided by the present invention;

[0041] Figure 3 This is a structural flow chart of the cyclic enhanced decoder module provided by the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, a space-based image denoising method based on a deformable Transformer provided by an embodiment of the present invention specifically includes the following steps:

[0044] S1. Input image: All images are processed into the same size of 640*640*3. The input image is defined as I v ; Input text: The input text is defined as I l ;

[0045] S2. Translate the text in S1 into l The input is sent to the mask enhancement module, which masks random words in the text according to carefully designed rules, and repeats this process up to three times to obtain a full text and three randomly masked texts. Finally, BERT is used to generate different text embeddings for these sentences.

[0046] S3. Image I in S1 v Input into the visual feature extractor to obtain the visual feature F v ;

[0047] S4. Text features output by S2 and the visual features F output by S3 vThe input is sent to the visual-text alignment module, which contains two multi-head attention mechanisms to enhance the feature map by collecting relevant semantic information from the corresponding textual expression, and secondly to process the visual-text fusion representation through center point supervision.

[0048] S5. Visual-text cross-modal attention representation F output by the visual-text alignment module vc (i,j) can be expressed as:

[0049]

[0050] Among them, W Q , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension, and R(·) is the sinusoidal position encoding of the relative position.

[0051] S6. Loss value L of the center point supervision output key It can be expressed as:

[0052]

[0053] Among them, CELoss is the cross entropy loss, To predict the center point, y is the center point truth obtained from the center of each ground-truth box.

[0054] S7. Output text features of S2 and the visual features F output by S3 v The input is fed into the iterative multi-level visual-text fusion module, which consists of four iterative stages. The first multi-head attention layer uses the visual feature map F v ∈R C×H×W As Query, use the text embedding F in the multi-level text feature enhancement module l ∈R C×L As Key and Value. Multi-head cross attention can integrate text information into the visual feature map to obtain F c ∈R C×H×W On the second floor, F c As query and key, and F v This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F vc ∈R C×H×W In the i-th iteration, the first multi-head attention The output can be expressed as:

[0055]

[0056] Among them, WQ , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension. First, according to and Get the verification score value through the pixel-level verification module:

[0057]

[0058] Among them, α and σ are learnable parameters. The verification score models the semantic relevance of each visual feature with the textual expression. Therefore, a more salient feature map can be built for the reference object by pixel-wise modulation of the visual features and the verification score. and Enter the second multi-head attention:

[0059]

[0060] In each iteration, the i-th visual feature map Become the initial feature map (i.e. ). After four iterations and using the element-by-element maximization strategy, the final fusion feature is obtained:

[0061]

[0062] S8. Retrieve relevant features for object localization via a recurrently enhanced decoder. The decoder consists of N stages, each consisting of the same network architecture (with non-shared weights) for iterative cross-modal reasoning. In the first stage, a learnable target query is used. As the initial representation of the target object, the target query is input into the decoder, visual features are extracted based on the textual expression, and its feature representation is iteratively updated in the subsequent stage to Specifically, in stage i, the target query As the query input to the first multi-head attention module, the semantic description information of the target object is collected from the text embedding to generate a text representation t l ∈R C×1 Then, in the second multi-head attention module, the text representation t l As a query, calculate its discriminant feature F established previously v The correlation is then obtained from the visual feature map F m Collect the features of interest in the l , obtain the visual features of the target object t v ∈R C×1 Afterwards, the collected features t vTo update the target query

[0063]

[0064] Here, LN(·) denotes layer normalization and FFN(·) is a feed-forward network consisting of two linear projection layers with ReLU activation.

[0065] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor, and the memory stores a computer program. When the computer program is executed, the method provided by the aforementioned embodiment of the present invention is implemented.

[0066] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the method provided by the aforementioned embodiment of the present invention.

[0067] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A cyclically enhanced visual-textual cross-modal attention representation method, characterized in that: The specific steps include: Step 1: Input the corresponding image and the corresponding original text; Step 2: Input the input text into the mask enhancement module, and mask the random words in the text according to the carefully designed rules. Repeat this process up to three times to obtain a full text and three randomly masked texts. Finally, use BERT to generate different text embeddings for these sentences. Step 3: Input the input image into the visual feature extractor to obtain visual features; Step 4: Process the output text features and image features through center point supervision to realize visual-text fusion representation; Step 5: Input the output text features and visual features into the iterative multi-level visual-text fusion module to implement iterative fusion processing; Step 6: The output visual-text fusion features are passed through a cyclic enhanced decoder to retrieve relevant features for object positioning.

2. The method for cyclically enhanced visual-textual cross-modal attention representation according to claim 1, characterized in that: In step 1, all images are processed into the same size of 640*640*3, and the input image is defined as I v ; Input text: The input text is defined as I l .

3. The method for cyclically enhancing visual-textual cross-modal attention representation according to claim 2, characterized in that: The text I l The input is sent to the mask enhancement module, which masks random words in the text according to carefully designed rules, and repeats this process up to three times to obtain a full text and three randomly masked texts. Finally, BERT is used to generate different text embeddings for these sentences.

4. The method for cyclically enhancing visual-textual cross-modal attention representation according to claim 3, characterized in that: Image I v Input into the visual feature extractor to obtain the visual feature F v .

5. The method for cyclically enhancing visual-textual cross-modal attention representation according to claim 4, characterized in that: The text features to be output and the visual features F output by S3 v The input is sent to the visual-text alignment module, which contains two multi-head attention mechanisms to enhance the feature map by collecting relevant semantic information from the corresponding textual expression, and secondly to process the visual-text fusion representation through center point supervision. The specific steps are as follows: (1) Visual-text cross-modal attention representation F output by the visual-text alignment module vc (i,j) can be expressed as: Among them, W Q , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension, and R(·) is the sinusoidal position encoding of the relative position. (2) The loss value L of the center point supervision output key It can be expressed as: Among them, CELoss is the cross entropy loss, To predict the center point, y is the center point truth obtained from the center of each ground-truth box.

6. The method for cyclically enhancing visual-textual cross-modal attention representation according to claim 5, characterized in that: The text features to be output And the output visual features F v The input is fed into the iterative multi-level visual-text fusion module, which consists of four iterative stages. The first multi-head attention layer uses the visual feature map F v ∈R C×H×W As Query, use the text embedding F in the multi-level text feature enhancement module l ∈R C×L As Key and Value. Multi-head cross attention can integrate text information into the visual feature map to obtain F c ∈R C×H×W On the second floor, F c As query and key, and F v This self-attention operator allows the model to collect key contextual features of the target object based on the text description, and the final feature is F vc ∈R C×H×W In the i-th iteration, the first multi-head attention The output can be expressed as: Among them, W Q , W K , W V is the linear projection weight of query, key and value, d k is the projection channel dimension. First, according to and Get the verification score value through the pixel-level verification module: Among them, α and σ are learnable parameters. The verification score models the semantic relevance of each visual feature with the textual expression. Therefore, a more salient feature map can be built for the reference object by pixel-wise modulation of the visual features and the verification score. and Enter the second multi-head attention: In each iteration, the i-th visual feature map Become the initial feature map (i.e. ). After four iterations and using the element-by-element maximization strategy, the final fusion feature is obtained:

7. The method for cyclically enhancing visual-textual cross-modal attention representation according to claim 6, characterized in that: The decoder is augmented with a recurrent decoder to retrieve relevant features for object localization. The decoder consists of N stages, each consisting of the same network architecture (with non-shared weights) for iterative cross-modal reasoning. In the first stage, a learnable target query is used As the initial representation of the target object, the target query is input into the decoder, visual features are extracted based on the textual expression, and its feature representation is iteratively updated in the subsequent stage to Specifically, in stage i, the target query As the query input to the first multi-head attention module, the semantic description information of the target object is collected from the text embedding to generate a text representation t l ∈R C×1 Then, in the second multi-head attention module, the text representation t l As a query, calculate its discriminant feature F established previously v The correlation is then obtained from the visual feature map F m Collect the features of interest in the l , obtain the visual features of the target object t v ∈R C×1 Afterwards, the collected features t v To update the target query Among them, LN(·) represents layer normalization, and FFN(·) is a feed-forward network composed of two linear projection layers with ReLU activation. The above cycle enhances the visual-text cross-modal attention representation method to achieve the fusion of visual features and text embedding.

Images