Grounding visual question-answering method and system based on bimodal prompt learning

Abstract

According to the grounded visual question-answering method and system based on bimodal prompt learning, visual prompt and text prompt are introduced, visual and text information is integrated into a coding process of a model, the ability of understanding and positioning a complex scene is improved, comprehensive answer reasoning is performed by adopting a multi-iteration fusion strategy, and the accuracy of question-answering is improved. And high-quality generation of text answers and grounded answers is ensured. The method comprises the steps of inputting a question and a related picture into a pre-training model CAT-ViL to obtain a visual knowledge prompt and a text knowledge prompt, extracting picture features, and performing vectorization processing on a text; guiding reasoning of multi-modal information including vision and text by using prompt information in a layered iterative fusion mode; fine vision and text information data obtained through reasoning are fused through a stacking iteration attention feature fusion module; the fused data and the visual and text prompt function after reasoning are decoded together; the dedicated classification head and the target detection head generate a final text answer and a grounded answer.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and natural language processing, and relates to visual question answering and multimodal semantic understanding technology, specifically a grounded visual question answering method and system based on bimodal prompt learning. Background Art

[0002] In the field of artificial intelligence, visual question answering is an important task that aims to enable machines to understand and answer questions about images. Traditional visual question answering methods mainly rely on deep neural networks to extract features from images and texts, and then combine these features through some fusion mechanism to generate answers to questions. However, in the existing technology, most visual question answering systems adopt single-modality or simple multimodal fusion strategies. Single-modal methods usually only focus on one type of information in the image or text, ignoring the complementarity between the two, resulting in an incomplete understanding of the problem and the reliability of the answer cannot be verified. Although simple multimodal fusion strategies take into account the combination of images and texts, they often lack in-depth interaction and reasoning mechanisms, so that the generated answers may not be accurate or lack details.

[0003] In addition, existing methods face greater challenges when dealing with grounded visual question answering. Grounded visual question answering requires not only that the machine can answer questions about the image, but also that the area in the image that is relevant to the answer is accurately located. This requires the model to not only understand the content of the image and text, but also to have the ability to perform precise spatial reasoning in complex scenes. However, existing visual question answering systems often have difficulty accurately understanding and locating visual information related to the answer in complex scenes, resulting in a lack of accuracy and reliability in the generated answers. Summary of the invention

[0004] In view of the above-mentioned problems existing in the prior art, the present invention proposes a grounded visual question answering method and system based on bimodal prompt learning, which introduces two complementary prompts, namely visual prompts and text prompts, and integrates visual and text information into the encoding process of the model, thereby improving the understanding and positioning capabilities of complex scenes. In addition, a multiple iterative fusion strategy is adopted for comprehensive answer reasoning to ensure high-quality generation of text answers and grounded answers.

[0005] To achieve the above objectives, the technical solution of the present invention is:

[0006] A grounded visual question answering method based on bimodal cue learning, comprising the following steps:

[0007] Step 1: First, input the question and related pictures into the pre-trained model CAT-ViL to obtain visual knowledge prompts and textual knowledge prompts, namely Box prompt features B i And Label prompt feature L i, extract image features and vectorize text at the same time;

[0008] Step 2: Use the cue information to guide the reasoning of multimodal information including vision and text through hierarchical iterative fusion;

[0009] Step 3: Fuse the inferred fine-grained visual and textual information data through the stacked iterative attention feature fusion module;

[0010] Step 4: Decode the fused data together with the inferred visual and textual cue features;

[0011] Step 5: The decoded information is passed to the dedicated classification head and object detection head to generate the final text answer and ground answer.

[0012] Based on the above scheme, this method uses the prompt knowledge generated by the pre-trained model to promote the joint encoding of cross-modal inputs, which can better understand and process visual information at different scales, thereby improving the understanding and positioning of complex scenes, and thus improving the accuracy of question answering. This method designs a visual supplement prompter and a text supplement prompter to better utilize the prompt knowledge of the two modalities. The visual supplement prompter supplements and improves the visual information related to the answer by strengthening the interaction between multimodal information; the text supplement prompter introduces visual area features, which are used to align with text features and text prompt features respectively, and then guide text information through visual information. An iterative fusion method is used between the prompt features and the original features to ensure more effective guidance. The stacked iterative attention feature fusion module is used to merge the refined visual and text data obtained through complex reasoning. This method can be applied to grounded visual question answering, providing an efficient and accurate method for machines to understand images and answer questions, and has potential applications in various fields such as autonomous driving, medical image analysis, image retrieval, etc.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1: First, input the question and related pictures into the pre-trained model CAT-ViL to obtain the Box prompt feature B i And Label prompt feature L i ;

[0015] Step 1.2: The image is then passed through Feature extractor to extract image features V i ;

[0016] Step 1.3: Finally, the question is vectorized into a text question through Tokenizer and outputs Q i ;

[0017] Furthermore, step 2 specifically includes:

[0018] Step 2.1: First, give feature B to Box i Downsampling is performed to limit the scope of relevant information and obtain the downsampled prompt feature B d , the image feature V i Smoothing is performed on all spatial dimensions to generate enhanced embedded image features V m , the specific formula is as follows:

[0019]

[0020] In formula (1), x represents the xth row; y represents the yth column; λ represents the learnable parameter;

[0021] Step 2.2: The next downsampled hint feature B d With enhanced embedded image features V m Effective integration: Two cross-fusion modules are used to integrate the enhanced embedded image features V m And the downsampled hint feature B d Fused together, one side is the embedded feature B obtained by cross-attention from the prompt (as query) to the image embedding c , and the other is to obtain the hint feature Y by cross-attention from the image embedding (as query) to the hint i The specific formula is as follows:

[0022]

[0023] Where T represents the matrix transpose operation; d k Indicates B d and V m The dimension of each element in ;

[0024] Step 2.3: Transfer the embedding B through the residual structure and the context-guided attention mechanism c Get visual information features M i ;

[0025] Step 2.4: Repeat the above process using an iterative fusion strategy, and then transform the visual information feature M i and prompt feature Y i As the new adjusted visual information features and prompt features, they are continuously input into the visual supplementary prompt module VCP for iterative update to generate the final visual prompt feature Y6. The specific formula is as follows:

[0026]

[0027] M i+1 =ψ <M i ,Y i>(i≤5) (5)

[0028] Y i+1 =ψ <M i ,Y i >(i≤5) (6)

[0029] Where Ψ represents the operation of the visual supplementary cue module; i represents the number of iterations;

[0030] Step 2.5: The label hint feature is initially encoded together with the text information feature using a self-attention mechanism, resulting in a label L s (for label hints) and Q s (for text information), the label hint feature L s Upsampling; using the attention fusion and alignment module AFF to combine the visual position information with the label hint feature L s and text information feature Q s Projecting into two joint feature spaces, the alignment of multimodal knowledge is realized to achieve more effective information interaction. The specific formula is as follows:

[0031] Q a =Γ AFF [[Q s ,V s ] (7)

[0032] C i =Γ AFF [β(L s ),Q a ] (8)

[0033] where Γ AFF represents AFF operation; β represents upsampling operation; V s Represents the input image features; Q a Indicates Q s Alignment features of C i Indicates L s Alignment features;

[0034] Step 2.6: Incrementally integrate the learned multimodal cues into the text feature space using a residual method: visually guided text feature R i By applying cross-attention to the text-encoded hint (as query), the multimodal information guides the hint feature G i It is obtained by applying cross attention to the text encoding of the prompt (as the query), and the specific formula is as follows:

[0035]

[0036] Where T represents the transpose operation of the matrix; L i It is the Label prompt feature; Qi It is the vectorized text question output feature;

[0037] Step 2.7: During the iteration, feature C i With feature R i and feature G i Through repeated interactions in the form of cross-attention, the text prompt function G6 and the text information function R6 are finally generated. The specific formula is as follows:

[0038] <C i+1 ,R i+1 ,G i+1 >=Φ <C i ,R i ,G i >(≤5) (11)

[0039] Where i represents the number of iterations; Ф represents the text supplement prompt module;

[0040] Furthermore, step 3 is specifically as follows: features R6 and M6 are fused together through a fusion module composed of two iterative attention feature fusion modules iAFF to generate a fused embedding feature F i , the specific formula is as follows:

[0041] F i =Γ iAFF {M6,Γ iAFF [M6,R6]} (12)

[0042] Furthermore, step 4 is as follows: embed the fusion feature F i Input into the pre-trained DeiT-base module, and connect F i Merge with Y6 and G6, and decode them together to further refine the relationship between features in each domain;

[0043] Furthermore, step 5 is specifically as follows: running the DeiT model on the feedforward network, the DeiT model receives the decoded data in step 4 as input, and the DeiT model predicts instance bounding boxes and answers related to the question through a dedicated classification head and a target detection head, and generates a final text answer and a grounded answer.

[0044] The present application also provides a grounded visual question answering system based on dual-modal prompt learning, the system comprising:

[0045] An input module, used for receiving questions and related pictures, and inputting the questions and related pictures into a pre-trained model CAT-ViL, wherein the pre-trained model CAT-ViL is a bimodal pre-trained model based on the Transformer architecture, which is used for jointly understanding visual and textual information;

[0046] Feature extraction module, used to extract visual knowledge cues and textual knowledge cues from the pre-trained model CAT-ViL, extract image features, and vectorize text;

[0047] The reasoning module uses the cue information to guide the reasoning of multimodal information including vision and text through hierarchical iterative fusion;

[0048] The fusion module includes two iterative attention feature fusion modules iAFF, which are used to fuse the inferred fine-grained visual and textual information data;

[0049] A decoding module, which is used to combine and decode the fused data with the inferred visual and textual cue functions;

[0050] The output module, including a dedicated classification head and an object detection head, is used to receive the decoded information, predict instance bounding boxes and answers related to the question through the DeiT model on the feed-forward network, and generate the final text answer and the ground answer.

[0051] Furthermore, the feature extraction module includes:

[0052] The Box cue feature extraction submodule is used to extract Box cue features from the pre-trained model CAT-ViL;

[0053] The Label prompt feature extraction submodule is used to extract Label prompt features from the pre-trained model CAT-ViL;

[0054] Image feature extraction submodule, used to extract features from images using Feature extractor;

[0055] The text vectorization processing submodule is used to use Tokenizer to perform vectorization processing on text questions.

[0056] Furthermore, the reasoning module further includes:

[0057] The downsampling submodule is used to downsample the Box hint features;

[0058] The image feature smoothing submodule is used to perform smoothing on the image features in all spatial dimensions to generate enhanced embedded image features;

[0059] A cross-fusion submodule for fusing the enhanced embedded image features with the downsampled cue features;

[0060] The iterative fusion submodule uses an iterative fusion strategy to repeat the cross-fusion process, and uses the visual information features and prompt features as new adjusted visual information features and prompt features, which are continuously input into the visual supplementary prompt module VCP for iterative update to generate the final visual prompt features;

[0061] A label hint feature encoding submodule is used to encode the label hint feature together with the text information feature using a self-attention mechanism;

[0062] The multimodal knowledge alignment submodule is used to use the attention fusion and alignment module AFF to project the visual position information, label prompt features and text information features into two joint feature spaces to achieve the alignment of multimodal knowledge;

[0063] a text feature guided submodule for obtaining visual guided text features by applying cross attention to the text encoded prompt as a query, and obtaining multimodal information guided prompt features by applying cross attention to the text encoded prompt as a query;

[0064] Text function generation module: used to generate text prompt function and text information function through the text supplement prompt module in the iteration process.

[0065] The beneficial effects of the present invention include: proposing a new method for visual question answering, introducing a bimodal prompt model, which consists of two complementary prompters, integrating vision and text into the encoding process of the model; the visual supplementary prompter combines visual prompt knowledge with visual information features to guide precise positioning in the image; the textual supplementary prompter aligns visual information with textual prompt knowledge and textual information, thereby promoting more accurate answer reasoning; and adopting a multi-iteration fusion strategy to ensure comprehensive answer reasoning, thereby generating high-quality text answers and fact-based answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 The network framework diagram of the grounded visual question answering method based on bimodal prompt learning;

[0067] Figure 2 is a schematic diagram of a visual supplement prompter module;

[0068] Figure 3 Schematic diagram of the text supplement prompter module. DETAILED DESCRIPTION

[0069] The embodiments of the present invention are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0070] The present invention provides a grounded visual question answering method and system based on bimodal cue learning, which aims to address the limitations of existing systems in accurately locating relevant content within an image. Specifically, in order to enable the visual question answering system to generate high-quality text answers and grounded answers, a bimodal cue model is proposed to enhance precise multimodal information interaction to guide accurate positioning and more accurate inference of answers. To ensure the generation of high-quality text and grounded answers, a multi-iteration fusion strategy is used for comprehensive answer reasoning.

[0071] The present invention is described in detail below in conjunction with embodiments so that those skilled in the art can implement the invention according to the description.

[0072] This embodiment uses Windows system as the development environment, Pycharm as the development platform, Python as the development language, and adopts the grounded visual question answering method based on dual-modal prompt learning of the present invention to complete the grounded visual question answering problem in robotic surgery.

[0073] In this embodiment, the grounded visual question answering method based on dual-modal prompt learning includes the following steps:

[0074] Step 1: First, input the question and related pictures into the pre-trained model CAT-ViL to obtain visual knowledge prompts and textual knowledge prompts, namely Box prompt features B i And Label prompt feature L i , extract image features and vectorize text at the same time;

[0075] Step 2: Use the cue information to guide the reasoning of multimodal information including vision and text through hierarchical iterative fusion;

[0076] Step 3: Fuse the inferred fine-grained visual and textual information data through the stacked iterative attention feature fusion module;

[0077] Step 4: Decode the fused data together with the inferred visual and textual cue features;

[0078] Step 5: The decoded information is passed to the dedicated classification head and object detection head to generate the final text answer and ground answer.

[0079] This embodiment uses the intersection-over-union ratio to measure the proximity of each bounding box to the true bounding annotation box; the text answer is evaluated using the two indicators of accuracy and F-score.

[0080] According to the above steps, the present invention is compared with VisualBERT model, VisualBERT R model, MCAN model, VQA-DeiT model, MUTAN model, MFH model, BlockTucker model and CAT-ViL DeiT model on two different test sets (EndoVis-18 and EndoVis-17) under two different experimental settings (Setting 1: FRCNN and Setting 2: ResNet18). It can be seen from Table 1 and Table 2 that the method proposed in the present invention is superior to other methods in all indicators on these two common test sets.

[0081] Table 1 Comparison of the above models with Setting 1 on the EndoVis-18 and EndoVis-17 datasets

[0082]

[0083]

[0084] Note: The best results are highlighted.

[0085] Table 2 Comparison of the above models with the EndoVis-18 and EndoVis-17 datasets under Setting2

[0086]

[0087]

[0088] Note: The best results are highlighted.

[0089] In summary, the present invention provides a grounded visual question-answering method and system based on bimodal prompt learning. The pre-trained model CAT-ViL is used to obtain visual knowledge prompts and textual knowledge prompts to promote the joint encoding of cross-modal inputs, which can better understand and process visual information at different scales, thereby improving the understanding and positioning of complex scenes; visual supplementary prompters and textual supplementary prompters are designed to better utilize the prompt knowledge of the two modalities. The visual supplementary prompter supplements and improves the visual information related to the answer by strengthening the interaction between multimodal information. The textual supplementary prompter introduces visual area features, which are used to align with text features and textual prompt features respectively, and then guide textual information through visual information; a multi-iteration fusion strategy is adopted to ensure comprehensive answer reasoning, thereby generating high-quality text answers and fact-based answers. The present invention adopts a bimodal prompt learning method to obtain visual prompt features and textual prompt features, realizes a more thorough interaction and alignment of visual information and textual information, and improves the performance of the model.

[0090] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A grounded visual question answering method based on bimodal cue learning, characterized in that: The method comprises the following steps: Step 1: First, input the question and related pictures into the pre-trained model CAT-ViL to obtain visual knowledge prompts and textual knowledge prompts, namely Box prompt features B i And Label prompt feature L i , extract image features and vectorize text at the same time; Step 2: Use the cue information to guide the reasoning of multimodal information including vision and text through hierarchical iterative fusion; Step 3: Fuse the inferred fine-grained visual and textual information data through the stacked iterative attention feature fusion module; Step 4: Decode the fused data together with the inferred visual and textual cue features; Step 5: The decoded information is passed to the dedicated classification head and object detection head to generate the final text answer and ground answer.

2. A grounded visual question answering method based on dual-modal prompt learning according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: First, input the question and related pictures into the pre-trained model CAT-ViL to obtain the Box prompt feature B i And Label prompt feature L i ; Step 1.2: The image is then passed through Feature extractor to extract image features V i ; Step 1.3: Finally, the question is vectorized into a text question through Tokenizer and outputs Q i。 3. A grounded visual question answering method based on dual-modal prompt learning according to claim 2, characterized in that: The step 2 specifically includes: Step 2.1: First, give feature B to Box i Downsample to obtain the downsampled prompt feature B d , the image feature V i Smoothing is performed on all spatial dimensions to generate enhanced embedded image features V m , the specific formula is as follows: In formula (1), x represents the xth row; y represents the yth column; λ represents the learnable parameter; Step 2.2: Secondly, two cross-fusion modules are used to transform the enhanced embedded image features V m And the downsampled hint feature B d Fusion, respectively, from the prompt as query to the image embedding to obtain the embedded feature B c , and the hint feature Y obtained by cross-attention from the image embedding as query to the hint i ; The specific formula is as follows: Where T represents the matrix transpose operation; d k Indicates B d and V m The dimension of each element in ; Step 2.3: Transfer the embedding B through the residual structure and the context-guided attention mechanism c Get visual information features M i ; Step 2.4: Repeat the above process using an iterative fusion strategy, and then combine the visual information feature M and the prompt feature Y i As the new adjusted visual information features and prompt features, they are continuously input into the visual supplementary prompt module VCP for iterative update to generate the final visual prompt feature Y6. The specific formula is as follows: M i+1 =ψ <Mi,Y i >(i≤5) (5) AND i+1 =ψ <M i ,AND i >(i≤5) (6) Where Ψ represents the operation of the visual supplementary cue module; i represents the number of iterations; Step 2.5: The label hint feature is initially encoded together with the text information feature using a self-attention mechanism, resulting in a label L s and Q s The characteristics of L s For labeling tips, Q s For text information; for label hint feature L s Upsampling is performed and the attention fusion and alignment module AFF is used to combine the visual position information with the label hint feature L s and text information feature Q s Projecting into two joint feature spaces, the alignment of multimodal knowledge is achieved. The specific formula is as follows: Q a =Γ AFF [Q s ,V s ] (7) C i =C AFF [β(L s ),Q a ] (8) where Γ AFF represents AFF operation; β represents upsampling operation; V s Represents the input image features; Q a Indicates Q s Alignment features of C i Indicates L s Alignment features; Step 2.6: Apply cross-attention to obtain visually guided text features R by using the hint encoded in the text as a query i By encoding the prompt text as a query, we apply cross attention to obtain multimodal information to guide the prompt feature G i , the specific formula is as follows: Where T represents the transpose operation of the matrix; L i It is the Label prompt feature; Q i It is the vectorized text question output feature; Step 2.7: During the iteration, feature C i With feature R i and feature G i Through repeated interactions in the form of cross-attention, the text prompt function G6 and the text information function R6 are finally generated. The specific formula is as follows: <C i+1 ,R i+1 ,C i+1 >=Φ<C i ,R i ,G i >(i≤5) (11) Where i represents the number of iterations; Φ represents the text supplement prompt module.

4. A grounded visual question answering method based on dual-modal prompt learning according to claim 3, characterized in that: Step 3 is as follows: features R6 and M6 are fused together through a fusion module composed of two iterative attention feature fusion modules iAFF to generate a fused embedding feature F i , the specific formula is as follows: F i =C iAFF {M6, C iAFF [M6, R6]} (12).

5. A grounded visual question answering method based on dual-modal prompt learning according to claim 4, characterized in that: Step 4 is as follows: embed the fusion feature F i Input into the pre-trained DeiT-base module, and connect F i Merge with Y6 and G6, and decode them together after merging.

6. A grounded visual question answering method based on dual-modal prompt learning according to claim 5, characterized in that: Specifically, step 5 includes: running the DeiT model on the feedforward network, the DeiT model receiving the decoded data in step 4 as input, and the DeiT model predicting instance bounding boxes and answers related to the question through a dedicated classification head and a target detection head, and generating a final text answer and a grounded answer.

7. A grounded visual question answering system based on dual-modal prompt learning, using the method according to any one of claims 1 to 6, characterized in that: The system includes: An input module, used for receiving questions and related pictures, and inputting the questions and related pictures into a pre-trained model CAT-ViL, wherein the pre-trained model CAT-ViL is a bimodal pre-trained model based on the Transformer architecture, which is used for jointly understanding visual and textual information; Feature extraction module, used to extract visual knowledge cues and textual knowledge cues from the pre-trained model CAT-ViL, extract image features, and vectorize text; The reasoning module uses the cue information to guide the reasoning of multimodal information including vision and text through hierarchical iterative fusion; The fusion module includes two iterative attention feature fusion modules iAFF, which are used to fuse the inferred fine-grained visual and textual information data; A decoding module, which is used to combine and decode the fused data with the inferred visual and textual prompt functions; The output module, including a dedicated classification head and an object detection head, is used to receive the decoded information, predict instance bounding boxes and answers related to the question through the DeiT model on the feed-forward network, and generate the final text answer and the ground answer.

8. A grounded visual question answering system based on dual-modal prompt learning according to claim 7, characterized in that: The feature extraction module comprises: The Box cue feature extraction submodule is used to extract Box cue features from the pre-trained model CAT-ViL; The Label prompt feature extraction submodule is used to extract Label prompt features from the pre-trained model CAT-ViL; Image feature extraction submodule, used to extract features from images using Feature extractor; The text vectorization processing submodule is used to use Tokenizer to perform vectorization processing on text questions.

9. A grounded visual question answering system based on dual-modal prompt learning according to claim 7, characterized in that: The reasoning module further comprises: The downsampling submodule is used to downsample the Box hint features; The image feature smoothing submodule is used to perform smoothing on the image features in all spatial dimensions to generate enhanced embedded image features; A cross-fusion submodule to fuse the enhanced embedded image features with the downsampled cue features; The iterative fusion submodule uses an iterative fusion strategy to repeat the cross-fusion process, and uses the visual information features and prompt features as new adjusted visual information features and prompt features, which are continuously input into the visual supplementary prompt module VCP for iterative update to generate the final visual prompt features; A label hint feature encoding submodule is used to encode the label hint feature together with the text information feature using a self-attention mechanism; The multimodal knowledge alignment submodule is used to use the attention fusion and alignment module AFF to project the visual position information, label prompt features, and text information features into two joint feature spaces to achieve the alignment of multimodal knowledge; a text feature guided submodule for obtaining visual guided text features by applying cross attention to the text encoded prompt as a query, and obtaining multimodal information guided prompt features by applying cross attention to the text encoded prompt as a query; Text function generation module: used to generate text prompt function and text information function through the text supplement prompt module in the iteration process.

Images