Single-stage gating multi-modal fusion method of two-stage cross attention

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a two-stage cross attention single-stage gating multi-modal fusion method, which comprises the following steps: enabling an input text to pass through an embedded layer and a text encoder to obtain a text vector; enabling an input image to pass through an image encoder to obtain an image vector; the text vector and the image vector are input into a modal feature fusion module, and the modal feature fusion module adopts a two-stage cross attention single-stage gating modal fusion mechanism to output a fusion vector; and enabling the fusion vector to pass through a decoder to obtain a predicted text. According to the method, the interaction effect between modes is improved, the illusion generated by the model is reduced, the calculation parameters of the model are effectively reduced, and the method has good universality and practicability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a two-level cross-attention single-stage gated multimodal fusion method. Background Art

[0002] Visual Question Answering (VQA) is a key task that combines traditional natural language processing with computer vision. Through the interaction between natural language questions and image content, VQA aims to obtain and understand the necessary information needed to solve the problem, thereby generating reasonable and accurate natural language answers. VQA strives to match or even surpass human capabilities in image understanding, enabling autonomous decision-making based on visual information or providing humans with the knowledge necessary to solve problems. VQA not only involves the integration of multimodal data but also requires models that possess sophisticated visual understanding, language reasoning, and commonsense reasoning capabilities. This makes it a key research topic in artificial intelligence, with broad application value across multiple fields. For example, in assisting the visually impaired, VQA technology can provide visual information about the surrounding environment to help users understand their surroundings and respond promptly to questions, enabling them to make better judgments and decisions. In the field of autonomous driving, VQA technology can provide vehicles with the ability to perceive complex traffic scenarios, allowing users to more easily understand the surrounding conditions of the vehicle and the environment. In medical image analysis, VQA technology can help doctors quickly obtain key information from medical images, providing details that may be overlooked, and providing more reliable support for doctors' diagnosis of diseases. VQA has greatly improved the information interaction capabilities of computer systems.

[0003] Although VQA technology has made significant progress in recent years, it still faces many challenges. First, free-form, open-ended questions require models to possess extensive world knowledge and reasoning capabilities to cope with diverse questions. Second, there are still deficiencies in the interactive fusion of visual and linguistic modalities, which play an almost dominant role in human cognition. This is primarily reflected in the difficulty in establishing correct semantic understanding relationships between information from different modalities. Furthermore, incomplete dataset types and sizes may lead to poor performance of learning models on specific types of questions and insufficient generalization capabilities. Therefore, improving the robustness, reasoning capabilities, and fairness of VQA models remains a key research focus.

[0004] The emergence of large language models such as GPT-3 and LLaMA has enabled significant breakthroughs in VQA technology for handling open-ended questions and few-shot learning, demonstrating superior performance across multiple natural language processing tasks. Some work has fine-tuned open-source language models, enabling them to perform even better on downstream tasks. In reasoning question answering tasks, these models only require a single step of reasoning to arrive at an answer, taking the question as input. However, due to the unknowable nature of the intermediate reasoning process, they often perform poorly on tasks requiring complex reasoning. To address this, researchers have proposed a thought chaining technique that mimics human cognitive thinking, prompting large models to generate intermediate inferences based on the question and ultimately arrive at the final answer. This approach fully leverages the knowledge reserves of large models, enabling them to tackle more complex reasoning tasks. In particular, the DeepSeek-R1 model, using thought chaining technology, has achieved reasoning performance comparable to the OpenAI o1 model using only a small model, demonstrating the critical role of thought chaining in unlocking model reasoning capabilities.

[0005] Thought chaining technology enables models to possess stronger reasoning capabilities without requiring additional training. The quality of the intermediate inferences generated by these models will affect the accuracy of the final answer. While these thought chaining models typically focus solely on using language modalities as input, humans, when handling complex reasoning question-answering tasks, can not only obtain information from the language modality of the question but also combine it with visual image data for comprehensive analysis and reasoning, resulting in more accurate answers.

[0006] To enable language models to achieve this, one approach is to use a summary model to summarize the image data, generating a brief textual description. This is then concatenated with the language modality data and encoded as input to a larger language model, enabling the model to interact with both modalities simultaneously. However, this approach essentially still involves interaction between the language model and the text modality. Its interaction with the visual modality is primarily accomplished through the results of the image summary model. The model's visual understanding is influenced by the summary model's perspective and accuracy of the image, significantly reducing its interactivity with the visual modality and, in turn, affecting the model's predictions.

[0007] Another method is to let the language model act directly on the visual features extracted from the image, enhance the interactivity of the model and the visual modality, and project the language modality features and the visual modality features into the same encoding space through a projection connection layer, which is then used as the input of the language model decoder to obtain the output prediction result. As a trainable layer, the projection connection layer enables the model to interact with visual features and language features at the same time, which can effectively enhance the model's multimodal interaction understanding ability. The structural design of the projection connection layer represents the way the model interacts with different modalities, which will directly affect the final output results. For this reason, a linear projection method is used in the prior art, which uses a linear projection layer to connect the features of different modalities as input and output to a space of input token length that can be accepted by the language model. Due to the simple connection method, this method usually makes it impossible for the model to effectively interact and understand multimodal features.

[0008] Among existing technologies, Enigma-CoT, building on the T5 architecture, proposes the use of multi-hop gated cross-attention for modal fusion. It also employs thought chaining techniques to demonstrate impressive performance in complex reasoning tasks. Using fewer than 250M parameters, the model surpasses BLIP-2 on the ScienceQA dataset. However, because it employs a gating mechanism behind each cross-attention mechanism, and cross-attention itself controls information flow, important information after fusion is excessively suppressed, reducing the effectiveness of modal fusion. Multimodal-CoT proposes a model that uses a single Transformer structure to connect the linguistic and visual modalities. It also applies thought chaining techniques to the reasoning process of the T5 model, demonstrating excellent performance in question-answering reasoning and achieving state-of-the-art results on the ScienceQA dataset. However, its single Transformer structure overly suppresses intermodal interaction information, and still suffers from hallucinations when generating reasoning content. Summary of the Invention

[0009] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the modal fusion effect is low, reasoning hallucinations are prone to occur, and the accuracy of answer prediction is reduced.

[0010] To solve the above technical problems, the present invention provides a two-level cross-attention single-stage gated multimodal fusion method, comprising:

[0011] Build a multimodal fusion model, including an embedding layer, a text encoder, an image encoder, a multimodal feature fusion module, and a decoder;

[0012] The process of the multimodal fusion model generating predicted text from input text and input image includes:

[0013] Pass the input text through the embedding layer and text encoder to obtain the text vector H l ; Pass the input image through the image encoder to obtain the image vector H v ;

[0014] The text vector H l and image vector H v Input modality feature fusion module, text vector H l After a single-head self-attention encoding, we get vector A l ; Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 ; Vector A v,0 and image vector H v Perform another single-head cross attention encoding to obtain the cross vector A v,l ; The text vector H l and the cross vector A v,l After concatenation, the gate vector is predicted through linear transformation and Sigmoid activation function; the text vector H l and the cross vector A v,l After proportional fusion with the gated vector, a fused vector is obtained;

[0015] Pass the fused vector through the decoder to obtain the predicted text.

[0016] Preferably, the process of reasoning using the multimodal fusion model includes:

[0017] The question text and image are input into the multimodal fusion model of the inference stage to obtain the intermediate inference text; after splicing the intermediate inference text and the question text, they are input into the multimodal fusion model of the inference stage with the image to obtain the answer text.

[0018] Preferably, the process of training the multimodal fusion model includes:

[0019] First, the reasoning data in the ScienceQA dataset is used as a supervisory signal, and the question text and image are used as input to train the text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the reasoning stage, and output the intermediate reasoning text; then the answer in the ScienceQA dataset is used as a supervisory signal, and the intermediate reasoning text, question text, and image are used as input to train the text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the inference stage, and output the answer text.

[0020] Preferably, the text vector H l After a single-head self-attention encoding, we get vector A l , the formula is:

[0021] A l =SHA1(H l ,H l ,H l )

[0022] Among them, SHA1 represents the single-head self-attention encoding, H l Represents the text vector, A l represents the output vector of a single-head self-attention encoding;

[0023] Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 , the formula is:

[0024] A v,0 =SHA2(A l ,H v ,H v )

[0025] Among them, SHA2 represents the first single-head cross attention encoding, H v represents the image vector, A v,0 represents the output vector of the first single-head cross attention encoding;

[0026] Vector A v,0 and image vector H v Perform another single-head cross attention encoding to obtain the cross vector A v,l , the formula is:

[0027] A v,l =SHA3(A v,0 ,H v ,H v )

[0028] Among them, SHA3 represents the second single-head cross attention encoding, A v,l Represents the output vector of the second single-head cross attention encoding;

[0029] The text vector H l and the cross vector A v,l After splicing, the gate vector is predicted through linear transformation and Sigmoid activation function. The formula is:

[0030] λ=Sigmoid(Linear(H l ,A v,l )),λ∈[0,1]

[0031] Where λ represents the gate vector, Linear(·) represents the linear transformation, and Sigmoid(·) represents the Sigmoid activation function;

[0032] The text vector H l and the cross vector A v,l After proportional fusion with the gated vector, the fusion vector is obtained, and the formula is:

[0033] H out =λ×A v,l +(1-λ)×H l

[0034] Among them, H out Represents the fusion vector.

[0035] Preferably, the image encoder adopts a VIT encoder.

[0036] Preferably, the image encoder is connected to a linear projection layer to map the image vector to the same dimensional space as the text vector.

[0037] Preferably, before the input text passes through the embedding layer, a word segmenter is used to split the input text into multiple subwords, and each subword is mapped to an ID.

[0038] Preferably, the word segmenter splits the input text into multiple subwords based on a Unigram language model.

[0039] Preferably, the text encoder adopts a T5 encoder, including multiple T5 blocks, each T5 block including a self-attention layer and a feedforward layer connected in sequence.

[0040] Preferably, the decoder adopts a T5 decoder, including multiple T5 blocks, each T5 block including a self-attention layer, a cross-attention layer and a feedforward layer connected in sequence.

[0041] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0042] The single-stage gated multimodal fusion method with two-level cross-attention described in the present invention uses only two cross-attentions to perform inter-modal interaction between text and visual modal information, and then completes the fusion of language and visual representations through a gating mechanism. It simplifies the existing modal fusion mechanism while retaining important information, thereby improving the effect of inter-modal interaction, reducing the hallucinations generated by the model, and improving the accuracy of VQA answers. In addition, the present invention effectively reduces the computational parameters of the model, reduces the computing resources required for prediction, and achieves a balance between low parameter count and high performance, so that the multimodal fusion model can be deployed on edge devices, with good versatility and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 It is a structural diagram of the multimodal fusion model of the present invention;

[0045] Figure 2 This is a structural comparison diagram of the multimodal fusion method of the present invention and the multimodal fusion method of the prior art, in which Figure 2 (a) is the structural diagram of Enigma-CoT's multimodal fusion method. Figure 2 (b) is a structural diagram of the multimodal fusion method of the present invention;

[0046] Figure 3 It is the structural diagram of the modal feature fusion module;

[0047] Figure 4 This is the classification statistics of the ScienceQA dataset;

[0048] Figure 5 This is a confusion matrix heat map of the prediction results of the model of the present invention drawn according to the ScienceQA test set option;

[0049] Figure 6 This is a comparative line chart of the prediction accuracy of thought chain models with similar parameter values ​​on the ScienceQA dataset. DETAILED DESCRIPTION

[0050] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0051] In the application of artificial intelligence and deep learning, multimodal learning is becoming a key technology for improving model understanding and generation capabilities. The effective integration of multimodal information is crucial for models to understand multimodal information and generate appropriate inference content. In text and image synthesis tasks, a well-designed framework can fully leverage the complementary information of the two modalities, improving the model's inference performance.

[0052] In order to alleviate the problem of reasoning hallucinations in reasoning question answering tasks and enable more effective interaction between multiple modalities, this paper proposes a multimodal fusion model combined two-level cross attention modal with single-level gate (M-TCM), which includes an embedding layer, a text encoder, an image encoder, a multimodal feature fusion module (Two-Level Cross Attention Modal with Single-Level Gated, TCM) and a decoder.

[0053] The first embodiment of the present invention proposes a two-level cross-attention single-stage gated multimodal fusion method, which uses a multimodal fusion model to generate predicted text from input text and input image. The process includes:

[0054] S1: Pass the input text through the embedding layer and text encoder to obtain the text vector H l .

[0055] The data involved in this invention mainly include text modality issues and corresponding image modality information, which need to be effectively processed to adapt to the input space of the model and enable the model to effectively understand the input information.

[0056] Before the input text passes through the embedding layer, the word segmenter is used to split the input text into multiple subwords, and each subword is mapped to an ID. This embodiment uses the tokenizers word segmenter to split based on the Unigram language model subword level encoding method. Unlike the traditional word segmentation method based on space splitting, subword splitting can be performed directly in the original text, which can better handle unseen words. Complex words can be automatically split into multiple subwords, reducing the size of the model vocabulary while retaining semantic information. The vocabulary size is 32100, and the words in the table will be mapped to unique IDs, and for special characters such as <pad>(ID:0, filler), <unk>(ID:2, unknown word),<extra_id_0> (ID:32099, mask word) is used to express special meanings. When the input text does not reach the set maximum length, it will be used <pad>Padding is performed to meet the input requirements of the model. After word segmentation is completed, each token will be mapped to a unique integer ID.

[0057] The embedding layer is used to convert the ID output by the tokenizer into a one-dimensional vector representation.

[0058] The text encoder uses a T5 encoder to extract text features from the one-dimensional vector representation output by the embedding layer to obtain a text vector H l The T5 encoder includes multiple T5 blocks, each of which includes a self-attention layer and a feedforward layer connected in sequence to learn the context information of the sequence and obtain the hidden layer text self-attention vector.

[0059] The decoder also adopts a T5 decoder, including multiple T5 blocks, each T5 block including a self-attention layer, a cross-attention layer and a feedforward layer connected in sequence.

[0060] After being decoded by the decoder, the vector is mapped to the vocabulary through a linear layer, and then converted into the corresponding text output through the mapping from ID to vocabulary.

[0061] The T5 architecture unifies all natural language processing tasks (such as machine translation and article summarization) into text-to-text conversion problems, using the standard Transformer encoder-decoder architecture. The encoder is responsible for understanding the input text and extracting high-level semantic features; the decoder is responsible for generating the target text, gradually outputting the final result using autoregressive generation. This architecture excels in a variety of tasks, possesses strong generalization capabilities, and can perform transfer learning across multiple tasks, eliminating the need to design separate model architectures for different tasks. This facilitates the development process of natural language processing tasks and simplifies the architecture by requiring only pre-training and fine-tuning for specific tasks, without requiring adjustments to the architecture itself.

[0062] S2: Pass the input image through the image encoder to obtain the image vector H v .

[0063] For feature extraction in image modalities, the most commonly used encoders are DETR, CLIP, and VIT. Since the VIT encoder is considered the best performing encoder in Multimodal-CoT and is widely used in various tasks, this embodiment also uses the VIT encoder as the image encoder for image feature extraction. All processed images are processed by VIT into high-dimensional features.

[0064] In order to be able to perform modal fusion with text features, the image encoder is connected to a linear projection layer to map the image vector to the same dimensional space as the text vector to meet the multimodal fusion requirements of the multimodal feature fusion module.

[0065] In the multimodal fusion model of the present invention, since T5 is the process of decomposing the codec into input text to decoded output text, the addition of visual features needs to be fully and effectively integrated with the text information. A more appropriate position is to introduce a modal feature fusion mechanism after the encoding is completed to fuse the encoded image and text features, with less invasion of the original semantic information of the trained T5 model, so that different modalities can learn from each other, thereby obtaining an effective semantic representation, which is convenient for the decoder to decode.

[0066] S3: Transform the text vector H l and image vector H v Input the modal feature fusion module to obtain the fusion vector.

[0067] In question-answering tasks, the addition of visual information allows the model to obtain more contextual information in the process of understanding the question, thereby generating higher-quality reasoning content, significantly reducing the proportion of hallucination errors in the model-generated reasoning, and improving the accuracy of the final predicted answer.

[0068] Figure 2 (a) in the figure is a structural diagram of the multimodal fusion method of Enigma-CoT. Enigma-CoT imitates the process of human repeated thinking about problems, and proposes to use a multi-hop cross-attention mechanism to fuse language and visual modal information, thereby improving the quality of model generation reasoning. This mechanism performs cross-attention encoding on the input text vector and image vector, and then uses a gating mechanism to fuse the encoded vector with the text vector. The fused vector is then repeatedly subjected to this cross-fusion process with the image vector to achieve the purpose of fully understanding and fusing the image content. In the gating mechanism, a gating vector is predicted to control the proportion of the fused output vector relative to the input vector in the fusion vector. However, due to the control effect of the cross-attention mechanism itself on the information flow, following a gating mechanism will suppress the expression of important information. Therefore, the present invention proposes a simplified single-stage gating module of two-stage cross-attention modal fusion as a modal feature fusion module to fuse different modal information. Figure 2 (b) is the structural diagram of the multimodal fusion method of the present invention. The modal feature fusion module proposed in the present invention removes the excessive suppression of important information, simplifies the network structure, and retains the gating mechanism at the last level to ensure faster and more stable network convergence.

[0069] Figure 3 This is the structural diagram of the modal feature fusion module. Specifically, the text vector H l After a single-head self-attention encoding, we get vector A l , the formula is:

[0070] A l =SHA1(H l ,H l ,H l )

[0071] Among them, SHA1 represents the single-head self-attention encoding, H l Represents the text vector, A l represents the output vector of a single-head self-attention encoding;

[0072] Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 , the formula is:

[0073] A v,0 =SHA2(A l ,H v ,H v )

[0074] Among them, SHA2 represents the first single-head cross attention encoding, H v represents the image vector, A v,0 represents the output vector of the first single-head cross attention encoding;

[0075] Vector A v,0 and image vector A v Perform another single-head cross attention encoding to obtain the cross vector A v,l , the formula is:

[0076] A v,l =SHA3(A v,0 ,H v ,H v )

[0077] Among them, SHA3 represents the second single-head cross attention encoding, A v,l Represents the output vector of the second single-head cross attention encoding;

[0078] The text vector H l and the cross vector A v,l After splicing, the gate weight is predicted through linear transformation and Sigmoid activation function. The formula is:

[0079] λ=Sigmoid(Linear(H l ,A v , l )),λ∈[0,1]

[0080] Where λ represents the gate weight, Linear(·) represents the linear transformation, and Sigmoid(·) represents the Sigmoid activation function;

[0081] The text vector H l and the cross vector A v,l After proportional fusion with gating weights, the fusion vector is obtained, and the formula is:

[0082] H out =λ×A v,l +(1-λ)×H l

[0083] Among them, H out Represents the fusion vector.

[0084] S4: Pass the fused vector through the decoder to obtain the predicted text.

[0085] In the reasoning question-answering task scenario under the Thinking Chain CoT model, the M-TCM model can be trained separately using different supervisory signals to produce qualified reasoning content. To utilize Thinking Chain technology to improve the model's reasoning performance, the reasoning data from the ScienceQA dataset is first used as a supervisory signal, with the question text and image as input. The text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the reasoning phase are trained. This allows the model to generate intermediate information about the solution to the corresponding problem based on the question and image information, and outputs the intermediate reasoning text. Then, the answers from the ScienceQA dataset are used as supervisory signals, with the intermediate reasoning text, question text, and image as input. The text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the inference phase are trained to output the answer text. By repeating this process, the multimodal fusion model can be applied to multiple steps of reasoning to generate more comprehensive answers.

[0086] The process of reasoning using the multimodal fusion model includes: inputting the question text and image into the multimodal fusion model of the reasoning stage to obtain the intermediate reasoning text; after splicing the intermediate reasoning text and the question text, inputting the image into the multimodal fusion model of the inference stage to obtain the answer text.

[0087] Specifically, consider the i-th problem P i and corresponding to n answer options {a i,1 ,a i,2 ,…,a i,n }, requiring the model to select the correct option a i,label In the process of reasoning generation, the goal is to i and {a i,1 ,a i,2 ,…,a i,n }, Generate inference data Y through multimodal fusion model i . Assume that the problem P i The language modality and visual modality data are X i,l and X i,v , then the target inference text Y i The probability of generating is expressed as:

[0088]

[0089] Among them, θ r are the learnable parameters of the multimodal fusion model in the inference phase, Represents the generated target reasoning text Y i length.

[0090] In the answer generation process, the goal is to i 、{a i,1 ,a i,2 ,…,a i,n } and the intermediate inference Y generated in the previous stage i Infer the final answer A i . Change the language mode of the question to X i，l and intermediate inference data Y i Splicing to get X i,ly , and then combined with the visual modality information X i,v Input the multimodal fusion model together to get the predicted answer A i . A i The probability of generating is expressed as:

[0091]

[0092] Among them, θ a represents the learnable parameters of the answer inference module, The generated target answer text A i length.

[0093] The multimodal fusion model constructed by the present invention mainly performs multimodal fusion on the preprocessed data, and then decodes and predicts the output results. First, during data preprocessing, the text modal information is embedded in the form of word vectors through the embedding layer, and then self-attention encoding is performed through the T5 encoder; for image modal information, the VIT encoder is used to extract image features. Subsequently, in the multimodal fusion part, after obtaining the extracted image features, due to the deviation between its feature dimensions and the text vector, it is not conducive to feature fusion by the multimodal feature fusion module. Therefore, a linear projection layer is used to transform the image vector feature dimensions to the same dimensional space as the text vector. Then, the features of the two modalities are cross-fused through the multimodal feature fusion module to enable full interaction and understanding of the information between the modalities, and obtain a feature vector that meets the input dimension of the decoder. Finally, in the decoding prediction part, the T5 block of the T5 decoder is similar to the T5 block structure of the encoder, except that a standard attention mechanism is added after each self-attention mechanism to focus on the output of the encoder. After decoding multiple T5 blocks, the output result of the final model can be obtained. It is worth noting that Figure 1 The dotted part in the figure indicates that the model can take the generated reasoning content as input and further generate the next step of reasoning content or get the answer to the question, thus realizing the thought chain reasoning process.

[0094] In summary, the single-stage gated multimodal fusion method with two-level cross-attention described in the present invention uses only two cross-attentions to perform inter-modal interaction between text and visual modal information, and then completes the fusion of language and visual representation through a gating mechanism. It simplifies the existing modal fusion mechanism while retaining important information, thereby improving the effect of inter-modal interaction, reducing the hallucinations produced by the model, and improving the accuracy of VQA answers; and the present invention effectively reduces the computational parameters of the model, reduces the computing resources required for prediction, and achieves a balance between low parameter count and high performance, so that the multimodal fusion model can be deployed on edge devices, with good versatility and practicality.

[0095] To verify the effectiveness of the method under limited computing resources, this example used only the Flan-Alpaca base model version with less than 300M parameters for training, rather than the larger 700M-level large version. All experiments in this example were conducted on a single NVIDIA RTX 4090 24G GPU.

[0096] 1. Dataset Selection

[0097] The ScienceQA dataset is a large-scale science question-answering dataset designed specifically for multimodal machine learning tasks, covering multiple-choice questions. The questions in this dataset are primarily sourced from elementary and middle school science curricula on the IXL resource learning platform. They are manually cleaned for formatting errors, sensitive information, or academic errors, and core concepts are refined and presented in a structured manner to ensure a complete logical chain of "premise-intermediate steps-conclusion." The annotated content is cross-reviewed by educational experts for scientific accuracy, logical rigor, and linguistic standardization. Human participants are recruited to answer the questions to verify that the annotated explanations are understandable. Explanations that confuse the majority of participants are revised to ensure the scientific and educational value of the questions. The ScienceQA dataset contains 21,208 question examples, of which 10,332 (48.7%) include images, 10,220 (48.2%) provide textual context, and 6,532 (30.8%) have both image and textual context. The design of this dataset fully considers the needs of multimodal learning, enabling the model to perform comprehensive understanding and inference at multiple levels such as vision, language, and knowledge reasoning.

[0098] In addition, the ScienceQA dataset covers three major subject areas: natural sciences, social sciences, and language sciences. It is subdivided into 26 topics, 127 categories, and involves 379 subject-related skills. This broad knowledge coverage makes the dataset not only suitable for general visual question answering (VQA) tasks, but also provides an important research foundation for intelligent tutoring systems, cross-modal reasoning tasks, and knowledge-driven question answering systems in the field of education. In order to more intuitively display the distribution of questions in the dataset, the dataset is divided by level and subject to statistical coverage of the questions, such as Figure 4 As shown, Figure 4 (a) in the figure is a pie chart of the data set classified by level. Figure 4 (b) is a bar chart of the dataset classified by subject. It can be seen that the main problems are concentrated between the middle levels 2 to 8. The total number of image data and non-image data by subject classification is almost equal, and natural science problems occupy the majority.

[0099] During the experiment, this example strictly followed the official data partitioning strategy, dividing the entire dataset into 12,726 training examples, 4,241 validation examples, and 4,241 test examples, with a partitioning ratio of approximately 3:1:1. It is worth noting that a key feature of the ScienceQA dataset is that most questions come with detailed answers and knowledge bases, allowing the model to not only learn how to choose the correct answer but also enhance its reasoning and explanation capabilities during training. This feature helps improve the interpretability of the model and provides valuable reference for further research.

[0100] The ScienceQA dataset, with its rich multimodal information, wide subject coverage, and detailed annotations, has become an important benchmark dataset in multimodal artificial intelligence research, science education assistance systems, and knowledge reasoning tasks.

[0101] 2. Selection of evaluation indicators

[0102] The ScienceQA dataset is a benchmark dataset specifically designed to evaluate the model's ability to answer questions in multiple disciplines. It consists of multiple multiple-choice questions. Since the answer to each question is certain and unique, this example uses accuracy, BLEU, Rouge-L, and F1-Score as evaluation metrics. To better analyze the performance of models in different fields, the ScienceQA dataset subdivides the evaluation metrics into the following eight categories:

[0103] (1) Natural Science (NAT): mainly covers issues in the fields of natural sciences such as physics, chemistry, and biology.

[0104] (2) Social Science (SOC): including issues in social science fields such as psychology, sociology, and economics.

[0105] (3) Language Science (LAN): involves language-related issues such as linguistics and semantics.

[0106] (4) Contains contextual clues (TXT): This type of question provides additional text information as a clue in the question stem.

[0107] (5) Contains images (IMG): The question contains image information and needs to be answered in combination with the image content.

[0108] (6) No contextual cues and images (NO): The question contains only the basic question stem without additional contextual cues or images.

[0109] (7) Low-complexity problems (G1-6): relatively simple problems suitable for primary school students in grades 1-6.

[0110] (8) High-complexity problems (G7-12): relatively complex problems suitable for middle school students in grades 7-12.

[0111] For each category, this embodiment calculates the prediction accuracy of the model in that category, thereby comprehensively evaluating the performance of the model in different fields and difficulty levels. Finally, this embodiment uses the Average indicator as the overall prediction accuracy evaluation indicator of the model, which is calculated as follows:

[0112]

[0113] Among them, k represents one of the 8 categories listed above, Correct Number k The number of questions that the model predicts correctly in category k, Total Number k is the total number of questions in category k, Accuracy k is the prediction accuracy under category k. The higher the accuracy of each category, the better the reasoning performance of the model in the corresponding category, and the more high-quality reasoning content can be generated to point to the correct answer.

[0114] In this way, we can more comprehensively measure the overall performance of the model on the ScienceQA dataset and provide direction for further model optimization.

[0115] BLEU is a metric commonly used in machine translation quality assessment, which measures the degree of n-gram matching between machine-generated text and reference text, as shown in the following formula:

[0116]

[0117]

[0118] Among them, p n is the precision of the n-gram, w n is the weight of the nth item, N is the total number of items, BLEU represents the final score, BP is the length penalty item, c is the length of the generated text, and r is the length of the reference text.

[0119] The Rouge metric is used to evaluate the degree of overlap between generated text and one or more reference texts in natural language generation tasks. It has multiple variations and can be measured using 1-gram, 2-gram, and longest common subsequence metrics. The longest common subsequence, or Rouge-L, approach can measure the structural consistency of the entire sentence and is unaffected by sentence length, making it suitable for complex natural language generation tasks. Its calculation formula is shown below:

[0120]

[0121] Among them, c is the length of the generated text, r is the length of the reference text, and LCS is the length of the longest common subsequence.

[0122] F1-Score is a commonly used metric for evaluating model performance in binary or multi-class classification problems, particularly in cases of class imbalance. It is the harmonic mean of precision and recall, calculated as shown below.

[0123]

[0124] Among them, Precision represents the precision rate and Recal l represents the recall rate.

[0125] In this way, we can more comprehensively measure the overall performance of the model on the ScienceQA dataset and provide direction for further model optimization.

[0126] In addition, to initialize the weights of the M-TCM model, this embodiment uses the Flan-Alpaca-Base model (223M) and further trains on this basis. The initial learning rate of the experiment is set to 8e-5. In the inference phase and the answer generation phase, the training batch size is 6 and 16 respectively, while the batch size of the evaluation phase is 32 and 64 respectively. The maximum output sequence length is set to 512 (inference phase) and 64 (answer generation phase). The entire training process lasted for 20 epochs.

[0127] 3. Analysis of experimental results

[0128] Tables 1 and 2 show the accuracy of the intermediate reasoning content generated by the model and the final predicted answers, including evaluation metrics such as BLEU1, BLEU4, Rouge-L, and accuracy. The BLEU1 and Rouge-L metrics were 92.32% and 94.04%, respectively, indicating that the overall sentence structure of the generated text and the reference text are relatively consistent, with a high number of shared vocabulary. The BLEU4 metric was 79.51%, indicating that the phrase-level match was accurate and coherent. This demonstrates that the model performed well in generating intermediate reasoning content, while the average accuracy of the final predictions reached 86.39%.

[0129] Table 1 BLEU1(%), BLEU4(%) and ROUGE-L(%) scores for intermediate reasoning

[0130]

[0131] Table 2 Prediction results on the ScienceQA test set (accuracy %)

[0132]

[0133] The options in the ScienceQA dataset are single-label multiple-choice questions with 5 options: A, B, C, D, and E. Each question has a specific answer. A confusion matrix heat map is drawn based on the statistical results of the ScienceQA test set options to evaluate the model prediction performance. The results are as follows: Figure 5 shown.

[0134] from Figure 5 It can be seen that the colors of the heat map are mainly concentrated in the diagonal part, indicating that the model prediction is relatively accurate, but there is also an uneven distribution of samples, mainly concentrated in the upper left corner, indicating that the distribution of options for the question is not uniform, and most questions have only 2-3 options, which will cause the statistical F1-Score indicator to be structurally distorted. Therefore, this embodiment calculates the F1-Score scores according to the number of different options to better measure the prediction performance of the model in different options. The group statistical results and the final weighted average results are shown in Table 3. After weighted averaging, it can be seen that the macro, micro and weighted scores are almost the same, indicating that the statistical results are relatively balanced and there is no obvious bias. The accuracy rate is highest on the two-choice questions, reaching 89.1% correct. The results on the four-choice questions are slightly lower than the two-choice questions, and also show good stability. The number of samples for the five-choice questions is small and cannot well reflect the prediction results of the model. The final weighted average result better reflects the overall prediction performance of the model.

[0135] Table 3 F1-Score statistics by number of options

[0136]

[0137] 4. Experimental Comparative Analysis

[0138] In the experiment, this example compared the performance of the proposed M-TCM model with a series of benchmark models. First, this example compared the modal alignment methods of several major models, categorizing the alignment methods according to the encoding methods of the text and image modalities and the modal fusion methods, as shown in Table 4. The model of the present invention almost uses the modal fusion method with the least number of trainable parameters and achieves good performance in complex reasoning tasks.

[0139] Table 4 Comparison of multimodal alignment methods

[0140]

[0141]

[0142] In the table, "Embedding" indicates embedding the model directly using word vectors, "Linear" indicates embedding followed by matrix projection, "&" indicates the concatenation of two operations, and "Cross" and "Self" represent the cross-attention and self-attention mechanisms, respectively. NFNet is a variant of ResNet. Note that since Patch-TRM does not specify specific parameter sizes, this example uses the Bert-Small model provided in the official code to calculate the trainable parameter size. For other models, the minimum parameter size model used in their respective papers is used.

[0143] Subsequently, this example compares some major modal fusion benchmark models and mainstream large language models such as GPT3.5 and GPT-4 with the M-TCM model proposed in this invention on the ScienceQA dataset. The results are summarized in Table 5.

[0144] Table 5 Comparison of prediction results with different multimodal fusion models (accuracy %)

[0145]

[0146] By comparing the experimental results in Table 5, we can find that the M-TCM multimodal fusion model of the present invention outperforms other mainstream fusion models, and has good performance in processing multimodal tasks while maintaining low parameters. The following is a detailed comparative analysis.

[0147] Since the OpenFlamingo model was not fine-tuned using instructions, it clearly did not fully understand the questions before giving answers when answering different questions, with an accuracy rate of only 39.27%. BLIP-2 was fine-tuned based on the Flan-T5-XXL model and used a frozen codec to train the projection layer. Since it was exposed to different instructions during training, it has strong adaptability to question-answering scenarios and even slightly exceeded GPT-3.5, achieving an accuracy rate of 74.17%.

[0148] The pre-trained visual fusion models ViLT, Patch-TRM, and VisualBERT are primarily used for classification tasks. Their key characteristic is that they only produce simple classification outputs when predicting answers, scoring each option and selecting the highest-scoring answer. ViLT uses a relatively lightweight modal fusion approach, while the Patch-TRM model employs a more complex image pyramid structure to fuse text and images at different levels. The VisualBERT model utilizes a bidirectional Transformer architecture based on BERT, inheriting some of BERT's language capabilities and achieving slightly higher accuracy than the previous two models. These three visual fusion models have a small number of parameters (approximately 100M) and limited expressive power, resulting in prediction accuracy of only approximately 61%. In contrast, the M-TCM model, which uses a single-stage gating mechanism with two-level cross-attention for modal fusion, achieves a prediction accuracy of 86.39%, significantly surpassing these visual question answering models.

[0149] LaVIN-13B is a fine-tuned model based on LLaMA-13B. By incorporating an adapter for training, it enables rapid adaptation to text and image commands. It achieves superior results compared to pre-trained vision models, with a prediction accuracy of 77.54%. Qwen-VL and LLaVA-1.5 are both models trained on large-scale image-text pairs for visual reasoning tasks. With a parameter of 7 bytes, both models achieved 67.1% and 71.6% accuracy, respectively, on image-only prediction tasks, demonstrating significant advantages over other pre-trained vision models. LLaVA-Mini utilizes modality pre-fusion and visual token compression to improve prediction performance with lower computational effort. It achieved an accuracy of 83.1% on IMG classification, second only to the Enigma-CoT model, demonstrating its significant advantage in multimodal fusion. DIEM separates the question and image separately, uses CLIP to match text and image modal information, and finally integrates the visual information and the answer to the sub-question to generate the final result. This method uses the existing CLIP model to process the correlation of multimodal information. The final result is only 68.88%, which is significantly lower than other large language models. This embodiment also compares the performance of the M-TCM model with open source and commercial large models such as GPT-3.5 and GPT-4. It can be found that GPT-3.5, GPT-3.5w / CoT and GPT-4w / CoT versions, as reference models, achieved accuracy rates of 73.97%, 75.17% and 83.99% respectively. After using the thinking chain technology, the prediction accuracy of GPT-3.5 increased by 1.2%. Obviously, due to the powerful knowledge reserve capacity of the basic model of GPT-4, the performance of its thinking chain version is more significantly improved than that of GPT-3.5, reaching 8.82%. The prediction results of these models are all lower than the model of the present invention, and the number of parameters is very large, which is not conducive to the lightweight deployment of the model.

[0150] In comparison, the M-TCM model achieved a prediction accuracy of 86.39% while maintaining a low number of parameters, surpassing models such as GPT-3.5 and GPT-4w / CoT.

[0151] The Hot-Base model employs a two-stage multimodal reasoning framework based on hypergraph thinking. It constructs a textual hypergraph and a visual hypergraph, respectively, and enables interaction between them through cross-modal collaborative attention graph learning. This significantly improves performance over traditional modal fusion models, achieving an accuracy rate of 81.42%. The Multimodal-CoT and Enigma-CoT models utilize thought chaining technology. The experimental results for the Multimodal-CoT model in Table 5 are actual prediction results after retraining for 20 epochs on an RTX 4090 GPU using the parameters provided in the paper. Since Enigma-CoT does not have open source training code, the experimental results in the original paper are used here. As can be seen in Table 5, the M-TCM model outperforms the Multimodal-CoT model on all classification metrics. In terms of overall average accuracy, the M-TCM model improved by 1.55% over Multimodal-CoT. Furthermore, in classifications with text and image context, Multimodal-CoT only achieved 83.04% and 80.66% accuracy, while M-TCM achieved 85.34% and 82.35%, improvements of 2.3% and 1.69%, respectively. These results demonstrate that the proposed modality fusion method and reasoning selection strategy have significant advantages in visual question answering tasks.

[0152] In order to facilitate a more intuitive comparison of the fusion effects of the Multimodal-CoT model and the Enigma-CoT model with similar parameters and using the thinking chain technology with the M-TCM multimodal fusion model proposed in this invention, this embodiment further graphically displays the results in Table 5, as shown in Figure 5. Figure 6 As shown in the figure, the line chart clearly shows that the M-TCM model surpasses the Multimodal-CoT model in all indicators, with the most significant improvements in LAN language classification and TXT text context classification. Although the Enigma-CoT model has advantages in multiple indicators, its SOC social science classification achieves very low results. This may be due to the ineffective processing of the problem feature model in this category. The overall average indicator is still lower than that of the M-TCM model, which indicates that the TCM fusion mechanism has been effectively optimized, thereby improving the fusion effect.

[0153] Overall, a comparison of the M-TCM model's performance with other baseline models shows that it outperforms other models in multimodal data fusion and reasoning tasks, achieving significant performance improvements across multiple classifications, particularly in multimodal problem handling. Even with a modest model size (256MB), it surpasses the performance of large language models like GPT-4, demonstrating its significant potential for application in multimodal reasoning tasks and providing valuable insights for subsequent research.

[0154] This embodiment further conducted an ablation experiment to more deeply explore the effectiveness of the module proposed in the present invention under different settings. Specifically, for the modal fusion mechanism, the ablation experiment was set as follows: ① using a single Transformer fusion mechanism, ② using a Multi-hop modal fusion mechanism, and ③ using a TCM modal fusion mechanism. In the first experiment, this embodiment sets the modal fusion mechanism to a single Transformer (Single Transformer) and does not consider the improved method proposed in the present invention; in the second experiment, in order to highlight the effect of the present invention on the improvement of the multi-hop cross-attention fusion method proposed in Enigma-CoT, this embodiment sets the modal fusion mechanism to a Multi-hop cross-attention mechanism for comparison; in the third experiment, the multimodal feature fusion module proposed in the present invention is used as the modal fusion mechanism. By comparing these three different modal fusion mechanisms, the advantages of TCM in multimodal tasks can be revealed. The experimental results are shown in Table 6.

[0155] Table 6 Experimental results of different modal fusion mechanisms (accuracy %)

[0156]

[0157] Table 6 shows the experimental results of different modal fusion mechanisms. By comparing the model performance of the Single Transformer mechanism, the multi-hop cross-attention mechanism, and the TCM fusion mechanism, it can be intuitively seen that the multi-hop cross-attention mechanism slightly improves the final results compared to the Single Transformer mechanism, and the TCM mechanism, which improves the multi-hop cross-attention mechanism, further has a positive impact on the model's accuracy. When using the Single Transformer module, the model relies solely on traditional modal fusion methods for reasoning. In the text modal classification (TXT) task, the model's prediction accuracy is 83.04%; in the multimodal classification (IMG) task that includes image information, the model's prediction accuracy is 80.66%. In this case, the overall average prediction accuracy is 84.84%. This shows the limitations of traditional modal fusion methods when faced with complex multimodal data. When using the multi-hop cross-attention mechanism, in text modality classification, the model prediction accuracy is 84.21%, which is 1.17% higher than the Single Transformer mechanism. In multimodal classification containing image information, the model prediction accuracy is 80.07%, which is 0.59% lower than the Single Transformer. The overall average prediction accuracy is 85.05%, an increase of 0.21%. This shows that although the multi-hop cross-attention mechanism has improved the overall effect, it has declined in the IMG classification involving multimodal fusion, indicating that the mechanism does not effectively handle the flow of image information.

[0158] However, after introducing the TCM module into the model, the experimental results showed a significant performance improvement. In the text modality classification (TXT) task, the model's prediction accuracy increased to 85.34%, an increase of 2.3 percentage points compared to the Single Transformer module and 1.13 percentage points compared to Multi-hop. More significantly, in the image modality classification (IMG) task, the model's accuracy increased to 82.35%, an increase of 1.69 percentage points compared to Single Transformer's 80.66% and 2.28 percentage points compared to Multi-hop's 80.07%. The overall average prediction accuracy of the TCM modality fusion mechanism reached 86.39%, an increase of 1.55 percentage points compared to Single Transformer's 84.84% and an increase of 1.34 percentage points compared to Multi-hop's 85.05%. This result shows that the TCM module has improved the IMG classification and overall average prediction accuracy after improving the multi-hop cross-attention mechanism, solving the problem of the multi-hop mechanism in inhibiting information flow. It fully demonstrates the effectiveness of the TCM module in multimodal data fusion and can better capture and integrate relevant information between different modalities, thereby improving the prediction performance of the model.

[0159] Through these ablation experiments, it is verified that the TCM module can effectively improve the model's ability to process multimodal data, thereby improving the overall prediction accuracy of the model.

[0160] The proposed multimodal fusion model design introduces a more efficient modal fusion method to fuse image and text modal information, providing a new solution for models to understand multimodal data. Overall, the M-TCM model achieved a prediction accuracy of 86.39% on the ScienceQA dataset using only 256M parameters, achieving an effective balance between parameter count and performance, and offering better adaptability in environments with lower computing power.

[0161] Based on the two-level cross-attention single-stage gated multimodal fusion method described in Example 1, Example 2 of the present invention proposes a two-level cross-attention single-stage gated multimodal fusion device, including:

[0162] Model building module, used to build a multimodal fusion model, including embedding layer, text encoder, image encoder, multimodal feature fusion module and decoder;

[0163] The prediction module is used to generate predicted text using the multimodal fusion model based on input text and input image. The process includes:

[0164] Pass the input text through the embedding layer and text encoder to obtain the text vector H l ; Pass the input image through the image encoder to obtain the image vector H v ;

[0165] The text vector H l and image vector H v Input modality feature fusion module, text vector H l After a single-head self-attention encoding, we get vector A l ; Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 ; Vector A v,0 and image vector H v Perform another single-head cross attention encoding to obtain the cross vector A v,l ; The text vector H l and the cross vector A v,l After concatenation, the gate weight is predicted through linear transformation and Sigmoid activation function; the text vector H l and the cross vector A v,l After proportional fusion with gating weights, a fusion vector is obtained;

[0166] Pass the fused vector through the decoder to obtain the predicted text.

[0167] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0168] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.< / pad> < / unk> < / pad>

Claims

1. A two-level cross-attention single-stage gated multimodal fusion method, characterized by: include: Build a multimodal fusion model, including an embedding layer, a text encoder, an image encoder, a multimodal feature fusion module, and a decoder; The process of the multimodal fusion model generating predicted text from input text and input image includes: Pass the input text through the embedding layer and text encoder to obtain the text vector H l ; Pass the input image through the image encoder to obtain the image vector H v ; The text vector H l and image vector H v Input modality feature fusion module, text vector H l After a single-head self-attention encoding, we get vector A l ; Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 ; Vector A v,0 and image vector H v Perform another single-head cross attention encoding to obtain the cross vector A v,l ; The text vector H l and the cross vector A v,l After splicing, After linear transformation and Sigmoid activation function, the gate vector is predicted; the text vector H l and the cross vector A v,l After proportional fusion with the gated vector, a fused vector is obtained; Pass the fused vector through the decoder to obtain the predicted text.

2. A two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: The process of reasoning using the multimodal fusion model includes: Input the question text and image into the multimodal fusion model of the reasoning stage to obtain the intermediate reasoning text; After splicing the intermediate reasoning text and the question text, the answer text is obtained by combining it with the multimodal fusion model of the image input inference stage.

3. The two-level cross-attention single-stage gated multimodal fusion method according to claim 2, characterized in that: The process of training the multimodal fusion model includes: First, the reasoning data in the ScienceQA dataset is used as a supervisory signal, and the question text and image are used as input to train the text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the reasoning stage, and output the intermediate reasoning text; then the answer in the ScienceQA dataset is used as a supervisory signal, and the intermediate reasoning text, question text, and image are used as input to train the text encoder, multimodal feature fusion module, and decoder in the multimodal fusion model in the inference stage, and output the answer text.

4. The two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: Text vector H l After a single-head self-attention encoding, we get vector A l , the formula is: A l =SHA1(H l ,H l ,H l ) Among them, SHA1 represents the single-head self-attention encoding, H l Represents the text vector, A l represents the output vector of a single-head self-attention encoding; Vector A l and image vector H v Perform a single-head cross attention encoding to obtain vector A v,0 , the formula is: A v,0 =SHA2(A l H v H v ) Among them, SHA2 represents the first single-head cross attention encoding, H v represents the image vector, A v,0 represents the output vector of the first single-head cross attention encoding; Vector A v,0 and image vector H v Perform another single-head cross attention encoding to obtain the cross vector A v,l , the formula is: A v,l =SHA3(A v,0 H v H v ) Among them, SHA3 represents the second single-head cross attention encoding, A v,l Represents the output vector of the second single-head cross attention encoding; The text vector H l and the cross vector A v,l After splicing, the gate vector is predicted through linear transformation and Sigmoid activation function. The formula is: λ=Sigmoid(Linear(H l ,l v,l )),λ∈[0,1] Where λ represents the gate vector, Linear(·) represents the linear transformation, and Sigmoid(·) represents the Sigmoid activation function; The text vector H l and the cross vector A v,l After proportional fusion with the gate vector, the fusion vector is obtained. The formula is: H out =λ×A v,k +(1-λ)×H l Among them, H out Represents the fusion vector.

5. The two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: The image encoder adopts a VIT encoder.

6. A two-level cross-attention single-stage gated multimodal fusion method according to claim 5, characterized in that: The image encoder is connected to a linear projection layer to map the image vector to the same dimensional space as the text vector.

7. The two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: Before the input text passes through the embedding layer, the word segmenter is used to split the input text into multiple subwords, and each subword is mapped to an ID.

8. The two-level cross-attention single-stage gated multimodal fusion method according to claim 7, characterized in that: The word segmenter splits the input text into multiple subwords based on the Unigram language model.

9. The two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: The text encoder adopts a T5 encoder, which includes multiple T5 blocks, each of which includes a self-attention layer and a feedforward layer connected in sequence.

10. A two-level cross-attention single-stage gated multimodal fusion method according to claim 1, characterized in that: The decoder adopts a T5 decoder, which includes multiple T5 blocks, each of which includes a self-attention layer, a cross-attention layer and a feedforward layer connected in sequence.

Images