Visual question and answer method based on multi-modal feature fusion and model thereof

Abstract

According to the multi-modal feature fusion-based visual question and answer method provided by the invention, the image features under different scales can be effectively extracted by generating images with different resolutions and a multi-scale information mining module, so that the image understanding precision is improved, and the image features and the text features are efficiently combined through a multi-modal feature guide fusion strategy; the method comprises the following steps of: performing deep interaction at a feature level by utilizing a self-attention and cross mechanism to generate feature representation with semantic and visual information fusion, inputting a full-connection neural network and a text into a large model by utilizing data features after multi-modal feature fusion to obtain a visual question and answer result, and training a prediction result in combination with answers in a data set. Compared with an existing method, the method has the advantages of good stability, high accuracy and the like.

Description

Technical Field

[0003] The present invention relates to the fields of natural language processing and computer vision, and in particular to a visual question answering method and model based on multimodal feature fusion. Background Art

[0004] In recent years, with the rapid development of artificial intelligence technology, the functions of intelligent robots have become increasingly powerful, covering many fields, such as image recognition, speech recognition, emotion recognition, and dialogue processing. Among them, the research on intelligent robots in visual question answering (VQA) is particularly noteworthy. Visual question answering is to accurately generate answers to questions by deeply fusing the visual information and text content of an image given an image and a question related to it. The model needs to understand the content of the image and combine the semantic information of the question to effectively infer the correct answer related to the image.

[0005] Early methods of visual question answering usually use pre-trained convolutional neural networks such as VGGNet or ResNet to extract visual features from images. These visual features can often only reflect the basic properties of objects in the image. For the text features of the question, a text encoding method based on a recurrent neural network such as LSTM or GRU is usually used to convert the question into a semantic embedding. For the feature fusion between the two modalities, simple feature fusion strategies such as concatenation, addition, or element multiplication are often used in the early days. Image features and text features are fused. Although this method is simple, it is difficult to capture the deep relationship between images and texts. The bilinear pooling methods Mutan or MLB proposed later improve the interactivity of features, making the association between image and question features richer, but it is difficult to bridge the information gap between images and texts. With the development of the attention mechanism, the model can gradually find the corresponding focus area in the image according to the question, which improves the performance of VQA and enables the model to pay more attention to the visual information related to the question. However, the above methods are mainly shallow interactions of multimodal features, and it is difficult to perform fusion representation and learning reasoning on the close relationship between images and questions.

[0006] In summary, it can be seen that the current visual model has problems such as difficulty in bridging the information gap between images and texts, and the inability of cross-modal features to interact closely, and performs poorly in actual citations. In response to the above key issues, the present invention designs a visual question answering model based on multimodal feature fusion. Summary of the invention

[0007] In order to solve the problems that the current visual question answering model has difficulty in bridging the information gap between images and texts, and that cross-model features cannot interact closely, the present invention discloses a visual question answering method based on multimodal feature fusion.

[0008] The present invention provides a visual question answering network based on multi-scale information mining and multi-modal feature guided fusion, which mainly includes a multi-scale information mining module MSIMM (Multi-scale Information Mining Module) for extracting features of an input image for subsequent network training; a multi-modal feature guided fusion module for performing deep guided fusion processing on processed image and text features, and then splicing them with text that has been vectorized by a large model and sending them into the large model to obtain the final result.

[0009] The present invention is mainly divided into the following steps:

[0010] 1. Obtain relevant visual question-answering annotated datasets and convert the label format of the datasets into the visual question-answering dataset format;

[0011] 2. Use the Qwen-7b large model to build a visual question answering network based on multimodal feature fusion;

[0012] 3. Train the constructed visual question answering model on the visual set in the prepared visual question answering visual set format, and save the trained weight file;

[0013] 4. Call the saved model weight file for the validation set in the dataset to test the performance of visual question answering.

[0014] In the above steps, the construction of the visual question answering network with multimodal feature fusion mainly includes: multi-scale information mining module, multimodal feature guided fusion module and large model connection module.

[0015] The construction of the multi-scale information mining module includes the following steps:

[0016] Step 1: Generate a medium-resolution image by bilinear interpolation from the input high-resolution image, and then generate a low-resolution image by the same method from the medium-resolution image;

[0017] Step 2: The high resolution, medium resolution and low resolution in step 1 are visually encoded through the VisualTransformer model respectively, and self-attention is performed on the low-resolution image features;

[0018] Step 3: Perform cross-attention on the medium-resolution image features after visual encoding in step 2 after flattening and the low-resolution image features after self-attention to obtain a fusion vector of low-resolution and medium-resolution image features. Then perform cross-attention on the low-resolution image features after flattening and the fusion vector of low-resolution and medium-resolution image features to finally obtain a fusion vector of image features.

[0019] The construction of the multimodal feature-guided fusion module is to modularly combine two different attention units, which is implemented in the following four steps:

[0020] Step 1: For the input image X in , through the multi-scale information mining module, the output image feature set is set to X m ×d For the input question text Y in First, the text is preprocessed to divide the words. Then the open source BERT model is used for word embedding. BERT converts each word into a context-related vector representation, and the output text feature vector Y n ×d ;

[0021] Step 2: Input the text and the initially generated query vector into the SA unit with shared weights, so that the query vector can focus on various parts of the text. This step helps the query vector extract the overall information of the text.

[0022] Step 3: Next, the query vector and image vector are introduced into the CA unit, with the query providing the K and V vectors and the image features providing the Q vector, so that the query vector not only focuses on the text features, but also interacts with the image features. This allows the query vector to consider the semantics of the text instructions when generating image features.

[0023] Step 4: Then pass the vectors output in step 1 and step 2 through FFN (feedforward neural network) respectively, and the output feature dimension is Rm×d, and the formula is:

[0024] FFN(x)=W2·f(W1·x+b1)+b2

[0025] Among them, and are the weight matrix and bias term of the first layer, and are the weight matrix and bias term of the second layer, which is the ReLU activation function.

[0026] Step 5: Get the Rm×d image information and the Rm×d text information fused in step 3 and input them repeatedly into step 1, step 2, and step 3, and repeat N times.

[0027] According to claim 1, the design of Self-Attention (SA) and Cross-Attention (CA) attention units includes:

[0028] Multi-head attention is an extension of the self-attention mechanism. It uses multiple attention heads to calculate attention in parallel, allowing the model to learn information in different subspaces. Its basic principles are as follows:

[0029] In basic self-attention, for the input sequence X, each element X i Will be mapped to the Q, K, V keys:

[0030] The formula for calculating self-attention is:

[0031]

[0032] In order to enhance the expressiveness of the model, multiple self-attention heads are introduced, and each head h is calculated independently through a different weight matrix:

[0033]

[0034] The attention of each head is calculated as follows:

[0035]

[0036] After multiple heads of attention are calculated, the outputs of all heads are concatenated together:

[0037] MultiHead(Q,K,V)=Concat(Attention1,Attention2,...,Attention h ) and then through a linear transformation matrix W O To make the final mapping:

[0038] Output=W O MultiHead(Q, K, V)

[0039] Based on the multi-head attention mechanism, a SA meta-attention unit is constructed. This unit uses only one modality X as input, and calculates Q, K and V in the Attention formula through this input. The unit implements self-attention through the multi-head attention layer to learn the relationship between paired samples in the feature set X. The features after attention processing are obtained by weighted summing the similarities of all paired instances. These feature information is then passed to a fully connected layer using the ReLU activation function and a fully connected layer for normalization to achieve spatial transformation of the features. The final output is CrossAttention (CA), which represents all x i The similarity reconstruction set between all features in the input modality X.

[0040] Similarly, we constructed a CA meta-attention unit. Different from the SA meta-attention unit, the input of the CA unit contains the feature values ​​of two modalities, namely modality X and modality Y. In this unit, the feature data of modality Y is used to guide the feature attention learning of modality X. Specifically, modality Y is used as K and V in the Attention calculation formula, while modality X is used as Q of the subject.

[0041] The CA unit focuses on learning the pairwise relationship between a single element in feature set X and all elements in feature set Y. By using modality Y to guide the learning process of modality X, the final output is Attention (CA). This output represents the input modality x. i The similarity reconstruction set formed by crossing all sample features in modality Y.

[0042] The connection to the large model module includes feature full connection and large model answer output. The output R after guiding attention learning of image feature X and question text feature Y m×d Need to be the same as the text feature Y generated in step 1 n×d First, insert " "Labels and the use of special tags" <bos>" to indicate "sequence start" and " <eoc>" to indicate "end of sequence" to handle R m×d and Y n×d To ensure the consistency of the length of each batch of data, padding symbols are added after the short feature vectors.

[0043] Get the large model to generate the description y, the goal is to make the generated description consistent with the true answer description y * The probability distribution of is as close as possible. Suppose the large model generates a word w t The conditional probability is P(w t |w ＜t , I), where w <t is the sequence of all words generated, and the cross entropy loss formula is as follows:

[0044]

[0045] Where T is the length of the true answer description. The smaller the loss value, the higher the match between the generated description and the true description.

[0046] This model proposes a multimodal feature fusion visual question answering model MF-VQA to address two problems: the current visual question answering model is prone to losing key information in the question text and cross-modal features cannot interact closely.

[0047] The multimodal feature fusion model proposed in the present invention mines image feature information through multi-scale information and uses text vectors, image vectors and query vectors to guide feature fusion through Self-Attention and Cross-Attention. After the fusion is completed, it is sent to the large model to obtain output.

[0048] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0049] 1. The query vector is used to bridge the gap between image features and text, significantly improving the accuracy of visual question answering;

[0050] 2. The self-attention and interactive attention mechanisms are used for text vectors and image vectors, so that the feature information of the two can interact well, providing more comprehensive semantic understanding and multimodal information fusion capabilities for subsequent large-scale model answers.

[0051] 3. By generating images of different resolutions (high, medium, and low resolution), the module can effectively extract image features at different scales. This enables the model to capture the details and global information of the image at multiple scales, thereby improving the accuracy of image understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0053] Figure 1 It is a schematic diagram of the model of visual question answering based on multimodal feature fusion of the present invention.

[0054] Figure 2 It is a schematic diagram of the MSIM model of the present invention.

[0055] Figure 3 It is a schematic diagram of the AGFFM attention mechanism model of the present invention.

[0056] Figure 4 It is a schematic diagram of the CA structure of the present invention.

[0057] Figure 5 It is a schematic diagram of the SA structure of the present invention. DETAILED DESCRIPTION

[0058] This paper proposes a visual question answering method based on multimodal feature fusion, and the specific implementation includes the following contents:

[0059] 1. Select the common visual question answering dataset VQA, which includes 265,016 images from the COCO dataset. Each image involves more than or equal to 3 question data, and each question contains 10 benchmark answers.

[0060] 2. Generate a medium-resolution image by bilinear interpolation for the input high-resolution image, and then generate a low-resolution image by the same method for the medium-resolution image. The high-resolution, medium-resolution and low-resolution images are visually encoded by the Visual Transformer model respectively, and self-attention is performed on the low-resolution image features; the medium-resolution image features after visual encoding are flattened and cross-attention is performed between the low-resolution image features after self-attention to obtain the low-resolution and medium-resolution image feature fusion vectors, and then the low-resolution image features are flattened and cross-attention is performed between the low-resolution and medium-resolution image feature fusion vectors to obtain the image feature fusion vector X. m×d Then, bert-base-chinese is selected as the text feature extractor for word embedding, which converts the text into a context-related vector representation. The output text feature vector Y n×d .

[0061] 3. Construct the Attention Guidance Feature Fusion Module (AGFFM).

[0062] Specifically include: Self-Attention (SA), Cross-Attention (CA) attention unit design:

[0063] Multi-head attention is an extension of the self-attention mechanism. It uses multiple attention heads to calculate attention in parallel, allowing the model to learn information in different subspaces. Its basic principles are as follows:

[0064] In basic self-attention, for the input sequence X, each element X i Will be mapped to Q, K, V keys: The formula for self-attention calculation is:

[0065]

[0066] In order to enhance the expressiveness of the model, multiple self-attention heads are introduced, and each head h is calculated independently through a different weight matrix:

[0067]

[0068] The attention of each head is calculated as follows:

[0069]

[0070] After multiple heads of attention are calculated, the outputs of all heads are concatenated together:

[0071] MultiHead(Q,K,V)=Concat(Attention1,Attention2,...,Attention h ) and then through a linear transformation matrix W O To make the final mapping:

[0072] Output=W O MultiHead(Q, K, V)

[0073] Based on the multi-head attention mechanism, a SA meta-attention unit is constructed. This unit uses only one modality X as input, and calculates Q, K, and V in the Attention formula through this input. The unit implements self-attention through the multi-head attention layer to learn the relationship between paired samples in the feature set X. The features after attention processing are obtained by weighted summing the similarities of all paired instances. These feature information is then passed to a fully connected layer using the ReLU activation function and a fully connected layer for normalization to achieve spatial transformation of the features. The final output is Attention(SA), which represents all x i The similarity reconstruction set between all features in the input modality X.

[0074] Similarly, we constructed a CA meta-attention unit. Different from the SA meta-attention unit, the input of the CA unit contains the feature values ​​of two modalities, namely modality X and modality Y. In this unit, the feature data of modality Y is used to guide the feature attention learning of modality X. Specifically, modality Y is used as K and V in the Attention calculation formula, while modality X is used as Q of the subject.

[0075] The CA unit focuses on learning the pairwise relationship between a single element in feature set X and all elements in feature set Y. By using modality Y to guide the learning process of modality X, the final output is Attention (CA). This output represents the input modality x. i The similarity reconstruction set formed by crossing all sample features in modality Y.

[0076] The construction of the guided attention network layer is a modular combination of two different attention units, which is implemented in the following three steps:

[0077] Step 2-1: Input the text and the initially generated query vector into the SA unit with shared weights, so that the query vector can focus on various parts of the text. This step helps the query vector extract the overall information of the text.

[0078] Step 2-2: Next, the query vector and image vector are introduced into the CA unit. The query provides the K and V vectors and the image features provide the Q vector, so that the query vector not only focuses on the text features, but also interacts with the image features. This allows the query vector to consider the semantics of the text instructions when generating image features.

[0079] Step 2-3: Then pass the vectors output in 2-1 and 2-2 through FFN (feedforward neural network) respectively, and the output feature dimension is R m×d , the formula is:

[0080] FFN(x)=W2·f(W1·x+b1)+b2

[0081] Among them, W1 and b1 are the weight matrix and bias term of the first layer, W2 and b2 are the weight matrix and bias term of the second layer, and f is the ReLU activation function.

[0082] 2-4: Get R which integrates the image information in 2-3 m×d and text information R m×d And repeat input to steps 2-1, 2-2, 2-3, repeat N times.

[0083] 4. Send the vector after feature fusion in step 3 to the large model.

[0084] The large model selected is Qwen-7b, of which the base model Qwen uses up to 3 trillion tokens of data for pre-training. The data mainly involves public network documents, encyclopedias, data, codes, etc. The data involves multiple languages, but mainly Chinese and English.

[0085] R is the output after guiding attention learning of image feature X and question text feature Y m×d , and the text feature Y generated in step 2 n×d To do the splicing. First, insert " "Labels and using special tags" <bos>" to indicate "sequence start" and " <eoc>" to indicate "end of sequence" to handle R m×d and Y n ×d To ensure the consistency of the length of each batch of data, padding symbols are added after the short feature vectors.

[0086] Get the large model to generate the description y, the goal is to make the generated description consistent with the true answer description y * The probability distribution of is as close as possible. Suppose the large model generates a word w t The conditional probability is P(w t |w <t , I), where w <t is the sequence of all words generated, and the cross entropy loss formula is as follows:

[0087]

[0088] Where T is the length of the true answer description. The smaller the loss value, the higher the match between the generated description and the true description.< / eoc> < / bos> < / eoc> < / bos>

Claims

1. A visual question answering method based on a multimodal feature fusion large model, characterized in that: By generating images of different resolutions, the multi-scale information mining module can effectively extract image features at different scales, thereby improving the accuracy of image understanding and guiding the fusion strategy through multimodal features, efficiently combining image and text features, and using self-attention and cross-feature mechanisms to deeply interact at the feature level to generate feature representations that fuse semantic and visual information, including the following steps: 1) For the input image Xin, the output image feature vector is obtained through the multi-scale information mining module MSIMM (Multi-scale Information Mining Module) and is set to Xm×d; for the input question text Yin, the text is first preprocessed to divide the words, and then the open source Chinese BERT model is used for word embedding, where the output text feature vector is Yn×d; 2) Use normal distribution initialization to generate a learnable feature vector Qn×d, and then send the image feature vector Xm×d and the text feature vector Yn×d in step 1 to the attention guidance feature fusion module (AGFFM), where the output fusion feature vector Rm×d; 3) Insert " " tag is used to mark the placement of the fused feature vector. After the text is segmented and vectorized by the large model, the fused feature vector Rm×d in step 2 is placed. It is sent to the large model. During the training phase, the question and answer answers in the text dataset are compared with the model output results. The large model uses the Low-Rank Adaptation of Large Language Models (LoRA) method to fine-tune and train the learnable feature vector Qn×d in step 2. The training goal is to optimize the large model parameters and learnable vectors by minimizing the loss function; 4) By constructing the completed visual question answering training data set, train the multimodal feature fusion model constructed in steps 1-3, adjust the parameters, and obtain the optimal model; 5) Input the image to be inferred and the corresponding question into the trained model to obtain the corresponding answer to the question.

2. A visual question answering method based on a multimodal feature fusion large model according to claim 1, characterized in that: In step 1, for the input image Xin, the process of sending it to the MSIMM module includes the following: 1) Generate a medium-resolution image by bilinear interpolation from the input high-resolution image, and then generate a low-resolution image by the same method from the medium-resolution image; 2) The high resolution, medium resolution and low resolution in step 1 are visually encoded through the Visual Transformer model respectively, and self-attention is performed on the low-resolution image features; 3) Perform cross-attention on the medium-resolution image features after visual encoding in step 2 after flattening and the low-resolution image features after self-attention to obtain a fusion vector of low-resolution and medium-resolution image features. Then perform cross-attention on the low-resolution image features after flattening and the fusion vector of low-resolution and medium-resolution image features to finally obtain a fusion vector of image features.

3. According to the visual question answering method based on a multimodal feature fusion large model as described in claim 1, it is characterized in that: In step 2, for the input learnable feature vector Qn×d, image feature vector Xm×d and text feature vector Yn×d, the process of feeding them into the AGFFM module with a total of N layers includes the following: 1) The input text feature vector Yn×d and the learnable feature vector Qn×d are self-attentioned separately, but the two share the weight value between the attentions, and the output vector is Mn×d; 2) Perform cross attention on the input image feature vector Xm×d and the vector Mn×d output in step 1, where the image feature vector Xm×d is linearly mapped as the Q vector, and Mn×d is linearly mapped as the K and V vectors, where the output vector is Tn×d; 3) The output Tn×d in step 2 and the output Mn×d in step 1 are respectively passed through FFN (feedforward neural network) to obtain output vectors Z1n×d and Z2n×d respectively; 4) After the above sub-steps 1 and 2, the output of the first layer is obtained; the network model structure of the other (N-1) layers is the same as that of the first layer. For the (N-2) intermediate layers and the last layer, the text feature vector Yn×d and the learnable feature vector Qn×d input in sub-step 1 are replaced with Z1n×d and Z2n×d obtained in the previous layer in the next layer respectively; after sub-steps 1-3, the output of the intermediate layer is obtained; 5) Obtain the Z1n×d output of the last layer as the final output fusion feature vector, set to Rm×d.

Images