Visual question-answering method and system cooperating with multi-modal alignment and counting problem optimization
By using self-attention and text-guided image attention mechanisms in the visual question-and-answer model for multimodal feature interaction, and combining the multimodal alignment enhancement module and counting module, the problem that existing models are difficult to capture deep semantic information in multimodal feature interaction and alignment is solved, and more accurate feature alignment and counting problems are achieved efficiently.
Patent Information
- Application Number
- CN202411773907.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-14
AI Technical Summary
Existing visual question-and-answer models are difficult to effectively capture deep semantic information in multimodal feature interaction and alignment, resulting in mismatch or matching errors between image features and text features, especially in poor performance in counting problems.
The multimodal feature interaction is performed using self-attention and text-guided image attention mechanism, combined with the multimodal alignment enhancement module and counting module, and optimize the model by comparing the loss function, so that the model can more accurately align and fuse the image and text features, and specially design the counting module to improve the processing performance of counting problems.
Improved deep interaction and alignment of multimodal features, improves the accuracy and efficiency of the model in terms of counting problems and multimodal alignment, and enhances the intrinsic connection between image and text features.
Smart Images
Figure CN119625334A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of visual question answering, and in particular to a visual question answering method and system for collaborative multimodal alignment and counting problem optimization. Background Art
[0002] Visual Question Answering (VQA) is an important research direction, which combines computer vision and computer natural language processing. Computer vision is a technology that studies how to make computers "see" images and further process images. It mainly involves tasks such as image recognition and classification to process given images. The natural language processing direction mainly processes natural language, that is, the text form of the language used by people in daily life (such as Chinese, etc.), including translation (such as Baidu Translate), information retrieval and other tasks. The visual question answering system inputs an image and an open-ended natural language question, and through a series of processing, a natural language answer is generated as output. In short, VQA is to answer based on given images and questions. The core challenge of this task is to efficiently extract, align and fuse features from images and natural languages, so that computers can understand the meaning of the question and generate accurate answers based on the image content.
[0003] In the field of VQA, existing models face multiple challenges. First, feature interaction usually adopts a unified feature processing strategy, which cannot finely filter features according to specific questions. This approach will lead to the redundancy of unnecessary information and inefficient use of computing resources, thus affecting the accuracy of VQA models. Multimodal alignment refers to the effective alignment and fusion of information from different modalities at the feature level when processing information from different data sources. Existing methods usually include simple entity correspondence and feature mapping, which can promote multimodal alignment and strengthen the intrinsic connection between image features and text features, but these methods are difficult to effectively grasp deep semantic information, resulting in problems such as mismatch between image features and text features and even matching errors. Counting problems are a subcategory of visual question answering tasks that involve extracting and understanding numerical information from images. These tasks usually include counting (for example, determining the number of specific objects in an image), measurement (estimating the size or distance of objects), and numerical comparison (comparing the number of different types of objects). In response to these challenges, the model must accurately identify elements in the image and effectively interpret numerically related queries. Summary of the invention
[0004] The purpose of the present invention is to provide a visual question answering method and system for collaborative multimodal alignment and counting problem optimization, which improves the deep interaction of multimodal features, multimodal alignment and counting problems.
[0005] In order to achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions.
[0006] The visual question answering method for collaborative multimodal alignment and counting problem optimization includes the following steps:
[0007] (1) Multimodal feature extraction: extract text features and capture the temporal relationship of text, extract image features and detect the object and position of the image;
[0008] (2) Multimodal feature interaction and alignment: self-attention and text-guided image attention are combined to achieve interaction between multimodal features. The processed image features and text features are input into the multimodal alignment enhancement module for alignment.
[0009] (3) The processed image features and the object box information extracted by Faster-RCNN are input into the counting module, the image features and object box information are subjected to attention operation, and then the linear layer is used for data integration;
[0010] (4) Multimodal feature fusion and answer prediction.
[0011] Furthermore, text feature extraction includes the steps of mapping vocabulary into vector space through GLOVE embedding and then processing these vectors using LSTM to capture the temporal relationship of the text.
[0012] Furthermore, the interaction between multimodal features includes the following steps:
[0013] First, use the self-attention mechanism to extract features;
[0014] The extracted features are input into the text-guided image attention for processing;
[0015] Perform a global average pooling operation to aggregate the spatial information of each channel and generate a one-dimensional representation graph containing only channel features;
[0016] After obtaining the global representation of each channel, a one-dimensional convolution is used to calculate a separate weight for each channel, and the calculated channel weights are recalibrated to the original feature map through multiplication.
[0017] Furthermore, the multimodal alignment enhancement module performs alignment including the following steps:
[0018] The processed image features and text features are input, and the model is optimized by calculating the contrast loss function and loss feedback, so that the model can shorten the distance between semantically related cross-modal data points and push away the distance between irrelevant data points.
[0019] Furthermore, the negative log-likelihood method is used to calculate the contrast loss. For each image x i , calculate its correlation with the text y iThe similarity of the positive samples is amplified by the exponential function exp, and the image x is calculated and amplified. i With all other text y in the batch j Based on the cosine similarity, a contrast loss function is constructed by minimizing the distance between image features and text features.
[0020] Furthermore, step 3 includes:
[0021] After the convolutional neural network extracts the image, it outputs two types of feature information, img_feat and img_bbox. img_feat represents the image feature information after feature interaction, and img_bbox represents the location information of the object in the image.
[0022] After the feature input module, firstly, the mask operation is performed on img_feat to obtain img_feat_mask, and the position embedding operation is performed on img_bbox to obtain the position weighted information;
[0023] Use img_feat as key, query, and value, and perform multiple attention operations to obtain the attention weights on img_feat;
[0024] Add the position weighted information of img_bbox and the attention weight of img_feat to get the final weight information W All , perform masking and filling operations on the weight information, multiply the final weight information by img_feat, and obtain the feature information enhanced by the weight information, called count_feat;
[0025] Perform linear layer and activation function operations on count_feat to obtain the final count_feat as the final output feature of the counting module.
[0026] Furthermore, the final output characteristics of the counting module are:
[0027]
[0028] Among them, C feat represents the final output feature of the counting module, R represents the Relu activation function, L represents the linear layer operation, S represents the softmax operation, img feat Represents feature information img_feat, and Masked_fill represents mask operation.
[0029] Further, step (4) includes:
[0030] The processed image features, text features and counting module features are flattened and linear multimodal fusion is performed to obtain the final multimodal features used for prediction. The obtained features are input into the linear layer. The output dimension of the linear layer is the number of label types. The prediction results can be obtained according to the numerical size of each dimension.
[0031] A visual question answering system for collaborative multimodal alignment and counting problem optimization, used to implement the above-mentioned visual question answering method for collaborative multimodal alignment and counting problem optimization, comprising:
[0032] Multimodal feature extraction module;
[0033] Multimodal feature interaction and alignment module;
[0034] Counting module;
[0035] Multimodal feature fusion and answer prediction module.
[0036] Furthermore, the multimodal feature extraction module includes a long short-term memory network LSTM combined with GLOVE embedding for extracting text features, a Faster-RCNN for extracting image features, and a region proposal network RPN for detecting objects and their locations in images.
[0037] The advantages of the present invention are that the method includes text-guided channel attention, a multimodal alignment enhancement module and a counting module, which improves the deep interaction, multimodal alignment and counting problems of multimodal features. Text-guided channel attention is used to improve deep feature interaction, and can select image features related to text and remove redundant information. This module can adapt to different problems and lay the foundation for subsequent multimodal alignment. In order to promote multimodal alignment, a multimodal alignment enhancement module is proposed. This component can not only measure the similarity between image and text features, but also promote further optimization of the model in terms of feature alignment. In addition, the counting module is specially designed to deal with counting problems in visual problem solving, aiming to improve the performance of the model when dealing with problems involving counting. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A framework diagram of a visual question answering system optimized for collaborative multimodal alignment and counting problems of the present invention;
[0039] Figure 2 Flowchart of the visual question answering method optimized for the collaborative multimodal alignment and counting problem of the present invention. DETAILED DESCRIPTION
[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0041] This embodiment proposes a visual question answering system that optimizes collaborative multimodal alignment and counting problems. Please refer to Figure 1 , including multimodal feature extraction module, multimodal feature interaction and alignment module, counting module, multimodal feature fusion and answer prediction module.
[0042] The multimodal feature extraction module includes a long short-term memory network (LSTM) combined with GLOVE embedding for extracting text features, a Faster-RCNN for extracting image features, and a region proposal network (RPN) for detecting objects and their locations in images.
[0043] Please refer to Figure 2 , first perform multimodal feature extraction. Combining LSTM and GLOVE for text feature extraction, usually by mapping vocabulary to vector space through GLOVE embedding, and then using LSTM to process these vectors to capture the temporal relationship of the text. The extracted text features can be used for further analysis or as input to other models. Faster-RCNN focuses on image processing, extracting image features through convolutional neural networks, and using region proposal networks (RPN) to detect objects and their locations in images. This method can achieve comprehensive understanding and processing of images and texts in the fusion task of images and texts. Figure 1 As shown in the first dotted box on the left.
[0044] Multimodal feature interaction and alignment, using self-attention (SA) and text-guided image attention (GA) to combine to achieve interaction between multimodal features. The formula is the classic attention formula, as follows:
[0045]
[0046] Among them, q, K, and V represent the query vector query, the key vector key, and the numerical vector value respectively, A represents the attention mechanism, T represents the transposed matrix, and d represents the dimension of K.
[0047] The extracted features are then input into the text-guided image attention for processing, and then a global average pooling operation is performed to aggregate the spatial information of each channel and generate a one-dimensional representation graph containing only channel features. The calculation formula is as follows:
[0048]
[0049] Where H and W represent the height and width of the feature map respectively, i,j,k represents the value of the input feature map at the i-th channel, j-th row and k-th column, y i ' represents the generated one-dimensional representation graph.
[0050] After obtaining the global representation of each channel, local cross-channel interactions are achieved by performing one-dimensional convolution on the interaction features. This step can effectively capture the local correlation between feature channels without introducing too many parameters, allowing the model to incorporate image feature information related to text features. By adopting one-dimensional convolution and applying nonlinear activation functions (such as the Sigmoid function), text-guided channel attention can calculate separate weights for each channel to capture their respective importance. Ultimately, these calculated channel weights will be recalibrated to the original feature map by multiplication in the TGCA module. The calculation formula is as follows:
[0051] Y i =X i ·σ(L(C1D k (y)))
[0052] Among them, Y i represents the final output feature map, X i represents the original input feature map, L represents the linear layer operation, C1D k (y) represents the use of 1D convolution of size k to process the interaction feature y, and σ represents the activation function.
[0053] The processed image features and text features are input into the multimodal alignment enhancement module for alignment. By calculating the contrast loss function and returning the loss to the optimization model, the model is forced to shorten the distance between semantically related cross-modal data points and push away the distance between irrelevant data points to learn the intrinsic connection between images and texts. In this embodiment, a negative log-likelihood method is used to calculate the contrast loss because cosine similarity is particularly effective in high-dimensional space, capturing the similarity in the direction of the vector rather than just the magnitude of the vector. For each image feature x i , calculate the text feature y of the relevant text i The similarity between the image and all other texts in the batch is calculated and amplified by the exponential function exp. j Based on the cosine similarity, a contrast loss function is constructed by minimizing the distance between image features and text features:
[0054]
[0055] Among them, cos(f(x i ),g(y i )) represents the image feature x i and text features y i The cosine similarity between them, N represents the number of samples in the batch, and τ represents the temperature parameter, which is used to adjust the smoothness of the similarity distribution and is set to 0.2.
[0056] The processed image features and the object box information extracted by Faster-RCNN are processed in some preparation, such as dimension transformation, and then input into the counting module. Then the image features and object box information are subjected to attention operation, and the linear layer is used for data integration. Specifically, after the convolutional neural network extracts the image, it outputs two kinds of feature information, img_feat and img_bbox. img_feat represents the image feature information after feature interaction, and img_bbox represents the location information of the object in the image.
[0057] After the feature input module, firstly, the mask operation is performed on img_feat to obtain img_feat_mask, and the position embedding operation is performed on img_bbox to obtain the position weighted information;
[0058] Use img_feat as key, query, and value, and perform multiple attention operations to obtain the attention weights on img_feat;
[0059] Add the position weighted information of img_bbox and the attention weight of img_feat to get the final weight information W All , perform masking and filling operations on the weight information, multiply the final weight information by img_feat, and obtain the feature information enhanced by the weight information, called count_feat;
[0060] Perform linear layer and activation function operations on count_feat to obtain the final count_feat as the final output feature of the counting module.
[0061] The formula is as follows;
[0062]
[0063] The processed image features, text features and features of the counting module are flattened, and then linear multimodal fusion is performed, that is, linear addition is performed to obtain the final multimodal features used for prediction. The obtained features are then input into the linear layer. The output dimension of the linear layer is the number of label types. The prediction result can be obtained according to the numerical size of each dimension.
[0064] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A visual question answering method for collaborative multimodal alignment and counting problem optimization, characterized in that: Includes steps: (1) Multimodal feature extraction: extract text features and capture the temporal relationship of text, extract image features and detect the object and position of the image; (2) Multimodal feature interaction and alignment: self-attention and text-guided image attention are combined to achieve interaction between multimodal features. The processed image features and text features are input into the multimodal alignment enhancement module for alignment. (3) The processed image features and the object box information extracted by Faster-RCNN are input into the counting module, the image features and object box information are subjected to attention operation, and then the linear layer is used for data integration; (4) Multimodal feature fusion and answer prediction.
2. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 1, characterized in that: The text feature extraction includes the steps of mapping vocabulary to vector space through GLOVE embedding and then processing these vectors using LSTM to capture the temporal relationship of the text.
3. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 1, characterized in that: The interaction between the multimodal features comprises the steps of: First, use the self-attention mechanism to extract features; The extracted features are input into the text-guided image attention for processing; Perform a global average pooling operation to aggregate the spatial information of each channel and generate a one-dimensional representation graph containing only channel features; After obtaining the global representation of each channel, a one-dimensional convolution is used to calculate a separate weight for each channel, and the calculated channel weights are recalibrated to the original feature map through multiplication.
4. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 1, characterized in that: The multimodal alignment enhancement module performs alignment including the following steps: The processed image features and text features are input, and the model is optimized by calculating the contrast loss function and loss feedback, so that the model can shorten the distance between semantically related cross-modal data points and push away the distance between irrelevant data points.
5. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 4, characterized in that: The contrast loss is calculated using the negative log-likelihood method. For each image x i , calculate its correlation with the text y i The similarity of the positive samples is amplified by the exponential function exp, and the image x is calculated and amplified. i With all other text y in the batch j Based on the cosine similarity, a contrast loss function is constructed by minimizing the distance between image features and text features.
6. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 1, characterized in that: The step 3 comprises: After the convolutional neural network extracts the image, it outputs two types of feature information, img_feat and img_bbox. img_feat represents the image feature information after feature interaction, and img_bbox represents the location information of the object in the image. After the feature input module, firstly, the mask operation is performed on img_feat to obtain img_feat_mask, and the position embedding operation is performed on img_bbox to obtain the position weighted information; Use img_feat as key, query, and value, and perform multiple attention operations to obtain the attention weights on img_feat; Add the position weighted information of img_bbox and the attention weight of img_feat to get the final weight information W All , perform masking and filling operations on the weight information, multiply the final weight information by img_feat, and obtain the feature information enhanced by the weight information, called count_feat; Perform linear layer and activation function operations on count_feat to obtain the final count_feat as the final output feature of the counting module.
7. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 6, characterized in that: The final output characteristics of the counting module are: Among them, C feat represents the final output feature of the counting module, R represents the Relu activation function, L represents the linear layer operation, S represents the softmax operation, img feat Represents feature information img_feat, and Masked_fill represents mask operation.
8. The visual question answering method for collaborative multimodal alignment and counting problem optimization according to claim 1, characterized in that: The step (4) comprises: The processed image features, text features and counting module features are flattened and linear multimodal fusion is performed to obtain the final multimodal features used for prediction. The obtained features are input into the linear layer. The output dimension of the linear layer is the number of label types. The prediction results can be obtained according to the numerical size of each dimension.
9. A visual question answering system for collaborative multimodal alignment and counting problem optimization, used to implement the visual question answering method for collaborative multimodal alignment and counting problem optimization as described in any one of claims 1 to 8, characterized in that: include: Multimodal feature extraction module; Multimodal feature interaction and alignment module; Counting module; Multimodal feature fusion and answer prediction module.
10. The visual question answering system for collaborative multimodal alignment and counting problem optimization according to claim 9, characterized in that: The multimodal feature extraction module includes a long short-term memory network LSTM combined with GLOVE embedding for extracting text features, a Faster-RCNN for extracting image features, and a regional proposal network RPN for detecting objects and their positions in images.