Question and answer reasoning method and system based on cross-modal implicit local learning
Through the cross-modal implicit local learning method, the Transformer encoder and cross-attention mechanism are used to fuse the features of images and problem description text, and the problem of insufficient fusion of features in the existing technology is solved, and the model's question-and-answer reasoning ability is improved. in the fields of image question and answer and object recognition.
Patent Information
- Application Number
- CN202410832864.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-26
- Publication Date
- 2024-11-26
AI Technical Summary
The existing technology lacks fine-grained alignment capabilities when fusing problem semantic features and image features, resulting in inaccurate model answers during high-level logical reasoning, and attention methods cannot effectively integrate text features and visual features, and cannot include images in the question-and-answer process. Characteristics, lack of relationship learning ability.
Using a question-answer reasoning method based on cross-modal implicit local learning, the features of images and problem description text are obtained, the Transformer encoder is used to extract features, and a cross-modal fusion module is built, using cross-modal attention mechanism and self-attention mechanism for Feature fusion, design the overall optimization objective function to improve the model's understanding and reasoning ability.
It effectively enhances the connection between images and text, improves the model's understanding and reasoning ability of problems, solves the problem of insufficient feature fusion, and improves the model's performance in the fields of image question-and-answer and object recognition.
Smart Images

Figure CN119026684A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a question-answering reasoning method and system based on cross-modal implicit local learning. Background Art
[0002] Although joint embedding methods, attention methods and other methods have proposed feature fusion strategies, they cannot deeply fuse the features of question semantics with image features and lack fine-grained alignment capabilities, resulting in the model often failing to give correct answers when high-level logical reasoning occurs. Most joint embedding methods combine text features with image features through simple mechanisms (e.g., concatenation, element-by-element addition or element-by-element multiplication) and feed the combined features into a linear classifier or neural network. However, the simple fusion mechanism mentioned above is not enough to capture and model all feature information, which greatly limits the model effect. The attention method analyzes the attention weights of the image so that the model pays more attention to the areas in the image related to the question or the most critical words in the question, providing the rationality of answering the question and more accurately capturing visual features and text features. However, the effective fusion of text features and visual features is not achieved, resulting in the failure to incorporate image features into the reasoning process in the process of answering questions. At the same time, the attention method has almost no help for "yes / no" questions, and the relationship learning ability of the model has not been improved. At the same time, in order to obtain better image processing capabilities, CNN neural networks are often used to extract image features in the field of visual question answering, but CNN has defects such as loss of effective information in the pooling layer and general solvability. Summary of the invention
[0003] Purpose of the invention: To propose a question-answering reasoning method based on cross-modal implicit local learning, and further propose a system for implementing the above method, so as to effectively solve the above problems existing in the prior art.
[0004] First, a question-answering reasoning method based on cross-modal implicit local learning is proposed, which includes the following steps:
[0005] Acquire a training feature data set, pair the training feature data set to obtain a number of samples, each pair of samples consisting of an image, a question description text corresponding to the image, and an answer corresponding to the question;
[0006] Extracting image features of the image, masking the problem description text to obtain masked problem description text; using a Transformer encoder to perform feature extraction on the masked problem description text to obtain masked problem description text features;
[0007] Constructing a cross-modal fusion module, taking the image features and the mask problem description text features as inputs of the cross-modal fusion module, and using a mask prediction module to predict characters replaced by the mask;
[0008] The similarity between the predicted characters replaced by the mask and the real characters is calculated, and an overall optimization objective function is designed to optimize the mask prediction module.
[0009] In a further embodiment of the first aspect, the process of extracting the image features of the image specifically includes:
[0010] Let image I∈R H×W×C Divided into N = H × W / P 2 A fixed-size non-overlapping image block sequence, where P is the size of the image block, H is the image height, and W is the image width, to obtain a N image block sequence of length;
[0011] Obtain image block features through linear mapping Add position information and CLS characters to get the image block feature sequence in, Indicates the image features with CLS characters added; f1 v Indicates the features of the first image block containing position information; Represents the features of the Nth image block containing position information;
[0012] The image block feature sequence As the input of the Transformer encoder, it is output through the last hidden layer. And the global representation of the image
[0013] In a further embodiment of the first aspect, the process of masking the problem description text to obtain the masked problem description text specifically includes:
[0014] Transformer encoder is used to extract the features of the problem description text T. Given the input text T, BPE is used to tokenize the input text T, and then special characters [SOS] and [EOS] are added as the beginning and end of the text encoding respectively. As the input of the Transformer encoder, Indicates the text encoding after adding the special character [SOS]; Indicates the text encoding after adding the special character [EOS]; f1 t Represents the features of the first character in text T;
[0015] Extracted from the highest layer of the Transformer encoder at the [EOS] marker The representation of is mapped to the joint image-text embedding space through linear projection to obtain the global text representation.
[0016] In a further embodiment of the first aspect, the cross-modal fusion module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks to achieve cross-modal feature fusion;
[0017] The output of the cross-modal fusion module is used as the input of the mask prediction module, and the mask prediction module predicts the mask.
[0018] In a further embodiment of the first aspect, the process of using the image features and the masked problem description text features as inputs of the cross-modal fusion module specifically includes:
[0019] Replace some characters in the problem description text T. The characters in T will be replaced with a probability of 15% to obtain the text The replaced characters are then replaced with [MASK] with a probability of 80%, replaced with other random characters with a probability of 10%, and remain unchanged with a probability of 10%. Special characters [SOS] and [EOS] are added as the beginning and end of the text encoding respectively to obtain the text
[0020] Using BPE to analyze text Encode and get As the input of the Transformer encoder, it is output through the last hidden layer. and text Global representation of
[0021] Will As the input of the cross-modal fusion module, As the query vector, As key vector and value vector, the calculation formula is:
[0022]
[0023] In the formula, It is the feature after the fusion of image and text features, LN(·) refers to layer normalization, MCA(`) refers to multi-head cross attention, Q represents the query vector, K represents the key vector, and V represents the value vector.
[0024] In a further embodiment of the first aspect, the multi-head cross attention calculation formula is as follows:
[0025]
[0026] Where d is the length of the mask character.
[0027] In a further embodiment of the first aspect, the process of predicting the characters replaced by the mask by the mask prediction module specifically includes:
[0028] For each mask character For inference, a multi-layer perceptron is used to predict the association probability between the masked characters and the original characters. The calculation formula is as follows:
[0029]
[0030] Where M represents the set of mask characters, |V| is the size of vocabulary V, and m i is the predicted distribution, y i is the true one-hot encoding of the word, represents the predicted distribution of the jth dimension; represents the predicted distribution of the kth dimension; represents the true encoding of the j-th dimension.
[0031] In a further embodiment of the first aspect, the process of calculating the similarity between the predicted character replaced by the mask and the real character specifically includes:
[0032] Given N image-text pairs, get the image-text representation pairs of these N image-text pairs where y i,j Indicates whether the image-text pair matches; y i,j =1 indicates that the text representation pair has the same features, y i,j =0 means that the text representation pair does not match; f i v Features of the i-th image; Represents the features of the jth text;
[0033] The distribution calculation formula of image-text similarity is:
[0034] In the formula, τ is used to adjust the peak value of the similarity matrix, sim(u,v)=u T v / ||u||||v|| is the cosine similarity.
[0035] In a further embodiment of the first aspect, the process of designing an overall optimization objective function to optimize the mask prediction module specifically includes:
[0036] Calculate text-image similarity and text-to-image loss:
[0037]
[0038] In the formula, represents the jth image feature; represents k image features; f it Represents i text features;
[0039] Get the final overall optimization goal:
[0040] L=L irr +(L i2t +L t2i ) / 2.
[0041] As a second aspect of the present invention, a question-answering reasoning system based on cross-modal implicit local learning is proposed. The system includes a sample acquisition unit, a feature extraction unit, a cross-modal fusion module, and an optimization unit.
[0042] The sample acquisition unit is used to acquire a training feature data set, and pair the training feature data set to obtain a plurality of samples, each pair of samples consisting of an image, a question description text corresponding to the image, and an answer corresponding to the question;
[0043] The feature extraction unit is used to extract image features of the image, mask the problem description text to obtain masked problem description text; use a Transformer encoder to extract features of the masked problem description text to obtain masked problem description text features;
[0044] The cross-modal fusion module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks; the output of the cross-modal fusion module is used as the input of the mask prediction module. The image features and the mask problem description text features are used as the input of the cross-modal fusion module, and the mask prediction module predicts the characters replaced by the mask;
[0045] The optimization unit is used to calculate the similarity between the predicted characters replaced by the mask and the real characters, and to design an overall optimization objective function to optimize the mask prediction module.
[0046] Compared with the prior art, the present invention has the following beneficial effects: the present invention considers an implicit local relationship learning based on cross-modality, designs an implicit inter-modal and intra-modal fine-grained relationship learning module, enhances the connection between images and texts, and improves the model's ability to understand and reason about problems. It can effectively avoid the defect that the joint embedding method cannot capture and model all feature information, and solves the problem that the traditional attention method only obtains more accurate visual features and text features without effective cross-modal feature fusion. In addition, the method of the present invention uses the ViT model trained by CLIP prediction when extracting image features, which can extract image features more accurately. In summary, the present invention has good application prospects in the fields of image question answering, object recognition, and pedestrian retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1It is a flowchart of a question-answering reasoning method based on cross-modal implicit local learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.
[0049] In order to better extract image and text features, achieve fine-grained alignment, effectively fuse image and text features, and enhance the model's ability to reason about image content based on questions, the present invention provides a question-answering reasoning method based on cross-modal implicit local learning. The main technical points are as follows:
[0050] Technical Point 1: Cross-modal implicit local learning. The image features obtained in the preprocessing step and the masked question description text features are used as the input of the cross-modal fusion module to achieve fine-grained relationship learning and question-answering reasoning. This method enhances the connection between images and text, improves the model's ability to understand and reason about problems, reduces additional supervision and reasoning costs, and achieves effective implicit inter-modal and intra-modal fine-grained relationship learning.
[0051] Technical Point 2: Use the ViT model pre-trained with CLIP to obtain image features. CNN neural networks are often used in the field of visual question answering to extract image features, but CNN has defects such as loss of effective information in the pooling layer and general solvability. Using the ViT model pre-trained with CLIP can effectively avoid the above problems.
[0052] To achieve the above purpose, the technical solution adopted by the present invention is shown in Figure 1 , the details are as follows:
[0053] Step 1: Get the training feature dataset and pair the dataset. Each pair of samples consists of an image, the question corresponding to the image, and the answer corresponding to the question, forming a triple of <image, question, answer>;
[0054] Step 2: For the images and question description texts in the paired dataset described in step 1, use the ViT model pre-trained by CLIP to obtain image features, mask the question description text, and use the Transformer encoder to extract features from the question description text and the masked question description text;
[0055] Step 3: The obtained image features and mask problem description text features are used as the input of the cross-modal fusion module. This module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks to achieve cross-modal feature fusion. Furthermore, the output of the cross-modal fusion module is used as the input of the mask prediction module, which predicts the mask to achieve cross-modal implicit reasoning.
[0056] Step 4: Design an objective function for the image and question text description to improve the model's understanding ability.
[0057] As a preferred embodiment, the specific steps of extracting image features using CLIP pre-trained ViT and extracting text features of the question description using the Transformer encoder in step 2 are as follows:
[0058] Step 2.1: Image feature learning step: image I∈R H×W×C Divided into N = H × W / P 2 A fixed-size non-overlapping image block sequence, where P is the size of the image block, obtains a sequence of N image blocks of length, and obtains the image block features through linear mapping Add position information and CLS characters to get the feature sequence of the image block The image feature sequence is used as the input of the Transformer encoder and output through the last hidden layer. And the global representation of the image
[0059] Step 2.2: Learning steps of the problem description text: Use the Transformer encoder to extract the features of the problem description text T. Given the input text T, use BPE to tokenize the input text T, then add special characters [SOS][EOS] as the beginning and end of the text encoding, construct the sentence structure of [SOS], text description, [EOS], and get As the input of the Transformer encoder, extract the [EOS] marker position from the highest layer of the Transformer The representation of is mapped to the joint image-text embedding space through linear projection to obtain the global text representation.
[0060] As a preferred embodiment, the calculation process of the cross-modal fusion module based on the cross-attention mechanism and the self-attention mechanism adopted in step 3 is as follows:
[0061] Step 3.1: First, replace some characters in the problem description text T. The characters in T will be replaced with a probability of 15% to obtain the text The replaced characters are then replaced with [MASK] with a probability of 80%, replaced with other random characters with a probability of 10%, and remain unchanged with a probability of 10%. Special characters [SOS][EOS] are then added to construct the sentence structure of [SOS], text description, and [EOS] to obtain the text. Using BPE to analyze text Encode and get As the input of the Transformer encoder, it is output through the last hidden layer. (L is text) and text Global representation of In order to more effectively fuse image and mask text representation, As the input of the cross-attention mechanism in the cross-modal fusion module, As Query(Q), As Key (K), Value (V), the calculation formula is:
[0062]
[0063] in It is the feature after the fusion of image and text features. LN(·) refers to layer normalization, and MCA(·) refers to multi-head cross attention. The specific calculation formula is:
[0064]
[0065] Where d is the mask character length.
[0066] Step 3.2: Use a mask prediction module, i.e., an MLM head, to predict each masked character. For inference, a multi-layer perceptron is used to predict the association probability between the masked characters and the original characters. The calculation formula is as follows:
[0067]
[0068] Where M represents the set of mask characters, |V| is the size of vocabulary V, and m i is the predicted distribution, y i is the true one-hot encoding of the word.
[0069] As a preferred embodiment, the image and question text description objective function proposed in step 4 is as follows:
[0070] Step 4.1: Given N image-text pairs, the image-text representation pairs of these N image-text pairs can be obtained by steps 1 and 2. where y i,j Indicates whether the image-text pair matches. i,j=1 indicates that the text representation pair has the same features, y i,j =0 means that the text representation pair does not match. The distribution calculation formula of image-text similarity is:
[0071]
[0072] Where τ is the temperature hyperparameter used to adjust the peak value of the similarity matrix, sim(u,v)=u T v / ||u||||v|| is the cosine similarity.
[0073] Step 4.2: Calculate the text-image similarity and text-to-image loss as above. The calculation formula is as follows:
[0074]
[0075] Step 4.3: Get the final overall optimization goal:
[0076] L=L irr +(L i2t +L t2i ) / 2.
[0077] As a preferred embodiment, a question-answering reasoning system based on cross-modal implicit local learning is further disclosed, which includes a sample acquisition unit, a feature extraction unit, a cross-modal fusion module, and an optimization unit.
[0078] The sample acquisition unit is used to acquire a training feature data set, and pair the training feature data set to obtain a number of samples, each pair of samples consisting of an image, a question description text corresponding to the image, and an answer corresponding to the question.
[0079] The feature extraction unit is used to extract image features of the image, mask the problem description text to obtain masked problem description text; use a Transformer encoder to extract features of the masked problem description text to obtain masked problem description text features.
[0080] The cross-modal fusion module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks; the output of the cross-modal fusion module is used as the input of the mask prediction module, the image features and the mask problem description text features are used as the input of the cross-modal fusion module, and the mask prediction module predicts the characters replaced by the mask.
[0081] The optimization unit is used to calculate the similarity between the predicted characters replaced by the mask and the real characters, and to design an overall optimization objective function to optimize the mask prediction module.
[0082] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims
1. A question-answering reasoning method based on cross-modal implicit local learning, characterized in that: The steps include: Acquire a training feature data set, pair the training feature data set to obtain a number of samples, each pair of samples consisting of an image, a question description text corresponding to the image, and an answer corresponding to the question; Extracting image features of the image, masking the problem description text to obtain masked problem description text; using a Transformer encoder to perform feature extraction on the masked problem description text to obtain masked problem description text features; Constructing a cross-modal fusion module, taking the image features and the mask problem description text features as inputs of the cross-modal fusion module, and using a mask prediction module to predict characters replaced by the mask; The similarity between the predicted characters replaced by the mask and the real characters is calculated, and an overall optimization objective function is designed to optimize the mask prediction module.
2. The question-answering reasoning method based on cross-modal implicit local learning according to claim 1, characterized in that: Extracting image features of the image specifically includes: Let image I∈R H×W×C Divided into N = H × W / P 2 A fixed-size non-overlapping image block sequence, where P is the size of the image block, H is the image height, and W is the image width, to obtain a N image block sequence of length; Obtain image block features through linear mapping Add position information and CLS characters to get the image block feature sequence in, Indicates the image features with CLS characters added; f1 v Indicates the features of the first image block containing position information; Represents the features of the Nth image block containing position information; The image block feature sequence As the input of the Transformer encoder, it is output through the last hidden layer. And the global representation of the image 3. The question-answering reasoning method based on cross-modal implicit local learning according to claim 1, characterized in that: Masking the problem description text to obtain a masked problem description text specifically includes: Transformer encoder is used to extract the features of the problem description text T. Given the input text T, BPE is used to tokenize the input text T, and then special characters [SOS] and [EOS] are added as the beginning and end of the text encoding respectively. As the input of the Transformer encoder, Indicates the text encoding after adding the special character [SOS]; Indicates the text encoding after adding the special character [EOS]; f1 t Represents the features of the first character in text T; Extracted from the highest layer of the Transformer encoder at the [EOS] marker The representation of is mapped to the joint image-text embedding space through linear projection to obtain the global text representation.
4. The question-answering reasoning method based on cross-modal implicit local learning according to claim 1, characterized in that: The cross-modal fusion module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks to achieve cross-modal feature fusion; The output of the cross-modal fusion module is used as the input of the mask prediction module, and the mask prediction module predicts the mask.
5. The question-answering reasoning method based on cross-modal implicit local learning according to claim 1, characterized in that: The image features and the mask problem description text features are used as inputs of the cross-modal fusion module, specifically including: Replace some characters in the problem description text T. The characters in T will be replaced with a probability of 15% to obtain the text The replaced characters are then replaced with [MASK] with a probability of 80%, replaced with other random characters with a probability of 10%, and remain unchanged with a probability of 10%. Special characters [SOS] and [EOS] are added as the beginning and end of the text encoding respectively to obtain the text Using BPE to analyze text Encode and get As the input of the Transformer encoder, it is output through the last hidden layer. and text Global representation of Will As the input of the cross-modal fusion module, As the query vector, As key vector and value vector, the calculation formula is: In the formula, It is the feature after the fusion of image and text features, LN(·) refers to layer normalization, MCA(·) refers to multi-head cross attention, Q represents the query vector, K represents the key vector, and V represents the value vector.
6. The question-answering reasoning method based on cross-modal implicit local learning according to claim 5, characterized in that: The multi-head cross attention calculation formula is as follows: Where d is the length of the mask character.
7. The question-answering reasoning method based on cross-modal implicit local learning according to claim 1, characterized in that: The mask prediction module predicts the characters to be replaced by the mask, including: For each mask character For inference, a multi-layer perceptron is used to predict the association probability between the masked characters and the original characters. The calculation formula is as follows: Where M represents the set of mask characters, |V| is the size of vocabulary V, and m i is the predicted distribution, y i is the true one-hot encoding of the word, represents the predicted distribution of the jth dimension; represents the predicted distribution of the kth dimension; represents the true encoding of the j-th dimension.
8. The question-answering reasoning method based on cross-modal implicit local learning according to claim 7, characterized in that: Calculate the similarity between the predicted masked characters and the real characters, including: Given N image-text pairs, get the image-text representation pairs of these N image-text pairs where y i,j Indicates whether the image-text pair matches; y i,j =1 indicates that the text representation pair has the same features, y i,j =0 means that the text representation pair does not match; f i v Features of the i-th image; Represents the features of the jth text; The distribution calculation formula of image-text similarity is: Wherein, τ is used to adjust the peak value of the similarity matrix, sim(u,v)=uTv / ||u||v||, which is the cosine similarity.
9. The question-answering reasoning method based on cross-modal implicit local learning according to claim 8, characterized in that: Designing an overall optimization objective function to optimize the mask prediction module specifically includes: Calculate text-image similarity and text-to-image loss: In the formula, represents the jth image feature; represents k image features; f i t Represents i text features; Get the final overall optimization goal: L=L irr +(L i2t +L t2i ) / 2.
10. A question-answering reasoning system based on cross-modal implicit local learning, characterized in that: include: A sample acquisition unit; the sample acquisition unit is used to acquire a training feature data set, pair the training feature data set, and obtain a number of samples, each pair of samples consisting of an image, a question description text corresponding to the image, and an answer corresponding to the question; Feature extraction unit; The feature extraction unit is used to extract image features of the image, mask the problem description text to obtain masked problem description text; use a Transformer encoder to extract features of the masked problem description text to obtain masked problem description text features; A cross-modal fusion module; the cross-modal fusion module adopts a cross-attention mechanism and stacks M self-attention mechanisms and forward networks; the output of the cross-modal fusion module is used as the input of the mask prediction module. The image features and the mask problem description text features are used as the input of the cross-modal fusion module, and the mask prediction module predicts the characters replaced by the mask; Optimization unit; The optimization unit is used to calculate the similarity between the predicted characters replaced by the mask and the real characters, and to design an overall optimization objective function to optimize the mask prediction module.
Citation Information
Patent Citations
Text image cross-modal pedestrian retrieval method and system based on implicit relation reasoning alignment
CN116383671A
Multi-modal pedestrian re-identification method based on multi-level cross-modal difference harmonic
CN116682144A
Video editing method and electronic device
WO2024040865A1
Cited By
Pedestrian description generation method based on multi-model local description integration
CN119251855A
Training method of implicit problem inference model and implicit problem inference method
CN120409712A
Training method and implicit question reasoning method of implicit question reasoning model
CN120409712B
Fish individual identification method based on contrast language-image pre-training model
CN120913239A
