A robust visual question answering model training method based on contrastive learning

Abstract

This invention presents a robust visual question-answering model training method based on contrastive learning, belonging to the interdisciplinary field of natural language processing and computer vision. In image enhancement, this invention employs a visual context perturbation-based enhancement method. It filters visual contexts with weak relevance to the question by analyzing the attention distribution of objects in the image and adds perturbations to these contexts to construct new image representations, allowing the model to learn visual context-independent image representations. For general question types, a strategy of removing interrogative auxiliary verbs is used for text enhancement; for other question types, a rewriting strategy is employed. Positive samples are constructed using these data enhancement methods, and then optimized using contrastive learning to learn an unbiased multimodal representation of the input information. This invention is applicable to fields such as artificial intelligence and natural language processing, enhancing model robustness and improving the accuracy of question answering on data with different scenarios or distributions.

Description

TECHNICAL FIELD

[0001] The present application relates to a visual question answering model training method, in particular to a robust visual question answering model training method based on contrast learning, and belongs to the technical field of cross-application of natural language processing and computer vision. BACKGROUND

[0002] The visual question answering task refers to giving a picture and a natural language question about the picture, hoping that the computer can predict the correct answer to the question. The computer not only needs to understand the natural language question, but also needs to understand the semantics of the image, and comprehensively reason the two kinds of information to predict the answer. The prior art has achieved good results in the visual question answering task.

[0003] However, with the rapid development of the visual question answering field, researchers have found that there is a phenomenon in this field: the visual question answering model tends to rely on single modal bias in the input information and rely on shortcuts to answer questions, making it difficult for the model to generalize to other data sets or real scenes, and lacking robustness to changes in data distribution.

[0004] Among the many improvement schemes for the bias problem in the visual question answering task, the common point is that these methods often only focus on the language bias between the question category information and the answer, and do not focus on the potential bias between the image information and the answer, so the problem has not been solved.

[0005] At the same time, with the development and wide application of the visual question answering field, researchers have increasingly realized that language bias is not the only bias, and more aspects need to be taken to eliminate bias to enhance the robustness of the model and reduce the dependence of the model on bias. SUMMARY

[0006] In view of the problem that the current visual question answering field tends to rely on single modal bias in the input information, the main purpose of the present application is to propose a robust visual question answering model training method based on contrast learning, using data enhancement methods on text and image two modalities to construct positive samples; then using the contrast learning method for optimization, learning the unbiased multi-modal representation of the input information for subsequent prediction, thereby enhancing the robustness of the visual question answering model, reducing the dependence of the model on bias, and improving the accuracy of the model in answering questions on different scenes or different distribution data.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] The application discloses a robust visual question answering model training method based on contrast learning, and in the aspect of image enhancement, an enhancement method based on visual context disturbance is used, visual contexts with weak relevance to a question are screened out through attention distribution of objects in an image, and disturbance is added to the visual contexts to construct new image representations, so that the model learns image representations irrelevant to visual contexts, so that it is more difficult to rely on biased information existing in the image, and the robustness of the model to image changes is increased. For questions of general interrogative sentence type, a strategy of deleting auxiliary verbs is adopted for text enhancement, so as to cut off the potential co-occurrence mode between auxiliary verbs and answers, reduce the bias dependence degree of the model on the questions, and improve the question and answer accuracy of the model on different scenes or different distributed data.

[0009] The application discloses a robust visual question answering model training method based on contrast learning, and in the aspect of image enhancement, an enhancement method based on visual context disturbance is used, visual contexts with weak relevance to a question are screened out through attention distribution of objects in an image, and disturbance is added to the visual contexts to construct new image representations, so that the model learns image representations irrelevant to visual contexts, so that it is more difficult to rely on biased information existing in the image, and the robustness of the model to image changes is increased. For questions of general interrogative sentence type, a strategy of deleting auxiliary verbs is adopted for text enhancement, so as to cut off the potential co-occurrence mode between auxiliary verbs and answers, reduce the bias dependence degree of the model on the questions, and improve the question and answer accuracy of the model on different scenes or different distributed data.

[0010] Step 1: rewriting the input question of visual question answering to obtain the enhanced question after rewriting.

[0011] Step 1.1: fine-tuning the T5 (Transfer Text-to-Text Transformer) model on multiple rewriting data sets.

[0012] Step 1.2: for general interrogative sentences, deleting the question category in the data set annotation on the basis of the original question, so as to delete the auxiliary verb corresponding to the question, and constructing a new enhanced question; for other questions, inputting the question into the fine-tuned T5 model to output the rewritten question corresponding to the question.

[0013] Step 2: inputting the question and image of visual question answering to obtain the question feature and image feature.

[0014] Step 2.1: using the GloVe word vector model to extract the text representation of the question and the rewritten question.

[0015] Firstly, the input question is segmented, the natural language form text is converted into an integer form recognizable by a computer, the maximum number of words is set and the question is truncated; then each word in the question is converted into a text representation vector Word_Embed, the first dimension of the representation vector is n, and is used to represent the number of words.

[0016] Step 2.2: inputting the text representation vector obtained in step 2.1 into the text encoder LSTM to extract the question feature.

[0017] The Word_Embed vector is transmitted into a single-layer LSTM network to obtain the question feature Y:

[0018] Y = LSTM(Word_Embed) (1)

[0019] Similarly, the same processing can be performed on the paraphrase question to obtain the paraphrase question feature Y pos .

[0020] Step 2.3: Extract image-based image features using the Faster R-CNN model.

[0021] First, the input image is subjected to target detection, and a Faster R-CNN model based on ResNet101 is used to extract image-based image features for each image.

[0022] X = Faster R-CNN(Input_Image) (2)

[0023] where Input_Image represents the input image, and X represents the image features, the first dimension of which is m, indicating the number of detected targets.

[0024] Step 3: Input the question features and image features obtained in Step 2 into the deep collaborative attention learning module to obtain the attention features after the interaction of the two modalities.

[0025] Step 3.1: The question features obtained in Step 2 are input into the self-attention unit of L layers to extract attention question features.

[0026] Step 3.2: The image features obtained in Step 2 are input into the self-attention unit and the guided attention unit of L layers to extract attention image features, where the guided attention is guided by the attention question features obtained in Step 3.1.

[0027] Step 4: Use the image enhancement method based on visual context perturbation to obtain the attention features of the enhanced image.

[0028] Step 4.1: Calculate the average value of each column vector of the attention weight matrix when performing the self-attention operation of the l-th layer in Step 3.2 as the saliency score of the current object.

[0029] Objects with higher attention weights have a strong correlation with the question and are essential for the model to correctly answer the question, and are considered key objects for the question. Objects with lower attention weights have weak or no correlation with the question and do not help the model answer the question, and are considered visual context for the question.

[0030] In the deep self-attention operation within the image modality information, the attention distribution will gradually stabilize as a result of multiple self-attention operation calculations, serving as a basis for screening visual context.

[0031] The attention weight matrix obtained by the first layer self-attention calculation wherein is a column vector, and each column represents the attention weight of the current object and m objects. In order to select the salient object or region in the image, the average value of each column vector in the attention weight matrix is calculated The average value of each object's attention weight is obtained as the salient score of the object, and the calculation process is shown in formula (3).

[0032]

[0033] wherein the salient score of the ith object is obtained by averaging the column vector corresponding to the object, represents the jth element of the column vector .

[0034] Step 4.2: Sort the salient scores of each object in step 4.1 according to the size, and mask the r objects with the smallest scores to obtain the attention features of the enhanced image.

[0035] For the r objects with the smallest salient scores, the attention features corresponding to these objects are assigned as the minimum value, so that the attention weight is 0 after the Softmax calculation, so as to mask these features. Data augmentation can be performed at the image representation level to obtain an enhanced image representation.

[0036] Step 5: Perform multi-modal fusion on the problem, image representation of step 3 and the enhanced problem, image representation of step 4 respectively, and obtain the multi-modal representation of the original sample and the positive sample respectively.

[0037] Step 5.1: Input the attention features of the two modalities in step 3 into the attention decay network and a fully connected layer, and add them to obtain the multi-modal representation of the original sample.

[0038] Step 5.2: Input the attention features of the enhanced text in step 3 and the attention features of the enhanced image in step 4 into the attention decay network and a fully connected layer, and add them to obtain the multi-modal representation of the positive sample.

[0039] Step 6: Use the contrastive learning loss function to optimize the multi-modal representation, and use the cross-entropy loss function to optimize the prediction ability of the robust visual question answering model, to obtain the trained robust visual question answering model, and realize high-robust visual question answering according to the trained robust visual question answering model.

[0040] Step 6.1: Use the InfoNCE loss function to optimize the multi-modal representation of the original sample, so that it is close to the multi-modal representation of the positive sample and far away from the multi-modal representation of other samples in the same batch.

[0041] Step 6.2: For the multi-modal representation of step 5.1, enter into a one-layer fully connected classifier to predict the answer; meanwhile, use the binary cross-entropy loss function for training.

[0042] Further comprising step 7: the robust visual question answering model trained through step 6 has stronger robustness when facing data distribution changes, reduces the bias dependence degree of the robust visual question answering model on such problems, improves the accuracy of the robust visual question answering model in different scenes or different distribution data, and improves the human-computer interaction performance.

[0043] Advantages

[0044] 1. The robust visual question answering training method based on contrast learning disclosed in the present application performs data augmentation on multiple modalities, and uses a visual context-based image enhancement method on the image modality, effectively suppressing the bias existing in image information, suitable for various scenes where bias exists in text or images, so that the model can exhibit superior performance in more scenes, improve the accuracy of the robust visual question answering model in different scenes or different distribution data, and improve the human-computer interaction performance.

[0045] 2. The robust visual question answering training method based on contrast learning disclosed in the present application is based on the contrast learning method, and uses the flexibility of the contrast learning method to combine the model with any visual question answering model without being limited to the model structure, so as to improve the robustness and performance of the model in the visual question answering task on the basis of various model structures.

[0046] 3. The robust visual question answering training method based on contrast learning disclosed in the present application uses the enhancement strategy of deleting auxiliary verbs to effectively suppress the bias dependence of the model on general interrogative sentences. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart of the robust visual question answering training method based on contrast learning disclosed in the present application;

[0048] Figure 2 The visual question answering model based on contrast learning of the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0049] The present application will be described in detail below with reference to the accompanying drawings and embodiments. At the same time, the technical problems solved by the technical scheme of the present application and the advantages are described, and it should be pointed out that the described embodiments are only intended to facilitate the understanding of the present application and do not limit the present application in any way.

[0050] If it is desired to train a visual question answering model as a life helper for visually impaired users, higher requirements are placed on the robustness and accuracy of the model. For example, as shown in FIG. 1, a robust visual question answering model training method based on contrast learning includes the following steps: Figure 1

[0051] Step 1: Rewrite the input question of visual question answering to obtain the rewritten enhanced question, so that the model can adapt to different language expression modes.

[0052] Step 1.1: Fine-tune the T5 (Transfer Text-to-Text Transformer) model on multiple rewriting datasets.

[0053] Specifically, fine-tune the T5 model on the Quora Question Pairs, Paraphrase Adversaries from WordScrambling, and Microsoft Research Paraphrase Corpus open source datasets.

[0054] Step 1.2: For general interrogative sentences, use the strategy of deleting interrogative auxiliary verbs to construct enhanced questions; for other questions, use the fine-tuned T5 model to generate rewritten questions for each question in the dataset.

[0055] For general interrogative sentences, delete the question category in the original question to delete the corresponding interrogative auxiliary verb, thereby constructing a new enhanced question; for other questions, input them into the fine-tuned T5 model to output the rewritten question corresponding to the question.

[0056] Step 2: Input the question and image of visual question answering to obtain the question feature and image feature.

[0057] Step 2.1: Use the GloVe word vector model to extract the text representation of the question and the rewritten question.

[0058] First, tokenize the input question to convert the natural language form of the text into an integer form recognizable by the computer, set the maximum number of words to 14, and truncate the question. Second, convert each word in the question into an n x 300-dimensional text representation vector Word_Embed, where n is the number of words in the question.

[0059] Step 2.2: Input the text representation obtained in step 2.1 into the text encoder LSTM to extract the question feature.

[0060] ​The Word_Embed vector is input into a single-layer LSTM network to obtain the question feature Y according to formula (1).

[0061] Similarly, the rewritten question can also be processed in the same way to obtain the rewritten question feature Y pos .

[0062] Step 2.3: Extracting image-based image features using the Faster R-CNN model;

[0063] First, the input image is subjected to target detection, and a Faster R-CNN model based on ResNet101 is used to extract target-based image features for each image as shown in formula (2).

[0064] where Input_Image represents the input image, X represents the m x 2048-dimensional image feature, and m represents the number of detected targets, which is generally set to 36.

[0065] Step 3: Input the question features obtained in step 2 and the image features into the deep collaborative attention learning module to obtain the attention features after the interaction of the two modalities.

[0066] Step 3.1: The question features obtained in step 2 are input into the self-attention unit of the L-layer cascade to extract attention question features;

[0067] On the question text side, the question feature Y is input into the L-layer cascaded multi-head self-attention unit. Assuming the number of heads is h, the l-th layer self-attention unit first maps the question feature to query value Q, key value K, and value V, which is calculated as follows:

[0068] Q = W Q ·Y l (4)

[0069] K = W K ·Y l (5)

[0070] V = W V ·Y l (6)

[0071] where W Q , W K , and W V represent the mapping matrix parameters of the query value Q, the key value K, and the value V, respectively; Y l represents the input question feature of the l-th layer.

[0072] Y l+1 = Attention(Q, K, V) = W attention V (7)

[0073]

[0074] where QK T can be seen as the dot product similarity calculation of two vectors, the normalized attention weight matrix W attention ∈R m×n , Y l+1 represents the output of the l-th layer of the problem features, and the output of the last layer of the problem features is taken as the attention problem feature Y′.

[0075] The rewritten problem can also be processed in the same way to obtain the attention rewritten problem feature Y′ pos .

[0076] Step 3.2: For the problem features obtained in step 2, the attention problem features are extracted by entering the self-attention unit of the L-layer cascade.

[0077] At the image end, the problem features X are input into the L-layer cascaded multi-head self-attention unit and multi-head guided attention unit. The number of heads is h, and the calculation method of the l-th layer self-attention unit is the same as that at the text end, and the guided attention unit is connected after the self-attention unit, and its calculation process is shown in the following formula:

[0078] Q=W Q ·X l (9)

[0079] K=W K ·Y′ (10)

[0080] V=W V ·Y′ (11)

[0081] where W Q , W K , and W V represent the mapping matrix parameters of the query value Q, the key value K, and the value V, respectively; X l represents the input image features of the l-th layer, and Y′ represents the attention problem features.

[0082] According to the calculation of formula (7) and (8) in step 3.1, the output image features of the last layer are taken as the attention image features X′.

[0083] Step 4: The attention features of the enhanced image are obtained using the image enhancement method based on visual context perturbation, as shown in Figure 2 .

[0084] Step 4.1: For the attention weight matrix when performing the l-th layer self-attention operation in step 3.2, the average value of each column vector is calculated as the saliency score of the current object.

[0085] Objects with higher attention weights have stronger relevance to the question, and are essential for the model to correctly answer the question, and are regarded as key objects of the question. Objects with lower attention weights have weaker or no relevance to the question, and will not help the model to answer the question, and are regarded as visual context of the question.

[0086] In the deep self-attention operation within the image modality information, the attention distribution gradually tends to be stable due to multiple self-attention operation calculations, and is very suitable as a basis for screening visual context.

[0087] For the attention weight matrix obtained by the self-attention calculation of the l-th layer wherein is a column vector, and each column represents the attention weight of the current object and the m objects. In order to select the salient objects or regions in the image, the average value of each column vector in the attention weight matrix is calculated to obtain the average attention weight of each object as the saliency score of the object, and the calculation process is shown in equation (3).

[0088] wherein the saliency score of the i-th object is obtained by averaging the column vector corresponding to the object, represents the j-th element of the column vector .

[0089] Step 4.2: Sort the saliency scores of each object in step 4.1 according to the size, and mask the r objects with the smallest scores to obtain the attention features of the enhanced image;

[0090] For the r objects with the smallest saliency scores, the attention features corresponding to these objects can be assigned to be the minimum value, so that the attention weight is 0 after the Softmax calculation, and thus these features are masked. This method can perform data augmentation at the image representation level to obtain an enhanced image representation X' pos .

[0091] Step 5: Perform multi-modal fusion on the question and image representation of step 3, and the enhanced question and image representation of step 4, respectively, to obtain the multi-modal representation of the original sample and the positive sample.

[0092] Step 5.1: Input the attention features of the two modalities in step 3 into the attention decay network and a fully connected layer, and add them to obtain the multi-modal representation of the original sample;

[0093] Specifically, after the deep collaborative attention learning stage, the output question feature Y' and the image feature X' have contained rich attention weight information of question words and image regions. Therefore, a multilayer perceptron (MLP) composed of two fully connected layers is used as the attention decay model, in which the first fully connected layer uses ReLU as the activation function and adds Dropout.

[0094] The Softmax function is applied to the decayed attention feature to calculate the new attention weight a, and the new attention feature is calculated based on this The calculation process is as follows:

[0095]

[0096] where X' is the input feature, a e [a1, a2,..., a m ] is the learned attention weight.

[0097] For the output attention feature and The following linear multimodal fusion function is used:

[0098]

[0099] where are two linear mapping matrices, d z represents the dimension of the fusion feature, and the layer normalization is used after fusion to stabilize the training and obtain the multimodal representation z.

[0100] Step 5.2: The attention features of the enhanced text in step 3 and the attention features of the enhanced image in step 4 are input into the attention decay network and a fully connected layer, and are added to obtain the multimodal representation of the positive sample;

[0101] Specifically, the same as the implementation method of step 5.1, the enhanced question feature Y' pos and the image feature X' pos can obtain the multimodal representation z pos as a positive sample.

[0102] Step 6: Use the contrastive learning loss function to optimize the multimodal representation and enhance its robustness to language changes and image changes, so that the model can obtain better generalization effect in real scenes and better help visually impaired users; use the cross-entropy loss function to optimize the prediction ability of the model.

[0103] Step 6.1: Use the InfoNCE loss function to optimize the multimodal representation of the original sample, so that it is close to the multimodal representation of the positive sample and far away from the multimodal representation of other samples in the same batch;

[0104] The multimodal representation of the current sample and its corresponding positive sample is M. i , These correspond to z and z obtained in the previous text, respectively. pos ,like Figure 2 As shown, the InfoNCE loss function is used for optimization to make the multimodal representation of the original sample as close as possible to the multimodal representation of the positive sample and as far away as possible from the multimodal representation of other negative samples in the same batch. The calculation process is as follows:

[0105]

[0106] Where N is the number of samples, τ is the temperature coefficient, and I [j≠i] ∈{0,1} is an indicator function, indicating that the value is 1 when j≠i, and 0 otherwise. M represents i , The similarity between two vectors is calculated as follows:

[0107]

[0108] Positive samples are constructed by introducing language and image variations, and then the multimodal representations learned by the visual question answering model are optimized through contrastive learning to enhance the robustness of the model.

[0109] Step 6.2: The multimodal representation from Step 5.1 is fed into a fully connected classifier to predict the answer; training is performed using the binary cross-entropy loss function.

[0110] Specifically, such as Figure 2 As shown, the multimodal representation z is mapped to a vector s∈R of vocabulary size via a fully connected layer FCN and a sigmoid function. G In the equation, G represents the number of candidate answers.

[0111] s = Sigmoid(FCN(z)) (16)

[0112] Finally, a binary cross-entropy loss function is used to train an N-class classifier, calculated as follows:

[0113]

[0114] where target i Let represent the target score corresponding to each answer label of the i-th sample.

[0115] Finally, the robust visual question answering model trained by the application can better serve as an assistant for visually impaired users to help them understand the world. When the user asks "Is the traffic light red now?", the model will not rashly answer "Yes" due to the existence of language bias in the training set, in which the answer to this type of question is mostly "Yes", but will actually observe the color of the signal light in the picture and give the correct answer, which undoubtedly increases the feasibility and safety of the application of this technology, and further expands its application value.

[0116] The above specific description further details the purpose, technical solutions and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for training a robust visual question answering model based on contrastive learning, characterized in that: The method comprises the following steps: Step 1: rewriting the input question of the visual question answering to obtain the rewritten enhanced question; Step 2: inputting the question and image of the visual question answering to obtain the question feature and the image feature; Step 3: inputting the question feature and the image feature obtained in step 2 into a deep collaborative attention learning module to obtain the attention feature after interaction of the two modalities; Step 4: obtaining the attention feature of the enhanced image by using a visual context perturbation-based image enhancement method; Step 5: performing multi-modal fusion on the question and image representation of step 3 and the enhanced question and image representation of step 4 respectively to obtain the multi-modal representation of the original sample and the positive sample respectively; Step 6: optimizing the multi-modal representation by using a contrastive learning loss function, optimizing the prediction ability of the robust visual question answering model by using a cross-entropy loss function, obtaining the trained robust visual question answering model, and realizing high-robust visual question answering according to the trained robust visual question answering model.

2. The robust visual question answering model training method based on contrastive learning according to claim 1, wherein: The implementation method of step 1 is as follows: Step 1: rewriting the input question of the visual question answering to obtain the rewritten enhanced question; Step 1.1: fine-tuning a T5 (Transfer Text-to-Text Transformer) model on multiple rewriting data sets; Step 1.2: for a general interrogative sentence, deleting the question category in the original question to delete the corresponding interrogative auxiliary verb, and constructing a new enhanced question; for other questions, inputting the question into the fine-tuned T5 model to output the rewritten question corresponding to the question.

3. The robust visual question answering model training method based on contrastive learning according to claim 2, characterized in that: The implementation method of step 2 is as follows: Step 2.1: extracting the text representation of the question and the rewritten question by using a GloVe word vector model; First, the input question is segmented, the natural language form text is converted into an integer form recognizable by a computer, the maximum number of words is set and the question is truncated; then, each word in the question is converted into a text representation vector The first dimension of the representation vector is , used to represent the number of words; Step 2.2: inputting the text representation vector obtained in step 2.1 into a text encoder LSTM to extract the question feature; will be described below. Vector pass into single-layer LSTM network to obtain question features : The same processing is performed on the paraphrase question to obtain paraphrase question features ; Step 2.3: extracting the target-based image feature of the image by using a Faster R-CNN model; First, target detection is performed on the input image, and a Faster R-CNN model based on ResNet101 is used to extract the target-based image feature for each image; wherein, represents an input image, represents an image feature, the first dimension of the feature being , for representing the number of detected targets.

4. The robust visual question answering model training method based on contrastive learning according to claim 3, characterized in that: The implementation method of step 3 is as follows: Step 3.1: For the question features obtained in Step 2, enter the cascade The self-attention unit of the layer extracts attention question features; Step 3.2: For the image features obtained in Step 2, enter the cascade The self-attention unit of the layer extracts attention image features with the guidance of the attention problem features obtained in Step 3.

1.

5. The robust visual question answering model training method based on contrastive learning according to claim 4, characterized in that: The implementation method of step 4 is as follows: Step 4.1: Perform the first layer self-attention operation on the output of step 3.2 The attention weight matrix in the layer self-attention operation, calculate the average value of each column vector as the saliency score of the current object; The object with higher attention weight has stronger relevance to the question, and is essential for the model to correctly answer the question, and is regarded as the key object of the question; and the object with lower attention weight has weaker or even no relevance to the question, and will not help the model to answer the question, and is regarded as the visual context of the question; In the deep self-attention operation within the image modality information, the attention distribution will gradually tend to be stable due to multiple self-attention operation calculations, and is used as a basis for screening the visual context; The first Layer self-attention calculates the attention weight matrix obtained , wherein is a column vector, and each column represents the attention weight of the current object and m objects; in order to select the salient object or region in the image, the average value of each column vector in the attention weight matrix is calculated , and the average attention weight of each object is obtained as the salient score of the object, and the calculation process is shown in the formula ​ wherein the saliency score of the th object is obtained by averaging the column vector corresponding to the object, denotes the th element of the column vector ; Step 4.2: Sort the saliency scores of each object in step 4.1 by size, and mask the object with the smallest score to obtain the attention feature of the enhanced image. Step 4.2: Sort the saliency scores of each object in step 4.1 by size, and mask the object with the smallest score to obtain the attention feature of the enhanced image. For the objects with the minimum significant score The attention features corresponding to the objects with the minimum significant score are assigned to be minimum values, so that the attention weights are 0 after Softmax calculation, thereby masking these features; data enhancement can be performed at the image representation level to obtain an enhanced image representation.

6. The robust visual question answering model training method based on contrastive learning according to claim 5, wherein: The implementation method of step 5 is as follows: Step 5.1: inputting the attention features of the two modalities in step 3 into an attention decay network and a fully connected layer, and adding them to obtain the multi-modal representation of the original sample; Step 5.2: The attention features of the enhanced text in step 3 and the attention features of the enhanced image in step 4 are input into an attention decay network and a fully connected layer, and are added to obtain the multi-modal representation of the positive sample.

7. The robust visual question answering model training method based on contrastive learning according to claim 6, wherein: The implementation method of step 6 is, Step 6.1: The multi-modal representation of the original sample is optimized using the InfoNCE loss function to be close to the multi-modal representation of the positive sample and far from the multi-modal representations of other samples in the same batch; Step 6.2: The multi-modal representation in step 5.1 is input into a fully connected classifier to predict the answer; at the same time, the binary cross-entropy loss function is used for training to obtain the trained robust visual question answering model.

Images