Visual common sense reasoning depolarization method and system based on answer guidance

Abstract

The invention discloses a visual common sense reasoning depolarization method and system based on answer guidance, and relates to the technical field of visual common sense reasoning, and the method comprises the following steps: obtaining a picture and an original text, pairing the picture and the original text to obtain a negative sample, and extracting a picture, a question and an answer text in the negative sample; inputting the picture and the question into a pre-established picture encoder, outputting to obtain picture encoding features, inputting the answer text into a pre-established text encoder, outputting to obtain text encoding features, and performing single-mode feature processing and inter-mode feature fusion on the picture encoding features and the text encoding features to obtain multi-mode features; receiving a visual common sense reasoning data set, extracting correct candidate answers in the visual common sense reasoning data set, generating a debiased feature sample set, and extracting features in the debiased feature sample set as answer features; and performing calculation on the multi-modal features and the answer features through an attention mechanism to obtain fusion features, performing calculation on the basis of the fusion features to obtain probabilities of candidate answers, and taking the candidate answer with the highest probability of the candidate answers as a prediction answer.

Description

Technical Field

[0001] The present invention relates to the technical field of visual common sense reasoning, and specifically to a method and system for debiasing visual common sense reasoning based on answer guidance. Background Art

[0002] In recent years, with the rapid development of science and technology, the impact of artificial intelligence on human life has greatly increased. At the same time, the interaction between machines and humans has also become one of the research hotspots in scientific research, and visual common sense reasoning is one of its sub-directions. Visual common sense reasoning refers to answering corresponding natural language questions based on pictures, and further explanations are required. At present, this field faces the problem of model prediction errors due to bias in data sets. Specifically, the model may only make predictions based on the co-occurrence of a single modality with the candidate answer or based on the frequency of answers guided by the same type of modal information, and perform poorly on the test set. Summary of the invention

[0003] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a visual common sense reasoning debiasing method and system based on answer guidance.

[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a visual common sense reasoning debiasing method based on answer guidance, the method comprising the following steps:

[0005] Get the image and the original text, pair the image with the original text to get the negative sample, and extract the image, question, and answer text in the negative sample;

[0006] Input the picture and question into a pre-established picture encoder, output the picture encoding features, input the answer text into a pre-established text encoder, output the text encoding features, and obtain the multi-modal features by processing the picture encoding features and the text encoding features through single-modal feature processing and inter-modal feature fusion;

[0007] receiving a visual common sense reasoning data set, extracting correct candidate answers in the visual common sense reasoning data set, generating a debiased feature sample set, and extracting features in the debiased feature sample set as answer features;

[0008] The multimodal features and answer features are processed through the attention mechanism to obtain the fused features. The probability of the candidate answers is calculated based on the fused features, and the candidate answer with the highest probability is taken as the predicted answer.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of constructing the negative sample is as follows:

[0010] The dataset is given a sample including an image v, a question q, a set of 4 candidate answers A = {a1, a2, a3, a4} and a reasoning set R = {r1, r2, r3, r4};

[0011] The visual common sense reasoning task consists of two subtasks: predicting the correct answer based on a given video and question, denoted as Q→A, and predicting the correct reasoning based on a given video, question, and correct answer, denoted as QA→R.

[0012] For each picture, another picture v- is randomly selected from the dataset to replace the picture v in the original sample set to obtain a negative sample. The negative sample contains another picture v-, question q, answer set A and reasoning set R.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of acquiring the multimodal feature:

[0014] In the Q→A task, the image v, question q, and answer set A are input into the image encoder and text encoder to obtain the corresponding image features f v , problem feature f q And the answer set feature f A ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m a =F(f v ,f q ,f A );

[0015] In the QA→R task, the image v, the concatenation of the question and the correct answer qa, and the reasoning set R are input into the image encoder and the text encoder to obtain the corresponding image features f v , question answer feature f qa And the inference set feature f R ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m r =F(f v ,f qa ,f R );

[0016] Negative samples are also input into the modal fusion module to obtain the corresponding multimodal features. The images in the negative samples are passed through the image encoder to obtain the negative sample image features f v- ; In the Q→A task, the fusion feature obtained by the negative sample is m a- =F(f v- ,f q ,f A ); In the QA→R task, the fusion feature obtained by the negative sample is m r- =F(f v- ,fqa ,f R ).

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of constructing the debiased feature sample set:

[0018] In the Q→A task, the correct answers from several samples are randomly selected as the answer debiasing feature sample set A;

[0019] In the QA→R task, the correct inferences from several samples are randomly selected as the inference debiasing feature sample set R.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting features from the debiased feature sample set:

[0021] Process the answer and reasoning debiased feature sample sets separately to obtain the corresponding debiased feature signatures;

[0022] For the answer debiasing feature sample set, the correct answers in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each answer are put into a vector in turn to obtain the answer debiasing feature feature D A ;

[0023] For the inference debiasing feature sample set, the correct inferences in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each inference are put into a vector in turn to obtain the inference debiasing feature feature D R .

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the fusion feature by using the attention mechanism to obtain the multimodal feature and the answer feature:

[0025] In the Q→A task, the fused feature is expressed as Where L = W1m a , K = W2D A , ⊙ represents element-wise multiplication, W1 and W2 are learnable parameters;

[0026] In the QA→R task, the fusion feature is expressed as Where L = W1m r , K = W2D R , ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters.

[0027] In combination with the first aspect, in some implementations of the first aspect, the method further includes: calculating the probability of the candidate answers based on the fusion features, and taking the candidate answer with the highest probability as the predicted answer:

[0028] In the Q→A task, the predicted answer probability is The predicted answer is a = argmax(P(A));

[0029] In the QA→R task, the predicted inference probability is Predictive reasoning is r = argmax(P(R));

[0030] For the fused features corresponding to the obtained negative samples, the answer is predicted directly without being fused with the answer features. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image.

[0031] The loss function of training is cross entropy loss, which is expressed as In QA→R, it is expressed as where y i represents the correct candidate answer in the candidate set, and the prediction loss corresponding to the negative sample in Q→A is In QA→R

[0032] Finally, the parameter λ is introduced to balance the training loss of normal samples and negative samples, expressed as L = L + λL neg .

[0033] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a visual common sense reasoning debiasing system based on answer guidance, comprising:

[0034] The negative sample construction module is used to obtain images and original texts, pair the images with the original texts to obtain negative samples, and extract the images, questions, and answer texts in the negative samples;

[0035] A feature fusion module is used to input the image and question into a pre-established image encoder to output image encoding features, input the answer text into a pre-established text encoder to output text encoding features, and obtain multi-modal features by processing the image encoding features and the text encoding features through single-modal feature processing and inter-modal feature fusion;

[0036] A feature extraction module is used to receive a visual common sense reasoning data set, extract the correct candidate answers in the visual common sense reasoning data set, generate a debiased feature sample set, and extract features in the debiased feature sample set as answer features;

[0037] The answer prediction module is used to calculate the fusion features by combining the multimodal features and the answer features through the attention mechanism, calculate the probability of the candidate answers based on the fusion features, and take the candidate answer with the highest probability as the predicted answer.

[0038] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the answer-guided visual common sense reasoning debiasing method as described above is adopted.

[0039] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a computer-readable storage medium is disclosed, in which a computer program is stored. When the computer program is loaded and executed by a processor, the above-mentioned answer-guided visual common sense reasoning debiasing method is adopted.

[0040] Beneficial effects of the present invention:

[0041] The present invention can effectively improve the prediction accuracy of the model and can be used on general visual common sense reasoning methods to achieve a debiasing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a schematic flow chart of the method of the present invention;

[0044] Figure 2 is a diagram of a negative sample construction method disclosed in an example of the present invention;

[0045] Figure 3 It is a diagram of the answer feature construction method disclosed in the example of the present invention;

[0046] Figure 4 It is a flow chart of the debiased reasoning prediction module disclosed in the example of the present invention;

[0047] Figure 5 It is a schematic diagram of the results of the present invention on the data set;

[0048] Figure 6 is an ablation experiment diagram of the present invention on the data set;

[0049] Figure 7It is a comparison chart of successful experimental sample cases of the present invention;

[0050] Figure 8 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1:

[0053] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0054] Visual Commonsense Reasoning (VCR) automatically answers natural language questions based on visual content and infers the evidence (Rational) or explanation (Explanation) of the answers. VCR requires visual learning and reasoning algorithms to achieve the leap from recognition (Recognition) to cognition (Cognition), and is a cutting-edge hot issue in the field of artificial intelligence.

[0055] like Figure 1 As shown, a visual common sense reasoning debiasing method based on answer guidance includes the following steps:

[0056] S101: Obtain images and original texts, pair the images with the original texts to obtain negative samples, and extract images, questions, and answer texts in the negative samples;

[0057] The construction process of negative samples is as follows:

[0058] The dataset is given a sample including an image v, a question q, a set of 4 candidate answers A = {a1, a2, a3, a4} and a reasoning set R = {r1, r2, r3, r4};

[0059] The visual common sense reasoning task consists of two subtasks: predicting the correct answer based on a given video and question, denoted as Q→A, and predicting the correct reasoning based on a given video, question, and correct answer, denoted as QA→R.

[0060] For each picture, another picture v- is randomly selected from the dataset to replace the picture v in the original sample set to obtain a negative sample. The negative sample contains another picture v-, question q, answer set A and reasoning set R.

[0061] S102: Input the picture and the question into a pre-established picture encoder, output the picture encoding features, input the answer text into a pre-established text encoder, output the text encoding features, and obtain the picture encoding features and the text encoding features through single-modal feature processing and inter-modal feature fusion to obtain multi-modal features;

[0062] The process of acquiring the multimodal features is as follows:

[0063] In the Q→A task, the image v, question q, and answer set A are input into the image encoder and text encoder to obtain the corresponding image features f v , problem feature f q And the answer set feature f A ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m a =F(f v ,f q ,f A );

[0064] In the QA→R task, the image v, the concatenation of the question and the correct answer qa, and the reasoning set R are input into the image encoder and the text encoder to obtain the corresponding image features f v , question answer feature f qa And the inference set feature f R ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m r =F(f v ,f qa ,f R );

[0065] Negative samples are also input into the modal fusion module to obtain the corresponding multimodal features. The images in the negative samples are passed through the image encoder to obtain the negative sample image features f v- ; In the Q→A task, the fusion feature obtained by the negative sample is m a- =F(f v- ,f q ,f A ); In the QA→R task, the fusion feature obtained by the negative sample is m r- =F(f v- ,f qa ,f R ).

[0066] S103: receiving a visual common sense reasoning data set, extracting correct candidate answers in the visual common sense reasoning data set, generating a debiased feature sample set, and extracting features in the debiased feature sample set as answer features;

[0067] The construction process of the debiased feature sample set is as follows:

[0068] In the Q→A task, the correct answers from several samples are randomly selected as the answer debiasing feature sample set A;

[0069] In the QA→R task, the correct inferences from several samples are randomly selected as the inference debiasing feature sample set R.

[0070] The process of extracting features from the debiased feature sample set:

[0071] Process the answer and reasoning debiased feature sample sets separately to obtain the corresponding debiased feature signatures;

[0072] For the answer debiasing feature sample set, the correct answers in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each answer are put into a vector in turn to obtain the answer debiasing feature feature D A ;

[0073] For the inference debiasing feature sample set, the correct inferences in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each inference are put into a vector in turn to obtain the inference debiasing feature feature D R .

[0074] S104: The multimodal features and the answer features are subjected to an attention mechanism to calculate a fusion feature, the probability of the candidate answers is calculated based on the fusion feature, and the candidate answer with the highest probability is taken as the predicted answer.

[0075] The process of calculating the fusion feature by using the attention mechanism to combine the multimodal features and the answer features is as follows:

[0076] In the Q→A task, the fused feature is expressed as Where L = W1m a , K = W2D A , ⊙ represents element-wise multiplication, W1 and W2 are learnable parameters;

[0077] In the QA→R task, the fusion feature is expressed as Where L = W1m r , K = W2D R , ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters.

[0078] The process of calculating the probability of candidate answers based on the fusion features and taking the candidate answer with the highest probability as the predicted answer:

[0079] In the Q→A task, the predicted answer probability is The predicted answer is a = argmax(P(A));

[0080] In the QA→R task, the predicted inference probability is Predictive reasoning is r = argmax(P(R));

[0081] For the fused features corresponding to the obtained negative samples, the answer is predicted directly without being fused with the answer features. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image.

[0082] The loss function of training is cross entropy loss, which is expressed as In QA→R, it is expressed as where y i represents the correct candidate answer in the candidate set, and the prediction loss corresponding to the negative sample in Q→A is In QA→R

[0083] Finally, the parameter λ is introduced to balance the training loss of normal samples and negative samples, expressed as L = L + λL neg .

[0084] Specifically, the present invention is further described below by way of embodiments:

[0085] See also Figure 2 As shown, this example gives a negative sample construction method, including: for each picture, randomly select another picture v- in the data set to replace the picture v in the original sample set to obtain a negative sample, and the negative sample contains another picture v-, question q, answer set A and reasoning set R.

[0086] See also Figure 3 As shown, this example provides a guided answer feature acquisition method, including: based on the visual common sense reasoning dataset, first obtaining the correct answer set of all samples, and then extracting some of the correct candidate answers to construct a debiased sample set. The debiased sample answer set is input into the text encoder RoBerta to obtain the corresponding answer feature.

[0087] See also Figure 4 As shown, this example provides a method for a debiased reasoning prediction module, including: taking multimodal features and answer features as input, and obtaining fused features through an attention mechanism. Specifically, the multimodal features and answer features are respectively passed through linear network layer 1 and linear network layer 2 and multiplied, and the matrix product result is element-wise multiplied with the answer feature to obtain the output feature. The feature is passed through a softmax function to obtain the predicted probability of the candidate set, and the candidate with the largest probability value is returned as the predicted answer.

[0088] The present invention uses examples to prove that it is effective for visual common sense reasoning debiasing, such as Figure 5 As shown. We target the visual commonsense reasoning data set, which contains a total of 11,045 samples, of which 10,000 samples constitute the training set and the other 1,045 constitute the test set. For each sample, we answer the corresponding natural language question based on the video content and give the reasoning reason. The experimental results include three accuracy indicators, namely, the probability of correctly predicting the two subtasks of Q→A and QA→R and the two tasks at the same time. Among them, the parameter λ is set to 3. Experiments have shown that the prediction accuracy of the two visual common sense reasoning basic models R2C and MSGT of the present invention has increased in all three indicators.

[0089] See also Figure 6 As shown, the present invention uses examples to prove the effectiveness of each part. We use the basic model MSGT to conduct experiments on the visual common sense reasoning ood dataset. The experimental results of our method are tested by adding two modules, negative samples and answer feature extraction. We found that after adding these two modules, the prediction accuracy of the model is improved in all three evaluation indicators, and the best effect is achieved after adding the two parts at the same time. This shows that each module designed by the present invention plays its own role in improving the prediction effect, and the modules promote each other.

[0090] See also Figure 7 As shown in the figure, the present invention uses successful experimental sample cases to compare and illustrate its effectiveness. Before adopting our method, the basic model was affected by bias and selected the wrong answer containing co-occurring words in the answer set. For example, in the two subtasks of Q→A and QA→R, the basic model selected (d) and (b) as the predicted answers, which are inconsistent with the correct answers. After being improved by our method, the model successfully predicted (b) and (c). Similar experimental results prove the improvement effect of the present invention on the basic model.

[0091] Embodiment 2: In the second aspect, as Figure 8 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a visual common sense reasoning debiasing system based on answer guidance, comprising:

[0092] A negative sample construction module 11 is used to obtain images and original texts, pair the images with the original texts to obtain negative samples, and extract images, questions, and answer texts in the negative samples;

[0093] The feature fusion module 12 is used to input the picture and the question into a pre-established picture encoder, output the picture encoding feature, input the answer text into a pre-established text encoder, output the text encoding feature, and obtain the multi-modal feature by processing the picture encoding feature and the text encoding feature through single-modal feature and inter-modal feature fusion;

[0094] A feature extraction module 13 is used to receive a visual common sense reasoning data set, extract correct candidate answers in the visual common sense reasoning data set, generate a debiased feature sample set, and extract features in the debiased feature sample set as answer features;

[0095] The answer prediction module 14 is used to calculate the fusion feature by combining the multimodal feature and the answer feature through the attention mechanism, calculate the probability of the candidate answer based on the fusion feature, and take the candidate answer with the highest probability as the predicted answer.

[0096] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0097] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0099] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A visual common sense reasoning debiasing method based on answer guidance, characterized in that: The method comprises the following steps: Get the image and the original text, pair the image with the original text to get the negative sample, and extract the image, question, and answer text in the negative sample; Input the picture and question into a pre-established picture encoder, output the picture encoding features, input the answer text into a pre-established text encoder, output the text encoding features, and obtain the multi-modal features by processing the picture encoding features and the text encoding features through single-modal feature processing and inter-modal feature fusion; receiving a visual common sense reasoning data set, extracting correct candidate answers in the visual common sense reasoning data set, generating a debiased feature sample set, and extracting features in the debiased feature sample set as answer features; The multimodal features and answer features are processed through the attention mechanism to obtain the fused features. The probability of the candidate answers is calculated based on the fused features, and the candidate answer with the highest probability is taken as the predicted answer.

2. The method for debiasing visual common sense reasoning based on answer guidance according to claim 1, characterized in that: The construction process of the negative sample is as follows: The dataset is given a sample including an image v, a question q, a set of 4 candidate answers A = {a1, a2, a3, a4} and a reasoning set R = {r1, r2, r3, r4}; The visual common sense reasoning task consists of two subtasks: predicting the correct answer based on a given video and question, denoted as Q→A, and predicting the correct reasoning based on a given video, question, and correct answer, denoted as QA→R. For each picture, another picture v- is randomly selected from the dataset to replace the picture v in the original sample set to obtain a negative sample. The negative sample contains another picture v-, question q, answer set A and reasoning set R.

3. The method for debiasing visual common sense reasoning based on answer guidance according to claim 1, characterized in that: The process of acquiring the multimodal features is as follows: In the Q→A task, the image v, question q, and answer set A are input into the image encoder and text encoder to obtain the corresponding image features f v , problem feature f q And the answer set feature f A ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m a =F(f v ,f q ,f A ); In the QA→R task, the image v, the concatenation of the question and the correct answer qa, and the reasoning set R are input into the image encoder and the text encoder to obtain the corresponding image features f v , question answer feature f qa And the inference set feature f R ; According to the visual common sense reasoning method, the feature fusion method is expressed as a function: m r =F(f v ,f qa ,f R ); Negative samples are also input into the modal fusion module to obtain the corresponding multimodal features. The images in the negative samples are passed through the image encoder to obtain the negative sample image features f v- ; In the Q→A task, the fusion feature obtained by the negative sample is m a- =F(f v- ,f q ,f A ); In the QA→R task, the fusion feature obtained by the negative sample is m r- =F(f v- ,f qa ,f R ).

4. The method for debiasing visual common sense reasoning based on answer guidance according to claim 1, characterized in that: The construction process of the debiased feature sample set is as follows: In the Q→A task, the correct answers from several samples are randomly selected as the answer debiasing feature sample set A; In the QA→R task, the correct inferences from several samples are randomly selected as the inference debiasing feature sample set R.

5. The method for debiasing visual common sense reasoning based on answer guidance according to claim 4, characterized in that: The process of extracting features from the debiased feature sample set: Process the answer and reasoning debiased feature sample sets separately to obtain the corresponding debiased feature signatures; For the answer debiasing feature sample set, the correct answers in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each answer are put into a vector in turn to obtain the answer debiasing feature feature D A ; For the inference debiasing feature sample set, the correct inferences in the sample are input into the pre-trained text encoder RoBerta in turn, and then the features corresponding to each inference are put into a vector in turn to obtain the inference debiasing feature feature D R .

6. The method for debiasing visual common sense reasoning based on answer guidance according to claim 1, characterized in that: The process of calculating the fusion feature by using the attention mechanism to combine the multimodal feature and the answer feature is as follows: In the Q→A task, the fused feature is expressed as Where L = W1m a , K = W2D A , ⊙ represents element-wise multiplication, W1 and W2 are learnable parameters; In the QA→R task, the fusion feature is expressed as Where L = W1m r , K = W2D R , ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters.

7. The method for debiasing visual common sense reasoning based on answer guidance according to claim 1, characterized in that: The process of calculating the probability of candidate answers based on the fusion features and taking the candidate answer with the highest probability as the predicted answer: In the Q→A task, the predicted answer probability is The predicted answer is a = argmax(P(A)); In the QA→R task, the predicted inference probability is Predictive reasoning is r = argmax(P(R)); For the fused features corresponding to the obtained negative samples, the answer is predicted directly without being fused with the answer features. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image. The loss function of training is cross entropy loss, which is expressed as In QA→R, it is expressed as where y i represents the correct candidate answer in the candidate set, and the prediction loss corresponding to the negative sample in Q→A is In QA→R Finally, the parameter λ is introduced to balance the training loss of normal samples and negative samples, expressed as L = L + λL neg .

8. A visual common sense reasoning debiasing system based on answer guidance, characterized in that: include: The negative sample construction module is used to obtain images and original texts, pair the images with the original texts to obtain negative samples, and extract the images, questions, and answer texts in the negative samples; A feature fusion module is used to input the image and question into a pre-established image encoder to output image encoding features, input the answer text into a pre-established text encoder to output text encoding features, and obtain multi-modal features by processing the image encoding features and the text encoding features through single-modal feature processing and inter-modal feature fusion; A feature extraction module is used to receive a visual common sense reasoning data set, extract the correct candidate answers in the visual common sense reasoning data set, generate a debiased feature sample set, and extract features in the debiased feature sample set as answer features; The answer prediction module is used to calculate the fusion features by combining the multimodal features and the answer features through the attention mechanism, calculate the probability of the candidate answers based on the fusion features, and take the candidate answer with the highest probability as the predicted answer.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, a visual common sense reasoning debiasing method based on answer guidance as described in any one of claims 1 to 7 is adopted.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by the processor, the answer-guided visual common sense reasoning debiasing method described in any one of claims 1 to 7 is adopted.

Images