An answer-guided visual common sense reasoning debiasing method and system

Abstract

The application discloses a visual common sense reasoning debiasing method and system based on answer guidance, relates to the technical field of visual common sense reasoning, and comprises the following steps: obtaining pictures and original texts, pairing the pictures and the original texts, obtaining negative samples, and extracting the pictures, questions and answer texts in the negative samples; inputting the pictures and the questions into a pre-established picture encoder, outputting picture coding features, inputting the answer texts into a pre-established text encoder, outputting text coding features, and obtaining multi-modal features by processing the picture coding features and the text coding features through single-modal feature processing and inter-modal feature fusion; receiving a visual common sense reasoning data set, extracting correct candidate answers in the visual common sense reasoning data set, generating a debiased feature sample set, extracting features in the debiased feature sample set as answer features; performing attention mechanism on the multi-modal features and the answer features, calculating fusion features, calculating the probability of a candidate answer based on the fusion features, and taking a candidate answer with the highest probability as a predicted answer.

Description

Technical Field

[0001] This invention relates to the field of visual common sense reasoning technology, specifically a bias-free method and system for visual common sense reasoning based on answer guidance. Background Technology

[0002] In recent years, with the rapid development of science and technology, artificial intelligence has had a significantly greater impact on human life. At the same time, machine-human interaction has become a research hotspot, and visual commonsense reasoning is one such sub-direction. Visual commonsense reasoning refers to answering natural language questions based on images and providing further explanations. Currently, this field faces the problem of model prediction errors due to dataset bias. Specifically, models may predict based solely on the co-occurrence of a single modality with candidate answers or on the frequency of answers guided by information from the same type of modality, resulting in poor performance on test sets. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, the present invention aims to provide a method and system for bias-free visual commonsense reasoning based on answer guidance.

[0004] Firstly, the objective of this invention can be achieved through the following technical solution: a bias-removing method for visual commonsense reasoning based on answer guidance, the method comprising the following steps:

[0005] Obtain the image and the original text, pair the image and the original text to obtain negative samples, and extract the image, question and answer text from the negative samples;

[0006] The image and question are input into a pre-built image encoder, and the output is the image encoded features. The answer text is input into a pre-built text encoder, and the output is the text encoded features. The image encoded features and text encoded features are processed through unimodal feature processing and intermodal feature fusion to obtain multimodal features.

[0007] Receive a visual commonsense reasoning dataset, extract the correct candidate answers from the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features from the biased feature sample set as answer features;

[0008] The multimodal features and answer features are processed through an attention mechanism to calculate the fused features. The probability of the candidate answer is calculated based on the fused features, and the candidate answer with the highest probability is selected as the predicted answer.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the construction process of the negative sample is as follows:

[0010] The dataset is given a sample consisting of an image v, a question q, a set of 4 candidate answers A = {a1, a2, a3, a4} and an inference set R = {r1, r2, r3, r4};

[0011] The visual commonsense reasoning task consists of two sub-tasks: predicting the correct answer based on a given video and a question (referred to as Q→A), and predicting the correct reasoning based on a given video, a question, and the correct answer (referred to as QA→R).

[0012] For each image, another image v- is randomly selected from the dataset to replace the original image v in the sample set to obtain a negative sample. The negative sample contains the other image v-, the question q, the answer set A, and the reasoning set R.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of acquiring the multimodal features:

[0014] In the Q→A task, the image v, the question q, and the answer set A are input into the image encoder and the text encoder, respectively, to obtain the corresponding image features f. v Problem characteristics f q and the feature f of the answer set A Based on visual commonsense reasoning, the feature fusion method is represented as a function: m a =F(f v ,f q ,f A );

[0015] In the QA→R task, the image v, the concatenation of the question and the correct answer qa, and the inference set R are input into the image encoder and the text encoder, respectively, to obtain the corresponding image features f. v Characteristics of the answer to the question f qa and the inference set feature f R Based on visual commonsense reasoning, the feature fusion method is represented as a function: m r =F(f v ,f qa ,f R );

[0016] Negative samples are also input into the modality fusion module to obtain corresponding multimodal features. The images in the negative samples are then processed by an image encoder to obtain negative sample image features f. v- In the Q→A task, the fused features obtained from negative samples are m. a- =F(f v- ,f q ,f A In the QA→R task, the fused features obtained from negative samples are m. r- =F(f v- ,fqa ,f R ).

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of constructing the biased feature sample set:

[0018] In the Q→A task, the correct answers are randomly selected from a number of samples as the answer bias-removed feature sample set A.

[0019] In the QA→R task, the correct inferences from a number of random samples are used as the inference debiasing feature sample set R.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting features within the biased feature sample set:

[0021] The biased feature sample sets of the answers and reasoning are processed separately to obtain the corresponding biased features;

[0022] For the answer-biased feature sample set, the correct answers in the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each answer are sequentially placed into a vector to obtain the answer-biased feature D. A ;

[0023] For the inference debiased feature sample set, the correct inferences from the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each inference are sequentially placed into a vector to obtain the inference debiased feature D. R .

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the fused features by applying an attention mechanism to the multimodal features and the answer features.

[0025] In the Q→A task, the fused features are represented as follows: Where L=W1m a K = W2D A , ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters;

[0026] In the QA→R task, the fused features are represented as: Where L=W1m r K = W2D R ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the probability of candidate answers based on fusion features, and selecting the candidate answer with the highest probability as the predicted answer:

[0028] In a Q→A task, the probability of predicting the answer is: The predicted answer is a = argmax(P(A));

[0029] In the QA→R task, the prediction inference probability is: Predictive inference is r = argmax(P(R));

[0030] The fusion features corresponding to the obtained negative samples are not fused with the answer features, and the answer is predicted directly. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image.

[0031] The training loss function is the cross-entropy loss, which is expressed as Q→A as In QA→R, it is represented as Where y i The correct candidate answer is represented in the candidate set, and the prediction loss for the negative sample in Q→A is... In QA→R,

[0032] Finally, a parameter λ is introduced to balance the training loss between normal and negative samples, expressed as L = L + λL neg .

[0033] Secondly, in order to achieve the above objectives, this invention discloses a bias-reduction system for visual commonsense reasoning based on answer guidance, comprising:

[0034] The negative sample construction module is used to acquire images and original text, pair the images and original text to obtain negative samples, and extract the images, questions and answer text within the negative samples;

[0035] The feature fusion module is used to input images and questions into a pre-built image encoder and output image-encoded features. It also inputs answer text into a pre-built text encoder and outputs text-encoded features. The image-encoded features and text-encoded features are processed through single-modal feature processing and inter-modal feature fusion to obtain multimodal features.

[0036] The feature extraction module is used to receive a visual commonsense reasoning dataset, extract the correct candidate answers from the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features in the biased feature sample set as answer features.

[0037] The answer prediction module is used to calculate the fused feature by passing the multimodal features and the answer features through an attention mechanism. Based on the fused feature, the probability of the candidate answer is calculated, and the candidate answer with the highest probability is selected as the predicted answer.

[0038] In another aspect of the present invention, in order to achieve the above-mentioned objective, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs a visual commonsense reasoning bias removal method based on answer guidance as described above.

[0039] In another aspect of the present invention, in order to achieve the above-mentioned objective, a computer-readable storage medium is disclosed, wherein a computer program is stored in the computer program, and when the computer program is loaded and executed by a processor, it employs a bias-removing method for visual common sense reasoning based on answer guidance as described above.

[0040] The beneficial effects of this invention are:

[0041] This invention can effectively improve the prediction accuracy of the model and can be used in general visual common sense reasoning methods, thereby achieving the effect of bias removal. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0044] Figure 2 This is a diagram of the negative sample construction method disclosed in this invention.

[0045] Figure 3 This is a diagram illustrating the answer feature construction method disclosed in this invention.

[0046] Figure 4 This is a flowchart illustrating the biased inference prediction module disclosed in this invention.

[0047] Figure 5 This is a schematic diagram of the results of this invention on a dataset;

[0048] Figure 6 This is an ablation experiment diagram of the present invention on a dataset;

[0049] Figure 7This is a comparison chart of successful experimental sample cases of the present invention;

[0050] Figure 8 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1:

[0053] The following is a description of the relevant terms used in the embodiments of this application:

[0054] Visual commonsense reasoning (VCR) automatically answers natural language questions based on visual content and infers the evidence (rational) or explanation (explanation) of the answer. To a certain extent, VCR requires visual learning and reasoning algorithms to achieve the leap from recognition to cognition, and it is a cutting-edge and hot topic in the field of artificial intelligence.

[0055] like Figure 1 As shown, a bias-reduction method for visual commonsense reasoning based on answer guidance is proposed, which includes the following steps:

[0056] S101: Obtain the image and the original text, pair the image and the original text to obtain negative samples, and extract the image, question and answer text from the negative samples;

[0057] The process of constructing negative samples is as follows:

[0058] The dataset is given a sample consisting of an image v, a question q, a set of 4 candidate answers A = {a1, a2, a3, a4} and an inference set R = {r1, r2, r3, r4};

[0059] The visual commonsense reasoning task consists of two sub-tasks: predicting the correct answer based on a given video and a question (referred to as Q→A), and predicting the correct reasoning based on a given video, a question, and the correct answer (referred to as QA→R).

[0060] For each image, another image v- is randomly selected from the dataset to replace the original image v in the sample set to obtain a negative sample. The negative sample contains the other image v-, the question q, the answer set A, and the reasoning set R.

[0061] S102: Input the image and question into a pre-built image encoder and output the image encoding features. Input the answer text into a pre-built text encoder and output the text encoding features. Combine the image encoding features and text encoding features through single-modal feature processing and inter-modal feature fusion to obtain multimodal features.

[0062] The process of acquiring the multimodal features:

[0063] In the Q→A task, the image v, the question q, and the answer set A are input into the image encoder and the text encoder, respectively, to obtain the corresponding image features f. v Problem characteristics f q and the feature f of the answer set A Based on visual commonsense reasoning, the feature fusion method is represented as a function: m a =F(f v ,f q ,f A );

[0064] In the QA→R task, the image v, the concatenation of the question and the correct answer qa, and the inference set R are input into the image encoder and the text encoder, respectively, to obtain the corresponding image features f. v Characteristics of the answer to the question f qa and the inference set feature f R Based on visual commonsense reasoning, the feature fusion method is represented as a function: m r =F(f v ,f qa ,f R );

[0065] Negative samples are also input into the modality fusion module to obtain corresponding multimodal features. The images in the negative samples are then processed by an image encoder to obtain negative sample image features f. v- In the Q→A task, the fused features obtained from negative samples are m. a- =F(f v- ,f q ,f A In the QA→R task, the fused features obtained from negative samples are m. r- =F(f v- ,f qa ,f R ).

[0066] S103: Receive the visual commonsense reasoning dataset, extract the correct candidate answers from the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features from the biased feature sample set as answer features.

[0067] The process of constructing the biased feature sample set:

[0068] In the Q→A task, the correct answers are randomly selected from a number of samples as the answer bias-removed feature sample set A.

[0069] In the QA→R task, the correct inferences from a number of random samples are used as the inference debiasing feature sample set R.

[0070] The process of extracting features from the biased feature sample set:

[0071] The biased feature sample sets of the answers and reasoning are processed separately to obtain the corresponding biased features;

[0072] For the answer-debiased feature sample set, the correct answers in the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each answer are sequentially placed into a vector to obtain the answer-debiased feature D. A ;

[0073] For the inference debiased feature sample set, the correct inferences from the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each inference are sequentially placed into a vector to obtain the inference debiased feature D. R .

[0074] S104: The multimodal features and answer features are processed through an attention mechanism to calculate the fused features. The probability of the candidate answer is calculated based on the fused features, and the candidate answer with the highest probability is selected as the predicted answer.

[0075] The process of calculating the fused features by applying an attention mechanism to multimodal features and answer features:

[0076] In the Q→A task, the fused features are represented as follows: Where L=W1m a K = W2D A , ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters;

[0077] In the QA→R task, the fused features are represented as: Where L=W1m r K = W2D R ⊙ represents element-wise multiplication, and W1 and W2 are learnable parameters.

[0078] The process of calculating the probability of candidate answers based on fusion features and selecting the candidate answer with the highest probability as the predicted answer:

[0079] In a Q→A task, the probability of predicting the answer is: The predicted answer is a = argmax(P(A));

[0080] In the QA→R task, the prediction inference probability is: Predictive inference is r = argmax(P(R));

[0081] The fusion features corresponding to the obtained negative samples are not fused with the answer features, and the answer is predicted directly. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image.

[0082] The training loss function is the cross-entropy loss, which is expressed as Q→A as In QA→R, it is represented as Where y i The correct candidate answer is represented in the candidate set, and the prediction loss for the negative sample in Q→A is... In QA→R,

[0083] Finally, a parameter λ is introduced to balance the training loss between normal and negative samples, expressed as L = L + λL neg .

[0084] Specifically, the present invention will be further illustrated below through embodiments:

[0085] Please see Figure 2 As shown, this example provides a method for constructing negative samples, including: for each image, randomly selecting another image v- from the dataset to replace the image v in the original sample set to obtain a negative sample. The negative sample contains another image v-, a question q, an answer set A, and an inference set R.

[0086] Please see Figure 3 As shown in the example, this example presents a guided method for obtaining answer features, including: based on a visual commonsense reasoning dataset, firstly, obtaining the set of correct answers for all samples, and then extracting a portion of the correct candidate answers to construct a bias-reduced sample set. This bias-reduced sample answer set is then input into the text encoder Roberta to obtain the corresponding answer features.

[0087] Please see Figure 4 As shown in the example, this example presents a method for a bias-free inference prediction module, including: taking multimodal features and answer features as input, and obtaining fused features through an attention mechanism. Specifically, the multimodal features and answer features are passed through linear network layer 1 and linear network layer 2 respectively and multiplied. The matrix product is then multiplied element-wise with the answer features to obtain the output features. This feature is then passed through a softmax function to obtain the predicted probability of the candidate set, and the candidate with the highest probability value is returned as the predicted answer.

[0088] This invention uses examples to demonstrate its effectiveness in correcting biases in visual commonsense reasoning, such as... Figure 5 As shown, we used the Visual Commonsense Reasoning OOD dataset, which contains 11,045 samples, with 10,000 samples forming the training set and 1,045 forming the test set. For each sample, we answered corresponding natural language questions based on the video content and provided the reasoning behind the reasoning. The experimental results include three accuracy metrics: the probability of correctly predicting both the Q→A and QA→R subtasks, and the probability of correctly predicting both tasks simultaneously. The parameter λ was set to 3. The experiments demonstrate that the present invention improves the prediction accuracy of the two basic visual commonsense reasoning models, R2C and MSGT, across all three metrics.

[0089] Please see Figure 6 As shown, this invention uses examples to demonstrate the effectiveness of each component. We conducted experiments using the basic model MSGT on the Visual Commonsense Reasoning OOD dataset. We tested the experimental results of our method by adding two modules: negative samples and answer feature extraction. We found that adding these two modules improved the model's prediction accuracy across all three evaluation metrics, and achieving optimal results when both modules were added simultaneously. This demonstrates that each module in this invention plays its respective role in improving prediction performance, and that the modules mutually reinforce each other.

[0090] Please see Figure 7 As shown, the effectiveness of this invention is illustrated using successful experimental sample cases. Before adopting our method, the base model was influenced by bias and selected incorrect answers containing co-occurring words in the answer set. For example, in the sub-tasks Q→A and QA→R, the base model selected (d) and (b) as predicted answers, respectively, which did not match the correct answers. After improvement using our method, the model successfully predicted (b) and (c). Similar experimental results demonstrate the improvement effect of this invention on the base model.

[0091] Example 2: Second aspect, such as Figure 8 As shown, in order to achieve the above objectives, this invention discloses a bias-reduction system for visual commonsense reasoning based on answer guidance, comprising:

[0092] Negative sample construction module 11 is used to acquire images and original text, pair images and original text to obtain negative samples, and extract images, questions and answer texts from negative samples;

[0093] The feature fusion module 12 is used to input the image and question into a pre-established image encoder and output the image encoding features. It inputs the answer text into a pre-established text encoder and outputs the text encoding features. The image encoding features and text encoding features are processed through single-modal feature processing and inter-modal feature fusion to obtain multimodal features.

[0094] Feature extraction module 13 is used to receive the visual commonsense reasoning dataset, extract the correct candidate answers in the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features in the biased feature sample set as answer features.

[0095] The answer prediction module 14 is used to calculate the fusion feature by passing the multimodal features and the answer features through an attention mechanism, calculate the probability of the candidate answer based on the fusion feature, and select the candidate answer with the highest probability as the predicted answer.

[0096] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0097] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0098] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A bias-correction method for visual commonsense reasoning based on answer guidance, characterized in that, The method includes the following steps: Obtain the image and the original text, pair the image and the original text to obtain negative samples, and extract the image, question and answer text from the negative samples; The image and question are input into a pre-built image encoder, and the output is the image encoded features. The answer text is input into a pre-built text encoder, and the output is the text encoded features. The image encoded features and text encoded features are processed through unimodal feature processing and intermodal feature fusion to obtain multimodal features. Receive a visual commonsense reasoning dataset, extract the correct candidate answers from the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features from the biased feature sample set as answer features; Multimodal features and answer features are processed through an attention mechanism to calculate fused features. The probability of candidate answers is calculated based on the fused features, and the candidate answer with the highest probability is selected as the predicted answer. The process of calculating the probability of candidate answers based on fusion features and selecting the candidate answer with the highest probability as the predicted answer: In a Q→A task, the probability of predicting the answer is: The predicted answer is ; In the QA→R task, the prediction inference probability is: Predictive reasoning is ; The fusion features corresponding to the obtained negative samples are not fused with the answer features, and the answer is predicted directly. The final result is input into the loss function as a constraint to make the complete model focus on the correct part of the image. The training loss function is the cross-entropy loss, which is expressed as Q→A as In QA→R, it is represented as ,in The correct candidate answer is represented in the candidate set, and the prediction loss for the negative sample in Q→A is... In QA→R, it is ; final A parameter λ is introduced to balance the training loss between normal and negative samples, expressed as: .

2. The method for bias removal in visual commonsense reasoning based on answer guidance according to claim 1, characterized in that, The process of constructing the negative samples is as follows: The dataset provides a sample consisting of one image. One question A set of 4 candidate answers and inference set ; The visual commonsense reasoning task consists of two sub-tasks: predicting the correct answer based on a given video and a question (referred to as Q→A), and predicting the correct reasoning based on a given video, a question, and the correct answer (referred to as QA→R). For each image, randomly select another image from the dataset. - Replace the images in the original sample set We obtain negative samples, which contain another image. -,question Answer set and inference set .

3. The method for bias removal in visual commonsense reasoning based on answer guidance according to claim 1, characterized in that, The process of acquiring the multimodal features: In the Q→A task, respectively, the images ,question and answer set The image features are input into the image encoder and text encoder to obtain the corresponding image features. Problem characteristics and characteristics of the answer set ; Based on visual commonsense reasoning methods, the feature fusion method can be represented as a function: ; In the QA→R task, the images are respectively... The combination of questions and correct answers and inference set The image features are input into the image encoder and text encoder to obtain the corresponding image features. Characteristics of Question Answers and inference set features ; Based on visual commonsense reasoning methods, the feature fusion method can be represented as a function: ; Negative samples are also input into the modality fusion module to obtain corresponding multimodal features. The images in the negative samples are then processed by an image encoder to obtain negative sample image features. In the Q→A task, the fused features obtained from negative samples are: In the QA→R task, the fused features obtained from negative samples are: .

4. The method for bias removal in visual commonsense reasoning based on answer guidance according to claim 1, characterized in that, The process of constructing the biased feature sample set: In a Q→A task, the correct answers are randomly selected from a number of samples to form the bias-removed feature set of the answers. ; In QA→R tasks, the correct inferences from a random sample are used as the inference bias removal feature sample set. .

5. The method for bias removal in visual commonsense reasoning based on answer guidance according to claim 4, characterized in that, The process of extracting features from the biased feature sample set: The biased feature sample sets of the answers and reasoning are processed separately to obtain the corresponding biased features; For the answer-biased feature sample set, the correct answers in the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each answer are sequentially placed into a vector to obtain the answer-biased features. ; For the inference debiased feature sample set, the correct inferences from the samples are sequentially input into the pre-trained text encoder RoBerta. Then, the features corresponding to each inference are sequentially placed into a vector to obtain the inference debiased features. .

6. The method for bias removal in visual commonsense reasoning based on answer guidance according to claim 1, characterized in that, The process of calculating the fused features by applying an attention mechanism to multimodal features and answer features: In the Q→A task, the fused features are represented as follows: ,in , , This indicates element-wise multiplication. and These are learnable parameters; In the QA→R task, the fused features are represented as: ,in , , This indicates element-wise multiplication. and These are learnable parameters.

7. A bias-reduction system for visual commonsense reasoning based on answer guidance, employing the bias-reduction method for visual commonsense reasoning based on answer guidance as described in any one of claims 1 to 6, characterized in that, include: The negative sample construction module is used to acquire images and original text, pair the images and original text to obtain negative samples, and extract the images, questions and answer text within the negative samples; The feature fusion module is used to input images and questions into a pre-built image encoder and output image-encoded features. It also inputs answer text into a pre-built text encoder and outputs text-encoded features. The image-encoded features and text-encoded features are processed through single-modal feature processing and inter-modal feature fusion to obtain multimodal features. The feature extraction module is used to receive a visual commonsense reasoning dataset, extract the correct candidate answers from the visual commonsense reasoning dataset, generate a biased feature sample set, and extract the features in the biased feature sample set as answer features. The answer prediction module is used to calculate the fused feature by passing the multimodal features and the answer features through an attention mechanism. Based on the fused feature, the probability of the candidate answer is calculated, and the candidate answer with the highest probability is selected as the predicted answer.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs a bias-removing method for visual common sense reasoning based on answer guidance, as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs a bias-removing method for visual common-sense reasoning based on answer guidance, as described in any one of claims 1 to 6.

Images