Visual common sense reasoning method, device, electronic device and storage medium

Abstract

The present invention provides a visual common sense reasoning method, device, electronic device and storage medium, wherein the method comprises: obtaining an image to be reasoned, and a reasoning question and multiple candidate reasoning answers corresponding to the image to be reasoned; inputting the image to be reasoned, the reasoning question and the multiple candidate reasoning answers into a visual common sense reasoning model, and obtaining a reasoning result output by the visual common sense reasoning model; wherein the visual common sense reasoning model is trained based on a sample reasoning image, and a sample reasoning question and multiple candidate sample reasoning answers corresponding to the sample reasoning image. The method, device, electronic device and storage medium provided by the present invention have powerful feature learning capabilities and understanding reasoning capabilities, and the error of the obtained reasoning result is small, thereby improving the accuracy of visual common sense reasoning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a visual common sense reasoning method, device, electronic device and storage medium. Background Art

[0002] With the rapid growth of multimodal data in social networks, many challenging tasks have been paid attention to and studied in order to effectively analyze heterogeneous modal data. Visual commonsense reasoning is one of these tasks. Given a question about an image, the visual commonsense reasoning model needs to provide not only the correct answer, but also reasonable reasons to prove the answer. In recent years, with the development of deep learning technology, visual commonsense reasoning tasks have made great progress. However, this task is still a challenging problem because it requires a comprehensive understanding of the diverse visual content in the image, semantically rich language expressions, and complex cross-modal relationships.

[0003] In the existing technology, visual common sense reasoning is performed through the overall attention mechanism, or by exploring Transformer-based models with large-scale pre-training. However, these models are prone to making predictions based on the bias of the data itself, and are unable to fully understand images and texts. In addition, the amount of calculation is huge, the error of visual common sense reasoning is large, and the accuracy is poor. Summary of the invention

[0004] The present invention provides a visual common sense reasoning method, device, electronic device and storage medium, which are used to solve the technical problems of large errors and poor accuracy in visual common sense reasoning in the prior art.

[0005] The present invention provides a visual common sense reasoning method, comprising:

[0006] Acquire an image to be inferred, as well as a reasoning question and multiple candidate reasoning answers corresponding to the image to be inferred;

[0007] Inputting the image to be inferred, the reasoning question and the multiple candidate reasoning answers into a visual common sense reasoning model to obtain a reasoning result output by the visual common sense reasoning model;

[0008] The visual common sense reasoning model is trained based on a sample reasoning image, a sample reasoning question corresponding to the sample reasoning image, and a plurality of candidate sample reasoning answers.

[0009] According to the visual common sense reasoning method provided by the present invention, the image to be reasoned, the reasoning question and the multiple candidate reasoning answers are input into a visual common sense reasoning model to obtain the reasoning result output by the visual common sense reasoning model, including:

[0010] Inputting the image to be inferred into the image coding layer of the visual common sense reasoning model to obtain image features output by the image coding layer;

[0011] Inputting the reasoning question into the text encoding layer of the visual common sense reasoning model to obtain the question features output by the text encoding layer;

[0012] The image features, the question features, and the multiple candidate reasoning answers are input into a result classification layer of the visual common sense reasoning model to obtain a reasoning result output by the result classification layer.

[0013] According to the visual common sense reasoning method provided by the present invention, the visual common sense reasoning model is trained based on the following steps:

[0014] Based on the sample inference image, determining instance-level positive and negative samples and image-level positive and negative samples;

[0015] Inferring answers based on the multiple candidate samples, determining semantic-level positive and negative samples;

[0016] Based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples, the initial model is trained to obtain the visual common sense reasoning model.

[0017] According to the visual common sense reasoning method provided by the present invention, the initial model is trained based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples to obtain the visual common sense reasoning model, including:

[0018] Inputting the sample inference image into the image coding layer of the initial model to obtain sample image features output by the image coding layer;

[0019] Inputting the instance-level positive and negative samples into the image coding layer of the initial model to obtain instance-level positive sample image features and instance-level negative sample image features output by the image coding layer;

[0020] Inputting the image-level positive and negative samples into the image coding layer of the initial model to obtain image-level positive sample image features and image-level negative sample image features output by the image coding layer;

[0021] Inputting the sample reasoning problem into the text encoding layer of the initial model to obtain the sample problem features output by the text encoding layer;

[0022] Inputting the semantic-level positive and negative samples into the text encoding layer of the initial model to obtain sample reasoning answer features output by the text encoding layer;

[0023] After fusing the sample image features, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features and image-level negative sample image features with the sample problem features respectively, a sample image fusion feature, an instance-level positive sample fusion feature, an instance-level negative sample fusion feature, an image-level positive sample fusion feature and an image-level negative sample fusion feature are obtained;

[0024] The sample image fusion features, the instance-level positive sample fusion features, the instance-level negative sample fusion features, the image-level positive sample fusion features and the image-level negative sample fusion features are respectively concatenated with the sample reasoning answer features and input into the result classification layer of the initial model. The initial model is trained for the purpose of minimizing the contrast loss between positive and negative samples and the cross entropy loss of model classification to obtain the visual common sense reasoning model.

[0025] According to the visual common sense reasoning method provided by the present invention, the loss function of the visual common sense reasoning model is determined based on the following steps:

[0026] Determine a basic classification loss function based on a cross entropy loss function, the sample image fusion feature, and the sample reasoning answer feature;

[0027] Determine an instance-level contrast loss function based on the contrast loss function, the sample image fusion feature, the instance-level positive sample fusion feature, and the instance-level negative sample fusion feature;

[0028] Determine an image-level contrast loss function based on the contrast loss function, the sample image fusion feature, the image-level positive sample fusion feature, and the image-level negative sample fusion feature;

[0029] Determining a semantic level contrast loss function based on the contrast loss function, the sample image fusion feature and the sample reasoning answer feature;

[0030] Determine an auxiliary classification loss function based on a cross entropy loss function, the instance-level positive sample image features, the instance-level negative sample image features, the image-level positive sample image features, the image-level negative sample image features, and the sample reasoning answer features;

[0031] Based on the basic classification loss function, the instance-level contrast loss function, the image-level contrast loss function, the semantic-level contrast loss function and the auxiliary classification loss function, a loss function of the visual common sense reasoning model is determined.

[0032] According to the visual common sense reasoning method provided by the present invention, determining instance-level positive and negative samples based on the sample reasoning image includes:

[0033] Based on the reasoning contribution of each object in any sample reasoning image, a set of objects that meet a first preset number are selected in descending order as instance-level positive samples corresponding to any sample reasoning image, and objects that meet the first preset number are selected in ascending order as instance-level negative samples corresponding to any sample reasoning image.

[0034] According to the visual common sense reasoning method provided by the present invention, the determining of image-level positive and negative samples based on the sample reasoning image includes:

[0035] Based on the reasoning contribution of each object in any sample reasoning image, objects satisfying a second preset number are selected in ascending order for masking, and the sample reasoning image obtained after masking is used as an image-level positive sample corresponding to the any sample reasoning image;

[0036] The image-level negative sample is determined based on other sample inference images except any one of the sample inference images.

[0037] The present invention provides a visual common sense reasoning device, comprising:

[0038] An acquisition unit, used to acquire an image to be inferred, and a reasoning question and a plurality of candidate reasoning answers corresponding to the image to be inferred;

[0039] An inference unit is used to input the image to be inferred, the inference question and the multiple candidate inference answers into a visual common sense inference model to obtain an inference result output by the visual common sense inference model; wherein the visual common sense inference model is trained based on a sample inference image, and a sample inference question and multiple candidate sample inference answers corresponding to the sample inference image.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the visual common sense reasoning method when executing the computer program.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the visual common sense reasoning method are implemented.

[0042] The visual common sense reasoning method, device, electronic device and storage medium provided by the embodiments of the present invention obtain a visual common sense reasoning model through training of sample reasoning images, sample reasoning problems corresponding to the sample reasoning images and multiple candidate sample reasoning answers. The obtained visual common sense reasoning model can process the image to be reasoned, the reasoning problem corresponding to the image to be reasoned and multiple candidate reasoning answers to obtain the reasoning result. Since the visual common sense reasoning model can comprehensively understand multimodal data such as images and texts, it can learn the correlation between the image to be reasoned, the reasoning problem and each candidate reasoning answer, and has strong feature learning ability and understanding reasoning ability. The error of the obtained reasoning result is small, which improves the accuracy of visual common sense reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of the visual common sense reasoning method provided by the present invention;

[0045] Figure 2 A schematic diagram of the structure of the visual common sense reasoning device provided by the present invention;

[0046] Figure 3 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Figure 1 A flow chart of the visual common sense reasoning method provided by the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0049] Step 110: Obtain an image to be inferred, a reasoning question corresponding to the image to be inferred, and a plurality of candidate reasoning answers.

[0050] Specifically, the image to be inferred may be a photo, painting, map, film and television screen, X-ray, electrocardiogram, etc., and its storage format may be PNG, BMP, JPEG, etc. The embodiment of the present invention does not specifically limit the type and format of the image to be inferred.

[0051] The reasoning question is an information request for the information in the image to be reasoned. The reasoning answer is the information corresponding to the content required to be answered in the reasoning question. The candidate reasoning answer is the option that may become the reasoning answer. For example, for an image, the person 1 in the image is holding a water cup and preparing to drink water. The reasoning question can be why the person 1 in the image is holding a water cup. There can be 4 candidate reasoning answers, namely, person 1 is preparing to eat, person 1 is preparing to drink water, person 1 is preparing to read, and person 1 is preparing to dance. Among them, the second candidate reasoning answer is closest to the actual situation recorded in the image.

[0052] Step 120, input the image to be inferred, the reasoning question and multiple candidate reasoning answers into the visual common sense reasoning model to obtain the reasoning result output by the visual common sense reasoning model; wherein the visual common sense reasoning model is trained based on the sample reasoning image, the sample reasoning question corresponding to the sample reasoning image and multiple candidate sample reasoning answers.

[0053] Specifically, the inference result is a selection of an inference answer from a plurality of candidate inference answers. The determination of the inference result can be achieved through a visual common sense reasoning model.

[0054] The visual common sense reasoning model can extract features from the inference images and reasoning questions to obtain image features and text features, and then learn the correlation between the image features, text features and each candidate reasoning answer, and take the candidate reasoning answer with the highest correlation as the reasoning answer.

[0055] The visual common sense reasoning model can be pre-trained, which can be obtained through the following training methods:

[0056] First, a large number of sample reasoning images, sample reasoning questions corresponding to the sample reasoning images, and multiple candidate sample reasoning answers are collected; secondly, multiple candidate sample reasoning answers are labeled according to the real reasoning answer to the sample reasoning question corresponding to each sample reasoning image, and the label of each candidate sample reasoning answer is determined; then, based on a large number of sample reasoning images, sample reasoning questions corresponding to the sample reasoning images, multiple candidate sample reasoning answers, and the label of each candidate sample reasoning answer, the initial model is trained to improve the initial model's prediction ability for reasoning results and obtain a visual common sense reasoning model.

[0057] The visual common sense reasoning method provided by the embodiment of the present invention obtains a visual common sense reasoning model through sample reasoning images, sample reasoning problems corresponding to the sample reasoning images, and multiple candidate sample reasoning answers. The obtained visual common sense reasoning model can process the image to be reasoned, as well as the reasoning problems corresponding to the image to be reasoned and multiple candidate reasoning answers to obtain the reasoning result. Since the visual common sense reasoning model can comprehensively understand multimodal data such as images and texts, it can learn the correlation between the image to be reasoned, the reasoning problem and each candidate reasoning answer, and has a strong feature learning ability and understanding reasoning ability. The error of the obtained reasoning result is small, which improves the accuracy of visual common sense reasoning.

[0058] Based on the above embodiment, step 120 includes:

[0059] Input the image to be inferred into the image encoding layer of the visual common sense reasoning model to obtain the image features output by the image encoding layer;

[0060] Input the reasoning question into the text encoding layer of the visual common sense reasoning model to obtain the question features output by the text encoding layer;

[0061] The image features, question features, and multiple candidate reasoning answers are input into the result classification layer of the visual common sense reasoning model to obtain the reasoning results output by the result classification layer.

[0062] Specifically, the visual common sense reasoning model may include three layers, namely, an image encoding layer, a text encoding layer, and a result classification layer.

[0063] The image coding layer is used to extract features of the image to be inferred, and obtain image features of the image to be inferred. For example, the image features are used to represent the information of each object in the image to be inferred, as well as the interaction information between each object.

[0064] The text encoding layer is used to extract features of the reasoning problem and obtain text features of the reasoning problem. For example, the text features are used to represent semantic information related to the object in the image to be reasoned in the reasoning problem.

[0065] The result classification layer is used to determine the inference answer from multiple candidate inference answers based on the image features and question features and output the inference result. For example, the result classification layer can output the inference answer or output each candidate inference answer as the predicted probability of the inference answer.

[0066] Based on any of the above embodiments, the visual common sense reasoning model is trained based on the following steps:

[0067] Based on the sample reasoning image, determine the instance-level positive and negative samples and the image-level positive and negative samples;

[0068] Infer answers based on multiple candidate samples and determine semantic-level positive and negative samples;

[0069] Based on sample reasoning images, sample reasoning problems, instance-level positive and negative samples, image-level positive and negative samples, and semantic-level positive and negative samples, the initial model is trained to obtain a visual common sense reasoning model.

[0070] Specifically, contrastive learning can be used to improve the feature learning ability of the visual common sense reasoning model.

[0071] We can infer images based on samples and construct instance-level positive and negative samples as well as image-level positive and negative samples.

[0072] Instance-level positive and negative samples are constructed from each object in the sample reasoning image. Among them, positive samples can be composed of objects that contribute more to the determination of the reasoning result, and negative samples can be composed of objects that contribute less to the determination of the reasoning result. By learning instance-level positive and negative samples, the ability of the visual reasoning model to learn the detailed information of objects in the image can be improved.

[0073] Image-level positive and negative samples are constructed from the sample reasoning image as a whole. Among them, the positive sample can be an image obtained after masking some objects that contribute less to the determination of the reasoning result, and the negative sample can be other sample reasoning images. By learning image-level positive and negative samples, the visual reasoning model's ability to learn the overall scene information of the image can be improved.

[0074] The answer can be inferred based on multiple candidate samples to determine semantic-level positive and negative samples.

[0075] Semantic-level positive and negative samples are constructed from the candidate sample reasoning answers. Among them, the positive sample can be the real sample reasoning answer in the candidate sample reasoning answer, and the negative sample is the reasoning answer of other candidate samples. By learning semantic-level positive and negative samples, the ability of the visual reasoning model to learn the semantic information of the candidate sample reasoning answer can be improved.

[0076] For example, the four candidate sample reasoning answers are recorded as a, b, c and d respectively. According to the naturally existing counterfactual situations, the correct option a is set as the positive sample, and the other three incorrect options are set as negative samples for comparative learning.

[0077] Based on any of the above embodiments, the initial model is trained based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples to obtain a visual common sense reasoning model, including:

[0078] Input the sample inference image into the image coding layer of the initial model to obtain the sample image features output by the image coding layer;

[0079] Input the instance-level positive and negative samples into the image coding layer of the initial model to obtain the instance-level positive sample image features and instance-level negative sample image features output by the image coding layer;

[0080] Input the image-level positive and negative samples into the image coding layer of the initial model, and obtain the image-level positive sample image features and the image-level negative sample image features output by the image coding layer;

[0081] Input the sample reasoning problem into the text encoding layer of the initial model to obtain the sample problem features output by the text encoding layer;

[0082] Input semantic-level positive and negative samples into the text encoding layer of the initial model to obtain the sample reasoning answer features output by the text encoding layer;

[0083] After fusing the sample image features, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features and image-level negative sample image features with the sample problem features respectively, the sample image fusion features, instance-level positive sample fusion features, instance-level negative sample fusion features, image-level positive sample fusion features and image-level negative sample fusion features are obtained;

[0084] The sample image fusion features, instance-level positive sample fusion features, instance-level negative sample fusion features, image-level positive sample fusion features and image-level negative sample fusion features are respectively concatenated with the sample reasoning answer features and input into the result classification layer of the initial model. The initial model is trained with the purpose of minimizing the contrast loss between positive and negative samples and the cross entropy loss of model classification to obtain a visual common sense reasoning model.

[0085] Specifically, a multi-level counterfactual contrastive learning method can be used to train a visual common sense reasoning model, that is, the visual common sense reasoning model is trained using positive and negative samples of different modal data types, and the positive and negative samples of the image data type also include two levels: instance level and image level.

[0086] The image encoding layer of the initial model can use a neural network model, such as a combination of ResNet101 and a feature fusion module.

[0087] The sample inference image is input into the image coding layer of the initial model to obtain the sample image features output by the image coding layer; the instance-level positive and negative samples are input into the image coding layer of the initial model to obtain the instance-level positive sample image features and the instance-level negative sample image features output by the image coding layer; the image-level positive and negative samples are input into the image coding layer of the initial model to obtain the image-level positive sample image features and the image-level negative sample image features output by the image coding layer.

[0088] The text encoding layer of the initial model can use a neural network model, such as a combination of BERT and a feature fusion module.

[0089] The sample reasoning question is input into the text encoding layer of the initial model to obtain the sample question features output by the text encoding layer; the semantic-level positive and negative samples are input into the text encoding layer of the initial model to obtain the sample reasoning answer features output by the text encoding layer.

[0090] For example, for the sample inference image in the training data, the sample inference question corresponding to the sample inference image and four candidate sample inference answers. The four candidate sample inference answers are recorded as a, b, c and d respectively. According to the sample inference image, the instance-level positive and negative samples and the image-level positive and negative samples are determined. According to the four candidate sample inference answers, the semantic-level positive and negative samples are determined.

[0091] Input the sample inference image into the image encoding layer to extract the sample image features Input the sample reasoning problem into the text encoding layer to extract the sample problem features Input semantic-level positive and negative samples into the text encoding layer of the initial model to obtain the sample reasoning answer feature R output by the text encoding layer t ∈{R a ,R b ,R c ,R d}, t∈{a,b,c,d}. Here, D q ,D v ,D r Represents the dimension of the feature, for example, D v =512,D q =D r =512.

[0092] The instance-level positive samples O ins+ and k instance-level negative samples Input to the image encoding layer to obtain the instance-level positive sample image feature V ins+ and instance-level negative sample image features Here i is the label of the instance-level negative sample, i∈[1,k].

[0093] The image-level positive samples I img+ and m image-level negative samples Input to the image encoding layer to obtain the image-level positive sample image feature V img+ and image-level negative sample image features Here j is the label of the image-level negative sample, j∈[1,m].

[0094] Next, the sample image feature V and the instance-level positive sample image feature Vins+ , instance-level negative sample image features Image-level positive sample image features V img+ and image-level negative sample image features After being fused with the sample problem feature Q, the sample image fusion feature is obtained Instance-level positive sample fusion features Instance-level negative sample fusion features Image-level positive sample fusion features And image-level negative sample fusion features

[0095] When fusing features, a linear cross-modal fusion method is used to fuse the sample image feature V and the sample problem feature Q to obtain the sample image fusion feature The specific implementation is as follows:

[0096]

[0097] Here W v and W q are two mapping matrices, LayerNorm is a layer normalization operation, which can stabilize the training. The acquisition of other fusion features can be carried out according to this method.

[0098] Finally, the sample image is fused with features Instance-level positive sample fusion features Instance-level negative sample fusion features Image-level positive sample fusion features And image-level negative sample fusion features Respectively with the sample reasoning answer feature R t After concatenation, the result classification layer is input to the initial model, and the initial model is trained to minimize the contrast loss between positive and negative samples and the cross entropy loss of model classification to obtain a visual common sense reasoning model. The result classification layer can be classified using the classification function f(·).

[0099] When the initial model is trained, its loss mainly comes from two aspects: one is the contrast loss between each positive and negative sample, and the other is the cross entropy loss when the model is classified. Therefore, the loss function of the visual common sense reasoning model can be determined based on the contrast loss between positive and negative samples and the cross entropy loss of model classification.

[0100] Based on any of the above embodiments, the loss function of the visual common sense reasoning model is determined based on the following steps:

[0101] Determine the basic classification loss function based on the cross entropy loss function, sample image fusion features and sample reasoning answer features;

[0102] Determine an instance-level contrast loss function based on the contrast loss function, sample image fusion features, instance-level positive sample fusion features, and instance-level negative sample fusion features;

[0103] Determine an image-level contrast loss function based on the contrast loss function, sample image fusion features, image-level positive sample fusion features, and image-level negative sample fusion features;

[0104] Determine the semantic level contrast loss function based on the contrast loss function, sample image fusion features and sample reasoning answer features;

[0105] Determine an auxiliary classification loss function based on a cross entropy loss function, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features, image-level negative sample image features, and sample reasoning answer features;

[0106] Based on the basic classification loss function, instance-level contrast loss function, image-level contrast loss function, semantic-level contrast loss function and auxiliary classification loss function, the loss function of the visual common sense reasoning model is determined.

[0107] Specifically, the cross-entropy loss is used to measure the ability of the visual common sense reasoning model to classify samples, and the contrast loss is used to measure the ability of the visual common sense reasoning model to compare and learn positive and negative samples.

[0108] According to the cross entropy loss function, sample image fusion features And sample reasoning answer feature R t , determine the basic classification loss function L base , expressed as:

[0109]

[0110] Where f(·) is the classification function, y t Is the option feature R t The real result.

[0111] The contrast loss function can adopt the contrast loss based on noise contrast estimation (InfoNCE).

[0112] According to the contrast loss function, sample image fusion feature V, instance-level positive sample fusion feature And instance-level negative sample fusion features Determine the instance-level contrast loss function L ins , expressed as:

[0113]

[0114] In the formula, the similarity measurement function Temperature parameter τ = 0.2. By optimizing the above loss, the features of positive sample pairs containing important objects are are guided to be close in feature space, and the features of negative sample pairs Therefore, the model can perceive more important objects in the original image, which is beneficial for learning fine-grained visual feature representations in visual commonsense reasoning.

[0115] According to the contrast loss function, sample image fusion features Image-level positive sample fusion features And image-level negative sample fusion features Determine the image-level contrast loss function L img , expressed as:

[0116]

[0117] In the formula, the similarity measurement function The temperature parameter τ = 0.2. By optimizing the above loss, the features of positive sample pairs with similar global scenes are Features of negative pairs with different scenes are guided to be close in feature space Therefore, the model can perceive visual information from a global perspective.

[0118] According to the contrast loss function, sample image fusion features And sample reasoning answer feature R t , determine the semantic level contrast loss function L sem , expressed as:

[0119]

[0120] In the formula, the similarity measurement function The temperature parameter τ = 0.1. By optimizing the above loss, the fusion features with similar semantics and feature R a are guided to move closer, while features with dissimilar semantics are guided to move away from each other. Therefore, semantic-level contrast loss helps the model understand high-level semantics and capture complex cross-modal relationships. a is the positive sample reasoning answer feature, {R b ,R c ,R d} is the negative sample reasoning answer feature.

[0121] In addition, according to contrastive learning, we can build a comparison between prediction results for positive and negative samples at the instance level and image level, and directly optimize the prediction results of visual common sense reasoning.

[0122] According to the cross entropy loss function, instance-level positive sample image features, instance-level negative sample image features and sample reasoning answer features, the cross entropy classification loss corresponding to the instance-level positive sample can be obtained: Cross entropy classification loss corresponding to instance-level negative samples The formula is:

[0123]

[0124]

[0125] According to the cross entropy loss function, image-level positive sample image features, image-level negative sample image features and sample reasoning answer features, the cross entropy classification loss corresponding to the image-level positive sample can be obtained: Cross entropy classification loss corresponding to image-level negative samples The formula is:

[0126]

[0127]

[0128] The classification results of positive samples should be closer to the classification results of the original samples than those of negative samples. Therefore, the reasoning results of positive samples should be more accurate than those of negative samples. Based on the above inferences, based on the classification losses corresponding to positive and negative samples, an auxiliary learning loss is constructed to directly optimize the answer prediction results in visual common sense reasoning. The specific loss function is defined as follows:

[0129]

[0130]

[0131]

[0132] Where, L aux is the auxiliary classification loss function, is the auxiliary classification loss function corresponding to the instance-level samples, is the auxiliary classification loss function corresponding to the image-level sample, with amplitude parameter Δ = 0.3 and weight parameter γ = 0.5. By optimizing the above function, the model can better distinguish between positive and negative samples, thereby improving the reasoning ability of visual common sense reasoning.

[0133] Integrating all the above loss functions, we get the loss function of the visual common sense reasoning model, which can be expressed as:

[0134] L=L base +λ1L ins +λ2Limg +λ3L sem +λ4L aux

[0135] Where L is the loss function of the visual common sense reasoning model, and λ1,λ2,λ3,λ4 are balancing parameters.

[0136] Based on any of the above embodiments, determining instance-level positive and negative samples based on the sample inference image includes:

[0137] Based on the reasoning contribution of each object in any sample reasoning image, a set of objects that meet a first preset number is selected in descending order as instance-level positive samples corresponding to any sample reasoning image, and objects that meet the first preset number are selected in ascending order as instance-level negative samples corresponding to any sample reasoning image.

[0138] Specifically, the gradient-based Grad-CAM method can be used to calculate the objects in the image Contribution to visual reasoning, where w is the object v in the sample reasoning image w The label of , n is the number of objects in the sample reasoning image.

[0139] The first preset number can be denoted as k, According to the reasoning contribution of each object in any sample reasoning image, a set of k objects is selected in descending order as the instance-level positive sample O ins+ , that is, select the first k objects with high contribution to construct the fact sample.

[0140] At the same time, select k objects in ascending order as instance-level negative samples corresponding to any sample inference image That is, select the first k objects with low contribution to construct counterfactual samples.

[0141] Using the image encoding layer, we can extract the features of instance-level positive and negative samples, which are recorded as and

[0142] Based on any of the above embodiments, determining image-level positive and negative samples based on the sample inference image includes:

[0143] Based on the reasoning contribution of each object in any sample reasoning image, objects satisfying a second preset number are selected in ascending order for masking, and the sample reasoning image obtained after masking is used as an image-level positive sample corresponding to any sample reasoning image;

[0144] Based on other sample inference images except any sample inference image, image-level negative samples are determined.

[0145] Specifically, according to the reasoning contribution of each object in any sample reasoning image, objects that meet the second preset number are selected in ascending order for masking, and the sample reasoning image obtained after masking is used as the image-level positive sample I corresponding to any sample reasoning image. img+ .

[0146] The second preset number is marked as y, and can be determined based on the total number n of objects in any sample inference image and the first preset number k. For example,

[0147] At the same time, in order to destroy the original image scene, m images are randomly selected from other sample inference images as negative samples

[0148] Using the image coding layer, the features of positive and negative samples at the image level can be extracted respectively, which are recorded as and

[0149] Based on any of the above embodiments, the present invention provides a visual common sense reasoning method based on multi-level counterfactual comparison, the method comprising:

[0150] Step 1: Extract the features of the original image and text and build a basic visual common sense reasoning module;

[0151] Step 2: Use counterfactuals to generate instance-level positive and negative samples, build instance-level contrastive learning, and make the model focus on the details of the image;

[0152] Step 3: Use counterfactuals to generate image-level positive and negative samples, build image-level contrastive learning, and make the model focus on the global scene information of the image;

[0153] Step 4: Consider the counterfactual samples of the correct option text at the semantic level, construct semantic-level contrastive learning, and enable the model to model cross-modal semantic relationships;

[0154] Step 5: Construct a comparison between the prediction results for instance-level and image-level positive and negative samples;

[0155] Step 6: Integrate all the above parts into a unified framework to perform overall training of the visual reasoning model.

[0156] The beneficial effects of the visual common sense reasoning method provided by the embodiment of the present invention include:

[0157] (1) The embodiment of the present invention proposes a multi-level counterfactual contrastive learning framework for visual common sense reasoning, which improves the model's ability to understand and reason about diverse visual content, high-level text semantics, and complex cross-modal relationships.

[0158] (2) The embodiments of the present invention introduce counterfactual thinking into contrastive learning to generate detailed factual and counterfactual samples, which enhances the perception ability of the model and helps the model to more effectively extract discriminative feature representations.

[0159] (3) The embodiment of the present invention integrates an auxiliary comparison module into the method to directly optimize answer prediction, further improving the performance of visual common sense reasoning.

[0160] Based on any of the above embodiments, Figure 2 A schematic diagram of the structure of the visual common sense reasoning device provided by the present invention is shown in FIG. Figure 2 As shown, the device comprises:

[0161] An acquisition unit 210 is used to acquire an image to be inferred, a reasoning question corresponding to the image to be inferred, and a plurality of candidate reasoning answers;

[0162] The reasoning unit 220 is used to input the image to be reasoned, the reasoning question and multiple candidate reasoning answers into the visual common sense reasoning model to obtain the reasoning result output by the visual common sense reasoning model; wherein the visual common sense reasoning model is trained based on the sample reasoning image, the sample reasoning question corresponding to the sample reasoning image and multiple candidate sample reasoning answers.

[0163] The visual common sense reasoning device provided by the embodiment of the present invention obtains a visual common sense reasoning model through sample reasoning images, sample reasoning problems corresponding to the sample reasoning images, and multiple candidate sample reasoning answers. The obtained visual common sense reasoning model can process the image to be reasoned, as well as the reasoning problems and multiple candidate reasoning answers corresponding to the image to be reasoned, to obtain the reasoning result. Since the visual common sense reasoning model can comprehensively understand multimodal data such as images and texts, it can learn the correlation between the image to be reasoned, the reasoning problem and each candidate reasoning answer, and has a strong feature learning ability and understanding reasoning ability. The error of the obtained reasoning result is small, which improves the accuracy of visual common sense reasoning.

[0164] Based on any of the above embodiments, the reasoning unit 220 is used to:

[0165] Input the image to be inferred into the image encoding layer of the visual common sense reasoning model to obtain the image features output by the image encoding layer;

[0166] Input the reasoning question into the text encoding layer of the visual common sense reasoning model to obtain the question features output by the text encoding layer;

[0167] The image features, question features, and multiple candidate reasoning answers are input into the result classification layer of the visual common sense reasoning model to obtain the reasoning results output by the result classification layer.

[0168] Based on any of the above embodiments, it also includes:

[0169] An image positive and negative sample determination unit, used to determine instance-level positive and negative samples and image-level positive and negative samples based on sample reasoning images;

[0170] A text positive and negative sample determination unit, used to infer answers based on multiple candidate samples and determine semantic-level positive and negative samples;

[0171] The training unit is used to train the initial model based on sample reasoning images, sample reasoning problems, instance-level positive and negative samples, image-level positive and negative samples, and semantic-level positive and negative samples to obtain a visual common sense reasoning model.

[0172] Based on any of the above embodiments, the training unit is used for:

[0173] Input the sample inference image into the image coding layer of the initial model to obtain the sample image features output by the image coding layer;

[0174] Input the instance-level positive and negative samples into the image coding layer of the initial model to obtain the instance-level positive sample image features and instance-level negative sample image features output by the image coding layer;

[0175] Input the image-level positive and negative samples into the image coding layer of the initial model, and obtain the image-level positive sample image features and the image-level negative sample image features output by the image coding layer;

[0176] Input the sample reasoning problem into the text encoding layer of the initial model to obtain the sample problem features output by the text encoding layer;

[0177] Input semantic-level positive and negative samples into the text encoding layer of the initial model to obtain the sample reasoning answer features output by the text encoding layer;

[0178] After fusing the sample image features, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features and image-level negative sample image features with the sample problem features respectively, the sample image fusion features, instance-level positive sample fusion features, instance-level negative sample fusion features, image-level positive sample fusion features and image-level negative sample fusion features are obtained;

[0179] The sample image fusion features, instance-level positive sample fusion features, instance-level negative sample fusion features, image-level positive sample fusion features and image-level negative sample fusion features are respectively concatenated with the sample reasoning answer features and input into the result classification layer of the initial model. The initial model is trained with the purpose of minimizing the contrast loss between positive and negative samples and the cross entropy loss of model classification to obtain a visual common sense reasoning model.

[0180] Based on any of the above embodiments, it also includes:

[0181] The loss function determination unit is used to determine the basic classification loss function based on the cross entropy loss function, the sample image fusion features and the sample reasoning answer features; determine the instance-level contrast loss function based on the contrast loss function, the sample image fusion features, the instance-level positive sample fusion features and the instance-level negative sample fusion features; determine the image-level contrast loss function based on the contrast loss function, the sample image fusion features, the image-level positive sample fusion features and the image-level negative sample fusion features; determine the semantic-level contrast loss function based on the contrast loss function, the sample image fusion features and the sample reasoning answer features; determine the auxiliary classification loss function based on the cross entropy loss function, the instance-level positive sample image features, the instance-level negative sample image features, the image-level positive sample image features, the image-level negative sample image features and the sample reasoning answer features; determine the loss function of the visual common sense reasoning model based on the basic classification loss function, the instance-level contrast loss function, the image-level contrast loss function, the semantic-level contrast loss function and the auxiliary classification loss function.

[0182] Based on any of the above embodiments, the image positive and negative sample determination unit includes:

[0183] An instance-level positive and negative sample determination unit is used to select a set of objects that meet a first preset number in descending order as instance-level positive samples corresponding to any sample reasoning image based on the reasoning contribution of each object in any sample reasoning image, and to select objects that meet the first preset number in ascending order as instance-level negative samples corresponding to any sample reasoning image.

[0184] Based on any of the above embodiments, the image positive and negative sample determination unit includes:

[0185] The image-level positive and negative sample determination unit is used to select objects that meet a second preset number in ascending order for masking based on the reasoning contribution of each object in any sample reasoning image, and use the sample reasoning image obtained after masking as the image-level positive sample corresponding to any sample reasoning image; and determine the image-level negative sample based on other sample reasoning images except any sample reasoning image.

[0186] Based on any of the above embodiments, Figure 3 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 3As shown, the electronic device may include: a processor (Processor) 310, a communication interface (Communications Interface) 320, a memory (Memory) 330 and a communication bus (Communications Bus) 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic command in the memory 330 to execute the following method:

[0187] Obtain an image to be inferred, as well as a reasoning question and multiple candidate reasoning answers corresponding to the image to be inferred; input the image to be inferred, the reasoning question and the multiple candidate reasoning answers into a visual common sense reasoning model to obtain a reasoning result output by the visual common sense reasoning model; wherein the visual common sense reasoning model is trained based on a sample reasoning image, as well as a sample reasoning question and multiple candidate sample reasoning answers corresponding to the sample reasoning image.

[0188] In addition, the logic commands in the above-mentioned memory 330 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0189] The processor in the electronic device provided in the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0190] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0191] Obtain an image to be inferred, as well as a reasoning question and multiple candidate reasoning answers corresponding to the image to be inferred; input the image to be inferred, the reasoning question and the multiple candidate reasoning answers into a visual common sense reasoning model to obtain a reasoning result output by the visual common sense reasoning model; wherein the visual common sense reasoning model is trained based on a sample reasoning image, as well as a sample reasoning question and multiple candidate sample reasoning answers corresponding to the sample reasoning image.

[0192] When the computer program stored on the non-transitory computer-readable storage medium provided by the embodiment of the present invention is executed, the above method is implemented. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual common sense reasoning method, characterized in that: include: Acquire an image to be inferred, as well as a reasoning question and multiple candidate reasoning answers corresponding to the image to be inferred; Inputting the image to be inferred, the reasoning question and the multiple candidate reasoning answers into a visual common sense reasoning model to obtain a reasoning result output by the visual common sense reasoning model; The visual common sense reasoning model is trained based on a sample reasoning image, a sample reasoning question corresponding to the sample reasoning image, and a plurality of candidate sample reasoning answers; The visual common sense reasoning model is trained based on the following steps: Based on the sample inference image, determining instance-level positive and negative samples and image-level positive and negative samples; Inferring answers based on the multiple candidate samples, determining semantic-level positive and negative samples; Based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples, the initial model is trained to obtain the visual common sense reasoning model; The initial model is trained based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples to obtain the visual common sense reasoning model, including: Inputting the sample inference image into the image coding layer of the initial model to obtain sample image features output by the image coding layer; Inputting the instance-level positive and negative samples into the image coding layer of the initial model to obtain instance-level positive sample image features and instance-level negative sample image features output by the image coding layer; Inputting the image-level positive and negative samples into the image coding layer of the initial model to obtain image-level positive sample image features and image-level negative sample image features output by the image coding layer; Inputting the sample reasoning problem into the text encoding layer of the initial model to obtain the sample problem features output by the text encoding layer; Inputting the semantic-level positive and negative samples into the text encoding layer of the initial model to obtain sample reasoning answer features output by the text encoding layer; After fusing the sample image features, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features and image-level negative sample image features with the sample problem features respectively, a sample image fusion feature, an instance-level positive sample fusion feature, an instance-level negative sample fusion feature, an image-level positive sample fusion feature and an image-level negative sample fusion feature are obtained; The sample image fusion feature, the instance-level positive sample fusion feature, the instance-level negative sample fusion feature, the image-level positive sample fusion feature and the image-level negative sample fusion feature are respectively concatenated with the sample reasoning answer feature and input into the result classification layer of the initial model, and the initial model is trained for the purpose of minimizing the contrast loss between positive and negative samples and the cross entropy loss of model classification, so as to obtain the visual common sense reasoning model; The instance-level positive and negative samples are positive and negative samples constructed based on each object in the sample reasoning image; the image-level positive and negative samples are positive and negative samples constructed based on the sample reasoning image as a whole; the semantic-level positive and negative samples are positive and negative samples constructed based on the candidate sample reasoning answer; The determining, based on the sample inference image, instance-level positive and negative samples comprises: Based on the reasoning contribution of each object in any sample reasoning image, a set of objects that meet a first preset number are selected in descending order as instance-level positive samples corresponding to the any sample reasoning image, and objects that meet the first preset number are selected in ascending order as instance-level negative samples corresponding to the any sample reasoning image; The determining of image-level positive and negative samples based on the sample inference image includes: Based on the reasoning contribution of each object in any sample reasoning image, objects satisfying a second preset number are selected in ascending order for masking, and the sample reasoning image obtained after masking is used as an image-level positive sample corresponding to the any sample reasoning image; Determining the image-level negative sample based on other sample inference images except any one of the sample inference images; The determining of semantic-level positive and negative samples based on the inference answers of the multiple candidate samples includes: Based on the true sample inference answers among the candidate sample inference answers, semantic-level positive samples are determined; based on the candidate sample inference answers other than the true sample inference answers, semantic-level negative samples are determined.

2. The visual common sense reasoning method according to claim 1, characterized in that: The step of inputting the image to be inferred, the inference question and the plurality of candidate inference answers into a visual common sense inference model to obtain an inference result output by the visual common sense inference model includes: Inputting the image to be inferred into the image coding layer of the visual common sense reasoning model to obtain image features output by the image coding layer; Inputting the reasoning question into the text encoding layer of the visual common sense reasoning model to obtain the question features output by the text encoding layer; The image features, the question features, and the multiple candidate reasoning answers are input into a result classification layer of the visual common sense reasoning model to obtain a reasoning result output by the result classification layer.

3. The visual common sense reasoning method according to claim 1, characterized in that: The loss function of the visual common sense reasoning model is determined based on the following steps: Determine a basic classification loss function based on a cross entropy loss function, the sample image fusion feature, and the sample reasoning answer feature; Determine an instance-level contrast loss function based on the contrast loss function, the sample image fusion feature, the instance-level positive sample fusion feature, and the instance-level negative sample fusion feature; Determine an image-level contrast loss function based on the contrast loss function, the sample image fusion feature, the image-level positive sample fusion feature, and the image-level negative sample fusion feature; Determining a semantic level contrast loss function based on the contrast loss function, the sample image fusion feature and the sample reasoning answer feature; Determine an auxiliary classification loss function based on a cross entropy loss function, the instance-level positive sample image features, the instance-level negative sample image features, the image-level positive sample image features, the image-level negative sample image features, and the sample reasoning answer features; Based on the basic classification loss function, the instance-level contrast loss function, the image-level contrast loss function, the semantic-level contrast loss function and the auxiliary classification loss function, a loss function of the visual common sense reasoning model is determined.

4. A visual common sense reasoning device, characterized in that: include: An acquisition unit, used to acquire an image to be inferred, and a reasoning question and a plurality of candidate reasoning answers corresponding to the image to be inferred; An inference unit, used for inputting the image to be inferred, the inference question and the multiple candidate inference answers into a visual common sense inference model, and obtaining an inference result output by the visual common sense inference model; wherein the visual common sense inference model is trained based on a sample inference image, a sample inference question corresponding to the sample inference image and multiple candidate sample inference answers; The device comprises: An image positive and negative sample determination unit, used to determine instance-level positive and negative samples and image-level positive and negative samples based on the sample inference image; A text positive and negative sample determination unit, used to infer answers based on the multiple candidate samples and determine semantic level positive and negative samples; A training unit, configured to train an initial model based on the sample reasoning image, the sample reasoning problem, the instance-level positive and negative samples, the image-level positive and negative samples, and the semantic-level positive and negative samples to obtain the visual common sense reasoning model; The training unit is used for: Inputting the sample inference image into the image coding layer of the initial model to obtain sample image features output by the image coding layer; Inputting the instance-level positive and negative samples into the image coding layer of the initial model to obtain instance-level positive sample image features and instance-level negative sample image features output by the image coding layer; Inputting the image-level positive and negative samples into the image coding layer of the initial model to obtain image-level positive sample image features and image-level negative sample image features output by the image coding layer; Inputting the sample reasoning problem into the text encoding layer of the initial model to obtain the sample problem features output by the text encoding layer; Inputting the semantic-level positive and negative samples into the text encoding layer of the initial model to obtain sample reasoning answer features output by the text encoding layer; After fusing the sample image features, instance-level positive sample image features, instance-level negative sample image features, image-level positive sample image features and image-level negative sample image features with the sample problem features respectively, a sample image fusion feature, an instance-level positive sample fusion feature, an instance-level negative sample fusion feature, an image-level positive sample fusion feature and an image-level negative sample fusion feature are obtained; The sample image fusion feature, the instance-level positive sample fusion feature, the instance-level negative sample fusion feature, the image-level positive sample fusion feature and the image-level negative sample fusion feature are respectively concatenated with the sample reasoning answer feature and input into the result classification layer of the initial model, and the initial model is trained for the purpose of minimizing the contrast loss between positive and negative samples and the cross entropy loss of model classification, so as to obtain the visual common sense reasoning model; The instance-level positive and negative samples are positive and negative samples constructed based on each object in the sample reasoning image; the image-level positive and negative samples are positive and negative samples constructed based on the sample reasoning image as a whole; the semantic-level positive and negative samples are positive and negative samples constructed based on the candidate sample reasoning answer; The image positive and negative sample determination unit is used for: Based on the reasoning contribution of each object in any sample reasoning image, a set of objects that meet a first preset number are selected in descending order as instance-level positive samples corresponding to the any sample reasoning image, and objects that meet the first preset number are selected in ascending order as instance-level negative samples corresponding to the any sample reasoning image; Based on the reasoning contribution of each object in any sample reasoning image, objects satisfying a second preset number are selected in ascending order for masking, and the sample reasoning image obtained after masking is used as an image-level positive sample corresponding to the any sample reasoning image; Determining the image-level negative sample based on other sample inference images except any one of the sample inference images; The text positive and negative sample determination unit is used for: Based on the true sample inference answers among the candidate sample inference answers, semantic-level positive samples are determined; based on the candidate sample inference answers other than the true sample inference answers, semantic-level negative samples are determined.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the visual common sense reasoning method according to any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual common sense reasoning method according to any one of claims 1 to 3 are implemented.

Images