A method and apparatus for generating visual dialogues based on counterfactual common sense causal reasoning

Abstract

This invention discloses a visual dialogue generation method and apparatus based on counterfactual common sense causal reasoning. The method includes: introducing common sense into the visual dialogue task, constructing a visual dialogue causal graph based on common sense fusion, and calculating the total effect of input features on answer prediction; constructing a corresponding counterfactual causal graph based on the visual dialogue causal graph; estimating the impact of harmful biases introduced by common sense on answer prediction based on the natural direct effect of the counterfactual causal graph, and removing them from the total effect; training a visual dialogue model using model ensemble, with cross-entropy and KL divergence loss as training objectives to obtain answer prediction results; integrating image features, dialogue history features, common sense features, and current question features, feeding them into a decoder, minimizing the loss function, optimizing network parameters, and finally providing realistic disaster environment information for disaster relief sites. The apparatus includes a processor and a memory.

Description

Technical Field

[0001] This invention relates to the field of visual dialogue generation, and more particularly to a method and apparatus for generating visual dialogue based on counterfactual common sense causal reasoning. Background Technology

[0002] With the rapid development of computer vision and natural language processing technologies, the multimodal field of visual and language interaction has received widespread attention. From image description... [1] Scene graph generation [2] Visual Q&A [3] From visual dialogue [4] Researchers are dedicated to improving the ability of computers to engage in sustained interaction with humans. Visual dialogue has been a key research focus in the multimodal field, requiring agents to continuously infer answers to current questions based on existing image information and textual information embedded in past question-and-answer sequences. The dialogue between humans and the agent can last for multiple rounds. Therefore, visual dialogue tasks demand strong human-computer interaction capabilities from the agent, making it highly valuable for applications in areas such as assisting the visually impaired and disaster relief.

[0003] In recent years, many excellent works have emerged in the field of visual dialogue. For example, methods based on recurrent neural networks... [4] Recurrent neural networks and their variants are used to encode multimodal features of visual-linguistic communication to obtain answers; attention-based methods are employed. [5][6] The primary approach utilizes attention mechanisms to extract more refined image information and contextual information from the dialogue history needed to answer the current question, based on graph structures. [7][8] The primary method uses graph structures to encode images, dialogue history, or the current question, endowing the agent with stronger reasoning abilities to generate answers. These methods all rely on existing image information and textual information embedded in historical question-and-answer sequences to deduce answers. However, in more complex dialogue scenarios, simply using this information is far from sufficient. The agent also needs to utilize external common-sense knowledge, much like humans, to assist in answer generation. This aspect is often overlooked by researchers, thus limiting the improvement of the agent's human-computer interaction capabilities. For example, at a fire rescue scene, the agent can enter the fire area beforehand to take photos and answer rescuers' questions in real time. When rescuers ask, "Are there gas cylinders at the fire scene?", if the agent lacks common-sense knowledge such as "the gas cylinders are in the kitchen," it will not focus on the kitchen area and will therefore be unable to answer the rescuers' question correctly.

[0004] Currently, only knowledge-based structured network methods (SKANet) are available. [9] And the common sense reasoning method based on multi-structure knowledge (RMK)

[10] Various approaches have introduced external common-sense knowledge into visual dialogue tasks, achieving significant progress. These approaches extract common-sense knowledge from external common-sense databases, encode it, and then integrate it with multimodal information to arrive at the answer. However, these frameworks are all based on the underlying assumption that common-sense knowledge will always positively influence answer generation. Although they filter out common-sense knowledge irrelevant to the current dialogue by calculating the semantic similarity between image descriptions and common-sense knowledge, constructing a common-sense knowledge graph using graph embedding algorithms (TransE algorithm), and calculating the cosine distance between nodes, some "harmful biases" inherent in common-sense knowledge are still present, negatively impacting answer generation. For example, keywords in image tags or descriptions used to retrieve common-sense knowledge that appear frequently in common-sense knowledge may interfere with the agent's answer generation or even cause the agent to generate incorrect answers containing these high-frequency words. For instance, if an agent possesses common-sense knowledge related to "gas cylinders," when answering rescuers' question, "Are there gas cylinders at the fire scene?", the agent might focus excessively on incorrect answers containing the high-frequency word "gas cylinder," thus failing to provide accurate information to rescuers and increasing the difficulty of the rescue.

[0005] Based on the current state of research, the challenges we face are mainly in the following three aspects: (1) how to more effectively select and utilize common sense knowledge related to images and current dialogues to assist in answer generation; (2) how to quantify the negative impact of "harmful biases" contained in common sense on answer generation; (3) how to remove the negative impact of common sense on answer generation in general and retain only the positive impact of common sense on answer generation, thereby improving the accuracy of answer prediction by the agent at the disaster relief site, so as to help rescuers better understand the disaster site environment, formulate rescue plans, and carry out rescue work. Summary of the Invention

[0006] This invention provides a visual dialogue generation method and apparatus based on counterfactual common sense causal reasoning. The invention constructs a visual dialogue fact-cause graph based on common sense fusion. The natural direct effect of common sense on answer generation is subtracted from the answer prediction score generated from the visual dialogue based on this causal graph. This process retains the positive influence of common sense on answer generation while removing the negative impact of "harmful biases" inherent in common sense, thereby improving the human-computer interaction capabilities of intelligent agents at disaster relief sites and providing rescuers with more realistic and detailed disaster environment information. See the description below for details:

[0007] A visual dialogue generation method based on counterfactual common sense causal reasoning, the method comprising:

[0008] For the extracted common sense triples, a common sense subgraph is constructed, and an attention graph convolutional network is used to encode the subgraph to obtain common sense features;

[0009] Common sense is introduced into the visual dialogue task, a visual dialogue causal graph based on common sense fusion is constructed, and the total effect of input features on answer prediction is calculated; based on the visual dialogue causal graph, its corresponding counterfactual causal graph is constructed.

[0010] For counterfactual causal graphs, the harmful bias introduced by common sense on answer prediction is estimated based on the natural direct effect, and then removed from the total effect;

[0011] A visual dialogue model is trained using model ensemble, with cross-entropy and KL divergence loss as training objectives to obtain answer prediction results.

[0012] The system integrates image features, dialogue history features, common sense features, and current problem features, feeds them into the decoder, minimizes the loss function, optimizes network parameters, and finally provides real disaster environment information for disaster relief sites.

[0013] The common sense triples are: common sense related to each visual dialogue unit extracted from the training set, validation set, and test set samples of the database; the specific operation of extracting the common sense triples is as follows:

[0014] The Faster R-CNN framework is used to detect object labels, and the top few object labels with the highest confidence scores are selected based on the confidence scores of each label; similarly, the top few keywords with the highest scores are selected for image descriptions.

[0015] The construction of the corresponding counterfactual causal graph based on the visual dialogue causal graph is as follows:

[0016] By assigning null values ​​to I, Q, H, and C respectively, i.e., I = i*, Q = q*, H = h*, and C = c*, and thus K = k*, the influence of I, Q, H, and K on the prediction of answer A is blocked.

[0017] In a counterfactual world, common sense C can be assigned two values ​​at the same time: C = c* and C = c. The former is used to obtain K = k*, while the latter is directly connected to the answer A to evaluate the natural direct effect of common sense C on the prediction of the answer A.

[0018] A second aspect is a visual dialogue generation device based on counterfactual common sense causal reasoning, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method described in any of the first aspects.

[0019] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the method described in any of the first aspects.

[0020] The beneficial effects of the technical solution provided by this invention are:

[0021] 1. This invention introduces external common sense knowledge into existing visual dialogue tasks. The agent can not only reason based on existing image information and dialogue history context information, but also generate answers using information from common sense knowledge. Existing visual dialogue methods often ignore the important role of common sense knowledge in the visual dialogue generation process. This invention focuses on the information sources required by the agent when reasoning for answers. The introduction of common sense knowledge continuously improves the agent's human-computer interaction ability and effectively enhances the accuracy of answer generation.

[0022] 2. This invention, while introducing common sense knowledge into the visual dialogue task, considers the negative impact of "harmful biases" inherent in common sense on answer generation. It constructs a counterfactual causal graph based on common sense fusion, quantifying this negative impact as the natural direct effect of common sense on answer generation. Existing visual dialogue generation methods based on common sense fusion merely introduce common sense into the visual dialogue task, making the agent susceptible to interference from "harmful biases" when generating answers. This invention focuses on the impact of common sense knowledge on answer generation in the visual dialogue task, using the natural direct effect in causal theory to quantify and remove the negative impact of "harmful biases" inherent in common sense on answer generation, thereby further improving the accuracy of answer generation.

[0023] 3. This invention constructs a fact-cause graph for visual dialogue based on common sense fusion. Existing visual dialogue frameworks based on common sense fusion often deduce from this graph. This invention derives the total effect of common sense on answer generation based on this graph, removes the natural direct effect of common sense on answer generation from the total effect, and further minimizes the KL-divergence loss function to make the answer probability distribution of the natural direct effect similar to that of the total effect. This better removes the negative impact of "harmful bias" inherent in common sense on answer generation while retaining the positive impact of common sense on answer generation. Existing methods often overlook this point. This invention focuses on removing the negative impact of "harmful bias" inherent in common sense on answer generation and improving the human-computer interaction capabilities of the intelligent agent.

[0024] 4. The present invention uses the counterfactual analysis method in causal theory to remove the negative impact of "harmful bias" contained in common sense on answer generation. It has a certain degree of generalization and can be applied to most visual dialogue models.

[0025] 5. Applying the generation results of this invention to disaster relief missions improves the accuracy of the intelligent agent's answer prediction when responding to rescuers' questions in real time, enabling rescuers to fully understand the environmental information at the disaster site, thereby better formulating rescue plans, reducing the difficulty of rescue, and avoiding rescue casualties. Attached Figure Description

[0026] Figure 1 A flowchart of a visual dialogue generation method based on counterfactual common sense causal reasoning;

[0027] Figure 2 A schematic diagram illustrating causal theory and counterfactual analysis;

[0028] Figure 3 A fact-cause graph for visual dialogue based on common sense fusion;

[0029] Figure 4 This is a counterfactual causal graph for visual dialogue based on common sense fusion.

[0030] Figure 5 This is a model diagram of a visual dialogue generation method based on counterfactual common sense causal reasoning. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0032] Example 1

[0033] To address the problem that current visual dialogue systems ignore harmful biases introduced by common sense when incorporating it, and thus improve the accuracy of answer prediction, commonly used metrics include recall and normalized loss cumulative gain. (See [link to relevant documentation]). Figure 1 This invention provides a visual dialogue generation method based on counterfactual causal reasoning, which includes the following steps:

[0034] 101: For the extracted common sense triples, construct a common sense subgraph, and use an attention graph convolutional network to encode the subgraph to obtain common sense features;

[0035] 102: Introduce common sense into the visual dialogue task, construct a visual dialogue causal graph based on common sense fusion, calculate the total effect of input features on answer prediction; based on the visual dialogue causal graph, construct its corresponding counterfactual causal graph;

[0036] 103: For counterfactual causal diagrams, estimate the impact of harmful biases introduced by common sense on answer prediction based on natural direct effects, and remove them from the total effect;

[0037] 104: The visual dialogue model is trained using model ensemble, with cross-entropy and KL divergence loss as training objectives to obtain answer prediction results;

[0038] 105: The image features, dialogue history features, common sense features, and current problem features are integrated and fed into the decoder to minimize the loss function, optimize network parameters, and finally provide real disaster environment information for disaster relief sites.

[0039] In summary, the embodiments of the present invention, through the above steps 101-105, achieve the removal of the negative impact of "harmful biases" inherent in common sense on answer generation, thereby improving the human-computer interaction capabilities of the intelligent agent at the disaster relief site and providing rescuers with more realistic and detailed disaster environment information.

[0040] Example 2

[0041] The scheme in Example 1 will be further described below with specific calculation formulas and examples:

[0042] 201: For the database used, extract common sense related to each visual dialogue unit from its training set, validation set, and test set samples;

[0043] The dataset used consists of visual dialogue units. Each visual dialogue unit includes: an image and its corresponding image description, historical question-answer pairs, and the question to be answered, with each question corresponding to 100 candidate answers. The commonly used VisDial v0.9 and VisDial v1.0 databases are used here, but the embodiments of the present invention are not limited to these databases and can use any database containing the required visual dialogue units.

[0044] For the required common sense source, this embodiment of the invention uses the commonly used ConceptNet common sense knowledge base. This knowledge base is based on the most basic things a person knows, and it consists of basic common sense triples. The specific form of a triple is <start node, relation label, end node>, where the start node and end node both refer to words or phrases in natural language, such as...<bus,capable of,transport people> Each triple is given a confidence score, and the higher the score, the more reliable the relationship between the start and end nodes of the triple.

[0045] First, in order to retrieve common knowledge related to each dialogue unit from the common knowledge base, search terms need to be generated. In this embodiment of the invention, the search terms are derived from the labels of objects in the image and the image description. Specifically, for the labels of objects in the image, the object labels are detected using the Faster R-CNN (Fast Region Convolutional Neural Network) framework, and the top 5 object labels with the highest confidence scores are selected based on the confidence scores corresponding to each label; similarly, for the image description and keywords, the top 5 keywords with the highest confidence scores are selected based on the confidence scores corresponding to each keyword. Thus, the extracted 5 object labels and 5 keywords together constitute the search terms used to retrieve common knowledge.

[0046] Next, relevant common sense is retrieved from the ConceptNet common sense knowledge base using search terms. Specifically, the search terms are matched sequentially with the start nodes of triples in the common sense knowledge base, and triples with the same search terms and start nodes are selected. Based on the confidence score given to each triple in the common sense knowledge base, the top 20 triples with the highest scores are selected.

[0047] 202: For the common sense triples extracted in step 201, first construct a common sense subgraph, and then use aGCN (note graph convolutional network) to encode the subgraph to obtain the common sense feature c;

[0048] First, the 20 common sense triples corresponding to each dialogue unit extracted in step 201 are used to construct a common sense subgraph G. c Among them, G c The included node features are the node features corresponding to the start and end nodes of all common sense triples. Specifically, GloVe word vector embedding is performed on the start and end nodes of each common sense triple to obtain the node features. Where d refers to the embedding dimension and i refers to the i-th node.

[0049] Then, aGCN (Attention Graph Convolutional Network) is used to further encode the commonsense subgraph G. c Update the current node feature z i . Specifically:

[0050]

[0051] Where W is a learnable linear transformation matrix; N(i) is the set of neighboring nodes of the i-th node; s is the non-linear activation function sigmoid; L is the number of convolutional layers in the aGCN network; a ij The feature z of the i-th node i With the feature z of the j-th node j Predefined weights between them;

[0052] Specifically:

[0053]

[0054] a ij =softmax(u ij )

[0055] Among them, W h and All are learnable parameters; [·,·] refers to concatenating node features; softmax is a non-linear activation function.

[0056] Finally, common sense subgraph G cEach node feature in the algorithm is updated through aGCN (note graph convolutional network), and it acquires feature information from its neighboring nodes. All updated node features are then averaged and pooled to output the commonsense feature c.

[0057] 203: The concepts of cause-and-effect diagrams and counterfactual analysis;

[0058] In this embodiment of the invention, a powerful graphical tool—the cause-effect graph—is used to explore the causal relationships behind variables. It is a directed acyclic graph (DAG). Each node in the cause-effect graph represents a variable, and the arrows between variables represent the causal relationships between them. Typically, the starting point of the arrow is the "cause," and the ending point is the "effect." Figure 2 As shown in (a), X→Y means that X has a direct influence on Y, where X is the "cause" and Y is the "effect". X→M→Y means that X indirectly influences Y through the intermediary M. When variables are assigned a certain value, such as X=x and X=x * (In this embodiment of the invention, lowercase letters are used to represent specific observed values ​​of variables, and lowercase letters are used to represent specific observed values ​​of variables.) * If "" indicates a null value, then the total effect (TE) of X on Y is calculated as follows:

[0059]

[0060] in, and They represent the values ​​at X = x and X = x respectively. * At that time, variable X affects variable Y through X→Y and X→M→Y.

[0061] Furthermore, the total effect TE can be decomposed into the natural direct effect (NDE) and the total indirect effect (TIE), namely:

[0062] TE = NDE + TIE

[0063] Specifically, in order to calculate the natural direct effects (NDEs), embodiments of the present invention construct a counterfactual world, such as... Figure 2 As shown in (d), in a counterfactual world, a variable can be assigned two values ​​at once (e.g., X = x or X = x). * This is clearly impossible in the real world, in the real world ( Figure 2 (b) Figure 2 In (c), a variable can only be assigned one value at a time (e.g., X = x or X = x). *When variable X directly affects variable Y through X→Y, variable X is assigned the value X=x; when variable X indirectly affects Y through X→M→Y, variable X is assigned the value X=x. * Then the total effect of variable X on Y is The method for calculating the natural direct effects NDE is as follows:

[0064]

[0065] Intuitively, NDE represents the extent of the direct influence (X→Y) of X on Y after excluding the indirect influence (X→M→Y) of X on Y.

[0066] Furthermore, the total indirect effect TIE is calculated as follows:

[0067]

[0068] 204: Introduce common sense into visual dialogue tasks and construct a causal graph for visual dialogue based on common sense fusion;

[0069] To address the limitations of most existing visual dialogue models based on common sense fusion, a causal graph for visual dialogue based on common sense fusion is constructed, as shown below. Figure 3 As shown.

[0070] First, visual features are extracted from the images using a visual encoder (most models use Faster-RCNN), and then processed in a causal graph ( Figure 3 In the causal graph, the visual feature is denoted as node I; the current question and dialogue history are usually encoded using a Long Short-Term Memory (LSTM) network to obtain the corresponding question features and history features, which are represented as nodes Q and H, respectively. The common sense feature output in step 202 is denoted as node C in the causal graph.

[0071] Then, the subgraph (I→K,Q→K,H→K,H→Q→K,C→K) represents the encoder input H,Q,I,C in the visual dialogue model, and the output multimodal feature K, where H→Q is the output since the question to be answered now originates from the previous dialogue history.

[0072] Finally, the problem features Q and multimodal features K are input into the discriminative decoder or the generative decoder to generate the answer A for the current problem. The corresponding subgraphs are Q→A and K→A.

[0073] Additionally, as described in step 201, although the search terms used to retrieve common sense originate from image descriptions in the image and dialogue history, i.e., I→C, H→C, the subgraphs I→C, H→C do not affect the visual dialogue causal graph based on common sense fusion. Figure 3 The causality between the input (H,Q,I,C) and output (A) of the visual dialogue model, as represented by the symbol, is simplified and therefore not included. Figure 3 The above is annotated. Although existing visual dialogue models based on common sense fusion may utilize common sense knowledge to improve answer prediction accuracy in different ways, without loss of generality, the causal relationship between common sense features and multimodal features can all be represented by the subgraph C→K. Meanwhile, this embodiment of the invention severs the direct connection between H and A, i.e., removes the subgraph H→A to avoid the visual dialogue model learning the linguistic biases inherent in H, which could negatively impact answer prediction.

[0074] 205: Based on the commonsense fusion-based visual dialogue causal graph from step 204, calculate the total effect TE of the input features (H,Q,I,C) on the prediction of the answer (A).

[0075] First, for the commonsense fusion-based visual dialogue causal graph in step 204, according to the calculation method of the total effect TE in step 203, specific observation values ​​and null values ​​are assigned to the input variables H, Q, I, and C respectively, i.e., I = i, Q = q, H = h, C = c and I = i * Q = q * H = h * C = c * Furthermore, for the visual dialogue model based on common sense fusion, observation i refers to the image features output by the visual encoder in step 204 for a specific image; observations q and h refer to the current question and dialogue history corresponding to the image, obtained by encoding them using a Long Short-Term Memory (LSTM) network in step 204, respectively; observation c refers to the common sense features extracted by the current visual dialogue unit in steps 201 and 202; null value i * ,q * ,h * ,c * These refer to features corresponding to visual dialogue models without input images, questions, dialogue history, and common sense.

[0076] Subsequently, when the input variables H, Q, I, and C are assigned specific observed values, their impact on the prediction of answer (A) is denoted as A(I=i, Q=q, H=h, C=c). When the input variables are assigned null values, their impact on the prediction of answer (A) is denoted as A(I=i). * Q = q * H = h * C = c * According to the calculation method of the total effect TE in step 103, we get:

[0077] TE=A(I=i,Q=q,H=h,C=c)-A(I=i * Q = q * H = h * C = c *)

[0078] For simplicity, in this embodiment of the invention, A (I=i,Q=q,H=h,C=c) is abbreviated as A i,q,h,c , convert A(I=i * Q = q * H = h * C = c * Abbreviated as A i*,q*,h*,c* .

[0079] like Figure 3 As shown, there are two direct paths connecting to answer (A), namely K→A and Q→A. Furthermore, in this embodiment of the invention, P(·) is used to represent the combined effect of K and Q on A. Then A... i,q,h,c It can also be written as P k,q , where P k,q =P(K=k,Q=q), where the observation k refers to the multimodal feature k output by the encoder in the visual dialogue model of step 204 after inputting I=i, Q=q, H=h, C=c; similarly, A i*,q*,h*,c* It can be denoted as P k*,q* , where P k*,q* =P(K=k*,Q=q*), where the null value k* refers to the encoder in the visual dialogue model of step 204 when the input null value I=i * Q = q * H = h * C = c * The output of the multimodal null value k* is then used to express the total effect TE as follows:

[0080]

[0081] Most existing visual dialogue methods based on common sense fusion rely on deduction. Figure 3 The causal graph shown is used to predict answers, but it ignores the harmful bias caused by the introduced common sense on the prediction of answers; that is, the total effect TE includes not only the "beneficial" effect of the introduced common sense on the prediction of answers, but also the negative effect of the harmful bias on the prediction of answers.

[0082] 206: In order to remove the harmful biases introduced by common sense in the visual dialogue task, based on the common sense fusion-based visual dialogue causal graph constructed in step 204, a corresponding counterfactual causal graph is constructed to estimate the negative impact of harmful biases on answer prediction.

[0083] Regarding the common-sense fusion-based visual dialogue causal graph constructed in step 204, common-sense C indirectly influences answer prediction through multimodal knowledge K, i.e., subgraph C→K→A. However, in practice, the model not only learns "beneficial" common-sense knowledge that helps improve answer prediction accuracy but also learns "harmful" biases that reduce answer prediction accuracy. For example, for the current question "What is the bus used for," the common-sense triples extracted in step 201 contain not only "beneficial" common-sense knowledge...<bus,capable of,transport people> This helps the model predict the correct answer "Transport people"; at the same time, the extracted common sense triples also contain...<bus,is a,public transport> ,<bus,is a,car> as well as<bus,at location,road> These factors create "harmful" biases that reduce the accuracy of answer prediction. For example, the starting node "bus" often appears in the extracted common sense triples, which may mislead the model to favor candidate answers containing "bus", such as "City bus" and "On a bus", and even make its prediction score higher than the prediction score of the correct answer "Transport people".

[0084] However, most existing visual dialogue models based on common sense fusion ignore the "harmful" biases introduced by common sense. In this embodiment of the invention, in order to estimate and remove the negative impact of "harmful" biases on answer prediction, a counterfactual causal graph is constructed based on the common sense fusion-based visual dialogue causal graph constructed in step 204. The core idea is to hypothesize "If only the common sense variable C is input to the visual dialogue model, while blocking the input of the other variables I, Q, H, and K, what impact will this have on answer prediction?" Specifically, this embodiment of the invention evaluates the natural direct effect (NDE) of common sense (C) on answer prediction (A) by blocking the influence of I, Q, H, and K on answer (A) prediction.

[0085] The counterfactual causal graph constructed by the embodiment of the present invention is as follows: Figure 4 As shown.

[0086] First, embodiments of the present invention utilize causal graphs based on common sense fusion (…). Figure 3 Based on this, by assigning null values ​​to I, Q, H, and C respectively, i.e., I = i*, Q = q*, H = h*, and C = c*, and then K = k*, the influence of I, Q, H, and K on the prediction of answer (A) is blocked.

[0087] Subsequently, as described in step 203, common sense C can be assigned two values ​​simultaneously in the counterfactual world: C = c* and C = c. The former is used to obtain K = k*, while the latter is directly connected to the answer A to evaluate the natural direct effect (NDE) of common sense (C) on the answer prediction (A). Since the model can only predict the answer through common sense (C) in the counterfactual world, the natural direct effect (NDE) preserves the influence of the harmful bias introduced by common sense on the answer prediction.

[0088] 207: For the counterfactual causal graph based on common sense fusion constructed in step 206, according to the calculation method of natural direct effect NDE in step 203, estimate the impact of harmful bias caused by the introduced common sense on answer prediction, and remove it from the total effect TE described in step 205.

[0089] First, regarding the counterfactual causal graph based on common sense fusion constructed in step 206, according to the calculation method of the natural direct effect NDE in step 203, the natural direct effect NDE of common sense on answer prediction is obtained as follows:

[0090]

[0091] in, This refers to the impact on answer prediction after blocking the input of question Q and multimodal knowledge K, while retaining only common sense C. This refers to the impact on answer prediction after assigning null values ​​to all input variables in a visual dialogue model.

[0092] Subsequently, to remove the harmful bias caused by introduced common sense from answer prediction, the natural direct effect (NDE) was removed from the total effect (TE), resulting in the total indirect effect (TIE):

[0093]

[0094] Finally, the effects of "beneficial" common-sense knowledge (i.e., TIE) are preserved, while the effects of "harmful" common-sense biases (i.e., NDE) are eliminated. It is important to clarify here that... Equivalent to Because they all refer to the impact on answer prediction after all inputs to the model are blocked. Furthermore, the counterfactual causal reasoning method proposed in this invention has strong generalization ability and can be applied to various visual dialogue models based on common sense fusion.

[0095] 208: Based on the total indirect effect TIE in step 207, the visual dialogue model is trained using the model ensemble method, and the answer prediction results are obtained by using cross-entropy loss and KL divergence loss as training objectives.

[0096] Regarding the total indirect effect TIE in step 207, in practical applications, this embodiment of the invention uses a model ensemble method for training, as follows: Figure 5 As shown. Specifically, P q,k The dialogue method can be decomposed into P q and P k Two components, respectively generated by neural network N Q and N K Calculation, i.e., P q =N Q (q) and P k =N K (i,q,h,c). N Q (q) represents the subgraph Q→A, where the features of the question after LSTM encoding are input into the softmax decoder to obtain the predicted scores of the candidate answers. N K (i,q,h,c) covers the subgraph {I→K,Q→K,H→K,H→Q→K,C→K}, where multimodal interactions between questions, history, images, and common sense are accomplished using an attention mechanism.

[0097] This invention integrates the outputs from two networks to generate P in the real world. q,k Final answer score:

[0098]

[0099] Where s is the non-linear activation function sigmoid.

[0100] Similarly, in a counterfactual world, Decomposed into and P c As described in step 206, in order to evaluate the direct effect of C on A, all signals, including Q and K, are blocked, i.e., N. Q and N K The input is invalid. In the counterfactual scenario, a reasonable strategy is to assume that both networks will guess and pick the answer without any specific processing. Specifically, if one or more of i, q, h, or c are not given, then... and It is assigned a learnable parameter m with a uniform distribution.

[0101] At the same time, P c By neural network N C Calculation, i.e., P c =N C (c), corresponding to subgraph C A. Specifically, common sense features are encoded using LSTM, and then a two-layer MLP is used to rank the candidate answers. Finally, all the network outputs are fused to obtain the counterfactual world... Answer score:

[0102]

[0103] During the reasoning phase, counterfactual reasoning is used for unbiased common-sense learning to facilitate VD prediction:

[0104]

[0105] The network is trained by minimizing the following loss function:

[0106]

[0107] Where a represents the ground truth answer; γ represents the loss weight; and CE represents the cross-entropy loss between the true answer and its predicted answer. Specifically, an ensemble model is used to perform counterfactual computation, where the three branches K→A, Q→A, and C→A are respectively located at P... k P q and P c Above. At the same time, in order to control The sharpness of the distribution is expressed using the KL divergence D. kl To optimize the learnable parameter m:

[0108]

[0109] Where, p(a|q,k)=softmax(P q,k ), Softmax is a non-linear activation function. When calculating D... kl At that time, only the learnable parameter m is optimized.

[0110] 209: The image features, dialogue history features, common sense features, and current question features are integrated and fed into the decoder. At the same time, the loss function is minimized, the network parameters are optimized, and finally the correct answer to the question posed by a human in the current scenario is generated.

[0111] Specifically, based on the fact-cause graph of common sense fusion, the visual dialogue network inputs real image features, dialogue history features, common sense features, and current question features, which are then fused and sent to the decoder. Based on the counterfactual causal graph of common sense fusion, the visual dialogue network still inputs real common sense features, while replacing the input of real image features, dialogue history features, and current question features with learnable parameters, which are then fused and sent to the same decoder. Based on the calculation method of the total indirect effect TIE in step 207 and the output results of the decoder in these two cases, the ranking prediction results of the candidate answers are obtained.

[0112] Simultaneously, by minimizing the loss function described in step 208 as the training objective, the parameters of the visual dialogue network are continuously optimized to achieve optimal performance in answer prediction, ultimately obtaining accurate answers to questions posed by humans in the current scenario. This invention has a certain degree of generalization and can be applied to most visual dialogue models. It improves the accuracy of answer prediction when the agent responds to questions from rescuers in real time, enabling rescuers to fully understand the environmental information at the disaster site, thereby better formulating rescue plans, reducing the difficulty of rescue operations, and avoiding rescue casualties.

[0113] Example 3

[0114] The feasibility of the schemes in Example 2 is verified below with specific examples, as detailed in Tables 1 and 2:

[0115] To verify the effectiveness of this invention, experiments were conducted on the Visdial-v1.0 and Visdial-v0.9 datasets using the method of this invention. Evaluation metrics for answer prediction included recall (R@1, R@5, R@10), normalized depreciation cumulative gain (NDCG), mean reciprocal rank (MRR) of the correct answer, and mean rank (Mean) of the correct answer. Higher recall (R@1, R@5, R@10), normalized depreciation cumulative gain (NDCG), and mean reciprocal rank (MRR) of the correct answer, coupled with a lower mean rank (Mean) of the correct answer, indicate a better visual dialogue generation method. Experimental results are shown in Table 1 (using the Visdial-v1.0 dataset) and Table 2 (using the Visdial-v0.9 dataset), respectively.

[0116] Table 1 shows the results of the method of the present invention on the Visdial-v1.0 test set.

[0117]

[0118]

[0119] Table 2 shows the results of the method of the present invention on the Visdial-v0.9 validation set.

[0120]

[0121] As can be seen from the results in Tables 1 and 2, this invention introduces common sense into the visual dialogue method, verifying that common sense knowledge can effectively improve the accuracy of answer prediction. At the same time, after introducing common sense knowledge, this invention uses counterfactual analysis to remove the negative impact of "harmful biases" contained in common sense on answer generation, while retaining the positive impact of common sense on answer generation. The experimental results verify the effectiveness of this invention.

[0122] This invention can be applied to disaster relief missions. When a disaster occurs, rescuers can send an intelligent agent to the disaster site in advance. The agent inputs images of the disaster site, relevant common knowledge, and question-and-answer history into the agent. The agent then answers the rescuers' questions in real time based on this information, enabling the rescuers to fully understand the disaster site environment, formulate reasonable rescue plans, and reduce casualties.

[0123] Example 4

[0124] A visual dialogue generation device based on counterfactual common sense causal reasoning, the device includes: a processor and a memory, the memory storing program instructions, and the processor calling the program instructions stored in the memory to cause the device to execute any one of the following method steps:

[0125] For the extracted common sense triples, a common sense subgraph is constructed, and an attention graph convolutional network is used to encode the subgraph to obtain common sense features;

[0126] Common sense is introduced into the visual dialogue task, a visual dialogue causal graph based on common sense fusion is constructed, and the total effect of input features on answer prediction is calculated; based on the visual dialogue causal graph, its corresponding counterfactual causal graph is constructed.

[0127] For counterfactual causal graphs, the harmful bias introduced by common sense on answer prediction is estimated based on the natural direct effect, and then removed from the total effect;

[0128] A visual dialogue model is trained using model ensemble, with cross-entropy and KL divergence loss as training objectives to obtain answer prediction results.

[0129] The system integrates image features, dialogue history features, common sense features, and current problem features, feeds them into the decoder, minimizes the loss function, optimizes network parameters, and finally provides real disaster environment information for disaster relief sites.

[0130] The common sense triples are derived by extracting common sense relevant to each visual dialogue unit from the training, validation, and test sets of the database. The specific operations for extracting the common sense triples are as follows:

[0131] The Faster R-CNN framework is used to detect object labels, and the top few object labels with the highest confidence scores are selected based on the confidence scores of each label; similarly, the top few keywords with the highest scores are selected for image descriptions.

[0132] Furthermore, the common sense features are obtained by encoding the subgraph using an attention graph convolutional network: the common sense subgraph G is encoded using aGCN. c Update the current node feature z i ;

[0133]

[0134] Where W is a learnable linear transformation matrix; N(i) is the set of neighboring nodes of the i-th node; s is the non-linear activation function sigmoid; L is the number of convolutional layers in the aGCN network; a ij The feature z of the i-th node i With the feature z of the j-th node j The predefined weights between nodes; after average pooling of all updated node features, the commonsense feature c is output.

[0135] Specifically, the introduction of common sense into the visual dialogue task and the construction of a visual dialogue causal graph based on common sense fusion are as follows:

[0136] Visual features are extracted from images using a visual encoder. In the causal graph, the visual features are denoted as node I. They are represented as nodes Q and H in the causal graph, respectively. Common sense features are denoted as node C in the causal graph.

[0137] Then, the subgraph (I→K,Q→K,H→K,H→Q→K,C→K) represents the encoder input H,Q,I,C in the visual dialogue model and the output multimodal feature K.

[0138] The problem features Q and multimodal features K are input into the discriminative decoder or generative decoder to produce the answer A for the current problem. The corresponding subgraphs are Q→A and K→A.

[0139] Furthermore, the total effect of the input features on answer prediction is calculated as follows:

[0140] First, specific observed values ​​and null values ​​are assigned to the input variables H, Q, I, and C, respectively, i.e., I = i, Q = q, H = h, C = c, and I = i. * Q = q * H = h * C = c * ;

[0141] When input variables H, Q, I, and C are assigned specific observed values, their impact on the prediction of answer A is denoted as A(I=i, Q=q, H=h, C=c). When input variables are assigned null values, their impact on the prediction of answer A is denoted as A(I=i). * Q = q * H = h * C = c * The total effect TE is calculated as follows:

[0142]

[0143] Among them, the corresponding counterfactual causal graph constructed based on the visual dialogue causal graph is as follows:

[0144] By assigning null values ​​to I, Q, H, and C respectively, i.e., I = i*, Q = q*, H = h*, and C = c*, and thus K = k*, the influence of I, Q, H, and K on the prediction of answer A is blocked.

[0145] In a counterfactual world, common sense C can be assigned two values ​​at the same time: C = c* and C = c. The former is used to obtain K = k*, while the latter is directly connected to the answer A to evaluate the natural direct effect of common sense C on the prediction of the answer A.

[0146] Furthermore, for the counterfactual causal diagram, the harmful bias introduced by common sense on answer prediction is estimated based on the natural direct effects, and then removed from the total effect as follows:

[0147]

[0148] in, This refers to the impact on answer prediction after blocking the input of question Q and multimodal knowledge K, while retaining only common sense C. This refers to the impact on answer prediction after assigning null values ​​to all input variables of a visual dialogue model;

[0149] The total indirect effect TIE is obtained as follows:

[0150]

[0151] Among them, the visual dialogue model is trained using model ensemble, and the answer prediction results are obtained by using cross-entropy and KL divergence loss as training objectives:

[0152]

[0153] Where s is the non-linear activation function sigmoid. Decomposed into and P c Each is composed of a neural network N Q NK and N C Calculate P q,k Decompose into P q and P k Two components, respectively generated by neural network N Q and N K calculate.

[0154] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0155] The aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, this embodiment of the invention does not limit the execution subject and selects it according to the needs of actual applications.

[0156] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0157] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0158] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0159] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0160] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0161] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0162] References:

[0163] [1]X.Yang,H.Zhang,and J.Cai,“Deconfounded image captioning:A causalretrospect,”IEEE Transactions on Pattern Analysis and Machine Intelligence,pp.1–1,2021.

[0164] [2]X.Han,X.Dong,X.Song,T.Gan,Y.Zhan,Y.Yan,and L.Nie,“Divide-and-conquer predictor for unbiased scene graph generation,”IEEE Trans.CircuitsSyst.Video Technol.,vol.32,pp.8611–8622,2022.

[0165] [3]Y.Niu,K.Tang,H.Zhang,Z.Lu,X.Hua,and J.Wen,“Counterfactual VQA:Acause-effect look at language bias,”in CVPR,2021,pp.12 700–12 710.

[0166] [4]A.Das,S.Kottur,K.Gupta,A.Singh,D.Yadav,J.M.F.Moura,D.Parikh,andD.Batra,“Visual dialog,”in CVPR,2017,pp.1080–1089.

[0167] [5]J.Lu,A.Kannan,J.Yang,D.Parikh,and D.Batra,“Best of both worlds:Transferring knowledge from discriminative learning to a generative visualdialog model,”in NIPS,2017,pp.314–324.

[0168] [6]Y.Niu,H.Zhang,M.Zhang,J.Zhang,Z.Lu,and J.Wen,“Recursive visualattention in visual dialog,”in CVPR,2019,pp.6679–6688.

[0169] [7]X.Jiang,S.Du,Z.Qin,Y.Sun,and J.Yu,“KBGN:knowledge-bridge graphnetwork for adaptive vision-text reasoning in visual dialogue,”in ACMMultimedia,2020,pp.1265–1273.

[0170] [8]D.Guo,H.Wang,and M.Wang,“Context-aware graph inference withknowledge distillation for visual dialog,”IEEE Trans.PatternAnal.Mach.Intell.,vol.44,no.10,pp.6056–6073,2022.[9]L.Zhao,H.Zhang,X.Li,S.Yang,and Y.Song,“You should know more:Learning external knowledge forvisual dialog,”Neurocomputing,vol.488,pp.54–65,2022.

[0171]

[10] S.Zhang,X.Jiang,Z.Yang,T.Wan,and Z.Qin,“Reasoning with multi-structure commonsense knowledge in visual dialog,”in CVPR Workshops,2022,pp.4599–4608.

[0172]

[11] R.Speer,J.Chin,and C.Havasi,“Conceptnet 5.5:An open multilingualgraph of general knowledge,”in AAAI,2017,pp.4444–4451.

[0173] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0174] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0175] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual dialogue generation method based on counterfactual common sense causal reasoning, characterized in that, The method includes: For the extracted common sense triples, a common sense subgraph is constructed, and an attention graph convolutional network is used to encode the subgraph to obtain common sense features; Common sense is introduced into the visual dialogue task, a visual dialogue causal graph based on common sense fusion is constructed, and the total effect of input features on answer prediction is calculated; based on the visual dialogue causal graph, its corresponding counterfactual causal graph is constructed. For counterfactual causal graphs, the harmful bias introduced by common sense on answer prediction is estimated based on the natural direct effect, and then removed from the total effect; A visual dialogue model is trained using model ensemble, with cross-entropy and KL divergence loss as training objectives to obtain answer prediction results. The system integrates image features, dialogue history features, common sense features, and current question features, feeds them into the decoder, minimizes the loss function, optimizes network parameters, and finally provides real disaster environment information for disaster relief sites. The specific steps for introducing common sense into the visual dialogue task and constructing a visual dialogue causal graph based on common sense fusion are as follows: Visual features are extracted from images using a visual encoder and then represented as nodes in a causal graph. I In a cause-effect graph, these are represented as nodes. Q and nodes H Common-sense characteristics are denoted as nodes in a cause-and-effect graph. C ; After that, subgraph ( ) represents the encoder input in the visual dialogue model. H, Q, I, C Output multimodal features K ; Problem characteristics Q With multimodal features K The input discriminative decoder or generative decoder produces the answer A to the current problem, and the corresponding subgraph is: and ; The visual dialogue model is trained using model ensemble, with cross-entropy and KL divergence loss as training objectives, to obtain the following answer prediction results: ； in, It is a non-linear activation function, sigmoid. Decomposed into , and Each is composed of a neural network , and calculate, Decomposed into and Two components, each generated by a neural network. and calculate.

2. The visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, The common sense triples are: common sense related to each visual dialogue unit extracted from the training set, validation set, and test set samples of the database; the specific operation of extracting the common sense triples is as follows: The Faster R-CNN framework is used to detect object labels, and the top few object labels with the highest confidence scores are selected based on the confidence scores of each label; similarly, the top few keywords with the highest scores are selected for image descriptions.

3. The visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, The common sense features obtained by encoding the subgraph using an attention graph convolutional network are as follows: common sense subgraphs are encoded using aGCN. Update current node features ; ； in, It is a learnable linear transformation matrix; It represents the number of convolutional layers in the aGCN network; It is the first Node characteristics With the Node characteristics Predefined weights between nodes; commonsense features are output after average pooling of all updated node features. .

4. The visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, The total effect of the calculated input features on answer prediction is: First, regarding the input variables H, Q, I, C Assign specific observed values ​​and null values ​​respectively, i.e. and ; Input variables H, Q, I, C When assigned specific observation values, the impact on the prediction of answer A is denoted as . The effect on the prediction of answer A when the input variable is assigned a null value is denoted as . The total effect TE is calculated as follows: 。 5. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, The counterfactual causal graph constructed based on the visual dialogue causal graph is as follows: By I, Q, H, C Each is assigned a null value, that is I =i Q = q H = h and C = c And thus K=k* , block I, Q, H and K Check the answers A The predicted impact; common sense C In a counterfactual world, two values ​​can be assigned simultaneously, that is... C = c and C = c The former is used to obtain K =k* The latter is related to the answer. A Direct connection to assess common sense C Predicting the answer A The natural direct effects.

6. The visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, The aforementioned method for counterfactual causal mapping estimates the harmful bias introduced by common sense on answer prediction based on the natural direct effects, and removes it from the total effect as follows: ； in, This indicates that the problem has been blocked. Q and multimodal knowledge K Input and retain only common sense C The subsequent impact on answer prediction This refers to the impact on answer prediction after all input variables of a visual dialogue model are assigned null values. The total indirect effect TIE is obtained as follows: .

7. A visual dialogue generation device based on counterfactual common sense causal reasoning, characterized in that, The device includes a processor and a memory, the memory storing program instructions, wherein the processor invokes the program instructions stored in the memory to cause the device to perform the method described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.

Images