Visual question answering method, medium and device based on causal guidance of visual attention representation

Through the combination of causal structure learning and attention mechanism, the visual object features are decomposed as the subject and support features, and cross-modal representation is generated, which solves the problem of insufficient visual feature representation in visual questions and answers, and improves the accuracy of the model's answers.

CN118840647BActive Publication Date: 2024-11-29HEFEI UNIV OF TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411310645.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-29
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

In visual question and answer tasks, standard attention mechanisms are difficult to effectively distinguish visual detection objects, resulting in insufficient quality of visual feature representation, affecting the accuracy of the model's answers, especially when there are a large number of overlapping objects.

Method used

The causal structure learning algorithm is used to construct a causal relationship diagram of visual objects. The subject features and support features are decomposed through the Markov discriminator, combined with the self-attention and cross-attention mechanism, generate a cross-modal representation of the visual subject and support objects, and through synergy The attention network layer and the fully connected layer perform feature fusion to generate the final multimodal feature representation.

Benefits of technology

The quality of cross-modal representation is improved, the accuracy of the answers of the visual question-and-answer model is improved, and the model performance is enhanced by appropriate use of background and false features, and the problem of insufficient quality of visual feature representation is solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118840647B_ABST
    Figure CN118840647B_ABST
Patent Text Reader

Abstract

The present invention discloses a visual question-answering method, medium and device based on causal guidance of visual attention representation, which relates to the field of artificial intelligence. The visual question-answering method based on causal guidance of visual attention representation mainly comprises the following steps: preprocessing data using Fast R-CNN and GloVe+LSTM, obtaining a visual object causal relationship graph using a causal structure learning algorithm, distinguishing a main feature causal relationship graph from a supporting feature causal relationship graph using a Markov discriminator, obtaining cross-modal representations of respective features using a self-attention mechanism and a cross-attention mechanism and a collaborative attention network composed of the same, fusing the cross-modal representations to obtain a final multi-modal feature representation, and finally obtaining the answer to the question using a trained classifier. The implementation of the visual question-answering method, medium and device based on causal guidance of visual attention representation provided by the present invention can improve the quality of cross-modal representation and the accuracy of answering questions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a method, medium and device for visual question answering based on causal guidance of visual attention representation. Background Art

[0002] With the introduction of the transformer architecture, multimodal data deep representation methods based on the attention mechanism have gradually matured and play an extremely important role in visual language processing tasks such as visual question answering (VQA). However, in the data representation process, over-reliance on the standard attention mechanism will lead to the risk of data misrepresentation. For example, in the VQA task, when there are a large number of overlapping visual detection objects, the representation weights learned by the visual self-attention mechanism are difficult to focus on related objects, resulting in the representation quality of visual features being insufficient to support the model to accurately answer questions.

[0003] Existing methods learn relevant features by removing background features or learning invariant features to simplify visual detection objects, that is, selecting the attention of some features for representation to alleviate the above-mentioned problems of the attention mechanism. However, this type of method is relatively conservative in feature selection. At the same time, due to the diversity and multi-source complexity of modal data, this type of method will sacrifice the model performance of other specific tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a visual question-answering method, medium and device based on causal-guided visual attention representation, which can improve the quality of cross-modal representation and improve the accuracy of answering questions.

[0005] The present invention provides a visual question-answering method based on causal guidance of visual attention representation, comprising the following steps: S1: pre-processing original image data and question text data to obtain a visual object feature vector and a question text feature vector; S2: obtaining a visual object causal relationship graph based on the visual object feature vector by using a causal structure learning algorithm; S3: obtaining a main feature causal relationship graph and a supporting feature causal relationship graph based on the visual object causal relationship graph by using a Markov discriminator; S4: obtaining a visual main object feature and a visual supporting object feature based on the main feature causal relationship graph, the supporting feature causal relationship graph and the visual object feature vector by using a self-attention mechanism; S5: obtaining a visual main object feature and a visual supporting object feature based on the visual main object feature, the visual supporting object feature and The question text feature vector uses the cross-attention mechanism to obtain the cross-modal representation of the visual main object feature and the cross-modal representation of the visual support object feature; S6: Based on the cross-modal representation of the visual main object feature, the cross-modal representation of the visual support object feature and the question text feature vector, the collaborative attention network layer and the fully connected layer are used to obtain the final cross-modal representation of the visual main feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text; S7: The final cross-modal representation of the visual main feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text are fused to obtain the final multimodal feature representation; S8: Based on the final multimodal feature representation, the trained classifier is used to obtain the answer to the question.

[0006] Furthermore, step S1 specifically includes: S11: obtaining original image data and question text data, using FastR-CNN to perform feature extraction on the original image data to obtain a visual object feature vector; S12: using GloVe encoding and long short-term memory network to perform feature extraction on the question text data to obtain a text feature vector.

[0007] Further, step S3 specifically includes: according to the visual object causal relationship graph, using a Markov discriminator, obtaining a main feature causal relationship graph and a supporting feature causal relationship graph, such as formula:

[0008] , ,

[0009] ,

[0010] ,

[0011] ,

[0012] ,

[0013] ,

[0014] ,

[0015] ,

[0016] ,

[0017] ,

[0018] ,

[0019] in, Represents the subject feature causal relationship diagram, represents the supporting feature causal relationship diagram, represents the feature vector of the visual object, The bootstrap matrix of the causal graph representing the subject feature set, represents the bootstrap matrix of the causal graph supporting the feature set, The first i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph supporting the feature set i Line j Column elements, represents the number of input features, The Markov blanket set representing the visual object is the MB set. Represents a collection of parent node objects. Represents a collection of child node objects. Represents the target object's spouse node object collection, Represents a collection of parent and child node objects. Represents a collection of child and spouse node objects, represents the adjacency matrix, Represents the adjacency matrix i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the parent node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the spouse node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of parent and child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child and spouse node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the MB set i Line j Column elements, The bootstrap matrix of the causal graph representing the set of parent node objects, The bootstrap matrix of the causal graph representing the set of child node objects, The bootstrap matrix of the causal graph representing the set of spouse node objects, The guidance matrix of the causal relationship graph representing the combination of parent and child node objects, The bootstrap matrix representing the causal graph for the collection of child and spouse node objects, The bootstrap matrix representing the causal graph of the MB set.

[0020] Further, step S4 specifically includes: according to the main feature causal relationship graph, the supporting feature causal relationship graph and the visual object feature vector, using the self-attention mechanism, obtaining the visual main object feature and the visual supporting object feature, such as the formula:

[0021] ,

[0022] ,

[0023] in, The input features represent the query, key and value matrices of the subject features respectively, The bootstrap matrix of the causal graph representing the subject feature set, Represents the characteristics of the visual subject object, represents the causality-guided self-attention mechanism, represents the normalized exponential function, represents the self-attention score matrix, represents the matrix dot product, denote the input features of the query, key and value matrices of the supporting features respectively, represents the bootstrap matrix of the causal graph supporting the feature set, Represents visual support object features.

[0024] Further, step S5 specifically includes: according to the visual subject object feature, the visual support object feature and the question text feature vector, using the cross attention mechanism, obtaining the visual subject object feature cross-modal representation and the visual support object feature cross-modal representation, such as formula:

[0025] ,

[0026] ,

[0027] ,

[0028] in, represents the feature vector of the visual object, represents the cross attention mechanism, represents the question text feature vector, represents the attention mechanism, represents the normalized exponential function, The input features of the query representing the visual object feature vector, Input features representing the keys of the question text feature vector, The input features are the matrix of values ​​representing the question text feature vector, are the lengths of the query, key, and value, The input features of the query represent the feature vector of the visual subject object, Represents the cross-modal representation of visual subject object features, The input features of the query representing the visual support object feature vector, Representing visually supported object features across modal representations.

[0029] Further, step S6 specifically includes: according to the cross-modal representation of the visual subject object feature, the cross-modal representation of the visual support object feature and the feature vector of the question text, using the collaborative attention network layer and the fully connected layer, obtaining the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text, such as the formula:

[0030] ,

[0031] ,

[0032] ,

[0033] ,

[0034] ,

[0035] ,

[0036] ,

[0037] in, Indicates l The cross-modal representation of deep visual subject object features at the layer, Indicates l Layer-by-layer deep visual subject object feature cross-modal representation, Indicates l The depth of the question text feature self-attention representation of the layer, Represents a deep causal-inspired collaborative attention network, Indicates l Layer i A cross-modal representation of deep visual subject object features, Indicates l Layer i Deep vision supports cross-modal representation of object features, Indicates l Layer i A deep question text feature self-attention representation, l represents the network depth, m Indicates the number of visual objects, Fully connected layer Weight, represents the normalized exponential function, represents the fully connected layer operation, Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text.

[0038] Further, step S7 specifically includes: fusing the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text to obtain a final multimodal feature representation, such as formula:

[0039] ,

[0040] ,

[0041] ,

[0042] in, represents the final multimodal feature representation, Represents the final multimodal feature representation of the subject feature, represents the final multimodal feature representation of the supporting features, are the linear projection matrices of the output features of the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text, respectively. is the common dimension of the fused features, It is a linear normalization process. Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text.

[0043] Furthermore, step S8 specifically includes:

[0044] S81: Obtain a question-answering training data set, construct a loss function, and train a classifier using the loss function and the question-answering training data set to obtain a trained classifier, such as formula:

[0045]

[0046]

[0047]

[0048]

[0049] in, To constrain the final multimodal feature representation of the subject feature The question answering classification loss is The final multimodal feature representation for constrained support features The question answering classification loss is To constrain the final multimodal feature representation The question answering classification loss is represents the binary cross entropy loss function, is the sigmoid function, It is a question that verifies the answer. The final question-answer classification loss is;

[0050] S82: Obtaining the answer to the question based on the final multimodal feature representation and using the trained classifier.

[0051] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned visual question-answering method based on causal guidance of visual attention representation.

[0052] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned visual question-answering method based on causal guidance of visual attention representation are implemented.

[0053] The implementation of the visual question-answering method, medium and device based on causal guidance of visual attention representation provided by the present invention has the following beneficial effects:

[0054] The present invention is based on the discovery that the performance of the model can be improved by properly using background and false features. The present invention divides visual object features into Influential Object Features (IOF) and Supportive Object Features (SOF) to improve the quality of cross-modal representation. Among them, IOF can represent object features that are related to or important to the target object features, and SOF can represent background or false object features that are complementary to the target object features. The present invention uses a causal relationship graph to model the dependency or independence between objects. At the same time, in terms of dependency, the present invention uses attention scores to represent the degree of dependency between objects without the guidance of additional entity feature relationship graphs. Finally, the present invention guides visual features into complementary dual-stream feature representations during cross-modal learning, thereby solving the problem of insufficient representation quality of visual features in existing methods, such as the single visual feature representation of the existing attention-based transformer architecture visual question answering model during cross-modal learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0056] Figure 1 is a flow chart of a visual question answering method based on causal guidance of visual attention representation provided by the present invention;

[0057] Figure 2 It is a schematic diagram of the standard attention mechanism, the graph-guided attention mechanism and the causal relationship-guided attention mechanism provided by the present invention;

[0058] Among them, (a) is a schematic diagram of the standard attention mechanism, (b) is a schematic diagram of the graph-guided attention mechanism, and (c) is a schematic diagram of the attention mechanism guided by causality;

[0059] Figure 3 It is a schematic diagram of a single visual feature representation and a complementary dual-stream feature representation provided by the present invention;

[0060] Figure 4 is an example of a cause-effect relationship diagram provided by the present invention;

[0061] Figure 5is a schematic diagram of a cross-modal visual representation method based on causal guidance provided by the present invention;

[0062] Figure 6 It is a structural block diagram of the computer device provided by the present invention. DETAILED DESCRIPTION

[0063] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0064] Figure 1 A schematic diagram of a visual question answering method based on causal guidance of visual attention representation in this embodiment is shown. In this embodiment, the visual question answering method based on causal guidance of visual attention representation includes the following steps:

[0065] S1: Preprocess the original image data and question text data to obtain visual object feature vectors and question text feature vectors;

[0066] Specifically, the above step S1 includes: S11: obtaining original image data and question text data, and using Fast R-CNN to perform feature extraction on the original image data to obtain a visual object feature vector; S12: using GloVe encoding and long short-term memory network to perform feature extraction on the question text data to obtain a text feature vector;

[0067] S2: obtaining a visual object causal relationship graph using a causal structure learning algorithm according to the visual object feature vector;

[0068] S3: According to the visual object causal relationship graph, using a Markov discriminator, a main feature causal relationship graph and a supporting feature causal relationship graph are obtained;

[0069] Specifically, the above step S3 includes: according to the visual object causal relationship graph, using the Markov discriminator, obtaining the main feature causal relationship graph and the supporting feature causal relationship graph, such as the formula:

[0070] , ,

[0071] ,

[0072] ,

[0073] ,

[0074] ,

[0075] ,

[0076] ,

[0077] ,

[0078] ,

[0079] ,

[0080] ,

[0081] in, Represents the subject feature causal relationship diagram, represents the supporting feature causal relationship diagram, represents the feature vector of the visual object, The bootstrap matrix of the causal graph representing the subject feature set, represents the bootstrap matrix of the causal graph supporting the feature set, The first i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph supporting the feature set i Line j Column elements, represents the number of input features, The Markov blanket set representing the visual object is the MB set. Represents a collection of parent node objects. Represents a collection of child node objects. Represents the target object's spouse node object collection, Represents a collection of parent and child node objects. Represents a collection of child and spouse node objects, represents the adjacency matrix, Represents the adjacency matrix i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the parent node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the spouse node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of parent and child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child and spouse node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the MB set i Line j Column elements, The bootstrap matrix of the causal graph representing the set of parent node objects, The bootstrap matrix of the causal graph representing the set of child node objects, The bootstrap matrix of the causal graph representing the set of spouse node objects, The guidance matrix of the causal relationship graph representing the combination of parent and child node objects, The bootstrap matrix representing the causal graph for the collection of child and spouse node objects, The bootstrap matrix representing the causal graph of the MB set;

[0082] S4: according to the main feature causal relationship graph, the supporting feature causal relationship graph and the visual object feature vector, using a self-attention mechanism, obtaining a visual main object feature and a visual supporting object feature;

[0083] Specifically, the above step S4 includes: according to the main feature causal relationship graph, the supporting feature causal relationship graph and the visual object feature vector, using the self-attention mechanism, obtaining the visual main object feature and the visual supporting object feature, such as the formula:

[0084] ,

[0085] ,

[0086] in, The input features represent the query, key and value matrices of the subject features respectively, The bootstrap matrix of the causal graph representing the subject feature set, Represents the characteristics of the visual subject object, represents the causality-guided self-attention mechanism, represents the normalized exponential function, represents the self-attention score matrix, represents the matrix dot product, denote the input features of the query, key and value matrices of the supporting features respectively, represents the bootstrap matrix of the causal graph supporting the feature set, Indicates visual support object features;

[0087] S5: According to the visual subject object feature, the visual support object feature and the question text feature vector, a cross-attention mechanism is used to obtain a cross-modal representation of the visual subject object feature and a cross-modal representation of the visual support object feature;

[0088] Specifically, the above step S5 includes: according to the visual main object feature, the visual support object feature and the question text feature vector, using the cross attention mechanism, obtaining the cross-modal representation of the visual main object feature and the cross-modal representation of the visual support object feature, such as formula:

[0089] ,

[0090] ,

[0091] ,

[0092] in, represents the feature vector of the visual object, represents the cross attention mechanism, represents the question text feature vector, represents the attention mechanism, represents the normalized exponential function, The input features of the query representing the visual object feature vector, Input features representing the keys of the question text feature vector, The input features are the matrix of values ​​representing the question text feature vector, are the lengths of the query, key, and value, The input features of the query represent the feature vector of the visual subject object, Represents the cross-modal representation of visual subject object features, The input features of the query representing the visual support object feature vector, Representing cross-modal representation of visually supported object features;

[0093] S6: According to the cross-modal representation of the visual subject object feature, the cross-modal representation of the visual support object feature and the feature vector of the question text, using a collaborative attention network layer and a fully connected layer, obtain a final cross-modal representation of the visual subject feature, a final cross-modal representation of the visual support feature and a final attention feature representation of the question text;

[0094] Specifically, the above step S6 includes: according to the cross-modal representation of the visual subject object feature, the cross-modal representation of the visual support object feature and the feature vector of the question text, using the collaborative attention network layer and the fully connected layer, obtaining the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text, as shown in the formula:

[0095] ,

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] ,

[0101] ,

[0102] in, Indicates l The cross-modal representation of deep visual subject object features at the layer, Indicates l Layer-by-layer deep visual subject object feature cross-modal representation, Indicates l The depth of the question text feature self-attention representation of the layer, Represents a deep causal-inspired collaborative attention network, Indicates l Layer i A cross-modal representation of deep visual subject object features, Indicates l Layer i Deep vision supports cross-modal representation of object features, Indicates l Layer i A deep question text feature self-attention representation, l represents the network depth, m Indicates the number of visual objects, Fully connected layer Weight, represents the normalized exponential function, represents the fully connected layer operation, Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text;

[0103] S7: fusing the final cross-modal representation of the visual main feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text to obtain a final multimodal feature representation;

[0104] Specifically, the above step S7 includes: fusing the final cross-modal representation of the visual main feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text to obtain a final multimodal feature representation, such as formula:

[0105] ,

[0106] ,

[0107] ,

[0108] in, represents the final multimodal feature representation, Represents the final multimodal feature representation of the subject feature, represents the final multimodal feature representation of the supporting features, are the linear projection matrices of the output features of the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text, respectively. is the common dimension of the fused features, It is a linear normalization process. Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text;

[0109] S8: Obtaining an answer to the question using the trained classifier according to the final multimodal feature representation;

[0110] Specifically, the above step S8 includes: S81: obtaining a question-answering training data set, constructing a loss function, and using the loss function and the question-answering training data set to train a classifier to obtain a trained classifier, such as formula:

[0111]

[0112]

[0113]

[0114]

[0115] in, To constrain the final multimodal feature representation of the subject feature The question answering classification loss is The final multimodal feature representation for constrained support features The question answering classification loss is To constrain the final multimodal feature representation The question answering classification loss is represents the binary cross entropy loss function, is the sigmoid function, It is a question that verifies the answer. The final question-answer classification loss is: S82: according to the final multimodal feature representation, using the trained classifier, obtaining the answer to the question;

[0116] In some embodiments, the above-mentioned visual question answering method based on causal guidance of visual attention representation can also be implemented in the following way.

[0117] Generally, the self-attention mechanism is used to learn the modal feature representation of visual object data; Figure 2 As shown in (a) of , assuming that the input features with query, key and value matrices are , in is the number of input features, is the length of query, key and value. Then the attention score of the standard attention mechanism is:

[0118]

[0119] Then, the modal features can be expressed as follows:

[0120]

[0121] If there is an adjacency matrix of the relationship graph corresponding to the object features ,like Figure 2 As shown in (b) in the present embodiment, the adjacency matrix of the relationship graph can be As a guide matrix, the attention mechanism is guided to learn the modal feature representation, namely the graph guided attention mechanism (GGA). In general, the graph guided visual feature representation can be expressed as follows:

[0122]

[0123] in is a renormalization operation. The graph-guided attention mechanism is widely used in VQA. For example, in the GQA task, it guides the attention mechanism to learn spatial feature representations through scene graphs.

[0124] Whether it is a scene graph or a semantic graph, both require entity features to define the relationship between visual objects. Unlike guided methods that require additional entity feature graphs, causal graphs model the dependency or independence between objects. Since there is no entity on the corresponding edge, the weight (causal effect) of the causal edge cannot be directly calculated. At the same time, in terms of dependency, the attention score can represent the degree of dependency between objects.

[0125] Based on this, this embodiment designs an attention mechanism guided by causality, using the attention score guided by the causal relationship graph to replace the causal effect between calculation objects.

[0126] like Figure 2 As shown in (c) in , the causal attention score is:

[0127]

[0128] in is the guidance matrix of the IOF and SOF causal graphs. Therefore, this embodiment reconstructs the self-attention module into a causality guided self-attention (CGS) module for causal attention representation learning. In order to obtain a more detailed representation, this embodiment extends CGS to a multi-head self-attention mechanism to learn modal representations, and then the attention features calculated by CGS are represented as follows:

[0129]

[0130] because Built-in normalization, Omitted. The guidance matrix based on different causal relationship graphs can form the causal relationship graph representation of IOF and SOF and , the CGS module can represent visual objects as main object features and support object features .

[0131] like Figure 3 As shown, unlike the existing transformer architecture visual question answering model based on the attention mechanism, which uses a single visual feature representation during cross-modal learning, this embodiment guides the visual features into a complementary dual-stream feature representation during cross-modal learning to solve the problem of insufficient representation quality of visual features in existing methods.

[0132] Combined with recent research, it is found that the appropriate use of background and false features can improve the performance of the model. Based on the above considerations, this embodiment improves the quality of cross-modal representation by dividing visual object features into Influential Object Features (IOF) and Supportive Object Features (SOF). Among them, IOF represents object features that are related to or important to the target object features; SOF represents background or false object features that are complementary to the target object features.

[0133] In this embodiment, Figure 5 As shown in FIG, the cross-modal visual representation method based on causal guidance includes the following steps:

[0134] Step 1: Preprocessing of visual and text data: The input text information is first segmented into words as tags (no more than 14 words per sentence). Each word is encoded using 300-D GloVe to generate a word embedding vector representation, so the text sentence can be formed into a A sequence of is the number of words. Then, all the words are passed through an LSTM network to obtain As text features, is the length of each problem feature. The input visual image is the regional object feature extracted by Faster R-CNN on the visual dataset. The objects are represented as , the input image can be represented as a visual feature matrix ,in is the number of visual object features, is the representation length of each visual object feature. This embodiment uses the maximum zero padding and masking method to solve the variable problem of text length and number of visual objects.

[0135] Step 2: Causal structure learning: In order to learn the influencing object features (IOFs) and supporting object features (SOFs) of the target object from the visual object, this embodiment introduces the causal relationship theory to model the relationship between object features. The structure of the causal graph is represented by a directed acyclic graph (DAG). In the DAG, nodes represent variables and directed edges represent the dependencies between variables. This embodiment uses the causal structure learning algorithm, the PC algorithm, to learn the global causal structure of the visual object and generate a causal graph.

[0136] Step 3: Markov discriminator of IOF and SOF: In the learned causal structure, this embodiment regards the core subset of the visual object MB set as IOF and the edge subset as SOF. The MB object set set for the target object includes a parent node object set P, a child node object set C, and a spouse node object set S of the target object, where P is the parent set of the target object, which is the direct cause object set. The P set represents the object set most related to the target object, which is the innermost hierarchical structure of the related object set. Although the causal relationship is unidirectional, from the perspective of correlation, the C set as the direct result object set of the target object still has a certain correlation and is the middle hierarchical structure of the related object set. The set S composed of other parent node objects of the C set of the target object except the P set is called the spouse node object set of the target object. Although the S set and the target object are conditionally independent with respect to the C set, when the target object and the set C coexist, the subset S still has a slight correlation with the target object, so the S set is the outermost hierarchical structure of the related object set.

[0137] For example, Figure 4 As shown, for the target variable Parent nodes are marked in green, child nodes are marked in orange, and spouse nodes are marked in red. Specifically, an edge Represented in DAG yes The parent node (or immediate cause) or yes Child nodes (or direct influences) of a node. About the structure , child node The parent node of yes Therefore, the IOF defined based on MB theory can be divided into the following hierarchical structures from inside to outside: , (a collection of parent and child node objects) and .

[0138] The graph representation of the visual object causal relationship in this embodiment is composed of visual object features. and the adjacency matrix Composition, where the adjacency matrix Represents the causal relationship between object features and is generated by the PC algorithm. It only represents the causal dependency between objects, so calculating the attention weight requires the introduction of a unitary matrix to ensure self-attention. The guidance matrix of the causal relationship graph of the MB set and MB subset The bootstrap matrix of the causal relationship diagram In the adjacency matrix So the set The bootstrap matrix of the causal relationship diagram It can be expressed as:

[0139]

[0140] gather The bootstrap matrix of the causal relationship diagram for The transpose of :

[0141]

[0142] Then the collection The bootstrap matrix of the causal relationship diagram for:

[0143]

[0144] based on , parent-child node object collection The bootstrap matrix of the causal relationship diagram It can be expressed as:

[0145]

[0146] Bootstrap Matrix for Causal Graph of Collections for:

[0147] That is, it is the guidance matrix of all possible IOF set node object causal relationship graphs based on MB theory.

[0148] For the SOF set, this embodiment sets different coarse and fine granularity ranges for the objects therein. Since in MB theory, the target object is related to the S set node only when the S set and the C set coexist, this means that when IOFs is set to P, the fine-grained set of SOF is C or C∪S. On the other hand, sof contains all object features except IOF at the coarse granularity. Therefore, the guidance matrix of the fine-grained SOF set node object causal relationship graph is: If there is a bootstrap matrix of the causal relationship graph of the IOF set: , then there is a guidance matrix of the coarse-grained SOF set node object causal relationship graph:

[0149]

[0150] Visual Features and pre-built MB-based visuals for cause-effect relationships Forming a causal graph representation of IOF , SOF is expressed as .

[0151] Step 4: Causal Attention Representation Learning: Causal Graph Representation Given IOF and the causal graph representation of SOF , through the visual feature representation of IOF in formula (5) Can be Characteristic , Learn as follows:

[0152]

[0153] Then the visual feature representation of SOF is :

[0154]

[0155] Step 5: Cross-modal representation learning: Representing visual features , Respectively and question text feature representation Enter the criss-cross attention mechanism:

[0156]

[0157] This allows us to learn cross-modal representations of visual features. The specific process is as follows:

[0158]

[0159]

[0160] Step 6: Deep cross-modal learning - causally guided collaborative attention network: Figure 3 As shown, steps 4 and 5 are combined into a collaborative attention network layer, and the network parameters are deeply iterated to optimize the quality of feature representation, thereby learning deep cross-modal feature representation of visual object subjects and cross-modal feature representation of visual object support. The specific process is as follows:

[0161]

[0162] So far, we can get and Then, a simple fully connected layer is used to approximate the dimension and learn the final attention feature representation .by For example, the final representation can be learned as follows:

[0163]

[0164]

[0165] in For full connection Weight, and Different from the single output visual modality, the model output of this embodiment is , The dual visual features are used to represent IOF and SOF.

[0166] Step 7: Multimodal feature fusion: Represent the cross-modal features of the visual object separately and visual objects support cross-modal feature representation and text representation features Fusion forms a dual-output multimodal feature representation and further fuses it into a final multimodal feature representation. In order to make full use of multimodal information, this embodiment designs a linear multimodal fusion function, and the specific process is as follows:

[0167]

[0168]

[0169] in, is the linear projection matrix of each output feature. It is the common dimension of the fused features. is a linear normalization process. Therefore, the final multimodal fusion feature for:

[0170]

[0171] Step 8: Train the classifier and answer the question: This embodiment is to give full play to the complementary role of IOF and SOF, while constraining and The question answering classification loss is used to optimize and reduce The loss of question-answering classification is calculated using Binary Cross-Entropy (BCE) loss:

[0172]

[0173]

[0174]

[0175] in is the sigmoid function, used for vector classification mapping, is the answer to the question verification. Then the final question-answering classification loss is:

[0176]

[0177] Train the question-answering classifier according to the above loss, fix the learning parameters of the deep model composed of each step, and answer visual questions.

[0178] Optionally, the question-answering classifier designed in step eight adjusts the loss function appropriately according to the classification requirements of different VQA data. For example, if a single classification task needs to be performed in VQA2.0, the binary cross entropy (BCE) loss function is used to design the loss function; for multi-classification question-answering tasks such as GQA, the loss function needs to be changed to the cross entropy design function.

[0179] In order to verify the effectiveness of the method (CGCN) of the present invention, this embodiment uses three public datasets commonly used in visual question answering: VQA 2.0, GQA, and GQA-OOD; each dataset uses question answering accuracy as an evaluation indicator, where Yes / No, Num, and Other in the experimental comparison results of the VQA 2.0 dataset represent the accuracy of three different types of question answers, and test-dev and test-std represent different test set divisions; GQA-OOD is an extension of the GQA dataset, which is specially designed to test the generalization ability of the model when facing scenarios and out-of-distribution questions. Acc-All, Acc-Tail, and Acc-Head represent the comprehensive question answering accuracy of the data test set (Acc-All), the question answering accuracy of the tail sample split data test set of rare distribution (Acc-Tail), and the question answering accuracy of the head sample split data test set of common distribution (Acc-Head).

[0180] Table 1: Comparison results of the proposed method and the comparative method on the VQA 2.0 dataset

[0181]

[0182] Table 2: Comparison results of the proposed method and the comparative method on the GQA and GQA-OOD datasets

[0183]

[0184] As shown in Table 1 and Table 2, the comparative experimental results on the VQA 2.0, GQA, and GQA-OOD datasets are shown respectively. It can be seen that on the three datasets, the method (CGCN) proposed in this embodiment is significantly better than many of the compared methods, thereby proving the feasibility of the method proposed in the present invention.

[0185] This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned visual question-answering method based on causal guidance of visual attention representation are implemented. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0186] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned visual question-answering method based on causal guidance of visual attention representation are implemented.

[0187] like Figure 6 As shown, the computer device may include: at least one processor 121, such as a CPU (Central Processing Unit), at least one communication interface 123, a memory 124, and at least one communication bus 122. Among them, the communication bus 122 is used to realize the connection and communication between these components. Among them, the communication interface 123 may include a display screen (Display), a keyboard (Keyboard), and the optional communication interface 123 may also include a standard wired interface and a wireless interface. The memory 124 may be a high-speed RAM memory (Random Access Memory, volatile random access memory), or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 124 may also be at least one storage device located away from the aforementioned processor 121. Among them, the memory 124 stores an application program, and the processor 121 calls the program code stored in the memory 124 to perform any of the above method steps. Among them, the communication bus 122 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 122 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6Only one line is used to represent it, but it does not mean that there is only one bus or one type of bus. Among them, the memory 124 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 124 may also include a combination of the above-mentioned types of memory. Among them, the processor 121 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP. Among them, the processor 121 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (English: application-specific integrated circuit, abbreviated: ASIC), a programmable logic device (English: programmable logic device, abbreviated: PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Optionally, the memory 124 is also used to store program instructions. The processor 121 may call the program instructions to implement the visual question-answering method based on causal guidance of visual attention representation as in the present embodiment.

[0188] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A visual question answering method based on causal guidance of visual attention representation, characterized in that: The following steps are involved: S1: Preprocess the original image data and question text data to obtain visual object feature vectors and question text feature vectors; S2: obtaining a visual object causal relationship graph using a causal structure learning algorithm according to the visual object feature vector; S3: According to the visual object causal relationship graph, using a Markov discriminator, a main feature causal relationship graph and a supporting feature causal relationship graph are obtained; S4: according to the main feature causal relationship graph, the supporting feature causal relationship graph and the visual object feature vector, using a self-attention mechanism, obtaining a visual main object feature and a visual supporting object feature; S5: According to the visual subject object feature, the visual support object feature and the question text feature vector, a cross-attention mechanism is used to obtain a cross-modal representation of the visual subject object feature and a cross-modal representation of the visual support object feature; S6: According to the cross-modal representation of the visual subject object feature, the cross-modal representation of the visual support object feature and the feature vector of the question text, using a collaborative attention network layer and a fully connected layer, obtain a final cross-modal representation of the visual subject feature, a final cross-modal representation of the visual support feature and a final attention feature representation of the question text; S7: fusing the final cross-modal representation of the visual main feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text to obtain a final multimodal feature representation; S8: Obtaining the answer to the question based on the final multimodal feature representation using the trained classifier.

2. The visual question answering method based on causal guidance of visual attention representation according to claim 1 is characterized in that: Step S1 specifically includes: S11: Obtain original image data and question text data, and use Fast R-CNN to perform feature extraction on the original image data to obtain a visual object feature vector; S12: Use GloVe encoding and long short-term memory network to extract features of the question text data to obtain a text feature vector.

3. The visual question answering method based on causal guidance of visual attention representation according to claim 1 is characterized in that: Step S3 specifically includes: according to the visual object causal relationship graph, using the Markov discriminator, obtaining the main feature causal relationship graph and the supporting feature causal relationship graph, such as the formula: , , , , , , , , , , , , in, Represents the subject feature causal relationship diagram, represents the supporting feature causal relationship diagram, represents the feature vector of the visual object, The bootstrap matrix of the causal graph representing the subject feature set, represents the bootstrap matrix of the causal graph supporting the feature set, The first i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph supporting the feature set i Line j Column elements, represents the number of input features, The Markov blanket set representing the visual object is the MB set. Represents a collection of parent node objects. Represents a collection of child node objects. Represents the target object's spouse node object collection, Represents a collection of parent and child node objects. Represents a collection of child and spouse node objects, represents the adjacency matrix, Represents the adjacency matrix i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the parent node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the spouse node object set i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of parent and child node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the set of child and spouse node objects i Line j Column elements, The first part of the bootstrap matrix representing the causal relationship graph of the MB set i Line j Column elements, The bootstrap matrix of the causal graph representing the set of parent node objects, The bootstrap matrix of the causal graph representing the set of child node objects, The bootstrap matrix of the causal graph representing the set of spouse node objects, The guidance matrix of the causal relationship graph representing the combination of parent and child node objects, The bootstrap matrix representing the causal graph for the collection of child and spouse node objects, The bootstrap matrix representing the causal graph of the MB set.

4. The visual question answering method based on causal guidance of visual attention representation according to claim 1, characterized in that: Step S4 specifically includes: according to the main feature causal relationship diagram, the supporting feature causal relationship diagram and the visual object feature vector, using the self-attention mechanism, obtaining the visual main object feature and the visual supporting object feature, such as the formula: , , in, The input features represent the query, key and value matrices of the subject features respectively, The bootstrap matrix of the causal graph representing the subject feature set, Represents the characteristics of the visual subject object, represents the causality-guided self-attention mechanism, represents the normalized exponential function, represents the self-attention score matrix, represents the matrix dot product, denote the input features of the query, key and value matrices of the supporting features respectively, represents the bootstrap matrix of the causal graph supporting the feature set, Represents visual support object features.

5. The visual question answering method based on causal guidance of visual attention representation according to claim 1, characterized in that: Step S5 specifically includes: according to the visual subject object feature, the visual support object feature and the question text feature vector, using the cross attention mechanism, obtaining the visual subject object feature cross-modal representation and the visual support object feature cross-modal representation, such as formula: , , , in, represents the feature vector of the visual object, represents the cross attention mechanism, represents the question text feature vector, represents the attention mechanism, represents the normalized exponential function, The input features of the query representing the visual object feature vector, Input features representing the keys of the question text feature vector, The input features are the matrix of values ​​representing the question text feature vector, are the lengths of the query, key, and value, The input features of the query represent the feature vector of the visual subject object, Represents the cross-modal representation of visual subject object features, The input features of the query representing the visual support object feature vector, Representing visually supported object features across modal representations.

6. The visual question answering method based on causal guidance of visual attention representation according to claim 1, characterized in that: Step S6 specifically includes: according to the cross-modal representation of the visual subject object feature, the cross-modal representation of the visual support object feature and the feature vector of the question text, using the collaborative attention network layer and the fully connected layer, obtaining the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature and the final attention feature representation of the question text, as shown in the formula: , , , , , , , in, Indicates l The cross-modal representation of deep visual subject object features at the layer, Indicates l Layer-by-layer deep visual subject object feature cross-modal representation, Indicates l The depth of the question text feature self-attention representation of the layer, Represents a deep causal-inspired collaborative attention network, Indicates l Layer i A cross-modal representation of deep visual subject object features, Indicates l Layer i Deep vision supports cross-modal representation of object features, Indicates l Layer i A deep question text feature self-attention representation, l represents the network depth, m Indicates the number of visual objects, Fully connected layer Weight, represents the normalized exponential function, represents the fully connected layer operation, Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text.

7. The visual question answering method based on causal guidance of visual attention representation according to claim 1 is characterized in that: Step S7 specifically includes: fusing the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text to obtain a final multimodal feature representation, such as formula: , , , in, represents the final multimodal feature representation, Represents the final multimodal feature representation of the subject feature, represents the final multimodal feature representation of the supporting features, are the linear projection matrices of the output features of the final cross-modal representation of the visual subject feature, the final cross-modal representation of the visual support feature, and the final attention feature representation of the question text, respectively. is the common dimension of the fused features, It is a linear normalization process. Represents the final cross-modal representation of the visual subject features, represents the final cross-modal representation of the visual support features, Represents the final attention feature representation of the question text.

8. The visual question answering method based on causal guidance of visual attention representation according to claim 1, characterized in that: Step S8 specifically includes: S81: Obtain a question-answering training data set, construct a loss function, and train a classifier using the loss function and the question-answering training data set to obtain a trained classifier, such as formula: in, To constrain the final multimodal feature representation of the subject feature The question answering classification loss is The final multimodal feature representation for the constraint support feature The question answering classification loss is To constrain the final multimodal feature representation The question answering classification loss is represents the binary cross entropy loss function, is the sigmoid function, It is a question that verifies the answer. The final question-answer classification loss is; S82: Obtaining the answer to the question based on the final multimodal feature representation and using the trained classifier.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of any one of the visual question answering methods based on causal guided visual attention representation as described in claims 1-8 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the visual question answering methods based on causal guidance of visual attention representation as described in claims 1-8 are implemented.

Citation Information

Patent Citations

  • Unmanned aerial vehicle visual reasoning and question answering method in near-ground security rescue scene

    CN118298335A

  • Continuous visual language navigation method based on causal reasoning and cognitive thinking

    CN118482720A

  • Image recognition reverse tuning test system

    US20200410082A1