A visual question answering system and method based on scene graph neural network reasoning mechanism

Abstract

This invention discloses a visual question answering system and method based on a scene graph neural network inference mechanism, including a question embedding module, a scene graph extraction module, a graph neural network attention inference module, and an answer prediction module. The question embedding module acquires text question features, which are then segmented into words using a tokenizer module and feature extracted using a Transformer network. The scene graph extraction module acquires image visual features and embeds these features into a scene graph containing rich image information. The graph neural network attention inference module combines scene graph features and question features to introduce a collaborative attention inference mechanism for iterative inference, simultaneously focusing on deep textual and visual information. The answer prediction module acquires the final iteration state of the graph neural network attention inference network and inputs it, along with global textual feature information, into a classifier for answer prediction. This invention significantly improves performance compared to traditional visual question answering models.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and natural language processing, and to a visual question answering system and method based on a scene graph neural network reasoning mechanism. Background Technology

[0002] Visual Question Answering (VQA) is a machine intelligence task proposed by the computer science community in recent years. Its task is to answer natural language questions posed by humans based on the content of a given image, as shown in the attached image. Figure 1 As shown, visual question answering involves inputs of multimodal information (i.e., visual images and natural language descriptions), requiring a more refined understanding of image content. The key lies in the joint semantic understanding of vision and natural language, and joint reasoning between images and language. Typical visual question answering tasks involve techniques such as fine-grained recognition, object recognition, and understanding the text contained in the question. A typical visual question answering model consists of three modules: a feature extraction module that uses convolutional neural networks to extract features from the image and recurrent neural networks to extract features from the question; a feature fusion module that fuses the question features and image features into an abstract feature that can represent the current task; and an answer classification module that takes the fused feature as input and classifies it, with the number of categories determined by the number of candidate answers. However, current visual question answering technologies often involve a simple fusion of visual and linguistic modal information, neglecting other rich information, including the semantic context of sentences, spatial relationships within the image, and internal connections between multiple entities. This leads to the answer classification module misinterpreting the language question and insufficiently recognizing image features, resulting in incorrect answer predictions. Another problem with current visual question answering methods is that the machine-provided answers lack interpretability: due to the black-box design of deep learning models, it is often difficult to provide the reasons and causes for the machine's answers. This situation greatly limits the promotion and use of intelligent visual question answering in the industrial field. Summary of the Invention

[0003] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a visual question answering system and method based on scene graph neural network reasoning mechanism, which gives full attention to information such as the semantics before and after the statement, the spatial position relationship in the image, and the inner connection between multiple entities. At the same time, a collaborative attention mechanism is introduced to enhance the model's reasoning ability to deal with complex language problems.

[0004] To address the aforementioned technical problems, the present invention provides a visual question answering system based on a scene graph neural network reasoning mechanism, comprising a question embedding module, a scene graph extraction module, a graph neural network attention reasoning module, and an answer prediction module;

[0005] In the question embedding module, the Tokenizer module segments the input natural language question to generate individual word segments, and retains only the individual word segments. Then, the multi-head attention mechanism in the Transformer concept is used to embed the word segments into a sequence of instruction vectors and a global feature vector of the question.

[0006] The scene graph extraction module first uses a pre-trained Faster-RCNN network to perform entity recognition on the input image, obtains the initial target feature vector of each target entity, and inputs the initial target feature vector into the SGG network to generate the initial state scene graph.

[0007] The graph neural network attention reasoning module performs graph neural network collaborative attention reasoning calculation based on the instruction vector sequence obtained by the question embedding module and the initial state scene graph obtained by the scene graph extraction module, and obtains the final state of all graph nodes after a set number of iterations.

[0008] The answer prediction module uses the Aggregate function to summarize the final state of the graph nodes to obtain the final state after message passing. The linear classifier predicts the answer based on the final state of message passing and the global feature vector of the question.

[0009] The present invention also includes a visual question-answering method employing the above-described system, comprising:

[0010] Step 1: Use the Tokenizer module to segment the input natural language question into individual tokens, retaining only the individual tokens. Then, use the multi-head attention mechanism from the Transformer concept to embed the tokens into a sequence of instruction vectors [g (1) ,g (2) ,g (3) ,…,g (m) [Q] and the global feature vector of the problem, where m represents the number of word segments;

[0011] Step 2: Use a pre-trained Faster-RCNN network to perform entity recognition on the input image to obtain the initial target feature vector X for each target entity. i ={v1,v2,…,v n}, where n is the number of target objects, and the initial target feature vector is input into the SGG network to generate the initial state scene graph [h1,h2,…,h k ], h i Let represent the i-th graph node corresponding to the i-th target entity in the initial state scene graph, and k represent the number of target entities;

[0012] Step 3: Process the instruction vector sequence [g] (1) ,g (2) ,g (3) ,…,g(m) ] and the initial state scene graph [h1,h2,…,h k Perform collaborative attention inference computation using a graph neural network, and obtain the final state of all graph nodes through iterative computation. m represents the number of layers in the iterative network;

[0013] Step 4: Use the Aggregate function to summarize the final state of the graph nodes. After obtaining the final state H after message passing, the linear classifier predicts the answer y based on the final state H and the global feature vector Q of the problem.

[0014] Furthermore, the initial state scene diagram [h1,h2,…,h] k Integrating spatial and semantic relationships, specifically:

[0015]

[0016] h i =[h′ i g (i) ]

[0017] Among them, h' i h represents the node feature representation of the i-th entity. i This indicates that the graph nodes are combined with the i-th problem guidance vector g from step one. (i) Preparation before reasoning, W t1 W t2 W t3 Let N be the linear transformation weight matrix, σ be the ReLU activation function, and N be the weight matrix. i Including all nodes except the i-th node itself, α ij This is an attention mechanism between nodes.

[0018] Furthermore, the final state of all graph nodes The i-th element satisfies:

[0019]

[0020] Where σ is the ReLU activation function, K represents the number of target features, and N i Including all nodes except the i-th node itself. For the attention mechanism a of the kth node k The normalized attention coefficient, W, is calculated. k The weight matrix is ​​the corresponding input linear transformation. This represents the j-th feature node of a single layer's attention layer.

[0021] Furthermore, the answer y described in step 4 satisfies:

[0022] y = Softmax(MLP(H,Q))

[0023] Wherein, the final state H satisfies:

[0024]

[0025] The beneficial effects of this invention: To achieve a deeper understanding of image and language information through collaborative attention learning of multimodal information in a scene, this invention proposes a visual question answering system and method based on a scene graph neural network reasoning mechanism. This method primarily involves extracting scene graphs from real images using graph neural networks and combining them with natural language questions for reasoning and question answering. The method generates scene graphs from images and uses a collaborative attention mechanism to reason with extracted question features to predict the answer to the input question. The model first represents the input image as a scene graph composed of multiple triples (initial state of the graph neural network), with a subject-object relationship. Specifically, an object is represented as a node with a category label, and the relationship between nodes is characterized by a directed edge with a specific predicate category between them, the direction of which specifies the subject and object in the triple. Then, the natural language question is transformed into multiple iterations of message passing between scene graph nodes. The final iteration state after message passing (final state of the graph neural network) represents the answer, while intermediate states reflect the reasoning process. Each intermediate message passing iteration state is completed by a graph neural network (GNN) layer. Through training and testing on the benchmark dataset VQA-v2, the model of this invention outperforms other models, verifying the effectiveness of generating scene graphs from images and performing collaborative attention reasoning on graph neural networks in conjunction with questions. Attached Figure Description

[0026] Figure 1 This is an example of the visual question answering task process;

[0027] Figure 2 It refers to the spatial and semantic relationships between entities in the scene graph;

[0028] Figure 3 This invention proposes a visual question-answering model framework based on a scene graph neural network reasoning mechanism.

[0029] Figure 4 It is the multi-head attention mechanism of the Transformer concept;

[0030] Figure 5 It is the attention iterative mechanism of graph neural networks;

[0031] Figure 6 It is a multi-head attention mechanism joint reasoning network;

[0032] Figure 7It is the bounding box of the entity region selected during the process of extracting image features. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] This invention is implemented according to the following process:

[0035] Step 1: Feature extraction of the problem, specifically:

[0036] The feature extraction module for the question primarily functions to obtain the question features from the text. The text question is mainly processed through word segmentation using the Tokenizer module and feature extraction using a Transformer network to enhance the accuracy of keyword localization. Its input is a natural language question, which is processed as follows: the question is split into m words, and then each word is embedded into an instruction vector and a global feature vector. The output is an m×300-dimensional instruction vector and a 1×300-dimensional global feature vector.

[0037] Step two, feature extraction of the image and construction of the scene graph, specifically:

[0038] For image feature extraction, a top-down attention approach is used to extract image features. The Faster R-CNN object detection model (a convolutional network based on ResNet-101) is used to extract image target region features. The entire image feature extraction model is trained on the Visual Genome dataset. Image feature extraction extracts K target features from each image, with each target object having a feature dimension of 2048. The acquired visual features are then embedded into a scene map containing rich image information using the SGG-Net network.

[0039] The scene graph is essentially a directed graph. Each node in the scene graph is a formal description of a related scene object, and each directed edge in the scene graph represents a formal relationship between two objects. A specific example is shown below. Figure 2 As shown, each relevant object is defined by its visibility, shape, color, material, openability, pose, and positional relationships, enabling the neural network model to effectively understand and manipulate objects. Regarding the positional relationships between objects, this model primarily focuses on spatial and semantic relationships, and introduces bounding boxes based on image feature extraction to enhance the reasoning ability between entities.

[0040] Step 3: Graph Neural Network Attention Inference, specifically:

[0041] The acquired scene graph features and problem features are introduced into the collaborative graph attention reasoning mechanism for joint attention iterative reasoning. The number of iteration layers of network reasoning is equal to the number of instruction vectors in the problem.

[0042] Step four, answer prediction, specifically:

[0043] The final iteration state of the network in step three and the global feature vector of the problem in step one are obtained and input into a softmax classifier for answer prediction. Then, the predicted answer is optimized using the cross-entropy loss function and the Adam optimizer.

[0044] Visual Question Answering (VQA) models, when handling scene-based text tasks, need to reason by reading both visual and textual content in an image, and by combining the visual and textual information to arrive at an answer. Poor text reading ability and insufficient visual reasoning ability are the main reasons why existing VQA models perform poorly. Figure 1 Traditional visual question answering methods only perform simple feature fusion of visual and textual information and lack a joint reasoning step. To address the above problems, this invention provides a visual question answering method based on a scene graph neural network reasoning mechanism.

[0045] The overall framework of the model in this paper is as follows: Figure 3 As shown, the entire model framework is mainly divided into four parts: question embedding, scene graph embedding, graph neural network attention inference, and answer prediction. In this framework, the main function of question embedding is to obtain the question features from the text. The text question is mainly segmented into words using the Tokenizer module and feature extracted using the Transformer network to enhance the accuracy of keyword localization. The main function of scene graph embedding is to first obtain the visual features of the image, and then embed these features into a scene graph containing rich image information (including entities and relationships between entities). Based on image feature extraction, bounding boxes are introduced to enhance the reasoning ability for relationships between entities. The main function of graph neural network attention inference is to combine scene graph features and question features and introduce a collaborative attention inference mechanism for iterative inference, simultaneously focusing on deep textual and visual information. The main function of the answer prediction module is to obtain the final iteration state of the graph neural network attention inference, and then input it along with the global textual feature information into the classifier for answer prediction.

[0046] The specific implementation process of this invention is as follows:

[0047] 1. The question is embedded in vector form.

[0048] The question embedding module first uses the Tokenizer module to segment the input natural language question, generating individual tokenized forms, and retaining only these individual tokens. It then employs a multi-head attention mechanism from the Transformer paradigm, such as... Figure 4As shown, by paying close attention to the semantics before and after the question, m words in the question are embedded into a sequence of m instruction vectors [g] (1) g (2) g (3) , ..., g (m) The vectorization process is represented by formulas (1) and (2), where the dimension of each vector is set to 300.

[0049] [g (1) g (2) g (3) , ..., g (m) ]=Transformer(q1, q2,..., q m (1)

[0050] Q=Transformer(q1, q2,..., q m (2)

[0051] 2. Image feature extraction and scene graph construction

[0052] The scene graph extraction module first uses the Faster-RCNN network to perform entity recognition on the input image, obtaining the target feature vector set X. i ={v1, v2, ..., v n}, v n ∈R 2048 , where n is the number of target objects, and each entity feature vector is set to 2048 dimensions.

[0053] After obtaining the entity features of the image, use them as the initial target feature vector, and then according to... Figure 2 The scene graph structure shown uses the target feature vector as input to the scene graph to construct a network that generates the initial state scene graph. (1) h (2) , ..., h (n) The spatial and semantic relationships are integrated and represented by formulas (3) and (4).

[0054]

[0055] h i =[h′ i g (i) (4)

[0056] Where h′ i h represents the node feature representation of the i-th entity. i This indicates that the graph nodes are combined with the i-th problem guidance vector g from step one. (i) Preparation before reasoning, w t1 W t2W t3 Let N be the linear transformation weight matrix, σ be the ReLU activation function, and N be the weight matrix. i Including all nodes except the i-th node itself, α ij This is an attention mechanism between nodes.

[0057] 3. Graph Neural Network Collaborative Attention Reasoning

[0058] This module obtains m question instruction vectors [g] derived from question embedding. (1) ,g (2) ,g (3) ,…,g (m) ] and the features of n initial nodes extracted from the scene graph [h (1) ,h (2) ,…,h (n) The graph neural network collaborative attention inference calculation is performed, and the final node state is obtained through iterative calculation. The specific process is as follows:

[0059] First, a single graph attention layer is described as the iterative attention inference layer in the graph neural network architecture. The input to this basic layer is a set of node features, represented by Equation (5).

[0060]

[0061] Where n is the number of nodes and F is the number of features in each node. This layer generates a new set of node features (which may have different numbers of features F'), represented by Equation (6) as the output of this layer.

[0062]

[0063] To obtain sufficient expressive power to transform the input features into higher-level features, at least one learnable linear transformation is required. Therefore, as an initial step, a transformation consisting of a weight matrix W∈R is applied to each node. F′×F Parameterized shared linear transformation. Then, self-attention is performed at the nodes, a shared attention mechanism a:R F′ ×R F′ →R, calculate the attention coefficient (node ​​and edge features), expressed by formula (7).

[0064]

[0065] This demonstrates the importance of node j's features to node i. The model allows each node to participate in the attention calculation for every other node while ignoring structural information. Graph structure is injected into this mechanism by performing masked attention, i.e., for node j∈N i Only calculate e ij , where N iIt is a neighborhood of node i in the graph. In the experiment, these are all first-order neighborhoods of node i (including node i). To make the coefficients easy to compare between different nodes, the softmax function is used to normalize all selected nodes j, as expressed by formula (8).

[0066]

[0067] In the experiment, the attention mechanism 'a' is a single-layer feedforward neural network consisting of a weight vector. Parameterization is performed, and LeakyReLU nonlinearity is applied (negative input slope α = 0.2). After full expansion, the coefficients are calculated by the attention mechanism (e.g.) Figure 5 As shown, it can be expressed as formula (9).

[0068]

[0069] Where T represents transpose and || represents cascading operation.

[0070] Obtain the normalized attention coefficient α ij Then, the linear combination of its corresponding features is calculated, and after applying the nonlinear activation function, it is used as the final output feature of each node, represented by formula (10).

[0071]

[0072] To stabilize the learning process of self-attention, multi-head attention collaborative computation (e.g., ...) is introduced into the single attention mechanism. Figure 2 (As shown). Specifically, the N-node independent attention mechanism performs the operation of formula (10), and then concatenates its features to produce an output feature representation, given by formula (11).

[0073]

[0074] Where || represents the cascade operation, α ij It is the attention mechanism of the kth node a k The normalized attention coefficient, W, is calculated. k Let h' be the weight matrix of the corresponding linear transformation of the input. After calculation by formula (11), the final output h' will contain the attention feature KF' of each node (instead of the attention feature F' of a single node).

[0075] Finally, multi-head attention is performed on the final (prediction) layer of the network. In this case, cascading operations are no longer reasonable. Instead, averaging operations are used, and the final nonlinear activation function (softmax or logisticsigmoid for classification problems) is applied with a delay, as expressed by formula (12).

[0076]

[0077] 4. Answer Prediction

[0078] After executing the graph reasoning module, the final state of all graph nodes was obtained after m iterations of message passing. First, summarize the final state H after message passing, and then input H into a linear classifier to predict the answer y using the global feature vector Q of the problem, as shown in formulas (13) and (14).

[0079]

[0080] y = Softmax(MLP(H,Q))(14).

Claims

1. A visual question answering system based on a scene graph neural network inference mechanism, characterized in that: The system comprises a question embedding module, a scene graph extraction module, a graph neural network attention reasoning module and an answer prediction module. In the question embedding module, a Tokenizer module divides an input natural language question into individual words, and only retains the individual words, and then uses a multi-head attention mechanism in a Transformer idea to embed the words into an instruction vector sequence and a question global feature vector. The scene graph extraction module first uses a pre-trained Faster-RCNN network to perform entity recognition on an input picture to obtain an initial target feature vector of each target entity, and inputs the initial target feature vector into an SGG network to generate an initial state scene graph. The graph neural network attention reasoning module performs graph neural network collaborative attention reasoning calculation according to the instruction vector sequence obtained by the question embedding module and the initial state scene graph obtained by the scene graph extraction module, and obtains final states of all graph nodes through a set number of iteration calculations. The answer prediction module utilizes The final state of the function summary graph node is obtained after message passing, and the linear classifier predicts the answer according to the final state after message passing and a question global feature vector. The visual question answering method using the system comprises the following steps: Step 1, use the Tokenizer module to split the input natural language question into individual words, and only keep the individual words, and then use the multi-head attention mechanism in the Transformer idea to embed the words into a sequence of instruction vectors and the question global feature vector , m represents the number of words; Step 2, using a pre-trained Faster-RCNN network to perform entity recognition on the input picture to obtain an initial target feature vector of each target entity , is the number of target objects, and the initial target feature vector is input into an SGG network to generate an initial state scene graph , represents the i-th graph node corresponding to the i-th target entity in the initial state scene graph, and k represents the number of target entities. Step 3, sequence of instruction vectors and initial state scenario graph perform graph neural network collaborative attention inference calculation, and obtain the final state of all graph nodes through iterative calculation m represents the number of iteration network layers The initial state scenario graph Integrating spatial and semantic relations, specifically: ; ;in, Indicates the first The node feature representation of an entity, This indicates that the first step in the process of joining graph nodes will be performed. Guiding vector for each problem Preparation before reasoning The weight matrix is ​​a linear transformation matrix. It is the ReLU activation function. Including except the first All nodes other than the node itself For the attention mechanism between nodes; the final state of all the graph nodes the first element satisfies: ; wherein, is a ReLU activation function, K denotes a target feature number, includes all nodes except the first node itself, is a normalized attention coefficient calculated by the first node attention mechanism , is a weight matrix of a corresponding input linear transformation, denotes a single graph layer attention layer jth feature node; Step 4, utilizing function summarizes the final state of the graph nodes get the final state after message passing , linear classifier predicts the answer based on the final state after message passing and the problem global feature vector predicts the answer .

2. The visual question answering system of claim 1, wherein: The answer from step 4 satisfies: ; wherein the final state satisfies: 。

Images