A visual question answering method based on scene graph relationship information enhancement

Abstract

A visual question answering method based on scene graph relationship information enhancement, first extracts the overall vector features and segmented vector features of the input question; then detects the visual target object and extracts the visual target features; then constructs a scene relationship graph focusing on the target relationship; the relationship node information is enhanced by using a multi-layer graph attention network, and a relationship selection module is designed to exclude and questions irrelevant or incorrect relationships; finally, the visual embedding, question embedding and scene graph relationship embedding are fused by using a multi-layer network with an attention mechanism to infer the answer to the question. In the training process, the method fully utilizes the supervision information provided by the pre-trained scene graph relationship decoder, designs a related loss function, and improves the ability of the model to understand the relationship between the image visual target based on the question itself. The invention improves the accuracy of visual question answering by enhancing the relationship node information of the scene graph and selectively selecting the scene graph relationship information related to the question.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision technology and natural language processing technology, and relates to a method for visual question answering, particularly a visual question answering method based on scene graph relation information enhancement. Background Technology

[0002] Visual Question Answering (VQA) aims to provide answers to natural language questions related to images based on information presented in the images. VQA models not only need to understand the image content, the semantics of the natural language question, and related knowledge, but also need to understand the relationship between the two modalities and make correct answer inferences. VQA technology has broad applications in real life, such as assisting visually impaired individuals in understanding and recognizing real-world or online image information, improving computers' image understanding capabilities, and supporting richer image retrieval. Most existing VQA methods mainly include image encoders, question encoders, feature fusion, and answer inference modules. These methods use convolutional neural networks (CNNs) to extract image features and recurrent neural networks (RNNs) to learn the question representation, then fuse the two features into the same space to predict the answer. In datasets such as GQA and VQA 2.0, most questions involve more than one object. Therefore, richer information about the relationship between the object and the question is more helpful.

[0003] Utilizing more abstract levels of image representation for visual question answering has become a recent research hotspot. Visual question answering using scene graphs not only offers excellent performance but also advantages in interpretability. However, it suffers from two main drawbacks: first, the scene graphs generated using existing scene graph generation models contain much irrelevant node and edge information, which may also contain erroneous noise; second, some existing models use implicit relational reasoning or simple spatial relationships to determine the relationships between objects in the image, failing to fully utilize the supervised information from the scene graph. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a visual question answering method based on scene graph relation information enhancement. This method enhances the relation information of the generated initial scene graph and selects relations in conjunction with the question, thereby reducing the influence of irrelevant and erroneous edges in the scene graph on the reasoning of visual question answering answers, and thus improving the accuracy of visual question answering.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A visual question answering method based on scene graph relationship information enhancement includes the following steps:

[0007] Step 1: For text-based input questions, extract their overall vector features and word segmentation vector features;

[0008] Step 2: For the input image, detect visual target objects and extract visual target features;

[0009] Step 3: Construct a scene relationship graph using the visual target features, and classify the relationships between visual target objects using a pre-trained scene graph relationship decoder to obtain supervised information on the relationships between visual target objects; in the scene relationship graph, the relationships between visual target objects are used as nodes, and the visual target objects are used as edges.

[0010] Step 4: Enhance the scene relationship graph using an attention network based on a graph message propagation mechanism;

[0011] Step 5: Use the overall vector features to select the relationship features in the scene relationship graph to obtain the problem-related scene graph relationship features;

[0012] Step 6: Infer the answer to the input question by integrating word segmentation vector features, visual target features, and question-related scene graph relationship features.

[0013] Compared with existing technologies, the advantages of this invention are: This invention utilizes graph neural networks to enhance relationships in the scene graph and selects relationships guided by the problem, reducing noise from irrelevant or erroneous edges in the scene graph, thereby achieving better performance. Furthermore, this invention fully leverages the capabilities of pre-trained scene graph models, using labels generated by the scene graph relationship decoder as supervision information, and designs two related loss functions to improve the capabilities of the enhancement modules on the model's scene relationship graph. Attached Figure Description

[0014] Figure 1 This is an overall flowchart of the visual question answering method based on scene graph relationship information enhancement of the present invention.

[0015] Figure 2 This is a block diagram of the overall model of the visual question answering method based on scene graph relationship information enhancement according to the present invention.

[0016] Figure 3 This is a schematic diagram of the multi-layer Transformer modal fusion inference network in this invention.

[0017] Figure 4 This is a sample result diagram of the visual question-answering method of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0019] refer to Figure 1 and Figure 2 The specific steps of this invention are as follows: This invention is a visual question answering method based on scene graph information enhancement, mainly comprising the following six parts: question embedding, visual target feature extraction, scene relationship graph construction, scene relationship graph enhancement, relationship selection, modality fusion, and answer reasoning, implemented according to the following steps:

[0020] Step 1: For text-based input problems, extract the overall vector features and word segmentation vector features.

[0021] This invention divides the question into independent words using WordPiece. After adding special tags to the word set, it uses the GloVe word vector model to vectorize the words in the input question. Based on the GloVe word vectors, it then uses a pre-trained BERT model to extract the overall vector features and word segmentation vector features of the input question. The specific steps are as follows:

[0022] Step 1.1: Use the WordPiece method to divide the input question into a word array W of subwords. That is, the input question sentence is transformed into a word array W, represented as:

[0023] W = [w1, w2, ..., w N ]

[0024] Where N is the number of words in the input question, w1, w2, ..., w N It consists of N individual subwords.

[0025] Step 1.2: Add special markers [CLS] and [SEP] to the word array W. The set of words transformed from the question sentence can be represented as {[CLS], w1, w2, ..., w...} N ,[SEP]}.

[0026] Step 1.3: Use the GloVe word vector model to obtain the set of word vectors H, represented as:

[0027] H = [h1, h2, ..., h N ]

[0028] Where H is the set of word vectors trained by the GloVe word vector model, h1, h2, ..., h N There are N word vectors. H is the set of word vectors trained using the GloVe word vector model, where h1, h2, ..., h... Nq For N q Each word vector.

[0029] Step 1.4: Input the word vectors into the pre-trained BERT model to obtain the overall vector features of the input question. and word segmentation vector features They are respectively associated with h1, h2, ..., h Nq The corresponding word segmentation vector features.

[0030] Step 2: For the input image, detect visual target objects and extract visual target features.

[0031] In this invention, the Faster R-CNN model framework is used for image visual information extraction, which is beneficial for obtaining multi-dimensional feature information in the image and effectively improves the accuracy of subsequent scene relationship graph construction. In this example, the X152-C4 network is selected as the backbone network of the Faster R-CNN object detection algorithm. This object detection algorithm is used to detect objects in the input image and predict N. v One target object, and obtain its object semantic features. Regional visual features and regional location characteristics A comprehensive feature is generated by combining the object's semantic features, region visual features, and region location features. The formula for calculating the comprehensive features is defined as follows: Wherein, for the i-th candidate visual target object, the formula for calculating its comprehensive features is defined as follows:

[0032] l i =σ(FC) l ([x i p i Emb l (s i )]))

[0033] Among them, l i Let p represent the comprehensive features of the i-th target object, xi represent the regional visual features of the i-th target object, and p represent the comprehensive features of the i-th target object. i s represents the regional location feature of the i-th target object. i Let [·] represent the semantic features of the i-th target object, [·] represent a concatenation operation, σ(·) represent the ReLU nonlinear transformation, and Emb l (·) represents a language word embedding transformation based on a pre-trained GloVe model, FC l (x) = Wx + b represents a fully connected layer.

[0034] In this invention, the top K with the highest confidence level are selected. v The candidate bounding boxes of objects serve as the set of visual target objects in the scene graph. Indicates the Kth v A visual target object.

[0035] After the above comprehensive feature calculations, the representation of the visual target object includes not only its own visual region features, but also the object's spatial location information in the image and abstract semantic descriptive features. With this information, subsequent models can more easily understand the semantic and spatial relationships between the learned objects, thus enabling more accurate reasoning about the relationships between target objects and, based on this, predicting the correct answer.

[0036] Step 3: Construct a scene relationship graph using the visual target features, and classify the relationships between visual target objects using a pre-trained scene graph relationship decoder to obtain supervised information on the relationships between visual target objects; in the scene relationship graph, the relationships between visual target objects are used as nodes, and the visual target objects are used as edges.

[0037] This invention uses the visual target features obtained in step 2 to construct an initial scene relationship graph. Unlike a typical scene graph, the scene relationship graph emphasizes the relationships between visual target objects. Therefore, after constructing fully connected initial relationships between visual target objects, these initial relationships are selected as nodes in the scene relationship graph, and nodes related to these initial relationships are used as edges. Simultaneously, to fully utilize the supervision information of the scene graph, a pre-trained scene graph relationship decoder is used to classify the relationships between visual targets generated in step 1, obtaining classification supervision information regarding the relationships between targets. The specific steps are as follows:

[0038] Step 3.1: Establishing the initial relationships between visual target objects.

[0039] Use the set of visual target objects V to construct an initial set of relationships E between the visual target objects, if e k ∈E, and the i-th visual target object v i and the j-th visual target object v j There exists a directed e k Relationships, Writing <v i -e k -v j >,e k The calculation formula is as follows:

[0040] e k =σ(FC) r3 ([FC r1 (v i ), FC r2 (v j )]))

[0041] Where [·] represents a concatenation operation, σ(·) represents the nonlinear transformation ReLU, FC r1 (x), FC r2 (x) and FC r3(x) represents a fully connected layer.

[0042] Step 3.2: Scene Relationship Diagram G rel The construction of.

[0043] Slightly different from common scene graph representations, this invention focuses on enhancing scene graph relationships, using relationships between target objects as nodes, while visual target objects are the edges of the scene relationship graph. V represents the visual target object v in the scene relationship graph. i The set of elements is also the set of edges in the scene relationship graph; E represents the relationship e obtained in step 3.1. i It is a set of nodes in the scene relationship graph.

[0044] Since the scene relationship graph is directed, for any v k If ∈V, then it can be represented as e i For e j There exists a directed relationship; let's write... <e i -v k -e j This allows for the construction of a new scene relationship diagram, using e i and v k Taking the direction as the reference, the adjacency matrices of the incoming and outgoing edges of the relation node set E are represented as A. in and A out The scene relationship diagram is defined as follows:

[0045] G rel ={E, V, A in A out}

[0046] The initial relation set E serves as the scene relation graph G. rel The set of nodes, the set of visual target objects V, serves as the scene relationship graph G. rel The set of edges, A in and A out It is the adjacency matrix of the incoming and outgoing edges of the node set.

[0047] Step 3.3: The prediction results of the scene graph relation decoder are used as relation supervision information.

[0048] A scene graph relation decoder, a classification network pre-trained on the Visual Genome dataset, is used to predict relationships between visual objects. This relation decoder predicts the relationship between any two visual objects in the set V, obtaining supervisory information about the relationships between them, which serves as the supervisory labels for the relationships.

[0049] Step 4: Enhance the scene relationship graph using an attention network based on a graph message propagation mechanism.

[0050] The scene relationship graph G obtained in step 3 of this invention rel The above approach enhances the scene relationship graph by updating the information of relationship nodes through an attention network based on a graph message propagation mechanism. To aggregate information from neighboring nodes and edges to better represent the relationships between visual target objects, an attention network method is employed, significantly enhancing the information of the relationship nodes.

[0051] Step 4.1: Evaluate the relevance of each node in the scene relationship graph to its neighboring nodes using an attention mechanism; node e i Its neighbor node e j Attention weight α ij The calculation formula is as follows:

[0052]

[0053] Where v k It is a connection e i and e j directed edges <e i v k e j >, It is a learnable mapping matrix, d h d is the dimension of the hidden layer vector set in the correlation calculation. e Let be the vector dimension of the node. It is a learnable mapping matrix, d v It is the vector dimension of the edge, and dir(·) is the connection node e. i To node e j v k The edge direction is divided into two directions: outgoing edge and incoming edge, N. i Represents node e i The set of all neighboring nodes.

[0054] Step 4.2: Update the node features using a graph messaging framework. Use the obtained attention weights to aggregate information from neighboring nodes and the node itself to update each node, node e. i The update formula is as follows:

[0055]

[0056] in, It is node e i Enhanced representation, and It is a learnable mapping matrix.

[0057] Step 4.3: Construct an attention network using steps 4.1 and 4.2, which consists of m layers; the input nodes of the first layer are equal to the nodes of the constructed initial scene relationship graph. The node is obtained after updating through a layer of attention network. The updated relation nodes of the attention network in layer m are represented as follows: The set of relation nodes enhanced by an m-layer attention network is represented as follows: n r Indicates the number of edges.

[0058] Step 4.4: Loss function for scene graph relationship classification. For the enhanced set of relationship nodes E... * Using fully connected layers and the sigmoid activation function to predict the classification of relation nodes. rel , is represented as:

[0059] y rel =Sigmoid(FC(E) * ))

[0060] To fully utilize the supervision information from the pre-trained scene graph relation decoder, the supervision information obtained in step 3.3 is used to construct the relation classification loss function L. cls for:

[0061]

[0062] Where, N rel This is the total number of relations (N in the preceding text represents the number of individual subwords). This refers to the classification results predicted after enhancing the relationship nodes. These are the supervision labels obtained by the relation decoder.

[0063] This model can not only supervise learning through the correctness of answers to visual questions, but also learn the training labels provided by the pre-trained relation decoder, thus gaining the ability to predict relationships in scene graphs.

[0064] Step 5: Use the overall vector features to select the relationship features in the scene relationship graph to obtain the problem-related scene graph relationship features.

[0065] In this step, guided by the overall vector features, attention operations are used to filter relationships relevant to the question, thereby eliminating the influence of irrelevant or erroneous relationships on visual question answering reasoning. The specific steps for selecting relationship features are as follows:

[0066] Step 5.1: Calculate the overall vector features using attention operations. With the enhanced set of relation nodes E * Similarity score for each node in the dataset:

[0067]

[0068] in, The dot product operation (<·>) uses the sigmoid function to restrict the similarity score to (0, 1), selecting the top K. r Each node represents a set of candidate relations.

[0069] Step 5.2: Relationship ranking loss function.

[0070] Using the supervised information, the pre-trained scene graph relation decoder determines the nodes related to two visual target objects as a set of positive relations. Irrelevant nodes are considered as a set of negative relationships. Constructing the relation ranking loss function:

[0071]

[0072] Where m is the adjustable threshold, Z is a normalization factor, and f p express The similarity score f of p sampled in step 5.1 n express The similarity score of n sampled in step 5.1.

[0073] Step 6: Infer the answer to the input question by integrating word segmentation vector features, visual target features, and question-related scene graph relationship features.

[0074] This invention utilizes a multi-layer network with an attention mechanism to fuse word segmentation vector features, visual target features, and question-related scene graph relationship features to infer the answer to the input question. The multi-layer network with the attention mechanism is a modal fusion network with a multi-layer Transformer structure. A multi-classifier selects the answer with the highest probability as the inference result. The specific steps for inferring the answer to the input question can be described as follows:

[0075] Step 6.1: Input Feature Concatenation. The question segmentation vector obtained in Step 1 is represented as Q = [q1, q2, ..., q...]. N As a question embedding, the set of visual target objects obtained in step 2 As a visual embedding, the scene graph relation representation obtained in step 5 As a relational embedding, special markers [CLS] and [SEP] are added, and the transformed concatenated vector O is... in It can be represented as:

[0076]

[0077] Step 6.2: Combine the concatenated vector O obtained in Step 6.1in The input is fed into a multi-layer Transformer fusion network, and the final output feature at the [CLS] marked input position is used as the fusion feature, represented as:

[0078] O F =Transformer(O in ,Θ)

[0079] Where Transformer(·) represents the training operation of a multi-layer fusion network, and Θ represents the trainable parameters.

[0080] The multi-layer Transformer fusion network architecture in this invention is as follows: Figure 3 As shown, it is mainly divided into three modules: input module, encoding module, and output module.

[0081] The input module refers to the embedding layer that processes the input; the encoding module consists of multiple stacked encoder layers, each encoder layer comprising two sub-layers. The first sub-layer contains a multi-head self-attention sub-layer, a normalization layer, and a residual connection; the second sub-layer contains a feedforward fully connected sub-layer, a normalization layer, and a residual connection; the output module includes a linearization layer and a layer normalization function, here using the Sigmoid function. The final output can be represented as:

[0082] u = Sigmoid(FC(O) F ))

[0083] Where Sigmoid(·) represents the Sigmoid function operation, FC(x) = Wx + b represents a fully connected layer, and O F This represents the fusion feature. The answer with the highest probability among the candidate answers is selected as the model's predicted answer, such as... Figure 4 This demonstrates a case study of the model's visual question answering capabilities.

[0084] Step 6.3: Loss Function. One component of the loss function is the standard cross-entropy used in the visual question answering part. This embodiment uses binary cross-entropy as the question answering loss function for training classification problems. The question answering loss function is as follows:

[0085]

[0086] y is the model's predicted answer. This is the marked answer, N ans This represents the total number of all answers;

[0087] The final total loss function is the question-answering loss function L. QA The classification loss function L in step 4.4 cls And the sorting loss function L in step 5.3 rank sum:

[0088] L total =L QA +L cls +L rank

[0089] Model training and experimental process of the present invention

[0090] 1. Dataset

[0091] This invention selects the GQA dataset, which contains 113,018 real-world images from the Visual Genome dataset and includes 22,669,678 questions, covering a wide range of reasoning abilities.

[0092] Evaluation Metrics: GQA treats visual question answering as a multi-class classification problem. A voting mechanism is used to calculate the model's accuracy.

[0093]

[0094] Here, N represents the number of answers that are consistent with those of the annotators. Therefore, in the dataset, the model's predicted answer must be consistent with at least 3 annotators for the model's prediction to be considered correct.

[0095] 2. Experimental Environment

[0096] This invention uses Python 3.7 and the PyTorch 1.5.0 deep learning framework. The experiment was conducted using eight Nvidia GTX Titan-V graphics cards with a batch size of 128. The Adam gradient descent algorithm was used with a learning rate of 0.0001. After 12,000 model iterations, the learning rate was multiplied by 0.2 every 5,000 iterations. The experiment was stopped after 27,000 iterations.

[0097] The best results of the model of this invention trained on the GQA dataset are compared with other state-of-the-art models, as shown in Table 1.

[0098] Table 1. Comparison results of the model of this invention with other models on the GQA test-dev dataset.

[0099]

[0100] The MAC model is a high-performing implicit relational multi-stage reasoning model that uses multiple attention-based guidance vectors to reason on visual features for the question. MCAN-large is a high-performing Trasformer-like structure model employing self-attention and interactive attention mechanisms. In the table, "Human" represents the test results of humans on the GQA dataset.

[0101] Ours is the model of this invention, which improves the overall accuracy by 13.16% compared to the benchmark model CNN+LSTM, improves the accuracy by 9.97% compared to the advanced BUTD model, and is also better in overall accuracy compared to the advanced MCAN-large model.

[0102] In summary, this invention fully extracts the features of visual targets in an image, initializes the relationships between these targets based on these features, models the image as a scene relationship graph focusing on these relationships, enhances the relationship node information using a multi-layer graph attention network, and then incorporates a relationship selection module designed to exclude irrelevant or incorrect relationships. Finally, it uses a multi-layer network with an attention mechanism to fuse visual embeddings, question embeddings, and scene graph relationship embeddings for reasoning to predict the answer to the question. Furthermore, during training, this invention fully utilizes the supervision information provided by a pre-trained scene graph relationship decoder and designs relevant loss functions to improve the model's ability to understand the relationships between visual targets in the image in conjunction with the question itself. This invention improves the accuracy of visual question answering by enhancing the relationship node information in the scene graph and selectively selecting question-related scene graph relationship information.

Claims

1. A visual question answering method based on scene graph relation information enhancement, characterized in that, Includes the following steps: Step 1: For text-based input questions, extract their overall vector features and word segmentation vector features; Step 2: For the input image, detect visual target objects and extract visual target features; Step 3: Construct a scene relationship graph using the visual target features, and classify the relationships between visual target objects using a pre-trained scene graph relationship decoder to obtain supervised information on the relationships between visual target objects; In the scene relationship diagram, the relationships between visual target objects are used as nodes, and the visual target objects are used as edges. Step 4: Enhance the scene relationship graph using an attention network based on a graph message propagation mechanism; Step 5: Use the overall vector features to select the relationship features in the scene relationship graph to obtain the problem-related scene graph relationship features; Step 6: Infer the answer to the input question by integrating word segmentation vector features, visual target features, and question-related scene graph relationship features; In step 4, the scene relationship graph is enhanced by updating the information of the relationship nodes through the attention network of the graph message propagation mechanism. The steps are as follows: Step 4.1: Evaluate the relevance of each node in the scene relationship graph to its neighboring nodes using an attention mechanism; Its neighboring nodes Attention weight The calculation formula is as follows: in It is a connection and The directed edge < , , >, It is a learnable mapping matrix. The hidden layer vector dimension set in the correlation calculation. Let be the vector dimension of the node. It is a learnable mapping matrix. It is the vector dimension of the edge. It is a connection node To the node of The direction of the edge is divided into two directions: outgoing edge and incoming edge. Represents a node The set of all neighboring nodes; Step 4.2: Update the node features using a graph messaging framework. Use the obtained attention weights to aggregate information from neighboring nodes and the node itself, thereby updating each node. The update formula is as follows: in, It is a node Enhanced representation, and It is a learnable mapping matrix; Step 4.3: Construct an attention network using steps 4.1 and 4.

2. This network has m layers, ultimately yielding an enhanced set of relational nodes. ; Step 4.4: Optimize the enhanced set of relation nodes. Fully connected layers and the sigmoid activation function are used to predict the classification of relation nodes. , is represented as: Using the aforementioned supervision information, a loss function for relation classification is constructed. , is represented as: in, It is the total number of relationships. This refers to the classification results predicted after enhancing the relationship nodes. These are the supervision labels obtained by the relation decoder.

2. The visual question answering method based on scene graph relationship information enhancement according to claim 1, characterized in that, In step 1, the words in the input question are vectorized using the GloVe word vector model. Based on the GloVe word vectors, the overall vector features and word segmentation vector features of the input question are extracted by combining the pre-trained BERT model.

3. The visual question answering method based on scene graph relationship information enhancement according to claim 1 or 2, characterized in that, Step 1, the steps for extracting the overall vector features and word segmentation vector features, are as follows: Step 1.1: Use WordPiece to divide the input question into an array of words containing sub-words. , is represented as: in To input the number of words in the question, for A single subword; Step 1.2: Add special markers and to word array In, it is represented as ; Step 1.3: Process the word array Obtain a set of word vectors using the GloVe word vector model. , is represented as: in, This is a set of word vectors trained using the GloVe word vector model. for Word vectors; Step 1.4: Input the word vectors into the pre-trained BERT model to obtain the overall vector features of the input question. and word segmentation vector features , They are respectively with The corresponding word segmentation vector features.

4. The visual question answering method based on scene graph relationship information enhancement according to claim 3, characterized in that, Step 2 involves using the Faster R-CNN object detection algorithm, with the X152-C4 network as its backbone, to perform object detection on the input image. For each target object, obtain the object's semantic features. Regional visual features and regional location characteristics Combine the object's semantic features, region's visual features, and region's location features to generate a comprehensive feature. The formula for calculating the comprehensive features is defined as follows: in, Indicates the first i The comprehensive characteristics of a target object Indicates the first i The visual characteristics of the target object's region Indicates the first i The regional location characteristics of a target object Indicates the first i The semantic features of a target object This represents a concatenation operation. Represents nonlinear transformation , This represents the language word embedding transformation based on the pre-trained GloVe model. To represent a fully connected layer, select the layer with the highest confidence level. The candidate bounding boxes of objects serve as the set of visual target objects in the scene graph. , Indicates the first A visual target object.

5. The visual question answering method based on scene graph relationship information enhancement according to claim 4, characterized in that, Step 3, which involves constructing a scene relationship graph and obtaining supervised information through classification, is as follows: Step 3.1: Establishing the initial relationship between visual target objects Use visual target object collection Construct an initial set of relationships between visual target objects. ,like And the first A visual target object and the A visual target object There exists a directed Relationships, Writing , The calculation formula is as follows: in, , and Both represent fully connected layers; Step 3.2: Scene Relationship Diagram Construction The scene relationship diagram is defined as follows: Initial set of relations As a scene relationship diagram The set of nodes, the set of visual target objects As a scene relationship diagram The set of edges, and It is the adjacency matrix of the incoming and outgoing edges of the node set; Step 3.3: The prediction results of the scene graph relation decoder are used as relation supervision information. A pre-trained scene graph relation decoder is used to predict the relationships between visual target objects. The relation decoder then processes the set of visual target objects. The relationship between any two visual target objects is predicted to obtain supervisory information on the relationship between visual target objects.

6. The visual question answering method based on scene graph relationship information enhancement according to claim 4, characterized in that, In step 5, guided by the overall vector features, attention operations are used to filter relationships relevant to the question, thereby eliminating the influence of irrelevant or erroneous relationships on visual question answering reasoning.

7. The visual question answering method based on scene graph relationship information enhancement according to claim 6, characterized in that, Step 5, the steps for selecting relation features, are as follows: Step 5.1: Calculate the overall vector features using attention operations. With the enhanced set of relation nodes Similarity score for each node in the dataset: in, , For the dot product operation, the sigmoid function is used to limit the similarity score within a certain range. Before choosing Each node represents a set of candidate relations. ; Step 5.2: Relationship Ranking Loss Function Using the supervised information, the pre-trained scene graph relation decoder determines the nodes related to two visual target objects as a set of positive relations. Irrelevant nodes are considered as a set of negative relationships. Construct the relation ranking loss function: in, It is an adjustable threshold. It is a normalization factor. express Mid-sampling The similarity score in step 5.1, express Mid-sampling The similarity score in step 5.

1.

8. The visual question answering method based on scene graph relationship information enhancement according to claim 7, characterized in that, In step 6, the answer to the input question is inferred by fusing word segmentation vector features, visual target features, and question-related scene graph relationship features using a multi-layer network with an attention mechanism. The multi-layer network with the attention mechanism is a modal fusion network with a multi-layer Transformer structure, and the answer with the highest probability is selected as the inference result by a multi-classifier.

9. The visual question answering method based on scene graph relationship information enhancement according to claim 8, characterized in that, Step 6, the steps for reasoning the answer to the input question, are as follows: Step 6.1: Input Feature Concatenation Word segmentation vector features As a problem embedding, the visual target object set As a visual embedding, a scene graph relationship set As a relation embedding, add special tags. and The concatenated vector is obtained as follows: : ； Step 6.2, concatenate the vectors The input is fed into a multi-layer Transformer modal fusion network, in The last output feature at the input position is used as the fused feature, represented as: in, This represents the training operation of a multi-layer fusion network. Indicates trainable parameters; After the linearization layer and normalization function of the output layer, the final output answer is expressed as: in, This indicates the operation of the Sigmoid function. This represents a fully connected layer. The fusion features are represented, and the candidate answer with the highest probability is selected as the reasoning answer; Step 6.3: Loss Function One component of the loss function is the standard cross-entropy used in the visual question answering part. For training classification problems, the question answering loss function is as follows: It is the model's predicted answer. It is to mark the answer. This represents the total number of all answers; The final total loss function is the question-answering loss function. Classification loss function and sorting loss function sum: 。

Images