Visual question answering method and device based on graph attention neural network and visual relationship

Abstract

The application provides a visual question answering method and device based on a graph attention neural network and a visual relationship, which comprises the following steps: inputting a target image into a feature extraction layer in a visual question answering model to obtain image features of the target image, and inputting a target question text into the feature extraction layer to obtain text features of the target question text; inputting the image features and the text features into a graph attention neural network layer in the visual question answering model to obtain semantic relationship features; inputting the semantic relationship features into a visual relationship network layer in the visual question answering model to obtain visual relationship features; and inputting the semantic relationship features and the visual relationship features into an answer prediction layer in the visual question answering model to obtain an answer corresponding to the target question text. The application realizes accurate reasoning of an answer corresponding to a target question text based on a target image and a target question text.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a visual question answering method and apparatus based on the relationship between graph attention neural networks and vision. Background Technology

[0002] With the rapid development of deep learning in recent years, a variety of problems and research directions have emerged. Deep learning can be broadly divided into two main research directions: Computer Vision (CV) and Natural Language Processing (NLP). Within these two fields, image annotation and text question answering each represent a specific research direction. Image annotation involves deep learning models learning from a large number of labeled images to annotate unknown new images. Text question answering, on the other hand, involves models responding to specific questions posed by a natural language sentence or paragraph. Visual Question Answering (VQA) is a new interdisciplinary field that has emerged with the rise of computer vision and natural language processing in recent years. This model not only processes image information but also provides a reasonable answer to a given question about the image.

[0003] Most existing methods rely on attention mechanisms to find key information in images, or extract image information such as entities or attributes as a supplement to semantic knowledge. However, the correlation between this image information and the question text is weak, and it cannot fully represent the features of visual question answering, resulting in inaccurate visual question answering results obtained based on this image information. Summary of the Invention

[0004] This invention provides a visual question answering method and apparatus based on graph attention neural networks and visual relationships. It addresses the shortcomings of existing technologies that rely on attention mechanisms to find key information in images or extract some entity or attribute information from images as a supplement to semantic knowledge. These methods cannot fully represent the features of visual question answering, resulting in inaccurate visual question answering results. This invention aims to achieve accurate visual question answering results.

[0005] This invention provides a visual question answering method based on graph attention neural networks and visual relationships, comprising:

[0006] The target image is input into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and the target question text is input into the feature extraction layer to obtain the text features of the target question text;

[0007] The image features and the text features are input into the graph attention neural network layer of the visual question answering model to obtain semantic relationship features;

[0008] The semantic relationship features are input into the visual relationship network layer in the visual question answering model to obtain visual relationship features;

[0009] The semantic relationship features and the visual relationship features are input into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text.

[0010] The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0011] According to the present invention, a visual question answering method based on graph attention neural networks and visual relationships is provided, wherein the step of inputting the image features and the text features into the graph attention neural network layer of the visual question answering model to obtain semantic relationship features includes:

[0012] Each region feature in the image features is concatenated with the text features to obtain the concatenation result corresponding to each region feature;

[0013] The concatenation result corresponding to the features of each region is used as the first node of the graph attention neural network layer.

[0014] Based on the graph attention neural network layer, entity relationships between multiple first nodes are extracted to obtain entity relationship features, and the entity relationship features are fused with the text features to obtain the semantic relationship features.

[0015] According to the present invention, a visual question answering method based on graph attention neural networks and visual relationships is provided, wherein the step of inputting the semantic relationship features into the visual relationship network layer in the visual question answering model to obtain visual relationship features includes:

[0016] The semantic relationship features are input into the visual relationship network layer;

[0017] Based on the visual relationship network layer, multiple candidate relationship features corresponding to the semantic relationship features are extracted, and each candidate relationship feature is fused with the semantic relationship features to obtain a first fused feature corresponding to each candidate relationship feature. Based on the first fused feature corresponding to each candidate relationship feature, the visual relationship features are obtained.

[0018] According to the present invention, a visual question answering method based on graph attention neural network and visual relationships is provided, wherein obtaining the visual relationship features according to the first fusion feature corresponding to each candidate relationship feature includes:

[0019] The first fusion feature corresponding to each candidate relationship feature is fused with each candidate relationship feature;

[0020] Based on the fusion results, the visual relationship features are obtained.

[0021] According to the present invention, a visual question answering method based on graph attention neural network and visual relationships is provided, wherein obtaining the visual relationship features according to the first fusion feature corresponding to each candidate relationship feature includes:

[0022] The first fusion feature corresponding to each candidate relation feature is used as the second node of the graph attention neural network layer in the visual relation network layer;

[0023] Based on the graph attention neural network layer in the visual relationship network layer, the entity relationships between multiple second nodes are extracted to obtain the visual relationship features.

[0024] According to the present invention, a visual question answering method based on graph attention neural network and visual relationship is provided, wherein the answer prediction layer includes an attention network layer and a prediction layer;

[0025] The step of inputting the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text includes:

[0026] Based on the semantic relationship features, obtain the first input information; based on the visual relationship features, obtain the second input information.

[0027] The first input information and the second input information are input into the attention network layer to obtain the second fusion feature; the attention network layer is used to perform multimodal fusion of the semantic relationship feature and the visual relationship feature based on the first input information and the second input information, and output the second fusion feature;

[0028] The second fusion feature is input into the prediction layer to obtain the answer corresponding to the target question text.

[0029] According to a visual question-answering method based on graph attention neural network and visual relationship provided by the present invention, the step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0030] The first input information is input to the first input port and the second input port of the attention network layer, respectively, and the second input information is input to the third input port of the attention network layer to obtain the second fusion feature.

[0031] According to a visual question-answering method based on graph attention neural network and visual relationship provided by the present invention, the step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0032] The second input information is input to the first input port and the second input port of the attention network layer, respectively, and the first input information is input to the third input port of the attention network layer to obtain the second fusion feature.

[0033] According to the present invention, a visual question answering method based on the relationship between graph attention neural network and vision is provided, wherein the attention network layer includes a first branch attention network layer and a second branch attention network layer;

[0034] The step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0035] The first input information is input to the first input port and the second input port of the first branch attention network layer, and the second input information is input to the third input port of the first branch attention network layer to obtain the third fusion feature;

[0036] The second input information is input to the first and second input ports of the second branch attention network layer, and the first input information is input to the third input port of the second branch attention network layer to obtain the fourth fusion feature;

[0037] The second fusion feature is obtained based on the third fusion feature and the fourth fusion feature.

[0038] According to the present invention, a visual question answering method based on graph attention neural network and visual relationship is provided, wherein the feature extraction layer includes a residual network layer and a recurrent network layer;

[0039] The process of inputting the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and inputting the target question text into the feature extraction layer to obtain the text features of the target question text, includes:

[0040] The target image is input into the residual network layer, and the image features are extracted based on the residual network layer.

[0041] The target question text is input into the recurrent network layer, and semantic features are extracted from the target question text based on the recurrent network layer to obtain the text features.

[0042] The present invention also provides a visual question answering device based on the relationship between graph attention neural networks and vision, comprising:

[0043] The first feature extraction module is used to input the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and input the target question text into the feature extraction layer to obtain the text features of the target question text;

[0044] The second feature extraction module is used to input the image features and the text features into the graph attention neural network layer in the visual question answering model to obtain semantic relationship features;

[0045] The third feature extraction module is used to input the semantic relationship features into the visual relationship network layer in the visual question answering model to obtain visual relationship features;

[0046] The answer prediction module is used to input the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text;

[0047] The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0048] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the visual question answering method based on the relationship between graph attention neural network and vision as described above.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual question answering method based on the relationship between graph attention neural network and vision as described above.

[0050] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the visual question answering method based on the relationship between graph attention neural network and vision as described above.

[0051] The present invention provides a visual question answering method and apparatus based on graph attention neural networks and visual relationships. It employs a feature extraction layer to extract image features of the target image and text features of the target question text. A graph attention neural network layer is then used to deeply explore the interaction between image and text features to obtain semantic relationship features. A visual relationship network layer is then used to extract visual relationships from these semantic relationship features. These acquired visual relationship features are used as external prior knowledge to supplement the visual question answering features. This allows the visual relationship model to not only acquire rich visual question answering features, but also to possess not only semantic relationship features between the image and the question text, but also prior knowledge. Based on these visual question answering features, the answer corresponding to the target question text can be accurately inferred. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the visual question answering method based on the relationship between graph attention neural networks and vision provided by this invention.

[0054] Figure 2 This is a schematic diagram of the structure of the visual question answering model in the visual question answering method based on the relationship between graph attention neural network and vision provided by the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of the classifier in the visual relationship network layer of the visual question answering method based on graph attention neural network and visual relationship provided by the present invention;

[0056] Figure 4 This is one of the schematic diagrams of the attention network layer in the visual question answering method based on the relationship between graph attention neural network and vision provided by the present invention;

[0057] Figure 5 This is the second schematic diagram of the structure of the attention network layer in the visual question answering method based on the relationship between graph attention neural network and vision provided by this invention;

[0058] Figure 6 This is the third schematic diagram of the structure of the attention network layer in the visual question answering method based on the relationship between graph attention neural network and vision provided by this invention;

[0059] Figure 7 This is a schematic diagram of the structure of the visual question answering device based on the relationship between graph attention neural network and vision provided by the present invention;

[0060] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] It should be noted that the subject executing this method can be an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This invention does not impose specific limitations.

[0063] The following is combined with Figures 1-6 This invention describes a visual question answering method based on the relationship between graph attention neural networks and vision.

[0064] like Figure 1 The diagram shown is a flowchart of the visual question answering method based on graph attention neural network and visual relationship provided in this embodiment, which specifically includes the following steps:

[0065] Step 101: Input the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and input the target question text into the feature extraction layer to obtain the text features of the target question text;

[0066] The target image is the image that needs to be used for visual question answering. It can be of various types, such as animal images, plant images, vehicle images, road images, etc. This implementation does not make specific limitations on this.

[0067] The target question text is the text that asks a question about the target image.

[0068] Visual Question Answering (VQA) is a new interdisciplinary field that emerged with the rise of computer vision and natural language processing. This model not only processes image information but also provides a reasonable answer to a given question about the image. For example, if an image shows a user wearing a mustache made of special objects (such as bananas), and the question text is "What is the mustache made of?", the visual question answering model should output the answer "bananas".

[0069] A visual question answering model is constructed based on at least a feature extraction layer, a graph attention neural network layer, a visual relation network layer, and an answer prediction layer. The feature extraction layer is used to extract image and text features, the graph attention neural network layer is used to extract semantic relation features, the visual relation network layer is used to extract visual relation features, and the answer prediction layer is used to obtain the answer corresponding to the target question text in the target image.

[0070] Optionally, before performing step 101, the visual question-answering model needs to be trained. The specific training steps include:

[0071] First, obtain the training dataset; this dataset consists of sample images, sample question texts, and their corresponding answers. The number of samples in the training dataset can be set according to actual needs.

[0072] Then, the visual question-answering model is optimized and trained using a training dataset. This allows the trained model to accurately output the answer corresponding to the target question text based on the target image and the target question text, thereby achieving accurate visual question answering and improving the user experience. The training method can be set according to actual needs, such as gradient descent and its derivative optimization algorithms, and genetic algorithms.

[0073] The feature extraction layer can be based on a single overall feature extraction network that can be used for both image and text feature extraction; or it can be based on one branch of a two-branch feature extraction network for image feature extraction and the other branch for text feature extraction. This embodiment does not impose specific limitations on this. It should be noted that when the feature extraction layer contains two branch feature extraction networks, the structures of the two branch feature extraction networks can be the same or different. For example, both can be built based on convolutional networks, residual networks, recurrent networks, etc., or one can be built based on a residual network and the other based on a recurrent network. This embodiment does not impose specific limitations on this.

[0074] like Figure 2As shown, after obtaining the target image and the target question text, the target image can be input into the feature extraction layer, which will extract image features from the target image to obtain image features; at the same time, the target question text can be input into the feature extraction layer, which will extract text features from the target image to obtain text features.

[0075] Step 102: Input the image features and the text features into the graph attention neural network layer in the visual question answering model to obtain semantic relationship features;

[0076] The graph attention neural network layer can be built based on one or more multi-head attention networks, and the specific structure can be set according to actual needs.

[0077] like Figure 2 As shown, after obtaining image features and text features, in order to learn the semantic relationship between image features and text features, the input information of the graph attention neural network layer can be obtained based on the image features and text features. The graph attention neural network layer then mines semantic relationship features based on the input information to obtain semantic relationship features.

[0078] Step 103: Input the semantic relationship features into the visual relationship network layer in the visual question answering model to obtain visual relationship features;

[0079] The visual relational network layer uses triples to supplement the visual question answering model for missing prior knowledge. This allows the visual question answering model to extract visual relational features from the fusion features of the question and image (i.e., semantic relational features) when common sense is required to answer questions in specific question scenarios. These features are then added to the visual question answering model as prior semantic knowledge, thereby obtaining richer features for visual question answering and improving the accuracy of the visual question answering results.

[0080] The visual relational network layer can be constructed based on a fully connected layer and a fusion layer. The fusion layer can be constructed based on an MLB (Multimodal Low-rank Bilinear Pooling) model or a graph attention neural network model. This embodiment does not specifically limit the construction of the fusion layer.

[0081] like Figure 2 As shown, after obtaining the semantic relationship features, the semantic relationship features can be input into the visual relationship network layer in the visual question answering model to extract multiple candidate relationship features. After obtaining multiple candidate relationship features, the multiple candidate relationship features are fused with the semantic relationship features to obtain the visual relationship features.

[0082] The fusion method can be fusion using an MLB model or fusion based on a graph attention neural network model, etc. This embodiment does not specifically limit this.

[0083] Step 104: Input the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text;

[0084] The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0085] Optionally, after obtaining the semantic relationship features and visual relationship features, the semantic relationship features and visual relationship features can be input into the answer prediction layer, and the answer prediction layer can directly output the answer corresponding to the target question text based on the semantic relationship features and visual relationship features; or the answer prediction layer can perform one or more fusion processing on the semantic relationship features and visual relationship features, and output the answer corresponding to the target question text based on the fusion processing result. This embodiment does not specifically limit this.

[0086] The visual question answering method based on graph attention neural networks and visual relationships provided in this embodiment employs a feature extraction layer to extract image features of the target image and text features of the target question text. A graph attention neural network layer is then used to delve into the interaction between image and text features to obtain semantic relationship features. Finally, a visual relationship network layer is used to extract visual relationships from these semantic relationship features. These acquired visual relationship features are then used as external prior knowledge to supplement the visual question answering features. This allows the visual relationship model to not only acquire rich visual question answering features, but also to possess not only semantic relationship features between the image and the question text, but also prior knowledge. Based on these visual question answering features, the answer corresponding to the target question text can be accurately inferred.

[0087] In some embodiments, step 102, inputting the image features and the text features into the graph attention neural network layer of the visual question answering model to obtain semantic relationship features, includes:

[0088] Each region feature in the image features is concatenated with the text features to obtain the concatenation result corresponding to each region feature;

[0089] The concatenation result corresponding to the features of each region is used as the first node of the graph attention neural network layer.

[0090] Based on the graph attention neural network layer, entity relationships between multiple first nodes are extracted to obtain entity relationship features, and the entity relationship features are fused with the text features to obtain the semantic relationship features.

[0091] Among them, the image features include multiple regional features, such as 14×14 regional features.

[0092] A graph attention neural network layer can contain two graph attention neural network layers, one pooling layer, and a feature fusion layer for semantic relation feature extraction.

[0093] Optionally, if image features are obtained, each region feature in the image features can be concatenated with the text features. The specific calculation formula is as follows:

[0094] v' i =W1(v i ||q);

[0095] Among them, v i Let v' be the feature of the i-th region, q be the text feature, || be the concatenation operation, W1 be the parameter matrix, and v' be the text feature. i ∈R 14 ×14×2048 This is the splicing result corresponding to the feature of the i-th region.

[0096] After obtaining the concatenation result corresponding to each region feature, the concatenated image features are input into the graph attention neural network layer. Each region feature's concatenation result serves as a node in the graph attention neural network layer. For example, if there are 14×14 region features, there are 14×14 nodes in the graph attention neural network layer. The feature scale of each node can be 2048. After two layers of graph attention neural network operations, a feature vector suitable for the target question text can be extracted from the image features. This feature vector still has a scale of 14×14×2048. In the first layer of GAT (Graph Attention Network), the size of the dimension transformation matrix is ​​2048×4096, and in the second layer of GAT, the size of the dimension transformation matrix is ​​4096×2048. Therefore, after two layers of dimension transformation, the dimension of the extracted feature vector does not change compared to before the feature transformation, but a non-linear factor is added, improving the learning ability. After two layers of GAT feature extraction, pooling can be performed on the feature vector to obtain the entity relationship feature P. The specific calculation formula is as follows:

[0097]

[0098] in, is the feature vector obtained after two layers of GAT feature extraction; AvgPooling(·) is the average pooling operation; p is the entity relation feature.

[0099] After feature extraction and pooling operations using a two-layer graph attention neural network, the entity relationship feature p is extracted. Then, the entity relationship feature p is fused with the text feature q to obtain the final semantic relationship feature J. The specific calculation formula is as follows:

[0100] J = f θ (p,q);

[0101] Where f is the feature fusion model of the feature fusion layer, and θ is the model parameter of the feature fusion layer. The feature fusion model can use a non-linear fusion method to fuse entity relationship features p and text features q, with the specific formula as follows:

[0102] J = W2p⊙q;

[0103] Where ⊙ represents the dot product operation, and W2 is the parameter matrix.

[0104] In this embodiment, the first step is to concatenate and fuse the features of each region in the target image with the text features, and then use this as a node in the graph attention neural network layer to mine semantic relationship features. After obtaining the image feature vector output by the graph attention neural network that is adapted to the question, it is then concatenated with the text features to obtain the fused features that combine the question text and the image. This process is used to mine the semantic relationships between different regions in the image adapted to the question. The obtained semantic relationship features not only have image features but also specific question text features, thus making the visual question answering results obtained based on semantic relationship features more accurate.

[0105] In some embodiments, step 103, inputting the semantic relationship features into the visual relationship network layer of the visual question answering model to obtain visual relationship features, includes:

[0106] The semantic relationship features are input into the visual relationship network layer;

[0107] Based on the visual relationship network layer, multiple candidate relationship features corresponding to the semantic relationship features are extracted, and each candidate relationship feature is fused with the semantic relationship features to obtain a first fused feature corresponding to each candidate relationship feature. Based on the first fused feature corresponding to each candidate relationship feature, the visual relationship features are obtained.

[0108] Optionally, semantic relationship features are input into a visual relationship network layer. The entity classifier, relationship classifier, and object classifier of the visual relationship network layer extract multiple subjects, relationships, and objects from the semantic relationship features, and select candidate relationship features from the permutations and combinations of subjects, relationships, and customers, choosing the rule with the highest probability mean. Each of the three classifiers uses a three-layer fully connected neural network (Dense) as the classification model, with the specific structure as follows: Figure 3 As shown, when the semantic relationship feature J is input into the first and second layers of the neural network, batch normalization (BN) is used as an optimization strategy to make it easier for the model to learn the patterns in the feature data. At the same time, ReLU is used as the activation function to prevent gradient explosion and gradient vanishing. The output layer uses the softmax activation function to classify the results, and the one with the highest accuracy is the predicted subject, relationship, or object.

[0109] For example, if the number of candidate relation features is 10, then the top 3 subjects, objects and relations with the highest probabilities are arranged and combined, resulting in 3×3×3=27 relation features. Then, the top 10 combinations with the highest average probability are selected as candidate relation features.

[0110] For the 10 selected candidate relation features, these relation features need to be used as semantic knowledge to supplement the model. The candidate relation features can be represented as M = [t1, t2, ..., t10], and each candidate relation feature consists of a subject, a relation, and an object, i.e., ti = (si, li, oi).

[0111] Optionally, in the process of fusing each candidate relation feature with semantic relation features, the feature information of the subject, object, and relation is first embedded into the semantic space to obtain the embedding vector s. i l i and o i Then based on the embedding vector s i l i and o i Obtain the embedded feature r i The method for obtaining the embedding features can be to directly use the average word vector method to obtain the embedding features; or to concatenate the corresponding word vectors and input them into a shallow neural network to obtain the embedding features, etc. This embodiment does not specifically limit this method.

[0112] The corresponding word vectors are concatenated and then input into a shallow neural network to obtain the embedded features r. i The calculation formula is:

[0113] r i =s i +l i +o i ;

[0114] Based on the embedding features, candidate relation features are obtained. These candidate relation features can be represented as R. k ∈[r1,r2,r3,......r 10 ].

[0115] Then, for multiple candidate relation features, each candidate relation feature can be fused with the semantic relation feature to obtain the first fused feature corresponding to each candidate relation feature; and the first fused features corresponding to multiple candidate relation features can be fused again to obtain the visual relation feature. The method of fusion again can be based on the weighted summation of the first fused features corresponding to multiple candidate relation features using the MLB model to obtain the visual relation feature R; or it can be fused using a graph attention neural network model to obtain the visual relation feature R, etc. This embodiment does not specifically limit the method.

[0116] In this embodiment, based on the visual relational network layer, end-to-end learning is performed using a relational database that has undergone text filtering and semantic relational features to extract relational features and fuse features from semantic relational features. Visual relational features that are appropriate for questions and images are extracted to obtain more accurate visual question answering results based on visual relational features.

[0117] In some embodiments, step 103, obtaining the visual relationship feature based on the first fusion feature corresponding to each candidate relationship feature, includes:

[0118] The first fusion feature corresponding to each candidate relationship feature is fused with each candidate relationship feature;

[0119] Based on the fusion results, the visual relationship features are obtained.

[0120] Optionally, an attention mechanism can be used to fuse candidate relation features and semantic relation features to obtain visual relation features. The specific steps are as follows:

[0121] First, the candidate relation features R k The feature is fused with semantic relation features J, and then R is calculated using the softmax function. k The weight parameter c k After obtaining the weight parameter c k Then, with R k The visual relationship feature R is obtained by performing a weighted summation, and the specific calculation formula is as follows:

[0122] f k =σ(R) k ,J);

[0123] c k=softamx(W3f k +b);

[0124]

[0125] Where σ represents the fusion model, specifically the MLB model can be used as the fusion model. k The first fusion feature; c k represents the weight parameters corresponding to the candidate relation features, W3 represents the weight parameters of the MLB model, b represents the bias of the MLB model, and softamx represents the activation function.

[0126] In this embodiment, the selection of candidate relation features uses an attention mechanism-based fusion method that combines candidate relation features R. k The visual relationship features are fused with semantic relationship features J to obtain visual relationship features obtained through the interaction between images and questions, thereby improving the accuracy of visual question answering results.

[0127] In some embodiments, step 103, obtaining the visual relationship feature based on the first fusion feature corresponding to each candidate relationship feature, includes:

[0128] The first fusion feature corresponding to each candidate relation feature is used as the second node of the graph attention neural network layer in the visual relation network layer;

[0129] Based on the graph attention neural network layer in the visual relationship network layer, the entity relationships between multiple second nodes are extracted to obtain the visual relationship features.

[0130] Optionally, for visual relationship feature extraction, a graph attention neural network model can be used to extract the corresponding visual relationship features. This ensures that the obtained visual relationship features not only possess the correlation between candidate relationship features but also the correlation between candidate relationship features and semantic relationship features, thereby compensating for the lack of connections between candidate relationship features and improving the accuracy of visual question answering results. The specific steps are as follows:

[0131] First, the candidate relation features R are... k The candidate relation features are fused with semantic relation features J to give them prior knowledge of the image and the question. The specific calculation formula is as follows:

[0132] S=σ(R k ,J);

[0133] Where S represents the first fused feature corresponding to the first candidate relation feature. σ indicates that the fusion model is a fusion model, which can be a nonlinear model or a linear concatenation model, that is, using a nonlinear or linear concatenation model to fuse each candidate relation feature with the semantic relation feature.

[0134] Then, each first fused feature, which integrates prior knowledge of the image and the question, serves as a second node in the GAT. The graph attention neural network layer in the visual relational network layer extracts the entity relationships between multiple second nodes to obtain visual relational features. The specific calculation formula is as follows:

[0135] R = MaxPooling(GAT) θ (S));

[0136] Where R represents the visual relationship features obtained based on the graph attention neural network layer; θ represents the parameters in the graph attention neural network; MaxPooling is adding a max pooling layer after the graph attention neural network. The visual relationship features obtained in this way not only contain the feature information of the candidate relationship features themselves, but also integrate the connection between them and other candidate relationships. Moreover, the first fused feature S is based on prior knowledge. In a sense, it achieves the effect of the attention mechanism, thereby making the visual question answering results of prediction reasoning more accurate.

[0137] In some embodiments, the answer prediction layer includes an attention network layer and a prediction layer;

[0138] Step 104, which involves inputting the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text, includes:

[0139] Based on the semantic relationship features, obtain the first input information; based on the visual relationship features, obtain the second input information.

[0140] The first input information and the second input information are input into the attention network layer to obtain the second fusion feature; the attention network layer is used to perform multimodal fusion of the semantic relationship feature and the visual relationship feature based on the first input information and the second input information, and output the second fusion feature;

[0141] The second fusion feature is input into the prediction layer to obtain the answer corresponding to the target question text.

[0142] The attention network layer can consist of multiple encoding layers and multiple decoding layers. The encoding layer comprises multiple sets of multi-head attention layers and feedforward neural networks, and the specific number can be set according to actual needs. Based on the attention mechanism, various interaction relationships can be discovered between features, enabling the model to focus on the most important information.

[0143] The attention layer contains multiple input ports, and the input information required by these ports are Q, K, and V, respectively; where Q represents the query vector, K represents the key vector, and V represents the queried vector. In the encoder, if K = Q = V, it is a self-attention mechanism; if K = V is one feature vector and Q is another feature vector, then an attention mechanism is used for fusion.

[0144] Optionally, before outputting the answer corresponding to the target question text, semantic relationship features and visual relationship features can be fused based on an attention network layer to adaptively mine the relationship features between the two types of features, and then obtain a more accurate visual question-answering result based on the fusion result. The fusion steps are as follows:

[0145] First, based on either semantic relation features or visual relation features, determine K and V in the input information of the attention network layer, and based on the other feature, determine Q in the input information of the attention network layer. The specific settings can be configured according to actual needs.

[0146] Then, the input information is fed into the attention network layer, which fuses semantic and visual relationship features based on the input information to obtain a second fused feature. This second fused feature is then fed into the prediction layer, which predicts and outputs the answer corresponding to the target question text based on the second fused feature.

[0147] In some embodiments, inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0148] The first input information is input to the first input port and the second input port of the attention network layer, respectively, and the second input information is input to the third input port of the attention network layer to obtain the second fusion feature.

[0149] Optionally, the semantic relationship features obtained above are labeled as J, and the visual relationship features obtained above are labeled as F, and the visual question answering method in this embodiment is described in detail.

[0150] like Figure 4 As shown, the first input information determined based on semantic relationship features can be used as the input information for the first and second input ports of the attention network layer, and the second input information determined based on visual relationship features can be used as the input information for the third input port of the attention network layer. This input is then fed into the attention network layer to obtain the second fused feature. That is, J and F are fused using the TA(J,F) fusion method based on the Transformer Attention (TA) network, i.e., J = K = V, F = Q, with n encoder layers, outputting the second fused feature Z.

[0151] In some embodiments, inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0152] The second input information is input to the first input port and the second input port of the attention network layer, respectively, and the first input information is input to the third input port of the attention network layer to obtain the second fusion feature.

[0153] like Figure 5 As shown, the second input information determined based on visual relationship features can be used as the input information for the first and second input ports of the attention network layer, while the first input information determined based on semantic relationship features can be used as the input information for the third input port of the attention network layer. This input is then fed into the attention network layer to obtain the second fused feature. That is, the attention network (Transformer Attention, TA) uses the TA(F,J) fusion method, where F = K = V, J = Q, to fuse features. The encoder has n layers and outputs the second fused feature Z.

[0154] In some embodiments, the attention network layer includes a first branch attention network layer and a second branch attention network layer;

[0155] The step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes:

[0156] The first input information is input to the first input port and the second input port of the first branch attention network layer, and the second input information is input to the third input port of the first branch attention network layer to obtain the third fusion feature;

[0157] The second input information is input to the first and second input ports of the second branch attention network layer, and the first input information is input to the third input port of the second branch attention network layer to obtain the fourth fusion feature;

[0158] The second fusion feature is obtained based on the third fusion feature and the fourth fusion feature.

[0159] like Figure 6As shown, the first input information determined based on semantic relationship features can be used as the input information of the first and second input ports of the first branch attention network layer, and the second input information determined based on visual relationship features can be used as the input information of the third input port of the first branch attention network layer to obtain the third fusion feature Z1; the second input information determined based on visual relationship features can be used as the input information of the first and second input ports of the second branch attention network layer, and the first input information determined based on semantic relationship features can be used as the input information of the third input port of the second branch attention network layer to obtain the fourth fusion feature Z2; then the third fusion feature Z1 and the fourth fusion feature Z2 are fused to obtain the second fusion feature Z, and the specific calculation formula is as follows:

[0160] Z = W(Z) 1 ⊙Z 2 );

[0161] To verify the effectiveness of the visual question answering method in this embodiment, a validation dataset was used. Specifically, the dataset used was the R-VQA (Relation Visual Question Answering) dataset, and an open-ended VQA task was used. The top 3000 most frequent answers were used as candidate answers for all experiments. As mentioned above, the semantic relationship feature J can have a dimension of K1×2048, where K1 can be 14×14. F represents the visual relationship feature, and its dimension can be K2×1500, where K2 can be 10, representing 10 candidate relationships.

[0162] The problem text was initialized using a pre-trained word vector model, GloVe, which maps all words into a 300-dimensional feature space. The image dimensions were adjusted to 448×448. The batch size for all experiments was 200. Adam was used as the optimizer during training with an initial learning rate of 0.0003. Dropout = 0.4 and early stopping were used as optimization strategies for all models.

[0163] The attention network layer uses a multi-head attention mechanism, with n=16. All three attention network models were trained for 30 epochs. The training computer configuration was p-100×8, the model parameters were 145M, and the training time was 31 hours. The experimental results are shown in Table 1.

[0164] Table 1. Experimental results of three fusion methods based on attention network layers.

[0165] Fusion method All Y / N Num Other TA(J,F) 62.32％ 88.73％ 44.21％ 55.43％ TA(F,J) 64.87％ 87.32％ 44.37％ 56.93％ TA(F,J)+TA(J,F) 65.23％ 89.33％ 45.72％ 60.22％

[0166] Among them, All, Num, Y / N, and Other represent four different verification tasks. As shown in Table 1, the three fusion methods based on the attention network layer all achieved good experimental results under different verification tasks.

[0167] Among the three different tasks (All, Num, Other), TA(F,J) achieves higher accuracy than TA(J,F). This is because TA(F,J) uses visual relationship features as the dominant feature information, and visual relationship features are superior to semantic relationship features extracted from images in reasoning about the answer. If both visual and semantic relationship features are used as the dominant feature information for answer reasoning, i.e., under TA(F,J)+TA(J,F), the accuracy of answer reasoning is optimal in both cases.

[0168] Table 2 shows the comparison results between the visual question answering model of this embodiment and other models.

[0169] Model All Y / N Num Other Q+I 53.74％ 78.94％ 35.32％ 36.42％ MLB 64.53％ 83.41％ 37.82％ 54.43％ MLAN 64.60％ 83.80％ 40.20％ 53.70％ TA(J,F) 65.11％ 83.91％ 39.41％ 56.99％ TA(F,J) 64.98％ 82.25％ 40.01％ 56.23％ TA(F,J)+TA(J,F) 65.34％ 82.21％ 40.78％ 57.91％

[0170] To further evaluate the superiority of the proposed visual question answering model, the three feature fusion methods based on attention networks mentioned above were used as benchmark models and further tested on the VQA dataset. The experimental parameters were the same as those set in the previous experiments, and the open-ended tasks All, Num, Y / N, and Other were still selected as the evaluation models. The comparison results are shown in Table 2.

[0171] In Table 2, Q+I represents the most basic visual question answering model, which uses only the fused features of the image and the question for answer reasoning. MLB represents a visual question answering model that uses the MLB fusion algorithm to fuse image and text features. The MLAN model is a model that adds external visual concept information to the MLB model. The TA(J,F) model, TA(F,J) model, and TA(F,J)+TA(J,F) model proposed in this embodiment outperform existing models in accuracy for question types "All", "Num", and "Other", with improvements of 0.74%, 0.58%, and 4.21%, respectively, with a more significant improvement on the "Other" question task. Because the visual question answering model in this embodiment uses visual relationships more effectively, it is more suitable for answering some complex questions that require prior knowledge. However, although the model's performance is not outstanding when answering "Y / N" type questions, it is consistent with the performance of existing models. Moreover, under the TA(J,F) model, the visual question answering model in this embodiment has a significant improvement effect, specifically an improvement of 0.11%.

[0172] In summary, graph attention neural networks can uncover missing semantic relationship features in the image adapted to the question. The fusion features supplemented by these features improve the accuracy of answer reasoning by approximately 2% compared to fusion features using attention mechanisms. When using graph attention neural networks to extract visual relationship features, answer accuracy is further improved. In the final fusion feature answer reasoning experiment, the proposed TA(F,J)+TA(J,F) model shows a certain degree of accuracy improvement in answer reasoning compared to existing visual question answering models, with a maximum improvement of 4.21%.

[0173] In some embodiments, the feature extraction layer includes a residual network layer and a recurrent network layer;

[0174] The step 101, which involves inputting the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and inputting the target question text into the feature extraction layer to obtain the text features of the target question text, further includes:

[0175] The target image is input into the residual network layer, and the image features are extracted based on the residual network layer.

[0176] The target question text is input into the recurrent network layer, and semantic features are extracted from the target question text based on the recurrent network layer to obtain the text features.

[0177] The feature extraction layer comprises two branches: a residual network layer for extracting region features from the target image, and a recurrent network layer for extracting semantic features from the target text. The residual network layer can be constructed based on models such as ResNet (Deep Residual Network); the recurrent network layer can be constructed based on models such as GRU (Gated Recurrent Neural Network).

[0178] Optionally, given the target image and the target question text, the target image can be input into a residual network layer. The residual network layer then extracts region features from the target image to obtain multiple region features, which are then used as the image features of the target image. The number and scale of the region features in the image features can be set according to actual needs. For example, if the image features contain 14×14 region features, and each region feature has a scale of 1×2048, the scale of the image features is 14×14×2048.

[0179] Simultaneously, the target question-and-answer text is input into a recurrent network layer, which extracts semantic features from the text to obtain text features. After obtaining the image and text features, steps 102-104 can be executed to obtain the answer corresponding to the target question text in the target image based on the text and image features.

[0180] In this embodiment, a residual network layer is used to extract deep features from the target image to extract more accurate image features, and a recurrent network layer is used to extract contextual semantics from the target question text to extract more accurate text features. Then, based on the accurate image features and text features, the answer corresponding to the target question text can be accurately predicted and inferred.

[0181] The visual question answering device based on graph attention neural network and visual relationship provided by the present invention will be described below. The visual question answering device based on graph attention neural network and visual relationship described below can be referred to in correspondence with the visual question answering method based on graph attention neural network and visual relationship described above.

[0182] like Figure 7 As shown, the device includes a first feature extraction module 701, a second feature extraction module 702, a third feature extraction module 703, and an answer prediction module 704, wherein:

[0183] The first feature extraction module 701 is used to input the target image into the feature extraction layer in the visual question answering model to obtain the image features of the target image, and input the target question text into the feature extraction layer to obtain the text features of the target question text;

[0184] The second feature extraction module 702 is used to input the image features and the text features into the graph attention neural network layer in the visual question answering model to obtain semantic relationship features;

[0185] The third feature extraction module 703 is used to input the semantic relationship features into the visual relationship network layer in the visual question answering model to obtain visual relationship features;

[0186] The answer prediction module 704 is used to input the semantic relationship features and the visual relationship features into the answer prediction layer in the visual question answering model to obtain the answer corresponding to the target question text;

[0187] The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0188] The visual question-answering device based on graph attention neural networks and visual relationships provided in this embodiment extracts image features of the target image and text features of the target question text using a feature extraction layer. It then uses a graph attention neural network layer to deeply explore the interaction between image and text features, obtaining semantic relationship features. Finally, a visual relationship network layer extracts visual relationships from these semantic relationship features, using the acquired visual relationship features as external prior knowledge to supplement the visual question-answering features. This allows the visual relationship model to not only acquire rich visual question-answering features, but also to possess not only semantic relationship features between the image and the question text, but also prior knowledge. Based on these visual question-answering features, the answer corresponding to the target question text can be accurately inferred.

[0189] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The processor 801 can call logical instructions in the memory 803 to execute a visual question answering method based on graph attention neural networks and visual relationships. This method includes: inputting a target image into a feature extraction layer of a visual question answering model to obtain image features of the target image; inputting a target question text into the feature extraction layer to obtain text features of the target question text; inputting the image features and the text features into a graph attention neural network layer of the visual question answering model to obtain semantic relationship features; inputting the semantic relationship features into a visual relationship network layer of the visual question answering model to obtain visual relationship features; and inputting the semantic relationship features and the visual relationship features into an answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text. The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0190] Furthermore, the logical instructions in the aforementioned memory 803 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual question answering method based on graph attention neural network and visual relations provided by the above methods. The method includes: inputting a target image into a feature extraction layer in a visual question answering model to obtain image features of the target image, and inputting a target question text into the feature extraction layer to obtain text features of the target question text; inputting the image features and the text features into a graph attention neural network layer in the visual question answering model to obtain semantic relation features; inputting the semantic relation features into a visual relation network layer in the visual question answering model to obtain visual relation features; and inputting the semantic relation features and the visual relation features into an answer prediction layer in the visual question answering model to obtain the answer corresponding to the target question text. The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the visual question-answering method based on graph attention neural networks and visual relations provided by the above methods. The method includes: inputting a target image into a feature extraction layer in a visual question-answering model to obtain image features of the target image; inputting a target question text into the feature extraction layer to obtain text features of the target question text; inputting the image features and the text features into a graph attention neural network layer in the visual question-answering model to obtain semantic relation features; inputting the semantic relation features into a visual relation network layer in the visual question-answering model to obtain visual relation features; and inputting the semantic relation features and the visual relation features into an answer prediction layer in the visual question-answering model to obtain the answer corresponding to the target question text. The visual question-answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual question answering method based on graph attention neural networks and visual relationships, characterized in that, include: The target image is input into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and the target question text is input into the feature extraction layer to obtain the text features of the target question text; The image features and the text features are input into the graph attention neural network layer of the visual question answering model to obtain semantic relationship features; The semantic relationship features are input into the visual relationship network layer in the visual question answering model to obtain visual relationship features; The step of inputting the semantic relationship features into the visual relationship network layer of the visual question answering model to obtain visual relationship features includes: The semantic relationship features are input into the visual relationship network layer; based on the visual relationship network layer, multiple candidate relationship features corresponding to the semantic relationship features are extracted, and each candidate relationship feature is fused with the semantic relationship features to obtain a first fused feature corresponding to each candidate relationship feature; the visual relationship features are obtained according to the first fused feature corresponding to each candidate relationship feature. The semantic relationship features and the visual relationship features are input into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text. The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

2. The visual question answering method based on graph attention neural network and visual relationship according to claim 1, characterized in that, The step of inputting the image features and the text features into the graph attention neural network layer of the visual question answering model to obtain semantic relationship features includes: Each region feature in the image features is concatenated with the text features to obtain the concatenation result corresponding to each region feature; The concatenation result corresponding to the features of each region is used as the first node of the graph attention neural network layer. Based on the graph attention neural network layer, entity relationships between multiple first nodes are extracted to obtain entity relationship features, and the entity relationship features are fused with the text features to obtain the semantic relationship features.

3. The visual question answering method based on graph attention neural network and visual relationship according to claim 1, characterized in that, The step of obtaining the visual relationship feature based on the first fusion feature corresponding to each candidate relationship feature includes: The first fusion feature corresponding to each candidate relationship feature is fused with each candidate relationship feature; Based on the fusion results, the visual relationship features are obtained.

4. The visual question answering method based on graph attention neural network and visual relationship according to claim 1, characterized in that, The step of obtaining the visual relationship feature based on the first fusion feature corresponding to each candidate relationship feature includes: The first fusion feature corresponding to each candidate relation feature is used as the second node of the graph attention neural network layer in the visual relation network layer; Based on the graph attention neural network layer in the visual relationship network layer, the entity relationships between multiple second nodes are extracted to obtain the visual relationship features.

5. The visual question answering method based on graph attention neural network and visual relationship according to any one of claims 1-4, characterized in that, The answer prediction layer includes an attention network layer and a prediction layer; The step of inputting the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text includes: Based on the semantic relationship features, obtain the first input information; based on the visual relationship features, obtain the second input information. The first input information and the second input information are input into the attention network layer to obtain the second fusion feature; the attention network layer is used to perform multimodal fusion of the semantic relationship feature and the visual relationship feature based on the first input information and the second input information, and output the second fusion feature; The second fusion feature is input into the prediction layer to obtain the answer corresponding to the target question text.

6. The visual question answering method based on graph attention neural network and visual relationship according to claim 5, characterized in that, The step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes: The first input information is input to the first input port and the second input port of the attention network layer, respectively, and the second input information is input to the third input port of the attention network layer to obtain the second fusion feature.

7. The visual question answering method based on graph attention neural network and visual relationship according to claim 5, characterized in that, The step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes: The second input information is input to the first input port and the second input port of the attention network layer, respectively, and the first input information is input to the third input port of the attention network layer to obtain the second fusion feature.

8. The visual question answering method based on graph attention neural network and visual relationship according to claim 5, characterized in that, The attention network layer includes a first branch attention network layer and a second branch attention network layer; The step of inputting the first input information and the second input information into the attention network layer to obtain the second fused feature includes: The first input information is input to the first input port and the second input port of the first branch attention network layer, and the second input information is input to the third input port of the first branch attention network layer to obtain the third fusion feature; The second input information is input to the first and second input ports of the second branch attention network layer, and the first input information is input to the third input port of the second branch attention network layer to obtain the fourth fusion feature; The second fusion feature is obtained based on the third fusion feature and the fourth fusion feature.

9. The visual question answering method based on graph attention neural network and visual relationship according to any one of claims 1-4, characterized in that, The feature extraction layer includes a residual network layer and a recurrent network layer; The process of inputting the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and inputting the target question text into the feature extraction layer to obtain the text features of the target question text, includes: The target image is input into the residual network layer, and the image features are extracted based on the residual network layer. The target question text is input into the recurrent network layer, and semantic features are extracted from the target question text based on the recurrent network layer to obtain the text features.

10. A visual question-answering device based on the relationship between graph attention neural networks and vision, characterized in that, include: The first feature extraction module is used to input the target image into the feature extraction layer of the visual question answering model to obtain the image features of the target image, and input the target question text into the feature extraction layer to obtain the text features of the target question text; The second feature extraction module is used to input the image features and the text features into the graph attention neural network layer in the visual question answering model to obtain semantic relationship features; The third feature extraction module is used to input the semantic relationship features into the visual relationship network layer in the visual question answering model to obtain visual relationship features; The step of inputting the semantic relationship features into the visual relationship network layer of the visual question answering model to obtain visual relationship features includes: The semantic relationship features are input into the visual relationship network layer; based on the visual relationship network layer, multiple candidate relationship features corresponding to the semantic relationship features are extracted, and each candidate relationship feature is fused with the semantic relationship features to obtain a first fused feature corresponding to each candidate relationship feature; the visual relationship features are obtained according to the first fused feature corresponding to each candidate relationship feature. The answer prediction module is used to input the semantic relationship features and the visual relationship features into the answer prediction layer of the visual question answering model to obtain the answer corresponding to the target question text; The visual question answering model is trained based on sample images, sample question texts, and the answers corresponding to the sample question texts.

Images