Visual question answering method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a visual question answering method and device, electronic equipment and a computer readable storage medium, wherein the method comprises: screening a target label related to a target question from labels corresponding to each entity included in a target image; wherein the target question is a question related to the target image; extracting a knowledge text corresponding to the target label from a target knowledge base; performing feature fusion on a visual feature vector of the target image, a question feature vector of the target question, and a knowledge feature vector of the knowledge text through a graph convolution network layer in a visual question answering model to obtain a first fusion feature matrix; performing feature fusion on the first fusion feature matrix and the knowledge feature vector through an attention module in the visual question answering model to obtain a second fusion feature matrix; and predicting a target answer to the target question according to the first fusion feature matrix and the second fusion feature matrix. Through the method, the accuracy of the visual question answering result can be improved.

Description

Visual question answering methods, devices, electronic devices, and computer-readable storage media Technical Field

[0001] This application relates to the field of visual question answering technology, and in particular to a visual question answering method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Visual question answering is a task where, given an image and a question related to that image, a visual question answering model predicts the answer to the question. Visual question answering has a wide range of applications, such as early childhood education, human-computer interaction, and helping blind people access external information.

[0003] Current visual question answering methods only extract image features and question features, and then fuse these features to predict the answer. However, because they only use image and question features for answer prediction, the information richness is insufficient, leading to low accuracy in visual question answering results. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a visual question answering method, apparatus, electronic device and computer-readable storage medium to improve the accuracy of visual question answering results.

[0005] In a first aspect, embodiments of this application provide a visual question-answering method, including:

[0006] From the labels corresponding to each entity contained in the target image, target labels related to the target question are filtered out; wherein, the target question is a question related to the target image.

[0007] Extract the knowledge text corresponding to the target tag from the target knowledge base;

[0008] The visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text are fused using the graph convolutional network layer in the visual question answering model to obtain a first fused feature matrix;

[0009] The first fusion feature matrix and the knowledge feature vector are fused using the attention module in the visual question answering model to obtain the second fusion feature matrix;

[0010] Based on the first fusion feature matrix and the second fusion feature matrix, predict the target answer to the target question.

[0011] In conjunction with the first aspect, this application provides a first possible implementation of the first aspect, wherein filtering target tags related to the target problem from the tags corresponding to each entity contained in the target image includes:

[0012] For the target image given in the visual question answering task, extract each entity contained in the target image to determine the label corresponding to each entity; the label includes entity category label and / or entity attribute label;

[0013] Based on the target question related to the target image given by the visual question answering task, the target labels related to the target question are filtered from the various labels.

[0014] In conjunction with the first aspect, this application provides a second possible implementation of the first aspect, wherein the method further includes:

[0015] Image recognition is performed on the entities contained in the target image to select each entity in the target image using a feature extraction bounding box;

[0016] The target image is segmented using the feature extraction boxes corresponding to each entity in the target image to obtain an entity segmentation image for each entity.

[0017] For each entity segmentation image, the entity segmentation image is converted into an entity feature vector;

[0018] The visual feature vector of the target image is generated based on the entity feature vectors corresponding to all entities in the target image.

[0019] In conjunction with the first aspect, this application provides a third possible implementation of the first aspect, wherein the method further includes:

[0020] Extract the first keyword from the target question to determine the first keyword vector corresponding to each first keyword;

[0021] Input all the first keyword vectors corresponding to the target problem into a long short-term memory artificial neural network to obtain the problem feature vector of the target problem.

[0022] In conjunction with the second possible implementation of the first aspect, this application provides a fourth possible implementation of the first aspect, wherein the graph convolutional network layer includes a first number of sequentially connected hidden layers; the feature fusion of the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain a first fusion feature matrix includes:

[0023] For each entity feature vector corresponding to the target image, the entity feature vector is concatenated with the problem feature vector to obtain the first concatenated feature vector corresponding to the entity feature vector;

[0024] For each of the first concatenated feature vectors, the first concatenated feature vector is input into the first nonlinear activation function to obtain the joint feature vector corresponding to the first concatenated feature vector;

[0025] All the joint feature vectors are concatenated to obtain a concatenation matrix, and an undirected graph adjacency matrix is ​​generated based on the concatenation matrix.

[0026] The dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector are aligned and corrected to make them have the same dimensions.

[0027] The knowledge feature vector, question feature vector, and visual feature vector with the same dimension are concatenated to obtain the second concatenated feature matrix;

[0028] For the first hidden layer in the graph convolutional network layer, the undirected graph adjacency matrix and the second concatenated feature matrix are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0029] For all hidden layers except the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix of the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0030] The output fusion feature matrix of the last hidden layer is determined as the first fusion feature matrix.

[0031] In conjunction with the first aspect, this application provides a fifth possible implementation of the first aspect, wherein the attention module includes a second number of attention units; the step of fusing the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain a second fusion feature matrix includes:

[0032] In each attention unit, the first fused feature matrix and the knowledge feature vector are calculated using the scaling dot product attention formula to obtain the head vector corresponding to that attention unit:

[0033] The head vectors corresponding to all attention units are superimposed to obtain a first superimposed vector;

[0034] The first superimposed vector is normalized to obtain a normalized vector;

[0035] Based on the dimension of the knowledge feature vector, the dimension of the normalized vector is aligned and corrected so that the dimension of the normalized vector is the same as the dimension of the knowledge feature vector.

[0036] The normalized vectors of the same dimension are superimposed with the knowledge feature vectors to obtain a second superimposed vector;

[0037] The second superimposed vector is normalized to obtain the second fused feature matrix.

[0038] In conjunction with the first aspect, this application provides a sixth possible implementation of the first aspect, wherein the visual question-answering model pre-sets multiple candidate answers corresponding to the target question; the step of predicting the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix includes:

[0039] The first fusion feature matrix and the second fusion feature matrix are input into the classification layer of the visual question answering model to obtain the probability corresponding to each candidate answer;

[0040] The candidate answer with the highest probability is determined as the target answer to the target question.

[0041] Secondly, embodiments of this application also provide a visual question-answering device, comprising:

[0042] The filtering module is used to filter out target labels related to the target question from the labels corresponding to each entity contained in the target image; wherein, the target question is a question related to the target image;

[0043] The first extraction module is used to extract the knowledge text corresponding to the target tag from the target knowledge base;

[0044] The first fusion module is used to fuse the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain a first fusion feature matrix;

[0045] The second fusion module is used to perform feature fusion on the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain the second fusion feature matrix.

[0046] The prediction module is used to predict the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix.

[0047] In conjunction with the second aspect, this application provides a first possible implementation of the second aspect, wherein the filtering module, when filtering target tags related to the target problem from the tags corresponding to each entity contained in the target image, is specifically used for:

[0048] For the target image given in the visual question answering task, extract each entity contained in the target image to determine the label corresponding to each entity; the label includes entity category label and / or entity attribute label;

[0049] Based on the target question related to the target image given by the visual question answering task, the target labels related to the target question are filtered from the various labels.

[0050] In conjunction with the second aspect, embodiments of this application provide a second possible implementation of the second aspect, which further includes:

[0051] The recognition module is used to perform image recognition on entities contained in the target image, so as to select each entity in the target image using a feature extraction box;

[0052] The segmentation module is used to segment the target image using the feature extraction boxes corresponding to each entity in the target image, so as to obtain an entity segmentation image corresponding to each entity;

[0053] The conversion module is used to convert each entity segmentation image into an entity feature vector.

[0054] The generation module is used to generate the visual feature vector of the target image based on the entity feature vectors corresponding to all entities in the target image.

[0055] In conjunction with the second aspect, this application provides a third possible implementation of the second aspect, which further includes:

[0056] The second extraction module is used to extract the first keyword from the target question to determine the first keyword vector corresponding to each first keyword.

[0057] The input module is used to input all the first keyword vectors corresponding to the target question into the long short-term memory artificial neural network to obtain the question feature vector of the target question.

[0058] In conjunction with the second possible implementation of the second aspect, this application provides a fourth possible implementation of the second aspect, wherein the graph convolutional network layer includes a first number of sequentially connected hidden layers; the first fusion module, when used to perform feature fusion on the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain a first fusion feature matrix, is specifically used for:

[0059] For each entity feature vector corresponding to the target image, the entity feature vector is concatenated with the problem feature vector to obtain the first concatenated feature vector corresponding to the entity feature vector;

[0060] For each of the first concatenated feature vectors, the first concatenated feature vector is input into the first nonlinear activation function to obtain the joint feature vector corresponding to the first concatenated feature vector;

[0061] All the joint feature vectors are concatenated to obtain a concatenation matrix, and an undirected graph adjacency matrix is ​​generated based on the concatenation matrix.

[0062] The dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector are aligned and corrected to make them have the same dimensions.

[0063] The knowledge feature vector, question feature vector, and visual feature vector with the same dimension are concatenated to obtain the second concatenated feature matrix;

[0064] For the first hidden layer in the graph convolutional network layer, the undirected graph adjacency matrix and the second concatenated feature matrix are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0065] For all hidden layers except the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix of the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0066] The output fusion feature matrix of the last hidden layer is determined as the first fusion feature matrix.

[0067] In conjunction with the second aspect, this application provides a fifth possible implementation of the second aspect, wherein the attention module includes a second number of attention units; the second fusion module, when used to perform feature fusion on the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain a second fusion feature matrix, is specifically used for:

[0068] In each attention unit, the first fused feature matrix and the knowledge feature vector are calculated using the scaling dot product attention formula to obtain the head vector corresponding to that attention unit:

[0069] The head vectors corresponding to all attention units are superimposed to obtain a first superimposed vector;

[0070] The first superimposed vector is normalized to obtain a normalized vector;

[0071] Based on the dimension of the knowledge feature vector, the dimension of the normalized vector is aligned and corrected so that the dimension of the normalized vector is the same as the dimension of the knowledge feature vector.

[0072] The normalized vectors of the same dimension are superimposed with the knowledge feature vectors to obtain a second superimposed vector;

[0073] The second superimposed vector is normalized to obtain the second fused feature matrix.

[0074] In conjunction with the second aspect, this application provides a sixth possible implementation of the second aspect, wherein the visual question-answering model pre-sets multiple candidate answers corresponding to the target question; and the prediction module, when used to predict the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix, is specifically used for:

[0075] The first fusion feature matrix and the second fusion feature matrix are input into the classification layer of the visual question answering model to obtain the probability corresponding to each candidate answer;

[0076] The candidate answer with the highest probability is determined as the target answer to the target question.

[0077] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in any of the possible implementations of the first aspect described above are performed.

[0078] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps in any of the possible implementations of the first aspect described above.

[0079] This application provides a visual question-answering method, apparatus, electronic device, and computer-readable storage medium. The method involves: filtering target tags related to a target question from the tags corresponding to each entity in a target image; the target question being a question related to the target image; extracting knowledge text corresponding to the target tags from a target knowledge base; fusing the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through a graph convolutional network layer in a visual question-answering model to obtain a first fused feature matrix; fusing the first fused feature matrix and the knowledge feature vector through an attention module in the visual question-answering model to obtain a second fused feature matrix; and predicting the target answer to the target question based on the first and second fused feature matrices. Compared to existing technologies that only fuse image feature vectors and question feature vectors and use the fused feature vectors for answer prediction, this embodiment not only fuses image feature vectors and question feature vectors but also fuses knowledge feature vectors corresponding to entities related to the target question contained in the target image. This results in richer information being incorporated during answer prediction, thereby improving the accuracy of the visual question-answering results.

[0080] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0081] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 shows a flowchart of a visual question-answering method provided in an embodiment of this application;

[0083] Figure 2 shows a flowchart of another visual question-answering method provided in an embodiment of this application;

[0084] Figure 3 shows a schematic diagram of the structure of a visual question-answering device provided in an embodiment of this application;

[0085] Figure 4 shows a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0087] In view of the problem that the accuracy of visual question answering result prediction is not high enough in the prior art, this application provides a visual question answering method, apparatus, electronic device, and computer-readable storage medium to improve the accuracy of visual question answering result prediction. The following embodiments describe these embodiments.

[0088] Example 1:

[0089] To facilitate understanding of this embodiment, a visual question-answering method disclosed in this application will first be described in detail. Figure 1 shows a flowchart of a visual question-answering method provided in this application embodiment, as shown in Figure 1, including the following steps S101-S106:

[0090] S101: Filter out target labels related to the target problem from the labels corresponding to each entity contained in the target image; wherein, the target problem is a problem related to the target image.

[0091] In this embodiment, the visual question answering task provides a target image and a target question associated with that image. For example, the target image might contain a woman holding a frisbee, and the target question could be: What color is the frisbee?

[0092] An entity refers to a physical object contained within the target image. Specifically, it can be people, objects, etc., such as vehicles, roads, trees, fruits, etc. A label is a tag used to describe the entity; each entity corresponds to at least one label.

[0093] For example, when entity A in the target image is a white iron bicycle, the labels for entity A could be "bicycle," "iron product," "white," etc. When the target question is "What is the material of the bicycle?", the target labels related to the target question can be "bicycle" and "iron product" from the labels (bicycle, iron product, white) corresponding to the target image.

[0094] In one possible embodiment, when performing step S101, the following steps can be specifically performed:

[0095] S1011: For a given target image in a visual question answering task, extract each entity contained in the target image to determine the corresponding label for each entity; the label includes entity category label and / or entity attribute label.

[0096] S1012: Based on the target question related to the target image given in the visual question answering task, filter out the target labels related to the target question from each label.

[0097] In this embodiment, the entity category label indicates the category to which the entity belongs, specifically the name of the entity. The entity attribute label identifies the attributes of the entity, specifically its color, material, size, etc. Continuing with the above embodiment, the entity category label for entity A is "bicycle," and the entity attribute labels for entity A are "iron products" and "white."

[0098] For example, if the target image contains entity B "a red apple" in addition to entity A, then the entities extracted from the target image are "bicycle" and "apple". The tags for "bicycle" include "bicycle, iron, white"; the tags for "apple" include "apple, red, one". When the target question is how many apples are contained in the target image, the target tags can be "apple, one".

[0099] S102: Extract the knowledge text corresponding to the target tag from the target knowledge base.

[0100] The target knowledge base contains the knowledge text corresponding to each entity. The knowledge text is used to represent the entity's attributes, common sense, and capabilities. For example, when the target label is "hat," the knowledge text corresponding to that target label can include the hat's attributes, uses, functions, etc.

[0101] Specifically, based on the target label, the knowledge text corresponding to the entity corresponding to the target label is extracted from the knowledge text corresponding to each entity contained in the target knowledge base.

[0102] S103: The visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text are fused through the graph convolutional network layer in the visual question answering model to obtain the first fused feature matrix.

[0103] Visual feature vectors are used to characterize the content features contained in the target image, question feature vectors are used to characterize the meaning features of the target question, and knowledge feature vectors are used to characterize the meaning features of the knowledge text.

[0104] In one possible implementation, before performing step S103, the visual feature vector of the target image can be determined in the following way:

[0105] S1061: Perform image recognition on entities contained in the target image to select each entity in the target image using a feature extraction box.

[0106] In this embodiment, the feature extraction box can be a rectangle, which is used to select each entity bounding box in the target image. Each entity corresponds to one feature extraction box. When there are many entities in the target image, the feature extraction boxes may overlap.

[0107] S1062: The target image is segmented using the feature extraction boxes corresponding to each entity in the target image to obtain the entity segmentation image corresponding to each entity.

[0108] The target image is segmented according to the bounding box of the feature extraction box, and the image inside the feature extraction box is used as the entity segmentation image corresponding to the entity. In this embodiment, the entity segmentation image is a part of the target image, and each entity corresponds to its own entity segmentation image.

[0109] S1063: For each entity segmentation image, convert the entity segmentation image into an entity feature vector.

[0110] Each entity segmentation image has its own entity feature vector, which is used to characterize the image content contained in its corresponding entity segmentation image.

[0111] S1064: Generate the visual feature vector of the target image based on the entity feature vectors corresponding to all entities in the target image.

[0112] In this embodiment, the entity feature vectors corresponding to all entities in the target image can be concatenated to obtain a visual feature vector.

[0113] In one possible implementation, before performing step S103, the problem feature vector of the target problem can be determined in the following way:

[0114] S1071: Extract the first keyword from the target question to determine the first keyword vector corresponding to each first keyword.

[0115] The target question is treated as text, containing at least one primary keyword. Each primary keyword is extracted from the target question, and for each primary keyword, a corresponding primary keyword vector is determined to obtain all primary keyword vectors for the target question. The primary keyword vector represents the meaning of the word associated with its corresponding primary keyword.

[0116] S1072: Input all the first keyword vectors corresponding to the target problem into the Long Short-Term Memory Artificial Neural Network (LSTM) to obtain the problem feature vector of the target problem.

[0117] In one possible implementation, before performing step S103, the knowledge feature vector of the knowledge text can be determined in the following way:

[0118] S1081: Extract the second keyword from the knowledge text to determine the second keyword vector corresponding to each second keyword.

[0119] In this embodiment, the knowledge text contains at least one second keyword. Each second keyword is extracted from the knowledge text, and for each second keyword, a corresponding second keyword vector is determined to obtain all second keyword vectors corresponding to the knowledge text. The second keyword vector is used to represent the meaning of the word corresponding to its second keyword.

[0120] S1082: Input all the second keyword vectors corresponding to the knowledge text into the long short-term memory artificial neural network to obtain the knowledge feature vectors corresponding to the knowledge text.

[0121] In one possible implementation, the graph convolutional network layer includes a first number of sequentially connected hidden layers, wherein the first number is a positive integer greater than 1; when executing step S103, it can be specifically implemented through the following steps S1031-S1038:

[0122] S1031: For each entity feature vector corresponding to the target image, concatenate the entity feature vector with the problem feature vector to obtain the first concatenated feature vector corresponding to the entity feature vector.

[0123] In this embodiment, when the entity feature vector and the question feature vector have the same vector dimension, they are directly concatenated. When the entity feature vector and the question feature vector have different vector dimensions, the vector dimensions of the entity feature vector and / or the question feature vector are aligned and corrected by padding with zeros, so that the entity feature vector and the question feature vector have the same vector dimension. Then, the entity feature vectors with the same vector dimension are concatenated with the question feature vectors. In this embodiment, the number of first concatenated feature vectors is the same as the number of entity feature vectors.

[0124] S1032: For each first concatenated feature vector, input the first concatenated feature vector into the first nonlinear activation function (ReLU) to obtain the joint feature vector corresponding to the first concatenated feature vector.

[0125] The number of concatenated feature vectors is the same as the number of joint feature vectors.

[0126] S1033: Concatenate all joint eigenvectors to obtain a concatenation matrix, and generate an undirected graph adjacency matrix based on the concatenation matrix.

[0127] In this embodiment, the adjacency matrix of the undirected graph can be generated using the following formula:

[0128] A = EE T

[0129] Where A represents the adjacency matrix of the undirected graph, E represents the concatenation matrix, and E T This represents the transpose of the concatenated matrix.

[0130] S1034: Align and correct the dimensions of the knowledge feature vector, problem feature vector, and visual feature vector to make them have the same dimensions.

[0131] In this embodiment, if the dimension of the knowledge feature vector (i.e., the vector dimension) is 5×6, the dimension of the question feature vector is 4×3, and the dimension of the visual feature vector is 4×9, in this application, the dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector can be aligned and corrected by padding with zeros. The dimensions of the aligned and corrected knowledge feature vector, the question feature vector, and the visual feature vector can be 5×9.

[0132] S1035: Concatenate the knowledge feature vector, question feature vector, and visual feature vector with the same dimension to obtain the second concatenated feature matrix.

[0133] For example, if the dimensions of the aligned and corrected knowledge feature vector, problem feature vector, and visual feature vector can be 5×9, then the dimensions of the second concatenated feature matrix can be 15×9 or 5×27.

[0134] S1036: For the first hidden layer in the graph convolutional network, the undirected graph adjacency matrix and the second concatenated feature matrix are input into the second non-linear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0135] S1037: For hidden layers other than the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix of the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0136] In this embodiment, the formula for the second nonlinear activation function is:

[0137]

[0138] Where α(·) represents the second nonlinear activation function (GELU), n represents the nth hidden layer, and n is a positive integer greater than or equal to 1 and less than or equal to the first number, H n Let H be the output fusion feature matrix of the nth hidden layer. When n is 1, H n-1 (i.e., H0) is the second concatenated feature matrix. When n is greater than 1, H n-1 This is the output fusion feature matrix of the (n-1)th hidden layer.

[0139]

[0140] A denotes the adjacency matrix of an undirected graph, and I is the identity matrix. yes The degree matrix of the diagonal nodes, W n These are the trainable parameters in the nth hidden layer.

[0141] S1038: Determine the output fusion feature matrix of the last hidden layer as the first fusion feature matrix.

[0142] The last hidden layer refers to the Nth hidden layer, where N is the first number and the first fusion feature matrix is ​​H. N , which is the output fusion feature matrix of the Nth hidden layer.

[0143] S104: The first fusion feature matrix and the knowledge feature vector are fused using the attention module in the visual question answering model to obtain the second fusion feature matrix.

[0144] In one possible implementation, the attention module includes a second number of attention units, where the second number is a positive integer greater than 1. Specifically, when executing step S104, this can be achieved through the following steps S1041-S1046:

[0145] S1041: In each attention unit, the first fusion feature matrix and knowledge feature vector are calculated using the scaling dot product attention formula to obtain the head vector corresponding to the attention unit.

[0146] In this embodiment, for each attention unit, the first fused feature matrix and the knowledge feature vector are calculated using the scaling dot product attention calculation formula in that attention unit. The scaling dot product attention calculation formula is as follows:

[0147] head i =Attention(qP) i Q kP i K ,vP i V )

[0148] Here, the first fusion feature matrix is ​​used as the key matrix K and the value matrix V, and the knowledge feature matrix is ​​used as the query matrix Q. i represents the i-th attention unit, and head... i P represents the head vector output by the i-th attention unit; q is a submatrix of the query matrix Q, specifically the i-th column submatrix of the query matrix Q; i Q Learnable parameters assigned to matrix q in the i-th attention unit; k is a submatrix of the key matrix K, specifically the i-th column submatrix of the key matrix K; P i K Learnable parameters assigned to matrix k in the i-th attention unit; v is a submatrix of the value matrix V, specifically the i-th column submatrix of the value matrix V; P i V Learnable parameters are assigned to matrix v in the i-th attention unit.

[0149] S1042: Perform vector superposition on the head vectors corresponding to all attention units to obtain the first superposition vector.

[0150] In this embodiment, each attention unit corresponds to its own head vector. The head vectors corresponding to all attention units are then superimposed, which means a second number of head vectors are superimposed.

[0151] S1043: Normalize the first superimposed vector to obtain the normalized vector.

[0152] S1044: Based on the dimension of the knowledge feature vector, the dimension of the normalized vector is aligned and corrected so that the dimension of the normalized vector is the same as the dimension of the knowledge feature vector.

[0153] In this embodiment, the dimension of the normalized vector is aligned and corrected to match the dimension of the knowledge feature vector. Specifically, when the dimension of the normalized vector is smaller than the dimension of the knowledge feature vector, the dimension of the normalized vector can be aligned and corrected by padding with zeros to make the dimension of the normalized vector the same as the dimension of the knowledge feature vector.

[0154] S1045: Superimpose the normalized vectors and knowledge feature vectors of the same dimension to obtain the second superimposed vector.

[0155] S1046: Normalize the second superposition vector to obtain the second fusion feature matrix.

[0156] S105: Based on the first fusion feature matrix and the second fusion feature matrix, predict the target answer to the target question.

[0157] In one possible implementation, the visual question-answering model pre-sets multiple candidate answers corresponding to the target question; Figure 2 shows a flowchart of another visual question-answering method provided by an embodiment of this application. As shown in Figure 2, when executing step S105, it can be achieved through the following steps S1051-S1052:

[0158] S1051: Input the first fusion feature matrix and the second fusion feature matrix into the classification layer in the visual question answering model to obtain the probability corresponding to each candidate answer.

[0159] S1052: Determine the candidate answer with the highest probability as the target answer to the target question.

[0160] In this embodiment, the probability corresponding to a candidate answer represents the likelihood that the candidate answer is the correct answer; the higher the probability, the greater the likelihood that the candidate answer is the correct answer; the lower the probability, the less likely that the candidate answer is the correct answer.

[0161] Example 2:

[0162] Based on the same technical concept, this application also provides a visual question-answering device. Figure 3 shows a schematic diagram of the structure of a visual question-answering device provided in this application embodiment. As shown in Figure 3, the device includes:

[0163] The filtering module 301 is used to filter out target labels related to the target question from the labels corresponding to each entity contained in the target image; wherein, the target question is a question related to the target image;

[0164] The first extraction module 302 is used to extract the knowledge text corresponding to the target tag from the target knowledge base;

[0165] The first fusion module 303 is used to fuse the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain a first fusion feature matrix;

[0166] The second fusion module 304 is used to perform feature fusion on the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain the second fusion feature matrix.

[0167] The prediction module 305 is used to predict the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix.

[0168] Optionally, when the filtering module 301 filters out target labels related to the target question from the labels corresponding to each entity contained in the target image, it is specifically used for:

[0169] For the target image given in the visual question answering task, extract each entity contained in the target image to determine the label corresponding to each entity; the label includes entity category label and / or entity attribute label;

[0170] Based on the target question related to the target image given by the visual question answering task, the target labels related to the target question are filtered from the various labels.

[0171] Optional, also includes:

[0172] The recognition module is used to perform image recognition on entities contained in the target image, so as to select each entity in the target image using a feature extraction box;

[0173] The segmentation module is used to segment the target image using the feature extraction boxes corresponding to each entity in the target image, so as to obtain an entity segmentation image corresponding to each entity;

[0174] The conversion module is used to convert each entity segmentation image into an entity feature vector.

[0175] The generation module is used to generate the visual feature vector of the target image based on the entity feature vectors corresponding to all entities in the target image.

[0176] Optional, also includes:

[0177] The second extraction module is used to extract the first keyword from the target question to determine the first keyword vector corresponding to each first keyword.

[0178] The input module is used to input all the first keyword vectors corresponding to the target question into the long short-term memory artificial neural network to obtain the question feature vector of the target question.

[0179] Optionally, the graph convolutional network layer includes a first number of sequentially connected hidden layers; the first fusion module 303, when used to perform feature fusion on the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain the first fusion feature matrix, is specifically used for:

[0180] For each entity feature vector corresponding to the target image, the entity feature vector is concatenated with the problem feature vector to obtain the first concatenated feature vector corresponding to the entity feature vector;

[0181] For each of the first concatenated feature vectors, the first concatenated feature vector is input into the first nonlinear activation function to obtain the joint feature vector corresponding to the first concatenated feature vector;

[0182] All the joint feature vectors are concatenated to obtain a concatenation matrix, and an undirected graph adjacency matrix is ​​generated based on the concatenation matrix.

[0183] The dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector are aligned and corrected to make them have the same dimensions.

[0184] The knowledge feature vector, question feature vector, and visual feature vector with the same dimension are concatenated to obtain the second concatenated feature matrix;

[0185] For the first hidden layer in the graph convolutional network layer, the undirected graph adjacency matrix and the second concatenated feature matrix are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0186] For all hidden layers except the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix of the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output.

[0187] The output fusion feature matrix of the last hidden layer is determined as the first fusion feature matrix.

[0188] Optionally, the attention module includes a second number of attention units; the second fusion module 304, when used to perform feature fusion on the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain the second fusion feature matrix, is specifically used for:

[0189] In each attention unit, the first fused feature matrix and the knowledge feature vector are calculated using the scaling dot product attention formula to obtain the head vector corresponding to that attention unit:

[0190] The head vectors corresponding to all attention units are superimposed to obtain a first superimposed vector;

[0191] The first superimposed vector is normalized to obtain a normalized vector;

[0192] Based on the dimension of the knowledge feature vector, the dimension of the normalized vector is aligned and corrected so that the dimension of the normalized vector is the same as the dimension of the knowledge feature vector.

[0193] The normalized vectors of the same dimension are superimposed with the knowledge feature vectors to obtain a second superimposed vector;

[0194] The second superimposed vector is normalized to obtain the second fused feature matrix.

[0195] Optionally, the visual question-answering model pre-sets multiple candidate answers corresponding to the target question; when the prediction module 305 predicts the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix, it is specifically used for:

[0196] The first fusion feature matrix and the second fusion feature matrix are input into the classification layer of the visual question answering model to obtain the probability corresponding to each candidate answer;

[0197] The candidate answer with the highest probability is determined as the target answer to the target question.

[0198] Example 3:

[0199] Based on the same technical concept, this application also provides an electronic device. Figure 4 shows a schematic diagram of the structure of an electronic device provided in this application. As shown in Figure 4, the electronic device 400 includes: a processor 401, a memory 402 and a bus 403. The memory stores machine-readable instructions that can be executed by the processor. When the electronic device is running, the processor 401 and the memory 402 communicate through the bus 403. The processor 401 executes the machine-readable instructions to perform the steps of the method described in Embodiment 1.

[0200] Example 4:

[0201] Based on the same technical concept, Embodiment 4 of this application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the steps of the method described in Embodiment 1.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, electronic devices, and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0206] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A visual question-answering method, characterized in that, include: From the labels corresponding to each entity in the target image, target labels related to the target question are selected; wherein, the target question is a question related to the target image; knowledge text corresponding to the target labels is extracted from the target knowledge base; the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text are fused through a graph convolutional network layer in the visual question answering model to obtain a first fused feature matrix; the first fused feature matrix and the knowledge feature vector are fused through an attention module in the visual question answering model to obtain a second fused feature matrix; the target answer to the target question is predicted based on the first fused feature matrix and the second fused feature matrix; the graph convolutional network layer includes a first number of sequentially connected hidden layers; the target image Each entity in the model corresponds to its own entity segmentation image, and each entity segmentation image corresponds to its own entity feature vector. The step of fusing the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through a graph convolutional network layer in the visual question answering model to obtain a first fused feature matrix includes: for each entity feature vector corresponding to the target image, concatenating the entity feature vector with the question feature vector to obtain a first concatenated feature vector corresponding to the entity feature vector; for each first concatenated feature vector, inputting the first concatenated feature vector into a first nonlinear activation function to obtain a joint feature vector corresponding to the first concatenated feature vector; concatenating all the joint feature vectors to obtain a concatenation matrix, and generating an undirected graph adjacency matrix using the following formula: Where A represents the adjacency matrix of the undirected graph, E represents the concatenation matrix, and E T This represents the transpose of the concatenation matrix; the dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector are aligned and corrected to ensure that they have the same dimensions; the knowledge feature vector, the question feature vector, and the visual feature vector with the same dimensions are concatenated to obtain the second concatenation feature matrix; for the first hidden layer in the graph convolutional network layer, the undirected graph adjacency matrix and the second concatenation feature matrix are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output; for other hidden layers besides the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix output by the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output; the formula for the second nonlinear activation function is: Where α(·) represents the second nonlinear activation function, n represents the nth hidden layer, and n is a positive integer greater than or equal to 1 and less than or equal to the first number, H n H is the output fusion feature matrix of the nth hidden layer; when n is 1, H n-1 This is the second concatenated feature matrix; when n is greater than 1, H n-1 The output fusion feature matrix is ​​the output of the (n-1)th hidden layer; A denotes the adjacency matrix of an undirected graph, and I is the identity matrix. yes The degree matrix of the diagonal nodes, W n These are the trainable parameters in the nth hidden layer; the output fusion feature matrix of the last hidden layer is determined as the first fusion feature matrix.

2. The method according to claim 1, characterized in that, The step of filtering target tags related to the target question from the tags corresponding to each entity contained in the target image includes: extracting each entity contained in the target image given by the visual question answering task to determine the tag corresponding to each entity; the tag includes entity category tag and / or entity attribute tag; and filtering target tags related to the target question from the tags according to the target question related to the target image given by the visual question answering task.

3. The method according to claim 1, characterized in that, The method further includes: performing image recognition on entities contained in the target image to select each entity in the target image using a feature extraction box; segmenting the target image using the feature extraction boxes corresponding to each entity in the target image to obtain an entity segmentation image corresponding to each entity; converting each entity segmentation image into an entity feature vector; and generating the visual feature vector of the target image based on the entity feature vectors corresponding to all entities in the target image.

4. The method according to claim 1, characterized in that, The method further includes: extracting a first keyword from the target question to determine a first keyword vector corresponding to each first keyword; and inputting all the first keyword vectors corresponding to the target question into a long short-term memory artificial neural network to obtain the question feature vector of the target question.

5. The method according to claim 1, characterized in that, The attention module includes a second number of attention units; the step of fusing the first fusion feature matrix and the knowledge feature vector through the attention module in the visual question answering model to obtain a second fusion feature matrix includes: in each attention unit, calculating the first fusion feature matrix and the knowledge feature vector using the scaling dot product attention calculation formula to obtain the head vector corresponding to the attention unit; performing vector superposition on the head vectors corresponding to all attention units to obtain a first superposition vector; normalizing the first superposition vector to obtain a normalized vector; aligning and correcting the dimension of the normalized vector based on the dimension of the knowledge feature vector so that the dimension of the normalized vector is the same as the dimension of the knowledge feature vector; superimposing the normalized vector with the knowledge feature vector to obtain a second superposition vector; and normalizing the second superposition vector to obtain the second fusion feature matrix.

6. The method according to claim 1, characterized in that, The visual question answering model pre-sets multiple candidate answers corresponding to the target question; predicting the target answer of the target question based on the first fusion feature matrix and the second fusion feature matrix includes: inputting the first fusion feature matrix and the second fusion feature matrix into the classification layer of the visual question answering model to obtain the probability corresponding to each candidate answer; and determining the candidate answer with the highest probability as the target answer of the target question.

7. A visual question-answering device, characterized in that, include: A filtering module is used to filter out target tags related to the target question from the tags corresponding to each entity contained in the target image; wherein, the target question is a question related to the target image; a first extraction module is used to extract the knowledge text corresponding to the target tags from the target knowledge base; a first fusion module is used to perform feature fusion on the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through a graph convolutional network layer in the visual question answering model to obtain a first fusion feature matrix; a second fusion module is used to perform feature fusion on the first fusion feature matrix and the knowledge feature vector through an attention module in the visual question answering model to obtain a second fusion feature matrix; a prediction module is used to predict the target answer to the target question based on the first fusion feature matrix and the second fusion feature matrix; the graph convolutional network layer includes a first number of The hidden layers are connected sequentially; each entity in the target image corresponds to its own entity segmentation image, and each entity segmentation image corresponds to its own entity feature vector; when the first fusion module is used to perform feature fusion on the visual feature vector of the target image, the question feature vector of the target question, and the knowledge feature vector of the knowledge text through the graph convolutional network layer in the visual question answering model to obtain the first fusion feature matrix, it is specifically used to: for each entity feature vector corresponding to the target image, concatenate the entity feature vector with the question feature vector to obtain the first concatenated feature vector corresponding to the entity feature vector; for each first concatenated feature vector, input the first concatenated feature vector into the first nonlinear activation function to obtain the joint feature vector corresponding to the first concatenated feature vector; concatenate all the joint feature vectors to obtain the concatenation matrix, so as to generate the undirected graph adjacency matrix by the following formula: Where A represents the adjacency matrix of the undirected graph, E represents the concatenation matrix, and E T This represents the transpose of the concatenation matrix; the dimensions of the knowledge feature vector, the question feature vector, and the visual feature vector are aligned and corrected to ensure that they have the same dimensions; the knowledge feature vector, the question feature vector, and the visual feature vector with the same dimensions are concatenated to obtain the second concatenation feature matrix; for the first hidden layer in the graph convolutional network layer, the undirected graph adjacency matrix and the second concatenation feature matrix are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output; for other hidden layers besides the first hidden layer, the undirected graph adjacency matrix and the output fusion feature matrix output by the previous hidden layer adjacent to the hidden layer are input into the second nonlinear activation function in the hidden layer, and the output fusion feature matrix corresponding to the hidden layer is output; the formula for the second nonlinear activation function is: Where α(·) represents the second nonlinear activation function, n represents the nth hidden layer, and n is a positive integer greater than or equal to 1 and less than or equal to the first number, H n H is the output fusion feature matrix of the nth hidden layer; when n is 1, H n-1 This is the second concatenated feature matrix; when n is greater than 1, H n-1 The output fusion feature matrix is ​​the output of the (n-1)th hidden layer; A denotes the adjacency matrix of an undirected graph, and I is the identity matrix. yes The degree matrix of the diagonal nodes, W n These are the trainable parameters in the nth hidden layer; the output fusion feature matrix of the last hidden layer is determined as the first fusion feature matrix.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 6.

Images