A Visual Question Answering Method and Device Structured and Guided by Problem Image Triples

Abstract

The present invention relates to a visual question-answering method and device guided by structured question-image triples. The method comprises: obtaining a target image and a target question for the target image; extracting the target question's global features, multiple question attribute triple features, and multiple question relationship triple features using a first target model; extracting multiple image attribute triple features of the target image using a second target model; determining the first correlation between each image attribute triple feature and each question attribute triple feature using a target attention model, and forming an attribute attention weight matrix from the multiple first correlations; and concatenating the target image attribute triple features, the target image relationship triple features, and the global question features, and inputting the concatenated features into a target answer classifier to obtain the answer information for the target question. This solution achieves multimodal feature alignment and fusion while maintaining the sequential structure of the question triples, thereby improving the accuracy of the output answer.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a visual question answering method and device guided by structured question image triples. Background Art

[0002] Visual question answering involves feeding a computer an image and asking a question based on the image. The computer then needs to understand the image and the question and provide an answer. Visual question answering involves two input modalities: visual, which contains rich scene information; and linguistic, which provides an abstract understanding and high-level summary of objects. Therefore, visual question answering requires both a thorough understanding of visual information and a comprehensive understanding of linguistic information.

[0003] The basic idea of the current visual question answering algorithm is: first extract image features and text features, then perform feature fusion or feature alignment, and finally send the fused features into the classifier to obtain the final answer.

[0004] During research into related technologies, the inventors discovered that questions in visual question answering tasks, as a crucial component of input data, serve as a guide for the task, making question analysis particularly crucial. However, existing algorithms only analyze questions at the word or phrase level, failing to fully utilize semantic information and text structure. Rich semantic representation of images is crucial for achieving visual question answering tasks, yet existing algorithms typically employ traditional object detection models to extract image features, which can lead to excessive noise and weak semantic relationships between the extracted features.

[0005] These problems lead to low accuracy of answers obtained in current visual question answering tasks. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0007] To this end, the first aspect of the present invention proposes a visual question answering method guided by structured question image triples, the method comprising:

[0008] Acquire a target image and a target question for the target image;

[0009] The first target model is used to extract the global problem features, multiple problem attribute triple features, and multiple problem relationship triple features of the target problem; the attribute triples are three fields representing the semantics of the object, and the relationship triples are three fields representing the relationship between the objects;

[0010] extracting a plurality of image attribute triple features of the target image using the second target model, and extracting a plurality of image relationship triple features of the target image using the third target model;

[0011] Determining, using the target attention model, a first correlation between each of the image attribute triple features and each of the question attribute triple features, and forming an attribute attention weight matrix from the plurality of the first correlations; and determining, using the target attention model, a second correlation between each of the image relationship triple features and each of the question relationship triple features, and forming a relationship attention weight matrix from the plurality of the second correlations;

[0012] Multiplying the attribute attention weight matrix by each of the image attribute triple features respectively to obtain target image attribute triple features, and multiplying the relationship attention weight matrix by each of the image relationship triple features respectively to obtain target image relationship triple features;

[0013] The target image attribute triplet feature, the target image relationship triplet feature and the question global feature are spliced and input into the target answer classifier to obtain the answer information of the target question.

[0014] Optionally, the target attention model is used to determine first correlations between each of the image attribute triple features and each of the question attribute triple features, and a plurality of the first correlations are combined into an attribute attention weight matrix, including:

[0015] Dimensionally aligning the triplet feature of the question attribute with the triplet feature of the image attribute;

[0016] Calculating inner products between the plurality of question attribute triple features and the plurality of image attribute triple features, and normalizing the inner products to obtain a first correlation between the image attribute triple features and the question attribute triple features;

[0017] Combine multiple of the first correlations to obtain an attribute attention weight matrix.

[0018] Optionally, the extracting of the target problem's global features, multiple problem attribute triple features, and multiple problem relationship triple features using the first target model includes:

[0019] Segmenting the target question into a plurality of tokens, and encoding the plurality of tokens into a plurality of feature vectors;

[0020] Inputting the multiple feature vectors into the question encoding layer of the first model, obtaining the semantic relationship between the multiple word units, and obtaining the global feature of the target question;

[0021] Input the global feature of the problem into the triple decoding layer of the first model to obtain multiple problem attribute triple features and multiple problem relationship triple features of the target problem

[0022] Optionally, the training methods of the first target model, the second target model, the third target model, the target attention model, and the target answer classifier are as follows:

[0023] Acquire training set data; the training set data includes sample questions and sample images, as well as triple feature labels of the sample questions and triple feature labels of the sample images;

[0024] Extracting a global feature of the prediction problem, a plurality of prediction problem attribute triple features, and a plurality of prediction problem relationship triple features of the sample problem using the first initial model;

[0025] Determining loss values between the predicted problem attribute triplet feature, the predicted problem relationship triplet feature, and the pre-calibrated feature label of the sample problem respectively to obtain a first loss value;

[0026] Extracting a plurality of predicted image attribute triple features of the sample image using the second initial model, and extracting a plurality of predicted image relationship triple features of the sample image using the third initial model;

[0027] Using the target attention model, respectively determining the predicted first correlation between each of the predicted image attribute triple features and each of the predicted question attribute triple features, and forming a predicted attribute attention weight matrix with multiple predicted first correlations; and using the target attention model, respectively determining the predicted second correlation between each of the predicted image relationship triple features and each of the predicted question relationship triple features, and forming a predicted relationship attention weight matrix with multiple predicted second correlations;

[0028] Determining loss values between the predicted attribute attention weight matrix, the predicted relationship attention weight matrix, and a pre-calibrated target attention matrix label respectively to obtain a second loss value;

[0029] Multiplying the target prediction attribute attention weight matrix by each of the predicted image attribute triple features respectively to obtain target prediction image attribute triple features, and multiplying the target prediction relationship attention weight matrix by each of the predicted image relationship triple features respectively to obtain target prediction image relationship triple features;

[0030] splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into the initial answer classifier to obtain the predicted answer to the sample question;

[0031] Determining a loss value between the predicted answer and a pre-calibrated target answer label to obtain a third loss value;

[0032] taking the sum of the first loss value, the second loss value and the third loss value as a target loss value;

[0033] When the target loss value is greater than the preset loss value threshold, the parameters of the first initial model, the second initial model, the third initial model, the initial attention model, and the initial answer classifier are modified and retrained until the target loss value is less than the preset loss value threshold, thereby obtaining the first target model, the second target model, the third target model, the target attention model, and the target answer classifier.

[0034] Optionally, the step of splicing the target predicted image attribute triplet feature, the target predicted image relationship triplet feature, and the predicted question global feature and inputting them into an initial answer classifier to obtain a predicted answer to the sample question includes:

[0035] splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into an initial answer classifier to obtain multiple candidate answers to the sample question and prediction scores of the candidate answers;

[0036] Determining the type of the target question using a question type classifier, wherein the type includes verification questions and inquiry questions; the answer space of the verification question includes yes and no, and the answer space of the inquiry question includes entity words, attribute words, and relation words;

[0037] Determine an answer mask vector according to the type of the target question, and perform element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector;

[0038] The candidate answer corresponding to the maximum score in the target prediction score vector is determined as the predicted answer to the sample question.

[0039] Optionally, determining an answer mask vector according to the type of the target question, and performing element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector includes:

[0040] When the type of the target problem is the verification problem, performing element-wise multiplication on a prediction score vector composed of a plurality of the prediction scores and a preset binary classification mask vector to obtain a target prediction score vector;

[0041] The binary classification mask vector is an N*1 vector composed of two values 0 and 1, 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of predicted answers.

[0042] Optionally, determining an answer mask vector according to the type of the target question, and performing element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector includes:

[0043] When the type of the target question is the query-type question, determining the number M of the attribute words in the answer space;

[0044] An N*1 vector including only two numerical values of 0 and 1 is set as the attribute classification mask vector in the multi-classification mask vector, wherein the attribute classification mask vector includes M 1s, and the positions of the M 1s respectively correspond to the position numbers of the attribute words in the answer space; wherein 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of the predicted answers.

[0045] Setting the entity classification mask vector and the relationship mask vector according to the attribute classification mask vector method;

[0046] The prediction score vector composed of multiple prediction scores is element-wise multiplied with the attribute classification mask vector, the entity classification mask vector, and the relationship mask vector to obtain the target prediction score vector corresponding to the attribute word, the target prediction score vector corresponding to the entity word, and the target prediction score vector corresponding to the relationship word.

[0047] Optionally, determining the candidate answer corresponding to the maximum score in the target prediction score vector as the predicted answer to the sample question includes:

[0048] Select the target attribute word corresponding to the maximum score in the target prediction score vector corresponding to the attribute word, the target entity word corresponding to the maximum score in the target prediction score vector corresponding to the entity word, and the target relation word corresponding to the maximum score in the target prediction score vector corresponding to the relation word respectively;

[0049] The answer consisting of the target attribute words, target entity words, and target relationship words is used as the predicted answer to the sample question.

[0050] A second aspect of an embodiment of the present invention provides a visual question answering device guided by structured question image triples, the device comprising:

[0051] An acquisition module, configured to acquire a target image and a target question for the target image;

[0052] A problem feature extraction module is used to extract the problem global feature, multiple problem attribute triple features and multiple problem relationship triple features of the target problem using the first target model; the attribute triple is three fields representing the semantics of the object, and the relationship triple is three fields representing the relationship between the objects;

[0053] an image feature extraction module, configured to extract a plurality of image attribute triple features of the target image using the second target model, and to extract a plurality of image relationship triple features of the target image using the third target model;

[0054] a weight matrix determination module, configured to use a target attention model to respectively determine a first correlation between each of the image attribute triple features and each of the question attribute triple features, and to form an attribute attention weight matrix with a plurality of the first correlations; and to respectively determine a second correlation between each of the image relationship triple features and each of the question relationship triple features, and to form a relationship attention weight matrix with a plurality of the second correlations;

[0055] a target feature acquisition module, configured to respectively multiply the attribute attention weight matrix by each of the image attribute triple features to obtain target image attribute triple features, and respectively multiply the relationship attention weight matrix by each of the image relationship triple features to obtain target image relationship triple features;

[0056] The answer acquisition module is used to splice the target image attribute triple feature, the target image relationship triple feature and the question global feature and input them into the target answer classifier to obtain the answer information of the target question.

[0057] Optionally, the weight matrix determination module is specifically used to:

[0058] Dimensionally aligning the triplet feature of the question attribute with the triplet feature of the image attribute;

[0059] Calculating inner products between the plurality of question attribute triple features and the plurality of image attribute triple features, and normalizing the inner products to obtain a first correlation between the image attribute triple features and the question attribute triple features;

[0060] Combine multiple of the first correlations to obtain an attribute attention weight matrix.

[0061] Optionally, the problem feature extraction module is specifically used to:

[0062] Segmenting the target question into a plurality of tokens, and encoding the plurality of tokens into a plurality of feature vectors;

[0063] Inputting the multiple feature vectors into the question encoding layer of the first model, obtaining the semantic relationship between the multiple word units, and obtaining the global feature of the target question;

[0064] Input the global feature of the problem into the triple decoding layer of the first model to obtain multiple problem attribute triple features and multiple problem relationship triple features of the target problem

[0065] Optionally, the device further includes a training module, wherein the training module is configured to:

[0066] Acquire training set data; the training set data includes sample questions and sample images, as well as triple feature labels of the sample questions and triple feature labels of the sample images;

[0067] Extracting a global feature of the prediction problem, a plurality of prediction problem attribute triple features, and a plurality of prediction problem relationship triple features of the sample problem using the first initial model;

[0068] Determining loss values between the predicted problem attribute triplet feature, the predicted problem relationship triplet feature, and the pre-calibrated feature label of the sample problem respectively to obtain a first loss value;

[0069] Extracting a plurality of predicted image attribute triple features of the sample image using the second initial model, and extracting a plurality of predicted image relationship triple features of the sample image using the third initial model;

[0070] Using the target attention model, respectively determining the predicted first correlation between each of the predicted image attribute triple features and each of the predicted question attribute triple features, and forming a predicted attribute attention weight matrix with multiple predicted first correlations; and using the target attention model, respectively determining the predicted second correlation between each of the predicted image relationship triple features and each of the predicted question relationship triple features, and forming a predicted relationship attention weight matrix with multiple predicted second correlations;

[0071] Determining loss values between the predicted attribute attention weight matrix, the predicted relationship attention weight matrix, and a pre-calibrated target attention matrix label respectively to obtain a second loss value;

[0072] Multiplying the target prediction attribute attention weight matrix by each of the predicted image attribute triple features respectively to obtain target prediction image attribute triple features, and multiplying the target prediction relationship attention weight matrix by each of the predicted image relationship triple features respectively to obtain target prediction image relationship triple features;

[0073] splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into the initial answer classifier to obtain the predicted answer to the sample question;

[0074] Determining a loss value between the predicted answer and a pre-calibrated target answer label to obtain a third loss value;

[0075] taking the sum of the first loss value, the second loss value and the third loss value as a target loss value;

[0076] When the target loss value is greater than the preset loss value threshold, the parameters of the first initial model, the second initial model, the third initial model, the initial attention model, and the initial answer classifier are modified and retrained until the target loss value is less than the preset loss value threshold, thereby obtaining the first target model, the second target model, the third target model, the target attention model, and the target answer classifier.

[0077] Optionally, the training module is further used to:

[0078] splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into an initial answer classifier to obtain multiple candidate answers to the sample question and prediction scores of the candidate answers;

[0079] Determining the type of the target question using a question type classifier, wherein the type includes verification questions and inquiry questions; the answer space of the verification question includes yes and no, and the answer space of the inquiry question includes entity words, attribute words, and relation words;

[0080] Determine an answer mask vector according to the type of the target question, and perform element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector;

[0081] The candidate answer corresponding to the maximum score in the target prediction score vector is determined as the predicted answer to the sample question.

[0082] Optionally, the training module is further used to:

[0083] When the type of the target problem is the verification problem, performing element-wise multiplication on a prediction score vector composed of a plurality of the prediction scores and a preset binary classification mask vector to obtain a target prediction score vector;

[0084] The binary classification mask vector is an N*1 vector composed of two values 0 and 1, 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of predicted answers.

[0085] Optionally, the training module is further used to:

[0086] When the type of the target question is the query-type question, determining the number M of the attribute words in the answer space;

[0087] An N*1 vector including only two numerical values of 0 and 1 is set as the attribute classification mask vector in the multi-classification mask vector, wherein the attribute classification mask vector includes M 1s, and the positions of the M 1s respectively correspond to the position numbers of the attribute words in the answer space; wherein 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of the predicted answers.

[0088] Setting the entity classification mask vector and the relationship mask vector according to the attribute classification mask vector method;

[0089] The prediction score vector composed of multiple prediction scores is element-wise multiplied with the attribute classification mask vector, the entity classification mask vector, and the relationship mask vector to obtain the target prediction score vector corresponding to the attribute word, the target prediction score vector corresponding to the entity word, and the target prediction score vector corresponding to the relationship word.

[0090] Optionally, the training module is further used to:

[0091] Select the target attribute word corresponding to the maximum score in the target prediction score vector corresponding to the attribute word, the target entity word corresponding to the maximum score in the target prediction score vector corresponding to the entity word, and the target relation word corresponding to the maximum score in the target prediction score vector corresponding to the relation word respectively;

[0092] The answer consisting of the target attribute words, target entity words, and target relationship words is used as the predicted answer to the sample question.

[0093] A third aspect of the present invention proposes an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the visual question answering method as described in the first aspect.

[0094] In a fourth aspect, the present invention proposes a computer-readable storage medium, which stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the visual question answering method as described in the first aspect.

[0095] The embodiments of the present invention have the following beneficial effects:

[0096] In an embodiment of the present invention, a target image and a target question for the target image are obtained;

[0097] The first target model is used to extract the problem global features, multiple problem attribute triple features and multiple problem relationship triple features of the target problem; the attribute triple is three fields that represent the semantics of the object, and the relationship triple is three fields that represent the relationship of the object; the second target model is used to extract multiple image attribute triple features of the target image, and the third target model is used to extract multiple image relationship triple features of the target image; the target attention model is used to determine the first correlation between each image attribute triple feature and each problem attribute triple feature, and the multiple first correlations are combined into an attribute attention weight matrix; and the target attention model is used to extract the multiple image attribute triple features and multiple problem attribute triple features. The target attention model determines the second correlation between each of the image relationship triple features and each of the question relationship triple features, and combines the plurality of the second correlations into a relationship attention weight matrix; multiplies the attribute attention weight matrix with each of the image attribute triple features to obtain the target image attribute triple features, and multiplies the relationship attention weight matrix with each of the image relationship triple features to obtain the target image relationship triple features; the target image attribute triple features, the target image relationship triple features, and the global question features are spliced and input into the target answer classifier to obtain the answer information of the target question. This solution adopts a method based on triple feature alignment, selects the relevant features of the image triples according to the order of the question triples, and realizes multimodal feature alignment and fusion while ensuring the sequential structure of the question triples. Finally, the focused image features and the global features of the question are used as the input of the answer classifier to obtain the final answer information, thereby improving the accuracy of the output answer.

[0098] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] To more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0100] Figure 1 A flowchart of the steps of a first visual question answering method guided by structured question-image triples provided by an embodiment of the present invention;

[0101] Figure 2 This is an overall framework diagram of a visual question answering system model provided by an embodiment of the present invention;

[0102] Figure 3 A flowchart of the steps of a visual question answering model training method provided by an embodiment of the present invention;

[0103] Figure 4 This is a structural block diagram of a visual question answering device with structured guidance of question image triples provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0104] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0105] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. When implemented in an actual system or server product, the methods shown in the embodiments or figures may be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0106] Figure 1 A flowchart of the steps of a visual question answering method guided by structured question-image triples provided by an embodiment of the present invention.

[0107] Step 101: Obtain a target image and a target question for the target image.

[0108] The target image is the image that is fed into the computer. The questioner asks the target question based on the target image and feeds it to the computer. The computer then needs to understand the target image and the target question and give the answer to the target question.

[0109] For example, the target image depicts multiple apples scattered on a table and the floor, and the target question is: What color are the apples on the table? Based on its understanding of the target image and the target question, the computer processes the following sequence of steps: find the table, find the apples on the table, check the color of the apples, and ultimately provide the answer to the target question.

[0110] Step 102: Use the first target model to extract the global problem features, multiple problem attribute triple features, and multiple problem relationship triple features of the target problem; the attribute triples are three fields that represent the semantics of the object, and the relationship triples are three fields that represent the relationship between the objects.

[0111] Existing algorithms only analyze questions at the word or phrase level. In order to fully explore the text structure and semantic information of the question, this solution extracts the global features, attribute triple features, and relationship triple features of the target question.

[0112] The attribute triple represents the semantics of an object and consists of the subject, is, and the attribute word that modifies the subject, i.e., the subject-category-table structure. For example: bird is white.

[0113] Relation triples represent the relationship between objects, including the subject, the relational term between the subject and the object, and the object. For example: tree near building.

[0114] For example, the attribute triples for the question "What color is the apple on the table?" are: apple, of, color, and the relation triples are: apple, on, table. Based on the overall structure of the question, these two types of triples are sequentially represented to form a sequence of question triples: apple, on, table, apple, of, color. The global feature of a question is an abstract summary of the entire question, encompassing both the attribute and relational features of the question, but not simply the sum of the attributes and relations.

[0115] A question contains a global feature of a question, including one or more question attribute triple features and one or more question relationship triple features.

[0116] The first target model may be a Transformer model, which is a trained model.

[0117] Step 103: Use the second target model to extract multiple image attribute triple features of the target image, and use the third target model to extract multiple image relationship triple features of the target image.

[0118] The second target model may be the trained Faster R-CNN model, and the third target model may be the trained codec part of PSGTR and ResNet-101 CNN.

[0119] Specifically, the target image is input into the Faster R-CNN model. The network's last layer output is extracted and non-maximum suppression is performed for each target category using an Intersection over Union (IoU) threshold. Regions with confidence exceeding a certain threshold are then selected. Average pooling is performed on each selected region to obtain its features, with each feature vector being 2048-dimensional. Furthermore, to better represent image semantics, the target region features and the ground-truth target label are concatenated as input to the attribute classifier head to obtain the target's attributes. By associating the target with its attributes, the image's attribute triples are obtained, such as "table is wooden." Text encoding of the attribute triples yields the target image's attribute triplet features.

[0120] This solution uses the encoder-decoder component of PSGTR and the ResNet-101 CNN to extract relation triple features from the GQA dataset. To generate semantically rich relation triple features, the image is input into a trained PSGTR model. The decoder output is taken and thresholded to obtain a series of 256-dimensional relation triplets, such as "tree near building." These relation triplets are then encoded using text encoding to obtain image relation triple features.

[0121] An image includes multiple target areas, and image attribute triple features and image relationship triple features can be extracted from each target area. Therefore, multiple image attribute triple features and multiple image relationship triple features can be extracted from the target image.

[0122] Step 104: Use the target attention model to determine the first correlation between each of the image attribute triple features and each of the question attribute triple features, and combine the multiple first correlations into an attribute attention weight matrix; and use the target attention model to determine the second correlation between each of the image relationship triple features and each of the question relationship triple features, and combine the multiple second correlations into a relationship attention weight matrix.

[0123] This scheme uses question triples to filter the image relationship triple features and image attribute triple features that are most relevant to the question.

[0124] Specifically, a question-attribute triple is input, and its correlation with all image-attribute triple features is calculated. The image-attribute triple feature with the highest similarity is the image-attribute triple feature most relevant to the question. Similarly, a question-relationship triple is input, and its correlation with all image-relationship triple features is calculated. The image-relationship triple feature with the highest similarity is the image-relationship triple feature most relevant to the question. This achieves feature alignment between question triples and image triples, or feature screening.

[0125] In terms of network design, this scheme uses cosine similarity, attention mechanism and its variant for feature alignment to filter out the image triplet feature vector that is most relevant to the problem feature vector.

[0126] Specifically, cosine similarity measures the similarity between two vectors by measuring the cosine value of the angle between them. The attention mechanism measures the similarity between two vectors by calculating the correlation coefficient between them. The variant of attention is based on the attention mechanism, which first performs linear mapping on each input vector and then calculates the correlation coefficient of the vector.

[0127] Experimental results show that the attention mechanism performs best, the cosine similarity method has poor feature expression ability, and the attention variant makes the model too complex and overfitting occurs. Therefore, this paper selects the attention mechanism as the feature alignment module.

[0128] The first correlations are combined to obtain the attribute attention weight matrix, and the second correlations are combined to obtain the relationship attention weight matrix.

[0129] The input of the feature alignment module is the question relation triplet feature vector, the image relation triplet vector, the question attribute triplet feature vector, and the image attribute triplet vector, and the output is the relation attention weight matrix and the attribute attention weight matrix.

[0130] Step 105: multiply the attribute attention weight matrix with each of the image attribute triple features to obtain the target image attribute triple features, and multiply the relationship attention weight matrix with each of the image relationship triple features to obtain the target image relationship triple features.

[0131] Multiplying the attribute attention weight matrix with the features of each image attribute triplet will enhance the feature expression of the part of the image attribute triplet that is highly correlated with the question attribute triplet. Using it as the input of the answer classifier can improve the output speed and accuracy of the answer.

[0132] Multiplying the relation attention weight matrix with each image relation triplet feature also has the same beneficial effect.

[0133] Taking the attribute attention weight matrix as an example, the question attribute triples will be correlated with the image attribute triples, and the importance of each image attribute triple feature will be determined based on the correlation.

[0134] When the attribute attention weight matrix is multiplied by the image attribute triplet feature, if the correlation is 1, which is equivalent to multiplying the image attribute triplet feature by 1, then the image triplet feature will not change. If the correlation is 0, which is equivalent to multiplying the image attribute triplet feature by 0, then the image triplet feature will be ignored. This is equivalent to strengthening the features with high correlation and weakening the features with low correlation.

[0135] The attribute attention weight matrix is multiplied by the old image attribute triplet feature, which changes the features of the image attribute triplet and obtains the target image attribute triplet feature.

[0136] Step 106: splice the target image attribute triple feature, the target image relationship triple feature, and the question global feature, and input the concatenated features into a target answer classifier to obtain answer information for the target question.

[0137] The target image attribute triplet features and the global features of the question are concatenated in dimension 0, and the target image relationship triplet features and the global features of the question are concatenated in dimension 0. The concatenated feature matrices are input into the answer classifier to obtain the answer to the target question.

[0138] Specifically, the answer classifier is a multi-layer perceptron consisting of dropout layers, fully connected layers, and nonlinear activation layers.

[0139] The specific idea of this solution is as follows: First, starting from the two modalities of image and text, the information of the image and question is structured to obtain the corresponding features, namely the relationship triplet features and attribute triplet features in the image and question. Secondly, the obtained question triplet features are used to guide the model to select the image area related to the question. This solution adopts a method based on triplet feature alignment, selecting the relevant features of the image triplets according to the order of the question triples. While ensuring the sequential structure of the question triples, multimodal feature alignment and fusion are achieved. Finally, the image features of interest and the global features of the question are used as the input of the answer classifier to obtain the final answer.

[0140] Figure 2 This is an overall framework diagram of a visual question answering system model provided by the present invention.

[0141] like Figure 2As shown, the bird image in the upper left corner is a sample image, and below the sample image is the sample question, "What color is the bird near the tree?" The sample image is input into the image attribute triplet feature extraction module and the image relationship triplet feature extraction module, and the sample question is input into the question triplet feature extraction module. After obtaining the image attribute triplet features and image relationship triplet features, feature fusion and inference, i.e., feature alignment, are performed. Specifically, the image attribute triplet features and image relationship triplet features are input into the question triplet-guided image attribute triplet feature extraction module, and the question triplet features are input into the question triplet-guided image relationship triplet feature extraction module to obtain the feature alignment result. The feature alignment result is input into the answer classifier to obtain the predicted answer: "Answer: White."

[0142] In summary, in an embodiment of the present invention, a target image and a target question for the target image are obtained; the first target model is used to extract the global feature of the target question, multiple question attribute triple features and multiple question relationship triple features; the attribute triple is three fields that represent the semantics of the object, and the relationship triple is three fields that represent the relationship of the object; the second target model is used to extract multiple image attribute triple features of the target image, and the third target model is used to extract multiple image relationship triple features of the target image; the target attention model is used to determine the relationship between each image attribute triple feature and each question attribute triple. The correlation between the group features and the correlation between the features of each image relationship triplet and the features of each question relationship triplet are calculated to obtain multiple attribute attention weight matrices and multiple relationship attention weight matrices; the attribute attention weight matrices are multiplied with the features of each image attribute triplet to obtain the target image attribute triplet features, and the relationship attention weight matrices are multiplied with the features of each image relationship triplet to obtain the target image relationship triplet features; the target image attribute triplet features, the target image relationship triplet features, and the global features of the question are concatenated and input into the target answer classifier to obtain the answer information of the target question. This solution adopts a method based on triplet feature alignment, selecting the relevant features of the image triples according to the order of the question triples, and realizing multimodal feature alignment and fusion while ensuring the sequential structure of the question triples. Finally, the focused image features and the global features of the question are used as the input of the answer classifier to obtain the final answer information, thereby improving the accuracy of the output answer.

[0143] In one possible implementation, step 104 includes:

[0144] Step 1041: Dimensionally align the question attribute triplet feature and the image attribute triplet feature.

[0145] A 1*1 convolution operation is performed on the question attribute triple feature and the image attribute triple feature, so that the image attribute triple feature is mapped from 256 dimensions to 512 dimensions, thereby achieving dimensional alignment of the question attribute triple feature and the image attribute triple feature.

[0146] Moreover, through dimension alignment, a feature transformation layer is added to the image triplet features.

[0147] Step 1042: Calculate the inner products between the multiple question attribute triple features and the multiple image attribute triple features, and normalize the inner products to obtain a first correlation between the image attribute triple features and the question attribute triple features.

[0148] Step 1043: Combine multiple of the first correlations to obtain an attribute attention weight matrix.

[0149] Step 1042 and step 1043 are the process of obtaining the attention weight matrix using the attention mechanism. Specifically, the formula for obtaining the attention weight matrix using the attention mechanism is as follows:

[0150]

[0151] Among them, Q represents the feature vector of the question triple, I represents the feature vector of the image triple, and d i represents the dimension of the image triplet feature vector, and Att(Q,I) represents the attention matrix.

[0152] QI T The inner product between the question triplet feature and the image triplet feature is normalized to obtain the first correlation. The first correlations are combined to obtain the attention weight matrix.

[0153] When using the attention mechanism to calculate the attribute attention weight matrix and the relationship attention weight matrix, both equation (1) can be used.

[0154] The values of the attribute attention weight matrix can represent the correlation between the question attribute triplet features and the image attribute triplet features. The larger the value, the greater the correlation. The values of the relationship attention weight matrix can represent the correlation between the question relationship triplet features and the image relationship triplet features. The larger the value, the greater the correlation.

[0155] In one possible implementation, step 102 includes:

[0156] Step 1021: Divide the target question into multiple word units, and encode the multiple word units into multiple feature vectors.

[0157] The question sequence is divided into multiple tokens, such as words or phrases, and then GloVe is used to encode each token into a feature vector.

[0158] Step 1022: Input the multiple feature vectors into the question encoding layer of the first model, obtain the semantic relationship between the multiple word units, and obtain the global feature of the target question.

[0159] The first model can be a Transformer. The encoding layer of the Transformer is used to obtain the contextual semantic relationship between word units and obtain one or more global features of the question.

[0160] Step 1023: Input the global feature of the question into the triple decoding layer of the first model to obtain multiple question attribute triple features and multiple question relationship triple features of the target question.

[0161] Using the idea of sequence generation, the decoding layer of Transformer is used to obtain the corresponding triple sequence, and the triple sequence is encoded into question attribute triple features and question relationship triple features.

[0162] Figure 3 A flowchart of the steps of a visual question answering model training method provided by an embodiment of the present invention.

[0163] like Figure 3 As shown, the method includes the following steps:

[0164] Step 201: Acquire training set data; the training set data includes sample questions and sample images, as well as triple feature labels of the sample questions and triple feature labels of the sample images.

[0165] A sample image may correspond to one or more sample questions, and there may be multiple sample images and sample questions with this correspondence.

[0166] The triple feature labels of the sample questions include: sample question relationship triple feature labels and attribute triple feature labels; the triple feature labels of the sample images include: sample image relationship triple feature labels and attribute triple feature labels.

[0167] Step 202: Using the first initial model, extract the prediction problem global features, multiple prediction problem attribute triple features, and multiple prediction problem relationship triple features of the sample problem.

[0168] Programmatic problem-solving is a common way humans solve problems. For example, when asked, "What color is the apple on the table?" the corresponding program chain is as follows: "Find the table, find the apple on the table, check the color of the apple." In visual question answering tasks, relational terms in questions bridge the gaps between entities and between entities and attributes. Therefore, it is necessary to extract triples from the question. Building on the sequential nature of program chains, this solution further optimizes them by designing question-attribute triples and question-relation triples. During training, this solution uses these triples as labels to guide feature extraction from question triples.

[0169] Specifically, the Transformer can be used as the first initial model. First, the question sequence is segmented into multiple tokens (words or phrases). Next, GLOVE is used to encode each token into a feature vector. The Transformer's encoding layer then uses the Transformer's encoding layer to capture the contextual semantic relationships between words, generating global features for the predicted question. Finally, leveraging the concept of sequence generation, the Transformer's decoding layer is used to generate the corresponding triple prediction sequence and features, resulting in multiple predicted question attribute triple features and multiple predicted question relationship triple features.

[0170] Step 203 : respectively determine the loss values between the predicted problem attribute triple feature, the predicted problem relationship triple feature, and the pre-calibrated feature label of the sample problem to obtain a first loss value.

[0171] In order to ensure the effectiveness of question feature extraction, this scheme designs a question triplet loss function, the specific formula is shown in (2):

[0172]

[0173] Among them, L question_triplet Represents the first loss value, Q out It represents the output of the question feature extraction module, i.e., the predicted question attribute triple feature and the predicted question relationship triple feature. target Represents the true triplet feature label.

[0174] Step 204: Use the second initial model to extract multiple predicted image attribute triple features of the sample image, and use the third initial model to extract multiple predicted image relationship triple features of the sample image.

[0175] This approach feeds sample images into a pretrained Faster R-CNN model, extracts the network's final layer output, and uses an Intersection over Union (IoU) threshold to perform non-maximum suppression on each target category. Regions with confidence exceeding a certain threshold are then selected. Average pooling is applied to each selected region to obtain its features, with each feature vector being 2048-dimensional. Furthermore, to better represent image semantics, the target region features and the true target label are concatenated as input to the attribute classifier head to obtain the target's attributes. By associating the target with its attributes, a predicted attribute triplet ("table is wooden") is obtained for the image. Text encoding of the predicted attribute triplet yields the predicted image attribute triplet features.

[0176] This solution uses the encoder-decoder component of PSGTR and the ResNet-101 CNN to extract relational triple features from sample images. To generate semantically rich relational triple features, the PSGTR model is first trained. Sample images are then fed into the model. The decoder output is extracted and thresholded to produce a series of 256-dimensional predicted relational triple features. Furthermore, the relational triple feature labels are stored, corresponding one-to-one to the predicted relational triple features. The predicted relational triple features are then text-encoded to obtain the predicted relational triple text features for the image.

[0177] Step 205: using the initial attention model to respectively determine the predicted first correlation between each of the predicted image attribute triple features and each of the predicted problem attribute triple features, and combining the multiple predicted first correlations into a predicted attribute attention weight matrix; and using the target attention model to respectively determine the predicted second correlation between each of the predicted image relationship triple features and each of the predicted problem relationship triple features, and combining the multiple predicted second correlations into a predicted relationship attention weight matrix.

[0178] Specifically, the formula for obtaining the predicted attention weight matrix using the attention mechanism is shown in formula (1) in step 1042.

[0179] Step 206: respectively determine the loss values between the predicted attribute attention weight matrix, the predicted relationship attention weight matrix and the pre-calibrated target attention matrix label to obtain a second loss value.

[0180] In terms of loss function design, Smooth L1 loss is used as relay supervision to supervise the alignment process of triple features. The input of Smooth L1 loss is the predicted attribute attention weight matrix, the predicted relationship attention weight matrix and the target attention matrix label. The specific loss function formula is shown in (3):

[0181] L alignment =Smooth_L1_Loss(Att out ,Att target ) (3)

[0182] Among them, L alignment is the second loss value, Att out To predict the attention weight matrix, Att target is the target attention matrix label.

[0183] Step 207: multiply the target prediction attribute attention weight matrix with each of the predicted image attribute triple features to obtain target prediction image attribute triple features, and multiply the target prediction relationship attention weight matrix with each of the predicted image relationship triple features to obtain target prediction image relationship triple features.

[0184] Multiplying the target predicted attribute attention weight matrix with each predicted image attribute triple feature will enhance the feature expression of the part of the image attribute triple feature that is highly correlated with the question attribute triple. Using it as the input of the answer classifier can improve the output speed and accuracy of the predicted answer.

[0185] Multiplying the target prediction relation attention weight matrix with each predicted image relation triplet feature has the same beneficial effect.

[0186] Step 208: splice the target prediction image attribute triple feature, the target prediction image relationship triple feature and the prediction question global feature and input them into the initial answer classifier to obtain the predicted answer to the sample question.

[0187] The target prediction image attribute triplet features and the prediction question global features are concatenated in dimension 0, and the target prediction image relationship triplet features and the prediction question global features are concatenated in dimension 0. The concatenated prediction feature matrices are input into the answer classifier to obtain the predicted answer to the target question.

[0188] Step 209: Determine the loss value between the predicted answer and the pre-calibrated target answer label to obtain a third loss value.

[0189] The loss function for predicting the answer is defined as follows:

[0190] L answer =CrossEntropy(A predict ,A target ) (4)

[0191] Among them, L answer Represents the third loss value, Apredict Indicates the predicted answer, A target Represents the target answer label.

[0192] Step 210: Taking the sum of the first loss value, the second loss value, and the third loss value as a target loss value.

[0193] The sum of the above loss values is used as the objective function of the model, as follows:

[0194] L total =L question_triplet +L alignment +L answer (5)

[0195] Among them, L total Represents the target loss value, L question_triplet Represents the first loss value, L alignment Represents the second loss value, L answer Represents the third loss value.

[0196] Step 211: When the target loss value is greater than the preset loss value threshold, modify the parameters of the first initial model, the second initial model, the third initial model, the initial attention model, and the initial answer classifier, and retrain until the target loss value is less than the preset loss value threshold, to obtain the first target model, the second target model, the third target model, the target attention model, and the target answer classifier.

[0197] The loss value threshold for model training is set in advance. When the target loss value is greater than the loss value threshold, the parameters of each model in the system are modified, and the sample questions and sample images are re-input for training until the target loss value is less than the loss value threshold. At this time, the model training is completed, and the first target model, the second target model, the third target model, the target attention model and the target answer classifier are obtained.

[0198] Existing technologies for visual question answering lack the ability to parse questions and decouple images. This approach to image and question parsing is a crucial step, helping us understand the semantics of the question and infer answers from the decoupled image.

[0199] Questions in visual question answering (VQA) are a crucial part of the input data and serve as a guide for the task, so question analysis is crucial. However, existing algorithms only analyze questions at the word or phrase level, underutilizing semantic information and text structure. This solution fully exploits the structural information of questions, enabling the model to employ human-like logical reasoning, effectively answer questions, and ultimately provide accurate predicted answers.

[0200] Rich semantic representation of images is key to achieving visual question answering. However, existing algorithms typically use traditional object detection models to extract image features, which is prone to excessive noise and weak semantic relationships between the extracted features. This solution fully utilizes the visual information in the image, allowing questions to be precisely localized to the corresponding visual area, improving the reliability and effectiveness of visual question answering.

[0201] Entities extracted from images and text do not exist in isolation; they rely on their corresponding attribute features and relationship features. Existing techniques, which focus solely on entity alignment, can easily disrupt these dependencies and increase the difficulty of learning feature alignment. Existing techniques, which focus on global image features, are not accurate enough to align global features with image features.

[0202] This solution, based on hierarchical structured information representation, addresses common issues with existing visual question models, such as weak image semantic representation, poor interpretability, and poor generalization. Specifically, for feature extraction, it addresses issues such as insufficient exploitation of structural information and semantic information in images; for triple feature alignment, it addresses issues such as entity semantic alignment destroying dependency information and global feature alignment ignoring detail information; and for answer space partitioning, it addresses the mismatch between answer predictions and question types.

[0203] In summary, this scheme extracts question triple features and image triple features from a multimodal and hierarchical perspective, making the extracted features more semantically richer and more structured.

[0204] In addition, this solution designs a multimodal feature alignment module guided by question triples from the perspectives of model design, loss function optimization and parameter tuning. Due to the use of modular design ideas and the introduction of relay supervision in each module, the model has interpretability and strong reasoning ability.

[0205] In one possible implementation, step 209 includes:

[0206] Step 2091: splice the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and input them into an initial answer classifier to obtain multiple candidate answers to the sample question and the prediction scores of the candidate answers;

[0207] Step 2092: Determine the type of the target question using a question type classifier, where the types include verification questions and inquiry questions; the answer space of the verification question includes yes and no, and the answer space of the inquiry question includes entity words, attribute words, and relation words;

[0208] Step 2093: Determine an answer mask vector according to the type of the target question, and perform element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector;

[0209] Step 2094: Determine the candidate answer corresponding to the maximum score in the target prediction score vector as the predicted answer to the sample question.

[0210] In steps 2091-2094, this solution divides the answer space into two categories: a binary answer space based on verification questions and a multi-category answer space based on query questions. To ensure a balanced distribution of questions, this solution further divides the binary answer space into two subspaces: yes and no, and the multi-category answer space into three subspaces: entity, attribute, and relationship. The binary answer space includes both yes and no answers. In the multi-category answer space, entities include tree, apple, and hat, attributes include red, wooden, and young, and relationships include sitting, to, the, right, and of.

[0211] Based on the question type and answer space mask, this solution trained a question type classifier. The question classifier is a linear classifier, which is equivalent to a function mapping process. The linear classifier maps the global feature 512 dimensions into a 4-dimensional vector. Each element in the vector represents the probability that the element belongs to this type of question. The index with the largest probability value is selected as the type of answer. The global features of the question are used to divide the questions into two categories: the first category is verification questions (whether the triple exists) and the second category is query questions (triple completion). This solution verified the pre-trained question type classifier on the test set, and the results showed good classification accuracy.

[0212] During the training process, the answer space mask is selected based on the question type, and the mask can be used to block out the scores of candidate answers that do not belong to the answer subspace, thereby ensuring the rationality of the answer.

[0213] In one possible implementation, step 2093 includes:

[0214] Step 20931. When the type of the target question is the verification question, perform element-wise multiplication on the predicted score vector composed of multiple predicted scores and the preset binary classification mask vector to obtain a target predicted score vector; wherein the binary classification mask vector is an N*1 vector composed of two numerical values 0 and 1, 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of the predicted answers.

[0215] Specifically, when the target question is a verification question, assuming the answer space has 1842 possible answers, a 1842*1 mask vector containing only the values 0 and 1 is set as the binary classification mask, where 1 indicates that the answer space is activated, and 0 indicates that it is not activated. The predicted score vector and the mask vector are element-wise multiplied to obtain the new target predicted score vector. This operation is equivalent to selecting only between the two answers yes / no.

[0216] In one possible implementation, step 2093 includes:

[0217] Step 20932: When the target question is a query-type question, determine the number M of the attribute words in the answer space.

[0218] Step 20933: Set an N*1 vector that only includes two numerical values, 0 and 1, as the attribute classification mask vector in the multi-classification mask vector, wherein the attribute classification mask vector includes M 1s, and the positions of the M 1s respectively correspond to the position numbers of the attribute words in the answer space; wherein 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of the predicted answers.

[0219] Step 20934: Set the entity classification mask vector and the relationship mask vector according to the attribute classification mask vector method;

[0220] Step 20935: perform element-wise multiplication of the prediction score vector composed of multiple prediction scores with the attribute classification mask vector, the entity classification mask vector, and the relationship mask vector to obtain the target prediction score vector corresponding to the attribute word, the target prediction score vector corresponding to the entity word, and the target prediction score vector corresponding to the relationship word.

[0221] Specifically, when the target question is a query question, taking attribute words as an example: assuming the answer space has 1842 possible answers and there are n attribute words in the answer space, set an 1842*1 mask vector containing only two values, 0 and 1, as the attribute classification mask in the multi-classification. This mask contains n 1s, each corresponding to the position of the attribute word in the answer space. For example, if the second answer space in the 1842 answer spaces is red, then the second number in the mask is 1, indicating that the answer space is activated, and 0 indicates that it is not activated. The attribute prediction score vector is element-wise multiplied by the attribute classification mask to obtain a new attribute prediction score vector.

[0222] This operation is equivalent to selecting the final answer only among the answers that are attribute words. The operations for entity words and relationship words are similar. Finally, the target attribute word corresponding to the maximum score, the target entity word corresponding to the maximum score, and the target relationship word corresponding to the maximum score are selected, and the answer composed of the target attribute word, target entity word, and target relationship word is used as the predicted answer to the sample question.

[0223] Existing models typically search for the correct answer within a large answer space, which is highly data-dependent, prone to language bias, and difficult to train. Choosing an appropriate answer space based on the type of question can reduce the model's computational cost and ensure the accuracy of the answer.

[0224] This solution divides the answer space based on statistics and analysis of the answers, and uses question types to guide the selection of the answer space, so that interfering answers can be quickly eliminated during the answer prediction process, which speeds up the network convergence and saves computing costs.

[0225] Figure 4 This is a structural block diagram of a visual question answering device with structured guidance of question image triples provided by an embodiment of the present invention.

[0226] like Figure 4 As shown, the visual question answering device 300 guided by structured question image triples includes:

[0227] An acquisition module 301 is configured to acquire a target image and a target question for the target image;

[0228] The problem feature extraction module 302 is used to extract the problem global feature, multiple problem attribute triple features and multiple problem relationship triple features of the target problem using the first target model; the attribute triple is three fields representing the semantics of the object, and the relationship triple is three fields representing the relationship between the objects;

[0229] An image feature extraction module 303 is configured to extract a plurality of image attribute triple features of the target image using the second target model, and to extract a plurality of image relationship triple features of the target image using the third target model;

[0230] The weight matrix determination module 304 is configured to determine, using the target attention model, a first correlation between each of the image attribute triple features and each of the question attribute triple features, and to form an attribute attention weight matrix from the plurality of the first correlations; and to determine, using the target attention model, a second correlation between each of the image relationship triple features and each of the question relationship triple features, and to form a relationship attention weight matrix from the plurality of the second correlations;

[0231] A target feature acquisition module 305 is configured to multiply the attribute attention weight matrix by each of the image attribute triple features to obtain target image attribute triple features, and to multiply the relationship attention weight matrix by each of the image relationship triple features to obtain target image relationship triple features;

[0232] The answer acquisition module 306 is used to splice the target image attribute triple feature, the target image relationship triple feature and the question global feature and input them into the target answer classifier to obtain the answer information of the target question.

[0233] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0234] In another embodiment provided by the present invention, a device is also provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the visual question answering method described in the embodiment of the present invention.

[0235] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the visual question answering method described in the embodiment of the present invention.

[0236] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0237] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0238] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0239] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A visual question answering method guided by structured question-image triples, characterized by: The method comprises: Acquire a target image and a target question for the target image; The first target model is used to extract the global problem features, multiple problem attribute triple features, and multiple problem relationship triple features of the target problem; the attribute triples are three fields representing the semantics of the object, and the relationship triples are three fields representing the relationship between the objects; extracting a plurality of image attribute triple features of the target image using the second target model, and extracting a plurality of image relationship triple features of the target image using the third target model; Determining, using the target attention model, a first correlation between each of the image attribute triple features and each of the question attribute triple features, and forming an attribute attention weight matrix from the plurality of the first correlations; and determining, using the target attention model, a second correlation between each of the image relationship triple features and each of the question relationship triple features, and forming a relationship attention weight matrix from the plurality of the second correlations; Multiplying the attribute attention weight matrix by each of the image attribute triple features respectively to obtain target image attribute triple features, and multiplying the relationship attention weight matrix by each of the image relationship triple features respectively to obtain target image relationship triple features; The target image attribute triplet feature, the target image relationship triplet feature and the question global feature are spliced and input into the target answer classifier to obtain the answer information of the target question; The extracting of the global problem feature, multiple problem attribute triple features, and multiple problem relationship triple features of the target problem using the first target model includes: Segmenting the target question into a plurality of tokens, and encoding the plurality of tokens into a plurality of feature vectors; Inputting the multiple feature vectors into the question encoding layer of the first model, obtaining the semantic relationship between the multiple word units, and obtaining the global feature of the target question; The global feature of the question is input into the triple decoding layer of the first model to obtain multiple question attribute triple features and multiple question relationship triple features of the target question.

2. The method according to claim 1, characterized in that The method of using the target attention model to respectively determine the first correlation between each of the image attribute triple features and each of the question attribute triple features, and forming an attribute attention weight matrix from the plurality of the first correlations, includes: Dimensionally aligning the triplet feature of the question attribute with the triplet feature of the image attribute; Calculating inner products between the plurality of question attribute triple features and the plurality of image attribute triple features, and normalizing the inner products to obtain a first correlation between the image attribute triple features and the question attribute triple features; Combine multiple of the first correlations to obtain an attribute attention weight matrix.

3. The method according to claim 1, characterized in that The training methods of the first target model, the second target model, the third target model, the target attention model, and the target answer classifier are as follows: Acquire training set data; the training set data includes sample questions and sample images, as well as triple feature labels of the sample questions and triple feature labels of the sample images; Extracting a global feature of the prediction problem, a plurality of prediction problem attribute triple features, and a plurality of prediction problem relationship triple features of the sample problem using the first initial model; Determining loss values between the predicted problem attribute triplet feature, the predicted problem relationship triplet feature, and the pre-calibrated feature label of the sample problem respectively to obtain a first loss value; Extracting a plurality of predicted image attribute triple features of the sample image using the second initial model, and extracting a plurality of predicted image relationship triple features of the sample image using the third initial model; Using the initial attention model, respectively determining the predicted first correlation between each of the predicted image attribute triple features and each of the predicted question attribute triple features, and forming a predicted attribute attention weight matrix with multiple predicted first correlations; and using the target attention model, respectively determining the predicted second correlation between each of the predicted image relationship triple features and each of the predicted question relationship triple features, and forming a predicted relationship attention weight matrix with multiple predicted second correlations; Determining loss values between the predicted attribute attention weight matrix, the predicted relationship attention weight matrix, and a pre-calibrated target attention matrix label respectively to obtain a second loss value; Multiplying the predicted attribute attention weight matrix by each of the predicted image attribute triple features respectively to obtain target predicted image attribute triple features, and multiplying the predicted relationship attention weight matrix by each of the predicted image relationship triple features respectively to obtain target predicted image relationship triple features; splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into the initial answer classifier to obtain the predicted answer to the sample question; Determining a loss value between the predicted answer and a pre-calibrated target answer label to obtain a third loss value; taking the sum of the first loss value, the second loss value and the third loss value as a target loss value; When the target loss value is greater than the preset loss value threshold, the parameters of the first initial model, the second initial model, the third initial model, the initial attention model, and the initial answer classifier are modified and retrained until the target loss value is less than the preset loss value threshold, thereby obtaining the first target model, the second target model, the third target model, the target attention model, and the target answer classifier.

4. The method according to claim 3, characterized in that The step of splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into the initial answer classifier to obtain the predicted answer to the sample question includes: splicing the target prediction image attribute triple feature, the target prediction image relationship triple feature, and the prediction question global feature and inputting them into an initial answer classifier to obtain multiple candidate answers to the sample question and prediction scores of the candidate answers; Determining the type of the target question using a question type classifier, wherein the type includes verification questions and inquiry questions; the answer space of the verification question includes yes and no, and the answer space of the inquiry question includes entity words, attribute words, and relation words; Determine an answer mask vector according to the type of the target question, and perform element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector; The candidate answer corresponding to the maximum score in the target prediction score vector is determined as the predicted answer to the sample question.

5. The method according to claim 4, characterized in that The step of determining an answer mask vector according to the type of the target question and performing element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector includes: When the type of the target problem is the verification problem, performing element-wise multiplication on a prediction score vector composed of a plurality of the prediction scores and a preset binary classification mask vector to obtain a target prediction score vector; Among them, the binary classification mask vector is an N*1 vector composed of two values 0 and 1, 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, N represents the total number of the predicted answers, and N is a natural number.

6. The method according to claim 4, characterized in that The step of determining an answer mask vector according to the type of the target question and performing element-wise multiplication of the predicted score vector composed of the predicted scores and the answer mask vector to obtain a target predicted score vector includes: When the type of the target question is the query-type question, determining the number M of the attribute words in the answer space, where M is a natural number; Set an N*1 vector containing only two values, 0 and 1, as the attribute classification mask vector in the multi-classification mask vector, wherein the attribute classification mask vector includes M 1s, and the positions of the M 1s respectively correspond to the position numbers of the attribute words in the answer space; wherein 1 indicates that the answer space is activated, 0 indicates that the answer space is not activated, and N represents the total number of the predicted answers; Setting the entity classification mask vector and the relationship mask vector according to the attribute classification mask vector method; The prediction score vector composed of multiple prediction scores is element-wise multiplied with the attribute classification mask vector, the entity classification mask vector, and the relationship mask vector to obtain the target prediction score vector corresponding to the attribute word, the target prediction score vector corresponding to the entity word, and the target prediction score vector corresponding to the relationship word.

7. The method according to claim 6, characterized in that Determining the candidate answer corresponding to the maximum score in the target prediction score vector as the predicted answer to the sample question includes: Select the target attribute word corresponding to the maximum score in the target prediction score vector corresponding to the attribute word, the target entity word corresponding to the maximum score in the target prediction score vector corresponding to the entity word, and the target relation word corresponding to the maximum score in the target prediction score vector corresponding to the relation word respectively; The answer consisting of the target attribute words, target entity words, and target relationship words is used as the predicted answer to the sample question.

8. A visual question answering device guided by structured question-image triples, characterized in that: The device comprises: An acquisition module, configured to acquire a target image and a target question for the target image; A problem feature extraction module is used to extract the problem global feature, multiple problem attribute triple features and multiple problem relationship triple features of the target problem using the first target model; the attribute triple is three fields representing the semantics of the object, and the relationship triple is three fields representing the relationship between the objects; an image feature extraction module, configured to extract a plurality of image attribute triple features of the target image using the second target model, and to extract a plurality of image relationship triple features of the target image using the third target model; a weight matrix determination module, configured to use a target attention model to respectively determine a first correlation between each of the image attribute triple features and each of the question attribute triple features, and to form an attribute attention weight matrix with a plurality of the first correlations; and to respectively determine a second correlation between each of the image relationship triple features and each of the question relationship triple features, and to form a relationship attention weight matrix with a plurality of the second correlations; a target feature acquisition module, configured to respectively multiply the attribute attention weight matrix by each of the image attribute triple features to obtain target image attribute triple features, and respectively multiply the relationship attention weight matrix by each of the image relationship triple features to obtain target image relationship triple features; The answer acquisition module is used to splice the target image attribute triple feature, the target image relationship triple feature and the global feature of the question and input them into the target answer classifier to obtain the answer information of the target question. The problem feature extraction module is specifically used for: Segmenting the target question into a plurality of tokens, and encoding the plurality of tokens into a plurality of feature vectors; Inputting the multiple feature vectors into the question encoding layer of the first model, obtaining the semantic relationship between the multiple word units, and obtaining the global feature of the target question; The global feature of the question is input into the triple decoding layer of the first model to obtain multiple question attribute triple features and multiple question relationship triple features of the target question.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the visual question answering method according to any one of claims 1 to 7.

Images