Visual Question Answering Method and System Based on Problem Guidance and Object - to - Object Difference Modeling

Abstract

The present invention relates to the technical field of visual question answering, and provides a visual question answering method and system based on question guidance and object - to - object difference modeling. The method includes: obtaining a to - be - detected image and a question, and extracting the visual features of the target region objects in the to - be - detected image and the word vectors in the question; constructing a question representation vector based on the word vectors; assigning different weight values to each word vector in the question representation vector to obtain a word attention representation vector; analyzing the implicit semantic relationship between words in the question representation vector to obtain the weights between words; constructing a word relationship representation vector based on the weights between words and the word vectors; constructing a question representation vector based on the word attention representation vector and the word relationship representation vector; comparing the differences between the target region objects under the guidance of a fine - grained question based on the visual features to obtain a graphic representation vector; predicting the answer to the question corresponding to the to - be - detected image based on the question representation vector and the graphic representation vector.

Description

Technical Field

[0001] The present invention relates to the technical field of visual question answering, and particularly relates to a visual question answering method and system based on question guidance and object - to - object difference modeling. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] Visual Question Answering (VQA) technology is a research direction that combines two major technologies: computer vision and natural language processing. Visual Question Answering technology refers to predicting the answer to a question corresponding to an image based on a given image and a natural language question related to the image.

[0004] Existing visual question answering technologies include:

[0005] 1. Visual Question Answering

[0006] Traditional VQA frameworks usually use pre - trained convolutional neural networks to extract visual features from images, and use recurrent convolutional neural networks to encode question features. The image and the question are encoded and represented as global features. Visual question answering work mainly explores and improves the performance of the basic model from four aspects. Multimodal embedding models such as bilinear pooling - based models focus on the feature fusion of pictures and questions. Attention - based models such as stacked attention gradually infer answers by querying the picture multiple times. Knowledge - based models attempt to use high - level attributes to improve visual understanding. Another direction is dedicated to solving spatial reasoning problems in abstract scenarios or developing composite networks.

[0007] 2. Visual Attention Mechanism

[0008] Visual attention is to solve the problem of "where to look" in the visual question answering model. Attention - based models aim to focus attention on the regions in the image related to the question or the most important words in the question, and concentrate limited attention on the most important parts of things according to actual needs. Attention methods in visual question answering include stacked attention, co - attention, multi - level attention, etc. These methods focus on locating important picture regions, attributes, or words in the question, so as to learn more fine - grained expressions.

[0009] 3. Graph Neural Networks

[0010] The VQA attention mechanism can focus on text keywords and visual objects, but it still needs to have a full understanding of the overall content and reason about complex questions. Graph neural networks (GNNs) are powerful frameworks for representing graph-structured data. GNNs follow an aggregation scheme that controls how the node representation vectors computed by neighboring nodes capture specific patterns of the graph. Recently, many variants of GNNs have been proposed to capture different types of patterns of the graph in many tasks. The graph network is a powerful model that can perform relational reasoning through message passing. The core idea is to enable communication between image regions to build context representations of these regions.

[0011] However, the currently proposed visual question answering technologies have the following problems:

[0012] 1. Visual question answering models focus on constructing new attention mechanisms and object recognition, ignoring the research on spatial reasoning and semantic relationships. The exploration of various visual relationships between objects in images still needs to be improved.

[0013] 2. Although some current graph models for VQA improve the interpretability of VQA models, previous methods ignore the importance of differences between objects, which is not conducive to establishing relationships between nodes in the graph.

[0014] 3. Currently, VQA models mostly simply use recurrent neural networks for feature extraction when processing questions, ignoring the fine-grained processing of questions. Due to the different importance of each word in solving problems, when there are a large number of objects in the image, simple feature extraction cannot effectively distinguish the relationships between objects in the image. Summary of the Invention

[0015] To solve the technical problems existing in the above background technology, the present invention provides a visual question answering method and system based on question guidance and modeling of differences between objects. First, the entire image and the entire question are encoded into high-level representations using a feature representation method, and then the question is processed in a fine-grained manner. A fusion object difference reasoning module is used to construct an object relationship graph based on the differences between objects under the guidance of the fine-grained question and update the node features; finally, multi-modal features are fused, and a multi-label loss function is used to predict the answer, thereby improving the accuracy of the visual question answering model.

[0016] To achieve the above object, the present invention adopts the following technical solutions:

[0017] The first aspect of the present invention provides a visual question answering method based on question guidance and modeling of differences between objects.

[0018] The visual question answering method based on question guidance and modeling of differences between objects includes:

[0019] Obtain the image to be detected and the question, and extract the visual features of the target region objects in the image to be detected and the word vectors in the question;

[0020] Based on the word vectors, construct a question representation vector; assign different weight values to each word vector in the question representation vector to obtain a word attention representation vector;

[0021] Analyze the implicit semantic relationships between words in the question representation vector to obtain the weights between words; based on the weights between words and the word vectors, construct a word relationship representation vector;

[0022] Based on the word attention representation vector and the word relationship representation vector, construct a question representation vector;

[0023] Based on the visual features, under the guidance of the fine-grained question, compare the differences between the target region objects to obtain a graphic representation vector;

[0024] Based on the question representation vector and the graphic representation vector, predict the answer to the question corresponding to the image to be detected.

[0025] Furthermore, the analysis of the implicit semantic relationships between words in the question representation vector adopts the following formula:

[0026] Use the dot product operation to calculate the correlation relationship m between the i-th word and the j-th word i,j :

[0027]

[0028] where W1, b1, W2, b2 are all learned parameters, and m i,j represents the proximity between two words, h i is the representation of word i, and h j is the representation of word j.

[0029] Furthermore, according to the correlation relationship m between the i-th word and the j-th word ij , calculate the weight between the i-th word and the j-th word.

[0030] Furthermore, the process of comparing the differences between the target region objects based on the visual features under the guidance of the fine-grained question to obtain a graphic representation vector includes:

[0031] Based on the comparison results of two different target region objects under the guidance of the question, use the attention mechanism to obtain the question-specific relationship between the two different target region objects;

[0032] Based on the question-specific relationship, use the attention graph convolutional layer to construct a graphic representation vector.

[0033] Further, the process of predicting the answer to the question corresponding to the image to be detected based on the question representation vector and the graphic representation vector includes: using a multimodal fusion inference model to fuse the question representation vector and the graphic representation vector to obtain a fused feature; based on the fused feature, using a fully connected network and a Sigmoid classifier to predict the answer to the question corresponding to the image to be detected.

[0034] Further, the multimodal fusion inference model includes a self-attention unit, a guided attention unit, a fully connected layer, and a multi-layer perceptron.

[0035] Further, the visual question answering method further includes a training process. During training, based on the fused feature, using a fully connected network and a Sigmoid classifier to predict the probability of each candidate answer; using a loss function to calculate the loss value between the true answer and the predicted candidate answer, and updating the network parameters of the multimodal fusion inference model according to the loss value.

[0036] The second aspect of the present invention provides a visual question answering system based on question guidance and object difference modeling.

[0037] The visual question answering system based on question guidance and object difference modeling includes:

[0038] A feature extraction module, which is configured to: obtain the image to be detected and the question, and extract the visual features of the target region objects in the image to be detected and the word vectors in the question;

[0039] A word attention representation module, which is configured to: based on the word vectors, construct a question representation vector; assign different weight values to each word vector in the question representation vector to obtain a word attention representation vector;

[0040] A word relationship representation module, which is configured to: analyze the implicit semantic relationship between words in the question representation vector to obtain the weights between words; based on the weights between words and the word vectors, construct a word relationship representation vector;

[0041] A question vector construction module, which is configured to: based on the word attention representation vector and the word relationship representation vector, construct a question representation vector;

[0042] An image fine-grained learning module, which is configured to: based on the visual features, under the guidance of the fine-grained question, compare the differences between the target region objects to obtain a graphic representation vector;

[0043] A prediction module, which is configured to: based on the question representation vector and the graphic representation vector, predict the answer to the question corresponding to the image to be detected.

[0044] The third aspect of the present invention provides a computer-readable storage medium.

[0045] A computer-readable storage medium stores a computer program thereon, and when the program is executed by a processor, it implements the steps in the visual question answering method based on question guidance and object-interrelationship modeling described in the first aspect above.

[0046] The fourth aspect of the present invention provides a computer device.

[0047] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual question answering method based on question guidance and object-interrelationship modeling described in the first aspect above.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] The two adaptive question attention modules designed by the present invention can explore the dependency relationships between keywords and other common words, and effectively identify the relationships between objects in an image.

[0050] The present invention can model the semantic relationships between objects based on the differences between objects, construct a question-specific relationship graph, reduce the redundancy of graph nodes, update the node features through a convolutional neural network, focus on the objects related to the question in the graph, and generate richer node information to improve the model prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0052] Figure 1 is a flowchart of the visual question answering method based on question guidance and object-interrelationship modeling shown by the present invention;

[0053] Figure 2 is a structural diagram of the network model shown by the present invention;

[0054] Figure 3 is a comparison graph of the ablation experiment results shown by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0057] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of methods and systems according to various embodiments of the present disclosure. It should be noted that each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. Similarly, it should be noted that each block in the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, may be implemented using a dedicated hardware-based system for performing the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0059] Embodiment 1

[0060] As Figure 1 shown, this embodiment provides a visual question answering method based on problem guidance and object difference modeling. This embodiment takes the application of this method to a server as an example. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, web servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here. In this embodiment, the method includes the following steps:

[0061] Obtain the image to be detected and the question, and extract the visual features of the target region object in the image to be detected and the word vectors in the question;

[0062] Based on word vectors, construct a problem representation vector; assign different weight values to each word vector in the problem representation vector to obtain a word attention representation vector;

[0063] Analyze the implicit semantic relationships between words in the problem representation vector to obtain the weights between words; based on the weights between words and word vectors, construct a word relationship representation vector;

[0064] Based on the word attention representation vector and the word relationship representation vector, construct a problem representation vector;

[0065] Based on visual features, under the guidance of fine-grained problems, compare the differences between object regions of interest to obtain a graphical representation vector;

[0066] Based on the problem representation vector and the graphical representation vector, predict the answer to the question corresponding to the image to be detected.

[0067] As Figure 2 shown, the specific solution of this embodiment can be implemented with reference to the following content:

[0068] (1) Data Embedding

[0069] The visual model uses the Faster RCNN network (with ResNet-101 as its backbone) pre-trained on the Visual Genome dataset to detect significant regions of interest in the image and extract the visual features in each region. Specifically, for each image, the top K most relevant regional visual features and their spatial features where represents the i th th visual feature, and c i = [x i , y i , w i , h i corresponds to the spatial location of the object, where (x i , y i ) is the lower left corner coordinate of the bounding box, and w i and h i are the width and height respectively.

[0070] For question embedding, first use the 300-D Glove word embedding method pre-trained on a large corpus to tokenize the input question consisting of n words into words, convert each word into a vector representation q f = (q1, q2,..., q n ), and then feed these vectors into a bidirectional GRU to obtain the question representation h q, where all questions are truncated or padded to the same length, and the encoding process is represented as follows.

[0071] h t = GRU(h t-1 , q f ), t ≤ n (1)

[0072] h q = (h1, h2, …, h n ), (2)

[0073] (2) Fine-grained question relation representation

[0074] To obtain a more expressive question representation, this embodiment designs two adaptive question attention modules, focusing on the key words and the implicit semantic relationships between words in the question.

[0075] In word attention, different weight values are assigned to each word according to its importance in the question. When answering this question, words that describe an object or a relationship are often more important. The weight vector p w and the question representation vector v w are calculated as follows:

[0076]

[0077] where, W w is the parameter vector, is the i th th element of the weight vector, representing the importance of the i th th word in the question.

[0078] In addition, the implicit semantic relationships between words are explored through the word relationship module. The importance of different words in the question varies. To obtain a fine-grained question representation, words with the characteristics of nouns, verbs, adjectives, and adverbs are used as core words, and the rest are used as regular words to identify all the characteristics of the question words. Words i and j are represented by h i , h j respectively.

[0079] The dot product operation is used to calculate the correlation relationship m i,j between words i and j as follows:

[0080]

[0081] where, W1, b1, W2, b2 are all learned parameters, and m i,j represents the proximity between two words. Then, the correlation m ij between the i i,jThe calculation is as follows:

[0082]

[0083] where n is the problem length.

[0084] To obtain the most valuable semantic relationships between words, based on the weight n i,j and the original word representations, new word representations are obtained as follows:

[0085]

[0086] Then the Maxpooling function is used to obtain the most relevant word representation v r . Thus, the overall word relationship representation is obtained:

[0087]

[0088] Finally, the word attention representations and word relationship representations obtained from the two adaptive problem attention modules are combined. After that, we effectively add these two bits together to generate the final representation of the problem.

[0089] To map the vectors of image features and text features into a common space, as follows, a one-dimensional convolutional layer and a fully connected layer are used to transform the image and problem features respectively.

[0090] V = ReLU(Conv1d(v f )) (8)

[0091] q = ReLU(Lineat(v q )) (9)

[0092] (3) Relationship reasoning for fusing object differences

[0093] Generally speaking, the object itself is often not sufficient to provide enough information. Due to the diversity of problems related to pictures, it is hoped that the relationships between objects are problem-specific. In this embodiment, a new object graph construction strategy is adopted. Under the guidance of fine-grained problems, object differences are compared to generate a graphical representation of the input image.

[0094] This process includes an operation of constructing an object relationship graph based on object differences and a graph convolution operation after constructing the relationship graph.

[0095] (3-1) Operation of constructing an object relationship graph based on object differences

[0096] An object relationship graph G=(V,E) is constructed based on object differences, where the nodes V = v1, v2,..., v kis defined in the previous section. Edge E represents the relationships between objects and their relevance to a given problem. The object relationship graph learned in (3-1) is passed to the graph convolutional layer in (3-2), and each node collects adjacent node information according to the adjacency matrix.

[0097] The comparison provides more information on how to assign different attention weights to different objects. The object targets that the model focuses on are two objects v i and v j . The problem-specific relationship between them can be learned by comparing them under the guidance of problem q through formula (10):

[0098] A ij = [(v i - v j ) ⊙ q] d W p , (10)

[0099] where is the comparison result between the i-th image object and the j-th image object under the guidance of problem q. It is an m×md matrix, and the i-th row of the matrix represents the comparison result of the i-th image object with all other image objects, and it is also the attention kernel in the relational attention. is the attention kernel, which is a learnable parameter that determines how to act on the object targets that the model focuses on and converts the difference comparison result of the image objects into an attention distribution.

[0100] For any object v i , v i is compared with all other objects under the guidance of problem q, which is crucial for measuring the importance of v i in answering the question.

[0101] Specifically, as Figure 2 shows, ⊙ represents element-wise multiplication, and the adjacency matrix A ij represents the relationship between the i-th object v i and the j-th object v j . To not ignore the relationship of the object itself, the identity matrix I is added after the Softmax operation on the original adjacency matrix A as in formula (11).

[0102] A = Softmax(A) + I, (11)

[0103] Since visual problems are related to a small subset of all objects and relationships in the visual scene, the graph sparsity is restricted. For each node, only the top m most relevant objects and its neighbors N i = Top(A i ) are selected.

[0104] Thus, the adjacency matrix as shown in Figure 2 is obtained. The objects V = v1, v2, …, v k-1 , v k in the matrix are represented as nodes in the object graph, and the relationship correlation between the objects in the matrix is represented as edges in the object graph. Finally, the encoded image will be converted into problem-adaptive object relationship graph learning and fed into the attention graph convolutional layer to update the representations of objects and visual relationships.

[0105] (3-2) Graph Convolutional Operations after Constructing the Relationship Graph

[0106] During the graph convolution process, as the number of convolutional layers deepens, the objects in the graph that are irrelevant to the problem will be assigned small weights that can be ignored until they are almost completely ignored. As shown in Equation (12), the contribution weight α i of an object is calculated by the dot product between the image object v i after Softmax normalization and the problem vector q.

[0107]

[0108] A set of K Gaussian kernels is used to learn the mean and covariance as patch operators to describe the influence of each neighborhood, where the mean can be interpreted as the direction and distance of the pseudo-coordinates of two nodes. After that, the kernel weight w n (u(c i , c j )) of each node k (k ∈ K) is obtained, representing the nth Gaussian kernel weight of the jth neighbor of node i, that is, the relative position information. As shown in Equation (13), g n (i) can be regarded as the weighted sum of the adjacent features of node i:

[0109]

[0110] where w n is the learned weight matrix, N (i) is the set of nodes adjacent to node i, v j represents the jth neighbor of node i, A ij represents the edge weight between node i and node j, and α j is the attention weight of the object v j . Then, the outputs {g1(i), g2(i), …, g N (i)} of all N kernels are concatenated as the convolution operation result g(i) of node i.

[0111] Finally, by referring to the method of aggregating node features with multiple convolutional kernels, the same operation is performed on each node as follows:

[0112]

[0113] Among them, G i is a learnable weight matrix, and h v is the updated image feature. In this way, in the context of a specific problem, each new object contains rich neighborhood information.

[0114] (4) Multimodal fusion and answer prediction

[0115] The multimodal fusion inference module consists of a self-attention unit SA, a guided attention unit GA, a fully connected layer Linear, and a multi-layer perceptron MLP. After L stacked SA and GA modules, the output image feature X L (the previously calculated h v ) and the question feature Y L (the previously calculated v q ) contain richer information in the text and image regions. Taking X L as an example, it is obtained through formulas (15) and (16) Similarly, it can be obtained Finally, the final classification-normalized fusion feature Z is obtained through formula (17):

[0116] s = softmax(MLP(X L )) (15)

[0117]

[0118]

[0119] The above takes X L as an example and is obtained through (15) and (16) Similarly, it can also be obtained from Y L through the same calculation process (15) and (16).

[0120] The final classification-normalized fusion feature Z is input into a fully connected network and a Sigmoid classifier to calculate the probability of each candidate answer:

[0121] p a = softmax(W a (σ(W b Z))) (18)

[0122] Among them, W a and W b are trainable parameters, and p a represents the probability vector of the answer.

[0123] Using KL-divergence as the loss function, the training classification problem is as follows:

[0124]

[0125] Among them, y i ∈[0,1] is the label of the i-th true answer.

[0126] In order to verify the effectiveness of the model proposed in this embodiment, an ablation experiment was conducted on the VQAv2.0 dataset. Baseline represents a baseline model. FQ+GCN represents the use of fine-grained word relationship representation to process words based on the baseline; OD_None-Att+GCN represents the use of object difference graph-based relationship learning without introducing a question-guided soft attention mechanism; OD+GCN: a method without fine-grained word representation, integrating object difference relationship reasoning; FQOD-GCN is the complete visual question answering model proposed in this embodiment. Figure 3 As shown in the figure, the results show that the fine-grained word representation module and the relational reasoning module that integrates object differences described in this embodiment have greatly improved compared with the baseline model, which illustrates the effectiveness of the modules proposed in this model.

[0127] Visual question answering can be applied to navigation for the blind. Figure 2 For example, consider the image and question "Is he walking dogs?" in the example. First, we identify and locate important object regions in the image relevant to the question, comparing these identified objects one by one. Next, we aim to focus on the key words "he" and "dogs" in the question, as well as the latent semantic relationship between them, "walking." Through these two steps, we extract question-adaptive relationships between objects based on the specific question and construct a relationship graph. This accurately focuses on the relevant objects and question-adaptive relationships, ultimately arriving at the answer "yes."

[0128] In addition, the solution of this embodiment can also be applied to early childhood education to teach children to "understand" the world, which will not be elaborated here.

[0129] Example 2

[0130] This embodiment provides a visual question answering system based on question guidance and inter-object difference modeling.

[0131] A visual question answering system based on question guidance and object difference modeling, including:

[0132] A feature extraction module is configured to: obtain an image to be detected and a question, extract visual features of a target area object in the image to be detected and a word vector in the question;

[0133] The word attention representation module is configured to: construct a question representation vector based on word vectors; assign different weight values to each word vector in the question representation vector to obtain a word attention representation vector;

[0134] The word relationship representation module is configured to: analyze the implicit semantic relationships between words in the question representation vector to obtain the weights between words; construct a word relationship representation vector based on the weights between words and word vectors;

[0135] The question vector construction module is configured to: construct a question representation vector based on the word attention representation vector and the word relationship representation vector;

[0136] The image fine-grained learning module is configured to: compare the differences between target region objects under the guidance of fine-grained questions based on visual features to obtain a graphic representation vector;

[0137] The prediction module is configured to: predict the answer to the question corresponding to the image to be detected based on the question representation vector and the graphic representation vector.

[0138] It should be noted here that the above feature extraction module, word attention representation module, word relationship representation module, question vector construction module, image fine-grained learning module, and prediction module have the same examples and application scenarios as those implemented in the steps of the first embodiment, but are not limited to the content disclosed in the first embodiment. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0139] Embodiment III

[0140] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in the visual question answering method based on question guidance and object difference modeling as described in the first embodiment above.

[0141] Embodiment IV

[0142] This embodiment provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual question answering method based on question guidance and object difference modeling as described in the first embodiment above.

[0143] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory and optical memory, etc.) that contain computer-usable program code.

[0144] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0145] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0147] Those of ordinary skill in the art can understand that to implement all or part of the processes in the above-described embodiment methods, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0148] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A visual question answering method based on problem guidance and object - to - object difference modeling, characterized in that, Including: Obtain the image to be detected and the question, and extract the visual features of the target region object in the image to be detected and the word vectors in the question; Based on the word vectors, construct a question representation vector; assign different weight values to each word vector in the question representation vector to obtain a word attention representation vector; Analyze the implicit semantic relationship between words in the question representation vector to obtain the weights between words; based on the weights between words and the word vectors, construct a word relationship representation vector; Based on the word attention representation vector and the word relationship representation vector, construct a question representation vector; Based on the visual features, under the guidance of the fine-grained question, compare the differences between the target region objects to obtain a graphical representation vector; Based on the question representation vector and the graphical representation vector, predict the answer to the question corresponding to the image to be detected The process of analyzing the implicit semantic relationship between words in the question representation vector adopts the following formula: Calculate the correlation between the th word and the j th word using the dot product operation : wherein, are all learned parameters, represents the proximity between two words, is the representation of the word , is the representation of the word ; The process of comparing the differences between the target region objects based on the visual features under the guidance of the fine-grained question to obtain a graphical representation vector includes: Based on the comparison results of two different target region objects under the guidance of the question, adopt an attention mechanism to obtain the question-specific relationship between the two different target region objects; Based on the question-specific relationship, adopt an attention graph convolutional layer to construct a graphical representation vector.

2. The visual question answering method based on problem guidance and object - to - object difference modeling according to claim 1, wherein, According to the relationship between the j th word and the th word, calculate the weight between the j th word and the th word.

3. The visual question answering method based on problem guidance and object - to - object difference modeling according to claim 1, characterized in that, The process of predicting the answer to the question corresponding to the image to be detected based on the question representation vector and the graphical representation vector includes: adopt a multi-modal fusion inference model to fuse the question representation vector and the graphical representation vector to obtain a fused feature; based on the fused feature, adopt a fully connected network and a Sigmoid classifier to predict the answer to the question corresponding to the image to be detected.

4. The visual question answering method based on problem guidance and object - to - object difference modeling according to claim 3, characterized in that, The multi-modal fusion inference model includes a self-attention unit, a guiding attention unit, a fully connected layer, and a multi-layer perceptron.

5. The visual question answering method based on problem guidance and object - to - object difference modeling according to claim 3, characterized in that, The visual question answering method further includes a training process. During training, based on the fused feature, adopt a fully connected network and a Sigmoid classifier to predict the probability of each candidate answer; adopt a loss function to calculate the loss value between the true answer and the predicted candidate answer, and update the network parameters of the multi-modal fusion inference model according to the loss value.

6. A visual question answering system based on problem guidance and object - to - object difference modeling, characterized in that, Including: A feature extraction module configured to: obtain the image to be detected and the question, and extract the visual features of the target region object in the image to be detected and the word vectors in the question; A word attention representation module configured to: based on the word vectors, construct a question representation vector; assign different weight values to each word vector in the question representation vector to obtain a word attention representation vector; A word relationship representation module configured to: analyze the implicit semantic relationship between words in the question representation vector to obtain the weights between words; based on the weights between words and the word vectors, construct a word relationship representation vector; A question vector construction module configured to: based on the word attention representation vector and the word relationship representation vector, construct a question representation vector; An image fine-grained learning module configured to: based on the visual features, under the guidance of the fine-grained question, compare the differences between the target region objects to obtain a graphical representation vector; A prediction module configured to: based on the question representation vector and the graphical representation vector, predict the answer to the question corresponding to the image to be detected; The implicit semantic relationship between words in the analysis problem representation vector adopts the following formula: Calculate the correlation between the th word and the j th word using the dot product operation : Among them, are all learned parameters, represents the proximity between two words, is the representation of the word ; is the representation of the word ; The process of obtaining the graphical representation vector based on visual features and comparing the differences of target region objects under the guidance of fine-grained problems includes: Based on the comparison results of two different target region objects under the guidance of the problem, an attention mechanism is adopted to obtain the problem-specific relationship between the two different target region objects; Based on the problem-specific relationship, an attention graph convolutional layer is adopted to construct the graphical representation vector.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the visual question answering method based on problem guidance and object difference modeling described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the visual question answering method based on problem guidance and object difference modeling described in any one of claims 1-5.

Images