A visual question answering method, system, device and storage medium based on thought chain

Abstract

The present invention discloses a visual question-answering method, system, device, and storage medium based on thought chaining, comprising the following steps: obtaining an image and a corresponding question; parsing the question based on the thought chaining to obtain corresponding visual and knowledge prompt triples to form question sub-triplets; jointly training the image and visual prompt triples based on the LXMERT model to extract target features; querying the knowledge prompt triples from the common sense knowledge graph using a knowledge-to-text strategy and a stem-based BM25 algorithm to obtain explicit knowledge; performing graph representation on the question sub-triplets, target features, and explicit knowledge to heterogeneously obtain a heterogeneous multimodal graph, and performing cross-modal explicit reasoning to generate the answer to the question, completing the training of one image; repeating the above steps until the training of all images is completed to obtain the target visual question-answering model; inputting the image to be tested and the question text to obtain the visual question-answering result. The embodiment of the present invention can improve the accuracy of visual question-answering.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a visual question-answering method, system, device and storage medium based on thought chaining. Background Art

[0002] In existing technologies, open-ended knowledge learned by large multimodal models is considered implicit knowledge, while knowledge retrieved from a fixed, closed-domain knowledge base is considered explicit knowledge. Knowledge-integrated visual question answering methods leverage common sense to answer open-domain questions by introducing both explicit and implicit knowledge.

[0003] However, these knowledge-based visual question answering methods, on the one hand, neglect the combined reasoning power of vision and knowledge guided by language, leading to the introduction of irrelevant information or the loss of knowledge useful for inferring the final answer. On the other hand, methods that embed implicit knowledge provided by large models lack interpretability. As a result, the answers obtained by visual question answering using existing technologies are low in accuracy. Summary of the Invention

[0004] The present invention provides a visual question answering method, system, device and storage medium based on thought chain to overcome the defects of the above-mentioned existing technologies and improve the accuracy of visual question answering.

[0005] To achieve the above objectives, an embodiment of the present invention provides a visual question answering method based on thought chaining, comprising:

[0006] Get images and corresponding questions for training visual question answering models;

[0007] Training a visual question answering model based on the image and the corresponding question to obtain a target visual question answering model;

[0008] Input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result;

[0009] The step of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model includes:

[0010] Step 1: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets;

[0011] Step 2: Based on the LXMERT model, the image and the visual cue triplet are jointly trained to extract target features;

[0012] Step 3: Using the knowledge-to-text strategy and the stem-based BM25 algorithm, the searcher queries the knowledge hint triples from the common sense knowledge graph to obtain explicit knowledge;

[0013] Step 4: Represent the question sub-triples, the target features, and the explicit knowledge respectively to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously represent the graph structures of the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph;

[0014] Step 5: Perform cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, completing the training of one image;

[0015] Step 6: Repeat steps 1 to 5 until all images are trained and the target visual question answering model is obtained.

[0016] As an improvement to the above solution, the problem is parsed based on the thought chain to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triples, including:

[0017] Based on the thought chain, the end-to-end Transformer architecture is used to parse the problem, that is, each different word sequence in the problem (q1,...,q Q ) is encoded into multiple inference instruction sequence vectors [i (1) ,...,i (M) ]:

[0018] [i (1) ,...,i (M) ]=Seq2Seq(q1,...,q Q )

[0019] Among them, i represents the inference instruction sequence, M represents the total number of generated inference instruction sequence vectors, i (1) Represents the first inference instruction sequence vector of the parsing, i (M) represents the Mth inference instruction sequence vector of the parsing; q represents the word sequence in the question, q1 represents the first word sequence to be parsed, q Q Represents the Qth word sequence to be parsed; Seq2Seq() represents the end-to-end neural network architecture;

[0020] The reasoning instruction sequence vector corresponding to the question is converted into a visual prompt triple and a knowledge prompt triple to form a question sub-triplet.

[0021] As an improvement to the above solution, the LXMERT model is used to jointly train the image and the visual cue triplet to extract target features, including:

[0022] A word segmenter is used to segment the features of the visual cue triples into words and use them as input to the LXMERT model. The words and their indexes are projected into vectors through the embedding sublayer of the model and added to the index-aware word embedding to obtain word-level feature embeddings.

[0023] Based on a top-down and bottom-up strategy, the position features of the objects detected in the image and their 2048-dimensional region of interest features are used as the embedding of the image, and the position-aware embedding is output through layer normalization processing of two fully connected layers to obtain the object-level feature embedding;

[0024] The word-level features and the target-level features are fed into an encoder to locate the visual entity objects of the triples in the question, and the corresponding visual feature sequence is output to obtain target features related to the question prompt.

[0025] As an improvement to the above solution, the word-level feature embedding is obtained according to the following formula:

[0026]

[0027] Among them, i represents the index of the absolute position of the word in the sentence; w i Represents the i-th word; WordEmbed() represents the operation of mapping words into embedding vectors; Represents the vector of the i-th word after word embedding representation; IdxEmbed() represents the operation of mapping the index to the embedding vector; Represents the vector of the index of the i-th word after index embedding representation; LayerNorm() represents the layer normalization operation; h i Represents the vector obtained by concatenating the two represented vectors and performing layer normalization.

[0028] As an improvement to the above solution, the target-level feature embedding is obtained according to the following formula:

[0029]

[0030] Among them, LayerNorm() represents the layer normalization operation; j represents the integer index subscript, f j Represents the characteristics of the region of interest, represents the feature vector after layer normalization; p j represents the position features (i.e., bounding box coordinates), represents the position vector after layer normalization; F represents feature (subscript), P represents position (subscript); b F Indicates feature offset, b P Indicates position offset; W F Represents the feature weight vector matrix, WP represents the position weight vector matrix; v j represents position-aware embedding.

[0031] As an improvement to the above solution, the knowledge-to-text strategy and the stem-based BM25 algorithm are used to query the knowledge hint triples from the common sense knowledge graph through a retriever to obtain explicit knowledge, including:

[0032] A knowledge-to-text strategy is adopted to convert the image into a caption and the knowledge hint triples into sentences to unify the modalities;

[0033] For each visual information question pair (v, q), the stem-based BM25 algorithm queries the stems of the keywords in the knowledge hint triples from the common sense knowledge graph through the retriever; the stem-based BM25 algorithm uses the stem as the smallest semantic unit, deletes the repeated stems in the visual information v before connecting it with the stem in the question q, and merges the stem information to obtain the stems with s1, s2, ..., s t The stem-based sequence Squery is used and the similarity score of the knowledge triples retrieved in each document is calculated according to the following formula:

[0034]

[0035] Among them, w i Indicates the i-th word; Squery indicates all the words being queried; s f Represents the knowledge triples retrieved from the document; s i represents the i-th word in the query; t represents the total number of words in the query; R(s i , s f ) indicates the measurement of word s i and the knowledge triples retrieved from the document s f The semantic relevance between them; Score() means calculating the similarity score; N means the total number of documents in the index; n(s i ) means containing word s i The number of documents; IDF(s i ) represents word s i Similarity to the document;

[0036] Retrieve documents based on the similarity score and place the Top-K s f Connect them so that s f Serves as external knowledge for each visual information question pair (v, q), acquiring explicit knowledge related to the question prompt.

[0037] As an improvement to the above solution, performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question further includes:

[0038] Aggregating information in the visual graph, the question graph, and the knowledge graph through an attention-based graph convolutional network, and obtaining a converted entity answer;

[0039] Cross-modal knowledge selection and knowledge reasoning are iteratively performed in multiple steps to generate an answer to the question.

[0040] To achieve the above objectives, an embodiment of the present invention further provides a visual question answering system based on thought chaining, comprising:

[0041] The data acquisition module is used to obtain images and corresponding questions for training the visual question answering model;

[0042] A model training module is used to train the visual question answering model based on the image and the corresponding question to obtain a target visual question answering model;

[0043] A model application module is used to input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result;

[0044] The step of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model includes:

[0045] Step 1: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets;

[0046] Step 2: Based on the LXMERT model, the image and the visual cue triplet are jointly trained to extract target features;

[0047] Step 3: Using the knowledge-to-text strategy and the stem-based BM25 algorithm, the searcher queries the knowledge hint triples from the common sense knowledge graph to obtain explicit knowledge;

[0048] Step 4: Represent the question sub-triples, the target features, and the explicit knowledge respectively to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously represent the graph structures of the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph;

[0049] Step 5: Perform cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, completing the training of one image;

[0050] Step 6: Repeat steps 1 to 5 until all images are trained and the target visual question answering model is obtained.

[0051] To achieve the above-mentioned purpose, an embodiment of the present invention also provides a visual question-answering device based on thought chain, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above-mentioned visual question-answering method based on thought chain when executing the computer program.

[0052] To achieve the above-mentioned purpose, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-mentioned visual question-answering method based on the thinking chain.

[0053] The implementation of the embodiments of the present invention has the following beneficial effects:

[0054] The thought chain-based visual question answering method, system, device and storage medium provided by the embodiments of the present invention first obtain images and corresponding questions for training the visual question answering model; then, the visual question answering model is trained based on the images and corresponding questions to obtain a target visual question answering model; finally, the image to be tested and the question text are input into the target visual question answering model to obtain a visual question answering result. Among them, the steps of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model include: step 1, based on the thought chain, parsing the question, obtaining the corresponding visual prompt triple and knowledge prompt triple, and forming a question sub-triplet; step 2, based on the LXMERT model, jointly training the image and the visual prompt triple to extract the target feature; step 3, using the knowledge-to-text strategy and the stem-based BM25 algorithm, querying the knowledge prompt triple from the common sense knowledge graph through the retriever to obtain explicit knowledge; step 4, respectively representing the question sub-triplet, the target feature and the explicit knowledge to obtain a question graph, a visual graph and a knowledge graph, and heterogeneously representing the graph structures of the question graph, the visual graph and the knowledge graph to obtain a heterogeneous multimodal graph; step 5, performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate the answer to the question and complete the training of an image; step 6, repeating steps 1 to 5 until the training of all images is completed to obtain the target visual question answering model. By adopting the embodiment of the present invention, the problem is parsed through the thought chain to obtain the real scene triples containing the image entity and the virtual scene triples related to the knowledge entity. This not only involves a deeper level of knowledge and understanding of the visual scene; it also guides the joint reasoning of the visual modality and the knowledge modality through the gradual analysis of the problem, and more accurately locates the specific target in the image, so that the system can obtain a series of knowledge related to the target and finally return the answer to the user, thereby improving the accuracy of visual question answering and obtaining reliable results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a visual question answering method based on thought chain provided by an embodiment of the present invention;

[0056] Figure 2 is a schematic diagram of parsing a problem into a structured problem graph provided by an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of analyzing a problem based on a thought chain provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of a visual language unimodal encoding module provided by an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of a visual language cross-modal encoding module provided by an embodiment of the present invention;

[0060] Figure 6 is a sample image for model training provided by an embodiment of the present invention;

[0061] Figure 7 is a schematic diagram of cross-modal explicit reasoning provided by an embodiment of the present invention;

[0062] Figure 8 is a structural block diagram of a visual question answering system based on thought chain provided by an embodiment of the present invention;

[0063] Figure 9 This is a structural block diagram of a visual question-answering device based on thought chain provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] In the description of the present invention, it should be noted that the step numbers in the text are only for the convenience of explaining the specific embodiments and are not used to limit the order in which the steps are executed.

[0066] See also Figure 1 , Figure 1 : is a flowchart of a visual question answering method based on thought chaining provided by an embodiment of the present invention. The visual question answering method based on thought chaining includes steps S1 to S3, which are specifically as follows:

[0067] S1. Obtain images and corresponding questions for training the visual question answering model;

[0068] S2. Training a visual question answering model based on the image and the corresponding question to obtain a target visual question answering model;

[0069] S3, input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result;

[0070] The step of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model includes:

[0071] S21. Based on the thought chain, the question is parsed to obtain corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets;

[0072] S22. Based on the LXMERT model, jointly train the image and the visual cue triplet to extract target features;

[0073] S23, using the knowledge-to-text strategy and the stem-based BM25 algorithm, querying the knowledge hint triples from the common sense knowledge graph through a retriever to obtain explicit knowledge;

[0074] S24, respectively performing graph representation on the question sub-triples, the target features, and the explicit knowledge to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously performing graph structure transformation on the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph;

[0075] S25. Performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, thereby completing the training of one image.

[0076] S26. Repeat steps S21 to S25 until all images are trained and the target visual question answering model is obtained.

[0077] It is understandable that you can obtain the images and corresponding questions used to train the visual question answering model by yourself to construct a training sample set, or you can use the CRIC (Compositional Reasoning on Vision and Commonsense) dataset as the training sample set. The overall construction of this dataset includes the task decomposition, scene graph, knowledge graph, reasoning steps related to each question, and rich annotations of the corresponding results of answering each question, which helps to improve the interpretability and robustness of the model.

[0078] In an embodiment of the present invention, an image and a corresponding question are first obtained; then, the question is parsed based on a thought chain to obtain corresponding visual and knowledge prompt triples to form question sub-triplets; based on the LXMERT model, the image and visual prompt triples are jointly trained to extract target features; a knowledge-to-text strategy and a stem-based BM25 algorithm are used to query knowledge prompt triples from a common sense knowledge graph to obtain explicit knowledge; the question sub-triplets, target features, and explicit knowledge are respectively graph-represented and heterogeneously obtained into a heterogeneous multimodal graph, and cross-modal explicit reasoning is performed to generate the answer to the question and complete the training of an image; the above steps are repeated until the training of all images is completed to obtain the target visual question answering model; finally, the image to be tested and the question text are input to obtain the visual question answering result. By adopting an embodiment of the present invention, the accuracy of visual question answering can be improved and reliable results can be obtained. By adopting the embodiment of the present invention, the problem is parsed through the thought chain to obtain the real scene triples containing the image entity and the virtual scene triples related to the knowledge entity. This not only involves a deeper knowledge and understanding of the visual scene, but also guides the joint reasoning of the visual modality and the knowledge modality through the gradual analysis of the problem, and more accurately locates the specific target in the image, so that the system can obtain a series of knowledge related to the target and finally return the answer to the user, thereby improving the accuracy of visual question answering and obtaining reliable results.

[0079] In an optional embodiment, the problem is parsed based on the thought chain to obtain corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets, including:

[0080] Based on the thought chain, the end-to-end Transformer architecture is used to parse the problem, that is, each different word sequence in the problem (q1,...,q Q ) is encoded into multiple inference instruction sequence vectors [i (1) ,...,i (M) ]:

[0081] [i (1) ,...,i (M) ]=Seq2Seq(q1,...,q Q )

[0082] Among them, i represents the inference instruction sequence, M represents the total number of generated inference instruction sequence vectors, i (1) Represents the first inference instruction sequence vector of the parsing, i (M) represents the Mth inference instruction sequence vector of the parsing; q represents the word sequence in the question, q1 represents the first word sequence to be parsed, q Q Represents the Qth word sequence to be parsed; Seq2Seq() represents the end-to-end neural network architecture;

[0083] The reasoning instruction sequence vector corresponding to the question is converted into a visual prompt triple and a knowledge prompt triple to form a question sub-triplet.

[0084] It is understood that the i-th instruction vector will accurately correspond to the i-th execution step in the semantic parser. In order to make the semantics of the instruction vectors understandable to humans, each instruction vector is further translated into human-readable text using a transformer-based instruction vector decoder. The M instruction vectors are passed to the neural execution module instead of human-readable text.

[0085] A thought chain is a series of intermediate natural language reasoning steps that lead to the final output. When solving problems involving knowledge and reasoning posed on images, the problem needs to be broken down and then divided according to the human thinking process.

[0086] In the embodiment of the present invention, based on the thought chain, the problem is decomposed into intermediate steps and each sub-step is solved before the final answer is given. The problem parsing module designed finally can parse the problem into a structured problem graph (such as Figure 2 As shown in ), it guides multi-step reasoning based on vision and knowledge. Specifically, Figure 3 As shown, the question is parsed by a thought chain generator (Chain-Of-Thought Generation), and a triple subsequence of the question is obtained according to the sequence vector of different relationship types. Finally, it is converted into a knowledge triplet (Visual Prompt Triplet) and a knowledge prompt triplet (Knowledge Prompt Triplet) in the form of a thought chain to form a question sub-triplet. The present invention performs question parsing through thought chain to obtain a real scene triplet containing an image entity and a virtual scene triplet related to the knowledge entity. It not only involves a deeper level of knowledge and understanding of the visual scene, but also guides the joint reasoning of the visual modality and the knowledge modality through the gradual analysis of the problem, more accurately locates the specific target in the image, enables the system to obtain a series of knowledge related to the target, and ultimately returns the answer to the user, thereby improving the accuracy of visual question answering and obtaining reliable results.

[0087] In an optional embodiment, the LXMERT model-based joint training of the image and the visual cue triplet to extract target features includes:

[0088] A word segmenter is used to segment the features of the visual cue triples into words and use them as input to the LXMERT model. The words and their indexes are projected into vectors through the embedding sublayer of the model and added to the index-aware word embedding to obtain word-level feature embeddings.

[0089] Based on a top-down and bottom-up strategy, the position features of the objects detected in the image and their 2048-dimensional region of interest features are used as the embedding of the image, and the position-aware embedding is output through layer normalization processing of two fully connected layers to obtain the object-level feature embedding;

[0090] The word-level features and the target-level features are fed into an encoder to locate the visual entity objects of the triples in the question, and the corresponding visual feature sequence is output to obtain target features related to the question prompt.

[0091] It is understandable that in order to better guide the joint reasoning of image modality and knowledge modality, it is necessary to represent the object features of the image modality and the relationship features of the interaction between objects in a fine-grained manner. For this purpose, the LXMERT model is selected as the image feature extractor.

[0092] LXMERT is a cross-modal model with self-attention and cross-attention layers. In its two-stream architecture, images and text are treated as two distinct data streams, fed into two separate Transformers. These are then multimodally fused via an interaction module. LXMERT is a framework for cross-modal encoder representation learning, with the primary goal of exploring the interactive relationship between vision and language, particularly when processing a single image and its associated descriptive sentence. The framework consists of three Transformer encoders: an object relation encoder, a language encoder, and a cross-modal encoder. During pre-training, LXMERT employs multiple loss functions, including masked language modeling, masked object prediction, cross-modal matching, and image question answering, to help the model learn the associations between a single modality and cross-modality. During fine-tuning, the LXMERT model can be applied to a variety of visual reasoning tasks, achieving state-of-the-art results on these tasks.

[0093] It can be understood that the encoder includes a language encoder, an object relationship encoder and a cross-modal encoder, and the attention layer mainly based on it includes a self-attention layer and a cross-attention layer. Among them, the attention layer is intended to obtain a set of context vectors y related to the query vector x. j The attention layer first calculates the query vector x and each context vector y j The matching score a between j , and then normalize the scores through softmax:

[0094] a j =score(x+y j )

[0095] α j =exp(a j ) / ∑ k exp(a k )

[0096] Where j represents the integer index subscript; a j represents the attention layer matching score; α j represents the softmax normalized score; x represents the query vector; y j represents the context vector; k represents an integer subscript; a k Represents the k-th level matching score; score() represents the calculated score; exp() represents the mathematical operation exponent.

[0097] The output of the attention layer is the weighted sum of the context vectors. j When the attention layer is in the set, it is called self-attention. Specifically, the multi-head attention after using Transformer

[0098] For the visual-language unimodal encoding module, after the visual and language unimodal encoders are embedded in the embedding layer, two Transformer encoders are used, namely the language encoder and the object relationship encoder. Each encoder only focuses on a single modality (i.e., language or vision) and is applied to the language input and visual input (as well as the cross-modal input described below). Figure 4 As shown in the figure, each layer in the unimodal encoder consists of a self-attention sublayer and a feed-forward sublayer, where the feed-forward sublayer is further composed of two fully connected sublayers, ultimately constructing a multi-layer language encoder and a visual object relationship encoder. Residual connections and layer normalization are added after each sublayer (indicated by the "+" symbol in the figure).

[0099] For the visual language cross-modal encoding module, such as Figure 5As shown in the figure, each cross-modal layer in the cross-modal encoder consists of two self-attention sublayers, a bidirectional cross-attention sublayer, and two feed-forward sublayers. In the encoder implementation, these layers are stacked (i.e., the output of layer k is used as the input to layer k+1). Within layer k, a bidirectional cross-attention sublayer is first applied, which contains two unidirectional cross-attention sublayers: one from language to vision and one from vision to language. The cross-attention modality encoding module fuses the visual-related question prompt triples and visual features, ultimately accurately locating the visual entity objects in the triples in the question and outputting the corresponding visual feature sequence.

[0100] In an embodiment of the present invention, the LXMERT model accepts two inputs: an image and a visual cue triplet obtained by question parsing. Each image is represented as a sequence of object features, and the visual cue triplet is represented as a sequence of word features. By carefully designing and combining these self-attention layers and cross-attention layers, the model is able to generate language representation, image representation, and cross-modal representation from the input. Since the scale of LXMERT's pre-training data is very large and involves a large number of tasks, the LXMERT model is used to jointly train image and text pairs. On the one hand, it can give full play to LXMERT's ability in feature extraction, thereby obtaining fine-grained object information and regional interaction information in the image. On the other hand, during the training process, implicit knowledge about image features can be learned, including potential cognition about information such as the attributes of objects in the images in the dataset. This joint training method helps to improve the model's ability to understand the relationship between images and text.

[0101] For example, input a picture (such as Figure 6 As shown) and a description text (There is a wooden chair in the picture...), through model learning, we can obtain the material information about the chair in this picture, "The chair is wooden".

[0102] In an optional embodiment, the word-level feature embedding is obtained according to the following formula:

[0103]

[0104] Among them, i represents the index of the absolute position of the word in the sentence; w i Represents the i-th word; WordEmbed() represents the operation of mapping words into embedding vectors; Represents the vector of the i-th word after word embedding representation; IdxEmbed() represents the operation of mapping the index to the embedding vector; Represents the vector of the index of the i-th word after index embedding representation; LayerNorm() represents the layer normalization operation; h i Represents the vector obtained by concatenating the two represented vectors and performing layer normalization.

[0105] In the embodiment of the present invention, the WordPiece word segmenter is first used to segment the features of the visual cue triples into words {w1, ..., w n} is used as the input of the LXMERT model with a length of n. The word w is then embedded in the embedding sublayer of the model. i and its index i are projected into a vector and added to the index-aware word embedding to obtain word-level feature embedding.

[0106] In an optional embodiment, the target-level feature embedding is obtained according to the following formula:

[0107]

[0108] Among them, LayerNorm() represents the layer normalization operation; j represents the integer index subscript, f j Represents the characteristics of the region of interest, represents the feature vector after layer normalization; p j represents the position features (i.e., bounding box coordinates), represents the position vector after layer normalization; F represents feature (subscript), P represents position (subscript); b F Indicates feature offset, b P Indicates position offset; W F Represents the feature weight vector matrix, W P represents the position weight vector matrix; v j represents position-aware embedding.

[0109] In the embodiment of the present invention, in order to obtain fine-grained feature information of the relationship between objects and regions in an image, a top-down and bottom-up strategy is adopted to extract the m objects {o1, ..., o m} position feature p j and its 2048-dimensional region of interest feature f j As the embedding of the image. In order to obtain the regional relationship features between objects, the position-aware embedding v is output by adding two fully connected layers and layer normalization processing. j Get the relationship information and obtain the target-level feature embedding.

[0110] In an optional embodiment, the knowledge-to-text strategy and the stem-based BM25 algorithm are used to query the knowledge hint triples from the common sense knowledge graph through a searcher to obtain explicit knowledge, including:

[0111] A knowledge-to-text strategy is adopted to convert the image into a caption and the knowledge hint triples into sentences to unify the modalities;

[0112] For each visual information question pair (v, q), the stem-based BM25 algorithm queries the stems of the keywords in the knowledge hint triples from the common sense knowledge graph through the retriever; the stem-based BM25 algorithm uses the stem as the smallest semantic unit, deletes the repeated stems in the visual information v before connecting it with the stem in the question q, and merges the stem information to obtain the stems with s1, s2, ..., s t The stem-based sequence Squery is used and the similarity score of the knowledge triples retrieved in each document is calculated according to the following formula:

[0113]

[0114] Among them, w i Indicates the i-th word; Squery indicates all the words being queried; s f Represents the knowledge triples retrieved from the document; s i represents the i-th word in the query; t represents the total number of words in the query; R(s i , s f ) indicates the measurement of word s i and the knowledge triples retrieved from the document s f The semantic relevance between them; Score() means calculating the similarity score; N means the total number of documents in the index; n(s i ) means containing word s i The number of documents; IDF(s i ) represents word s i Similarity to the document;

[0115] Retrieve documents based on the similarity score and place the Top-K s f Connect them so that s f Serves as external knowledge for each visual information question pair (v, q), acquiring explicit knowledge related to the question prompt.

[0116] Understandably, the hyperparameter of 0.5 is primarily used for computational smoothing. Since only the stems of the search keywords are needed during knowledge retrieval, incorporating stem information enriches the retrieved knowledge and avoids the influence of different inflected words. Therefore, a slight improvement is made to BM25 to adapt to using stems as the smallest semantic unit. In particular, extra words with repeated stems in the visual information v are removed before concatenating them with the stems in the question q, rather than removing duplicates in the final query, thereby preserving the important information emphasized by the question q.

[0117] In an embodiment of the present invention, question parsing is performed based on thought chains to obtain sub-triples with knowledge prompts to guide knowledge retrieval. The knowledge graph is queried through the stem-based BM25 algorithm to obtain explicit knowledge related to the question prompt, thereby capturing the knowledge most relevant to the visual object and further improving the accuracy of relevant visual knowledge retrieval.

[0118] In an optional embodiment, performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question further includes:

[0119] Aggregating information in the visual graph, the question graph, and the knowledge graph through an attention-based graph convolutional network, and obtaining a converted entity answer;

[0120] Cross-modal knowledge selection and knowledge reasoning are iteratively performed in multiple steps to generate an answer to the question.

[0121] It can be understood that the heterogeneous multimodal graph contains subgraphs of multiple modalities, and the heterogeneous graph is composed of nodes and edges from different modalities. Since knowledge facts are usually triples, and questions and answers are in text format. In order to achieve the unification of data of three different modalities (i.e., vision, unstructured language and structured knowledge), the question sub-triples, the target features and the explicit knowledge are respectively represented by graphs to obtain question graphs, visual graphs and knowledge graphs, and the graph structures of the three modalities are heterogeneous, and cross-modal explicit reasoning is performed on the final heterogeneous multimodal graph, as shown in the following example. Figure 7 As shown, circles of different colors represent entity objects from different modalities, and edges represent the relationships between entities.

[0122] In an embodiment of the present invention, questions are divided into knowledge prompt triples and visual prompt triples based on thought chains, which better filters out irrelevant factual information, thereby selecting a set of candidate knowledge related to both the question and the image in the knowledge base. From globally comparing all entities and selecting the best entity as the answer, to screening knowledge as the best candidate knowledge based on triple-centered entity-relationship pairs, the representation of each entity in the resulting knowledge graph aggregates information oriented towards the question sub-triples. In order to jointly evaluate the possibility of each entity, an attention-based graph convolutional network is used to aggregate the information in the visual graph, the question graph, and the knowledge graph, and obtain a converted entity answer; cross-modal knowledge selection and knowledge reasoning are iteratively performed in multiple steps, and the model is able to select the best one from all candidate entities to generate the answer to the question. This process enables the model to fully consider visual, semantic, and question generation knowledge, thereby answering questions more accurately.

[0123] The visual question answering method based on thought chain provided by an embodiment of the present invention first obtains images and corresponding questions for training a visual question answering model; then, the visual question answering model is trained based on the images and corresponding questions to obtain a target visual question answering model; finally, the image to be tested and the question text are input into the target visual question answering model to obtain a visual question answering result. Among them, the steps of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model include: step 1, based on the thought chain, parsing the question, obtaining the corresponding visual prompt triple and knowledge prompt triple, and forming a question sub-triplet; step 2, based on the LXMERT model, jointly training the image and the visual prompt triple to extract the target feature; step 3, using the knowledge-to-text strategy and the stem-based BM25 algorithm, querying the knowledge prompt triple from the common sense knowledge graph through the retriever to obtain explicit knowledge; step 4, respectively representing the question sub-triplet, the target feature and the explicit knowledge to obtain a question graph, a visual graph and a knowledge graph, and heterogeneously representing the graph structures of the question graph, the visual graph and the knowledge graph to obtain a heterogeneous multimodal graph; step 5, performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate the answer to the question and complete the training of an image; step 6, repeating steps 1 to 5 until the training of all images is completed to obtain the target visual question answering model. By adopting the embodiment of the present invention, the problem is parsed through the thought chain to obtain the real scene triples containing the image entity and the virtual scene triples related to the knowledge entity. This not only involves a deeper level of knowledge and understanding of the visual scene; it also guides the joint reasoning of the visual modality and the knowledge modality through the gradual analysis of the problem, and more accurately locates the specific target in the image, so that the system can obtain a series of knowledge related to the target and finally return the answer to the user, thereby improving the accuracy of visual question answering and obtaining reliable results.

[0124] See also Figure 8 , Figure 8 is a structural block diagram of a thought chain-based visual question answering system 10 provided in an embodiment of the present invention. The thought chain-based visual question answering system 10 includes:

[0125] Data acquisition module 11, used to acquire images and corresponding questions for training the visual question answering model;

[0126] A model training module 12 is used to train a visual question answering model based on the image and the corresponding question to obtain a target visual question answering model;

[0127] A model using module 13 is used to input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result;

[0128] The step of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model includes:

[0129] Step 1: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets;

[0130] Step 2: Based on the LXMERT model, the image and the visual cue triplet are jointly trained to extract target features;

[0131] Step 3: Using the knowledge-to-text strategy and the stem-based BM25 algorithm, the searcher queries the knowledge hint triples from the common sense knowledge graph to obtain explicit knowledge;

[0132] Step 4: Represent the question sub-triples, the target features, and the explicit knowledge respectively to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously represent the graph structures of the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph;

[0133] Step 5: Perform cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, completing the training of one image;

[0134] Step 6: Repeat steps 1 to 5 until all images are trained and the target visual question answering model is obtained.

[0135] Optionally, the problem is parsed based on the thought chain to obtain corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets, including:

[0136] Based on the thought chain, the end-to-end Transformer architecture is used to parse the problem, that is, each different word sequence in the problem (q1,...,q Q ) is encoded into multiple inference instruction sequence vectors [i (1) ,...,i (M) ]:

[0137] [i (1) ,...,i (M) ]=Seq2Seq(q1,...,q Q )

[0138] Among them, i represents the inference instruction sequence, M represents the total number of generated inference instruction sequence vectors, i (1) Represents the first inference instruction sequence vector of the parsing, i (M) represents the Mth inference instruction sequence vector of the parsing; q represents the word sequence in the question, q1 represents the first word sequence to be parsed, q Q Represents the Qth word sequence to be parsed; Seq2Seq() represents the end-to-end neural network architecture;

[0139] The reasoning instruction sequence vector corresponding to the question is converted into a visual prompt triple and a knowledge prompt triple to form a question sub-triplet.

[0140] Optionally, the LXMERT model-based joint training of the image and the visual cue triplet to extract target features includes:

[0141] A word segmenter is used to segment the features of the visual cue triples into words and use them as input to the LXMERT model. The words and their indexes are projected into vectors through the embedding sublayer of the model and added to the index-aware word embedding to obtain word-level feature embeddings.

[0142] Based on a top-down and bottom-up strategy, the position features of the objects detected in the image and their 2048-dimensional region of interest features are used as the embedding of the image, and the position-aware embedding is output through layer normalization processing of two fully connected layers to obtain the object-level feature embedding;

[0143] The word-level features and the target-level features are fed into an encoder to locate the visual entity objects of the triples in the question, and the corresponding visual feature sequence is output to obtain target features related to the question prompt.

[0144] Optionally, the word-level feature embedding is obtained according to the following formula:

[0145]

[0146] Among them, i represents the index of the absolute position of the word in the sentence; w i Represents the i-th word; WordEmbed() represents the operation of mapping words into embedding vectors; Represents the vector of the i-th word after word embedding representation; IdxEmbed() represents the operation of mapping the index to the embedding vector; Represents the vector of the index of the i-th word after index embedding representation; LayerNorm() represents the layer normalization operation; h i Represents the vector obtained by concatenating the two represented vectors and performing layer normalization.

[0147] Optionally, the target-level feature embedding is obtained according to the following formula:

[0148]

[0149] Among them, LayerNorm() represents the layer normalization operation; j represents the integer index subscript, f j Represents the characteristics of the region of interest, represents the feature vector after layer normalization; p j represents the position features (i.e., bounding box coordinates), represents the position vector after layer normalization; F represents feature (subscript), P represents position (subscript); b F Indicates feature offset, b P Indicates position offset; W F Represents the feature weight vector matrix, W P represents the position weight vector matrix; v j represents position-aware embedding.

[0150] Optionally, the method adopts a knowledge-to-text strategy and a stem-based BM25 algorithm to query the knowledge hint triples from the common sense knowledge graph through a retriever to obtain explicit knowledge, including:

[0151] A knowledge-to-text strategy is adopted to convert the image into a caption and the knowledge hint triples into sentences to unify the modalities;

[0152] For each visual information question pair (v, q), the stem-based BM25 algorithm queries the stems of the keywords in the knowledge hint triples from the common sense knowledge graph through the retriever; the stem-based BM25 algorithm uses the stem as the smallest semantic unit, deletes the repeated stems in the visual information v before connecting it with the stem in the question q, and merges the stem information to obtain the stems with s1, s2, ..., s tThe stem-based sequence Squery is used and the similarity score of the knowledge triples retrieved in each document is calculated according to the following formula:

[0153]

[0154] Among them, w i Indicates the i-th word; Squery indicates all the words being queried; s f Represents the knowledge triples retrieved from the document; s i represents the i-th word in the query; t represents the total number of words in the query; R(s i , s f ) indicates the measurement of word s i and the knowledge triples retrieved from the document s f The semantic relevance between them; Score() means calculating the similarity score; N means the total number of documents in the index; n(s i ) means containing word s i The number of documents; IDF(s i ) represents word s i Similarity to the document;

[0155] Retrieve documents based on the similarity score and place the Top-K s f Connect them so that s f Serves as external knowledge for each visual information question pair (v, q), acquiring explicit knowledge related to the question prompt.

[0156] It is worth noting that the working process of each module in the visual question answering system 10 based on the thinking chain described in the embodiment of the present invention can refer to the working process of the visual question answering method based on the thinking chain described in the above embodiment, and will not be repeated here.

[0157] The thought chain-based visual question answering system provided by an embodiment of the present invention first obtains images and corresponding questions for training a visual question answering model; then, the visual question answering model is trained based on the images and corresponding questions to obtain a target visual question answering model; finally, the image to be tested and the question text are input into the target visual question answering model to obtain a visual question answering result. Among them, the steps of training the visual question answering model according to the image and the corresponding question to obtain the target visual question answering model include: step 1, based on the thought chain, parsing the question, obtaining the corresponding visual prompt triple and knowledge prompt triple, and forming a question sub-triplet; step 2, based on the LXMERT model, jointly training the image and the visual prompt triple to extract the target feature; step 3, using the knowledge-to-text strategy and the stem-based BM25 algorithm, querying the knowledge prompt triple from the common sense knowledge graph through the retriever to obtain explicit knowledge; step 4, respectively representing the question sub-triplet, the target feature and the explicit knowledge to obtain a question graph, a visual graph and a knowledge graph, and heterogeneously representing the graph structures of the question graph, the visual graph and the knowledge graph to obtain a heterogeneous multimodal graph; step 5, performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate the answer to the question and complete the training of an image; step 6, repeating steps 1 to 5 until the training of all images is completed to obtain the target visual question answering model. By adopting the embodiment of the present invention, the problem is parsed through the thought chain to obtain the real scene triples containing the image entity and the virtual scene triples related to the knowledge entity. This not only involves a deeper level of knowledge and understanding of the visual scene; it also guides the joint reasoning of the visual modality and the knowledge modality through the gradual analysis of the problem, and more accurately locates the specific target in the image, so that the system can obtain a series of knowledge related to the target and finally return the answer to the user, thereby improving the accuracy of visual question answering and obtaining reliable results.

[0158] In addition, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-mentioned visual question answering method based on the thinking chain.

[0159] See also Figure 9 , Figure 9This is a block diagram of a visual question-answering device 20 based on thought chaining, provided in an embodiment of the present invention. The visual question-answering device 20 based on thought chaining includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps of the visual question-answering method embodiment based on thought chaining are implemented. Alternatively, when the processor 21 executes the computer program, the functions of the modules / units in the aforementioned device embodiments are implemented.

[0160] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the thought chain-based visual question answering device 20.

[0161] The visual question-answering device 20 based on thought chain may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the visual question-answering device 20 based on thought chain and does not limit the visual question-answering device 20 based on thought chain. The device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the visual question-answering device 20 based on thought chain may also include input and output devices, network access devices, buses, and the like.

[0162] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the visual question-answering device 20 based on the thinking chain, and uses various interfaces and lines to connect various parts of the entire visual question-answering device 20 based on the thinking chain.

[0163] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the thought chain-based visual question-answering device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0164] Wherein, if the module / unit integrated in the visual question-answering device 20 based on the thinking chain is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0165] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A visual question answering method based on thought chain, characterized in that: include: Get images and corresponding questions for training visual question answering models; Training a visual question answering model based on the image and the corresponding question to obtain a target visual question answering model; Input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result; The step of training the visual question answering model based on the image and the corresponding question to obtain a target visual question answering model includes: Step 1: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets; Step 2: Based on the LXMERT model, the image and the visual cue triplet are jointly trained to extract target features; Step 3: Using the knowledge-to-text strategy and the stem-based BM25 algorithm, the searcher queries the knowledge hint triples from the common sense knowledge graph to obtain explicit knowledge; Step 4: Represent the question sub-triples, the target features, and the explicit knowledge respectively to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously represent the graph structures of the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph; Step 5: Perform cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, completing the training of one image; Step 6: Repeat steps 1 to 5 until all images are trained and the target visual question answering model is obtained.

2. The visual question answering method based on thought chain according to claim 1, characterized in that: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets, including: Based on the thought chain, the end-to-end Transformer architecture is used to parse the problem, that is, each different word sequence in the problem (q1,...,q Q ) is encoded into multiple inference instruction sequence vectors [i (1) ,...,i (M) ]: [i (1) ,...,i (M) ]=Seq2Seq(q1,...,q Q ) Among them, i represents the inference instruction sequence, M represents the total number of generated inference instruction sequence vectors, i (1) Represents the first inference instruction sequence vector of the parsing, i (M) represents the Mth inference instruction sequence vector of the parsing; q represents the word sequence in the question, q1 represents the first word sequence to be parsed, q Q Represents the Qth word sequence to be parsed; Seq2Seq() represents the end-to-end neural network architecture; The reasoning instruction sequence vector corresponding to the question is converted into a visual prompt triple and a knowledge prompt triple to form a question sub-triplet.

3. The visual question answering method based on thought chain according to claim 1, characterized in that: The LXMERT model is based on which the image and the visual cue triplet are jointly trained to extract target features, including: A word segmenter is used to segment the features of the visual cue triples into words and use them as input to the LXMERT model. The words and their indexes are projected into vectors through the embedding sublayer of the model and added to the index-aware word embedding to obtain word-level feature embeddings. Based on a top-down and bottom-up strategy, the position features of the objects detected in the image and their 2048-dimensional region of interest features are used as the embedding of the image, and the position-aware embedding is output through layer normalization processing of two fully connected layers to obtain the object-level feature embedding; The word-level features and the target-level features are fed into an encoder to locate the visual entity objects of the triples in the question, and the corresponding visual feature sequence is output to obtain target features related to the question prompt.

4. The visual question answering method based on thought chain according to claim 3, characterized in that: The word-level feature embedding is obtained according to the following formula: Among them, i represents the index of the absolute position of the word in the sentence; w i Represents the i-th word; WordEmbed() represents the operation of mapping words into embedding vectors; Represents the vector of the i-th word after word embedding representation; IdxEmbed() represents the operation of mapping the index to the embedding vector; Represents the vector of the index of the i-th word after index embedding representation; LayerNorm() represents the layer normalization operation; h i Represents the vector obtained by concatenating the two represented vectors and performing layer normalization.

5. The visual question answering method based on thought chain according to claim 3, characterized in that: The target-level feature embedding is obtained according to the following formula: Among them, LayerNorm() represents the layer normalization operation; j represents the integer index subscript, f j Represents the characteristics of the region of interest, represents the layer-normalized feature vector; p j represents the position features (i.e., bounding box coordinates), represents the position vector after layer normalization; F represents feature (subscript), P represents position (subscript); b F Indicates feature offset, b P Indicates position offset; W F Represents the feature weight vector matrix, W P represents the position weight vector matrix; v j represents position-aware embedding.

6. The visual question answering method based on thought chain according to claim 1, characterized in that: The knowledge-to-text strategy and the stem-based BM25 algorithm are used to query the knowledge hint triples from the common sense knowledge graph through a retriever to obtain explicit knowledge, including: A knowledge-to-text strategy is adopted to convert the image into a caption and the knowledge hint triples into sentences to unify the modalities; For each visual information question pair (v, q), the stem-based BM25 algorithm queries the stems of the keywords in the knowledge hint triples from the common sense knowledge graph through the retriever; the stem-based BM25 algorithm uses the stem as the smallest semantic unit, deletes the repeated stems in the visual information v before connecting it with the stem in the question q, and merges the stem information to obtain the stems with s1, s2, ..., s t The stem-based sequence Squery is used and the similarity score of the knowledge triples retrieved in each document is calculated according to the following formula: Among them, w i Indicates the i-th word; Squery indicates all the words being queried; s f Represents the knowledge triples retrieved from the document; s i represents the i-th word in the query; t represents the total number of words in the query; R(s i , s f ) indicates the measurement of word s i and the knowledge triples retrieved from the document s f The semantic relevance between them; Score() means calculating the similarity score; N means the total number of documents in the index; n(s i ) means containing word s i The number of documents; IDF(s i ) represents word s i Similarity to the document; Retrieve documents based on the similarity score and place the Top-K s f Connect them so that s f Serves as external knowledge for each visual information question pair (v, q), acquiring explicit knowledge related to the question prompt.

7. The visual question answering method based on thought chain according to claim 1, characterized in that: The performing cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question further includes: Aggregating information in the visual graph, the question graph, and the knowledge graph through an attention-based graph convolutional network, and obtaining a converted entity answer; Cross-modal knowledge selection and knowledge reasoning are iteratively performed in multiple steps to generate an answer to the question.

8. A visual question answering system based on thought chain, characterized in that: include: The data acquisition module is used to obtain images and corresponding questions for training the visual question answering model; A model training module is used to train the visual question answering model based on the image and the corresponding question to obtain a target visual question answering model; A model application module is used to input the image to be tested and the question text into the target visual question answering model to obtain the visual question answering result; The step of training the visual question answering model based on the image and the corresponding question to obtain a target visual question answering model includes: Step 1: Based on the thought chain, the question is parsed to obtain the corresponding visual prompt triples and knowledge prompt triples to form question sub-triplets; Step 2: Based on the LXMERT model, the image and the visual cue triplet are jointly trained to extract target features; Step 3: Using the knowledge-to-text strategy and the stem-based BM25 algorithm, the searcher queries the knowledge hint triples from the common sense knowledge graph to obtain explicit knowledge; Step 4: Represent the question sub-triples, the target features, and the explicit knowledge respectively to obtain a question graph, a visual graph, and a knowledge graph, and heterogeneously represent the graph structures of the question graph, the visual graph, and the knowledge graph to obtain a heterogeneous multimodal graph; Step 5: Perform cross-modal explicit reasoning on the heterogeneous multimodal graph to generate an answer to the question, completing the training of one image; Step 6: Repeat steps 1 to 5 until all images are trained and the target visual question answering model is obtained.

9. A visual question-answering device based on thought chain, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the visual question answering method based on thought chain according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the visual question answering method based on thought chain according to any one of claims 1 to 7.

Images