A Visual Question Answering Method for Cognitive Reasoning Based on Cognitive Dual Channels

Abstract

The present invention relates to the fields of computer multi-modal information processing and cognitive science, and mainly relates to a visual question answering method for cognitive reasoning based on a cognitive dual-channel. It mainly includes the following steps: Step 1, construct cognitive content, extract the question keywords and the label content of the image target area as the index for retrieving a large knowledge base, and construct a task knowledge base through the retrieved content; Step 2, perform prior cognitive calculation, and train the visual text content representation through a multi-modal pre-training model; Step 3, construct the spatio-temporal characteristics of the task reasoning. By performing syntactic analysis and part-of-speech analysis on the question sentence, a topological graph between the question keyword words is constructed, and the association degree of the internal area of the image is calculated using the visual representation content in Step 2 to construct the image content spatial graph structure; Step 4, lock the image content related to the question and answer content, and calculate the image content concerned by each text vector according to the visual vector and text vector of the joint representation in Step 2 to construct the question and answer related image content; Step 5, perform hierarchical cognitive reasoning, combine the knowledge content constructed in Step 1 with the prior calculation in Step 2, re-encode the question and answer content, and combine the re-encoded representation content with the spatio-temporal characteristics analyzed in Step 3 to construct a question and answer instruction set, and perform question and answer reasoning through the question and answer instructions to obtain the result of visual question answering. The present invention improves the accuracy of the open-domain visual question answering model, continuously corrects the cognitive understanding of the question and answer content according to the external knowledge content during the reasoning process, and makes the visual question answering process robust and interpretable.

Description

Technical Field

[0001] The present invention relates to the fields of computer multi-modal information processing and cognitive science, and particularly relates to a visual question answering method for cognitive reasoning based on a cognitive dual-channel. Background Art

[0002] In ordinary visual question answering tasks, good results have been achieved through the learning of multi-modal information representation, but the reasoning nature of the question answering process has been ignored, resulting in a lack of interpretability in the question answering process. At the same time, when encountering complex visual question answering tasks, that is, the question answering relationship exceeds the given conditions and prior knowledge or general facts are needed to answer the question, there will be a large deviation in this kind of visual question answering, resulting in an off-topic answer. In the dual-channel theory of cognitive science, there are two systems in the cognitive system of the human brain: System1 and System2. System1 is an intuitive system that can find answers through an intuitive match of relevant information by a person, which is rapid and simple. While System2 is an analytical system that finds answers through certain reasoning and logic. The cognitive idea of the dual-channel can quickly understand the task through the way of task representation for System1. For complex visual question answering tasks, based on the understanding of 1, analysis and reasoning are carried out through 2, thus retaining the computer's understanding of the task and enabling the computer to realize the reasoning process of the task.

[0003] The method based on representation learning can enable the computer to well understand the task content. Through the joint representation of the task content, it cannot process information irrelevant to the task. At the same time, the represented content can only be limited to the given conditions themselves and cannot understand the correlation relationships outside the content. The method based on retrieval can search a large amount of corpus information as evidence for question answering and can solve the limitation of representation brought by content limitation. However, the corpus retrieved by it increases the computational cost. At the same time, the introduction of irrelevant corpus will deviate the distance between the original computed feature representation and the correct answer, affecting the final computation result. Summary of the Invention

[0004] The present invention is made to solve the above problems, and aims to provide a visual question answering algorithm for cognitive reasoning based on a cognitive dual-channel, which is used to solve the visual question answering method in the open domain, dynamically learn the question answering method, and improve the accuracy and interpretability of question answering. The present invention adopts the following technical solutions:

[0005] The present invention provides a visual question answering method for cognitive reasoning based on a cognitive dual-channel, which is characterized by the following steps: Step S1, extract question keywords and image target region labels, and construct an index set based on the keywords and the target region labels; Step S2, based on the index set, retrieve and construct a knowledge base related to the visual question answering task to obtain cognitive content; Step S3, through a cross-modal model pre-trained in multiple modalities, calculate and train a cross-modal representation model for the text modality and the image modality; Step S4, by fine-tuning the cross-modal representation model in the visual question answering task, obtain a primary cognitive model, and use the cognitive model to calculate various representation vectors; Step S5, through syntactic analysis and part-of-speech analysis of the question sentence, construct a topological structure between the question keywords, and update the topological structure using the part-of-speech of the question keywords, so as to obtain the time characteristics of question reasoning; Step S6, based on the visual question answering task, calculate the spatial characteristics of the image target region; Step S7, based on the spatial characteristics of the image target region, obtain a correlation matrix of semantic matching of the image target region, and construct a task association graph structure between cross-modalities; Step S8, based on the cognitive content and the various representation vectors, re-encode the visual question answering content to obtain re-encoded representation content; Step S9, based on the re-encoded representation content, the time characteristics of question reasoning, and the spatial characteristics of the image target region, construct a question and answer instruction set, and perform question and answer reasoning through the question and answer instruction set, so as to obtain the result of the visual question answering.

[0006] In the visual question answering method for cognitive reasoning based on a cognitive dual-channel provided by the present invention, it may further have the following technical feature: in Step S1, for the input question, word segmentation is first performed to obtain question words and phrases, then the corresponding image is input for the input question, the target region of the image is divided, and the target region is classified, so as to construct the index set.

[0007] In the visual question answering method for cognitive reasoning based on a cognitive dual-channel provided by the present invention, it may further have the following technical feature: in Step S2, to construct the knowledge base related to the visual question answering task, source nodes and target nodes are first constructed based on the relationships between parts of speech and cross-modal symbolic representations, then an algorithm is used to find all the shortest path sets from the source nodes to the target nodes, and then based on the commonality of the question and answer content, the shortest path sets are merged, so as to construct the knowledge base related to the visual question answering task.

[0008] The visual question answering method based on cognitive dual channels for cognitive reasoning provided by the present invention may further have the following technical features. Among them, the various representation vectors at least include a text modality vector, an image modality vector, and a linear mapping vector. In step S6, calculating the spatial characteristics includes the following sub-steps: Step S6-1, by jointly representing the text modality vector and the image modality vector, calculating the linear mapping vector, and the formula is as follows: JoinR k =f([R k |S]), k ∈ Ω, where f is a linear mapping function, and R k is the feature representation vector of the Kth target region on the input image, and S represents the sentence feature vector calculated by the primary cognitive model; Step S6-2, calculating the correlation matrix within the target region through matrix multiplication, and the formula is as follows: Relation = JoinR × JoinR T ; Step S6-3, according to the rectangular coordinate system of the target region, calculating the corresponding polar coordinates (r, θ) as its physical spatial relationship:

[0009] The visual question answering method based on cognitive dual channels for cognitive reasoning provided by the present invention may further have the following technical features. Among them, step S7 further includes the following sub-steps: Step S7-1, calculating the overall similarity Sam SF of the cross-modal task, and the expression is as follows: where S is the question semantic vector and F is the image i representation vector; Step S7-2, using the mean pooling method to calculate the keyword feature representation vector EP k , and the formula is as follows: Step S7-3, based on an attention-based interrogation mechanism, thereby calculating the correlation matrix.

[0010] The visual question answering method based on cognitive dual channels for cognitive reasoning provided by the present invention may further have the following technical features. Among them, the correlation matrix includes a weight matrix of the question and answer task and the image target region, and an attention score matrix of the keyword and the content of the image target region. Step S7-3 further includes: Step A1, by splicing the question representation vector and the image representation feature for task representation: T = σ[S, F]

[0011] where σ is an activation function; Step A2, calculating the weight matrix and processing the weight matrix through a smoothing method, and the formula is as follows: W tr =g(S, R); Step A3, calculating the keyword linear mapping vectors K, Q, V, and setting learning parameters W q 、W v : K = R r×W tr , Q = Key r ×W q , V = Key r ×W v Wherein, K represents the query vector of the image for the keyword, Q represents the query vector of the keyword for the target region of the image, and V represents the mapped value vector of the keyword; Step A4, calculate each attention score matrix Att: Step A5, calculate the discreteness between each attention score matrix and the Q&A task itself, and the formula is as follows: Where n is the dimension of the vector; Step A6, based on the overall similarity Sam SF , select TopN(Scorre).

[0012] The visual question answering method based on cognitive dual channels and cognitive reasoning provided by the present invention may also have the following technical features. Among them, the re-encoding of the visual question answering content in step S8 includes: calculating the similarity between the representation vector of the Q&A task and the relationship or tail entity in the tuple of the Q&A task knowledge base, and the expression is as follows: Where n is the vector dimension.

[0013] Functions and effects of the invention

[0014] According to the visual question answering method based on cognitive dual channels and cognitive reasoning of the present invention, since a novel machine-based hierarchical cognitive question answering algorithm is proposed, mainly through pre-training the model to enable the model itself to have an understanding of the task, constructing the first intuitive system System1 of the cognitive system, and through fine-tuning the task model, making it adaptable to new tasks; then by splitting the task, obtaining the symbolic language inherent in the task itself, and constructing a knowledge base related to the task through retrieval as a supplement to the content of the task to improve the intuition of the cognitive system; finally constructing the cognitive System2, including the content of two stages. In the first stage, the task is re-understood. For the intuitive understanding of System1, some of the question answering content is solved, but for non-intuitive questions, it is necessary to first re-define the task representation on the basis of System1, and under the conditions of System1, calculate the degree of association with the corpus evidence, calculate the content related to the task, and limit the content representation. In the second stage, the reasoning content is visualized according to the content of the cognitive knowledge base, a knowledge graph representing the task is constructed, and for each cognitive graph, through the constructed spatio-temporal order, the relationship reasoning between levels is carried out to obtain the final question answering answer. The present invention greatly improves the accuracy of the question answering result and continuously corrects the cognitive understanding of the question answering content according to the external knowledge content during the reasoning process, making the visual question answering process robust and interpretable. Brief Description of the Drawings

[0015] Figure 1 is a schematic flowchart of the visual question answering method for cognitive reasoning based on cognitive dual channels in an embodiment of the present invention; Detailed Embodiments

[0016] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the visual question answering method and device for cognitive reasoning based on cognitive dual channels of the present invention will be specifically described below in conjunction with embodiments and drawings.

[0017] <Embodiment>

[0018] In this embodiment, two systems are constructed, namely an intuitive system and a cognitive system. The intuitive system facilitates the rapid recognition, understanding, and analysis of complex visual question answering tasks; the cognitive system is based on the intuitive system and retains the computer's understanding of the question answering task through analysis and reasoning, thereby realizing the reasoning process of the computer for the question answering task.

[0019] Figure 1 is a schematic flowchart of the visual question answering method for cognitive reasoning based on cognitive dual channels in an embodiment of the present invention.

[0020] As Figure 1 shown, the visual question answering method for cognitive reasoning based on cognitive dual channels mainly includes the following steps:

[0021] Step S1, extract question keywords and image target region labels, and construct an index set based on the keywords and the target region labels;

[0022] In this embodiment, for the construction of the index set in step S1, the input question is first subjected to word segmentation to obtain question words and phrases, and then the corresponding image of the input question is input, and the target region of the image is divided, and the target region is classified, so as to construct the index set. Specifically, the steps for constructing the index set are as follows: For the question Q, perform word segmentation to obtain question words W k , phrases P k , where the length of P k is greater than that of W k ; for the image I corresponding to the question, through target detection, the image I is divided into multiple target boxes O k , calculate the classification Tag corresponding to each target k ; use a part-of-speech tagging tool to label and classify the sets W, P, and Tag respectively, delete the stop words therein, and construct a keyword index set K;

[0023] Step S2: Based on the index set, retrieve and construct a knowledge base related to the visual question answering task to obtain cognitive content;

[0024] In this embodiment, in step S2, when constructing the knowledge base related to the visual question answering task, source nodes and target nodes are first constructed based on the relationships between parts of speech and cross-modal symbolic representations, and then an algorithm is used to find all the shortest path sets from the source nodes to the target nodes. Then, based on the commonality of the question-and-answer content, the shortest path sets are merged to construct the knowledge base related to the visual question answering task. The specific steps are as follows: Construct source nodes and target nodes based on the relationships between parts of speech and cross-modal symbolic representations: Based on the commonality of the question-and-answer content, merge Construct the task knowledge base TKG. According to the structure to which the composition index belongs, divide the knowledge base TKG into a word-level knowledge base KG w , a text structure knowledge base KG s , and a cross-modal knowledge base KG c .

[0025] Step S3: Through a cross-modal model pre-trained with multiple modalities, calculate and train a cross-modal representation model for the text modality and the image modality;

[0026] Step S4: By fine-tuning the cross-modal representation model in the visual question answering task, a primary cognitive model is obtained, and various representation vectors are calculated using the cognitive model;

[0027] In this embodiment, the specific steps are as follows: First, use a pre-trained model to calculate a model for cross-modal representation of the text modality and the image modality; then, according to the obtained model fine-tuning task, obtain a primary cognitive model, and finally calculate the cross-modal representation vectors of the content in the primary cognitive stage.

[0028] Step S5: Through syntactic analysis and part-of-speech analysis of the question, construct a topological structure between the question keywords, and update the topological structure using the part-of-speech of the question keywords to obtain the temporal characteristics of question reasoning;

[0029] Step S6: Based on the visual question answering task, calculate the spatial characteristics of the image target region;

[0030] In this embodiment, step S6 includes the following sub-steps: Step S6-1, calculate the linear mapping vector by jointly representing the text modality vector and the image modality vector; Step S6-2, calculate the correlation matrix within the target region through matrix multiplication; Step S6-3, calculate the corresponding polar coordinates (r, θ) according to the rectangular coordinate system of the target region as the physical space relationship of the target region. Specifically, construct the topological structure of the question-and-answer reasoning through the syntactic information of the question sentence, and use the part of speech as the node label content to update the topological structure as the temporal characteristics of the question reasoning. Finally, calculate the spatial characteristics within the image region based on the question-and-answer task. The specific steps for calculating the spatial characteristics within the image region are as follows: Use the sentence feature vector representation S calculated by the primary cognitive model and the set R of feature representations of the image target region; Jointly represent the vectors of the text modality and the image modality, and find their linear mapping vector:

[0031] JoinR k =f([R k |S]),k∈Ω

[0032] where the function f is a linear mapping function, and R k is the feature vector representation of the k-th target region on the input image; Calculate the correlation matrix within the target region through matrix multiplication:

[0033] Relation=JoinR×JoinR T

[0034] According to the rectangular coordinate system of the target box, calculate the corresponding polar coordinates (r, θ) as its physical space relationship:

[0035]

[0036]

[0037] where: x and y are the abscissa and ordinate of the target O k in the rectangular coordinate system within the image;

[0038] Step S7, based on the spatial characteristics of the image target region, obtain the correlation matrix of semantic matching of the image target region, and construct a task association graph structure across modalities;

[0039] In this embodiment, step S7 includes the following sub-steps: Step S7-1, calculating the overall similarity expression of the cross-modal task; Step S7-2, calculating the keyword feature representation vector by using the mean pooling method; Step S7-3, based on the attention-based inquiry mechanism, thereby calculating the correlation matrix; where step S7-3 further includes the following sub-steps: Step A1, splicing the question representation vector and the image representation feature for task representation; Step A2, calculating the correlation matrix including the weight matrix of the question-and-answer task and the image target region and processing the weight matrix by a smoothing method; Step A3, calculating the linear mapping vectors K, Q, V of the keyword and setting the learning parameter W q , W v ; Step A4, calculating the attention score matrix Att of each keyword and the content of the image target region; Step A5, calculating the degree of dispersion of each attention score matrix and the question-and-answer task itself; Step A6, based on the overall similarity Sam SF , selecting TopN(Scorre). Specifically, based on the correlation matrix calculated in the foregoing steps, select TopN(JoinR k ), where N = 8, that is, select the 8 target regions with the highest correlation degree for each target region; The specific steps for constructing the target content locking between cross-modalities are as follows: First, calculate the overall similarity of the cross-modal task that needs to be calculated first in the preliminary cognitive encoding process:

[0040]

[0041] where S is the semantic vector of the question Q, and F is the image I representation vector;

[0042] Secondly, according to the segmented vector EW of the joint encoding, calculate the vector representation of the relevant phrase group by mean pooling:

[0043]

[0044] where EP k is the representation vector of the keyword feature;

[0045] Then, use the attention-based inquiry mechanism to calculate the correlation matrix of semantic matching. The specific steps are as follows: Task joint representation, splicing the sentence representation vector and the image representation feature for task representation:

[0046] T = σ[S,F]

[0047] where S is the question semantic vector, F is the image representation vector, and σ is the activation function;

[0048] Calculate the weight matrix of the task and the image region, and smooth the weight matrix Wtr :

[0049] W tr = g(S, R)

[0050] Where S is the semantic vector of the question, R is the vector representation of the target region within the image, and g calculates the similarity function;

[0051] Calculate the keyword linear mapping vectors K, Q, V, and set the learning parameter W q 、W v :

[0052] K = R r ×W tr

[0053] Q = Key r ×W q

[0054] V = Key r ×W v

[0055] Where K represents the query vector of the image for the keyword, Q represents the query vector of the keyword for the target region of the image, and V represents the mapped value vector of the keyword;

[0056] Calculate the attention score matrix Att for the content of each keyword and the target keyword of the image:

[0057]

[0058] Calculate the degree of dispersion of each attention score Att from the task itself:

[0059]

[0060] Where n is the dimension of the vector;

[0061] Finally, based on the overall similarity Sam of the joint representation SF , select TopN(Scorre), where N = 2 is the number of target regions of the image associated with the keyword;

[0062] Step S8, based on the cognitive content and the various types of representation vectors, re-encode the visual question and answer content to obtain the re-encoded representation content;

[0063] Step S9, based on the re-encoded representation content, the time characteristics of the question reasoning, and the spatial characteristics of the image target region, construct a question and answer instruction set, and perform question and answer reasoning through the question and answer instruction set to obtain the result of the visual question and answer.

[0064] In this embodiment, the specific steps for obtaining the re-encoded representation content are as follows: First, the task knowledge graph KG constructed by encoding w is used as the vector representation EKG w . Then, calculate the similarity between the task representation vector T and the relationships or tail entities in the tuples in the task knowledge base EKG w . The expression is as follows, where n is the vector dimension:

[0065]

[0066] Then, eliminate the tuples irrelevant to the task to obtain the task sub-graph SubKG w of KG w ; Finally, use Key as the index, structurally associate the content nodes of the graph, and use the graph attention network to update EW to EW' and take the average pooling of EW' as the text representation S g of the task-related graph knowledge.

[0067] Based on the spatio-temporal characteristics of the reasoning constructed by the question and answer, the specific steps for constructing the reasoning instruction D are as follows: According to the topological structure and inverse topological structure of the graph, the temporal route TimeRout of the reasoning is constructed; for each node in time, a spatial graph at the construction time point is constructed.

[0068] Based on the time points of the time route of the reasoning, the specific steps for calculating the reasoning instruction D are as follows: Construct the vector combination of the time connection node and the task representation where T k is the vector representation of the time node k; calculate the Gram matrix of ; Determine the reasoning sets D + , D - according to the magnitude and positive / negative of the corresponding Gram values;

[0069] Based on the instruction and the spatio-temporal graph structure constructed in the previous steps, the specific steps for obtaining the question and answer result by reasoning are as follows: First, start the forward reasoning D g at the time point T0 according to the text representation S + 0, and retain the state vector Sta at T0; Secondly, calculate the content of the spatial graph at T0 according to the D + 0 instruction, and update the representation vector h0 at the T0 moment based on the graph attention network. For the T k moment, based on the state quantity Sta before T k-1 and the instruction of D + k , update the representation vector h k . For the T k moment, update the representation vector h - according to the k instruction of D k‘; Finally, after the end of an inference set, calculate the similarity score between the task representation state quantity h and the spatial graph nodes updated at each time point, and take the maximum value as the answer to the visual question and answer.

[0070] Functions and effects of the embodiments

[0071] According to the visual question and answer method of cognitive reasoning based on cognitive dual channels of the present invention, since a novel machine-based hierarchical cognitive question and answer algorithm is proposed, mainly first train through a pre-trained model to enable the model itself to have an understanding of the task, and construct the first intuitive system System1 of the cognitive system. By fine-tuning the task model, it can be adapted to new tasks; then, by splitting the task, obtain the symbolic language inherent in the task, and construct a knowledge base related to the task through retrieval as a supplement to the content of the task to improve the intuition of the cognitive system; finally, construct the cognitive System2, including the content of two stages. In the first stage, re-understand the task. For the intuitive understanding of System1, solve part of the question and answer content. However, for non-intuitive questions, it is necessary to re-define the task representation on the basis of System1 first. Under the conditions of System1, calculate the degree of association with the corpus evidence, calculate the content related to the task, and limit the content representation. In the second stage, concretize the inference content according to the content of the cognitive knowledge base, construct a knowledge graph representing the task, and perform hierarchical relationship reasoning on each cognitive graph according to the constructed space-time order to obtain the final question and answer answer. The present invention greatly improves the accuracy of the question and answer results and continuously corrects the cognitive understanding of the question and answer content according to the external knowledge content during the reasoning process, making the visual question and answer process robust and interpretable.

[0072] The above embodiments are only used to illustrate the specific implementation manners of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A visual question answering method for cognitive reasoning based on a cognitive dual-channel, which is used to solve the visual question answering method in the open domain, dynamically learn the question answering method, and improve the accuracy and interpretability of question answering. It is characterized in that Including the following steps: Step S1: Extract question keywords and image target area labels, and construct an index set based on the keywords and the target area labels; Step S2: Based on the index set, retrieve and construct a knowledge base related to the visual question-answering task, and obtain cognitive content; Step S3: Through a cross-modal model pre-trained in multiple modalities, calculate and train a cross-modal representation model for the text modality and the image modality; Step S4: By fine-tuning the cross-modal representation model in the visual question-answering task, obtain a primary cognitive model, and calculate various representation vectors using the cognitive model; Step S5: By performing syntactic analysis and part-of-speech analysis on the question, construct a topological structure between the question keywords, and update the topological structure using the part-of-speech of the question keywords, so as to obtain the temporal characteristics of question reasoning; Step S6: Based on the visual question-answering task, calculate the spatial characteristics of the image target area; Step S7: Based on the spatial characteristics of the image target area, obtain an association matrix of semantic matching of the image target area, and construct a task association graph structure between cross-modalities; Step S8: Based on the cognitive content and the various representation vectors, re-encode the visual question-answering content to obtain re-encoded representation content; Step S9: Based on the re-encoded representation content, the temporal characteristics of question reasoning, and the spatial characteristics of the image target area, construct a question-answering instruction set, and perform question-answering reasoning through the question-answering instruction set, so as to obtain the result of the visual question-answering.

2. The visual question-answering method based on cognitive reasoning with a cognitive dual-channel according to claim 1, wherein: Among them, In step S1, for the constructed index set, the input question is first tokenized to obtain question words and phrases, then the corresponding image is input for the input question, and the target area of the image is divided, and the target area is classified, so as to construct the index set.

3. The visual question-answering method based on cognitive reasoning with a cognitive dual-channel according to claim 1, wherein: Among them, In step S2, for the construction of the knowledge base related to the visual question-answering task, source nodes and target nodes are first constructed based on the relationship between parts of speech and cross-modal symbolic representations, then the Dijkstra algorithm is used to find all the shortest path sets from the source nodes to the target nodes, and then based on the commonality of the question-answering content, the shortest path sets are merged, so as to construct the knowledge base related to the visual question-answering task.

4. The visual question-answering method based on cognitive reasoning with a cognitive dual-channel according to claim 1, wherein: Among them, The various representation vectors at least include a text modality vector, an image modality vector, and a linear mapping vector; Calculating the spatial characteristics in step S6 includes the following sub-steps: Step S6-1, calculate the linear mapping vector JoinR by jointly representing the text modality vector and the image modality vector, and the formula is as follows: k , as follows: JoinR k = f([R k |S]), k ∈ Ω, where f is a linear mapping function, and R k is the feature representation vector of the K-th target region on the input image, and S represents the sentence feature vector calculated by the primary cognitive model; Step S6-2: Calculate the association matrix Relation within the target area through matrix multiplication, and the formula is as follows: Relation = JoinR × JoinR T ; Step S6-3: According to the rectangular coordinate system of the target area, calculate the corresponding polar coordinates (r, θ) as the physical spatial relationship of the target area; 5. The visual question answering method for cognitive reasoning based on a cognitive dual-channel according to claim 1, It is characterized in that: Wherein, Step S7 further includes the following sub-steps: Step S7-1, calculate the overall similarity Sam of the cross-modal task SF , and the expression is as follows: Wherein, S represents the sentence feature vector calculated by the primary cognitive model, and F is the vector representing image i; Step S7-2, calculating the keyword feature representation vector EP based on the jointly encoded token vectors EW by using the average pooling method k , and the formula is as follows: where ew i is a sub-vector of the word segmentation vector EW; Step S7-3, an attention-based inquiry mechanism is used to calculate the correlation matrix.

6. The visual question answering method based on cognitive dual-channel cognitive reasoning according to claim 5, wherein: Among them, The correlation matrix includes a weight matrix of the question answering task and the image target area, and an attention score matrix of the keyword and the content of the image target area. Step S7-3 further includes: Step A1, by splicing the question representation vector and the image representation feature for task representation: T = σ[S,F] Wherein, σ is an activation function; Step A2, calculate the weight matrix and process the weight matrix W by a smoothing method tr , and the formula is as follows: W tr = g(S, R); Step A3, calculate the linear mapping vectors K, Q, V of the keyword, and set the learning parameter W q and W v : K = R r × W tr Q = Key r × W q V = Key r × W v Wherein, K represents the inquiry vector of the image for the keyword, Q represents the inquiry vector of the keyword for the image target area, and V represents the value vector mapped by the keyword; Step A4, calculate each attention score matrix Att: Step A5, calculate the discreteness of each attention score matrix from the question answering task itself, and the formula is as follows: Where n is the dimension of the vector; Step A6, based on the overall similarity Sam SF , select TopN(Scorre).

7. The visual question answering method based on cognitive dual-channel cognitive reasoning according to claim 1, wherein: Among them, The re-encoding of the visual question answering content in step S8 includes: Calculate the similarity between the question answering task representation vector and the relationship or tail entity in the tuple in the question answering task knowledge base, and the expression is as follows: Where n is the vector dimension.

Images