Visual common sense reasoning method and system based on knowledge-aware attention network

Abstract

The application discloses a visual common sense reasoning method and system based on a knowledge-aware attention network, relates to the cross technical field of computer vision and natural language processing, and comprises the following steps: training a visual common sense reasoning model by using multiple groups of sample images, sample questions and annotated sample answer candidates; the training process comprises the following steps: acquiring visual features and initial context sentence representations of the sample images, the sample questions and the sample answer candidates; learning knowledge-based sentence representations and knowledge-aware attention vectors of the sample questions and the sample answer candidates according to visual context guidance and knowledge embedding; fusing the knowledge-aware attention vectors and the visual features to obtain fused features, so as to train the visual common sense reasoning model; inputting an image to be reasoned, a question to be reasoned and multiple reasoning answer candidates into the trained visual common sense reasoning model, and outputting a reasoning result, so that accurate reasoning is realized, and the performance of a visual common sense reasoning task is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and natural language processing, and particularly relates to a visual common sense reasoning method and system based on a knowledge perception attention network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art that is already known to those skilled in the art.

[0003] In a multimedia world constructed by a large amount of different modal data, the data of different modalities have a high degree of relevance in the process of specific application. Single-modal learning refers to representing information as a numerical vector that can be processed by a computer or further abstracting it into a higher layer feature vector; while multi-modal learning aims to learn better feature representation by using the complementarity between multi-modalities and eliminating the redundancy between modalities through machine learning methods, which requires embedding the data of different modalities into a common representation space for alignment, comparison and fusion.

[0004] In recent years, a large number of researches on multi-modal tasks have emerged, and such tasks can better reflect the understanding ability of computers for various modal data. Among them, a representative task is the visual common sense reasoning (VCR) task, which not only requires selecting the correct answer from the answer candidate according to the image and the question, but also requires selecting a correct reason from the reason candidate to explain the answer. In this task, answering natural language questions not only requires identifying objects, behaviors, etc. in the image, but also requires a large amount of common sense knowledge to help understand the image. However, the existing visual common sense reasoning method does not focus on the background information that plays a key role in understanding the image and the text, and the content hidden outside the context, resulting in poor accuracy and performance of the visual common sense reasoning task. SUMMARY

[0005] To solve the above problems of the prior art, the present application provides a visual common sense reasoning method and system based on a knowledge perception attention network, which adaptively extracts important information in the question and answer candidate according to the visual context guidance, extracts the candidate word group related to the vision, and further integrates the initial features of the natural language sentence using a knowledge embedding method to learn the final knowledge perception attention vector. The fusion feature based on the fusion of the knowledge perception attention vector and the visual feature is used to complete the visual common sense reasoning task, thereby improving the accuracy and performance of the visual common sense reasoning task.

[0006] In a first aspect, the present disclosure provides a visual common sense reasoning method based on a knowledge perception attention network, comprising:

[0007] The visual common sense reasoning model is trained by using a plurality of sets of corresponding sample images, sample questions and annotated sample answer candidates as a sample training set; the training process comprises:

[0008] The visual features of the visual objects in the sample images are obtained, and initial context sentence representations of the sample questions and sample answer candidates are obtained;

[0009] According to the visual context guidance, the important information related to the vision in the sample questions and sample answer candidates is adaptively extracted, the knowledge-based sentence representations of the sample questions and sample answer candidates are learned, and the knowledge perception attention vectors of the sample questions and sample answer candidates are learned through knowledge embedding;

[0010] The obtained knowledge perception attention vectors and visual features are fused to obtain fused features, and the visual common sense reasoning model is trained in this way;

[0011] The to-be-reasoned image, the to-be-reasoned question and a plurality of reasoning answer candidates are input into the trained visual common sense reasoning model, and a reasoning result is output.

[0012] Further technical solutions, the initial context sentence representation of the sample question and the sample answer candidate comprises:

[0013] The pre-trained language representation model is used to extract the word embedding of the natural language text of the sample question and the sample answer candidate;

[0014] The extracted word embedding is input into the bidirectional long short-term memory network to generate the initial context sentence representation of the sample question and the sample answer candidate.

[0015] Further technical solutions, the visual features of the visual objects in the sample images comprise:

[0016] The pre-trained target detection model based on the ResNet101 network is used to perform target detection on the sample image, and the visual features of a plurality of visual objects in the image are extracted.

[0017] Further technical solutions, the visual context guidance is used to adaptively extract the important information related to the vision in the sample questions and sample answer candidates, and the knowledge-based sentence representations of the sample questions and sample answer candidates are learned, comprising:

[0018] Under the guidance of visual information, the pre-trained language model ZEN based on BERT is used to filter the word groups related to the visual features in the sample questions and sample answer candidates through a syntax matching algorithm, so as to construct a word group candidate set;

[0019] The embedding representation of each phrase in the phrase candidate set in the visual feature is learned through context guidance, and the knowledge-based sentence representation of the sample question and the sample answer candidate is learned.

[0020] In a further technical solution, the embedding representation of each candidate phrase in the phrase candidate set in the visual feature is learned by using a pre-trained TransE model.

[0021] On the basis of obtaining the initial context sentence representation of the sample question and the sample answer candidate, the knowledge-based sentence representation of the sample question and the sample answer candidate is learned through context guidance in combination with the learned phrase embedding representation.

[0022] In a further technical solution, the knowledge-aware attention vector of the sample question and the sample answer candidate is learned through knowledge embedding, and the knowledge-aware attention vector includes:

[0023] According to the initial context sentence representation and the knowledge-based sentence representation of the sample question and the sample answer candidate, the attention matrix and the knowledge-based attention matrix of the sample question and the sample answer candidate are calculated respectively.

[0024] The maximum set of rows and columns on the attention matrix and the knowledge-based attention matrix is determined, and the visual context-based attention vector and the knowledge-based attention vector of the sample question and the sample answer candidate are generated respectively.

[0025] The two attention vectors are combined to obtain the initial knowledge-aware attention vector of the sample question and the sample answer candidate.

[0026] The initial knowledge-aware attention vector is dot multiplied with the entire sentence vector to obtain the final knowledge-aware attention vector of the sample question and the sample answer candidate.

[0027] In a further technical solution, the fusion feature is obtained, and the visual common sense reasoning model is trained based on the fusion feature.

[0028] A multilayer perceptron with a piecewise linear function is used as a classifier, the fusion feature is used as input, and the cross-entropy loss between the true label and the predicted label of the sample answer candidate is used to train the visual common sense reasoning model.

[0029] In a second aspect, the present disclosure provides a visual common sense reasoning system based on a knowledge-aware attention network, comprising:

[0030] A visual common sense reasoning model construction and training module is configured to use a plurality of sets of mutually corresponding sample images, sample questions and labeled sample answer candidates as a sample training set to train the constructed visual common sense reasoning model.

[0031] Acquire visual features of visual objects in the sample image, acquire initial context sentence representations of the sample question and sample answer candidate;

[0032] According to the visual context guidance, the important information related to the vision in the sample question and the sample answer candidate is adaptively extracted, the knowledge-based sentence representation of the sample question and the sample answer candidate is learned, and the knowledge perception attention vector of the sample question and the sample answer candidate is learned through knowledge embedding;

[0033] The acquired knowledge perception attention vector is fused with the visual features to acquire fusion features, so as to train the visual common sense reasoning model;

[0034] The reasoning module is used for inputting the to-be-reasoned image, the to-be-reasoned question and the plurality of reasoning answer candidates into the trained visual common sense reasoning model, and outputting a reasoning result.

[0035] In a third aspect, the present disclosure further provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of the first aspect are completed.

[0036] In a fourth aspect, the present disclosure further provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the steps of the method of the first aspect are completed.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] 1. The present application provides a visual common sense reasoning method and system based on knowledge perception attention network, which uses knowledge perception attention network to construct a visual common sense reasoning model, adaptively extracts important information of questions and answer candidates according to visual context, extracts candidate word groups related to vision, and further integrates initial features of natural language sentences of questions and answer candidates using knowledge embedding method, learns final knowledge perception attention vector, obtains fusion features by fusing knowledge perception attention vector and visual features, and realizes visual common sense reasoning based on fusion features, further improves the accuracy and performance of visual common sense reasoning task.

[0039] 2. The visual common sense reasoning method based on knowledge perception attention network can make the computer understand the semantics between images more deeply and effectively improve the performance of the visual common sense reasoning task. BRIEF DESCRIPTION OF DRAWINGS

[0040] The drawings accompanying the specification of the present application form part of the present application and serve to provide a further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application and do not constitute an improper limitation of the present application.

[0041] Figure 1 The flowchart of the visual common sense reasoning method based on the knowledge-aware attention network according to the embodiment one of the present application is shown in Fig. 1.

[0042] Figure 2 The example diagram of the visual common sense reasoning task process according to the embodiment one of the present application is shown in Fig. 2. DETAILED DESCRIPTION

[0043] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0044] It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0045] Embodiment one

[0046] The visual common sense reasoning task is a task in which not only the correct answer is selected from the answer candidates according to the image and the natural language question, but also a reasonable reason is selected according to the answer. Generally, the visual common sense reasoning task includes three sub-tasks, Q→A: given a natural language question Q, select the correct answer A from the answer candidates; QA→R: given a natural language question Q and the answer QA of the question, select a reasonable reason R from the reason candidates; Q→AR: including the two tasks of Q→A and QA→R, the correct answer A and the reasonable reason R are required to be selected at the same time.

[0047] In order to improve the accuracy and performance of the visual common sense reasoning task, the present embodiment provides a visual common sense reasoning method based on a knowledge-aware attention network, a visual common sense reasoning model is constructed, an image to be reasoned and a reasoning question corresponding to the image to be reasoned, a plurality of reasoning answer candidates and a plurality of reasoning reason candidates are obtained, the image to be reasoned, the reasoning question, the plurality of reasoning answer candidates and the plurality of reasoning reason candidates are input into the trained visual common sense reasoning model, and the reasoning result output by the visual common sense reasoning model is that the correct answer is selected from the plurality of reasoning answer candidates according to the reasoning question, and then the reasonable reason is selected from the plurality of reasoning reason candidates according to the selected answer.

[0048] The method according to the embodiment specifically includes the following steps:

[0049] Firstly, a visual common sense reasoning model is trained by using a plurality of groups of sample images, sample questions and annotated sample answer candidates as a sample training set. In this embodiment, each group of sample images, sample questions and sample answer candidates correspond to each other, and the number of sample answer candidates is set to 4. Each sample answer candidate is annotated with an answer label relative to the sample image and the sample question, that is, each sample answer candidate is annotated as a correct answer or an incorrect answer.

[0050] Then, the to-be-reasoned image, the to-be-reasoned question and a plurality of reasoning answer candidates are input into the trained visual common sense reasoning model, and a reasoning result is output.

[0051] The training process of the above visual common sense reasoning model is shown in Figure 1 and includes the following steps:

[0052] Step 1: obtaining visual features of visual objects in a sample image, and obtaining language features of a sample question and a sample answer candidate, that is, an initial context sentence representation.

[0053] First, for the sample image, a pre-trained target detection model based on a ResNet101 network is used to perform target detection on the sample image, and visual features of m objects in the image are extracted, that is, m visual feature vectors v i are extracted. That is, the sample image can be represented by a set composed of visual feature vectors v i (i = 1,...,m), wherein v i is composed of a bounding box b i and a class label l i .

[0054] Secondly, for the sample question and the sample answer candidate, which are both natural language sentences, a pre-trained language representation model (Bidirectional Encoder Representations from Transformer, BERT) is used to extract word embeddings in the natural language sentences, and initial sentence representations are generated respectively. Then, the extracted word embeddings are input into a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, Bi-LSTM), and word features of the natural language sentences are extracted by using the Bi-LSTM to generate initial context sentence representations of the sample question and the sample answer candidate.

[0055] In this embodiment, taking the sample question q and a sample answer candidate a as an example, after the initial sentence representations of the sample question q and the sample answer candidate a are extracted by using BERT respectively, the initial context sentence representations of the sample question q and the sample answer candidate a are generated by using Bi-LSTM. and wherein L and d l is the length and dimension of the sentence. By using the bidirectional long short-term memory network, the feature data obtained at the current moment has information of the previous moment and the next moment, so that information can be captured from the previous context and the text information of the current sentence can be obtained.

[0056] Step 2, under the guidance of the visual context, the important information related to the vision in the sample question and the sample answer candidate is adaptively extracted, and the knowledge-based sentence representation of the sample question and the sample answer candidate is learned. Step 2 specifically includes the following steps:

[0057] Step 2.1, under the guidance of the visual information, the BERT-based pre-training language model ZEN is used to screen the words or word groups related to the visual features in the sample question and the sample answer candidate through a grammar matching algorithm, so as to construct a word group candidate set. In this embodiment, the sample question and the sample answer candidate are input into the BERT-based pre-training language model ZEN, and the words or word groups in the natural language sentence are identified through the grammar matching algorithm, and the visual words or word groups related to the visual features in the sample question and the sample answer candidate are screened out from the visual features. In fact, the visual words or word groups obtained above correspond to the visual objects in the sample image.

[0058] Step 2.2, the embedding representation of each word group in the visual features in the word group candidate set is aggregated, and the knowledge-based sentence representation of the sample question and the sample answer candidate is learned under the guidance of the context.

[0059] Specifically, first, the embedding representation of each candidate word group in the visual features in the word group candidate set is learned by using the pre-trained TransE model, that is, wherein k represents the number of word groups, d p is the dimension of the embedding representation. The TransE model is used to learn the low-dimensional embedding vectors of entities and relationships. In the TransE model, for an input set of triples (h, l, t), the embedding vector of the head entity h is added to the embedding vector of the tail entity t, which is close to the embedding vector of the dependent relationship l. In this embodiment, the visual object corresponding to the candidate word group is regarded as an entity, and the embedding representation of the candidate visual word group in the visual features is learned and optimized by using the TransE model.

[0060] Secondly, on the basis of the initial context sentence representation of the sample question and the sample answer candidate extracted by using the Bi-LSTM in the above step 1, the knowledge-based sentence representation of the sample question and the sample answer candidate is learned under the guidance of the context by combining the learned word group embedding representation. Taking the knowledge-based sentence representation of the sample question as an example, the context guidance is:

[0061] n e = w er1 ph e + w er2 P q

[0062] a e = exp(w er σ(n e ))

[0063]

[0064] where w er , w er1 and w er2 are learnable weight parameters, n e is the context-guided knowledge vector embedding the candidate phrase, and a e is the attention weight applied to the context-guided knowledge vector embedding the candidate phrase. Through the above scheme, a context-guided vector representation is generated for each visual phrase mentioned in the natural language sentence of the sample question, and a knowledge-based sentence representation in the sample question is integrated from the context guidance by capturing local information, i.e. where L is the length of the natural language sentence of the sample question, and d p represents the dimension of the sentence.

[0065] Based on the same method described above, a knowledge-based sentence representation P′ a of the sample answer candidate is obtained.

[0066] Step 3, based on the knowledge-based sentence representation of the sample question and the sample answer candidate, learning the knowledge-aware attention vector of the sample question and the sample answer candidate through knowledge embedding.

[0067] There are two different sentence-level representation vectors for the sample question and the sample answer candidate, i.e. P q and P a are initial context sentence representations, and P′ q and P′ a are knowledge-based sentence representations, i.e. the sentence representations derived from step 2 above. The initial sentence representations and derived sentence representations of the sample question and the sample answer candidate are input into the knowledge embedding layer, and the knowledge embedding method is used to integrate knowledge into natural language, perceive context information and hidden relationships outside the text, and learn the final knowledge-aware attention vector. The step 3 specifically comprises the following steps:

[0068] Step 3.1, according to the initial context sentence representation and the knowledge-based sentence representation of the sample question and the sample answer candidate, respectively, calculate the attention matrix M and the knowledge-based attention matrix M' of the sample question and the sample answer candidate, which is:

[0069]

[0070]

[0071] wherein, and are learnable parameter matrices.

[0072] Step 3.2, determine the maximum collection of rows and columns on the attention matrix M, and generate visual context-based attention vectors for the sample question and the sample answer candidate, respectively;

[0073] Similarly, on the knowledge-based attention matrix M', determine the maximum collection of rows and columns, and generate knowledge-based attention vectors for the sample question and the sample answer candidate, respectively;

[0074] In order to integrate the visual knowledge perception into the attention representation of the sentence, the embodiment combines the above two attention vectors to obtain the initial knowledge perception attention vector A q and A a of the sample question and the sample answer candidate, that is:

[0075]

[0076]

[0077] Step 3.3, point product the initial knowledge perception attention vector with the whole sentence vector, respectively obtain the final knowledge perception attention vector of the sample question and the sample answer candidate, that is:

[0078] S q =[P q :P′ q ] T A q

[0079] S a =[P a :P′ a ] T A a

[0080] Wherein, [:] represents the connection operation.

[0081] Step 4, fuse the obtained knowledge perception attention vector with the visual feature to obtain the fusion feature, and train the visual common sense reasoning model.

[0082] In step 4, the final knowledge-aware attention vector obtained is fused with the visual features to obtain fused features, and a joint representation of the visual and language is learned. The fused features are input into a classifier, and a prediction value is output, that is: where θ represents trainable parameters. Further, in the embodiment, the classifier is a multilayer perceptron with a ReLU activation function.

[0083] Further, in the embodiment, LayerNorm is also used to stabilize training, and a stable prediction value is output That is: where w′ v and w′ r are two linear projection matrices.

[0084] The cross-entropy is used as a loss function, and the cross-entropy loss between the true label and the predicted label of the sample answer candidate is used to train the visual common sense reasoning model. The loss function is:

[0085]

[0086] In the above formula, L loss represents a classification loss, y i represents a true value, represents a prediction value, where i represents the number of sample answer candidates, in the embodiment, the sample answer candidates are set to 4, so i∈{1,2,3,4}, y i represents the true value of the answer label labeled by the i-th sample answer candidate, corresponding to y i , represents the prediction value of the answer label of the i-th sample answer candidate.

[0087] The above visual common sense reasoning model is trained based on the sample image, the sample question, and the sample answer candidate, until the loss function converges, to obtain a trained visual common sense reasoning model.

[0088] Finally, the to-reason image, the to-reason question, and the plurality of reasoning answer candidates are input into the trained visual common sense reasoning model, and a reasoning result is output.

[0089] The above scheme is further explained and verified by the following examples.

[0090] The VCR task includes an image, a question, and an answer candidate. The question and the answer candidate are natural language sentences, and are both sets composed of words and visual words. For a natural language sentence, the Figure 2For example, the question "What is [person3] upset about?" and the answer "[person3] is upset that [person3] can not read music." in the dataset, first replace the name tags in the sentence with the corresponding person names annotated in the dataset, and replace the object tags with the corresponding category names. The natural language sentences of the question and a certain answer candidate are input into the BERT model and encoded into 768-dimensional vectors. Then, each word sequence is input into the Bi-LSTM with a hidden layer and encoded into a 512-dimensional feature vector to generate the initial context sentence representation of the question and the answer candidate.

[0091] The ResNet-101 is used to extract the feature vector of the target in the image, which is encoded into a 2048-dimensional vector. Each target in the image is determined by the Bound bounding box. The RoiAlign method is used to obtain the feature vector of each image target in the last layer of ResNet-101. The fully connected down-sampling network is used to project it into a 512-dimensional embedding vector.

[0092] Then, the pre-trained language model ZEN based on BERT is used to filter the words or phrases related to the visual features in the question and answer candidates by using the syntax matching algorithm, so as to construct a phrase candidate set. For example, for the question sentence "What is [person3] upset about?", two candidate phrases "person3" and "upset about" are matched, which constitute the phrase candidate set. The TransE model is used to learn the embedding representation of each candidate phrase in the visual feature. Through context guidance, the knowledge-based sentence representation of the question and the answer candidate is learned.

[0093] Secondly, the initial context sentence representation and the knowledge-based sentence representation of the question and the answer candidate are input into the knowledge embedding layer to learn the final knowledge-aware attention vector.

[0094] Finally, the knowledge-aware attention vector is fused with the visual feature to obtain the fusion feature. The fusion feature is input into the classifier to calculate the prediction value of the answer candidate as the correct answer output. Based on the above scheme, the correct answer candidate "He is upset that he can not read music." is finally output.

[0095] Based on the above scheme, the reasoning process of QA→R can also be realized, which will not be described here.

[0096] In summary, the visual common sense reasoning method based on the knowledge perception attention network provided in the embodiment uses the knowledge perception attention network to construct a visual common sense reasoning model, extracts important information of a question and an answer candidate according to visual context adaptation, extracts a candidate word group related to vision, and further integrates initial features of natural language sentences of the question and the answer candidate using a knowledge embedding method, learns a final knowledge perception attention vector, obtains a fusion feature by fusing the knowledge perception attention vector and visual features, and implements visual common sense reasoning based on the fusion feature, thereby further improving the accuracy and performance of the visual common sense reasoning task.

[0097] Embodiment Two

[0098] The embodiment provides a visual common sense reasoning system based on a knowledge perception attention network, and the system comprises:

[0099] A visual common sense reasoning model construction and training module is configured to use a plurality of groups of sample images, sample questions and annotated sample answer candidates corresponding to each other as a sample training set to train the constructed visual common sense reasoning model; the training process comprises:

[0100] Visual features of visual objects in the sample images are obtained, and initial context sentence representations of the sample questions and the sample answer candidates are obtained;

[0101] Important information related to vision in the sample questions and the sample answer candidates is adaptively extracted according to visual context guidance, knowledge-based sentence representations of the sample questions and the sample answer candidates are learned, and knowledge perception attention vectors of the sample questions and the sample answer candidates are learned through knowledge embedding;

[0102] The obtained knowledge perception attention vectors and the visual features are fused to obtain fusion features, and the visual common sense reasoning model is trained;

[0103] A reasoning module is configured to input an image to be reasoned, a question to be reasoned and a plurality of reasoning answer candidates into the trained visual common sense reasoning model, and output a reasoning result.

[0104] Embodiment Three

[0105] The embodiment provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps in the visual common sense reasoning method based on the knowledge perception attention network are completed.

[0106] Embodiment Four

[0107] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the visual common sense reasoning method based on the knowledge-aware attention network.

[0108] The steps involved in the above Embodiment 2 to Embodiment 4 correspond to the method of Embodiment 1, and the specific implementation can refer to the relevant description of Embodiment 1. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying the instruction set for execution by a processor and causing the processor to perform any method in the present application.

[0109] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0110] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0111] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not used to limit the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A visual common sense reasoning method based on knowledge-aware attention network, characterized in that, The method comprises the steps of: training a visual common sense reasoning model constructed by using a plurality of sets of mutually corresponding sample images, sample questions and annotated sample answer candidates as a sample training set; the training process comprises the steps of: obtaining visual features of visual objects in the sample images, obtaining initial context sentence representations of the sample questions and sample answer candidates; adaptively extracting important information related to the vision from the sample questions and sample answer candidates based on visual context guidance, learning to obtain knowledge-based sentence representations of the sample questions and sample answer candidates, and then learning knowledge perception attention vectors of the sample questions and sample answer candidates through knowledge embedding; fusing the obtained knowledge perception attention vectors and the visual features to obtain fused features, and training the visual common sense reasoning model based on the fused features; inputting an image to be reasoned, a question to be reasoned and a plurality of reasoning answer candidates into the trained visual common sense reasoning model to output a reasoning result; the step of adaptively extracting important information related to the vision from the sample questions and sample answer candidates based on visual context guidance and learning to obtain knowledge-based sentence representations of the sample questions and sample answer candidates comprises the steps of: under the guidance of visual information, using a pre-trained language model ZEN based on BERT, and screening word groups related to the visual features in the sample questions and sample answer candidates through a syntax matching algorithm to construct a word group candidate set; 2.The knowledge-aware attention network based visual common sense reasoning method of claim 1, wherein, aggregating embedding representations of the word groups in the visual features in the word group candidate set, and learning knowledge-based sentence representations of the sample questions and sample answer candidates based on context guidance; the step of learning knowledge perception attention vectors of the sample questions and sample answer candidates through knowledge embedding comprises the steps of: calculating attention matrices and knowledge-based attention matrices of the sample questions and sample answer candidates based on initial context sentence representations and knowledge-based sentence representations of the sample questions and sample answer candidates, respectively; 3.The knowledge-aware attention network based visual common sense reasoning method of claim 1, wherein, determining the maximum aggregation of rows and columns on the attention matrices and the knowledge-based attention matrices to generate visual context-based attention vectors and knowledge-based attention vectors for the sample questions and sample answer candidates, respectively; merging the two attention vectors to obtain initial knowledge perception attention vectors of the sample questions and sample answer candidates; and 4.The knowledge-aware attention network based visual common sense reasoning method of claim 1, wherein, performing dot product of the initial knowledge perception attention vectors and the entire sentence vectors to obtain final knowledge perception attention vectors of the sample questions and sample answer candidates, respectively. the step of obtaining initial context sentence representations of the sample questions and sample answer candidates comprises the steps of: extracting word embeddings of natural language texts of the sample questions and sample answer candidates by using a pre-trained language representation model; inputting the extracted word embeddings into a bidirectional long short-term memory network to generate initial context sentence representations of the sample questions and sample answer candidates. the step of obtaining visual features of visual objects in the sample images comprises the steps of: using a pre-trained target detection model based on a ResNet101 network to perform target detection on the sample images to extract visual features of a plurality of visual objects in the images. learning embedding representations of candidate word groups in the visual features by using a pre-trained TransE model; and On the basis of obtaining initial context sentence representations of sample question and sample answer candidate, combining learned word group embedding representations, and through context guidance, learning knowledge-based sentence representations of sample question and sample answer candidate.

5. The visual common sense reasoning method based on knowledge-aware attention network according to claim 1, characterized in that, The fusion features are obtained to train the visual common sense reasoning model. A multilayer perceptron with a segmented linear function is used as a classifier, the fusion features are used as inputs, and the cross-entropy loss between the true label and the predicted label of the sample answer candidate is used to train the visual common sense reasoning model.

6. A visual common sense reasoning system based on a knowledge-aware attention network, characterized in that it comprises: A visual common sense reasoning model construction and training module is configured to use a plurality of sets of mutually corresponding sample images, sample questions, and labeled sample answer candidates as a sample training set to train the constructed visual common sense reasoning model. The training process includes: Obtaining visual features of visual objects in sample images, obtaining initial context sentence representations of sample questions and sample answer candidates; According to the visual context guidance, the important information related to the vision in the sample question and the sample answer candidate is adaptively extracted, the knowledge-based sentence representation of the sample question and the sample answer candidate is learned, and the knowledge-aware attention vector of the sample question and the sample answer candidate is learned through knowledge embedding; The obtained knowledge-aware attention vector and visual features are fused to obtain fusion features, which are used to train the visual common sense reasoning model; A reasoning module is configured to input a to-be-reasoned image, a to-be-reasoned question, and a plurality of reasoning answer candidates into the trained visual common sense reasoning model and output a reasoning result. The visual context guidance is used to adaptively extract important information related to the vision in the sample question and the sample answer candidate, and the knowledge-based sentence representation of the sample question and the sample answer candidate is learned, which includes: under the guidance of visual information, a pre-trained language model ZEN based on BERT is used to filter word groups related to visual features in the sample question and the sample answer candidate through a syntax matching algorithm to construct a word group candidate set; the embedding representations of the word groups in the visual features are aggregated, and the knowledge-based sentence representation of the sample question and the sample answer candidate is learned through context guidance. The knowledge embedding is used to learn the knowledge-aware attention vector of the sample question and the sample answer candidate, which includes: according to the initial context sentence representation and the knowledge-based sentence representation of the sample question and the sample answer candidate, the attention matrix and the knowledge-based attention matrix of the sample question and the sample answer candidate are calculated respectively; the maximum aggregation of rows and columns on the attention matrix and the knowledge-based attention matrix is determined to generate the visual context-based attention vector and the knowledge-based attention vector of the sample question and the sample answer candidate respectively; the two attention vectors are combined to obtain the initial knowledge-aware attention vector of the sample question and the sample answer candidate; and the initial knowledge-aware attention vector is dot multiplied with the entire sentence vector to obtain the final knowledge-aware attention vector of the sample question and the sample answer candidate.

7. An electronic device, characterized by comprising: A computer program product, comprising a memory and a processor, and computer instructions stored on the memory and run on the processor, when the computer instructions are run by the processor, complete the steps of the visual common sense reasoning method based on knowledge-aware attention network according to any one of claims 1-5.

8. A computer readable storage medium characterized by, A computer program product for storing computer instructions, when the computer instructions are executed by a processor, complete the steps of the visual common sense reasoning method based on knowledge-aware attention network according to any one of claims 1-5.

Images