Visual question answering method and platform based on knowledge enhancement

By acquiring and splicing external knowledge in the visual question-and-answer model and training with internal knowledge, the problem of not being able to effectively utilize external and internal knowledge in the existing technology is solved, and the accuracy and reliability of visual question-and-answer are improved.

CN117972044BActive Publication Date: 2024-11-26INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311868499.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-29
Publication Date
2024-11-26
Estimated Expiration
2043-12-29

AI Technical Summary

Technical Problem

The existing pre-trained model knowledge enhancement methods cannot reasonably utilize the retrieved external knowledge and the internal knowledge implicitly of the model itself, affecting the accuracy of visual question and answer.

Method used

By obtaining the user-entered images and question text, after processing, inputting them to the searcher of the pre-built visual question and answer model, obtaining relevant external knowledge, and splicing it with the input text, inputting it to the answer generator, combining training samples and samples Tags jointly train the initial searcher and answer generator, making rational use of external and internal knowledge.

Benefits of technology

It improves the accuracy of visual question-and-answer, avoids the influence of noise caused by excessive reliance on external knowledge, and enhances the reliability of answers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117972044B_ABST
    Figure CN117972044B_ABST
Patent Text Reader

Abstract

The present invention provides a visual question answering method and platform based on knowledge enhancement, belonging to the field of artificial intelligence technology, wherein the method comprises: obtaining an image and a question text input by a user to obtain an input text; inputting the input text into a retriever of a visual question answering model to obtain a plurality of related external knowledge; splicing the plurality of related external knowledge with the input text respectively to obtain a plurality of spliced ​​input texts, inputting the plurality of spliced ​​input texts into an answer generator of a visual question answering model to obtain an answer text corresponding to each spliced ​​input text, inputting the input text into an answer generator to obtain an answer text corresponding to the input text, and determining a final target answer text corresponding to the input text; wherein the visual question answering model is obtained by jointly training an initial answer generator and an initial retriever of an initial visual question answering model. The present invention can reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the visual question answering model itself.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual question answering method and platform based on knowledge enhancement. Background Art

[0002] With the rapid update and development of Internet technology, visual question answering can provide users with image-oriented question answering services. This service allows users to input images and ask questions about the images to get answers. Visual question answering services can help users provide multimodal interactive experiences, so they have received widespread attention from researchers. In recent years, more and more work has focused on how to combine retrieval methods with pre-trained models. At present, traditional pre-trained model knowledge enhancement methods cannot reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the model itself, which affects the accuracy of visual question answering. Summary of the invention

[0003] The present invention provides a visual question answering method and platform based on knowledge enhancement, which is used to solve the defect that the pre-training model knowledge enhancement method in the prior art cannot reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the model itself, thus affecting the accuracy of visual question answering.

[0004] In a first aspect, the present invention provides a visual question answering method based on knowledge enhancement, which is applied to a visual question answering platform based on knowledge enhancement, comprising:

[0005] Acquire an image and a question text input by a user, and process the image and the question text to obtain an input text;

[0006] Inputting the input text into a pre-built visual question answering model retriever to obtain a plurality of relevant external knowledge corresponding to the input text output by the retriever;

[0007] splicing the multiple related external knowledge with the input text respectively to obtain multiple spliced ​​input texts, inputting the multiple spliced ​​input texts into the answer generator of the visual question answering model to obtain the answer text corresponding to each spliced ​​input text output by the answer generator, inputting the input text into the answer generator to obtain the answer text corresponding to the input text output by the answer generator;

[0008] Determine the final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text;

[0009] The visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model using sample input text as training samples and answer text labels corresponding to the sample input text as sample labels.

[0010] In some embodiments, the retriever includes a knowledge encoding layer, a query encoding layer, and a matching layer;

[0011] Correspondingly, the input text is input into a retriever of a pre-built visual question answering model, and a plurality of relevant external knowledge corresponding to the input text output by the retriever is obtained, including:

[0012] Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0013] Inputting the input text into the query encoding layer to obtain a feature vector of the input text output by the query encoding layer;

[0014] The feature vector of the input text and the feature vector of each piece of knowledge are input into the matching layer, and a plurality of relevant external knowledge corresponding to the input text is obtained as output by the matching layer.

[0015] In some embodiments, the process of determining the external knowledge base includes:

[0016] Obtain an open source visual question answering dataset, and concatenate the question text and the answer text of each data sample in the dataset to obtain the query text corresponding to each data sample;

[0017] Input each query text into a search engine to obtain multiple query results corresponding to each query text;

[0018] Calculating the relevance of each query result to each query text, filtering the multiple query results to obtain filtered query results;

[0019] Performing text extraction on the screened query results to obtain a plurality of knowledge text fragments, and processing the plurality of knowledge text fragments to obtain processed knowledge text fragments;

[0020] Based on the processed knowledge text fragments, the external knowledge base is obtained.

[0021] In some embodiments, determining the final target answer text corresponding to the input text from the answer text corresponding to each concatenated input text and the answer text corresponding to the input text includes:

[0022] Based on the visual question answering model, calculating the uncertainty degree of the answer text corresponding to each concatenated input text and the answer text corresponding to the input text;

[0023] According to the uncertainty degree of each of the answer texts, a final target answer text corresponding to the input text is determined from each of the answer texts.

[0024] In some embodiments, the processing of the image and the question text to obtain the input text includes:

[0025] Processing the image to obtain an image context corresponding to the image, wherein the image context includes image description text, recognition text, object information, and attribute information;

[0026] The image context and the question text are concatenated to obtain the input text.

[0027] In some embodiments, the process of determining the visual question answering model includes:

[0028] Acquire a sample image and a sample question text, process the sample image and the sample question text to obtain a sample input text, and determine an answer text label corresponding to the sample input text;

[0029] Inputting the sample input text into the initial retriever of the initial visual question answering model, and obtaining a plurality of sample-related external knowledge corresponding to the sample input text output by the initial retriever;

[0030] Splicing the multiple sample-related external knowledge with the sample input text respectively to obtain multiple spliced ​​sample input texts, inputting the multiple spliced ​​sample input texts into the initial answer generator of the initial visual question-answering model, obtaining the answer text prediction result corresponding to each spliced ​​sample input text output by the initial answer generator, inputting the sample input text into the initial answer generator, and obtaining the answer text prediction result corresponding to the sample input text output by the initial answer generator;

[0031] Determine the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each of the spliced ​​sample input texts and the answer text prediction results corresponding to the sample input text;

[0032] Calculate the optimization objective function value based on the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text, and the answer text label corresponding to the sample input text;

[0033] Based on the optimization objective function value, training the initial visual question answering model, and performing parameter optimization iterations on the initial visual question answering model to obtain the visual question answering model;

[0034] The process of determining the retriever includes:

[0035] Based on the answer text label corresponding to the sample input text, comparing the answer text prediction results corresponding to the multiple spliced ​​sample input texts with the answer text prediction result corresponding to the sample input text to obtain a comparison result;

[0036] Based on the comparison result, a supervised training signal is obtained to train the initial retriever, and after the initial retriever training is completed, the retriever is obtained.

[0037] In some embodiments, the initial retriever includes a knowledge encoding layer, an initial query encoding layer, and an initial matching layer;

[0038] Correspondingly, the inputting the sample input text into the initial retriever of the initial visual question answering model to obtain a plurality of sample-related external knowledge corresponding to the sample input text output by the initial retriever includes:

[0039] Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0040] Inputting the sample input text into the initial query coding layer to obtain a feature vector of the sample input text output by the initial query coding layer;

[0041] The feature vector of the sample input text and the feature vector of each piece of knowledge are input into the initial matching layer, and a plurality of sample-related external knowledge corresponding to the sample input text output by the initial matching layer is obtained.

[0042] In some embodiments, determining the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each of the spliced ​​sample input texts and the answer text prediction results corresponding to the sample input text includes:

[0043] Based on the initial visual question answering model, calculating the uncertainty degree of the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text;

[0044] According to the uncertainty degree of each answer text prediction result, the final target answer text corresponding to the sample input text is determined from each answer text prediction result.

[0045] In some embodiments, the calculation formula of the optimization objective function value is:

[0046]

[0047] Where P represents the optimization objective function value, x represents the sample input text, A represents the answer text label set corresponding to the sample input text, θ represents the parameters of the initial answer generator, φ represents the parameters of the initial retriever, s j represents the external knowledge related to the jth sample, represents the answer text label corresponding to the jth sample-related external knowledge in the answer text label set, R P represents the set of sample-related external knowledge corresponding to the positive supervision training signal, R N Represents the set of sample-related external knowledge corresponding to the negative supervision training signal.

[0048] In a second aspect, an embodiment of the present invention further provides a visual question answering platform based on knowledge enhancement, including:

[0049] An acquisition module, used for acquiring an image and a question text input by a user, and processing the image and the question text to obtain an input text;

[0050] A retrieval module, used for inputting the input text into a pre-built visual question answering model retriever, and obtaining a plurality of relevant external knowledge corresponding to the input text output by the retriever;

[0051] An answer generation module is used to concatenate the multiple related external knowledge with the input text respectively to obtain multiple concatenated input texts, input the multiple concatenated input texts into the answer generator of the visual question answering model to obtain an answer text corresponding to each concatenated input text output by the answer generator, input the input text into the answer generator to obtain an answer text corresponding to the input text output by the answer generator;

[0052] A determination module, used to determine a final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text;

[0053] The visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model using sample input text as training samples and answer text labels corresponding to the sample input text as sample labels.

[0054] The present invention provides a visual question answering method and platform based on knowledge enhancement, which obtains an image and a question text input by a user, processes the image and the question text to obtain an input text, inputs the input text into a retriever of a pre-built visual question answering model to obtain a plurality of relevant external knowledge, splices the plurality of relevant external knowledge with the input text respectively to obtain a plurality of spliced ​​input texts, inputs the plurality of spliced ​​input texts and the input text respectively into an answer generator of the visual question answering model to obtain a plurality of answer texts, and determines a target answer text, wherein the visual question answering model is obtained by jointly training an initial retriever and an initial answer generator based on training samples and sample labels, and the present invention can reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the visual question answering model itself, thereby improving the accuracy of the visual question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a flowchart of a visual question answering method based on knowledge enhancement provided by an embodiment of the present invention;

[0057] Figure 2 is a flowchart of a process for determining a visual question answering model provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a flow chart of a method for training a visual question answering model provided by an embodiment of the present invention;

[0059] Figure 4 Schematic diagram of a knowledge-enhanced visual question answering model provided by an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of the structure of a visual question answering platform based on knowledge enhancement provided by an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The terms "first", "second", etc. in the description of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0064] In recent years, pre-training technology has developed rapidly, and visual question-answering services based on pre-trained models can generate higher-quality answers. In addition, in order to enhance the implicit knowledge contained in the pre-trained model, some researchers have designed knowledge-based pre-training tasks to better incorporate common sense and factual knowledge into the parameters of the model. Although these methods can enhance the model's learning of knowledge entities, when it is necessary to deal with new knowledge, the model must be retrained. Recently, relying on self-supervised learning on massive data, large-scale pre-trained models such as the third-generation generative pre-trained (Generative Pre-trained-3Transformer, GPT-3) model have been proven to learn massive knowledge into their parameters. These models can achieve good results in knowledge-based question answering and are applied to visual question answering tasks. However, in practice, large-scale pre-trained models have the problem of hallucination output, that is, they produce prediction results that seem reasonable but are actually incorrect, and related studies have found that this problem cannot be solved simply by increasing the model size. In addition, this method of relying on parameter memory knowledge cannot cope with changes and additions in knowledge.

[0065] Visual question answering in practical applications is usually open-ended. Some questions are difficult to answer from the information in the image alone and require the help of relevant external knowledge. Therefore, model enhancement methods based on knowledge retrieval have been studied and used in visual question answering models, that is, using the retriever to obtain and input relevant external knowledge fragments to help the model make predictions. Since it is impossible to directly obtain the supervisory signal of the retrieved knowledge, the current related knowledge retrieval-based methods usually use weak supervisory signals to train the retriever, that is, if the retrieved knowledge contains the target answer, a positive supervisory signal is given, and these methods require that the output must rely on the retrieved external knowledge, even if the retrieved external knowledge is wrong. Therefore, this method greatly hinders its application in practice. In practical applications, if the retrieved relevant knowledge does not play a positive role in correct prediction, then their addition will interfere with the model's prediction, and some questions can be answered correctly using the implicit knowledge inside the model.

[0066] To this end, an embodiment of the present invention provides a visual question answering method and platform based on knowledge enhancement, which obtains an image and a question text input by a user, processes the image and the question text, obtains an input text, inputs the input text into a pre-built visual question answering model retriever, obtains multiple related external knowledge, splices the multiple related external knowledge with the input text, obtains multiple spliced ​​input texts, inputs the multiple spliced ​​input texts and the input text into the answer generator of the visual question answering model, obtains multiple answer texts, and determines the target answer text, wherein the visual question answering model is obtained by jointly training the initial retriever and the initial answer generator based on training samples and sample labels. The present invention can reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the visual question answering model itself, thereby improving the accuracy of visual question answering.

[0067] Figure 1 FIG. 1 is a flow chart of a knowledge-enhanced visual question answering method provided in an embodiment of the present invention. Figure 1 As shown, a knowledge-enhanced visual question answering method is provided, which is applied to a knowledge-enhanced visual question answering platform, and includes the following steps: step 110, step 120, step 130 and step 140. The steps of the method flow are only a possible implementation of the present invention.

[0068] Step 110: Acquire the image and question text input by the user, process the image and question text, and obtain the input text.

[0069] The image and the question text are associated with each other, for example, the image shows a train and the question text is "Who invented this first?"

[0070] Optionally, the image uploaded by the user may be directly received, or a URL link of the image provided by the user may be obtained, and the corresponding image may be obtained according to the URL link of the image.

[0071] Optionally, the question text input by the user may be directly received, or voice data input by the user may be received, and the voice data may be recognized to obtain the question text.

[0072] Optionally, the image and question text data input by the user can be collected through a terminal device, and the terminal device can be a mobile phone, a personal computer, a tablet computer and other devices.

[0073] Optionally, computer vision technology can be used to extract features from the image, and natural language processing technology can be used to process the question text.

[0074] It should be noted that for visual question answering tasks, by processing images and question texts, the images and question texts input by the user can be converted into input text that can be understood by the machine.

[0075] In some embodiments, step 110 processes the image and the question text to obtain the input text, including:

[0076] Step 111, processing the image to obtain an image context corresponding to the image, the image context including image description text, recognition text, object information and attribute information;

[0077] Step 112: Concatenate the image context and the question text to obtain the input text.

[0078] Optionally, the image may be input into an image description model to obtain an image description text corresponding to the image output by the image description model.

[0079] Optionally, the image may be input into a target recognition model to obtain object information and attribute information corresponding to the image output by the target recognition model.

[0080] Optionally, the image may be input into an optical character recognition (OCR) model to obtain a recognized text corresponding to the image output by the OCR model.

[0081] For example, the image context can be represented as c i =(caption i ,object i ,ocr i ), i represents the i-th user input, caption i Indicates the image description text corresponding to the i-th image, object iRepresents the object information and attribute information corresponding to the i-th image, OCR i Represents the recognized text corresponding to the i-th image.

[0082] For example, for the i-th user input x i , its corresponding image context c i and question text q i After splicing, we get the input text (c i ,q i ).

[0083] Step 120: input the input text to the retriever of the pre-built visual question answering model, and obtain a plurality of relevant external knowledge corresponding to the input text output by the retriever.

[0084] It can be understood that inputting the input text into the retriever of a pre-built visual question answering model and obtaining the relevant external knowledge output by the retriever can provide the visual question answering model with more contextual information and background knowledge, thereby helping the visual question answering model to understand and answer questions more accurately.

[0085] In some embodiments, the retriever includes a knowledge encoding layer, a query encoding layer, and a matching layer;

[0086] Correspondingly, step 120 inputs the input text into the retriever of the pre-built visual question answering model, and obtains a plurality of relevant external knowledge corresponding to the input text output by the retriever, including:

[0087] Step 121, input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0088] Step 122, input the input text to the query coding layer, and obtain the feature vector of the input text output by the query coding layer;

[0089] Step 123: Input the feature vector of the input text and the feature vector of each piece of knowledge into the matching layer, and obtain a plurality of related external knowledge corresponding to the input text output by the matching layer.

[0090] It should be noted that the knowledge encoding layer remains fixed, and all the knowledge in the external knowledge base can be encoded in advance at one time to obtain the feature vector of each knowledge, which is stored in the open source indexing engine and indexed, thereby reducing the computing cost; taking the input text as a query, the input text can be input into the query encoding layer in real time, and the query encoding layer can dynamically generate the feature vector of the input text (that is, the feature vector of the query).

[0091] Optionally, the semantic similarity between the feature vector of the input text and the feature vector of each piece of knowledge may be calculated based on the matching layer by means of inner product, and multiple relevant external knowledge corresponding to the input text may be determined according to the semantic similarity.

[0092] For example, the corresponding multiple external knowledge may be sorted in descending order of semantic similarity, and the top k-1 relevant external knowledge may be selected.

[0093] In some embodiments, the process of determining the external knowledge base includes:

[0094] Obtain an open source visual question answering dataset, concatenate the question text and answer text of each data sample in the dataset, and obtain the query text corresponding to each data sample;

[0095] Input each query text into the search engine to obtain multiple query results corresponding to each query text;

[0096] Calculate the relevance of each query result and each query text, filter multiple query results, and obtain filtered query results;

[0097] Performing text extraction on the filtered query results to obtain multiple knowledge text fragments, and processing the multiple knowledge text fragments to obtain processed knowledge text fragments;

[0098] Based on the processed knowledge text fragments, an external knowledge base is obtained.

[0099] Optionally, the relevances corresponding to different query results may be sorted in descending order, and query results whose relevance ranking exceeds a preset threshold may be screened out.

[0100] Optionally, the filtered query results may be the top 10 most relevant query results.

[0101] Optionally, multiple knowledge text fragments may be screened, incomplete and repeated knowledge text fragments may be deleted to obtain screened knowledge text fragments, and knowledge identifiers may be assigned to the screened knowledge text fragments to obtain processed knowledge text fragments.

[0102] Optionally, the processed knowledge text fragment is stored in an external knowledge base.

[0103] Step 130: splice the multiple related external knowledge with the input text respectively to obtain multiple spliced ​​input texts, input the multiple spliced ​​input texts into the answer generator of the visual question answering model, obtain the answer text corresponding to each spliced ​​input text output by the answer generator, input the input text into the answer generator, and obtain the answer text corresponding to the input text output by the answer generator;

[0104] It can be understood that by concatenating multiple relevant external knowledge with the input text to obtain multiple concatenated input texts, and inputting the multiple concatenated input texts and the input text into the answer generator of the visual question answering model respectively, the visual question answering model can generate multiple candidate answer texts.

[0105] Step 140: Determine the final target answer text corresponding to the input text from the answer text corresponding to each concatenated input text and the answer text corresponding to the input text.

[0106] Among them, the visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model with the sample input text as the training sample and the answer text label corresponding to the sample input text as the sample label.

[0107] In some embodiments, determining the final target answer text corresponding to the input text from the answer text corresponding to each concatenated input text and the answer text corresponding to the input text includes:

[0108] Based on the visual question answering model, the uncertainty of the answer text corresponding to each concatenated input text and the answer text corresponding to the input text are calculated;

[0109] According to the uncertainty degree of each answer text, the final target answer text corresponding to the input text is determined from each answer text.

[0110] Optionally, the answer text with the smallest degree of uncertainty is selected from each answer text as the final target answer text.

[0111] For example, k-1 related external knowledge are concatenated with the input text to obtain k-1 concatenated input texts, the first answer text is obtained based on the first concatenated input text, the second answer text is obtained based on the second concatenated input text, ..., the k-1th answer text is obtained based on the k-1th concatenated input text, and the kth answer text is obtained based on the implicit internal knowledge of the visual question and answer model; the uncertainty degree of the k prediction results is calculated, and the answer text with the smallest uncertainty is selected as the final target answer text.

[0112] In an embodiment of the present invention, an image and a question text input by a user are obtained, the image and the question text are processed to obtain an input text, the input text is input into a retriever of a pre-built visual question answering model to obtain a plurality of relevant external knowledge, the plurality of relevant external knowledge are respectively spliced ​​with the input text to obtain a plurality of spliced ​​input texts, the plurality of spliced ​​input texts and the input text are respectively input into an answer generator of the visual question answering model to obtain a plurality of answer texts, and a target answer text is determined, wherein the visual question answering model is obtained by jointly training an initial retriever and an initial answer generator based on training samples and sample labels, and the present invention can reasonably utilize the retrieved external knowledge and the implicit internal knowledge of the visual question answering model itself, thereby improving the accuracy of the visual question answering.

[0113] Figure 2 FIG. 1 is a flow chart of the process of determining the visual question answering model provided by an embodiment of the present invention. Figure 2 As shown, the determination process of the visual question answering model includes the following steps: step 210, step 220, step 230, step 240, step 250 and step 260.

[0114] Step 210: Obtain a sample image and a sample question text, process the sample image and the sample question text to obtain a sample input text, and determine an answer text label corresponding to the sample input text.

[0115] Optionally, the sample image is processed to obtain a sample image context corresponding to the sample image, where the sample image context includes a sample image description text, a sample identification text, sample object information, and sample attribute information.

[0116] Optionally, the sample image context and the sample question text are concatenated to obtain a sample input text.

[0117] Step 220: input the sample input text into the initial retriever of the initial visual question answering model, and obtain a plurality of sample-related external knowledge corresponding to the sample input text output by the initial retriever.

[0118] In some embodiments, the initial retriever includes a knowledge encoding layer, an initial query encoding layer, and an initial matching layer;

[0119] Correspondingly, the sample input text is input into the initial retriever of the initial visual question answering model, and multiple sample-related external knowledge corresponding to the sample input text output by the initial retriever is obtained, including:

[0120] Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0121] Input the sample input text into the initial query encoding layer to obtain a feature vector of the sample input text output by the initial query encoding layer;

[0122] The feature vector of the sample input text and the feature vector of each knowledge are input into the initial matching layer, and a plurality of sample-related external knowledge corresponding to the sample input text output by the initial matching layer is obtained.

[0123] Optionally, the process of determining the external knowledge base includes:

[0124] Obtain an open source visual question answering dataset, concatenate the question text and answer text of each data sample in the dataset, and obtain the query text corresponding to each data sample;

[0125] Input each query text into the search engine to obtain multiple query results corresponding to each query text;

[0126] Calculate the relevance of each query result and each query text, filter multiple query results, and obtain filtered query results;

[0127] Performing text extraction on the filtered query results to obtain multiple knowledge text fragments, and processing the multiple knowledge text fragments to obtain processed knowledge text fragments;

[0128] Based on the processed knowledge text fragments, an external knowledge base is obtained.

[0129] It should be noted that the knowledge encoding layer remains fixed, and all knowledge in the external knowledge base can be encoded in advance at one time, and only the initial query encoder can be trained, thereby reducing the training cost.

[0130] Step 230: Concatenate multiple sample-related external knowledge with the sample input text respectively to obtain multiple concatenated sample input texts, input the multiple concatenated sample input texts into the initial answer generator of the initial visual question-answering model, obtain the answer text prediction result corresponding to each concatenated sample input text output by the initial answer generator, input the sample input text into the initial answer generator, and obtain the answer text prediction result corresponding to the sample input text output by the initial answer generator.

[0131] Step 240: Determine the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each concatenated sample input text and the answer text prediction results corresponding to the sample input text.

[0132] In some embodiments, determining the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each concatenated sample input text and the answer text prediction results corresponding to the sample input text includes:

[0133] Based on the initial visual question answering model, calculate the uncertainty of the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text;

[0134] According to the uncertainty degree of each answer text prediction result, the final target answer text corresponding to the sample input text is determined from each answer text prediction result.

[0135] Step 250: Calculate the optimization objective function value based on the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text, as well as the answer text label corresponding to the sample input text.

[0136] Step 260: Based on the optimization objective function value, train the initial visual question answering model, perform parameter optimization iterations on the initial visual question answering model, and obtain the visual question answering model.

[0137] The process of determining the retriever includes:

[0138] Based on the answer text label corresponding to the sample input text, the answer text prediction results corresponding to the multiple concatenated sample input texts are compared with the answer text prediction result corresponding to the sample input text to obtain a comparison result;

[0139] Based on the comparison result, a supervised training signal is obtained to train the initial retriever. After the initial retriever training is completed, the retriever is obtained.

[0140] It should be noted that, given the training data set D = {v i ,q i ,a i}, where each sample contains a sample image v i ∈V, sample question text q i ∈Q, answer text label a i ∈A. The model needs to be trained to learn the mapping function:

[0141]

[0142] It should be noted that in practical applications, some questions can be answered directly using the implicit knowledge contained in the pre-trained model parameters, while some questions require additional help from retrieving relevant explicit knowledge from external knowledge sources. Therefore, the optimization goal of the visual question answering model can be expressed as follows:

[0143] argmaxp(a i |v i ,q i ;θ,K)

[0144] Among them, θ represents the implicit knowledge contained in the visual question answering model, K represents the set of external knowledge related to the sample, and K = {s j}, where s j Represents the external knowledge related to the jth sample, j∈[1,k-1]. The visual question answering model needs to be able to actively select more appropriate knowledge for prediction.

[0145] In some embodiments, the calculation formula for the optimization objective function value is:

[0146]

[0147] Among them, P represents the optimization objective function value, x represents the sample input text, A represents the answer text label set corresponding to the sample input text, θ represents the parameters of the initial answer generator, φ represents the parameters of the initial retriever, and s j represents the external knowledge related to the jth sample, represents the case text label corresponding to the external knowledge related to the jth sample, R P represents the set of sample-related external knowledge corresponding to the positive supervision training signal, R N Represents the set of sample-related external knowledge corresponding to the negative supervision training signal.

[0148] It should be noted that in this optimization objective function, the first item is to improve the answer generation quality of the answer generator, the second item is to encourage the retriever to obtain correct knowledge, and the third item is to prevent the retriever from obtaining incorrect knowledge.

[0149] Optionally, in the training data set, each question may include 10 manually given answer text labels, represented by the answer text label set A. In order to obtain better training results, the training target corresponding to each concatenated input text is determined separately.

[0150] Optionally, for the concatenated sample input text obtained by concatenating the j-th sample-related external knowledge with the sample input text, select an answer that appears in the j-th sample-related external knowledge from the answer label set A as the answer generation target for the input; if the j-th sample-related external knowledge does not contain any answer in A, the most frequent answer label in A is used as the answer generation target corresponding to the concatenated sample input text.

[0151] It should be noted that the first prediction result is obtained based on the concatenated sample input text corresponding to the external knowledge related to the first sample, the second prediction result is obtained based on the concatenated sample input text corresponding to the external knowledge related to the second sample, and so on, the k-1th prediction result is obtained based on the concatenated sample input text corresponding to the external knowledge related to the k-1th sample, and the kth prediction result is obtained based on the implicit internal knowledge of the visual question answering model.

[0152] It should be noted that valuable external knowledge should be able to help the model generate the correct answer when it cannot generate the correct answer. In an embodiment of the present invention, the kth prediction output can be used to determine which knowledge is truly valuable for the model prediction, because the kth prediction output does not use external knowledge to help generate the answer. For example, when the prediction result obtained using a certain knowledge is correct but the kth prediction obtained using the implicit knowledge of the model is wrong, then the knowledge can be determined to be valuable and a positive supervisory signal can be assigned to it; conversely, when the prediction result obtained using a certain external knowledge is wrong but the kth prediction obtained using the implicit knowledge of the model is correct, then the external knowledge can be determined to be noisy and a negative supervisory signal can be assigned to it. The training signal set of the constructed initial retriever is as follows:

[0153]

[0154]

[0155]

[0156] Among them, H(s j ,A)=1 indicates sample-related external knowledge s j Contains an answer from the manually annotated answer set, H(s j ,A)=0 means sample-related external knowledge s j Does not contain any answer from the manually annotated answer set; It means that the correct answer can be generated with the help of external knowledge related to the jth sample. It means that only using the implicit internal knowledge of the visual question answering model cannot generate the correct answer; R I Indicates that it does not belong to R P Nor does it belong to R N The knowledge set R I The knowledge in the training does not calculate the loss.

[0157] It should be noted that the first k-1 sample-related external knowledge is concatenated with the sample input text to generate k-1 prediction results {y i1 ,y i2 ,,...,y i(k-1)}, based on the internal knowledge, the kth prediction result is obtained, and the final prediction result is determined based on the uncertainty of each prediction result. The goal of the retriever training is to reduce R N The score of knowledge in the collection improves R P The fraction of knowledge in the set.

[0158] Optionally, the loss function value corresponding to each prediction result can be calculated. The loss function value corresponding to each prediction result represents the degree of uncertainty of the visual question answering model about the prediction result. The larger the loss function value, the greater the uncertainty. The prediction result with the smallest loss function value is selected as the final prediction result.

[0159] Figure 3 Schematic diagram of the flow of the training method of the visual question answering model provided by the embodiment of the present invention. Figure 3 As shown, the training method of the visual question answering model includes the following steps:

[0160] S301, start;

[0161] S302, building a knowledge base;

[0162] Optionally, external knowledge can be obtained and a knowledge base can be constructed based on open source visual question answering datasets and search engines.

[0163] S303, input a sample image and a sample question text;

[0164] Optionally, before inputting the sample image and the sample question text, training data is obtained, the training data including the sample image and the sample question text, and the corresponding answer text label.

[0165] S304, retrieving k-1 external knowledge in the knowledge base based on the sample input text;

[0166] The sample input text includes a sample image and a sample question text.

[0167] S305: The visual question answering model makes predictions based on each piece of external knowledge and obtains the top k-1 prediction results.

[0168] S306: The visual question answering model makes predictions without using external knowledge, and obtains the kth prediction result.

[0169] It should be noted that the kth prediction result is obtained by the visual question answering model based on the implicit internal knowledge of the model.

[0170] S307, determining a final prediction result from the k prediction results based on uncertainty;

[0171] S308, comparing the prediction result based on the external knowledge with the kth prediction result to determine the value of the external knowledge;

[0172] S309, jointly training the visual question answering model based on the prediction result loss and the external knowledge loss;

[0173] S310, using the gradient back propagation method to iterate the parameters of the visual question answering model;

[0174] S311, judging whether the model training has converged;

[0175] If the training converges, then execute step 312 ; if the training does not converge, then continue to return to execute step S303 .

[0176] S312, end.

[0177] Figure 4 The schematic diagram of the knowledge-enhanced visual question answering model provided by the embodiment of the present invention. Figure 4 As shown, the visual question answering model includes a retriever and an answer generator, and the retriever includes a knowledge encoder and a query encoder; an image and a question text input by a user are obtained, and an image context is obtained based on the image. The image context includes an image description, an object in the image, and a text in the image, and the question text and the image context are spliced ​​to obtain the input text; the input text is input to the retriever, and the retriever encodes the input text through the query encoder to obtain a feature vector of the input text, and all knowledge in the external knowledge base is pre-encoded through the knowledge encoder to obtain a feature vector of each knowledge, and the correlation between the feature vector of the input text and the feature vector of each knowledge is calculated according to the correlation calculation function to obtain the most relevant k-1 external knowledge; multiple retrieved external knowledge are spliced ​​with the input text to obtain multiple spliced ​​input texts, and the multiple spliced ​​input texts and the input text are respectively input to the answer generator, and k-1 prediction results are obtained based on the k-1 spliced ​​input texts, and the kth prediction result is obtained based on the internal knowledge, and the final prediction result is determined based on the uncertainty degree of each prediction result.

[0178] It should be noted that the training mechanism of the visual question answering model is to jointly train the retriever and the answer generator to obtain the correctness of k prediction results, determine the training signal of each external knowledge, and improve the quality of the external knowledge retrieved by the retriever, thereby improving the accuracy of the visual question answering model prediction.

[0179] In an embodiment of the present invention, by judging whether external knowledge can truly provide prediction performance gain as a supervisory signal for model training, the initial retriever and the initial answer generator of the initial visual question answering model are jointly trained, so that the retriever can be trained based on a strong supervisory signal, thereby improving the quality of external knowledge retrieved by the retriever, enabling the visual question answering model to reasonably utilize external knowledge and internal knowledge, improving the reliability of answer prediction, and avoiding the visual question answering model from over-relying on external knowledge during the prediction process and being affected by external noise knowledge.

[0180] The following describes a visual question answering platform based on knowledge enhancement provided by an embodiment of the present invention. The visual question answering platform based on knowledge enhancement described below and the visual question answering method based on knowledge enhancement described above can refer to each other.

[0181] Figure 5 A schematic diagram of the structure of a knowledge-enhanced visual question-answering platform provided in an embodiment of the present invention is shown in FIG. Figure 5 As shown, the knowledge-enhanced visual question answering platform 500 includes:

[0182] The acquisition module 510 is used to acquire the image and question text input by the user, and process the image and question text to obtain the input text;

[0183] A retrieval module 520 is used to input the input text into a pre-built visual question answering model retriever, and obtain a plurality of relevant external knowledge corresponding to the input text output by the retriever;

[0184] The answer generation module 530 is used to concatenate multiple related external knowledge with the input text respectively to obtain multiple concatenated input texts, input the multiple concatenated input texts into the answer generator of the visual question answering model, obtain the answer text corresponding to each concatenated input text output by the answer generator, input the input text into the answer generator, and obtain the answer text corresponding to the input text output by the answer generator;

[0185] A determination module 540, for determining a final target answer text corresponding to the input text from the answer text corresponding to each concatenated input text and the answer text corresponding to the input text;

[0186] Among them, the visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model with the sample input text as the training sample and the answer text label corresponding to the sample input text as the sample label.

[0187] Optionally, the retriever includes a knowledge encoding layer, a query encoding layer and a matching layer;

[0188] Correspondingly, the input text is input into the retriever of the pre-built visual question answering model, and multiple relevant external knowledge corresponding to the input text output by the retriever is obtained, including:

[0189] Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0190] Input the input text to the query encoding layer, and obtain the feature vector of the input text output by the query encoding layer;

[0191] The feature vector of the input text and the feature vector of each piece of knowledge are input into the matching layer, and the matching layer outputs a plurality of related external knowledge corresponding to the input text.

[0192] Optionally, the process of determining the external knowledge base includes:

[0193] Obtain an open source visual question answering dataset, concatenate the question text and answer text of each data sample in the dataset, and obtain the query text corresponding to each data sample;

[0194] Input each query text into the search engine to obtain multiple query results corresponding to each query text;

[0195] Calculate the relevance of each query result and each query text, filter multiple query results, and obtain filtered query results;

[0196] Performing text extraction on the filtered query results to obtain multiple knowledge text fragments, and processing the multiple knowledge text fragments to obtain processed knowledge text fragments;

[0197] Based on the processed knowledge text fragments, an external knowledge base is obtained.

[0198] Optionally, determining a final target answer text corresponding to the input text from the answer text corresponding to each concatenated input text and the answer text corresponding to the input text comprises:

[0199] Based on the visual question answering model, the uncertainty of the answer text corresponding to each concatenated input text and the answer text corresponding to the input text are calculated;

[0200] According to the uncertainty degree of each answer text, the final target answer text corresponding to the input text is determined from each answer text.

[0201] Optionally, the image and the question text are processed to obtain input text, including:

[0202] Process the image to obtain the image context corresponding to the image, which includes image description text, recognition text, object information and attribute information;

[0203] Concatenate the image context and question text to get the input text.

[0204] Optionally, the process of determining the visual question answering model includes:

[0205] Obtaining a sample image and a sample question text, processing the sample image and the sample question text to obtain a sample input text, and determining an answer text label corresponding to the sample input text;

[0206] Input the sample input text into the initial retriever of the initial visual question answering model, and obtain multiple sample-related external knowledge corresponding to the sample input text output by the initial retriever;

[0207] Splicing multiple sample-related external knowledge with sample input texts respectively to obtain multiple spliced ​​sample input texts, inputting the multiple spliced ​​sample input texts into the initial answer generator of the initial visual question answering model, obtaining the answer text prediction result corresponding to each spliced ​​sample input text output by the initial answer generator, inputting the sample input text into the initial answer generator, and obtaining the answer text prediction result corresponding to the sample input text output by the initial answer generator;

[0208] Determine the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each concatenated sample input text and the answer text prediction results corresponding to the sample input text;

[0209] Calculate the optimization objective function value based on the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text, as well as the answer text label corresponding to the sample input text;

[0210] Based on the optimization objective function value, the initial visual question answering model is trained, and the parameters of the initial visual question answering model are iterated to obtain the visual question answering model;

[0211] The process of determining the retriever includes:

[0212] Based on the answer text label corresponding to the sample input text, the answer text prediction results corresponding to the multiple concatenated sample input texts are compared with the answer text prediction result corresponding to the sample input text to obtain a comparison result;

[0213] Based on the comparison result, a supervised training signal is obtained to train the initial retriever. After the initial retriever training is completed, the retriever is obtained.

[0214] Optionally, the initial retriever includes a knowledge encoding layer, an initial query encoding layer, and an initial matching layer;

[0215] Correspondingly, the sample input text is input into the initial retriever of the initial visual question answering model, and multiple sample-related external knowledge corresponding to the sample input text output by the initial retriever is obtained, including:

[0216] Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer;

[0217] Input the sample input text into the initial query encoding layer to obtain a feature vector of the sample input text output by the initial query encoding layer;

[0218] The feature vector of the sample input text and the feature vector of each knowledge are input into the initial matching layer, and a plurality of sample-related external knowledge corresponding to the sample input text output by the initial matching layer is obtained.

[0219] Optionally, determining the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each concatenated sample input text and the answer text prediction results corresponding to the sample input text includes:

[0220] Based on the initial visual question answering model, calculate the uncertainty of the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text;

[0221] According to the uncertainty degree of each answer text prediction result, the final target answer text corresponding to the sample input text is determined from each answer text prediction result.

[0222] Optionally, the calculation formula for the optimization objective function value is:

[0223]

[0224] Among them, P represents the optimization objective function value, x represents the sample input text, A represents the answer text label set corresponding to the sample input text, θ represents the parameters of the initial answer generator, φ represents the parameters of the initial retriever, and s j represents the external knowledge related to the jth sample, represents the answer text label corresponding to the jth sample related external knowledge in the answer text label set, R P represents the set of sample-related external knowledge corresponding to the positive supervision training signal, R N Represents the set of sample-related external knowledge corresponding to the negative supervision training signal.

[0225] Optionally, the knowledge-enhanced visual question-answering platform 500 further includes an interactive interface, which is used to obtain an image and question text input by a user, display a target answer text, or display the target answer text and related external knowledge.

[0226] It should be noted here that the knowledge-enhanced visual question-answering platform provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned knowledge-enhanced visual question-answering method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0227] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 820, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a visual question answering method based on knowledge enhancement, which includes: obtaining an image and a question text input by a user, processing the image and the question text to obtain an input text; inputting the input text into a retriever of a pre-built visual question answering model to obtain a plurality of relevant external knowledge corresponding to the input text output by the retriever; splicing the plurality of relevant external knowledge with the input text respectively to obtain a plurality of spliced ​​input texts, inputting the plurality of spliced ​​input texts into an answer generator of the visual question answering model to obtain an answer text corresponding to each spliced ​​input text output by the answer generator, inputting the input text into the answer generator to obtain an answer text corresponding to the input text output by the answer generator; determining the final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text; wherein the visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model using the sample input text as the training sample and the answer text label corresponding to the sample input text as the sample label.

[0228] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0229] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual question answering method based on knowledge enhancement, characterized in that: Applied to knowledge-enhanced visual question answering platforms, including: Acquire an image and a question text input by a user, and process the image and the question text to obtain an input text; Inputting the input text into a pre-built visual question answering model retriever to obtain a plurality of relevant external knowledge corresponding to the input text output by the retriever; splicing the multiple related external knowledge with the input text respectively to obtain multiple spliced ​​input texts, inputting the multiple spliced ​​input texts into the answer generator of the visual question answering model to obtain the answer text corresponding to each spliced ​​input text output by the answer generator, inputting the input text into the answer generator to obtain the answer text corresponding to the input text output by the answer generator; Determine the final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text; The visual question answering model is obtained by jointly training an initial answer generator and an initial retriever of the initial visual question answering model using sample input text as training samples and answer text labels corresponding to the sample input text as sample labels; The process of determining the visual question answering model includes: Acquire a sample image and a sample question text, process the sample image and the sample question text to obtain a sample input text, and determine an answer text label corresponding to the sample input text; Inputting the sample input text into the initial retriever of the initial visual question answering model, and obtaining a plurality of sample-related external knowledge corresponding to the sample input text output by the initial retriever; Splicing the multiple sample-related external knowledge with the sample input text respectively to obtain multiple spliced ​​sample input texts, inputting the multiple spliced ​​sample input texts into the initial answer generator of the initial visual question-answering model, obtaining the answer text prediction result corresponding to each spliced ​​sample input text output by the initial answer generator, inputting the sample input text into the initial answer generator, and obtaining the answer text prediction result corresponding to the sample input text output by the initial answer generator; Determine the final target answer text corresponding to the sample input text from the answer text prediction results corresponding to each of the spliced ​​sample input texts and the answer text prediction results corresponding to the sample input text; Calculate the optimization objective function value based on the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text, and the answer text label corresponding to the sample input text; Based on the optimization objective function value, training the initial visual question answering model, and performing parameter optimization iterations on the initial visual question answering model to obtain the visual question answering model; The process of determining the retriever includes: Based on the answer text label corresponding to the sample input text, comparing the answer text prediction results corresponding to the multiple spliced ​​sample input texts with the answer text prediction result corresponding to the sample input text to obtain a comparison result; Based on the comparison result, a supervised training signal is obtained to train the initial retriever, and after the initial retriever training is completed, the retriever is obtained.

2. The visual question answering method based on knowledge enhancement according to claim 1, characterized in that: The retriever includes a knowledge encoding layer, a query encoding layer and a matching layer; Correspondingly, the input text is input into a retriever of a pre-built visual question answering model, and a plurality of relevant external knowledge corresponding to the input text output by the retriever is obtained, including: Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer; Inputting the input text into the query encoding layer to obtain a feature vector of the input text output by the query encoding layer; The feature vector of the input text and the feature vector of each piece of knowledge are input into the matching layer, and a plurality of relevant external knowledge corresponding to the input text is obtained as output by the matching layer.

3. The visual question answering method based on knowledge enhancement according to claim 2, characterized in that: The process of determining the external knowledge base includes: Obtain an open source visual question answering dataset, and concatenate the question text and the answer text of each data sample in the dataset to obtain the query text corresponding to each data sample; Input each query text into a search engine to obtain multiple query results corresponding to each query text; Calculating the relevance of each query result to each query text, filtering the multiple query results to obtain filtered query results; Performing text extraction on the screened query results to obtain a plurality of knowledge text fragments, and processing the plurality of knowledge text fragments to obtain processed knowledge text fragments; Based on the processed knowledge text fragments, the external knowledge base is obtained.

4. The knowledge-enhanced visual question answering method according to claim 1, characterized in that: The determining the final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text comprises: Based on the visual question answering model, calculating the uncertainty degree of the answer text corresponding to each concatenated input text and the answer text corresponding to the input text; According to the uncertainty degree of each answer text, a final target answer text corresponding to the input text is determined from each answer text.

5. The knowledge-enhanced visual question answering method according to any one of claims 1 to 4, characterized in that: The step of processing the image and the question text to obtain the input text includes: Processing the image to obtain an image context corresponding to the image, wherein the image context includes image description text, recognition text, object information, and attribute information; The image context and the question text are concatenated to obtain the input text.

6. The knowledge-enhanced visual question answering method according to claim 1, characterized in that: The initial retriever includes a knowledge encoding layer, an initial query encoding layer and an initial matching layer; Correspondingly, the inputting the sample input text into the initial retriever of the initial visual question answering model to obtain a plurality of sample-related external knowledge corresponding to the sample input text output by the initial retriever includes: Input each piece of knowledge in the pre-built external knowledge base into the knowledge encoding layer in advance, and obtain the feature vector of each piece of knowledge output by the knowledge encoding layer; Inputting the sample input text into the initial query coding layer to obtain a feature vector of the sample input text output by the initial query coding layer; The feature vector of the sample input text and the feature vector of each piece of knowledge are input into the initial matching layer, and a plurality of sample-related external knowledge corresponding to the sample input text output by the initial matching layer is obtained.

7. The knowledge-enhanced visual question answering method according to claim 1, characterized in that: The determining the final target answer text corresponding to the sample input text from the answer text prediction result corresponding to each of the spliced ​​sample input texts and the answer text prediction result corresponding to the sample input text comprises: Based on the initial visual question answering model, calculating the uncertainty degree of the answer text prediction result corresponding to each concatenated sample input text and the answer text prediction result corresponding to the sample input text; According to the uncertainty degree of each answer text prediction result, the final target answer text corresponding to the sample input text is determined from each answer text prediction result.

8. The knowledge-enhanced visual question answering method according to claim 1, characterized in that: The calculation formula of the optimization objective function value is: Where P represents the optimization objective function value, x represents the sample input text, A represents the answer text label set corresponding to the sample input text, θ represents the parameters of the initial answer generator, φ represents the parameters of the initial retriever, s j represents the external knowledge related to the jth sample, represents the answer text label corresponding to the jth sample-related external knowledge in the answer text label set, R P represents the set of sample-related external knowledge corresponding to the positive supervision training signal, R N Represents the set of sample-related external knowledge corresponding to the negative supervision training signal.

9. A visual question answering platform based on knowledge enhancement, characterized in that: The platform is used to execute the knowledge-enhanced visual question answering method according to any one of claims 1 to 8, and the platform includes: An acquisition module, used for acquiring an image and a question text input by a user, and processing the image and the question text to obtain an input text; A retrieval module, used for inputting the input text into a pre-built visual question answering model retriever, and obtaining a plurality of relevant external knowledge corresponding to the input text output by the retriever; An answer generation module is used to concatenate the multiple related external knowledge with the input text respectively to obtain multiple concatenated input texts, input the multiple concatenated input texts into the answer generator of the visual question answering model to obtain an answer text corresponding to each concatenated input text output by the answer generator, input the input text into the answer generator to obtain an answer text corresponding to the input text output by the answer generator; A determination module, used to determine a final target answer text corresponding to the input text from the answer text corresponding to each spliced ​​input text and the answer text corresponding to the input text; The visual question answering model is obtained by jointly training the initial answer generator and the initial retriever of the initial visual question answering model using sample input text as training samples and answer text labels corresponding to the sample input text as sample labels.