Visual question-answering system based on multi-modal knowledge autonomous learning and construction method

Abstract

The invention discloses a visual question-answering system based on multi-modal knowledge autonomous learning and a construction method, and belongs to the technical field of artificial intelligence. The invention provides a visual question and answer model construction method fusing image information, natural language questions and external knowledge. A knowledge dynamic screening and replacement strategy is constructed by introducing an autonomous evolution mechanism, and whether knowledge is inserted or updated can be judged according to a current training state and knowledge similarity, so that the problem that a traditional static knowledge base cannot adapt to task changes is solved. According to the method, through pseudo data construction and multi-modal triple extraction, structured and high-association-degree'image-question-answer 'knowledge representation is established, so that knowledge has better semantic expression ability, and redundancy and noise are effectively reduced. According to the method, rapid retrieval and active calling of knowledge are achieved by means of the vector indexing technology, key support information can still be obtained in a complex scene, and the expression of the model in a visual question-answer task based on knowledge is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a visual question-answering system based on multimodal knowledge autonomous learning and a construction method. Background Art

[0002] Visual question answering is a typical cross-modal understanding task, requiring the model to generate accurate answers based on an input image and a natural language question. With the increasing proportion of knowledge-based questions, it is difficult to answer complex questions relying solely on information from the image and the question itself. Therefore, incorporating external knowledge has become an important means of improving model capabilities.

[0003] Most existing knowledge-enhanced VQA methods use structured knowledge graphs (such as ConceptNet and DBpedia) or unstructured text (such as Wikipedia and image descriptions) as background knowledge, integrating this knowledge into the model through methods such as retrieval or graph neural networks. While these methods have achieved some success in improving the commonsense reasoning capabilities of the models, they still face challenges: Knowledge sources are single and static, unable to dynamically adapt to sample diversity and task changes; knowledge redundancy and noise are severe, impacting reasoning accuracy; and multimodal knowledge acquisition is difficult, preventing image information from being fully incorporated into knowledge modeling. Summary of the Invention

[0004] The following technical problems are addressed: how to automatically construct multimodal knowledge triples with semantic expression capabilities from image-question-answer pairs; how to efficiently store and dynamically filter multimodal knowledge to achieve continuous evolution and optimization of knowledge; and how to effectively integrate retrieved knowledge with image region features and question representation to improve the reasoning accuracy of visual question answering models.

[0005] This paper provides a method for constructing a visual question-answering model that integrates image information, natural language questions, and external knowledge. It introduces an autonomous evolutionary mechanism to enable dynamic screening and continuous learning of multimodal knowledge. The method is primarily used in intelligent systems such as intelligent question-answering, decision support, and human-computer interaction.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A visual question answering system based on autonomous learning of multimodal knowledge, comprising: a multimodal knowledge accumulation framework, an external knowledge base, a multimodal knowledge storage and retrieval framework based on a vector database, and an encoder-decoder framework;

[0008] The multimodal knowledge accumulation framework is used to enhance the knowledge coverage and cross-modal semantic understanding capabilities in visual question answering tasks;

[0009] The multimodal knowledge storage and retrieval framework based on the vector database is used to enable the model to dynamically adjust the stored knowledge representation during continuous interaction and learning, filter out more representative multimodal information, and improve the quality and stability of knowledge retrieval;

[0010] The encoder-decoder framework is used to encode the image description text knowledge and the multimodal knowledge retrieved from the vector library to generate answer predictions that integrate visual perception and knowledge reasoning.

[0011] A method for constructing a visual question answering system based on autonomous learning of multimodal knowledge includes the following steps:

[0012] Step 1: Use the visual question answering model to learn deep representations of image-question pairs, automatically identify key visual entities and their semantic associations through the cross-modal attention mechanism, and construct structured knowledge triples.

[0013] Step 2: Adopt an adaptive update strategy to store the extracted knowledge triples in an extensible vector space, and achieve continuous optimization and autonomous evolution of knowledge through a dynamic knowledge evolution mechanism;

[0014] In step 3, the retrieved multimodal knowledge representation and regional visual features are combined to perform joint semantic encoding based on the encoder-decoder architecture to generate answer predictions that integrate visual perception and knowledge reasoning.

[0015] Furthermore, the step 1 includes the following steps:

[0016] Step 1.1: Automatically generate weakly labeled samples in the form of question-answer pairs through pseudo-data construction to assist in expanding the training corpus.

[0017] Step 1.2: Extract multimodal knowledge and use the constructed pseudo data to generate multimodal triple knowledge to improve the comprehensive ability of the model in semantic fusion.

[0018] Furthermore, in step 1.1, weakly labeled samples in the form of question-answer pairs are automatically generated by constructing pseudo data to assist in expanding the training corpus:

[0019] A strategy for automatically generating pseudo data based on image descriptions to increase knowledge coverage. This strategy uses the Image Caption dataset to assist in building an external knowledge base and participate in the training process.

[0020] Multiple description sentences for each image are spliced ​​together to form a semantically complete context, which serves as the corpus basis for subsequent question and answer generation; then, combining the generation capabilities of the large language model with the prompt mechanism, question-answer pairs that are consistent with the style of the visual question and answer task are automatically generated.

[0021] The prompt mechanism uses explicit template constraints to guide the language model to generate concise questions and short answers that focus on image content. The template constrains the type and scope of the model's generated results by limiting the length of the answer and focusing on visually visible content (such as objects, colors, quantities, etc.), thereby improving the effectiveness and practicality of the question-answer pairs.

[0022] During the question-answer pair extraction phase, a result parsing mechanism based on regular expression matching was designed. Specifically, multiple sets of parsing rules were constructed to perform item-by-item matching and extraction based on the various formats that may appear in the model output (such as "Question: … Answer: …" or "Q: … A: …"). Furthermore, candidate question-answer pairs were screened based on multiple criteria, including length, keywords, and repetition rate. This included, but was not limited to, limiting the answer length to three words or less, avoiding excessive duplication with the question, and preventing invalid information such as function words or placeholders, thereby ensuring the quality and discriminability of the question-answer pairs.

[0023] Furthermore, the step 1.2 of extracting multimodal knowledge and using the constructed pseudo data to generate multimodal triplet knowledge to improve the comprehensive ability of the model in semantic fusion is specifically as follows:

[0024] The bottom-up Fast R-CNN method is used to extract the target area features of the input image, which are divided into visual feature vectors and spatial eigenvectors ; Then the visual feature vector and spatial eigenvectors , and the questions are input into the pre-trained model to obtain visual embedding and question embedding .

[0025] Furthermore, the extraction of knowledge triples includes: head entity extraction, relationship extraction, and tail entity extraction;

[0026] The head entity extraction is the visual area in the image that is most relevant to the question, that is, the visual entity; by calculating the visual area-question correlation matrix To evaluate the relevance of each visual region in the image to the question, it is expressed as: ;in, and is a learnable parameter; T represents transposition; under the action of the visual region-question correlation matrix, a visual region most relevant to the question is selected. Due to the pre-trained model's ability to model the implicit relationship between regions in the image, the selected region most relevant to the question has implicitly integrated its contextual information. This provides key clues for solving complex problems involving multiple regions. Perform row-by-row maximum pooling operations to find the visual area most relevant to the problem. The calculation formula is expressed as: ;according to The most relevant region is selected as the head entity, and then hard attention is used with the help of Gumbel-Softmax to obtain the attention weight of the visual region, which is calculated as follows:

[0027]

[0028] in, are samples drawn from an independent and identically distributed Gumbel (0,1) distribution, is the temperature parameter, which is used to control the smoothness of the Gumbel-Softmax distribution;

[0029] According to the attention distribution of the visual area, all target area features are weighted and aggregated to obtain the head entity representation, which is calculated as follows:

[0030]

[0031] in, ,FFN represents a feed-forward neural network, which contains two fully connected layers;

[0032] The relation extraction is the visual-linguistic connection between the visual region in the image that is most relevant to the question and its corresponding answer. After being encoded by the pre-trained model, the [CLS] vector output by the model is used as the cross-modal representation and input into the feed-forward neural network layer to obtain the final relation embedding, i.e., multimodal implicit knowledge.

[0033] The tail entity extraction is the answer in the image-question-answer, and the predicted answer is obtained through the encoder-decoder;

[0034] Finally, each knowledge unit contains visual entities , semantic relationship , text answers and loss value information , expressed as: .

[0035] Furthermore, the adaptive update strategy in step 2 is specifically: through a multimodal knowledge storage and retrieval framework based on a vector database combined with a dynamic threshold update strategy, the system can autonomously evolve knowledge; this mechanism enables the model to dynamically adjust the stored knowledge representation during continuous interaction and learning, screen out more representative multimodal information, and improve the quality and stability of knowledge retrieval.

[0036] Furthermore, the step 2 adopts an adaptive update strategy to store the extracted knowledge triples in an extensible vector space, and realizes continuous optimization and autonomous evolution of knowledge through a dynamic knowledge evolution mechanism; specifically:

[0037] Introducing a dynamic knowledge screening strategy to ensure autonomous evolution of knowledge, allowing stored knowledge to be continuously optimized during the training process, avoiding redundant data interference, and enhancing the model's ability to adapt to new knowledge;

[0038] For new knowledge units , calculate the similarity of the most similar historical knowledge units ,in is the most similar knowledge in the historical knowledge base, For the existing knowledge base;

[0039] Then, the knowledge insertion threshold T is dynamically adjusted based on the maximum similarity mean th , specifically expressed as:

[0040]

[0041] Among them, J b Indicates the recent J b The cosine similarity set of knowledge units, this paper takes J b = 50. d represents the maximum cosine similarity; Indicates the training loss value of the current knowledge, which is used to feedback the adaptability of the model to the sample; if the distance of the new knowledge If it is smaller than the dynamic threshold Tth-dy, it is inserted into the knowledge base; e represents the base of the natural logarithm, and tanh represents the hyperbolic tangent function.

[0042] Furthermore, the encoder-decoder framework in step 3 generates the answer, specifically: the encoder part encodes the image description text knowledge and the multimodal knowledge retrieved from the vector library, and the decoder generates the answer prediction that integrates visual perception and knowledge reasoning.

[0043] Furthermore, the textual knowledge describing the image is specifically:

[0044] Format the input text as: "question:{Question}context:{Context}",

[0045] Among them, question represents the question, and context is the question-guided image description generated by the PNP-VQA framework;

[0046] The Plug-and-Play VQA (PNP-VQA) framework utilizes saliency map-based interpretability techniques to identify the image region most relevant to the question and generate descriptions from only that region.

[0047] Then use the Transform encoder to encode the knowledge in text format, recorded as ;

[0048]

[0049] in, To describe the text, , represents the dimension of the embedding vector;

[0050] The encoding of the multimodal knowledge retrieved from the vector library is specifically to use an encoder to process the multimodal knowledge embedding retrieved from the vector library, and perform a nearest neighbor search for a given query vector CLS:

[0051] ;

[0052] Where K represents the multimodal knowledge queried, represents the amount of retrieved knowledge;

[0053] Using Transform encoder to process visual entities , visual embedding and multimodal knowledge vector, and then input the fully connected layer to stack the output to form a vector sequence, which is input into the encoder and recorded as ;

[0054]

[0055] in, , is a fully connected layer, Represents a splicing operation;

[0056] The answer prediction of generating the fusion of visual perception and knowledge reasoning is specifically: to obtain the encoding of text knowledge , encoding of multimodal knowledge Splice and input it into the T5 decoder, recorded as ;

[0057] ;in, represents the generated answer;

[0058] The cross entropy loss function is used to train the model.

[0059]

[0060] in, is the length of the true answer text, It's location The real text, are model parameters.

[0061] This method enables the automatic construction of question-answer pairs based on image semantic descriptions. The generated questions are mostly factual and visually decipherable, effectively expanding the OK-VQA training corpus and enhancing the model's knowledge generalization and visual-language alignment capabilities. Typical question types include "color recognition," "human behavior," and "object function," with coverage highly aligned with practical VQA application scenarios. The large-scale question-answer pairs constructed in this way are not only semantically clear but also highly controllable and diverse.

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] Continuous updating and autonomous optimization of knowledge are achieved: This invention introduces an autonomous evolution mechanism to construct a dynamic knowledge screening and replacement strategy, which can determine whether knowledge should be inserted or updated based on the current training status and knowledge similarity, thereby overcoming the problem that traditional static knowledge bases cannot adapt to task changes.

[0064] Improved the representation quality and effectiveness of multimodal knowledge: This invention establishes a structured, highly correlated "image-question-answer" knowledge representation through pseudo-data construction and multimodal triple extraction, which enables knowledge to have better semantic expression capabilities and effectively reduces redundancy and noise.

[0065] Enhanced generalization capabilities of visual question-answering models: This invention leverages vector indexing technology to enable rapid retrieval and active invocation of knowledge, enabling access to key supporting information even in complex scenarios, significantly improving the model's performance in knowledge-based visual question-answering tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0068] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0069] Example

[0070] A method for constructing a visual question answering system based on autonomous learning of multimodal knowledge, such as Figure 1 As shown, the key steps include the following:

[0071] S1. Use the visual question answering model to learn deep representations of image-question pairs, automatically identify key visual entities and their semantic associations through the cross-modal attention mechanism, and construct structured knowledge triples.

[0072] S2. Adopt an adaptive update strategy to store the extracted knowledge triples in an extensible vector space, and realize the continuous optimization and autonomous evolution of knowledge through the dynamic evolution mechanism of knowledge.

[0073] S3. Combining the retrieved multimodal knowledge representation and regional visual features, joint semantic encoding is performed based on the encoder-decoder architecture to generate answer predictions that integrate visual perception and knowledge reasoning.

[0074] S1, described here, utilizes a multimodal knowledge accumulation framework to enhance the knowledge coverage and cross-modal semantic understanding capabilities in visual question answering tasks.

[0075] Step S1.1 first automatically generates weakly labeled samples in the form of question-answer pairs through pseudo-data construction to assist in expanding the training corpus.

[0076] To increase knowledge coverage, we propose a strategy for automatically generating pseudo-data based on image descriptions. Using the Image Caption dataset, we assist in building an external knowledge base and participate in the training process. We first concatenate multiple descriptions of each image to form a semantically complete context, which serves as the corpus for subsequent question-answering. Subsequently, we combine the generative capabilities of a large language model with a prompting mechanism to automatically generate question-answer pairs tailored to the style of the visual question-answering task.

[0077] In terms of prompt design, explicit template constraints are employed to guide the language model in generating concise, image-focused questions and short answers. By limiting answer length and focusing on visually visible content (such as objects, colors, and quantities), the template constrains the type and scope of the model's generated results, improving the effectiveness and practicality of question-answer pairs.

[0078] During the question-answer pair extraction phase, a result parsing mechanism based on regular expression matching was designed. Specifically, multiple sets of parsing rules were constructed to perform item-by-item matching and extraction based on the various formats that may appear in the model output (such as "Question: … Answer: …" or "Q: … A: …"). Furthermore, candidate question-answer pairs were screened based on multiple criteria, including length, keywords, and repetition rate. This included, but was not limited to, limiting the answer length to three words or less, avoiding excessive duplication with the question, and preventing invalid information such as function words or placeholders, thereby ensuring the quality and discriminability of the question-answer pairs.

[0079] Step S1.2, extracting multimodal knowledge. In this stage, the constructed pseudo data is used to generate multimodal triple knowledge to improve the comprehensive ability of the model in semantic fusion.

[0080] In this module, the bottom-up Fast R-CNN method is used to extract the target area features of the input image, which are divided into visual feature vectors and spatial eigenvectors Then the visual feature vector and spatial eigenvectors , and the questions are input into the pre-trained model to obtain visual embedding and question embedding The specific triplet knowledge extraction is as follows:

[0081] (1) Head entity extraction

[0082] The head entity is defined as the visual region in the image that is most relevant to the question, i.e., the visual entity. By calculating the visual region-question correlation matrix to evaluate the relevance of each visual region in the image to the question,

[0083]

[0084] in, and is a learnable parameter and T represents transpose.

[0085] Under the action of the visual region-question correlation matrix, a visual region that is most relevant to the question is selected. Due to the pre-trained model's ability to model the implicit relationship between regions in the image, the selected region that is most relevant to the question has implicitly integrated its context information. This provides key clues for solving complex problems involving multiple regions. Specifically, Perform row-by-row max pooling to find the visual area most relevant to the problem. The calculation method is as follows:

[0086]

[0087] according to The most relevant region is selected as the head entity, and then hard attention is used with the help of Gumbel-Softmax to obtain the attention weight of the visual region, which is calculated as follows:

[0088]

[0089] in, are samples drawn from an independent and identically distributed Gumbel (0,1) distribution, is the temperature parameter, which is used to control the smoothness of the Gumbel-Softmax distribution.

[0090] Finally, all target area features are weighted and aggregated according to the attention distribution of the visual area to obtain the head entity representation, which is calculated as follows:

[0091]

[0092] in, FFN represents a feed-forward neural network, which contains two fully connected layers.

[0093] (2) Relationship Extraction

[0094] The relation is defined as the visual-linguistic connection between the visual region in the image that is most relevant to the question and its corresponding answer. After being encoded by the pre-trained model, the [CLS] vector output by the model is used as the cross-modal representation and input into the feed-forward neural network layer to obtain the final relation embedding, i.e., multimodal implicit knowledge.

[0095] (3) Tail entity extraction: The tail entity is defined as the answer in the image-question-answer sequence, and the predicted answer is obtained through the encoder-decoder.

[0096] Finally, each knowledge unit contains visual entities , semantic relationship , text answers and loss value information , expressed as: .

[0097] Among them, S2, in order to ensure the autonomous evolution of knowledge, introduces a dynamic knowledge screening strategy to continuously optimize the stored knowledge during the training process, avoid redundant data interference, and enhance the model's adaptability to new knowledge. , calculate the similarity of the most similar historical knowledge units ,in is the most similar knowledge in the historical knowledge base, For the existing knowledge base.

[0098] Then the knowledge insertion threshold T is dynamically adjusted based on the maximum similarity mean th , specifically:

[0099]

[0100] Among them, J b Indicates the recent J b The cosine similarity set of knowledge units, take J b = 50. d represents the maximum cosine similarity. Indicates the training loss value of the current knowledge, which is used to feedback the adaptability of the model to the sample. Less than the dynamic threshold T th-dy , then insert it into the knowledge base; e represents the base of the natural logarithm, and tanh represents the hyperbolic tangent function.

[0101] Some candidate knowledge is initially deemed redundant due to its high cosine similarity to existing knowledge. However, to account for performance differences during training, the system allows replacement when new knowledge outperforms old knowledge. This dynamic replacement strategy, based on similarity and training loss, allows the knowledge base to continuously optimize its representational quality, achieving progressive evolution and continuous adaptation.

[0102] In step S3, an encoder-decoder framework is used to generate the answer.

[0103] S3.1, the encoder part needs to encode the image description text knowledge and the multimodal knowledge retrieved in the vector library.

[0104] (1) Textual knowledge

[0105] The input text is formatted as: "question:{Question}context:{Context}", where question represents the question and context is a question-guided image description generated by the Plug-and-Play VQA (PNPVQA) framework. This framework uses interpretability techniques based on saliency maps to identify the image region most relevant to the question and generates descriptions only from that region. The knowledge in the text format is encoded using the Transform encoder, denoted as .

[0106]

[0107] in, To describe the text, , Represents the dimension of the embedding vector.

[0108] (2) Multimodal knowledge

[0109] Use the encoder to process the multimodal knowledge embedding retrieved from the vector library and perform a nearest neighbor search for a given query vector CLS:

[0110]

[0111] Among them, K represents the multimodal knowledge queried, Represents the amount of retrieved knowledge.

[0112] Using Transform encoder to process visual entities , visual embedding and multimodal knowledge vectors, input them into the fully connected layer, stack the outputs to form a vector sequence, and input it into the encoder, recorded as .

[0113]

[0114] in, , is a fully connected layer, Represents a splicing operation.

[0115] (3) Generate answers

[0116] S3.2. Encoding of textual knowledge , encoding of multimodal knowledge Splice and input it into the T5 decoder, recorded as .

[0117]

[0118] in, Represents the generated answer.

[0119] The cross entropy loss function is used to train the model.

[0120]

[0121] in, is the length of the true answer text, It's location The real text, are model parameters.

[0122] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A visual question answering system based on autonomous learning of multimodal knowledge, characterized by: The system includes: a multimodal knowledge accumulation framework, an external knowledge base, a multimodal knowledge storage and retrieval framework based on a vector database, and an encoder-decoder framework; The multimodal knowledge accumulation framework is used to enhance the knowledge coverage and cross-modal semantic understanding capabilities in visual question answering tasks; The multimodal knowledge storage and retrieval framework based on the vector database is used to enable the model to dynamically adjust the stored knowledge representation during continuous interaction and learning, filter out more representative multimodal information, and improve the quality and stability of knowledge retrieval; The encoder-decoder framework is used to encode the image description text knowledge and the multimodal knowledge retrieved from the vector library to generate answer predictions that integrate visual perception and knowledge reasoning.

2. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 1, characterized in that: The following steps are involved: Step 1: Use the visual question answering model to learn deep representations of image-question pairs, automatically identify key visual entities and their semantic associations through a cross-modal attention mechanism, and construct structured knowledge triples; Step 2: Adopt an adaptive update strategy to store the extracted knowledge triples in an extensible vector space, and achieve continuous optimization and autonomous evolution of knowledge through a dynamic knowledge evolution mechanism; In step 3, the retrieved multimodal knowledge representation and regional visual features are combined to perform joint semantic encoding based on the encoder-decoder architecture to generate answer predictions that integrate visual perception and knowledge reasoning.

3. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The step 1 comprises the following steps: Step 1.1: Automatically generate weakly labeled samples in the form of question-answer pairs through pseudo-data construction to assist in expanding the training corpus. Step 1.2: Extract multimodal knowledge and use the constructed pseudo data to generate multimodal triple knowledge to improve the comprehensive ability of the model in semantic fusion.

4. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: In step 1.1, weakly labeled samples in the form of question-answer pairs are automatically generated by constructing pseudo data to assist in expanding the training corpus. Specifically, A strategy for automatically generating pseudo data based on image descriptions to increase knowledge coverage. This strategy uses the Image Caption dataset to assist in building an external knowledge base and participate in the training process. Multiple descriptions of each image are concatenated to form a semantically complete context, which serves as the corpus for subsequent question-answer generation. Then, by combining the generative capabilities of the large language model with the prompt mechanism, question-answer pairs that are consistent with the style of the visual question answering task are automatically generated; The prompt mechanism uses an explicit template constraint method to guide the language model to generate concise questions and short answers that focus on the image content.

5. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The step 1.2, extracting multimodal knowledge and using the constructed pseudo data to generate multimodal triplet knowledge to improve the comprehensive ability of the model in semantic fusion, is specifically as follows: The bottom-up Fast R-CNN method is used to extract the target area features of the input image, which are divided into visual feature vectors and spatial eigenvectors ; Then the visual feature vector and spatial eigenvectors , and the questions are input into the pre-trained model to obtain visual embedding and question embedding .

6. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The extraction of knowledge triples includes: head entity extraction, relationship extraction and tail entity extraction; The head entity extraction is the visual area in the image that is most relevant to the question, that is, the visual entity; by calculating the visual area-question correlation matrix To evaluate the relevance of each visual region in the image to the question, it is expressed as: ;in, and is a learnable parameter; T represents transpose; Visual area-question correlation matrix Perform row-by-row maximum pooling operations to find the visual area most relevant to the problem. The calculation formula is expressed as: ;according to The most relevant region is selected as the head entity, and then hard attention is used with the help of Gumbel-Softmax to obtain the attention weight of the visual region, which is calculated as follows: ； in, are samples drawn from an independent and identically distributed Gumbel (0,1) distribution, is the temperature parameter, which is used to control the smoothness of the Gumbel-Softmax distribution; According to the attention distribution of the visual area, all target area features are weighted and aggregated to obtain the head entity representation, which is calculated as follows: ； in, ,FFN represents a feed-forward neural network, which contains two fully connected layers; The relationship extraction is the visual-linguistic connection between the visual area in the image that is most relevant to the question and its corresponding answer. After being encoded by the pre-trained model, the semantic relationship output by the model is converted to The vector is used as a cross-modal representation and fed into a feed-forward neural network layer to obtain the final relation embedding, i.e., multimodal implicit knowledge. The tail entity extraction is the answer in the image-question-answer, and the predicted answer is obtained through the encoder-decoder; Finally, each knowledge unit contains visual entities , semantic relationship , text answers and loss value information , expressed as: .

7. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The adaptive update strategy in step 2 is specifically: through a multimodal knowledge storage and retrieval framework based on a vector database combined with a dynamic threshold update strategy, the system performs autonomous evolution of knowledge.

8. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The step 2 adopts an adaptive update strategy to store the extracted knowledge triples in an extensible vector space, and realizes continuous optimization and autonomous evolution of knowledge through a dynamic knowledge evolution mechanism; Specifically: Introducing a dynamic knowledge screening strategy to ensure autonomous evolution of knowledge, allowing stored knowledge to be continuously optimized during the training process, avoiding redundant data interference, and enhancing the model's ability to adapt to new knowledge; For new knowledge units , calculate the similarity of the most similar historical knowledge units ,in is the most similar knowledge in the historical knowledge base, For the existing knowledge base; Then, the knowledge insertion threshold T is dynamically adjusted based on the maximum similarity mean th , specifically expressed as: ； Among them, J b Indicates the recent J b The cosine similarity set of knowledge units, d represents the maximum cosine similarity; Indicates the training loss value of the current knowledge, which is used to feedback the adaptability of the model to the sample; if the distance of the new knowledge If it is less than the dynamic threshold Tth-dy, it will be inserted into the knowledge base; e represents the base of natural logarithm, and tanh represents the hyperbolic tangent function.

9. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 2, characterized in that: The encoder-decoder framework in step 3 generates the answer, specifically: the encoder part encodes the image description text knowledge and the multimodal knowledge retrieved from the vector library, and the decoder generates the answer prediction that integrates visual perception and knowledge reasoning.

10. The method for constructing a visual question answering system based on multimodal knowledge autonomous learning according to claim 9, characterized in that: The textual knowledge describing the image is specifically: Format the input text as: "question:{Question}context:{Context}", Among them, question represents the question, and context is the question-guided image description generated by the PNP-VQA framework; Then use the Transform encoder to encode the knowledge in text format, recorded as ; ； in, To describe the text, , represents the dimension of the embedding vector; The encoding of the multimodal knowledge retrieved from the vector library is specifically to use an encoder to process the multimodal knowledge embedding retrieved from the vector library, and perform a nearest neighbor search for a given query vector CLS: ； Where K represents the multimodal knowledge queried, represents the amount of retrieved knowledge; Using Transform encoder to process visual entities , visual embedding and multimodal knowledge vector, and then input the fully connected layer to stack the output to form a vector sequence, which is input into the encoder and recorded as ; ； in, , is a fully connected layer, Represents a splicing operation; The answer prediction of generating the fusion of visual perception and knowledge reasoning is specifically: to obtain the encoding of text knowledge , encoding of multimodal knowledge Splice and input it into the decoder, recorded as ; ;in, represents the generated answer; The cross entropy loss function is used to train the model. ； in, is the length of the true answer text, It's location The real text, are model parameters.

Images