Multi-modal question and answer method based on large language model and knowledge graph

Abstract

The invention discloses a multi-modal question and answer method based on a large language model and a knowledge graph, and belongs to the technical field of artificial intelligence and the knowledge graph. Aiming at the problems of insufficient multi-modal information processing, insufficient semantic understanding depth, limited cross-modal reasoning ability and the like in the existing question-answering system, the invention provides a multi-modal question-answering engine, a multi-modal knowledge graph is constructed for multi-modal data, a semantic association network in modals and between the modals is established, and the multi-modal question-answering engine is used for solving the problems of multi-modal information processing, semantic understanding depth, limited cross-modal reasoning ability and the like. And indexes are generated for elements such as entities and relationships of the knowledge graph so as to support subsequent efficient retrieval. The knowledge graph is subjected to structured information retrieval, the powerful natural language understanding ability of a large language model and the ability of a specific modal question and answer model are combined, the efficiency and accuracy of information retrieval are improved, the cross-modal reasoning ability is enhanced, and therefore efficient and accurate question and answer in the complex multi-modal environment is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of knowledge graphs, large language models, and data retrieval, and in particular to a multimodal question-answering method based on a large language model and a knowledge graph. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems have become one of the important tools for solving complex problems and providing accurate information. Traditional question-answering systems usually rely on single-modal data input, mainly based on natural language text. However, in the real world, information often appears in multiple forms. For example, diagnosis in the medical field depends not only on the doctor's description and the patient's medical records, but also on X-rays, MRI images, etc. In addition, scientific papers, financial statements, product specifications, etc. often contain rich tabular information, and the dynamically changing data and images contained in videos can also provide a more intuitive understanding. Traditional question-answering systems can only process data from a single modality and cannot effectively integrate multimodal information. With the further development of technologies such as deep learning and graph neural networks (GNNs), multimodal question-answering systems have emerged, which improve the breadth and depth of question-answering systems by processing and integrating data from different modalities (such as text, images, videos, tables, etc.). Some existing multimodal question-answering methods are divided into two strategies: one is to convert other modalities into textual modalities and use existing textual question-answering systems to complete reasoning and answers; the other is to process each modality independently, conduct questions and answers separately, and then integrate the answers of each modality. Regardless of modality conversion or separate processing, current multimodal question-answering systems face the following three main challenges in practical applications:

[0003] 1. Preprocessing of multimodal information. Existing question-answering systems are mostly based on a single modality (mainly text), and are unable to fully utilize the potential of non-text data such as images and tables, resulting in the system being unable to provide comprehensive and accurate answers. In multimodal scenarios, how to effectively integrate different types of modal data (such as images, videos, text, tables, audio, etc.), and how to achieve unified processing while maintaining their respective advantages while maintaining their respective advantages is an urgent problem to be solved.

[0004] 2. Weak cross-modal reasoning capabilities. The cross-modal reasoning capabilities of current question-answering systems are still insufficient, and they are unable to effectively establish information associations and reason between different modalities. This lack of capability makes it difficult for the system to be comprehensive and accurate when answering questions involving multimodal data. With the popularization of multimodal data, cross-modal reasoning capabilities will become increasingly important. Only by achieving efficient reasoning between different modalities can the system perform well on complex problems.

[0005] 3. Efficiency of large-scale data processing. Multimodal question-answering systems usually need to process a large amount of data sources such as text, images, and videos. This requires the system to be able to quickly and efficiently retrieve, process, and generate answers when faced with massive amounts of information. How to optimize the processing efficiency of large-scale data, especially in the conversion, fusion, and reasoning process between modalities, is still a difficult problem that needs to be solved.

[0006] Therefore, the present invention proposes a multimodal question-answering method based on a large language model (LLM) combined with a knowledge graph. This method constructs a multimodal knowledge graph and establishes an index, uses a large language model for natural language understanding, and combines a specific modality question-answering model for deep reasoning, significantly improving the ability of the multimodal question-answering system in handling complex problems, cross-modal reasoning, and efficient retrieval. This multimodal question-answering engine can not only process multimodal data, but also provide more accurate and comprehensive answers through efficient semantic retrieval and reasoning. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and provide a multimodal question answering method based on a large language model and a knowledge graph, which solves the problems existing in the existing multimodal question answering system, such as insufficient multimodal information processing, limited cross-modal reasoning capability, and large-scale data processing efficiency.

[0008] The object of the present invention is achieved through the following technical solution: a multimodal question answering method based on a large language model and a knowledge graph, comprising:

[0009] Acquire multimodal data to build a multimodal knowledge graph;

[0010] Decomposing a natural language query into a plurality of subtasks, determining dependencies between the subtasks, and generating a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks;

[0011] Search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes the modality type, entity name, entity description and the data source ID corresponding to the entity;

[0012] The large language model generates the first answer based on the retrieved entities and relations;

[0013] Calling a corresponding modal question-answering model according to the modal type in the retrieved entity, wherein the modal question-answering model analyzes source data corresponding to the entity to generate a second answer;

[0014] Generate a final answer based on the first answer and the second answer.

[0015] Furthermore, the multimodal data includes one or more of text data, image data, audio data, video data and table data.

[0016] Furthermore, the step of acquiring multimodal data to construct a multimodal knowledge graph includes:

[0017] Perform entity recognition on data of different modalities and generate corresponding entity descriptions; the entity includes modality type, entity name, entity description and data source id corresponding to the entity;

[0018] Extracting the relationship between entities and generating a relationship description; the relationship includes a head entity, a head entity description, a relationship, a relationship description, a tail entity and a tail entity description;

[0019] Index construction: Generate embedding vectors for entities and relationships and create indexes for them. The index is used to record the correspondence between entities and their vectors, entity descriptions and their vectors, relationships and their vectors, and relationship descriptions and their vectors.

[0020] Furthermore, if the multimodal data includes tabular data, the entity also includes the structure of the table: table name, column name, and description of the table.

[0021] Furthermore, decomposing the natural language query into a plurality of subtasks, determining the dependency relationship between the subtasks, and generating a task dependency graph includes:

[0022] There are four types of natural language queries: comparison, intersection, combination, and original. If a natural language query includes one or more of the three types, comparison, intersection, and combination, the natural language query is decomposed.

[0023] Score the decomposed subtasks. If the score is greater than the preset threshold, the decomposition is successful. Otherwise, decompose again.

[0024] Dependencies between subtasks are determined based on the dependent task items of each subtask, and a task dependency graph is generated.

[0025] Furthermore, the multimodal knowledge graph is searched to obtain entities and relationships related to the subtasks, including:

[0026] Based on the multimodal knowledge graph, entities and their descriptions are jointly searched to generate candidate entity items; relationships and their descriptions are jointly searched to generate candidate relationship items;

[0027] The candidate entities and candidate relations are analyzed through a large language model to screen out the entities and relations that are most relevant to the subtask.

[0028] Furthermore, based on the multimodal knowledge graph, the entity and its description are jointly searched to generate candidate entity items; the relationship and its description are jointly searched to generate candidate relationship items, including:

[0029] Each subtask corresponds to a query entity. The entities in the multimodal knowledge graph are sorted according to their similarity with the query entity. At the same time, another sorting is performed according to the similarity between the subtask and the entity description. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the entity. The top n entities are taken as candidate entity items.

[0030] The relationships in the multimodal knowledge graph are sorted according to their similarity with the entity relationships corresponding to the query entity. At the same time, another sorting is performed according to the similarity between the subtask and the relationship description. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the relationship. The top n relationships are taken as candidate relationship items.

[0031] Furthermore, the large language model generates a first answer based on the retrieved entities and relations, including:

[0032] If the entities and relations currently retrieved are not sufficient to support the answer to the natural language query, continue to recursively search in the knowledge graph until enough information is obtained to generate the first answer.

[0033] Further, generating a final answer based on the first answer and the second answer includes:

[0034] If the large language model cannot generate the first answer based on the retrieved entities and relations, the second answer is the final answer;

[0035] If the first answer and the second answer are inconsistent, the two answers are given to the large language model to decide and generate the final answer.

[0036] The present invention also provides a multimodal question-answering device based on a large language model and a knowledge graph, comprising:

[0037] A knowledge graph construction module, used to obtain multimodal data to construct a multimodal knowledge graph;

[0038] A query parsing module is used to decompose a natural language query into several subtasks, determine the dependencies between the subtasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks;

[0039] A retrieval module is used to search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes a modality type, an entity name, an entity description, and a data source ID corresponding to the entity;

[0040] The answer generation module is used for the large language model to generate a first answer based on the retrieved entities and relationships; call the corresponding modal question and answer model according to the modal type in the retrieved entity, and the modal question and answer model analyzes the source data corresponding to the entity to generate a second answer; and generate a final answer based on the first answer and the second answer.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0042] 1. Multimodal knowledge graph and index construction, that is, to establish a semantic association network between modalities and data within modalities, and to establish an index at the same time. Technology for constructing multimodal knowledge graphs using LLM and different modal processing algorithms. By performing entity recognition on multimodal data such as text, images, and tables, and generating a detailed description of the entity, LLM is used to extract the relationship between entities from the data to form a knowledge graph. At the same time, an embedding vector is generated for each entity and relationship, and an efficient index is constructed to support fast retrieval. This process enables multimodal data to be associated and represented in a unified graph, overcoming the problems of decentralized data processing and difficulty in unifying modal information in traditional methods.

[0043] 2. By decomposing natural language queries into multiple subtasks and building a task dependency graph for this purpose, we can handle complex queries. We can identify different types of queries (such as comparison, intersection, combination, etc.) and decompose them into multiple simple subtasks. By generating a task dependency graph to determine the execution order, we ensure that complex queries can be reasonably disassembled and correctly executed.

[0044] 3. By combining coarse-grained and fine-grained retrieval, the natural language processing capabilities of LLM are fully utilized. First, candidate entities are generated by jointly retrieving entities and their description information, and then the most relevant entities and relationships are screened out through in-depth analysis of context and relationships. This ensures that the method of the present invention can quickly find content that is highly matched with the query in a huge knowledge graph, solving the problem of low efficiency in multimodal data retrieval.

[0045] 4. Complete reasoning and answer generation by combining LLM and the corresponding modal question-answering model. During the reasoning process, LLM is responsible for preliminary reasoning on the retrieved entities and relationships, and the corresponding modal question-answering model performs in-depth analysis on non-text modal data such as images and videos to make up for the shortcomings of LLM in non-text modal reasoning. By combining the natural language answers generated by LLM with the analysis results of the corresponding modal question-answering model, more accurate and comprehensive answers can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0047] Figure 1This is a flow chart of a multimodal question answering method based on a large language model and a knowledge graph provided by an embodiment of the present invention.

[0048] Figure 2 It is a flow chart of query execution and answer generation provided by an embodiment of the present invention.

[0049] Figure 3 It is a multimodal knowledge graph construction flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0051] A multimodal question answering method based on a large language model and a knowledge graph of the present invention, such as Figure 1 As shown, the following steps are included:

[0052] (1) acquiring multimodal data to construct a multimodal knowledge graph; wherein the multimodal data includes one or more of text data, image data, audio data, video data, and table data;

[0053] In one embodiment, see Figure 3 , the step (1) comprises the following sub-steps:

[0054] (1.1) Perform entity recognition on data of different modalities and generate corresponding entity descriptions. Each entity structure contains the entity name, modality type, entity description, and the data source id corresponding to the entity.

[0055] Specifically, data in text modality are subject to entity recognition based on LLM; data in image modality use the image object detection VinVL algorithm to identify entities on the image, and the Oscar algorithm is used to generate image subtitle descriptions; data in table modality use decision trees to identify key columns and entity columns and their corresponding column values ​​as entities based on the features of column headers and data content, and generate a description of the table; data in audio modality use the Whisper algorithm to convert audio into text, and then use LLM for entity recognition; data in video modality use the VideoLLaMA 2 algorithm to generate video subtitles and extract key objects in the video, and LLM is used to perform entity recognition on the video subtitles.

[0056] (1.2) Based on LLM, relations are extracted and relation descriptions are generated. Each relation structure contains a head entity, a head entity description, a relation, a relation description, a tail entity, and a tail entity description.

[0057] (1.3) Index construction. Generate embedding vectors for entity structures and relationship structures, and create indexes for them. Each index structure records samples (i.e., entities and their descriptions, relationships and their descriptions) and the corresponding relationships between vectors to support subsequent entity retrieval and relationship retrieval.

[0058] (2) decomposing the natural language query into a plurality of subtasks, determining the dependency relationships between the subtasks, and generating a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks;

[0059] In one embodiment, the step (2) includes the following sub-steps:

[0060] (2.1) Natural language query type identification. There are four types of queries: comparison, intersection, combination, and primitive.

[0061] (2.2) Query decomposition. If the natural language query includes one or more of the following three types: comparison, intersection, and combination, then decomposition is performed, and each subtask is ensured to be a meta-task and cannot be decomposed further. Based on the given typical sample set as an example for model learning, the natural language query is decomposed into multiple simple subtasks. Each decomposed subtask contains the tasks it depends on, so as to determine the dependency relationship between subtasks.

[0062] The step (2.2) also includes: scoring the decomposed subtasks. LLM scores the decomposed subtasks based on the requirements of query decomposition, and if the score is less than a threshold, the decomposition is repeated. The requirements for query decomposition include: (a) all decomposed subtasks are meta-tasks, that is, they cannot be decomposed any further; (b) all decomposed subtasks are directly related to the original question (natural language query), that is, background knowledge or other irrelevant subtasks do not meet the requirements; (c) all decomposed subtasks can be combined to answer the original question; (d) the dependencies of the decomposed subtasks are all correct.

[0063] (2.3) Determine the dependency relationship between subtasks based on the dependent task items of each subtask and generate a task dependency graph.

[0064] For example, the original query "Which movie did Ben Piazza star in first, Nighthawks or the movie with half a woman's face on the poster?" is a combination query type, where Ben Piazza starred in and the movie poster shows half a woman's face, Ben Piazza starred in and the movie is called Nighthawks are intersection queries, and the comparison of the release dates of Nighthawks and the movie with half a woman's face on the poster is a comparison query. This query can be divided into the following subtasks:

[0065] a) What movies has Ben Piazza starred in? Dependents: None

[0066] b) Did Ben Piazza star in the movie Nighthawks? Dependent task item: a)

[0067] c) When was the movie Nighthawks released? Dependent tasks: None

[0068] d) Which movie poster shows half of a woman's face? Dependent items: None

[0069] e) Does this movie star Ben Piazza? Dependent task items: a) d)

[0070] f) When will the movie be released? Dependent task item: d)

[0071] g) Which of the following was the first film in which Ben Piazza acted: Nighthawks or a film showing half a woman's face? Dependent items: b) c) e) f)

[0072] A task dependency graph is generated based on the subtasks decomposed above and the dependencies between them.

[0073] (3) Search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes the modality type, entity name, entity description, and the data source ID corresponding to the entity;

[0074] In one embodiment, the search includes coarse-grained search and fine-grained search.

[0075] (3.1) Coarse-grained retrieval. Perform a joint search on entities and their descriptions, perform a weighted sum of the entity similarity ranking and the entity description similarity ranking to obtain the final ranking, and generate candidate entity items; perform a joint search on relationships and their descriptions, perform a weighted sum of the relationship similarity ranking and the relationship description similarity ranking to obtain the final ranking, and generate candidate relationship items.

[0076] Specifically, each subtask corresponds to a query entity, and the entities in the multimodal knowledge graph are sorted according to the similarity between the query entity and the subtask. At the same time, another sorting is performed according to the similarity between the subtask and the entity description. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the entity. The top n entities are taken as candidate entity items.

[0077] The relationships in the multimodal knowledge graph are sorted according to the similarity with the entity relationships corresponding to the query entity (preferably, the relationships in the multimodal knowledge graph associated with the entities in the candidate entity items are sorted according to the similarity with the most relevant entity relationships corresponding to the query entity). At the same time, another sorting is performed according to the similarity between the subtasks and the relationship descriptions. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the relationships. The top n relationships are taken as candidate relationship items.

[0078] Among them, according to the context information of the subtask, the entities related to the subtask (i.e., the query entities) and their corresponding relationships (i.e., the entity relationships corresponding to the query entities) are extracted. For the query instance in the figure, according to the dependency relationship between subtask 1 and subtask 2, subtask 1 should be executed first and then subtask 2; specifically, the query entity Ben Piazza that appears in the decomposed subtask 1 is extracted. For subtask 2, which depends on the answer generated by subtask 1, the entities in subtask 2 and the entities in the generated answer are used as query entities. In this example, since subtask 2 itself does not extract an entity, the query entity of subtask 2 is the answer entity generated by subtask 1. The entity relationships corresponding to the query entity Ben Piazza include release time, movie name, and role. According to the context information of the subtask, the movie name is the most relevant entity relationship.

[0079] (3.2) Fine-grained retrieval. LLM analyzes candidate entities and candidate relationships one by one to further select the entities and relationships that are most relevant to the subtask. LLM comprehensively considers the contextual relationships, attributes and other information of the entity to ensure that the selected candidate items are most relevant to the subtask.

[0080] (4) The large language model generates the first answer based on the retrieved entities and relations;

[0081] Specifically, LLM generates a series of candidate answers (i.e., first answers) based on the retrieved entities and relations. Each candidate answer includes the reasoning process and answer confidence. If the currently retrieved entities and relations are not sufficient to support an accurate answer to the natural language query, it continues to recursively search in the multimodal knowledge graph until enough information is obtained to generate a complete answer.

[0082] (5) calling a corresponding modal question-answering model according to the modal type in the retrieved entity, wherein the modal question-answering model analyzes source data corresponding to the entity and generates a second answer;

[0083] In order to compensate for the loss of other modal information besides text modality, the corresponding modal question-answering model is called according to the modality type of the entity at this stage. For data involving non-text modalities such as images, videos, audio, tables, etc. (such as medical images, charts, etc.), the corresponding modal question-answering model will analyze the source data and generate the corresponding reasoning result, that is, the second answer, which also contains as many reasoning processes as possible; the source data is obtained according to the data source id corresponding to the entity.

[0084] Specifically, for image modality data, the BLIP model is used for question answering; for video and audio modality data, the VideoLLaMA 2 model is used for question answering; for table modality data, the Tapas model is used for question answering. Figure 2 In the example in , for the query entity "Ben Piazza", the most relevant entity representation is found in the knowledge graph and its attribute information is obtained, but since the current information is not enough for LLM to generate an answer, it will continue to search until the answer is generated. On the other hand, knowing that this entity is a table modal type and table structure, it will generate corresponding questions for the table and call the Tapas model for question and answer. Inference is continued until the answer is generated.

[0085] (6) Generate a final answer based on the first answer and the second answer.

[0086] Specifically, if the LLM cannot generate an answer, the answer from the corresponding modal question-answering model is used; if the answers generated by the two are inconsistent, the two answers are given to another LLM to decide and generate the final answer. The LLM and the corresponding modal question-answering model are combined to generate a more complete and accurate answer. Figure 2 In the example, since the answers generated by the LLM and the corresponding modal question answering model are inconsistent, another LLM is used to make a decision to generate the final answer. The LLM answer reasoning process only mentioned that the movie poster has the object "face", and did not clearly determine that it is half a woman's face. The answer obtained by the corresponding modal question answering model clearly mentioned whether it contains half a woman's face, so the final answer is "Smile", not "Smile" and "Bears".

[0087] The present invention also provides a multimodal question-answering device based on a large language model and a knowledge graph, comprising:

[0088] A knowledge graph construction module, used to obtain multimodal data to construct a multimodal knowledge graph;

[0089] A query parsing module is used to decompose a natural language query into several subtasks, determine the dependencies between the subtasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks;

[0090] A retrieval module is used to search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes a modality type, an entity name, an entity description, and a data source ID corresponding to the entity;

[0091] The answer generation module is used for the large language model to generate a first answer based on the retrieved entities and relationships; call the corresponding modal question and answer model according to the modal type in the retrieved entity, and the modal question and answer model analyzes the source data corresponding to the entity to generate a second answer; and generate a final answer based on the first answer and the second answer.

[0092] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0093] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A multimodal question answering method based on a large language model and a knowledge graph, characterized in that: include: Acquire multimodal data to build a multimodal knowledge graph; Decompose natural language queries into several subtasks, determine the dependencies between subtasks, and generate a task dependency graph; The task dependency graph is used to determine the execution order of subtasks; Search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes the modality type, entity name, entity description and the data source ID corresponding to the entity; The large language model generates the first answer based on the retrieved entities and relations; Calling a corresponding modal question-answering model according to the modal type in the retrieved entity, wherein the modal question-answering model analyzes source data corresponding to the entity to generate a second answer; Generate a final answer based on the first answer and the second answer.

2. The method according to claim 1, characterized in that The multimodal data includes one or more of text data, image data, audio data, video data and table data.

3. The method according to claim 1, characterized in that The acquiring of multimodal data to construct a multimodal knowledge graph includes: Perform entity recognition on data of different modalities and generate corresponding entity descriptions; the entity includes modality type, entity name, entity description and data source id corresponding to the entity; Extracting the relationship between entities and generating a relationship description; the relationship includes a head entity, a head entity description, a relationship, a relationship description, a tail entity and a tail entity description; Index construction: Generate embedding vectors for entities and relationships and create indexes for them. The index is used to record the correspondence between entities and their vectors, entity descriptions and their vectors, relationships and their vectors, and relationship descriptions and their vectors.

4. The method according to claim 1, characterized in that: If the multimodal data includes tabular data, the entity also includes the structure of the table: table name, column names, and description of the table.

5. The method according to claim 1, characterized in that The step of decomposing a natural language query into a plurality of subtasks, determining dependencies between the subtasks, and generating a task dependency graph includes: There are four types of natural language queries: comparison, intersection, combination, and original. If a natural language query includes one or more of the three types, comparison, intersection, and combination, the natural language query is decomposed. Score the decomposed subtasks. If the score is greater than the preset threshold, the decomposition is successful. Otherwise, decompose again. Dependencies between subtasks are determined based on the dependent task items of each subtask, and a task dependency graph is generated.

6. The method according to claim 1, characterized in that The multimodal knowledge graph is searched to obtain entities and relationships related to the subtasks, including: Based on the multimodal knowledge graph, entities and their descriptions are jointly searched to generate candidate entity items; relationships and their descriptions are jointly searched to generate candidate relationship items; The candidate entities and candidate relations are analyzed through a large language model to screen out the entities and relations that are most relevant to the subtask.

7. The method according to claim 6, characterized in that Based on the multimodal knowledge graph, the entity and its description are jointly searched to generate candidate entity items; the relationship and its description are jointly searched to generate candidate relationship items, including: Each subtask corresponds to a query entity. The entities in the multimodal knowledge graph are sorted according to their similarity with the query entity. At the same time, another sorting is performed according to the similarity between the subtask and the entity description. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the entity. The top n entities are taken as candidate entity items. The relationships in the multimodal knowledge graph are sorted according to their similarity with the entity relationships corresponding to the query entity. At the same time, another sorting is performed according to the similarity between the subtask and the relationship description. The two sortings are weighted and summed according to the set weights to obtain the final ranking of the relationship. The top n relationships are taken as candidate relationship items.

8. The method according to claim 1, characterized in that The large language model generates a first answer based on the retrieved entities and relations, including: If the entities and relations currently retrieved are not sufficient to support the answer to the natural language query, continue to recursively search in the knowledge graph until enough information is obtained to generate the first answer.

9. The method according to claim 1, characterized in that: Generating a final answer based on the first answer and the second answer comprises: If the large language model cannot generate the first answer based on the retrieved entities and relations, the second answer is the final answer; If the first answer and the second answer are inconsistent, the two answers are given to the large language model to decide and generate the final answer.

10. A multimodal question-answering device based on a large language model and a knowledge graph, characterized in that: include: A knowledge graph construction module, used to obtain multimodal data to construct a multimodal knowledge graph; The query parsing module is used to decompose the natural language query into several subtasks, determine the dependencies between the subtasks, and generate a task dependency graph; The task dependency graph is used to determine the execution order of subtasks; A retrieval module is used to search the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes a modality type, an entity name, an entity description, and a data source ID corresponding to the entity; The answer generation module is used by the large language model to generate the first answer based on the retrieved entities and relations; Calling a corresponding modal question-answering model according to the modal type in the retrieved entity, wherein the modal question-answering model analyzes source data corresponding to the entity to generate a second answer; Generate a final answer based on the first answer and the second answer.

Images