A multi-modal question answering method based on a large language model and a knowledge graph

Abstract

The application discloses a kind of multi-modal question answering method based on large language model and knowledge graph, belong to artificial intelligence and knowledge graph technical field. In view of the problem such as insufficient multi-modal information processing, insufficient semantic understanding depth and limited cross-modal reasoning ability in existing question answering system, the application proposes a kind of multi-modal question answering engine, constructs multi-modal knowledge graph for multi-modal data, establishes the semantic association network between modal and modal, and generates index for the elements such as entity, relationship and the like of knowledge graph, to support subsequent efficient retrieval. Through structured information retrieval on knowledge graph, combined with the powerful natural language understanding ability of large language model and the ability of specific modal question answering model, the efficiency and accuracy of information retrieval are improved, and the cross-modal reasoning ability is enhanced, so as to realize efficient and accurate question answering in complex multi-modal environment.

Description

Technical Field

[0001] This invention relates to the fields of knowledge graphs, large language models, and data retrieval, and particularly to a multimodal question answering method based on large language models and knowledge graphs. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems have become one of the important tools for solving complex problems and providing accurate information. Traditional question-answering systems typically rely on single-modal data input, primarily based on natural language text. However, in the real world, information often presents itself in multiple forms. For example, diagnosis in the medical field relies not only on doctors' descriptions and patients' medical records but also on X-rays, MRI images, etc. Furthermore, scientific papers, financial statements, product specifications, etc., often contain rich tabular information, and the dynamic data and images contained in videos can provide a more intuitive understanding. Traditional question-answering systems can only process data from a single modality and cannot effectively integrate multimodal information. With the further development of technologies such as deep learning and graph neural networks (GNNs), multimodal question-answering systems have emerged, improving the breadth and depth of question-answering systems by processing and integrating data from different modalities (such as text, images, videos, tables, etc.). Existing multimodal question answering methods fall into two categories: one is to convert other modalities into text modalities and utilize existing text-based question answering systems for reasoning and response; the other is to process each modality independently, performing question and answer tasks separately, and then integrating the answers from each modality. Regardless of whether modality conversion or separate processing is adopted, current multimodal question answering systems face the following three main challenges in practical applications:

[0003] 1. Preprocessing of Multimodal Information. Existing question-answering systems are mostly based on a single modality (mainly text), failing to fully utilize the potential of non-textual data such as images and tables, resulting in the system's inability to provide comprehensive and accurate answers. In multimodal scenarios, how to effectively integrate different types of modal data (such as images, videos, text, tables, audio, etc.), and how to achieve unified processing while maintaining the advantages of each modality, is an urgent problem to be solved.

[0004] 2. Weak cross-modal reasoning ability. Current question-answering systems still lack sufficient cross-modal reasoning capabilities, failing to effectively establish information connections and perform reasoning across different modalities. This deficiency makes it difficult for the system to provide comprehensive and accurate answers to questions involving multimodal data. With the increasing prevalence of multimodal data, cross-modal reasoning capabilities will become increasingly important. Only by achieving efficient reasoning across different modalities can the system perform well on complex problems.

[0005] 3. Efficiency of large-scale data processing. Multimodal question-answering systems typically need to process large amounts of data sources such as text, images, and videos. This requires the system to be able to quickly and efficiently retrieve, process, and generate answers when faced with massive amounts of information. How to optimize the processing efficiency of large-scale data, especially in the process of modal conversion, fusion, and reasoning, remains a pressing problem to be solved.

[0006] Therefore, this invention proposes a multimodal question answering method based on a combination of a large language model (LLM) and a knowledge graph. This method constructs and indexes a multimodal knowledge graph, utilizes a large language model for natural language understanding, and combines it with a specific modality question answering model for deep reasoning, significantly improving the multimodal question answering system's ability to handle complex problems, perform cross-modal reasoning, and achieve efficient retrieval. This multimodal question answering engine can not only process multimodal data but also provide more accurate and comprehensive answers through efficient semantic retrieval and reasoning. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a multimodal question answering method based on a large language model and knowledge graph, which solves the problems of insufficient multimodal information processing, limited cross-modal reasoning capabilities, and large-scale data processing efficiency in existing multimodal question answering systems.

[0008] The objective of this invention is achieved through the following technical solution: a multimodal question-answering method based on a large language model and knowledge graph, comprising:

[0009] Acquire multimodal data to construct a multimodal knowledge graph;

[0010] The natural language query is decomposed into several subtasks, the dependencies between the subtasks are determined, and a task dependency graph is generated; the task dependency graph is used to determine the execution order of the subtasks.

[0011] The multimodal knowledge graph is searched to obtain entities and relationships related to the subtask; each entity includes modality type, entity name, entity description, and the data source ID corresponding to the entity.

[0012] The large language model generates the first answer based on the retrieved entities and relationships;

[0013] Based on the modal type of the retrieved entity, the corresponding modal question-answering model is invoked. The modal question-answering model analyzes the source data corresponding to the entity and generates a second answer.

[0014] The final answer is generated based on the first and second answers.

[0015] Furthermore, the multimodal data includes one or more of text data, image data, audio data, video data, and tabular data.

[0016] Furthermore, the acquisition of multimodal data to construct a multimodal knowledge graph includes:

[0017] Entity recognition is performed on data of different modalities, and corresponding entity descriptions are generated; each entity includes modality type, entity name, entity description, and the data source ID corresponding to the entity.

[0018] Extract the relationships between entities and generate relationship descriptions; the relationships include head entity, head entity description, relationship, relationship description, tail entity, and tail entity description;

[0019] Index building: Generate embedding vectors for entities and relations, and build indexes for them. The indexes are used to record the correspondence between entities and their vectors, entity descriptions and their vectors, relations and their vectors, and relation descriptions and their vectors.

[0020] Furthermore, if the multimodal data includes tabular data, the entity also includes the table structure: table name, column names, and table description.

[0021] Furthermore, the step of decomposing the natural language query into several sub-tasks, determining the dependencies between the sub-tasks, and generating a task dependency graph includes:

[0022] Natural language query types are divided into four categories: comparison, intersection, combination, and original. If a natural language query includes one or more of these three types, then the natural language query is decomposed.

[0023] The subtasks after decomposition are scored. If the score is greater than the preset threshold, the decomposition is successful; otherwise, the decomposition is repeated.

[0024] The dependencies between subtasks are determined based on the dependent task items of each subtask, and a task dependency graph is generated.

[0025] Furthermore, the retrieval of the multimodal knowledge graph to obtain entities and relationships related to the subtask includes:

[0026] Based on multimodal knowledge graphs, joint retrieval of entities and their descriptions is performed to generate candidate entity items; joint retrieval of relations and their descriptions is performed to generate candidate relation items.

[0027] By analyzing candidate entities and relations using a large language model, the entities and relations most relevant to the subtask are selected.

[0028] Furthermore, the method of jointly retrieving entities and their descriptions based on a multimodal knowledge graph to generate candidate entity items; and jointly retrieving relations and their descriptions to generate candidate relation items, includes:

[0029] Each subtask corresponds to a query entity. The entities in the multimodal knowledge graph are sorted according to their similarity to the query entity. At the same time, the subtask is sorted again according to its similarity to the entity description. The two sorts are weighted and summed according to a set weight to obtain the final ranking of the entities. The top n entities are selected as candidate entity items.

[0030] The relationships in the multimodal knowledge graph are sorted based on their similarity to the entity relationships corresponding to the query entity. At the same time, another sort is performed based on the similarity between the subtask and the relationship description. The two sorts are then weighted and summed according to a set weight to obtain the final ranking of the relationships. The top n relationships are selected as candidate relationship items.

[0031] Furthermore, the large language model generates a first answer based on the retrieved entities and relations, including:

[0032] If the currently retrieved entities and relationships are insufficient to support an answer to the natural language query, the search continues recursively within the knowledge graph until enough information is obtained to generate the first answer.

[0033] Furthermore, the generation of the final answer based on the first and second answers includes:

[0034] If the large language model cannot generate the first answer based on the retrieved entities and relations, then the second answer will be the final answer;

[0035] If the first and second answers are inconsistent, then these two answers are fed to the large language model to make a decision and generate the final answer.

[0036] This invention also provides a multimodal question-answering device based on a large language model and knowledge graph, comprising:

[0037] The knowledge graph construction module is used to acquire multimodal data to construct a multimodal knowledge graph;

[0038] The query parsing module is used to decompose natural language queries into several subtasks, determine the dependencies between subtasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks.

[0039] The retrieval module is used to retrieve entities and relationships related to the subtask from the multimodal knowledge graph; the entity includes modality type, entity name, entity description and the data source ID corresponding to the entity;

[0040] The answer generation module is used by the large language model to generate a first answer based on the retrieved entities and relationships; to call the corresponding modal question answering model based on the modal type in the retrieved entity, and the modal question answering model analyzes the source data corresponding to the entity to generate a second answer; and to generate a final answer based on the first and second answers.

[0041] Compared with the prior art, the beneficial effects of the embodiments of the present invention are:

[0042] 1. Multimodal knowledge graph and index construction involves establishing a semantic association network between and within modalities, while simultaneously building an index. This technique utilizes LLM (Limited Language Management) and various modality processing algorithms to construct a multimodal knowledge graph. Entity recognition is performed on various modalities such as text, images, and tables, generating detailed descriptions of entities. LLM is then used to extract relationships between entities, forming a knowledge graph. Simultaneously, embedding vectors are generated for each entity and relationship, and an efficient index is built to support fast retrieval. This process enables multimodal data to be associated and represented within a unified graph, overcoming the problems of fragmented data processing and difficulty in unifying modal information in traditional methods.

[0043] 2. By decomposing natural language queries into multiple subtasks and constructing a task dependency graph, we have achieved the processing of complex queries. We can identify different types of queries (such as comparisons, intersections, combinations, etc.) and decompose them into multiple simple subtasks. By generating a task dependency graph, we determine the execution order, ensuring that complex queries can be reasonably decomposed and correctly executed.

[0044] 3. By combining coarse-grained and fine-grained retrieval methods, the natural language processing capabilities of LLM are fully utilized. First, candidate entities are generated by jointly retrieving entities and their descriptions. Then, the most relevant entities and relationships are selected through in-depth analysis of context and relationships. This ensures that the method of this invention can quickly find content that highly matches the query in a huge knowledge graph, thus solving the problem of low efficiency in multimodal data retrieval.

[0045] 4. Reasoning and answer generation are accomplished by combining LLM (Limited Language Modeling) with a corresponding modal question-answering model. During the reasoning process, LLM performs initial reasoning on the retrieved entities and relationships, while the corresponding modal question-answering model conducts in-depth analysis of non-textual modal data such as images and videos to compensate for the limitations of LLM in non-textual modal reasoning. By combining the natural language answers generated by LLM with the analysis results of the corresponding modal question-answering model, more accurate and comprehensive answers can be generated. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1This is a flowchart of a multimodal question-answering method based on a large language model and knowledge graph provided in an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of query execution and answer generation provided by an embodiment of the present invention.

[0049] Figure 3 This is a flowchart of a multimodal knowledge graph construction process provided by an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0051] This invention provides a multimodal question-answering method based on a large language model and knowledge graph, such as... Figure 1 As shown, it includes the following steps:

[0052] (1) Acquire multimodal data to construct a multimodal knowledge graph; wherein, multimodal data includes one or more of text data, image data, audio data, video data and tabular data;

[0053] In one embodiment, see Figure 3 Step (1) includes the following sub-steps:

[0054] (1.1) Perform entity recognition on data of different modalities and generate corresponding entity descriptions. Each entity structure includes entity name, modality type, entity description, and the data source ID corresponding to the entity.

[0055] Specifically, text-based data is used for entity recognition based on LLM; image-based data uses the VinVL algorithm to identify entities in the image and the Oscar algorithm to generate image captions; table-based data uses decision trees to identify key columns and entity columns and their corresponding column values ​​as entities based on the features of column headers and data content, and generates a description of the table; audio-based data uses the Whisper algorithm to convert audio into text, and then uses LLM for entity recognition; video-based data uses the VideoLLaMA2 algorithm to generate video captions and extract key objects in the video, and then uses LLM for entity recognition of the video captions.

[0056] (1.2) Relation extraction and relation description generation are performed based on LLM. Each relation structure contains a head entity, a head entity description, a relation, a relation description, a tail entity, and a tail entity description.

[0057] (1.3) Index Construction. Embedded vectors are generated for entity and relation structures, and indexes are built for them. Each index structure records the correspondence between samples (i.e., entities and their descriptions, relations and their descriptions) and vectors to support subsequent entity and relation retrieval.

[0058] (2) Decompose the natural language query into several sub-tasks, determine the dependencies between the sub-tasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the sub-tasks.

[0059] In one embodiment, step (2) includes the following sub-steps:

[0060] (2.1) Natural Language Query Type Recognition. There are four types of queries: comparison, intersection, combination, and original.

[0061] (2.2) Query Decomposition. If a natural language query includes one or more of the three types of operations—comparison, intersection, and combination—it is decomposed, ensuring that each subtask is a meta-task and cannot be further decomposed. The natural language query is decomposed into multiple simple subtasks based on a given set of typical samples used as examples for model learning. Each decomposed subtask includes the tasks it depends on, thus determining the dependencies between subtasks.

[0062] Step (2.2) further includes: scoring the decomposed subtasks. LLM scores the decomposed subtasks based on the query decomposition requirements. If the score is less than a threshold, the subtasks are decomposed again. The query decomposition requirements include: (a) all decomposed subtasks are meta-tasks, i.e., they cannot be further decomposed; (b) all decomposed subtasks are directly related to the original question (natural language query), i.e., background knowledge or other irrelevant subtasks do not meet the requirements; (c) all decomposed subtasks combined can answer the original question; and (d) all dependencies of the decomposed subtasks are correct.

[0063] (2.3) Determine the dependencies between subtasks based on the dependent task items of each subtask, and generate a task dependency graph.

[0064] For example, the original query: "Did Ben Piazza first star in *Nighthawks* or the movie whose poster shows half a woman's face?" is a combined query. The queries that Ben Piazza starred in the movie and whose poster shows half a woman's face, and that Ben Piazza starred in the movie and whose title is *Nighthawks*, are intersection queries. The comparison of the release dates of *Nighthawks* and the movie whose poster shows half a woman's face is a comparison query. This query can be divided into the following sub-tasks:

[0065] a) What movies has Ben Piazza starred in? Dependency: None

[0066] b) Does the movie *Nighthawks* star Ben Piazza? (Dependency: a)

[0067] c) When was the movie "Night Hawk" released? Dependency: None

[0068] d) Which movie poster shows half a woman's face? Dependency: None

[0069] e) Does Ben Piazza star in this movie? (Dependency: a) d)

[0070] f) When is that movie released? (Dependency: d)

[0071] g) Which was Ben Piazza's earliest film appearance: *Nighthawks* or a movie showing half a woman's face? (Dependency: b) c) e) f)

[0072] Generate a task dependency graph based on the subtasks decomposed above and their dependencies.

[0073] (3) Retrieve the multimodal knowledge graph to obtain entities and relationships related to the subtask; the entity includes modality type, entity name, entity description and the data source ID corresponding to the entity;

[0074] In one embodiment, the retrieval includes coarse-grained retrieval and fine-grained retrieval.

[0075] (3.1) Coarse-grained retrieval. Joint retrieval of entities and their descriptions is performed, and the final ranking is obtained by weighted summation of entity similarity ranking and entity description similarity ranking, generating candidate entity items; joint retrieval of relations and their descriptions is performed, and the final ranking is obtained by weighted summation of relation similarity ranking and relation description similarity ranking, generating candidate relation items.

[0076] Specifically, each subtask corresponds to a query entity. The entities in the multimodal knowledge graph are sorted according to their similarity to the query entity. At the same time, the subtask is sorted again according to its similarity to the entity description. The two sorts are weighted and summed according to a set weight to obtain the final ranking of the entities. The top n entities are selected as candidate entity items.

[0077] The relationships are sorted based on the similarity between the relationships in the multimodal knowledge graph and the entity relationships corresponding to the query entity (preferably, sorted based on the similarity between the relationships in the multimodal knowledge graph that are associated with the entities in the candidate entity items and the most relevant entity relationships corresponding to the query entity). At the same time, another sort is performed based on the similarity between the subtask and the relationship description. The two sorts are weighted and summed according to the set weights to obtain the final ranking of the relationships. The top n relationships are selected as candidate relationship items.

[0078] Based on the context information of each subtask, the entities related to that subtask (i.e., query entities) and their corresponding relationships (i.e., entity relationships corresponding to the query entities) are extracted. For the query example in the diagram, based on the dependency between subtask 1 and subtask 2, subtask 1 should be executed first, followed by subtask 2. Specifically, the query entity "Ben Piazza" appearing in the decomposed subtask 1 is extracted. Since subtask 2 depends on the answer generated by subtask 1, the entities in subtask 2 and the entities in the generated answer are used as query entities. In this example, since no entity is extracted from subtask 2 itself, the query entity for subtask 2 is the answer entity generated by subtask 1. The entity relationships corresponding to the query entity "Ben Piazza" include release date, movie name, and character; based on the context information of the subtask, the movie name is the most relevant entity relationship.

[0079] (3.2) Fine-grained retrieval. Candidate entities and relationships are analyzed one by one using LLM to further filter out the entities and relationships most relevant to the subtask. LLM comprehensively considers the contextual relationships and attributes of entities to ensure that the selected candidates are most relevant to the subtask.

[0080] (4) The large language model generates the first answer based on the retrieved entities and relationships;

[0081] Specifically, LLM generates a series of candidate answers (i.e., first answers) based on the retrieved entities and relationships. Each candidate answer includes the reasoning process and the answer's confidence level. If the currently retrieved entities and relationships are insufficient to support an accurate answer to the natural language query, the search continues recursively within the multimodal knowledge graph until enough information is obtained to generate a complete answer.

[0082] (5) Based on the modal type of the retrieved entity, call the corresponding modal question answering model, which analyzes the source data corresponding to the entity and generates a second answer;

[0083] To compensate for the loss of information in modalities other than text, the corresponding modal question-answering model is invoked at this stage based on the entity's modal type. For non-text modal data such as images, videos, audio, and tables (e.g., medical images, charts), the corresponding modal question-answering model analyzes the source data and generates a corresponding reasoning result, i.e., the second answer, which also includes as much reasoning as possible; the source data is obtained based on the data source ID corresponding to the entity.

[0084] Specifically, for image-based data, the BLIP model is used for question answering; for video and audio-based data, the VideoLLaMA 2 model is used; and for tabular-based data, the Tapas model is used. Figure 2 In the example, for the query entity "Ben Piazza," the most relevant entity representation and its attribute information are found in the knowledge graph. However, since this information is insufficient for the LLM to generate an answer, the search continues until an answer is generated. On the other hand, knowing that the entity is a table modality and has a table structure, corresponding questions about the table are generated, and the Tapas model is invoked for question answering. This reasoning continues until an answer is generated.

[0085] (6) Generate the final answer based on the first and second answers.

[0086] Specifically, if an answer cannot be generated using the LLM (Local Mode Model), the answer from the corresponding modal question-answering model is used; if the answers generated by the two models are inconsistent, both answers are fed to another LLM model to determine the final answer. Combining the LLM with the corresponding modal question-answering model generates a more complete and accurate answer. Figure 2 In the example above, because the answers generated by the LLM and the corresponding modal question answering model were inconsistent, another LLM was used to decide on the final answer. The LLM's answer reasoning process only mentioned that the movie poster had an object called "face," without explicitly identifying it as half of a woman's face. However, the corresponding modal question answering model's answer explicitly mentioned whether it contained half of a woman's face. Therefore, the final answer was "Smile," not "Smile" or "Bear Team."

[0087] This invention also provides a multimodal question-answering device based on a large language model and knowledge graph, comprising:

[0088] The knowledge graph construction module is used to acquire multimodal data to construct a multimodal knowledge graph;

[0089] The query parsing module is used to decompose natural language queries into several subtasks, determine the dependencies between subtasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks.

[0090] The retrieval module is used to retrieve entities and relationships related to the subtask from the multimodal knowledge graph; the entity includes modality type, entity name, entity description and the data source ID corresponding to the entity;

[0091] The answer generation module is used by the large language model to generate a first answer based on the retrieved entities and relationships; to call the corresponding modal question answering model based on the modal type in the retrieved entity, and the modal question answering model analyzes the source data corresponding to the entity to generate a second answer; and to generate a final answer based on the first and second answers.

[0092] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, and will not be repeated here.

[0093] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A multimodal question-answering method based on a large language model and knowledge graph, characterized in that, include: Acquire multimodal data to construct a multimodal knowledge graph; The natural language query is broken down into several sub-tasks, the dependencies between the sub-tasks are determined, and a task dependency graph is generated. The task dependency graph is used to determine the execution order of subtasks; The multimodal knowledge graph is searched to obtain entities and relationships related to the subtask; each entity includes modality type, entity name, entity description, and the data source ID corresponding to the entity. The large language model generates the first answer based on the retrieved entities and relationships; Based on the modal type of the retrieved entity, the corresponding modal question-answering model is invoked. The modal question-answering model analyzes the source data corresponding to the entity and generates a second answer. The final answer is generated based on the first and second answers; The acquisition of multimodal data to construct a multimodal knowledge graph includes: Perform entity recognition on data of different modalities and generate corresponding entity descriptions; Extract the relationships between entities and generate relationship descriptions; the relationships include head entity, head entity description, relationship, relationship description, tail entity, and tail entity description; Index building: Generate embedding vectors for entities and relations, and build indexes for them. The indexes are used to record the correspondence between entities and their vectors, entity descriptions and their vectors, relations and their vectors, and relation descriptions and their vectors.

2. The method according to claim 1, characterized in that, The multimodal data includes one or more of the following: text data, image data, audio data, video data, and tabular data.

3. The method according to claim 1, characterized in that, If the multimodal data includes tabular data, then the entity also includes the table structure: table name, column names, and table description.

4. The method according to claim 1, characterized in that, The process of decomposing natural language queries into several subtasks, determining the dependencies between subtasks, and generating a task dependency graph includes: Natural language query types are divided into four categories: comparison, intersection, combination, and original. If a natural language query includes one or more of these three types, then the natural language query is decomposed. The subtasks after decomposition are scored. If the score is greater than the preset threshold, the decomposition is successful; otherwise, the decomposition is repeated. The dependencies between subtasks are determined based on the dependent task items of each subtask, and a task dependency graph is generated.

5. The method according to claim 1, characterized in that, The process of retrieving the multimodal knowledge graph to obtain entities and relationships related to the subtask includes: Based on multimodal knowledge graphs, joint retrieval of entities and their descriptions is performed to generate candidate entity items; joint retrieval of relations and their descriptions is performed to generate candidate relation items. By analyzing candidate entities and relations using a large language model, the entities and relations most relevant to the subtask are selected.

6. The method according to claim 5, characterized in that, The method based on multimodal knowledge graphs performs joint retrieval of entities and their descriptions to generate candidate entity items; A joint retrieval of the relations and their descriptions is performed to generate candidate relation items, including: Each subtask corresponds to a query entity. The entities in the multimodal knowledge graph are sorted according to their similarity to the query entity. At the same time, the subtask is sorted again according to its similarity to the entity description. The two sorts are weighted and summed according to a set weight to obtain the final ranking of the entities. The top n entities are selected as candidate entity items. The relationships in the multimodal knowledge graph are sorted based on their similarity to the entity relationships corresponding to the query entity. At the same time, another sort is performed based on the similarity between the subtask and the relationship description. The two sorts are then weighted and summed according to a set weight to obtain the final ranking of the relationships. The top n relationships are selected as candidate relationship items.

7. The method according to claim 1, characterized in that, The large language model generates a first answer based on the retrieved entities and relations, including: If the currently retrieved entities and relationships are insufficient to support an answer to the natural language query, the search continues recursively within the knowledge graph until enough information is obtained to generate the first answer.

8. The method according to claim 1, characterized in that, The process of generating the final answer based on the first and second answers includes: If the large language model cannot generate the first answer based on the retrieved entities and relations, then the second answer will be the final answer; If the first and second answers are inconsistent, then these two answers are fed to the large language model to make a decision and generate the final answer.

9. A multimodal question-answering device based on a large language model and knowledge graph, characterized in that, include: The knowledge graph construction module is used to acquire multimodal data to construct a multimodal knowledge graph; The acquisition of multimodal data to construct a multimodal knowledge graph includes: Perform entity recognition on data of different modalities and generate corresponding entity descriptions; Extract the relationships between entities and generate relationship descriptions; the relationships include head entity, head entity description, relationship, relationship description, tail entity, and tail entity description; Index building: Generate embedding vectors for entities and relations, and build indexes for them. The indexes are used to record the correspondence between entities and their vectors, entity descriptions and their vectors, relations and their vectors, and relation descriptions and their vectors. The query parsing module is used to decompose natural language queries into several subtasks, determine the dependencies between subtasks, and generate a task dependency graph; the task dependency graph is used to determine the execution order of the subtasks. The retrieval module is used to retrieve entities and relationships related to the subtask from the multimodal knowledge graph; the entity includes modality type, entity name, entity description and the data source ID corresponding to the entity; The answer generation module is used by the large language model to generate a first answer based on the retrieved entities and relationships; to call the corresponding modal question answering model based on the modal type in the retrieved entity, and the modal question answering model analyzes the source data corresponding to the entity to generate a second answer; and to generate a final answer based on the first and second answers.

Images