Multi-modal knowledge graph construction and question answering method and system based on query driving

Abstract

The invention provides a multi-modal knowledge graph construction and question answering method and system based on query driving, and relates to the field of knowledge graphs, and the method comprises the steps: constructing a mode graph of user query; constructing a multi-modal knowledge base; constructing a clue map through a mode constraint retrieval method and a query focusing clue refining method in combination with the mode map; and through a cross-modal reasoning iteration mode, supplementing missing information of the cue graph to obtain a retrieval answer result queried by the user. According to the method, a query-driven multi-modal framework is constructed, the local knowledge graph of user query can be dynamically constructed, and rapid and accurate retrieval of user query problems is realized.

Description

Technical Field

[0001] This application relates to the field of knowledge graphs, and in particular, to a query-driven multi-modal knowledge graph construction and question-answering method and system. Background Art

[0002] The increasing popularity of large language models (LLMs) in complex reasoning tasks has made their interpretability and reliability crucial. To this end, recent advances include retrieval-augmented generation (RAG) and knowledge graph-augmented RAG (GraphRAG). However, they are both restricted by static knowledge bases and inefficient multi-modal data integration.

[0003] Combining knowledge graphs (KGs) with large language models (LLMs) through retrieval-augmented generation (RAG), i.e., GraphRAG, has become a promising paradigm to improve fact accuracy and reasoning ability (Pan et al., 2024; Edge et al., 2024). Traditional RAG systems use unstructured text retrieval, while recent GraphRAG methods attempt to combine structured knowledge graphs (KGs) with LLMs to capture complex entity relationships and enable multi-hop reasoning (Peng et al., 2024).

[0004] To achieve this, existing methods typically construct knowledge graphs in a bottom-up, information-driven manner (Cohen et al., 2023). This approach first identifies available data sources and then integrates the data sources into triples through entity extraction, relation extraction, and knowledge fusion. However, this may introduce additional complexity and runtime overhead. Specifically, these methods lack awareness of user needs during the knowledge graph construction process (Cohen et al., 2023), resulting in generated graphs that are either too refined and computationally expensive or too rough and lose key semantic details.

[0005] For example, in a multi-modal environment, parsing complex visual information without query context either leads to over-extraction of details or incomplete knowledge integration. This rigidity is particularly problematic for queries that require cross-modal understanding because existing frameworks have difficulty effectively combining text, visual, and tabular data into a unified knowledge representation. Summary of the Invention

[0006] The purpose of the present invention is to provide a query-driven multi-modal knowledge graph construction and question-answering method and system to solve the technical problem that existing frameworks have difficulty effectively combining text, visual, and tabular data into a unified knowledge representation.

[0007] The above object of this application is achieved by the following technical solutions:

[0008] S1: Construct a schema graph for the user query;

[0009] S2: Construct a multimodal knowledge base; construct a clue graph by means of a schema constraint retrieval method and a query focus clue refinement method, in combination with the schema graph;

[0010] S3: Supplement the missing information in the clue graph through cross-modal inference iteration to obtain the retrieval answer result of the user query.

[0011] Optionally, step S1 further includes:

[0012] Use a large language model LLM to generate a preliminary schema graph;

[0013] Receive the user query Q as the input of the large language model LLM;

[0014] Design a prompt template P pattern Guide the large language model LLM to extract the entity types and relationships of the user query Q and generate the schema graph of the user query Q: {T, R} = LLM(P pattern (Q));

[0015] The schema graph is a structured representation of the user query. Define the schema graph G1 as G1 = (T, R), where T = {t1, t2,..., t n} is a set of key entity types; R = {r1, r2,..., r m} is a set of semantic relationship types.

[0016] Optionally, step S2 includes:

[0017] Construct a retrieval graph G2 under the constraint guidance of the schema graph G1, in combination with the multimodal knowledge base D;

[0018] The retrieval graph is a constrained graph spectrum, where represents a resource lightweight graph spectrum construction operator, V represents type filtering, and E represents relationship pruning;

[0019] The retrieval graph G2 is subject to type filtering and relationship pruning;

[0020] Type filtering: V = {v ∈ D | φ type (v) ∈ T}, v represents an entity, and φ type represents mapping the entity to the entity type;

[0021] Relationship pruning: v i represents the i-th entity, v j represents the j-th entity, and r represents the entity v i and the entity vj The relationship between;

[0022] Extract entities and relationships that conform to the constraints of the pattern graph G1 from the text data D in the multimodal knowledge base D text to construct a retrieval graph G2;

[0023] Extract from D text The entities extracted where d represents a temporary variable; P extract (T, R) represents a hint template;

[0024] Treat each picture and table in the picture modality data D and the table modality data D in the multimodal knowledge base D image as independent nodes, and establish connections with V according to the similarity table E T = {(v M , related_to, v j , related_to, v i ) ∣ v j ∈V M , v i ∈V, ED(v i , v j ) < τ M}, where M ∈ {Image, Table}, V M represents a picture modality entity or a table modality entity, m represents a table or a picture, D M represents the table data or picture data in the multimodal knowledge base D, where ED represents an edit distance function, and τ M is a similarity threshold;

[0025] Based on the filtering of the user query Q, combined with the retrieval graph G2, construct a clue graph G3;

[0026]

[0027] Among them, represents a graph construction operator; Q represents a user query, V C represents an entity node in the clue graph; E C represents a node relationship in the clue graph; F DPER represents a two-path entity retrieval; V represents all entity nodes in the clue graph; V I represents a picture entity node in the clue graph; F VLM represents a visual evidence integration process; e represents a relationship in the retrieval graph G2; h(e) represents the head node of the edge e; t(e) represents the tail node of the edge e;

[0028] The two-path entity retrieval includes: lexical filtering and semantic filtering.

[0029] Optionally, the specific steps of the lexical filtering include:

[0030] Use the large language model LLM to extract a set of keywords W = {w1, w2, …, w n} from the user query Q;

[0031] Calculate the edit distance between each keyword w and the name of each node v in the retrieval graph G2;

[0032] Select the top-k nodes with the smallest edit distance from the keywords to obtain the entities after lexical filtering, as follows:

[0033] V lex = Top-k v∈V [min w∈W ED(v name , w)]

[0034] where Top-k represents the first k elements with the smallest edit distance; ED(v name , w) represents the edit distance between each node name and the keyword; v name is the node name; w represents a specific keyword in the set W, and V lex represents the entities after lexical filtering.

[0035] Optionally, the specific steps of the semantic filtering include:

[0036] Use the pre-trained language model BERT to encode the user query Q and the description v of each node v in the retrieval graph G2 desc , v desc represents the description of the node;

[0037] Calculate the semantic cosine similarity between the user query Q and the description v desc : sim cos (BERT(Q), BERT(v desc ));

[0038] Select the top-k nodes with the highest semantic cosine similarity: V sem = Top-k v∈V [sim cos (BERT(Q), BERT(v desc ))], V sem represents the result of semantic filtering.

[0039] Optionally, the specific steps of the visual evidence integration process are as follows:

[0040] For a given user query Q, use the large language model LLM to extract a set of image-related description information D = {d1, d2, …, dn};

[0041] Use the CLIP model to select the top-k images related to each description d from the image nodes retrieved from graph G2, specifically including: calculating the similarity score S(d i , v i for each description d and each image node v j ), and selecting the top-k images with the highest scores; i , v j ) and select the top-k images with the highest scores;

[0042] Input the top-k images selected by the CLIP model into the multimodal large model VLM for in-depth semantic matching.

[0043] Optionally, step S6 includes:

[0044] Dynamically expand and refine the clue graph through cross-modal inference iteration to obtain the final evidence graph G4 as follows:

[0045]

[0046] where represents the graph construction operator; V E represents the entity nodes in the evidence graph; E E represents the edges in the evidence graph; and E img represent enhanced visual evidence and edges, and E table represent table components; E expand represents the graph expansion operator;

[0047] The evidence graph G4 is the retrieval answer result of the user's query;

[0048] The specific steps of visual evidence enhancement include:

[0049] Use the large language model LLM to analyze the user's query Q, identify the image nodes v in the clue graph G3 related to the user's query k , and generate an answer q related to the image for each image node v k , expressed as: k where

[0050]

[0051] where represents the image nodes in the clue graph G3, LLM vis represents generating a visual query, and VLM represents the vision-language large model;

[0052] Dynamically extract the relationships related to the user's query from the table data of the clue graph G3

[0053] Use the large language model LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V related to the user's query req , expressed as:

[0054]

[0055] where represents the set of table entities in the clue graph; t represents a table entity, E table represents the mined table relationships; Join represents the operation of connecting the table node t with the entities extracted by the LLM based on the query Q for the table t;

[0056] E expand represents a graph expansion operator that performs structural expansion through the neighborhood, as follows:

[0057]

[0058] where u represents a one-hop neighbor node of v; N(v) represents retrieving the one-hop neighbors in the graph G2.

[0059] A query-driven multi-modal knowledge graph construction and question answering system, the system includes: a data acquisition module, a data processing module, and a display module;

[0060] The data acquisition module, the data processing module, and the display module are sequentially connected;

[0061] The data acquisition module is used to construct the data required for the pattern graph of the user's query;

[0062] The data processing module is used to construct the pattern graph of the user's query;

[0063] The data processing module is also used to construct a multi-modal knowledge base; through the pattern constraint retrieval method and the query-focused clue refinement method, combined with the pattern graph, construct a clue graph;

[0064] The data processing module is also used to supplement the missing information of the clue graph through the cross-modal reasoning iteration method to obtain the retrieval answer result of the user's query;

[0065] The display module is used to visually display the retrieval answer result

[0066] The beneficial effects brought by the technical solution provided by this application are:

[0067] This application proposes a query-driven multimodal GraphRAG framework that can dynamically construct a local knowledge graph for query semantics. It extracts graph patterns from query semantics to guide knowledge extraction, adopts a multi-path retrieval strategy to accurately locate core knowledge, and temporarily supplements missing multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The following will further illustrate this application in conjunction with the drawings. In the drawings:

[0069] Figure 1 is a step diagram in an embodiment of this application;

[0070] Figure 2 is an algorithm diagram in an embodiment of this application;

[0071] Figure 3 is the first comparison effect diagram in an embodiment of this application;

[0072] Figure 4 is the second comparison effect diagram in an embodiment of this application;

[0073] Figure 5 is a schematic diagram of the structure of an electronic device in an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] In order to have a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the drawings.

[0075] The embodiments of this application provide a method for constructing and answering questions for a multimodal knowledge graph based on query driving.

[0076] Please refer to Figure 1 , Figure 1 which is a step diagram of a method for constructing and answering questions for a multimodal knowledge graph based on query driving in an embodiment of this application, including:

[0077] S1: Construct a schema graph for the user query;

[0078] As an embodiment, dynamic graph schema construction: By analyzing query semantics, entity-relationship patterns are derived to guide knowledge extraction, ensuring focused and selective retrieval.

[0079] S2: Construct a multimodal knowledge base; Through a schema-constrained retrieval method and a query-focusing clue refinement method, combined with the schema graph, a clue graph is constructed;

[0080] As an example, a multi-path filtering strategy is implemented to identify the most relevant information across text, visual, and tabular modalities. Through two progressive filtering steps, relevant information is selectively extracted and integrated from the multi-modal knowledge base. The framework implements a neurally-inspired selective attention mechanism through two progressive filtering stages.

[0081] S3: Through the way of cross-modal reasoning iteration, supplement the missing information in the clue graph to obtain the retrieval answer result of the user query.

[0082] Step S1 further includes:

[0083] Use the large language model LLM to generate a preliminary schema graph;

[0084] Receive the user query Q as the input of the large language model LLM;

[0085] Design a prompt template P pattern Guide the large language model LLM to extract the entity types and relationships of the user query Q, and generate the schema graph of the user query Q: {T, R} = LLM(P pattern (Q));

[0086] As an example, P pattern (Q) = Analyze the following user query 'Q', extract all entity types and relationship types. Output format: Entity types: [type1, type2,...] Relationships: (relationship description, source entity type, target entity type).

[0087] The schema graph is a structured representation of the user query. Define the schema graph G1 as G1 = (T, R), where T = {t1, t2,..., t n} is the set of key entity types; R = {r1, r2,..., r m} is the set of semantic relationship types.

[0088] As an example, given the user query "Which actors have appeared in Marvel movies?", identify the entity types T = actor, movie; the semantic relationship type R = appear in. Construct the schema graph G1 = {actor, appear in, movie}.

[0089] Step S2 includes:

[0090] Under the constraint guidance of the schema graph G1, combine with the multi-modal knowledge base D to construct the retrieval graph G2;

[0091] The retrieval graph is a constrained graph, where represents a resource-lightweight graph construction operator, V represents type filtering, and E represents relationship pruning;

[0092] The retrieval graph G2 is subject to type filtering and relationship pruning;

[0093] Type filtering: V = {v ∈ D | φ type (v) ∈ T}, where v represents an entity and φ type represents mapping an entity to an entity type;

[0094] Relationship pruning: v i represents the i-th entity, v j represents the j-th entity, and r represents the relationship between entity v i and entity v j ;

[0095] Extract entities and relationships that conform to the constraints of the schema graph G1 from the text data D in the multimodal knowledge base D text to construct the retrieval graph G2;

[0096] Entities extracted from D text where d is a temporary variable; P (T, R) represents a hint template; extract As an example, the hint template P

[0097] (T, R) = "Extract entities and relationships that match the specified schema from the following text: Entity type: T Relationship type: R, Text: d. Output format: (entity, entity name, entity type, entity description)(relationship, source entity, target entity, relationship description)". extract (T, R) = "Extract entities and relationships that match the specified schema from the following text: Entity type: T Relationship type: R, Text: d. Output format: (entity, entity name, entity type, entity description)(relationship, source entity, target entity, relationship description)".

[0098] Treat each picture and table in the picture modality data D and the table modality data D in the multimodal knowledge base D image as independent nodes and establish connections with V according to the similarity. E table T M = {(v j , related_to, v i ) | v j ∈ V M , v i ∈ V, ED(v i , v j ) < τ M}, where M ∈ {Image, Table}, V M represents a picture modality entity or a table modality entity, m represents a table or a picture, and D M represents the table data or picture data in the multimodal knowledge base D, where ED represents the edit distance function and τ M is the similarity threshold; is the similarity threshold;

[0099] Based on the filtering of the user query Q, combined with the retrieval graph G2, construct the clue graph G3;

[0100]

[0101] Among them, represents the graph construction operator; Q represents the user query, V C represents the entity nodes in the clue graph; E C represents the node relationships in the clue graph; F DPER represents the dual-path entity retrieval; V represents all the entity nodes in the clue graph; V I represents the picture entity nodes in the clue graph; F VLM represents the visual evidence integration process; e represents the relationship in the retrieval graph G2; h(e) represents the head node of the edge e; t(e) represents the tail node of the edge e;

[0102] As an example, the clue graph is a subgraph of G2, obtained by filtering based on the user query Q, and only retaining the entities and relationships that are highly relevant to the query semantics. Dual-path entity retrieval: A dual-path entity retrieval (DPER) method that combines lexical filtering and semantic filtering.

[0103] The dual-path entity retrieval includes: lexical filtering and semantic filtering.

[0104] The specific steps of the lexical filtering include:

[0105] Use the large language model LLM to extract the keyword set W = {w1, w2,..., w n} from the user query Q;

[0106] Calculate the edit distance between each keyword w and the name of each node v in the retrieval graph G2;

[0107] Select the top-k nodes with the smallest edit distance from the keywords to obtain the entities after lexical filtering, as follows:

[0108] V lex = Top-k v∈V [min w∈W ED(v name , w)],

[0109] where Top-k represents the first k elements with the smallest edit distance; ED(v name , w) represents the edit distance between each node name and the keyword; v name node name; w represents a specific keyword in the set W, and V lex represents the entities after lexical filtering.

[0110] The specific steps of the semantic filtering include:

[0111] Use the pre-trained language model BERT to encode the user query Q and the description v of each node v in the retrieval graph G2 desc , v desc representing the description of the node;

[0112] Calculate the semantic cosine similarity between the user query Q and the description v desc : sim cos (BERT(Q), BERT(v desc ));

[0113] Select the top-k nodes with the highest semantic cosine similarity: V sem = Top-k v∈V [sim cos (BERT(Q), BERT(v desc ))], V sem representing the result of semantic filtering.

[0114] The specific steps of the visual evidence integration process are as follows:

[0115] As an example, a multi-stage image matching strategy is adopted to combine visual information with the content of the user query to improve the matching accuracy of image nodes.

[0116] For a given user query Q, use the large language model LLM to extract a set of description information D = {d1, d2,..., d n};

[0117] As an example, for the user query "In which movie did Ben Piazza make his first appearance: 'Nightwing' or the movie with half a woman's face shown on the poster?", the extracted description might be "half a woman's face shown on the poster".

[0118] Use the CLIP model to select the top-k images related to each description d i from the image nodes of the retrieval graph G2, specifically including: calculating the similarity score S(d i with each image node v j ) and selecting the top-k images with the highest scores; i , v j )

[0119] Input the top-k images selected by the CLIP model into the multimodal large model VLM for in-depth semantic matching.

[0120] Step S6 includes:

[0121] As an example, in this stage, the clue graph is dynamically extended and refined through iterative loops to form the final evidence graph (G4). This process simulates the iterative refinement process of temporary mental models in human cognition, and improves the cognitive framework by continuously accumulating evidence.

[0122] Through the way of cross-modal reasoning iteration, the clue graph is dynamically extended and refined to obtain the final evidence graph G4, as follows:

[0123]

[0124] Among them represents the graph construction operator; V E represents the entity nodes in the evidence graph; E E represents the edges in the evidence graph; and E img represent enhanced visual evidence and edges, and E table represent table components; E expand represents the graph expansion operator;

[0125] The evidence graph G4 is the retrieval answer result of the user query;

[0126] The specific steps of visual evidence enhancement include:

[0127] Use the large language model LLM to analyze the user query Q, identify the image nodes v k in the clue graph G3 that are relevant to the user query, and generate an answer q k related to the image for each image node v k , expressed as:

[0128]

[0129] Among them represents the image nodes in the clue graph G3, LLM vis represents generating a visual query, and VLM represents a vision-language large model;

[0130] Dynamically extract the relationships related to the user query from the tabular data in the clue graph G3

[0131] Use the large language model LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V req , expressed as:

[0132]

[0133] Among them represents the set of table entities in the clue graph; t represents the table entity, E tablerepresents the mined table relationships; Join represents the operation of joining the table node t with the entities extracted by the LLM based on the query Q for the analysis of the table t;

[0134] E expand represents the graph expansion operator, which expands the structure through the neighborhood as follows:

[0135]

[0136] where u represents the one-hop neighbor node of v; N(v) represents retrieving the one-hop neighbors in the graph G2.

[0137] As an example, the evidence graph expands the knowledge content by analyzing the images and tables in the clue graph, so as to be able to respond to user queries more comprehensively. For example, for the query "Who are the main actors in Marvel movies?", the evidence graph may include movie posters and actor information. However, if the actor photos and names on the posters are not further processed, they may still be static information and cannot be effectively utilized. In the process of constructing the final answer graph, analyzing the images and tables can extract hidden information from multi-modal data, so as to respond to the query more completely.

[0138] As an example, the construction of the evidence graph follows the evaluation-enhancement-expansion cycle, as Figure 2 shown, the enhancement operator in step 6. Visual evidence enhancement: Use a vision-language large model (VLM) to enhance the information of the image nodes as needed, and extract relevant visual details from the images to overcome the limitation of the lack of fine-grained visual information in the current knowledge graph. Specifically, use the LLM to analyze the query Q and identify the image nodes v in G3 related to the query k , and generate an image-related answer q for each image node v k . Table relationship mining: Although the table nodes contain rich relationship information, statically pre-modeling all table relationships may lead to an overly complex graph structure. Therefore, it is chosen to dynamically extract the relationships related to the query from the table data according to the specific requirements of the query during the inference stage. Specifically, use the LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V k that may be related to the query. req .

[0139] As an example, as Figure 2 shown, the graph expansion operator in step 7: Expand the structure through the neighborhood: This operation allows the system to dynamically expand the knowledge graph based on the existing evidence and discover potential relevant information.

[0140] To evaluate the effectiveness, experiments were conducted on two multimodal multi-hop question-answering datasets, MultimodalQA (Gupta et al., 2018) and WebQA (Chang et al., 2022). MultimodalQA is evaluated using EM and F1 metrics, and WebQA is evaluated using QA-FL, QA-ACC, and QA-F1 metrics. As Figure 3 and Figure 4 shown, the method of this application significantly outperforms supervised and unsupervised baseline methods on both the MultimodalQA and WebQA datasets. On the Multi-modalQA dataset, our method achieved an F1 value of 68.0% and an EM value of 60.3% without using labeled data, outperforming the supervised baseline SKURG (F1 value of 64.0% and EM value of 59.8%) and the unsupervised method.

[0141] Please refer to Figure 5 , Figure 5 which is a module diagram of a query-driven multimodal knowledge graph construction and question-answering system. The system includes: a data acquisition module, a data processing module, and a display module;

[0142] The data acquisition module, the data processing module, and the display module are sequentially connected in order;

[0143] The data acquisition module is used to construct the data required for the schema graph of the user query;

[0144] The data processing module is used to construct the schema graph of the user query;

[0145] The data processing module is also used to construct a multimodal knowledge base; through a schema constraint retrieval method and a query focus clue refinement method, combined with the schema graph, a clue graph is constructed;

[0146] The data processing module is also used to supplement the missing information of the clue graph through cross-modal inference iteration to obtain the retrieval answer result of the user query;

[0147] The display module is used to visually display the retrieval answer result

[0148] The above are only exemplary embodiments of the present disclosure and should not be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure.

[0149] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include well-known knowledge or conventional technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A query-driven multi-modal knowledge graph construction and question answering method, characterized in that The method includes the following steps: S1: Construct a schema graph for the user query; S2: Construct a multimodal knowledge base; Construct a clue graph by combining the schema graph through a schema-constrained retrieval method and a query-focused clue refinement method; S3: Supplement the missing information in the clue graph through cross-modal reasoning iteration to obtain the retrieval answer result of the user query.

2. The multimodal knowledge graph construction and question answering method based on query driving according to claim 1, characterized in that, Step S1 further includes: Use a large language model LLM to generate a preliminary schema graph; Receive the user query Q as the input of the large language model LLM; Design Prompt Template P pattern Guide the large language model LLM to extract the entity types and relationships of the user query Q, and generate a schema graph of the user query Q: {T, R} = LLM(P pattern (Q)); The schema graph is a structured representation of the user's query. Define the schema graph G1 as G1 = (T, R), where T = {t1, t2, …, t n} is a set of key entity types; R = {r1, r2, …, r m} is a set of semantic relation types.

3. A query-driven multi-modal knowledge graph construction and question answering method according to claim 2, characterized in that Step S2 includes: Construct a retrieval graph G2 by combining the multimodal knowledge base D under the constraint guidance of the schema graph G1; Retrieval graph is a constrained graph, where represents a resource lightweight graph construction operator, V represents the entities after type filtering, and E represents the entities after relationship pruning; The retrieval graph G2 is subject to type filtering and relation pruning; Type filtering: V = {v ∈ D | φ type (v) ∈ T}, where v represents an entity, and φ type represents mapping an entity to an entity type; Relationship pruning: v i represents the i-th entity, v j represents the j-th entity, and r represents the relationship between entity v i and entity v j therebetween; Extract entities and relationships that conform to the constraints of the schema graph G1 from the text data D in the multimodal knowledge base D text and construct a retrieval graph G2; Based on the filtering of the user query Q, construct a clue graph G3 by combining the retrieval graph G2; Among them, represents a graph construction operator; Q represents a user query, and V C represents an entity node in the clue graph; E C represents the node relationship in the clue graph; F DPER represents two-path entity retrieval; V represents all entity nodes in the clue graph; V I represents the picture entity node in the clue graph; F VLM represents the visual evidence integration process; e represents the relationship in the retrieval graph G2; h(e) represents the head node of edge e; t(e) represents the tail node of edge e; The two-path entity retrieval includes: lexical filtering and semantic filtering.

4. A query-driven multi-modal knowledge graph construction and question answering method according to claim 3, characterized in that The specific steps of the lexical filtering include: Use the large language model LLM to extract a set of keywords W = {w1, w2, …, w n} from the user query Q; Calculate the edit distance between each keyword w and the name of each node v in the retrieval graph G2; Select the top-k nodes with the smallest edit distance from the keyword to obtain the entities after lexical filtering, as follows: V lex = Top-k v∈V [min w∈W ED(v name , w)], Among them, Top-k represents the k smallest elements in terms of the edit distance; ED(v name , w) represents the edit distance between each node name and the keyword; v name is the node name; w represents a specific keyword in the set W, and V lex represents the entity after lexical filtering.

5. The multimodal knowledge graph construction and question answering method based on query driving according to claim 3, characterized in that The specific steps of the semantic filtering include: Encode the user query Q and the description v of each node v in the retrieval graph G2 using the pre-trained language model BERT desc , v desc represents the description of the node; Calculate the semantic cosine similarity between the user query Q and the description v desc : sim cos (BERT(Q), BERT(v desc )); Select the top-k nodes with the highest semantic cosine similarity: V sem = Top-k v∈V [sim cos (BERT(Q), BERT(v desc ))], V sem Indicates the result of semantic filtering.

6. The multimodal knowledge graph construction and question answering method based on query driving according to claim 3, characterized in that The specific steps of the visual evidence integration process are as follows: For a given user query Q, use a large language model (LLM) to extract a set of image-related descriptive information D = {d1, d2, …, d n}; Use the CLIP model to select the top-k images related to each description d from the image nodes retrieved from graph G2, specifically including: calculating the similarity score S(d i , related to each description d i and each image node v j as S(d i , v j ), and selecting the top-k images with the highest scores; Input the top-k images selected by the CLIP model into the multimodal large model VLM for in-depth semantic matching.

7. The multimodal knowledge graph construction and question answering method based on query drive according to claim 3, wherein Step S6 includes: Dynamically expand and improve the clue graph through cross-modal reasoning iteration to obtain the final evidence graph G4, as follows: Among them represents the graph construction operator; V E represents the entity node in the evidence graph; E E represents the edge in the evidence graph; and E img represent the enhanced visual evidence and edge, and E table represent the table component; E expand represents the graph expansion operator; The evidence graph G4 is the retrieval answer result of the user query; The specific steps of visual evidence enhancement include: Use the large language model LLM to analyze the user query Q, identify the image nodes v in the clue graph G3 that are relevant to the user query k , and for each image node v k generate an image-related answer q k , expressed as: Among them represents an image node in the clue graph G3, and LLM vis represents generating a visual query, and VLM represents a vision-language large model; Dynamically extract the relationships related to the user query from the tabular data of the clue graph G3 Use the large language model LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V related to the user's query req , expressed as: Among them represents the set of table entities in the clue graph; t represents a table entity, E table represents the mined table relationships; Join represents the operation of connecting the table node t with the entities extracted by the LLM after analyzing the table t based on the query Q; E expand Represents a graph expansion operator that expands the structure through the neighborhood as follows: where u represents that it is a one-hop neighbor node of v; N(v) represents the single-hop neighbors in the retrieval graph G2.

8. A query-driven multimodal knowledge graph construction and question answering system for implementing a query-driven multimodal knowledge graph construction and question answering method as described in any one of claims 1-7, characterized in that, The system includes: a data acquisition module, a data processing module, and a display module; The data acquisition module, the data processing module, and the display module are connected in sequence; The data acquisition module is used for the data required to construct the schema graph of the user query; The data processing module is used to construct the schema graph of the user query; The data processing module is also used to construct a multimodal knowledge base; construct a clue graph by combining the schema graph through a schema-constrained retrieval method and a query-focused clue refinement method; The data processing module is also used to supplement the missing information in the clue graph through cross-modal reasoning iteration to obtain the retrieval answer result of the user query; The display module is used to visually display the retrieval answer result.

Images