Query-driven multimodal knowledge graph construction and question-answering method and system

Abstract

The present application provides a query-driven multimodal knowledge graph construction and question-answering method and system, which relates to the field of knowledge graphs. The method includes: constructing a pattern graph of user queries; constructing a multimodal knowledge base; combining the pattern graph with a pattern constraint retrieval method and a query focus clue refinement method to construct a clue graph; supplementing the missing information of the clue graph through cross-modal reasoning iteration to obtain the retrieval answer result of the user query. The present application constructs a query-driven multimodal framework that can dynamically construct a local knowledge graph of user queries, enabling fast and accurate retrieval of user query questions.

Description

Technical Field

[0001] This application relates to the field of knowledge graphs, and in particular to a query-driven multimodal knowledge graph construction and question-answering method and system. Background Art

[0002] The increasing popularity of large language models (LLMs) in complex reasoning tasks makes their interpretability and reliability crucial. To this end, recent advances include retrieval-augmented generation (RAG) and knowledge graph-augmented RAG (GraphRAG), however, they are both limited by static knowledge bases and inefficient multimodal data integration.

[0003] Combining knowledge graphs (KGs) with large language models (LLMs) through retrieval-augmented generation (RAG), also known as GraphRAG, has become a promising paradigm for improving factual accuracy and reasoning capabilities (Panet al., 2024; Edge et al., 2024). Traditional RAG systems utilize unstructured text retrieval, while recent GraphRAG methods attempt to combine structured knowledge graphs (KGs) with LLMs to capture complex entity relationships and enable multi-hop reasoning (Peng et al., 2024).

[0004] To achieve this, existing methods typically construct knowledge graphs in a bottom-up, information-driven manner (Cohen et al., 2023). This approach first identifies available data sources and then integrates them into triples through entity extraction, relationship extraction, and knowledge fusion. However, this can introduce additional complexity and runtime overhead. Specifically, these methods lack awareness of user needs during the knowledge graph construction process (Cohen et al., 2023), resulting in the generated graphs being either too detailed and computationally expensive, or too coarse and missing key semantic details.

[0005] For example, in multimodal settings, parsing complex visual information without query context either leads to over-extraction of details or incomplete knowledge integration. This rigidity is particularly problematic for queries requiring cross-modal understanding, as existing frameworks struggle to effectively combine textual, visual, and tabular data into a unified knowledge representation. Summary of the Invention

[0006] The purpose of the present invention is to provide a query-driven multimodal knowledge graph construction and question-answering method and system in order to solve the technical problem that existing frameworks find it difficult to effectively combine text, visual and tabular data into a unified knowledge representation.

[0007] The above-mentioned purpose of this application is achieved through the following technical solutions:

[0008] S1: Construct a pattern graph of user queries;

[0009] S2: Build a multimodal knowledge base; build a clue graph by combining the pattern graph with the pattern constraint retrieval method and the query focus clue refinement method;

[0010] S3: Through cross-modal reasoning iteration, the missing information of the clue graph is supplemented to obtain the retrieval answer results of the user query.

[0011] Optionally, step S1 further includes:

[0012] Use large language model LLM to generate preliminary pattern graph;

[0013] Receive user query Q as input to the large language model LLM;

[0014] Design Tips Template P pattern Guide the large language model LLM to extract the entity types and relations of the user query Q and generate a pattern graph of the user query Q: {T, R} = LLM(P pattern (Q));

[0015] The pattern graph is a structured representation of the user query, and the pattern graph G1 is defined as G1=(T,R), where T={t1,t2,…,t n} is a set of key entity types; R = {r1, r2, ..., r m} is a collection of semantic relationship types.

[0016] Optionally, step S2 includes:

[0017] Through the constraint guidance of the pattern graph G1 and combined with the multimodal knowledge base D, the retrieval graph G2 is constructed;

[0018] Search Graph is a constrained graph where Represents a resource-lightweight graph construction operator, V represents type filtering, and E represents relationship pruning;

[0019] The retrieval graph G2 is subject to type filtering and relation pruning;

[0020] Type filtering: V = {v∈D|φ type (v)∈T}, v represents the entity, φ type Indicates mapping an entity to an entity type;

[0021] Relationship pruning: v i represents the i-th entity, v j represents the jth entity, r represents entity v i With entity vj the relationship between;

[0022] From the text data D in the multimodal knowledge base D text Extract entities and relationships that meet the constraints of the pattern graph G1 and construct the retrieval graph G2;

[0023] From D text Entities extracted from Where d represents a temporary variable; P extract (T, R) indicates the prompt template;

[0024] The image modality data D in the multimodal knowledge base D image and the table modal data D table Each image and table in the table is treated as an independent node and is compared with V according to its similarity. T Establish a connection. M ={(v j ,related_to,v i )|v j ∈V M ,v i ∈V,ED(v i ,v j )<τ M}, where M∈{Image,Table}, V M Indicates a picture modal entity or a table modal entity, m indicates a table or a picture, D M represents the tabular data or image data in the multimodal knowledge base D, where ED represents the edit distance function, τ M is the similarity threshold;

[0025] Based on the filtering of user query Q, combined with the retrieval graph G2, a clue graph G3 is constructed;

[0026]

[0027] in, represents the graph construction operator; Q represents the user query, V C Represents the entity node in the clue graph; E C Indicates the node relationship in the clue graph; F DPER Represents dual-path entity retrieval; V represents all entity nodes in the clue graph; V I Represents the picture entity node in the clue graph; F VLM represents the visual evidence integration process; e represents the relationship in the retrieval graph G2; h(e) represents the head node of edge e; t(e) represents the tail node of edge e;

[0028] Dual-path entity retrieval includes: lexical filtering and semantic filtering.

[0029] Optionally, the specific steps of the vocabulary filtering include:

[0030] Use the large language model LLM to extract the keyword set W = {w1,w2,…,w n};

[0031] Calculate the edit distance between each keyword w and each node v name in the search graph G2;

[0032] Select the top-k nodes with the smallest edit distance to the keyword and get the entities after vocabulary filtering, as follows:

[0033] V lex =Top-k v∈V [min w∈W ED(v name ,w)],

[0034] Top-k represents the k smallest elements in edit distance; ED(v name ,w) represents the edit distance between each node name and the keyword; v name Node name; w represents a specific keyword in the set W, V lex Represents an entity after vocabulary filtering.

[0035] Optionally, the specific steps of the semantic filtering include:

[0036] Use the pre-trained language model BERT to encode the user query Q and retrieve the description v of each node v in the graph G2 desc , v desc Represents the description of the node;

[0037] Calculate user query Q and description v desc Semantic cosine similarity: sim cos (BERT(Q),BERT(v desc ));

[0038] Select the top-k nodes with the highest semantic cosine similarity: V sem =Top-k v∈V [sim cos (BERT(Q),BERT(v desc ))],V sem Indicates the result of semantic filtering.

[0039] Optionally, the specific steps of the visual evidence integration process are as follows:

[0040] For a given user query Q, a large language model LLM is used to extract a set of image-related description information D = {d1, d2, ..., dn};

[0041] Use the CLIP model to select image nodes from the retrieval graph G2 that are associated with each description d i Related Top-k images, specifically including: calculating each description d i With each image node v j The similarity score S(d i ,v j ) and select the top-k images with the highest scores;

[0042] The top-k images selected by the CLIP model are input into the multimodal large model VLM for deep semantic matching.

[0043] Optionally, step S6 includes:

[0044] Through cross-modal reasoning iteration, the clue graph is dynamically expanded and improved to obtain the final evidence graph G4, as shown below:

[0045]

[0046] in represents the graph construction operator; V E Represents the entity node in the evidence graph; E E represents the edges in the evidence graph; and E img represents enhanced visual evidence and edges, and E table Represents a table component; E expand Represents a graph expansion operator;

[0047] Evidence graph G4 is the retrieval answer result of the user query;

[0048] The specific steps of visual evidence enhancement include:

[0049] Use the large language model LLM to analyze the user query Q and identify the image nodes v related to the user query in the clue graph G3 k , and for each image node v k Generate an answer q associated with the image k , expressed as:

[0050]

[0051] in Represents the image node in the clue graph G3, LLM vis represents generating visual queries, and VLM stands for Vision-Language Large Model;

[0052] Dynamically extract relationships related to user queries from the table data in the clue graph G3

[0053] Use the large language model LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V related to the user query req , expressed as:

[0054]

[0055] in Represents the set of table entities in the clue graph; t represents the table entity, E table Represents the mined table relationship; Join represents the connection operation between table node t and the entity extracted by LLM after analyzing table t based on query Q;

[0056] E expand Represents the graph expansion operator, which expands the structure through the neighborhood, as follows:

[0057]

[0058] Where u represents a one-hop neighbor node of v; N(v) represents a one-hop neighbor in the retrieval graph G2.

[0059] A query-driven multimodal knowledge graph construction and question-answering system, comprising: a data acquisition module, a data processing module, and a display module;

[0060] The data acquisition module, data processing module and display module are connected in sequence;

[0061] The data acquisition module is used to construct the data required for the pattern graph of the user query;

[0062] The data processing module is used to construct a pattern graph of user queries;

[0063] The data processing module is also used to build a multimodal knowledge base; through the pattern constraint retrieval method and the query focus clue refinement method, combined with the pattern graph, a clue graph is constructed;

[0064] The data processing module is also used to supplement the missing information of the clue graph through cross-modal reasoning iteration to obtain the retrieval answer results of the user query;

[0065] The display module is used to visualize the search answer results

[0066] The beneficial effects of the technical solution provided by this application are:

[0067] This application proposes a query-driven multimodal GraphRAG framework that dynamically constructs a local knowledge graph tailored to the query semantics. It extracts graph patterns from the query semantics to guide knowledge extraction, employs a multi-path retrieval strategy to precisely locate core knowledge, and temporarily supplements missing multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present application will be further described below with reference to the accompanying drawings and embodiments, in which:

[0069] Figure 1 is a step diagram in an embodiment of the present application;

[0070] Figure 2 It is an algorithm diagram in the embodiment of the present application;

[0071] Figure 3 This is the first comparison effect diagram in the embodiment of the present application;

[0072] Figure 4 This is the second comparison effect diagram in the embodiment of the present application;

[0073] Figure 5 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to have a clearer understanding of the technical features, purposes and effects of this application, the specific implementation methods of this application are now described in detail with reference to the accompanying drawings.

[0075] The embodiments of the present application provide a query-driven multimodal knowledge graph construction and question-answering method.

[0076] Please refer to Figure 1 , Figure 1 This is a step diagram of a query-driven multimodal knowledge graph construction and question-answering method in an embodiment of the present application, including:

[0077] S1: Construct a pattern graph of user queries;

[0078] As an embodiment, dynamic graph model construction: by analyzing query semantics, entity-relationship models are derived to guide knowledge extraction and ensure focused and selective retrieval.

[0079] S2: Build a multimodal knowledge base; build a clue graph by combining the pattern graph with the pattern constraint retrieval method and the query focus clue refinement method;

[0080] As an example, a multi-path filtering strategy is implemented to identify the most relevant information across textual, visual, and tabular modalities. Relevant information is selectively extracted and integrated from the multimodal knowledge base through two progressive filtering steps. The framework implements a neurocognitive-inspired selective attention mechanism through two progressive filtering stages.

[0081] S3: Through cross-modal reasoning iteration, the missing information of the clue graph is supplemented to obtain the retrieval answer results of the user query.

[0082] Step S1 further includes:

[0083] Use large language model LLM to generate preliminary pattern graph;

[0084] Receive user query Q as input to the large language model LLM;

[0085] Design Tips Template P pattern Guide the large language model LLM to extract the entity types and relations of the user query Q and generate a pattern graph of the user query Q: {T, R} = LLM(P pattern (Q));

[0086] As an example, P pattern (Q)=Analyze the following user query 'Q' and extract all entity types and relationship types. Output format: Entity type: [Type 1, Type 2, ...] Relationship: (Relationship description, source entity type, target entity type).

[0087] The pattern graph is a structured representation of the user query, and the pattern graph G1 is defined as G1=(T,R), where T={t1,t2,…,t n} is a set of key entity types; R = {r1, r2, ..., r m} is a collection of semantic relationship types.

[0088] As an example, given a user query "Which actors have appeared in Marvel movies?", entity types T = actor, movie and semantic relationship type R = appeared are identified. A pattern graph G1 = {actor, appeared, movie} is constructed.

[0089] Step S2 includes:

[0090] Through the constraint guidance of the pattern graph G1 and combined with the multimodal knowledge base D, the retrieval graph G2 is constructed;

[0091] Search Graph is a constrained graph where Represents a resource-lightweight graph construction operator, V represents type filtering, and E represents relationship pruning;

[0092] The retrieval graph G2 is subject to type filtering and relation pruning;

[0093] Type filtering: V = {v∈D|φ type (v)∈T}, v represents the entity, φ type Indicates mapping an entity to an entity type;

[0094] Relationship pruning: v i represents the i-th entity, v j represents the jth entity, r represents entity v i With entity v j the relationship between;

[0095] From the text data D in the multimodal knowledge base D text Extract entities and relationships that meet the constraints of the pattern graph G1 and construct the retrieval graph G2;

[0096] From D text Entities extracted from Where d represents a temporary variable; P extract (T,R) indicates the prompt template;

[0097] As an embodiment, the prompt template P extract (T,R) = "Extract entities and relationships that match the specified pattern from the following text: entity type: T, relationship type: R, text: d. Output format: (entity, entity name, entity type, entity description) (relationship, source entity, target entity, relationship description)".

[0098] The image modality data D in the multimodal knowledge base D image and the table modal data D table Each image and table in the table is treated as an independent node and is compared with V according to its similarity. T Establish a connection. M ={(v j ,related_to,v i )|v j ∈V M ,v i ∈V,ED(v i ,v j )<τ M}, where M∈{Image,Table}, V M Indicates a picture modal entity or a table modal entity, m indicates a table or a picture, D M represents the tabular data or image data in the multimodal knowledge base D, where ED represents the edit distance function, τ M is the similarity threshold;

[0099] Based on the filtering of user query Q, combined with the retrieval graph G2, a clue graph G3 is constructed;

[0100]

[0101] in, represents the graph construction operator; Q represents the user query, V C Represents the entity node in the clue graph; E C Indicates the node relationship in the clue graph; F DPER Represents dual-path entity retrieval; V represents all entity nodes in the clue graph; V I Represents the picture entity node in the clue graph; F VLM represents the visual evidence integration process; e represents the relationship in the retrieval graph G2; h(e) represents the head node of edge e; t(e) represents the tail node of edge e;

[0102] As an example, the clue graph is a subgraph of G2, obtained by filtering based on the user query Q, retaining only entities and relations that are highly semantically relevant to the query. Dual-Path Entity Retrieval: The Dual-Path Entity Retrieval (DPER) method combines lexical filtering and semantic filtering.

[0103] Dual-path entity retrieval includes: lexical filtering and semantic filtering.

[0104] The specific steps of the vocabulary filtering include:

[0105] Use the large language model LLM to extract the keyword set W = {w1,w2,…,w n};

[0106] Calculate the edit distance between each keyword w and each node v name in the search graph G2;

[0107] Select the top-k nodes with the smallest edit distance to the keyword and get the entities after vocabulary filtering, as follows:

[0108] V lex =Top-k v∈V [min w∈W ED(v name ,w)],

[0109] Top-k represents the k smallest elements in edit distance; ED(v name ,w) represents the edit distance between each node name and the keyword; v name Node name; w represents a specific keyword in the set W, V lex Represents an entity after vocabulary filtering.

[0110] The specific steps of the semantic filtering include:

[0111] Use the pre-trained language model BERT to encode the user query Q and retrieve the description v of each node v in the graph G2 desc , v desc Represents the description of the node;

[0112] Calculate user query Q and description v desc Semantic cosine similarity: sim cos (BERT(Q),BERT(v desc ));

[0113] Select the top-k nodes with the highest semantic cosine similarity: V sem =Top-k v∈V [sim cos (BERT(Q),BERT(v desc ))],V sem Indicates the result of semantic filtering.

[0114] The specific steps of the visual evidence integration process are as follows:

[0115] As an embodiment, a multi-stage image matching strategy is adopted to combine visual information with the content of the user query to improve the matching accuracy of image nodes.

[0116] For a given user query Q, a large language model LLM is used to extract a set of image-related description information D = {d1, d2, ..., d n};

[0117] As an example, for example, for the user query "In which movie did Ben Piazza make his debut: Nightwing or the movie where half a woman's face is shown on the poster?", the extracted description might be "half a woman's face is shown on the poster."

[0118] Use the CLIP model to select image nodes from the retrieval graph G2 that are associated with each description d i Related Top-k images, specifically including: calculating each description d i With each image node v j The similarity score S(d i ,v j ) and select the top-k images with the highest scores;

[0119] The top-k images selected by the CLIP model are input into the multimodal large model VLM for deep semantic matching.

[0120] Step S6 includes:

[0121] As an example, this stage dynamically expands and refines the clue graph through a cyclical and iterative process to form the final evidence graph (G4). This process simulates the iterative refinement of temporary mental models in human cognition, which continuously accumulates evidence to improve the cognitive framework.

[0122] Through cross-modal reasoning iteration, the clue graph is dynamically expanded and improved to obtain the final evidence graph G4, as shown below:

[0123]

[0124] in represents the graph construction operator; V E Represents the entity node in the evidence graph; E E represents the edges in the evidence graph; and E img represents enhanced visual evidence and edges, and E table Represents a table component; E expand Represents a graph expansion operator;

[0125] Evidence graph G4 is the retrieval answer result of the user query;

[0126] The specific steps of visual evidence enhancement include:

[0127] Use the large language model LLM to analyze the user query Q and identify the image nodes v related to the user query in the clue graph G3 k , and for each image node v k Generate an answer q associated with the image k , expressed as:

[0128]

[0129] in Represents the image node in the clue graph G3, LLM vis represents generating visual queries, and VLM stands for Vision-Language Large Model;

[0130] Dynamically extract relationships related to user queries from the table data in the clue graph G3

[0131] Use the large language model LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V related to the user query req , expressed as:

[0132]

[0133] in Represents the set of table entities in the clue graph; t represents the table entity, E tableRepresents the mined table relationship; Join represents the connection operation between table node t and the entity extracted by LLM after analyzing table t based on query Q;

[0134] E expand Represents the graph expansion operator, which expands the structure through the neighborhood, as follows:

[0135]

[0136] Where u represents a one-hop neighbor node of v; N(v) represents a one-hop neighbor in the retrieval graph G2.

[0137] As an example, the evidence graph expands the knowledge content by analyzing the images and tables in the clue graph, enabling a more comprehensive response to user queries. For example, for the query "Who are the main actors in Marvel movies?", the evidence graph might include movie posters and actor information. However, without further processing, the actor photos and names on the posters may remain static information and cannot be effectively utilized. In the process of constructing the final answer graph, analyzing images and tables can extract hidden information from multimodal data, thereby more completely responding to the query.

[0138] As an example, the construction of the evidence graph follows an evaluation-enhancement-expansion cycle, such as Figure 2 As shown in Figure 6, the enhancement operator in step 6. Visual Evidence Enhancement: A visual-language large model (VLM) is used to enhance the information of image nodes on demand and extract relevant visual details from the image to overcome the limitation of the lack of fine-grained visual information in the current graph. Specifically, LLM is used to analyze the query Q and identify the image nodes v in G3 that are relevant to the query. k , and for each image node v k Generate an answer q associated with the image k Table Relationship Mining: Although table nodes contain rich relational information, statically pre-modeling all table relationships may lead to an overly complex graph structure. Therefore, we choose to dynamically extract query-related relationships from table data according to the specific needs of the query during the reasoning phase. Specifically, we use LLM to analyze the content of the table nodes in the clue graph G3 and identify a set of candidate entities V that may be relevant to the query. req .

[0139] As an example, Figure 2 As shown, the graph expansion operator in step 7: Structural expansion through neighborhood: This operation allows the system to dynamically expand the knowledge graph based on existing evidence and discover potentially relevant information.

[0140] To evaluate the effectiveness, experiments were conducted on two multimodal multi-hop question answering datasets, MultimodalQA (Gupta et al., 3382018) and WebQA (Chang et al., 2022). MultimodalQA was evaluated using EM and F1 metrics, while WebQA was evaluated using QA-FL, QA-ACC, and QA-F1 metrics. Figure 3 and Figure 4 As shown in the results, our method significantly outperforms both supervised and unsupervised baseline methods on both MultimodalQA and WebQA datasets. On the Multi-modalQA dataset, our method achieves an F1 score of 68.0% and an EM score of 60.3% without using labeled data, outperforming the supervised baseline SKURG (F1 score of 64.0% and EM score of 59.8%) and the unsupervised method.

[0141] Please refer to Figure 5 , Figure 5 It is a module diagram of query-driven multimodal knowledge graph construction and question-answering system. The system includes: data acquisition module, data processing module and display module;

[0142] The data acquisition module, data processing module and display module are connected in sequence;

[0143] The data acquisition module is used to construct the data required for the pattern graph of the user query;

[0144] The data processing module is used to construct a pattern graph of user queries;

[0145] The data processing module is also used to build a multimodal knowledge base; through the pattern constraint retrieval method and the query focus clue refinement method, combined with the pattern graph, a clue graph is constructed;

[0146] The data processing module is also used to supplement the missing information of the clue graph through cross-modal reasoning iteration to obtain the retrieval answer results of the user query;

[0147] The display module is used to visualize the search answer results

[0148] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. In other words, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure.

[0149] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not described herein. The description and examples are to be considered as exemplary only, and the scope and spirit of the present disclosure are to be defined by the claims.

Claims

1. A query-driven multimodal knowledge graph construction and question-answering method, characterized by: The method comprises the following steps: S1: Construct a pattern graph of user queries; Step S1 further includes: Use large language model LLM to generate preliminary pattern graph; Receive user queries As input to the Large Language Model (LLM); Design Tips Template Guide the Large Language Model (LLM) to extract user queries Entity types and relationships, generating user queries Schematic diagram: ; The pattern graph is a structured representation of the user query, defining the pattern graph for ,in, is a collection of key entity types; is a set of semantic relationship types; S2: Build a multimodal knowledge base; build a clue graph by combining the pattern graph with the pattern constraint retrieval method and the query focus clue refinement method; Step S2 includes: Through the pattern diagram Constraint guidance, combined with multimodal knowledge base , build a search graph ; Search Graph is a constrained graph where Represents a resource-lightweight graph construction operator, Represents an entity after type filtering, Represents the entity after relationship pruning; Search Graph Subject to type filtering and relationship pruning; Type Filter: , Represents an entity, Indicates mapping an entity to an entity type; Relationship pruning: , represents the i-th entity, represents the jth entity, Representing an entity With entity the relationship between; From the multimodal knowledge base Text data in Extract the pattern diagram Constrained entities and relationships, building a retrieval graph ; Based on user query Filtering, combined with retrieval graph , build a clue map ; ， ， in, Represents the graph construction operator; Represents a user query, Represents an entity node in the clue graph; Represents the node relationship in the clue graph; Indicates dual-path entity retrieval; Represents all entity nodes in the clue graph; Represents the picture entity node in the clue graph; represents the visual evidence integration process; Represents a retrieval graph the relationship between Represents an edge The head node of Represents an edge The tail node of Dual-path entity retrieval includes: lexical filtering and semantic filtering; S3: Through cross-modal reasoning iteration, the missing information of the clue graph is supplemented to obtain the retrieval answer results of the user query.

2. The query-driven multimodal knowledge graph construction and question-answering method according to claim 1, characterized in that: The specific steps of the vocabulary filtering include: Use Large Language Model (LLM) to query from user Extract keyword set ; Count each keyword and retrieval graph Each node in edit distance between names; Select the one with the smallest edit distance to the keyword nodes, and get the entities after vocabulary filtering, as follows: ， in Represents the k smallest elements in edit distance; Indicates the edit distance between each node name and the keyword; Node name; Representing a collection A specific keyword in Represents an entity after vocabulary filtering.

3. The query-driven multimodal knowledge graph construction and question-answering method according to claim 1, characterized in that: The specific steps of the semantic filtering include: Encode user queries using the pre-trained language model BERT and retrieval graph Each node in Description , Represents the description of the node; Counting user queries and description Semantic cosine similarity: ; Select the one with the highest semantic cosine similarity Nodes: , Indicates the result of semantic filtering.

4. The query-driven multimodal knowledge graph construction and question-answering method according to claim 1, characterized in that: The specific steps of the visual evidence integration process are as follows: For a given user query , using a large language model LLM to extract a set of image-related description information ; Using CLIP model to retrieve graph The image node is selected with each description Related images, specifically including: calculating each description With each image node Similarity score , and select the one with the highest score images; Select the CLIP model The images are input into the multimodal large model VLM for deep semantic matching.

5. The query-driven multimodal knowledge graph construction and question-answering method according to claim 1, characterized in that: Step S6 includes: Through cross-modal reasoning iteration, the clue graph is dynamically expanded and improved to obtain the final evidence graph. ,as follows: in Represents the graph construction operator; Represents an entity node in the evidence graph; represents the edges in the evidence graph; and represents enhanced visual evidence and edges, and Represents a table component; Represents a graph expansion operator; Evidence map That is, the search answer result of the user's query; The specific steps of visual evidence enhancement include: Analyze user queries using the Large Language Model (LLM) , identification clue graph Image nodes related to the user query , and for each image node Generate an answer related to the image , expressed as: in Representation clue diagram The image node in represents the generation of visual queries, Representing the visual-linguistic big model; From the clue diagram Dynamically extract relationships related to user queries from tabular data; Analyze clue graphs using the Large Language Model (LLM) and identify a set of candidate entities relevant to the user query. , expressed as: ， in Represents a collection of table entities in a clue graph; Represents a table entity, Indicates the mined table relationship; Indicates a join operation between table node t and the entity extracted by LLM after analyzing table t based on query Q; Represents the graph expansion operator, which expands the structure through the neighborhood, as follows: in Indicates yes One-hop neighbor node; Represents a retrieval graph One-hop neighbors in .

6. A query-driven multimodal knowledge graph construction and question-answering system, used to implement a query-driven multimodal knowledge graph construction and question-answering method according to any one of claims 1 to 5, characterized in that: The system includes: a data acquisition module, a data processing module and a display module; The data acquisition module, data processing module and display module are connected in sequence; The data acquisition module is used to construct the data required for the pattern graph of the user query; The data processing module is used to construct a pattern graph of user queries; The data processing module is also used to build a multimodal knowledge base; through the pattern constraint retrieval method and the query focus clue refinement method, combined with the pattern graph, a clue graph is constructed; The data processing module is also used to supplement the missing information of the clue graph through cross-modal reasoning iteration to obtain the retrieval answer results of the user query; The display module is used to visualize the search answer results.

Images