Multi-modal data query semantic analysis method based on fusion of ontology atlas and large language model

Abstract

The invention discloses a multi-modal data query semantic analysis method based on fusion of an ontology map and a large language model. According to the method, through a double-layer semantic architecture, the semantic comprehension ability of a large language model and the structured knowledge of an ontology map are combined, and accurate semantic analysis of user query is achieved. The method specifically comprises the steps of generating LLM semantic vectors through a pre-training language model, extracting entities from a query, mapping the entities to a domain ontology map through a multi-level alignment algorithm to generate ontology semantic vectors, and carrying out weighted fusion on the two layers of semantic vectors through an adaptive weight learning mechanism based on a query type to generate a final query semantic vector. According to the method, an interactive feature capture mechanism is proposed to enhance a synergistic effect, a multi-level entity alignment algorithm is designed to improve mapping accuracy, a degradation processing mechanism is established to guarantee robustness when body coverage is insufficient, and a context enhancement disambiguation algorithm solves the problem of entity ambiguity. According to the method, the query semantic understanding accuracy under the multi-modal data environment is remarkably improved, a high-quality semantic basis is provided for intelligent database scheduling, and the method can be widely applied to supply chain management, scientific research data platforms, medical information integration and other scenes needing cross-multi-source heterogeneous database intelligent query.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and database technology, and in particular to a multimodal data query semantic parsing method based on the fusion of ontology graph and large language model. This method achieves accurate understanding of user queries through a two-layer semantic architecture, providing a high-quality semantic foundation for intelligent scheduling of multi-source heterogeneous databases. Background Technology

[0002] With the diversification and heterogeneity of enterprise data, multi-source database environments have become the norm, including various types such as relational databases (SQL), graph databases, vector databases (VectorDB), and document databases (NoSQL). In this environment, accurately understanding the semantic intent of user queries and selecting the most suitable database for the query has become a pressing technical problem to be solved.

[0003] Existing semantic parsing technologies mainly fall into the following categories: The first is rule-based semantic parsing, which uses predefined rules and templates for query parsing. The limitations of this method are: rules require manual maintenance, which is costly; it struggles to handle the diversity and ambiguity of natural language; and it cannot adapt to new query patterns and changes in domain knowledge.

[0004] The second approach is a semantic parsing method based purely on large language models, utilizing the powerful semantic understanding capabilities of pre-trained language models (such as BERT and GPT) for query parsing. The drawbacks of this method are: a lack of domain-specific structured knowledge, potentially leading to inaccurate understanding; poorer performance than knowledge graph-based methods for factual queries requiring precise matching; and difficulty in guaranteeing the interpretability of the parsing results.

[0005] The third approach is semantic parsing based on ontology graphs, which performs structured parsing by mapping query entities to the ontology graph. The limitations of this method are: the ontology graph has limited coverage and cannot handle concepts not included in the graph; the mapping accuracy is low for semantically ambiguous queries; and building and maintaining a high-quality ontology graph requires significant investment from domain experts.

[0006] Some researchers have attempted to combine large language models with ontology graphs, but existing methods mostly employ simple vector concatenation or fixed-weight fusion. The problems with this simple fusion approach are: it fails to consider the varying degrees of dependence of different query types on the two layers of semantic information; it lacks the ability to capture the synergistic effect of the two layers of information; it cannot dynamically adjust the fusion strategy based on the ontology alignment quality; and it lacks an effective degradation mechanism when ontology coverage is insufficient.

[0007] Therefore, there is an urgent need for a semantic parsing method that can deeply integrate the semantic understanding capabilities of large language models and the structured knowledge of ontology graphs. This method should have an adaptive fusion strategy, a robust degradation mechanism, and good interpretability to meet the intelligent query needs in multimodal data environments. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal data query semantic parsing method based on the fusion of ontology graph and large language model, so as to solve the problems of inaccurate semantic understanding, inflexible fusion strategy and insufficient robustness in the existing technology.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal data query semantic parsing method based on the fusion of ontology graph and large language model, comprising the following steps: Step 1: Receive user query request Q, preprocess the query and identify the query type, which includes fact query, semantic query, reasoning query and mixed query; Step 2: Encode the query Q using a pre-trained language model to generate the LLM semantic vector V_llm; Step 3: Extract entities and relations from query Q, map entities to the domain ontology graph using a multi-level alignment algorithm, extract relevant subgraphs and generate ontology semantic vector V_onto, and calculate alignment confidence. Step 4: Based on the query type, calculate the fusion weights α, β and γ through an adaptive weight learning mechanism, and perform weighted fusion on V_llm and V_onto to generate the final query semantic vector Q'; Step 5: Output the query semantic vector Q' to the database scheduling unit for subsequent database selection and query plan generation.

[0010] Furthermore, the two-layer semantic fusion in step 4 employs an interactive feature enhancement mechanism, specifically including: calculating the interactive feature vector V_interaction = MLP([V_llm ⊙ V_onto]), where ⊙ represents element-wise multiplication; and performing the fusion calculation Q' = α·V_llm + β·V_onto + γ·V_interaction. This fusion method not only considers the linear combination of the two-layer semantic vectors but also captures the synergistic effect of the two layers of information through the interaction term, significantly improving the accuracy of semantic representation.

[0011] Furthermore, the adaptive weight learning mechanism in step 4 dynamically adjusts the fusion strategy based on the query type. For fact queries, the ontology layer weights are increased (typical values: α=0.3, β=0.7) to leverage the precise matching capabilities of the ontology; for semantic queries, the LLM layer weights are increased (typical values: α=0.7, β=0.3) to leverage the semantic understanding capabilities of the large language model; for inference and hybrid queries, balanced weights (typical values: α=0.5, β=0.5) are used. Weight prediction is achieved through the neural network WeightNet, which performs supervised learning using historical query data to optimize the multi-objective loss function.

[0012] Furthermore, the multi-level alignment algorithm in step 3 employs a progressive matching strategy. Level 1 performs exact string matching; Level 2 performs fuzzy matching using edit distance and TF-IDF similarity when exact matching fails; Level 3 calculates the semantic vector cosine similarity when multiple candidates exist; and Level 4 uses relational constraints for disambiguation. Each level returns a corresponding confidence score for subsequent downgrade processing.

[0013] Furthermore, this invention designs a degradation processing mechanism for insufficient ontology coverage. When the alignment confidence is less than 0.5, it completely degrades to using only LLM layer semantics; when the confidence is in the range of [0.5, 0.8), the fusion weights are dynamically adjusted, reducing the proportion of ontology layer weights; when the confidence is greater than or equal to 0.8, the normal fusion process is executed. This mechanism ensures that the system can still function normally when ontology coverage is insufficient, while recording uncovered entities for subsequent ontology expansion.

[0014] Furthermore, this invention proposes a context-enhanced disambiguation algorithm to address entity ambiguity. When an entity has multiple candidate meanings in the ontology graph, keyword and relational features from the query context are extracted, and a comprehensive matching score is calculated for each candidate, with the relational matching score having the highest weight. This algorithm comprehensively considers domain consistency, relational constraints, and keyword co-occurrence, providing an interpretable disambiguation decision process.

[0015] Furthermore, query type identification is based on text features for classification. Fact queries contain explicit factual question words such as numerical values, times, and names; semantic queries contain vague descriptive words such as adjectives and degree adverbs; inference queries contain causal relationship words, hypothesis words, and inference verbs; and mixed queries contain multiple features simultaneously. A text classifier fine-tuned based on BERT is used for type identification.

[0016] The beneficial effects of this invention include: (1) Advanced technology: It is the first to propose a two-layer semantic fusion mechanism with enhanced interactive features. It is not a simple linear combination, but captures the synergistic effect of LLM semantics and ontology knowledge through MLP network, which significantly improves the quality of semantic representation. (2) Strong adaptability: The adaptive weight learning mechanism based on query type dynamically adjusts the fusion strategy according to different query characteristics and continuously optimizes the weight parameters through historical data, which has a high degree of intelligence. (3) Good robustness: The multi-level entity alignment algorithm adapts to different matching scenarios, from exact matching to fuzzy matching to semantic matching, and goes deeper layer by layer; it can gracefully degrade when the ontology coverage is insufficient, and the system will not fail; the polysemy disambiguation mechanism effectively handles entity ambiguity problems. (4) Strong interpretability: The clear fusion weight can trace the semantic source, the alignment confidence quantifies uncertainty, and the disambiguation process is based on interpretable features, which meets the transparency requirements of enterprise-level applications. (5) Significant actual effect: According to actual tests, the accuracy of query semantic understanding is significantly improved, with an improvement of more than 20% compared with the method of using LLM or ontology alone; the database selection accuracy is significantly improved; the system robustness is enhanced, and it covers a wider range of query scenarios. Attached Figure Description

[0017] Figure 1 The flowchart of the overall method of this invention shows the complete process from user query to final semantic vector output, including five main stages: query preprocessing, LLM layer semantic generation, ontology layer semantic generation, adaptive fusion, and output.

[0018] Figure 2 This is a diagram of the two-layer semantic fusion architecture of the present invention, which shows in detail the parallel processing of the LLM layer and the ontology layer, as well as the specific mechanisms for interactive feature calculation and weighted fusion.

[0019] Figure 3 This is a flowchart of the multi-level ontology entity alignment process of the present invention, which shows a progressive alignment process with four levels: exact matching, fuzzy matching, semantic matching, and relation disambiguation.

[0020] Figure 4 This invention presents a polysemous disambiguation decision tree, demonstrating how to disambiguate when an entity has multiple candidate meanings through a comprehensive score combining keyword matching, relation matching, and domain consistency. Detailed Implementation

[0021] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Embodiment 1: Semantic parsing in a supply chain query scenario.

[0022] This embodiment demonstrates the complete parsing process of the present invention's method for complex mixed queries in a supply chain management scenario. User query: "Find suppliers of medical-grade silicone with the lowest cost within the last three months." Domain ontology graph structure: - Entity Types: Supplier, Material, Contract - Relationship types: SUPPLY (supply relationship), HAS_PROPERTY (attribute relationship) - Material attributes: grade, cost, date

[0023] Step 1: Query preprocessing and type identification Upon receiving the original query, the system first performs preprocessing, including basic operations such as word segmentation, stop word removal, and keyword extraction. Query type identification process: - Time constraint detected: "Last three months" → Factual conditional feature - Material type detected: "Medical-grade silicone" → Entity precise matching feature - Optimization objective detected: "Lowest cost" → Ranking condition feature - Query target detected: "Supplier" → Target entity characteristics Output type probability distribution of a text classifier based on BERT fine-tuning: - Fact check: 0.35 - Semantic query: 0.20 - Inference query: 0.05 - Mixed queries: 0.40 (maximum) The system determines the query type as a hybrid query because it includes both factual conditions and semantic understanding requirements.

[0024] Step 2: Generation of semantic vectors in the LLM layer Construct a task-specific prompt: """ Please analyze the semantic intent and key information of the following data query: Search: Find suppliers of medical-grade silicone with the lowest cost within the last three months. Please identify: 1. Query the target entity type 2. Filtering criteria (time, type, cost) 3. Sorting criteria """ The BERT-base model (768 hidden layer dimensions) is invoked, and the query text is input into the model to obtain the last hidden state of the output sequence. The hidden state of the [CLS] token is extracted as the overall semantic representation of the query, denoted as V_llm ∈ R^768. The V_llm vector encodes the semantic information of the query, including the query intent (finding a supplier), constraints (time, material type, cost), and optimization objective (lowest cost).

[0025] Step 3: Ontology Layer Semantic Vector Generation, 3.1 Entity Recognition Stage: Extracting entities from the query using the Named Entity Recognition (NER) model: - Entity 1: "Medical-grade silicone", Type: Material - Entity 2: "Supplier", Type: Supplier - Time constraint: "last three months", type: TimeConstraint 3.2 Entity Alignment Stage (Execution of Multi-Level Alignment Algorithm). Alignment of the entity "Medical Grade Silicone": Level 1 (Exact Match): Searching the ontology graph for nodes whose tags exactly match "medical-grade silicone" yielded no results. Level 2 (fuzzy matching): Fuzzy matching is performed using edit distance (Levenshtein Distance) and TF-IDF similarity: - Candidate Node 1: Material.MedicalGradeSilicone - Tag: "Medical-grade silicone" - Edit distance: 2 ("medical" vs "medical") - TF-IDF similarity: 0.92 - Candidate Node 2: Material.IndustrialSilicone - Tag: "Industrial Grade Silicone" - Edit distance: 5 - TF-IDF similarity: 0.65 Candidates with edit distance ≤ 3 and TF-IDF similarity ≥ 0.8 are filtered to obtain the candidate set: {Material.MedicalGradeSilicone} Level 3 (Semantic Matching): Calculate the cosine similarity between the BERT vector of the entity "medical-grade silicone" and the candidate node embedding. A high similarity match result is obtained; if the similarity is greater than the preset threshold of 0.75, the alignment is successful. Alignment result: Material.MedicalGradeSilicone, Confidence: High Alignment of entity "Supplier" (simplified process): Level 1 exact match successful, mapped to ontology type Supplier, confidence: very high.

[0026] 3.3 Relation extraction stage: Identify relational terms between entities from the query: "provide". Map the relational terms to ontology relation types: - "Provide" → Ontology Relationship: SUPPLY - Relationship triple: (Supplier, SUPPLY, Material.MedicalGradeSilicone) The ontology graph was checked to confirm the existence of the relationship type; verification was successful. 3.4 Subgraph Extraction Stage Extract query-related subgraphs from the ontology graph: - Central node: Material.MedicalGradeSilicone - 1-hop neighbor node: - All Supplier nodes connected via the SUPPLY relationship - Material property nodes: Property:grade, Property:cost, Property:date - 2-hop neighbor nodes: - Contract node (contract information) - Supplier's attribute nodes (contact information, qualifications, etc.) The extracted subgraph contains several related nodes and edges, forming a query-related knowledge subnetwork. 3.5 In the graph embedding generation stage, the extracted subgraphs are vectorized using the TransE (Translation-based Embedding) model: - Represent nodes and edges as points and vectors in a vector space. Training objective: h + r ≈ t (head entity + relation ≈ tail entity) - Aggregate the subgraphs to generate a total graph representation vector. Further extract higher-order features using a graph convolutional network (GCN): - Input layer: Node feature matrix of the subgraph - Hidden layers: 2 layers of GCN, each with a dimension of 512 - Output layer: Fully connected layer mapped to 768 dimensions Generate ontology semantic vector: V_onto ∈ R^768 Overall alignment confidence: The weighted average of the alignment confidence of each entity is taken, and the result is the high confidence score.

[0027] Step 4: Adaptive Weight Fusion 4.1 Basic weights corresponding to query types Based on the query type "mixed query", obtain the basic weights from the weight strategy library: - α_base = 0.5 (Base weight of LLM layer) - β_base = 0.5 (Base weight of ontology layer) - γ_base = 0.1 (base weight of interaction items) 4.2 Weighted Adaptive Prediction Fine-tuning of weights using the WeightNet weight prediction neural network: Input feature vector: - Query the embedding (the dimensionality reduction representation of V_llm, 128 dimensions). - Query type one-hot encoding: [0, 0, 0, 1] (mixed query) - Ontology alignment confidence: 0.93 - Query length: 18 characters - Number of entities: 2 WeightNet network structure: - Input layer: 128 + 4 + 1 + 1 + 1 = 135 dimensions - Hidden Layer 1: Fully connected layer, 256-dimensional, ReLU activation - Hidden Layer 2: Fully connected layer, 128-dimensional, ReLU activation - Output layer: Fully connected layer, 3D, Softmax normalized Network output weights: - α ≈ 0.45 (LLM layer weights) - β ≈ 0.45 (ontology layer weight) - γ ≈ 0.10 (interaction term weight) Because of the high alignment confidence, the system adopts a relatively balanced weight allocation to make full use of the two layers of semantic information.

[0028] 4.3 Interaction Feature Calculation. Calculate the element-wise multiplication of the LLM semantic vector and the ontology semantic vector: V_element_wise = V_llm ⊙ V_onto The element-wise multiplication results are input into an MLP network to extract interaction features: MLP network structure: - Input layer: 768 dimensions - Hidden layer: Fully connected layer, 512-dimensional, ReLU activation - Output layer: Fully connected layer, 768-dimensional, Tanh activation Output interaction feature vector: V_interaction ∈ R^768

[0029] 4.4 Final fusion calculation. Perform weighted fusion: Q' = α·V_llm + β·V_onto + γ·V_interaction Where α ≈ 0.45, β ≈ 0.45, γ ≈ 0.10 Generate the final query semantic vector: Q' ∈ R^768

[0030] Step 5: Semantic Vector Output and Effect Verification. The final query semantic vector Q' is output to the intelligent scheduling unit. The intelligent scheduling unit calculates the cosine similarity between Q' and the metadata of each database: Database candidates: - MySQL database (stores purchase contracts and cost data): High similarity - Neo4j graph database (storage vendor relationship network): highest similarity - MongoDB Document Repository (Storage Vendor Qualification Documents): Medium Similarity - Milvus Vector Database (stores semantic vectors describing materials): High similarity The intelligent scheduling unit selects the two databases with the highest similarity: Neo4j + MySQL. Analysis of the rationality of scheduling decisions: Neo4j graph database can efficiently query supplier network relationships and find suppliers that provide specific materials. - MySQL relational databases can perform precise filtering and sorting based on time constraints and cost fields. - The combination of the two databases can perfectly meet query requirements.

[0031] Comparative experiment to verify the effectiveness of the method of the present invention: Method 1: Using only LLM (α=1, β=0, γ=0) - Query semantic vector: Q'_LLM = V_llm - Database selection result: Milvus + MongoDB - Actual execution result: Milvus found relevant materials through semantic search, but could not precisely match "medical grade". MongoDB document matching is not precise enough and contains a lot of noisy data. - Query accuracy: Low - Analysis: It focuses on semantic similarity and ignores the need for precise matching of structured conditions. Method 2: Use only the ontology (α=0, β=1, γ=0) - Query semantic vector: Q'_Onto = V_onto - Database selection result: MySQL + Neo4j - Actual execution result: - MySQL can accurately match material type and cost conditions. Neo4j can query supply relationships However, the semantic interpretation of "the past three months" is not flexible enough (it is strictly calculated as 90 days). - Query accuracy: Medium Analysis: The structured matching is accurate, but the semantic understanding capability is insufficient. Method 3: The method of the present invention (α≈0.45, β≈0.45, γ≈0.1) - Query semantic vector: Q' = α·V_llm + β·V_onto + γ·V_interaction - Database selection result: Neo4j + MySQL - Actual execution result: Neo4j uses graph lookup to find the supply network - MySQL performs precise filtering and sorting based on time, type, and cost. - The result set is accurate and complete. - Query accuracy: Significantly improved, significantly better than single methods. - Analysis: It combines the semantic understanding of LLM with precise ontology matching to achieve the best results.

[0032] This embodiment demonstrates the complete execution flow of the method of the present invention in a complex mixed query scenario. Through two-layer semantic fusion and adaptive weight adjustment, it significantly improves the accuracy of query semantic understanding and the rationality of database selection. Embodiment 2: Ambiguity Disambiguation Processing This embodiment demonstrates how the method of the present invention handles the problem of entity ambiguity, ensuring that the queried entity is mapped to the correct ontology node. User query: "Find Apple's supplier contact information" Problem Analysis: The entity "apple" has two candidate meanings in the ontology graph: - Fruit.Apple (Fruit category, Apple fruit) - Company.AppleInc (Technology company, Apple Inc.) This embodiment demonstrates the disambiguation process. Step 1-2: Query preprocessing and LLM semantic generation (same as Embodiment 1, omitted here) Query type recognition result: Fact query (finding specific contact information) Step 3: Ontology layer semantic generation (highlighting the disambiguation process) 3.1 Entity recognition: - Entity 1: "Apple" (ambiguous) - Entity 2: "Supplier" 3.2 Ontology entity alignment (polysemy disambiguation): Perform alignment on the entity "Apple": Level 1 (exact match): Search for nodes with the label "Apple" in the ontology and find two candidates: - Candidate 1: Fruit.Apple (label: "Apple") - Candidate 2: Company.AppleInc (label: "Apple") Since there are multiple candidates and it cannot be directly determined, enter the disambiguation process. Level 4 (relationship disambiguation): Extract query context features: - Keywords: ["Supplier", "Contact Information"] - Relationship word: "of" → indicating a belonging relationship - Query intent: Find supplier information Calculate the context matching score for each candidate: Candidate 1: Fruit.Apple (Fruit Apple) Calculate the keyword matching score: - Is "Supplier" in the set of relevant keywords of Fruit.Apple? - Relevant keywords of Fruit.Apple: ["Fruit", "Agricultural products", "Planting", "Wholesale", "Supplier", "Purchasing"] - "Supplier" matches successfully → +1 - Is "Contact Information" in the relevant keywords? - Not in the core keywords, but belongs to general business terms → +0.5 - KeywordScore = 1.5 Relationship matching score calculation: - Implicit relationship in the query: the relationship between "supplier" and "Apple" - Check if the following exists in the main file: Supplier -SUPPLY-> Fruit.Apple - This relationship exists (fruit supplier) → +2 - Check for other supporting relationships: - Distributor -DISTRIBUTE-> Fruit.Apple (Distributor Relationship) → +1 - RelationScore = 3 Domain consistency score calculation: - Query domain determination: Commercial procurement domain Fruit.Apple belongs to the agriculture / food sector. - Domain relevance: 0.6 (Some relevance, but not a primary domain) - DomainScore = 0.6 Overall Score: Score_Fruit = 0.3×1.5 + 0.5×3 + 0.2×0.6 = 0.45 + 1.5 + 0.12 = 2.07 Candidate 2: Company. Apple Inc. Keyword matching score calculation: Is "supplier" a relevant keyword for Company.AppleInc? Keywords related to Company.AppleInc: ["Technology Company", "Electronics", "Supply Chain", "Suppliers", "Partners"] - Supplier successfully matched → +1 - "Contact Information" Matching: Businesses typically need to record contact information → +1 - KeywordScore = 2 Relationship matching score calculation: - Check if the following exists in the ontology: Supplier -SUPPLY_TO->Company.AppleInc - This relationship exists (a supplier to Apple) → +2 - Check if there are any "Contact Information" related attributes: - Company.AppleInc has the attribute HAS_CONTACT_INFO → +1 - RelationScore = 3 Domain consistency score calculation: - Query area: Business procurement - Company.Apple Inc. Industry: Business / Technology - Domain consistency: 0.9 - DomainScore = 0.9 Overall Score: Score_Company = 0.3×2 + 0.5×3 + 0.2×0.9 = 0.6 + 1.5 + 0.18 = 2.28 Disambiguation Decision Making: Compare the combined scores of the two candidates: - Fruit.Apple: 2.07 - Company.AppleInc: 2.28 Select the candidate with the highest score: Company.AppleInc Alignment result: Maps "Apple" to the ontology node Company.AppleInc Alignment confidence calculation (normalized): confidence = 2.28 / (2.07 + 2.28), resulting in a medium confidence level. Since the confidence level is in the medium range [0.5, 0.8), the system will adjust the weights during subsequent fusion. Step 4: Adaptive weight fusion (considering confidence level) Since the ontology alignment confidence level is medium, the weight adjustment mechanism is triggered: Original base weights (fact query): - α_base ≈ 0.3 - β_base ≈ 0.7 Dynamic adjustments are made based on the confidence adjustment formula, and the weights of the LLM layer are increased to compensate for the uncertainty of ontology alignment. Adjusted weights: - α_final ≈ 0.58 (increase) - β_final ≈ 0.42 (decreased) - γ_final ≈ 0 (reduces interaction terms when disambiguation is uncertain) Actual integration: Q' = α·V_llm + β·V_onto This adjustment increases the weight of the LLM layer and decreases the weight of the ontology layer to compensate for the uncertainty of ontology alignment. Effect Analysis: Although disambiguation involves some uncertainty, the system can still generate reasonable semantic vectors through the weight adjustment mechanism. - If disambiguation is correct (Apple), the ontology layer provides the correct structured knowledge. - If the disambiguation is incorrect (it's actually fruit), the high weights of the LLM layer can provide compensatory semantic understanding. Subsequent database scheduling results: - We selected a CRM system (storage company contact information) and MongoDB (storage vendor profile). - The search results include Apple's supplier contact information. - Verify the correctness of the disambiguation decision This embodiment demonstrates that the context-enhanced disambiguation algorithm of the present invention can effectively handle entity polysemy and address disambiguation uncertainty through a weight adjustment mechanism. Example 3: Degradation processing for insufficient body coverage This embodiment demonstrates how the method of the present invention maintains system robustness through a degradation mechanism when the ontology graph does not cover the query entity. User query: "Find biological laboratories that support CRISPR gene editing technology" Problem Analysis: The ontology map does not include the emerging concept of "CRISPR gene editing technology," which is a typical scenario of insufficient ontology coverage. Steps 1-2: Query preprocessing and LLM semantic generation Query type identification: Semantic query (requires understanding the semantics of "CRISPR gene editing technology" and "support") The LLM semantic vector V_llm was successfully generated, and the BERT model was able to understand the semantics of "CRISPR" based on pre-trained knowledge. Step 3: Ontology layer semantic generation (encountering insufficient coverage issues) 3.1 Entity Recognition: Entity 1: "CRISPR gene editing technology" Entity 2: "Biological Laboratory" 3.2 Ontology entity alignment (failed): Alignment with the entity "CRISPR gene editing technology": Level 1 (Exact Match): A search for "CRISPR gene editing technology" in the ontology yielded no matching nodes. Level 2 (Fuzzy Matching): Fuzzy matching using edit distance and TF-IDF: - Candidate 1: "Gene technology" (edit distance 8, TF-IDF similarity 0.58) - Candidate 2: "Biotechnology" (edit distance 10, TF-IDF similarity 0.51) - All are below the threshold (edit distance ≤ 3 or TF-IDF ≥ 0.8) Fuzzy matching failed. Level 3 (Semantic Matching): Calculate the cosine similarity between the BERT vector of "CRISPR gene editing technology" and all ontology nodes: - Moderate similarity to Technology.GeneticEngineering - Moderate similarity to Technology.Biotechnology - Low similarity to Technology.MolecularBiology The highest similarity did not reach the preset threshold of 0.75. Semantic matching failed. Alignment result: Unable to map "CRISPR gene editing technology" to ontology node. Alignment confidence: Low Alignment of the physical "biological laboratory": Level 2 fuzzy matching successful: Organization.BiologyLab Confidence level: High Overall alignment confidence level: The overall confidence level is moderately low after comprehensive calculation. Step 4: Trigger the downgrade mechanism Because the overall alignment confidence level is at a moderately low level, partial degradation is triggered: Original weights (semantic query): - α_base ≈ 0.7 - β_base ≈ 0.3 However, given the low confidence level (< 0.5) of the key entity "CRISPR gene editing technology", the system decided to further downgrade. Downgrade strategy selection: Because the primary query entity failed to align, the system selected a full degradation strategy: - α_final = 1.0 (fully dependent on LLM) - β_final = 0 (ignore the ontology layer) - γ_final = 0 Final Fusion: Q' = V_llm Records not covered entities: The system will log entities that are not successfully aligned to the log file: { "entity": "CRISPR gene editing technology", "query": "Find biological laboratories that support CRISPR gene editing technology", "timestamp": "2025-11-13 14:23:45", "alignment_confidence": "low", "closest_match": "Technology.GeneticEngineering", "occurrence_count": 1 } Step 5: Semantic Vector Output and Results Output Q' = V_llm to the intelligent scheduling unit. Database selection results: - Milvus Vector Database (stores semantic vectors describing technologies): Highest similarity - MongoDB document repository (stores lab archives and research directions): High similarity - Neo4j graph database (stores organizational relationships): Medium similarity Top 2 choices: Milvus + MongoDB Query execution result: Milvus uses semantic search to find technology descriptions containing keywords such as "CRISPR" and "gene editing". MongoDB uses text matching to find labs whose research focuses include gene editing. - The results set includes several relevant laboratories, and the accuracy is slightly lower than the full fusion protocol, but still within acceptable limits. Online learning and ontology extension The system continuously monitors the frequency of occurrence of uncovered entities. When the cumulative occurrence of "CRISPR gene editing technology" exceeds a preset threshold, the ontology expansion process is triggered. Automatic expansion suggestion generation: - Use LLM to generate entity definitions: Entity Name: CRISPR Gene Editing Technology Superior concept: Technology.GeneticEngineering Definition: A precise gene editing technology based on the CRISPR-Cas9 system Correlation: - IS_A: Technology.GeneticEngineering - USED_IN: Research.MolecularBiology - APPLIED_BY: Organization.BiologyLab - Submit to manual review queue - Add ontology graph after approval - Update ontology embedding The next time the same query is encountered, alignment is successful, and system performance returns to normal. Comparison of downgrade effects Comparing the effects of full fusion (assuming the ontology covers "CRISPR") versus downgrade processing: Hypothetical scenario (ontology covering CRISPR): Alignment successful, high confidence level - Fusion weights: α≈0.7, β≈0.3 - Database selection: Neo4j + MongoDB (structured relational database + document database) - Expected accuracy: High Real-world scenario (downgrade handling): Alignment failed, complete degradation - Fusion weights: α=1.0, β=0 - Database selection: Milvus + MongoDB (semantic search + document search) - Actual accuracy: Moderate Accuracy decreased, but the system did not fail and could still provide valuable results. This validates the effectiveness of the degradation mechanism. This embodiment demonstrates that the degradation processing mechanism of the present invention can effectively address situations with insufficient ontology coverage, ensuring that the system remains robust when facing emerging concepts or long-tail entities. Simultaneously, through online learning and ontology expansion mechanisms, the system possesses continuous evolution capabilities.

Claims

1. A multimodal data query semantic parsing method based on the fusion of ontology graph and large language model, characterized in that, Includes the following steps: Step 1: Receive user query request Q, preprocess the query and identify the query type, which includes fact query, semantic query, reasoning query and mixed query; Step 2: Encode the query Q using a pre-trained language model to generate the LLM semantic vector V_llm; Step 3: Extract entities and relations from query Q, map entities to the domain ontology graph using a multi-level alignment algorithm, extract relevant subgraphs and generate ontology semantic vector V_onto, and calculate alignment confidence. Step 4: Based on the query type, calculate the fusion weights α, β and γ through an adaptive weight learning mechanism, and perform weighted fusion on V_llm and V_onto to generate the final query semantic vector Q'; Step 5: Output the query semantic vector Q' to the database scheduling unit for subsequent database selection and query plan generation.

2. The semantic parsing method according to claim 1, characterized in that, The two-layer semantic fusion in step 4 includes: Calculate the interaction feature vector: V_interaction = MLP([V_llm ⊙ V_onto]), where ⊙ represents element-wise multiplication, and MLP is a multilayer perceptron network; Perform fusion calculation: Q' = α·V_llm + β·V_onto + γ·V_interaction, where α is the LLM layer weight, β is the ontology layer weight, and γ is the interaction feature weight; The fusion algorithm captures the synergistic effect of LLM semantics and ontology structured knowledge by introducing the interaction term V_interaction, rather than a simple linear combination.

3. The semantic parsing method according to claim 1, characterized in that, The adaptive weight learning mechanism in step 4 includes: Define a set of query types: {fact query, semantic query, inference query, and mixed query}; The query type Q_type is obtained by identifying the type of query Q using a lightweight text classifier; The weight prediction neural network WeightNet predicts fusion weights based on query features and query type: α, β, γ = WeightNet(Q_features, Q_type); Supervised learning is performed using historical query datasets to optimize the multi-objective loss function: L = Σ[λ1·(1-Accuracy)+ λ2·Latency + λ3·BCE(DB_actual, DB_pred)], where λ1, λ2, and λ3 are the multi-objective optimization weights, Accuracy is the database selection accuracy, Latency is the query latency, and BCE is the binary cross-entropy loss.

4. The semantic parsing method according to claim 1, characterized in that, The multi-level alignment algorithm in step 3 includes: Level 1: Perform exact string matching on the extracted entities and search for identical node labels in the ontology graph; Level 2: If exact matching fails, use edit distance and TF-IDF similarity to perform fuzzy matching to obtain a set of candidate nodes; Level 3: If the number of candidate nodes is greater than 1, calculate the cosine similarity between the entity description and the semantic vector of the candidate node, and filter candidates with similarity higher than a preset threshold. Level 4: If multiple candidates still exist, disambiguation is performed using the relational constraints of other entities in the query, and the ontology node with the best relational structure is selected. Returns the final aligned ontology node and alignment confidence, which is used for subsequent fusion weight adjustment.

5. The semantic parsing method according to claim 4, characterized in that, When the ontology alignment confidence level falls below a preset threshold, a degradation processing mechanism is triggered: If the confidence level is less than 0.5, then it is completely downgraded to using only LLM layer semantics: Q' = V_llm; If the confidence level is in the range [0.5, 0.8), the fusion weights are dynamically adjusted to reduce the weight ratio of the ontology layer. α_adj = α + (1-α)·(0.8-confidence) / 0.3 β_adj = β - β·(0.8-confidence) / 0.3 Q' = α_adj·V_llm + β_adj·V_onto If the confidence level is greater than or equal to 0.8, then proceed with the normal fusion process; The degradation mechanism ensures that the system can still function normally when the ontology coverage is insufficient, and records the uncovered entities for subsequent ontology expansion.

6. The semantic parsing method according to claim 1, characterized in that, When an entity has multiple candidate meanings in the ontology graph, execute the context-enhanced disambiguation algorithm: Extract keyword and relational features from the query context; Calculate the context matching score for each candidate meaning: Score = w1·KeywordScore + w2·RelationScore + w3·DomainScore, where the weight of the relation matching score w2 is higher than the weight of the keyword w1 and the weight of the domain w3; The candidate with the highest overall score is selected as the final mapping result for the entity; The disambiguation algorithm comprehensively considers the domain consistency of entities, relational constraints, and keyword co-occurrence characteristics, providing an interpretable disambiguation decision process.

7. The semantic parsing method according to claim 3, characterized in that, Query type identification is based on the following features for classification: Fact query characteristics: The query contains explicit factual question words such as numerical values, time, and names, as well as a query intent of exact match type; Semantic query characteristics: The query contains vague descriptive words such as adjectives and degree adverbs, which require semantic understanding to be accurately parsed; Inference query characteristics: The query contains causal relation words, hypothesis words, and inference verbs, and requires relational reasoning and knowledge derivation; Hybrid query features: The query contains two or more of the above features simultaneously; A text classifier based on BERT fine-tuning is used to identify the type of the query and output the probability distribution of the query type.

8. A semantic parsing system based on the fusion of ontology graph and large language model, characterized in that, The system employs the semantic parsing method as described in any one of claims 1-7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; The memory is used to store program data, and the processor is used to execute the program data to implement the semantic parsing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the semantic parsing method as described in any one of claims 1-7.

Images