Knowledge graph generation method and system combined with dynamic semantic association mining

Abstract

The invention provides a knowledge graph generation method and system combined with dynamic semantic association mining in the technical field of artificial intelligence and knowledge engineering, and the method comprises the steps: S1, collecting basic data which is used for generating a knowledge graph and comprises relational data and non-relational data, and carrying out the preprocessing of each basic data through a kettle tool; s2, reasoning each piece of basic data through a LangChain framework in combination with a large model so as to dynamically and semantically associate and mine ontology information from each piece of basic data; s3, initializing a knowledge graph, converting the ontology information into a graph structure of the knowledge graph, and dynamically updating the knowledge graph based on the newly mined ontology information; and step S4, storing the knowledge graph in a graph database Neo4j in real time, and performing visual display on the knowledge graph stored in the graph database Neo4j through a Neo4jBLOOM tool. The knowledge graph generation method has the advantages that the flexibility, the automation degree, the efficiency, the semantic consistency, the dynamic adaptability and the expansibility of knowledge graph generation are greatly improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and knowledge engineering technology, and in particular to a knowledge graph generation method and system that combines dynamic semantic association mining. Background Technology

[0002] Knowledge graphs, as a structured knowledge representation method, present knowledge through a graph structure of entities, concepts, and their relationships, enabling the systematic organization and efficient utilization of knowledge. Currently, the mainstream methods for generating knowledge graphs fall into three categories:

[0003] 1. Top-down generation method:

[0004] The schema layer is constructed based on a predefined ontology model (Schema), and the data layer entities are populated step by step, specifically including:

[0005] Ontology modeling: relies on domain experts or structured knowledge bases (such as Wikidata) to define the hierarchical system of entities / relationships (e.g., in the medical field, "disease-symptom-treatment" association rules need to be preset);

[0006] Data population: Extracted entities (such as "lung cancer") and relations (such as "belongs to - cancer") are mapped to the ontology framework to form structured knowledge.

[0007] Technical shortcomings: High reliance on manual labor leads to high construction costs and poor flexibility; low data filling efficiency causes graph sparsity; rigidity of the pattern layer results in weak cross-domain migration ability and high risk of error propagation.

[0008] 2. Bottom-up generation method:

[0009] Knowledge elements are automatically extracted from unstructured data and summarized into a pattern layer, specifically including:

[0010] Entity extraction: Using NLP techniques (such as BERT, CRF models) to identify named entities (such as people's names, drug names) in text.

[0011] Relation extraction: Mining semantic relationships between entities (e.g., "drug-treatment-disease") through dependency parsing or deep learning models (e.g., GNN).

[0012] Attribute extraction: Extract the attribute values ​​of an entity (e.g., "disease-incubation period-3 days").

[0013] Knowledge fusion: Resolving conflicts between multiple data sources (such as entity alignment and attribute value merging), for example, unifying the record of "aspirin" in different databases.

[0014] Pattern induction: Automatically generate hierarchical structures based on clustering algorithms (e.g., discovering "infectious disease" as a subclass of "disease").

[0015] Technical drawbacks: Data noise leads to poor semantic consistency; lack of global constraints results in uneven knowledge coverage; limited inductive ability makes it difficult to form high-level abstractions; and high dynamic maintenance costs.

[0016] 3. Hybrid generation method:

[0017] Combining ontology-guided and data-driven approaches, the human-machine collaborative optimization model layer specifically includes:

[0018] Preliminary ontology construction: Define core concepts based on domain knowledge (such as "e-commerce domain - products - brands").

[0019] Data-driven extension: Extract new entities and relationships from the data and dynamically update the ontology (e.g., discover that "smart home" is a new subclass of "product").

[0020] Human-machine collaborative verification: Correcting errors through expert review or rule engine (such as correcting "Apple" being incorrectly classified as "Company").

[0021] Technical drawbacks: High implementation complexity, high technical threshold, high consumption of computing resources, low operating efficiency, difficulty in completely eliminating data and pattern conflicts, still high demand for manual intervention, limited degree of automation, limited scalability, difficulty in adapting to ultra-large-scale data, high difficulty in evaluation and debugging, and difficulty in quality assurance.

[0022] Therefore, how to provide a knowledge graph generation method and system that combines dynamic semantic association mining to improve the flexibility, automation, efficiency, semantic consistency, dynamic adaptability and scalability of knowledge graph generation has become an urgent technical problem to be solved. Summary of the Invention

[0023] The technical problem to be solved by this invention is to provide a knowledge graph generation method and system that combines dynamic semantic association mining, thereby improving the flexibility, automation, efficiency, semantic consistency, dynamic adaptability and scalability of knowledge graph generation.

[0024] In a first aspect, the present invention provides a knowledge graph generation method combining dynamic semantic association mining, comprising the following steps:

[0025] Step S1: Collect basic data, including relational and non-relational data, for generating the knowledge graph, and preprocess the basic data using the Kettle tool;

[0026] Step S2: Using the LangChain framework in conjunction with a large model, reason about each of the basic data to dynamically mine ontological information including entities, relationships, and attributes from each of the basic data through semantic association.

[0027] Step S3: Initialize a knowledge graph using LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information;

[0028] Step S4: Store the knowledge graph in real time to the graph database Neo4j, and use the Neo4jBLOOM tool to visualize the knowledge graph stored in the graph database Neo4j.

[0029] Furthermore, step S1 specifically includes:

[0030] The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs.

[0031] The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

[0032] Furthermore, step S2 specifically includes:

[0033] A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association.

[0034] The semantic parsing process is specifically as follows:

[0035] A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

[0036] Furthermore, step S3 specifically includes:

[0037] A knowledge graph is initialized using an LLM Knowledge Graph Builder. The ontology information is semantically normalized. The LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure that includes node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

[0038] Furthermore, step S4 specifically includes:

[0039] The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j performs structured storage of the knowledge graph and visualizes the knowledge graph stored in Neo4j using the Neo4jBLOOM tool.

[0040] Secondly, this invention provides a knowledge graph generation system that combines dynamic semantic association mining, comprising the following modules:

[0041] The basic data acquisition module is used to collect basic data, including relational and non-relational data, for generating knowledge graphs, and to preprocess the basic data using the Kettle tool.

[0042] The ontology information mining module is used to perform reasoning on the basic data using the LangChain framework and a large model, so as to dynamically mine ontology information, including entities, relationships and attributes, from the basic data through semantic association.

[0043] The knowledge graph generation module is used to initialize a knowledge graph through the LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information.

[0044] The knowledge graph storage and display module is used to store the knowledge graph in real time to the graph database Neo4j, and to visualize the knowledge graph stored in the graph database Neo4j using the Neo4jBLOOM tool.

[0045] Furthermore, the basic data acquisition module is specifically used for:

[0046] The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs.

[0047] The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

[0048] Furthermore, the ontology information mining module is specifically used for:

[0049] A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association.

[0050] The semantic parsing process is specifically as follows:

[0051] A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

[0052] Furthermore, the knowledge graph generation module is specifically used for:

[0053] A knowledge graph is initialized using an LLM Knowledge Graph Builder. The ontology information is semantically normalized. The LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure that includes node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

[0054] Furthermore, the knowledge graph storage and display module is specifically used for:

[0055] The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j performs structured storage of the knowledge graph and visualizes the knowledge graph stored in Neo4j using the Neo4jBLOOM tool.

[0056] The advantages of this invention are:

[0057] 1. Basic data, including relational and non-relational data, is collected for generating the knowledge graph. This data is preprocessed using the Kettle tool. Then, the LangChain framework, combined with a large model, is used to reason about the basic data, dynamically mining ontology information, including entities, relationships, and attributes, through semantic associations. A knowledge graph is initialized using an LLM Knowledge Graph Builder, converting ontology information into the graph structure of the knowledge graph. The knowledge graph is dynamically updated based on newly mined ontology information. Finally, the knowledge graph is stored in real-time in the Neo4j graph database, and visualized using the Neo4jBLOOM tool. In essence, the LangChain framework schedules a large model to perform intelligent reasoning on multi-source heterogeneous basic data, automatically mining explicit and implicit semantic associations between entities and generating standardized ontology information, replacing manually predefined ontology patterns. This approach overcomes domain limitations, improves flexibility, and reduces manual costs through automated semantic parsing. The integration of the Kettle tool with the lake repository achieves efficient data preprocessing and LLM Knowledge Graph integration. Builder's dynamic graph update mechanism ensures semantic consistency of entity attributes while processing incremental data in real time and adaptively adjusting the graph structure, significantly enhancing dynamic adaptability and operational efficiency. Ultimately, relying on the distributed storage and automated verification capabilities of the Neo4j graph database, it supports real-time structured storage and visual monitoring of massive heterogeneous data, comprehensively optimizing scalability and quality controllability. In other words, by integrating dynamic semantic association mining and an end-to-end automated toolchain, it significantly improves the overall performance of knowledge graph generation, ultimately greatly enhancing the flexibility, automation, efficiency, semantic consistency, dynamic adaptability, and scalability of knowledge graph generation.

[0058] 2. By collecting and preprocessing relational data (such as database data, GIS vector data, and Excel files) and non-relational data (such as text, images, and PDFs), and using the Kettle tool to perform extraction, cleaning, deduplication, and format unification operations, "lake-warehouse integration" processing was achieved. This supports the comprehensive processing of heterogeneous data sources, avoids the dependence on a single data type in traditional methods, and improves the data foundation quality of the knowledge graph. Automated preprocessing reduces the impact of erroneous data, ensures the consistency of input in subsequent steps, reduces the need for manual intervention in knowledge graph construction, improves overall efficiency, and solves the problems of scattered data sources and incomplete cleaning in knowledge graph construction.

[0059] 3. Using the LangChain framework combined with a large model as the inference engine, semantic parsing is performed on the preprocessed basic data. Based on a preset Prompt template, ontological information (including explicit and implicit relationships) of entities, relations, and attributes is dynamically mined to generate a set of structured triples. The application of the large model can identify implicit relationships (such as inferring implicit entity associations from unstructured text), while traditional rule engines or statistical methods only handle explicit relationships. This significantly increases the information richness and accuracy of the knowledge graph. The semantic parsing process is automated based on the Prompt template, eliminating the need for manual rule definition. It can adapt to the dynamic changes of multi-source data, improving the flexibility and scalability of the solution. Through dynamic semantic association mining, end-to-end inference is performed using an LLM (Large Language Model), solving the problems of shallow semantic mining and strong reliance on manual intervention in existing knowledge graphs.

[0060] 4. The knowledge graph is initialized using the LLM Knowledge Graph Builder. After semantic normalization, the ontology information is converted into a graph structure (including node types, attributes, relation constraints, etc.) and updated in real time based on newly mined ontology information. Semantic normalization ensures the consistency and standardization of ontology information and reduces the risk of conflicts introduced by heterogeneous data. The dynamic update mechanism enables the knowledge graph to continuously absorb new data, avoiding the long cycle and high cost of rebuilding the graph in traditional methods and improving applicability.

[0061] 5. By storing the knowledge graph in real-time to the Neo4j graph database (including data type validation, integrity validation, and constraint validation) and visualizing it using the Neo4jBLOOM tool, Neo4j's validation mechanism ensures data integrity and consistency. Real-time storage supports fast indexing of large-scale graphs, improving query performance. The integration of the Neo4jBLOOM tool enables interactive visualization, facilitating users' intuitive understanding of the graph structure and relationships, and enhancing decision support capabilities. This end-to-end storage and visualization design is practical and technologically innovative (such as seamlessly integrating validation mechanisms with visualization), solving the pain points of lagging graph display and complex user operations.

[0062] 6. By integrating multi-stage technologies (such as Kettle data processing, LangChain inference, large model mining, LLMKnowledge Graph Builder construction, Neo4j storage and visualization), an efficient end-to-end process is formed; the integration of various tools (such as LangChain and large models) optimizes the data flow, reduces redundant calculations, realizes a high degree of automation in knowledge graph construction, adapts to big data scenarios (such as heterogeneous data in the financial and medical fields), and the dynamic update and real-time storage mechanism supports long-term applications.

[0063] 7. By combining semantic normalization processing with the real-time parsing capabilities of LLM Knowledge Graph Builder, newly mined ontology information can be seamlessly updated to update the graph structure. Compared with traditional methods that periodically rebuild the graph (such as T+1 update), it supports streaming data response and effectively reduces decision latency.

[0064] 8. By integrating the Kettle tool, LangChain framework, and large model, it innovatively achieves dynamic semantic deep mining of multi-source heterogeneous data (including GIS spatial information and unstructured text / images), breaking through the technical bottlenecks of traditional methods in implicit relationship identification and real-time updates. Its automated graph construction mechanism based on LLM Knowledge Graph Builder and Neo4j's three-level verification storage architecture not only significantly improves the cognitive depth and structural accuracy of knowledge graphs (such as inferring disease concurrency patterns or financial risk correlations from complex data), but also supports industrial-grade real-time decision-making with streaming update response. At the same time, the Neo4jBLOOM visualization tool greatly reduces the usage threshold, ultimately forming an end-to-end solution integrating efficient data processing, intelligent semantic parsing, dynamic graph construction, and reliable visualization. Attached Figure Description

[0065] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0066] Figure 1 This is a flowchart of a knowledge graph generation method combining dynamic semantic association mining according to the present invention.

[0067] Figure 2 This is a schematic diagram of the structure of a knowledge graph generation system that combines dynamic semantic association mining according to the present invention.

[0068] Figure 3 This is a schematic diagram of the knowledge graph construction process of this invention. Detailed Implementation

[0069] The overall approach of the technical solution in this application is as follows: The LangChain framework is used to schedule large models to perform intelligent reasoning on multi-source heterogeneous basic data, automatically mining explicit and implicit semantic relationships between entities and generating standardized ontology information. This replaces manually predefined ontology patterns, breaking through domain limitations and improving flexibility, while reducing manual costs through automated semantic parsing. Combined with the Kettle tool, efficient data preprocessing integrating a lake repository and the dynamic graph update mechanism of the LLM KnowledgeGraph Builder are implemented. While ensuring semantic consistency of entity attributes, incremental data is processed in real time and the graph structure is adaptively adjusted, significantly enhancing dynamic adaptability and operational efficiency. Finally, relying on the distributed storage and automated verification capabilities of the Neo4j graph database, real-time structured storage and visual monitoring of massive heterogeneous data are supported, comprehensively optimizing scalability and quality controllability, thereby improving the flexibility, automation, efficiency, semantic consistency, dynamic adaptability, and scalability of knowledge graph generation.

[0070] Please refer to Figures 1 to 3 As shown, a preferred embodiment of the knowledge graph generation method combining dynamic semantic association mining of the present invention includes the following steps:

[0071] Step S1: Collect basic data, including relational and non-relational data, for generating the knowledge graph, and preprocess the basic data using the Kettle tool;

[0072] Step S2: Using the LangChain framework in conjunction with large models (such as deepseek, Qianwen, ChatGPT, etc.) to perform reasoning on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association.

[0073] Step S3: Initialize a knowledge graph using LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information;

[0074] Step S4: Store the knowledge graph in real time to the graph database Neo4j, and use the Neo4jBLOOM tool to visualize the knowledge graph stored in the graph database Neo4j.

[0075] Step S1 specifically involves:

[0076] The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs.

[0077] The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

[0078] By collecting and preprocessing relational data (such as database data, GIS vector data, and Excel files) and non-relational data (such as text, images, and PDFs), and using the Kettle tool to perform extraction, cleaning, deduplication, and format unification operations, a "lake-warehouse integration" processing was achieved. This supports the comprehensive processing of heterogeneous data sources, avoids the dependence on a single data type in traditional methods, and improves the data foundation quality of the knowledge graph. Automated preprocessing reduces the impact of erroneous data, ensures the consistency of input for subsequent steps, reduces the need for manual intervention in knowledge graph construction, improves overall efficiency, and solves the problems of scattered data sources and incomplete cleaning in knowledge graph construction.

[0079] Step S2 specifically involves:

[0080] A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association.

[0081] The semantic parsing process is specifically as follows:

[0082] A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

[0083] In practice, the topological structure of the ontology information is dynamically maintained based on an incremental update mechanism, including: retrieving associated existing knowledge nodes (ontology information) through vector similarity; using conflict resolution algorithms to process contradictory data; and using graph neural networks to optimize the graph path of the ontology information.

[0084] Using the LangChain framework combined with a large model as the inference engine, semantic parsing is performed on the preprocessed basic data. Based on a preset Prompt template, ontological information (including explicit and implicit relationships) of entities, relations, and attributes is dynamically mined to generate a set of structured triples. The application of the large model can identify implicit relationships (such as inferring implicit entity associations from unstructured text), while traditional rule engines or statistical methods only handle explicit relationships. This significantly increases the information richness and accuracy of the knowledge graph. The semantic parsing process is automated based on the Prompt template, eliminating the need for manual rule definition. It can adapt to the dynamic changes of multi-source data, improving the flexibility and scalability of the solution. Through dynamic semantic association mining, end-to-end inference is performed using an LLM (Large Language Model), which solves the problems of shallow semantic mining and strong reliance on manual intervention in existing knowledge graphs.

[0085] Step S3 specifically involves:

[0086] A knowledge graph is initialized using LLM Knowledge Graph Builder. The ontology information is semantically normalized. LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure of the knowledge graph, including node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

[0087] The knowledge graph is initialized using the LLM Knowledge Graph Builder. After semantic normalization, the ontology information is converted into a graph structure (including node types, attributes, relation constraints, etc.) and updated in real time based on newly mined ontology information. Semantic normalization ensures the consistency and standardization of ontology information and reduces the risk of conflicts introduced by heterogeneous data. The dynamic update mechanism enables the knowledge graph to continuously absorb new data, avoiding the long cycle and high cost of rebuilding the graph in traditional methods, and improving applicability.

[0088] By combining semantic normalization processing with the real-time parsing capabilities of LLM Knowledge Graph Builder, newly mined ontology information can be seamlessly updated to update the graph structure. Compared with traditional methods that periodically rebuild the graph (such as T+1 updates), it supports streaming data response and effectively reduces decision latency.

[0089] Step S4 specifically involves:

[0090] The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j performs structured storage of the knowledge graph and visualizes the knowledge graph stored in Neo4j using the Neo4jBLOOM tool.

[0091] By storing the knowledge graph in real-time to the Neo4j graph database (including data type validation, integrity validation, and constraint validation) and visualizing it using the Neo4jBLOOM tool, the Neo4j validation mechanism ensures data integrity and consistency. Real-time storage supports fast indexing of large-scale graphs, improving query performance. The integration of the Neo4jBLOOM tool enables interactive visualization, allowing users to intuitively understand the graph structure and relationships, enhancing decision support capabilities. This end-to-end storage and visualization design is practical and technologically innovative (such as seamlessly integrating validation mechanisms with visualization), solving the pain points of lagging graph display and complex user operations.

[0092] By integrating multi-stage technologies (such as Kettle data processing, LangChain inference, large model mining, LLMKnowledge Graph Builder construction, Neo4j storage and visualization), an efficient end-to-end process is formed. The integration of various tools (such as LangChain and large models) optimizes the data flow, reduces redundant computation, and achieves a high degree of automation in knowledge graph construction, adapting to big data scenarios (such as heterogeneous data in the financial and medical fields). Dynamic updates and real-time storage mechanisms support long-term applications.

[0093] By integrating the Kettle tool, the LangChain framework, and large models, this system innovatively achieves dynamic semantic deep mining of multi-source heterogeneous data (including GIS spatial information and unstructured text / images), breaking through the technical bottlenecks of traditional methods in implicit relationship identification and real-time updates. Its automated graph construction mechanism based on LLM Knowledge Graph Builder and Neo4j's three-level verification and storage architecture not only significantly improves the cognitive depth and structural accuracy of the knowledge graph (such as inferring disease concurrency patterns or financial risk correlations from complex data), but also supports industrial-grade real-time decision-making with streaming update responses. At the same time, the Neo4jBLOOM visualization tool greatly reduces the barrier to entry, ultimately forming an end-to-end solution integrating efficient data processing, intelligent semantic parsing, dynamic graph construction, and reliable visualization.

[0094] A preferred embodiment of the knowledge graph generation system combining dynamic semantic association mining according to the present invention includes the following modules:

[0095] The basic data acquisition module is used to collect basic data, including relational and non-relational data, for generating knowledge graphs, and to preprocess the basic data using the Kettle tool.

[0096] The ontology information mining module is used to perform reasoning on the basic data by combining the LangChain framework with large models (such as deepseek, Qianwen, ChatGPT, etc.) to dynamically mine ontology information, including entities, relationships and attributes, from the basic data through semantic association.

[0097] The knowledge graph generation module is used to initialize a knowledge graph through LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information.

[0098] The knowledge graph storage and display module is used to store the knowledge graph in real time to the graph database Neo4j, and to visualize the knowledge graph stored in the graph database Neo4j using the Neo4jBLOOM tool.

[0099] The basic data acquisition module is specifically used for:

[0100] The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs.

[0101] The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

[0102] By collecting and preprocessing relational data (such as database data, GIS vector data, and Excel files) and non-relational data (such as text, images, and PDFs), and using the Kettle tool to perform extraction, cleaning, deduplication, and format unification operations, a "lake-warehouse integration" processing was achieved. This supports the comprehensive processing of heterogeneous data sources, avoids the dependence on a single data type in traditional methods, and improves the data foundation quality of the knowledge graph. Automated preprocessing reduces the impact of erroneous data, ensures the consistency of input for subsequent steps, reduces the need for manual intervention in knowledge graph construction, improves overall efficiency, and solves the problems of scattered data sources and incomplete cleaning in knowledge graph construction.

[0103] The ontology information mining module is specifically used for:

[0104] A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association.

[0105] The semantic parsing process is specifically as follows:

[0106] A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

[0107] In practice, the topological structure of the ontology information is dynamically maintained based on an incremental update mechanism, including: retrieving associated existing knowledge nodes (ontology information) through vector similarity; using conflict resolution algorithms to process contradictory data; and using graph neural networks to optimize the graph path of the ontology information.

[0108] Using the LangChain framework combined with a large model as the inference engine, semantic parsing is performed on the preprocessed basic data. Based on a preset Prompt template, ontological information (including explicit and implicit relationships) of entities, relations, and attributes is dynamically mined to generate a set of structured triples. The application of the large model can identify implicit relationships (such as inferring implicit entity associations from unstructured text), while traditional rule engines or statistical methods only handle explicit relationships. This significantly increases the information richness and accuracy of the knowledge graph. The semantic parsing process is automated based on the Prompt template, eliminating the need for manual rule definition. It can adapt to the dynamic changes of multi-source data, improving the flexibility and scalability of the solution. Through dynamic semantic association mining, end-to-end inference is performed using an LLM (Large Language Model), which solves the problems of shallow semantic mining and strong reliance on manual intervention in existing knowledge graphs.

[0109] The knowledge graph generation module is specifically used for:

[0110] A knowledge graph is initialized using LLM Knowledge Graph Builder. The ontology information is semantically normalized. LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure of the knowledge graph, including node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

[0111] The knowledge graph is initialized using the LLM Knowledge Graph Builder. After semantic normalization, the ontology information is converted into a graph structure (including node types, attributes, relation constraints, etc.) and updated in real time based on newly mined ontology information. Semantic normalization ensures the consistency and standardization of ontology information and reduces the risk of conflicts introduced by heterogeneous data. The dynamic update mechanism enables the knowledge graph to continuously absorb new data, avoiding the long cycle and high cost of rebuilding the graph in traditional methods, and improving applicability.

[0112] By combining semantic normalization processing with the real-time parsing capabilities of LLM Knowledge Graph Builder, newly mined ontology information can be seamlessly updated to update the graph structure. Compared with traditional methods that periodically rebuild the graph (such as T+1 updates), it supports streaming data response and effectively reduces decision latency.

[0113] The knowledge graph storage and display module is specifically used for:

[0114] The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j performs structured storage of the knowledge graph and visualizes the knowledge graph stored in Neo4j using the Neo4jBLOOM tool.

[0115] By storing the knowledge graph in real-time to the Neo4j graph database (including data type validation, integrity validation, and constraint validation) and visualizing it using the Neo4jBLOOM tool, the Neo4j validation mechanism ensures data integrity and consistency. Real-time storage supports fast indexing of large-scale graphs, improving query performance. The integration of the Neo4jBLOOM tool enables interactive visualization, allowing users to intuitively understand the graph structure and relationships, enhancing decision support capabilities. This end-to-end storage and visualization design is practical and technologically innovative (such as seamlessly integrating validation mechanisms with visualization), solving the pain points of lagging graph display and complex user operations.

[0116] By integrating multi-stage technologies (such as Kettle data processing, LangChain inference, large model mining, LLMKnowledge Graph Builder construction, Neo4j storage and visualization), an efficient end-to-end process is formed. The integration of various tools (such as LangChain and large models) optimizes the data flow, reduces redundant computation, and achieves a high degree of automation in knowledge graph construction, adapting to big data scenarios (such as heterogeneous data in the financial and medical fields). Dynamic updates and real-time storage mechanisms support long-term applications.

[0117] By integrating the Kettle tool, the LangChain framework, and large models, this system innovatively achieves dynamic semantic deep mining of multi-source heterogeneous data (including GIS spatial information and unstructured text / images), breaking through the technical bottlenecks of traditional methods in implicit relationship identification and real-time updates. Its automated graph construction mechanism based on LLM Knowledge Graph Builder and Neo4j's three-level verification and storage architecture not only significantly improves the cognitive depth and structural accuracy of the knowledge graph (such as inferring disease concurrency patterns or financial risk correlations from complex data), but also supports industrial-grade real-time decision-making with streaming update responses. At the same time, the Neo4jBLOOM visualization tool greatly reduces the barrier to entry, ultimately forming an end-to-end solution integrating efficient data processing, intelligent semantic parsing, dynamic graph construction, and reliable visualization.

[0118] In summary, the advantages of this invention are as follows:

[0119] 1. Basic data, including relational and non-relational data, is collected for generating the knowledge graph. This data is preprocessed using the Kettle tool. Then, the LangChain framework, combined with a large model, is used to reason about the basic data, dynamically mining ontology information, including entities, relationships, and attributes, through semantic associations. A knowledge graph is initialized using an LLM Knowledge Graph Builder, converting ontology information into the graph structure of the knowledge graph. The knowledge graph is dynamically updated based on newly mined ontology information. Finally, the knowledge graph is stored in real-time in the Neo4j graph database, and visualized using the Neo4jBLOOM tool. In essence, the LangChain framework schedules a large model to perform intelligent reasoning on multi-source heterogeneous basic data, automatically mining explicit and implicit semantic associations between entities and generating standardized ontology information, replacing manually predefined ontology patterns. This approach overcomes domain limitations, improves flexibility, and reduces manual costs through automated semantic parsing. The integration of the Kettle tool with the lake repository achieves efficient data preprocessing and LLM Knowledge Graph integration. Builder's dynamic graph update mechanism ensures semantic consistency of entity attributes while processing incremental data in real time and adaptively adjusting the graph structure, significantly enhancing dynamic adaptability and operational efficiency. Ultimately, relying on the distributed storage and automated verification capabilities of the Neo4j graph database, it supports real-time structured storage and visual monitoring of massive heterogeneous data, comprehensively optimizing scalability and quality controllability. In other words, by integrating dynamic semantic association mining and an end-to-end automated toolchain, it significantly improves the overall performance of knowledge graph generation, ultimately greatly enhancing the flexibility, automation, efficiency, semantic consistency, dynamic adaptability, and scalability of knowledge graph generation.

[0120] 2. By collecting and preprocessing relational data (such as database data, GIS vector data, and Excel files) and non-relational data (such as text, images, and PDFs), and using the Kettle tool to perform extraction, cleaning, deduplication, and format unification operations, "lake-warehouse integration" processing was achieved. This supports the comprehensive processing of heterogeneous data sources, avoids the dependence on a single data type in traditional methods, and improves the data foundation quality of the knowledge graph. Automated preprocessing reduces the impact of erroneous data, ensures the consistency of input in subsequent steps, reduces the need for manual intervention in knowledge graph construction, improves overall efficiency, and solves the problems of scattered data sources and incomplete cleaning in knowledge graph construction.

[0121] 3. Using the LangChain framework combined with a large model as the inference engine, semantic parsing is performed on the preprocessed basic data. Based on a preset Prompt template, ontological information (including explicit and implicit relationships) of entities, relations, and attributes is dynamically mined to generate a set of structured triples. The application of the large model can identify implicit relationships (such as inferring implicit entity associations from unstructured text), while traditional rule engines or statistical methods only handle explicit relationships. This significantly increases the information richness and accuracy of the knowledge graph. The semantic parsing process is automated based on the Prompt template, eliminating the need for manual rule definition. It can adapt to the dynamic changes of multi-source data, improving the flexibility and scalability of the solution. Through dynamic semantic association mining, end-to-end inference is performed using an LLM (Large Language Model), solving the problems of shallow semantic mining and strong reliance on manual intervention in existing knowledge graphs.

[0122] 4. The knowledge graph is initialized using the LLM Knowledge Graph Builder. After semantic normalization, the ontology information is converted into a graph structure (including node types, attributes, relation constraints, etc.) and updated in real time based on newly mined ontology information. Semantic normalization ensures the consistency and standardization of ontology information and reduces the risk of conflicts introduced by heterogeneous data. The dynamic update mechanism enables the knowledge graph to continuously absorb new data, avoiding the long cycle and high cost of rebuilding the graph in traditional methods and improving applicability.

[0123] 5. By storing the knowledge graph in real-time to the Neo4j graph database (including data type validation, integrity validation, and constraint validation) and visualizing it using the Neo4jBLOOM tool, Neo4j's validation mechanism ensures data integrity and consistency. Real-time storage supports fast indexing of large-scale graphs, improving query performance. The integration of the Neo4jBLOOM tool enables interactive visualization, facilitating users' intuitive understanding of the graph structure and relationships, and enhancing decision support capabilities. This end-to-end storage and visualization design is practical and technologically innovative (such as seamlessly integrating validation mechanisms with visualization), solving the pain points of lagging graph display and complex user operations.

[0124] 6. By integrating multi-stage technologies (such as Kettle data processing, LangChain inference, large model mining, LLMKnowledge Graph Builder construction, Neo4j storage and visualization), an efficient end-to-end process is formed; the integration of various tools (such as LangChain and large models) optimizes the data flow, reduces redundant calculations, realizes a high degree of automation in knowledge graph construction, adapts to big data scenarios (such as heterogeneous data in the financial and medical fields), and the dynamic update and real-time storage mechanism supports long-term applications.

[0125] 7. By combining semantic normalization processing with the real-time parsing capabilities of LLM Knowledge Graph Builder, newly mined ontology information can be seamlessly updated to update the graph structure. Compared with traditional methods that periodically rebuild the graph (such as T+1 update), it supports streaming data response and effectively reduces decision latency.

[0126] 8. By integrating the Kettle tool, LangChain framework, and large model, it innovatively achieves dynamic semantic deep mining of multi-source heterogeneous data (including GIS spatial information and unstructured text / images), breaking through the technical bottlenecks of traditional methods in implicit relationship identification and real-time updates. Its automated graph construction mechanism based on LLM Knowledge Graph Builder and Neo4j's three-level verification storage architecture not only significantly improves the cognitive depth and structural accuracy of knowledge graphs (such as inferring disease concurrency patterns or financial risk correlations from complex data), but also supports industrial-grade real-time decision-making with streaming update response. At the same time, the Neo4jBLOOM visualization tool greatly reduces the usage threshold, ultimately forming an end-to-end solution integrating efficient data processing, intelligent semantic parsing, dynamic graph construction, and reliable visualization.

[0127] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A knowledge graph generation method combining dynamic semantic association mining, characterized in that: Includes the following steps: Step S1: Collect basic data, including relational and non-relational data, for generating the knowledge graph, and preprocess the basic data using the Kettle tool; Step S2: Using the LangChain framework in conjunction with a large model, reason about each of the basic data to dynamically mine ontological information including entities, relationships, and attributes from each of the basic data through semantic association. Step S3: Initialize a knowledge graph using LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information; Step S4: Store the knowledge graph in real time to the graph database Neo4j, and use the Neo4jBLOOM tool to visualize the knowledge graph stored in the graph database Neo4j.

2. The knowledge graph generation method combining dynamic semantic association mining as described in claim 1, characterized in that: Step S1 specifically involves: The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs. The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

3. The knowledge graph generation method combining dynamic semantic association mining as described in claim 1, characterized in that: Step S2 specifically involves: A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association. The semantic parsing process is specifically as follows: A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

4. The knowledge graph generation method combining dynamic semantic association mining as described in claim 1, characterized in that: Step S3 specifically involves: A knowledge graph is initialized using an LLM Knowledge Graph Builder. The ontology information is semantically normalized. The LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure that includes node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

5. The knowledge graph generation method combining dynamic semantic association mining as described in claim 1, characterized in that: Step S4 specifically involves: The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j stores the knowledge graph in a structured manner and uses the Neo4jBLOOM tool to visualize the knowledge graph stored in Neo4j.

6. A knowledge graph generation system combining dynamic semantic association mining, characterized in that: Includes the following modules: The basic data acquisition module is used to collect basic data, including relational and non-relational data, for generating knowledge graphs, and to preprocess the basic data using the Kettle tool. The ontology information mining module is used to perform reasoning on the basic data by combining the LangChain framework with a large model, so as to dynamically mine ontology information including entities, relationships and attributes from the basic data through semantic association. The knowledge graph generation module is used to initialize a knowledge graph through LLM Knowledge Graph Builder, convert the ontology information into the graph structure of the knowledge graph, and dynamically update the knowledge graph based on the newly mined ontology information. The knowledge graph storage and display module is used to store the knowledge graph in real time to the graph database Neo4j, and to visualize the knowledge graph stored in the graph database Neo4j using the Neo4jBLOOM tool.

7. The knowledge graph generation system combining dynamic semantic association mining as described in claim 6, characterized in that: The basic data acquisition module is specifically used for: The basic data collected for generating the knowledge graph includes both relational and non-relational data. The relational data includes at least database data, GIS vector data, and Excel files, while the non-relational data includes at least text, images, and PDFs. The basic data is preprocessed using the Kettle tool, including extraction and cleaning, to remove duplicate data, remove erroneous data, and unify the format, thereby achieving lake warehouse integration.

8. A knowledge graph generation system combining dynamic semantic association mining as described in claim 6, characterized in that: The ontology information mining module is specifically used for: A multi-source data processing pipeline is constructed using the LangChain framework. Heterogeneous basic data are loaded through each of the data processing pipelines. A pre-trained large model is used as an inference engine to perform semantic parsing on each of the basic data, so as to dynamically mine ontological information including entities, relationships and attributes from each of the basic data through semantic association. The semantic parsing process is specifically as follows: A structured query instruction is generated based on a preset Prompt template. Based on the structured query instruction, entities and type tags in the basic data are identified. The explicit and implicit relationships between the entities are mined and the attributes are described. Then, a set of triples of <entity-relationship-attribute> is generated, which is the ontology information.

9. A knowledge graph generation system combining dynamic semantic association mining as described in claim 6, characterized in that: The knowledge graph generation module is specifically used for: A knowledge graph is initialized using an LLM Knowledge Graph Builder. The ontology information is semantically normalized. The LLM Knowledge Graph Builder parses the semantically normalized ontology information using a large language model and uses it as input to convert it into a graph structure that includes node types, node attributes, relation attributes, and relation constraints. The knowledge graph is then dynamically updated based on newly mined ontology information.

10. A knowledge graph generation system combining dynamic semantic association mining as described in claim 6, characterized in that: The knowledge graph storage and display module is specifically used for: The configuration of the graph database Neo4j includes connection parameters such as the database server address, port number, and authorization authentication information. Based on the connection parameters, the knowledge graph is transmitted to the graph database Neo4j in real time. After performing at least data type verification, integrity verification, and constraint verification on the received knowledge graph, Neo4j stores the knowledge graph in a structured manner and uses the Neo4jBLOOM tool to visualize the knowledge graph stored in Neo4j.

Images