General knowledge graph construction method, system, equipment and medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a general knowledge graph construction method, system and device and a medium, and the method comprises the steps: constructing a general framework based on a modular hierarchical architecture, the general framework comprises a basic data layer, a model service layer and a knowledge service layer, and each layer is decoupled through a standardized interface; field self-adaptive fine tuning is carried out on the core extraction model in the model service layer through a field adapter, and efficient adaptation to a specific field is achieved; based on a dynamic Schema mapping mechanism, generating a knowledge extraction task according to an entity and relationship type defined by a user; calling a core extraction model to execute a task, and extracting domain knowledge from the multi-source heterogeneous data; knowledge is stored in the basic data layer, and derivative knowledge is generated by using an inference model of the model service layer, so that a complete knowledge graph is constructed. According to the method, the construction efficiency and the cross-domain adaptation capability are remarkably improved, and the deployment period is greatly shortened.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a general knowledge graph construction method, system, device, and medium. Background Technology

[0002] Knowledge graphs, as structured semantic knowledge bases, are the core infrastructure supporting intelligent search and decision-making reasoning. Currently, building domain-specific knowledge graphs faces significant challenges. First, data sources are highly heterogeneous, encompassing various forms such as databases, text, and streaming data. Traditional preprocessing methods lack versatility, requiring customized transformation rules for each data source, leading to high integration costs and low efficiency. Second, the construction process is rigid, making it difficult to balance generality and domain specificity: general models lack sufficient extraction accuracy in specialized domains (such as finance and healthcare), while domain-customized models suffer from "catastrophic forgetting" and cannot be quickly migrated. Furthermore, system architectures are often tightly coupled with specific storage engines or models, lacking modular design, making component replacement and cross-domain reuse extremely difficult. These problems severely restrict the efficiency of knowledge graph construction and its widespread application. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a general knowledge graph construction method, system, device and medium to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a general knowledge graph construction method, comprising: Construct a general knowledge graph construction framework, which is implemented through a modular layered architecture. The architecture includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled from each other through standardized interfaces. The core extracted model of the model service layer in the general framework is adaptively fine-tuned by a domain adapter to adapt the general framework to the domain corresponding to the domain adapter. Through a dynamic schema mapping mechanism, a task is generated to extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain. The core extraction model is invoked to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; The domain knowledge is stored in the basic data layer. Based on the basic data layer, the reasoning model encapsulated in the model service layer is used to generate derived knowledge and construct the knowledge graph.

[0005] In one optional implementation, a general knowledge graph construction framework is built, including: A five-layer modular architecture is constructed, including: a basic data layer for storing knowledge triples and raw data; an entity aggregation layer for disambiguating and fusing entities in the basic data layer; a model service layer for encapsulating knowledge processing models and providing services through standardized interfaces; a knowledge service layer for providing knowledge query, retrieval, and reasoning services; and an application adaptation layer for providing configurable domain templates to quickly adapt to upper-layer applications. The layers communicate and exchange data through defined application programming interfaces, enabling the pluggable and independent replacement of technical components; When the entities are grouped into a single layer for entity disambiguation, an entity discrimination model trained on a generative adversarial network is used to distinguish different entities with similar or identical names. The domain configuration templates provided by the application adaptation layer encapsulate the entity types, relationship types, business rules, and visual configurations for a specific domain.

[0006] In an optional implementation, the method further includes: A structured configuration file based on JSON or YAML is used, which explicitly defines: pipeline components, used to declare the various knowledge processing models included in the processing pipeline; execution topology, used to define the execution order and data flow between the models; and parameter configuration, used to configure the specific parameters required for the execution of each model. The model service layer parses the structured configuration file, calls the corresponding model services in sequence according to the defined topology, and uses the output of the upstream model as the input of the downstream model, thereby automatically executing the complete knowledge processing flow.

[0007] In an optional implementation, the core extraction model of the model service layer in the general framework is subjected to domain-adaptive fine-tuning via a domain adapter, including: The large language model, which serves as the core extraction model, is continuously pre-trained using a pre-trained corpus containing corpora from multiple domains. When adapting to a new domain, a parameter-efficient fine-tuning method is adopted, which fine-tunes the basic large language model by inserting a domain adapter, while keeping the core parameters of the basic large language model unchanged; The training of the domain adapter adopts a progressive domain expansion strategy. By calculating the semantic similarity between the new domain and the existing domain, the most relevant existing domain adapter is selected as the initialization basis for incremental training. Among them, the domain adapter is a pluggable, lightweight neural network module specifically designed to adapt a general model to a specific domain; The efficient parameter fine-tuning method specifically adopts any one of LoRA, adapter, or prefix fine-tuning methods; The progressive domain expansion strategy specifically includes: measuring domain similarity by calculating the cosine similarity of embedding vectors between domains, and selecting the existing domain adapter with the highest similarity as the initialization parameter for training the new domain.

[0008] In an optional implementation, a knowledge extraction task is dynamically generated based on the entity and relationship types defined by the user for a specific domain, using a dynamic schema mapping mechanism. This includes: A predefined instruction template is provided, which includes task instruction fields, schema list fields, and input data placeholders. Receive the entity types and relationship types defined by the user for a specific domain, and populate them into the Schema list field of the instruction template; Acquire multi-source heterogeneous data and perform general hierarchical preprocessing on the multi-source heterogeneous data to convert structured, semi-structured and unstructured data into a unified standardized format; The standardized data is filled into the input data placeholders to generate structured task description data.

[0009] In one optional implementation, multi-source heterogeneous data is acquired, and general hierarchical preprocessing is performed on the multi-source heterogeneous data, including: Multi-source heterogeneous data is accessed through a data source adaptation layer, which is implemented through a configurable API gateway. Relational databases, unstructured documents, and streaming data are accessed using JDBC protocol, Apache Tika parser, and Kafka message queue, respectively. The data standardization layer performs unified processing on the incoming data, including data cleaning and deduplication based on a common metadata model, and converts all data into JSON-LD format; The data augmentation layer performs semantic enhancement on the standardized data, including generating text semantic vectors using pre-trained language models and generating feature vectors for structured data using knowledge embedding techniques.

[0010] In an optional implementation, the domain knowledge is stored in the basic data layer, and derived knowledge is generated based on the basic data layer using the inference model encapsulated in the model service layer, including: The extracted domain knowledge is stored in the graph database of the basic data layer in the form of triples; Through the reasoning model encapsulated in the model service layer, logical reasoning or vector calculation is performed on the triples stored in the basic data layer to generate implicit, derived knowledge that is not explicitly expressed in the original data. The derived knowledge is written back to the basic data layer, and together with the original extracted knowledge, it forms a complete knowledge graph. Enhanced knowledge query and reasoning services are then provided to the outside world through the knowledge service layer. The reasoning models include a logical reasoning model based on predefined rules and a vector reasoning model based on knowledge graph embedding representation.

[0011] Secondly, the present invention provides a general knowledge graph construction system, comprising: The framework building module is used to build a general knowledge graph construction framework. This framework is implemented through a modular layered architecture, which includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled from each other through standardized interfaces. The domain fine-tuning module is used to perform domain adaptive fine-tuning on the core extracted model of the model service layer in the general framework through the domain adapter, so as to adapt the general framework to the domain corresponding to the domain adapter. The task generation module is used to generate tasks that extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain through a dynamic schema mapping mechanism. The knowledge extraction module is used to call the core extraction model to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; The knowledge processing module is used to store the domain knowledge in the basic data layer, and based on the basic data layer, generate derived knowledge using the reasoning model encapsulated in the model service layer to construct the knowledge graph.

[0012] Thirdly, a device is provided, comprising: The memory is used to store the general knowledge graph building program; A processor is configured to implement the steps of the general knowledge graph construction method as provided in the first aspect when executing the general knowledge graph construction program.

[0013] Fourthly, a computer-readable medium is provided, on which a general knowledge graph construction program is stored, which, when executed by a processor, implements the steps of the general knowledge graph construction method provided in the first aspect.

[0014] The beneficial effects of this invention are as follows: the general knowledge graph construction method, system, device, and medium provided by this invention achieve seamless access and standardization of multi-source heterogeneous data through a general hierarchical preprocessing mechanism, significantly reducing data integration costs. By employing a "domain adapter" and dynamic schema mapping, high-precision extraction can be maintained even under zero-sample / few-sample conditions, effectively resolving the contradiction between general models and domain-specific requirements. The modular, layered architecture gives the system high flexibility, supporting plug-and-play replacement of storage engines and processing models, significantly shortening cross-domain deployment cycles. These innovations collectively drive the transformation of knowledge graphs from customized projects to standardized services. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0021] The general knowledge graph construction method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the general knowledge graph construction system runs on the computer device.

[0022] Figure 1This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a general knowledge graph construction system. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.

[0023] like Figure 1 As shown, the method includes: S1. Construct a general knowledge graph construction framework, which is implemented through a modular layered architecture. The architecture includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled through standardized interfaces. S2. The core extraction model of the model service layer in the general framework is adaptively fine-tuned by the domain adapter to adapt the general framework to the domain corresponding to the domain adapter; S3. Through a dynamic schema mapping mechanism, a task is generated to extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain; S4. Invoke the core extraction model to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; S5. Store the domain knowledge in the basic data layer, and based on the basic data layer, generate derived knowledge using the reasoning model encapsulated in the model service layer to construct the knowledge graph.

[0024] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0025] 1. Overall Architecture Design The system adopts a five-layer decoupled design, with each layer exchanging data and calling services through a unified RESTful API. This design allows any technical component in any layer to be replaced without modifying the code in other layers, achieving a truly pluggable architecture.

[0026] 2. Specific implementation at each level The basic data layer uses the Neo4j graph database to store (entity, relation, entity) triples, while using the HDFS distributed file system to back up the original data. By defining a unified data access interface, the underlying storage engine can be replaced with other graph databases such as NebulaGraph and JanusGraph.

[0027] Entity Merging Layer: Implements the core entity disambiguation and fusion functions. Specifically, it employs an entity discrimination model based on Generative Adversarial Networks (GANs): the generator is responsible for generating indistinguishable negative samples (such as entity pairs with the same name but different actual entities), while the discriminator learns to distinguish these samples. Through adversarial training, the model can accurately identify entities with "same name but different actual entities" (such as Apple Inc. and apple fruit) and "different name but same actual entities" (such as full name and abbreviation), achieving accurate entity merging.

[0028] Model Service Layer: This layer encapsulates algorithmic models such as entity extraction, relation classification, and knowledge reasoning into independent microservices, providing standardized API interfaces. The key innovation lies in the introduction of a model pipeline description language: through YAML-formatted configuration files, users can declare pipeline components (such as bert_ner and relation_classifier), define the execution topology (ner -> relation_classifier), and configure the parameters of each component. After parsing this file, the system automatically schedules the corresponding model services according to the topology order, forming an end-to-end knowledge processing pipeline.

[0029] Knowledge Service Layer: Based on the knowledge stored in the basic data layer, this layer provides Cypher query interfaces, semantic retrieval APIs, and visualization services. It integrates derived knowledge generated by the model service layer (such as implicit relationships inferred through rules) to offer an enhanced knowledge query experience.

[0030] Application Adaptation Layer: Provides reusable domain configuration templates. Taking the financial risk control template as an example, it predefines entity relationship types such as "enterprise," "individual," "holding," and "guarantee"; encapsulates business rules such as "identification of the same controller"; and includes a visual configuration of risk transmission paths. Users can quickly deploy their own risk control knowledge graph with slight adjustments through the visual interface.

[0031] 3. Key Process Examples Taking the construction of a medical diagnostic atlas as an example, when a user selects the "Medical" template in the application adaptation layer, the system loads predefined schemas for diseases, symptoms, and drugs. Through the YAML configuration file in the model service layer, the "medical entity recognition model" and the "diagnosis-treatment relationship classification model" are combined to form an extraction pipeline. The entity grouping layer uses a trained medical entity discrimination model to accurately distinguish between variations and aliases of different disease names. Finally, the knowledge service layer provides a reliable atlas query service for the clinical decision-making system.

[0032] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0033] S201. Basic Concepts and Positioning of Domain Adapters A domain adapter is a pluggable, lightweight neural network module whose core function is to inject domain-specific expertise into a basic large language model (such as LLaMA-13B) without changing its main parameters. Essentially, it's a parameter-efficient fine-tuning interface, like a "domain expert plugin." By loading different adapters (such as a financial adapter or a medical adapter), the system can enable the same basic model to perform knowledge extraction tasks in different domains.

[0034] S202. Basic Pre-training Phase First, a large-scale, multi-domain general pre-training corpus is constructed, containing approximately 0.4B tokens and covering at least five core domains, including finance, healthcare, and law. This corpus is then used to continuously pre-train a selected foundational large language model (such as LLaMA-13B). The goal of this stage is to establish a solid foundation in general language understanding and multi-domain knowledge, preparing the model for subsequent refined domain adaptation.

[0035] S203. Implementation of Efficient Parameter Fine-Tuning When a new domain (such as "medical") needs to be adapted to the system, a parameter-efficient fine-tuning method is used. In practice, the LoRA method is preferred.

[0036] Operational procedure: Insert the LoRA adapter in parallel into the attention mechanism module of the Transformer layer of the base model (such as LLaMA-13B). During fine-tuning, freeze all original parameters of the base model and train only the small number of low-rank matrix parameters introduced by the LoRA adapter.

[0037] Data and Training: Using labeled or unlabeled text data from the target domain (medical), perform supervised or self-supervised training and update the LoRA adapter parameters. After training, save the domain-specific adapter file (typically only a few megabytes), while the base model remains unchanged.

[0038] S204. Implementation of a Progressive Domain Expansion Strategy This is the key innovation of this solution. When adapting to a completely new field (such as "biomedicine"), a progressive scaling strategy is adopted to improve training efficiency and effectiveness: Similarity calculation: First, the Sentence-BERT model is used to convert the corpora of the new domain (biomedicine) and all existing domains (such as finance and healthcare) into domain embedding vectors respectively. Then, the cosine similarity between the new domain embedding and each existing domain embedding is calculated.

[0039] Adapter initialization: Select the existing domain adapter with the highest similarity to the new domain as the initialization basis. For example, if the calculation finds that the cosine similarity between the "biomedicine" and "medical" domains is the highest (e.g., 0.82), the system will automatically copy the parameters of the "medical adapter" as the initial state of the "biomedicine adapter".

[0040] Incremental training: Using data from the "biomedical" field, incremental training is performed based on the pre-initialized adapter. This strategy effectively utilizes existing domain-specific knowledge, achieving smooth knowledge transfer and accumulation, avoiding training from scratch, and significantly mitigating the "catastrophic forgetting" problem.

[0041] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0042] S301. Predefinition and configuration of instruction templates The system predefines a structured instruction template in JSON format, which contains three core fields: instruction: The task instruction field has a fixed content of "Extract all entities and relationships of the specified type from the following text".

[0043] schema: A schema list field used to dynamically receive user-defined entity and relationship types.

[0044] input: Input data placeholder, used to receive standardized data to be processed.

[0045] S302. Execution flow of dynamic schema mapping When a user needs to build a knowledge graph for a specific domain (such as "medical treatment"), they first define the entity types (such as "disease", "symptom", "medicine") and relationship types (such as "cause" and "treatment") required for that domain through the configuration interface provided by the system. After receiving the domain schema, the system automatically populates it into the schema field of the instruction template.

[0046] S303. Unified Preprocessing of Multi-Source Data The data source adaptation layer enables multi-source data access through a configurable API gateway: For hospital relational databases (such as MySQL), structured data such as patient information tables and diagnostic record tables can be directly extracted by configuring a JDBC connector.

[0047] For unstructured PDF documents such as clinical medical records and medical literature, the Apache Tika parser is used to extract the text content.

[0048] Streaming data generated by real-time medical monitoring devices is efficiently accessed through a Kafka message queue.

[0049] The data standardization layer establishes a common metadata model and performs standardization processing on all accessed data: Perform data cleaning: validate the format of fields such as date and ID number using regular expressions.

[0050] The SimHash algorithm is used to approximate deduplication of large-scale medical record texts.

[0051] Finally, all data is converted into JSON-LD format and associated with metadata tags such as data source and credibility weight.

[0052] The data augmentation layer performs semantic augmentation on the standardized data: For medical record text data, a 768-dimensional semantic vector is generated using the BERT-base model.

[0053] For structured data such as test index tables, the TransE algorithm is used to generate feature vector representations.

[0054] S304. Generation of Task Description The standardized JSON-LD data obtained after the above three-level preprocessing is used to populate the input field of the instruction template, generating a complete structured task description. This description will serve as input to the domain-adaptive model, guiding the model to accurately perform knowledge extraction tasks in the medical field.

[0055] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0056] S401. Model Loading and Initialization Once the system receives the generated structured task description, it first automatically loads the corresponding domain adapter in the model service layer based on the domain information (such as schema type) implicit in the task description. For example, when the system detects that the schema contains types such as "disease" or "symptom", it loads the pre-trained "medical domain adapter" and combines it with the basic large language model (such as LLaMA-13B) to form a domain-specific core extraction model.

[0057] S402. Task Analysis and Instruction Understanding The core extraction model parses the structured task description: it reads the instruction field to understand that the goal of this task is to perform joint entity and relation extraction. It parses the schema field to clarify the set of entity types to be identified (e.g., {"disease", "symptom", "medicine"}) and the set of relation types (e.g., {"cause", "treatment"}). It then obtains the preprocessed, standardized data from the input field, ready for knowledge extraction.

[0058] S403. Two-stage joint extraction execution The model employs a two-stage architecture of "entity recognition - relation classification" to perform the extraction task: Phase 1: Entity Recognition The input text is encoded using a model that incorporates a domain adapter to generate a context-aware character representation.

[0059] The entity boundaries are identified through the Conditional Random Field (CRF) decoding layer, and all candidate entities and their types are output.

[0060] For example, from the text "diabetic patients may develop retinopathy", we can accurately identify "diabetes" (disease) and "retinopathy" (disease).

[0061] Phase Two: Relationship Classification The entities identified in the first stage are paired up to generate candidate entity pairs.

[0062] For each entity pair, a relational attention mechanism is used to analyze the contextual semantic relationships between entities.

[0063] Based on attention weights and semantic features, the classifier determines whether a predefined relationship exists between entity pairs.

[0064] Continuing with the previous example, for the entity pair ("diabetes", "retinopathy"), the model identifies a "cause" relationship.

[0065] S404. Few-shot suggestion enhancement mechanism In zero-sample / few-sample scenarios (such as processing novel diseases not present in the training data), the system automatically initiates the Few-Shot Prompting mechanism: Retrieve 1-3 samples from the example library that are semantically similar to the current input.

[0066] These sample input-output pairs are used as demonstration examples and inserted before the task description to form an enhanced cue.

[0067] By leveraging the contextual learning capabilities of large models, the extraction performance of unfamiliar schemas is significantly improved.

[0068] S405. Global Consistency Optimization After the basic extraction is completed, the system further employs a graph neural network (GNN) for globally consistent inference: Construct a local subgraph from the entities and relations extracted from the current sentence.

[0069] By combining relevant entities already existing in the knowledge graph, information dissemination and reasoning are carried out on the global graph.

[0070] Detect and correct potential entity ambiguities and relational contradictions in the local extraction results to ensure the logical consistency between the newly extracted knowledge and the existing graph.

[0071] S406. Results Processing and Output Finally, the model organizes the extracted results into a standardized (entity, relation, entity) triple format and attaches a confidence score.

[0072] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0073] S501. Knowledge Storage Implementation The extracted domain knowledge is stored in the graph database of the basic data layer as (entity, relation, entity) triples. In specific implementation: Using Neo4j graph database as the storage engine, a knowledge network with a basic "entity-relationship-entity" topology is established.

[0074] Each entity node contains attribute fields such as "Entity ID", "Entity Name", "Entity Type", "Creation Time", and "Data Source".

[0075] Each relation edge contains attributes such as "relation type", "confidence level", and "extraction model version".

[0076] The extracted medical knowledge, such as triples ("diabetes", "cause", "retinopathy"), is persistently stored through the batch import interface.

[0077] S502. Implementation of Derivative Knowledge Generation By using the inference model encapsulated in the model service layer, deep analysis is performed on the triples stored in the basic data layer to generate implicit knowledge: Implementation of a logical reasoning model based on predefined rules: Establish a rule base in the medical field, containing deductive rules such as "If disease A causes symptom B, and symptom B is a clinical manifestation of disease C, then disease A may be complicated by disease C"; Use the Drools rule engine to perform forward chain reasoning, automatically deriving new relationships from existing knowledge.

[0078] For example, from the known ("diabetes", "cause", "blurred vision") and ("blurred vision", "is a clinical manifestation", "cataract"), we can infer the derived knowledge ("diabetes", "may cause", "cataract").

[0079] The vector reasoning model based on knowledge graph embedding representation is implemented as follows: the TransE algorithm is used to map all entities and relations in the graph to a low-dimensional vector space; potential semantic associations are discovered through vector computation, such as calculating vector distance: vector("diabetes") + vector("complications") ≈ vector("retinopathy"); when the calculated similarity score exceeds the threshold of 0.85, hypothetical derived knowledge ("diabetes", "possible complications", "retinopathy") is generated.

[0080] S503. Implementation of Knowledge Integration and Service Provision The complete process of writing derived knowledge back to the underlying data layer includes: The confidence level of the derived knowledge obtained through reasoning is evaluated. The confidence level of the logical reasoning result is 1.0, and the confidence level of the vector reasoning result is calculated as a score.

[0081] The consistency verification module ensures that derived knowledge does not logically conflict with existing knowledge.

[0082] An asynchronous write mechanism is adopted, and qualified derived knowledge is written to the graph database in batches through a message queue.

[0083] The specific implementation of providing enhanced services to external entities through the knowledge service layer: Extend the Cypher query interface to support complex queries that include derived knowledge, such as "MATCH (d: disease)-[r: possible complications]->(c: disease) WHERE d.name='diabetes'RETURN c.name,r.confidence".

[0084] It provides a semantic retrieval API that not only returns exact matching results, but also inferred related entities and relationships.

[0085] Build a knowledge graph visualization service, using different colors and lines to distinguish between original extracted knowledge and derived knowledge, thereby enhancing interpretability.

[0086] In some embodiments, the general knowledge graph construction system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the general knowledge graph construction system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of general knowledge graph construction.

[0087] In this embodiment, the general knowledge graph construction system can be divided into multiple functional modules based on its functions, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0088] The framework building module is used to build a general knowledge graph construction framework. This framework is implemented through a modular layered architecture, which includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled from each other through standardized interfaces. The domain fine-tuning module is used to perform domain adaptive fine-tuning on the core extracted model of the model service layer in the general framework through the domain adapter, so as to adapt the general framework to the domain corresponding to the domain adapter. The task generation module is used to generate tasks that extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain through a dynamic schema mapping mechanism. The knowledge extraction module is used to call the core extraction model to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; The knowledge processing module is used to store the domain knowledge in the basic data layer, and based on the basic data layer, generate derived knowledge using the reasoning model encapsulated in the model service layer to construct the knowledge graph.

[0089] Figure 3 The general knowledge graph construction method provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0090] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0091] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0092] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0093] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0094] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0096] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0097] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0098] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0100] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A general knowledge graph construction method, characterized in that, include: Construct a general knowledge graph construction framework, which is implemented through a modular layered architecture. The architecture includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled from each other through standardized interfaces. The core extracted model of the model service layer in the general framework is adaptively fine-tuned by a domain adapter to adapt the general framework to the domain corresponding to the domain adapter. Through a dynamic schema mapping mechanism, a task is generated to extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain. The core extraction model is invoked to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; The domain knowledge is stored in the basic data layer. Based on the basic data layer, the reasoning model encapsulated in the model service layer is used to generate derived knowledge and construct the knowledge graph.

2. The method according to claim 1, characterized in that, To construct a general knowledge graph construction framework, including: A five-layer modular architecture is constructed, including: a basic data layer for storing knowledge triples and raw data; an entity aggregation layer for disambiguating and fusing entities in the basic data layer; a model service layer for encapsulating knowledge processing models and providing services through standardized interfaces; a knowledge service layer for providing knowledge query, retrieval, and reasoning services; and an application adaptation layer for providing configurable domain templates to quickly adapt to upper-layer applications. The layers communicate and exchange data through defined application programming interfaces, enabling the pluggable and independent replacement of technical components; When the entities are grouped into a single layer for entity disambiguation, an entity discrimination model trained on a generative adversarial network is used to distinguish different entities with similar or identical names. The domain configuration templates provided by the application adaptation layer encapsulate the entity types, relationship types, business rules, and visual configurations for a specific domain.

3. The method according to claim 2, characterized in that, The method further includes: A structured configuration file based on JSON or YAML is used, which explicitly defines: pipeline components, used to declare the various knowledge processing models included in the processing pipeline; execution topology, used to define the execution order and data flow between the models; and parameter configuration, used to configure the specific parameters required for the execution of each model. The model service layer parses the structured configuration file, calls the corresponding model services in sequence according to the defined topology, and uses the output of the upstream model as the input of the downstream model, thereby automatically executing the complete knowledge processing flow.

4. The method according to claim 1, characterized in that, Domain-adaptive fine-tuning of the core extraction model in the model service layer of the general framework is performed through a domain adapter, including: The large language model, which serves as the core extraction model, is continuously pre-trained using a pre-trained corpus containing corpora from multiple domains. When adapting to a new domain, a parameter-efficient fine-tuning method is adopted, which fine-tunes the basic large language model by inserting a domain adapter, while keeping the core parameters of the basic large language model unchanged; The training of the domain adapter adopts a progressive domain expansion strategy. By calculating the semantic similarity between the new domain and the existing domain, the most relevant existing domain adapter is selected as the initialization basis for incremental training. Among them, the domain adapter is a pluggable, lightweight neural network module specifically designed to adapt a general model to a specific domain; The efficient parameter fine-tuning method specifically adopts any one of LoRA, adapter, or prefix fine-tuning methods; The progressive domain expansion strategy specifically includes: measuring domain similarity by calculating the cosine similarity of embedding vectors between domains, and selecting the existing domain adapter with the highest similarity as the initialization parameter for training the new domain.

5. The method according to claim 1, characterized in that, Through a dynamic schema mapping mechanism, knowledge extraction tasks are dynamically generated based on the entity and relationship types defined by the user for a specific domain, including: A predefined instruction template is provided, which includes task instruction fields, schema list fields, and input data placeholders. Receive the entity types and relationship types defined by the user for a specific domain, and populate them into the Schema list field of the instruction template; Acquire multi-source heterogeneous data and perform general hierarchical preprocessing on the multi-source heterogeneous data to convert structured, semi-structured and unstructured data into a unified standardized format; The standardized data is filled into the input data placeholders to generate structured task description data.

6. The method according to claim 5, characterized in that, Acquire multi-source heterogeneous data and perform general hierarchical preprocessing on the multi-source heterogeneous data, including: Multi-source heterogeneous data is accessed through a data source adaptation layer, which is implemented through a configurable API gateway. Relational databases, unstructured documents, and streaming data are accessed using JDBC protocol, Apache Tika parser, and Kafka message queue, respectively. The data standardization layer performs unified processing on the incoming data, including data cleaning and deduplication based on a common metadata model, and converts all data into JSON-LD format; The data augmentation layer performs semantic enhancement on the standardized data, including generating text semantic vectors using pre-trained language models and generating feature vectors for structured data using knowledge embedding techniques.

7. The method according to claim 1, characterized in that, The domain knowledge is stored in the basic data layer, and derived knowledge is generated based on the basic data layer using the inference model encapsulated in the model service layer, including: The extracted domain knowledge is stored in the graph database of the basic data layer in the form of triples; Through the reasoning model encapsulated in the model service layer, logical reasoning or vector calculation is performed on the triples stored in the basic data layer to generate implicit, derived knowledge that is not explicitly expressed in the original data. The derived knowledge is written back to the basic data layer, and together with the original extracted knowledge, it forms a complete knowledge graph. Enhanced knowledge query and reasoning services are then provided to the outside world through the knowledge service layer. The reasoning models include a logical reasoning model based on predefined rules and a vector reasoning model based on knowledge graph embedding representation.

8. A general knowledge graph construction system, characterized in that, include: The framework building module is used to build a general knowledge graph construction framework. This framework is implemented through a modular layered architecture, which includes at least a basic data layer, a model service layer, and a knowledge service layer, and the layers are decoupled from each other through standardized interfaces. The domain fine-tuning module is used to perform domain adaptive fine-tuning on the core extracted model of the model service layer in the general framework through the domain adapter, so as to adapt the general framework to the domain corresponding to the domain adapter. The task generation module is used to generate tasks that extract knowledge from pre-acquired multi-source heterogeneous data based on the entity and relationship types defined by the user for the corresponding domain through a dynamic schema mapping mechanism. The knowledge extraction module is used to call the core extraction model to execute the task in order to extract domain knowledge from the multi-source heterogeneous data; The knowledge processing module is used to store the domain knowledge in the basic data layer, and based on the basic data layer, generate derived knowledge using the reasoning model encapsulated in the model service layer to construct the knowledge graph.

9. A general-purpose knowledge graph construction device, characterized in that, include: The memory is used to store the general knowledge graph building program; A processor, configured to implement the steps of the general knowledge graph construction method as described in any one of claims 1-7 when executing the general knowledge graph construction program.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a general knowledge graph construction program, which, when executed by a processor, implements the steps of the general knowledge graph construction method as described in any one of claims 1-7.

Images