A large model driven automatic knowledge graph construction method

Abstract

The application discloses a large model driven automatic knowledge graph construction method based on a confidence feedback mechanism, aiming to improve the structural accuracy and semantic consistency in the structured triple generation process, using a few-shot prompt mechanism and a cross-validation mechanism of a heterogeneous large model to guide the structural constraint, and strengthening the control ability of the large language model on the triple format, so as to reduce the format deviation and semantic redundancy in the generation process. At the same time, a multi-dimensional confidence evaluation system is constructed, the model consensus judgment, semantic rationality analysis and knowledge consistency check are fused, and the fine quantification and screening of the triple quality are realized. The application introduces a confidence backtracking feedback strategy, constructs a closed loop process of generation-verification-optimization, enhances the expression and correction ability of the system to the complex knowledge structure, effectively reduces the dependence on external API, reduces the consumption of computing resources, and significantly improves the system running efficiency and the feasibility of engineering deployment.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and in particular to a method for constructing automated knowledge graphs driven by a large model based on a confidence feedback mechanism. Background Technology

[0002] Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing in recent years, and have been widely used in tasks such as question answering systems, text generation, and machine translation, gradually becoming an important technical support for information extraction and knowledge construction tasks. Especially in scenarios where structured information is scarce or annotation costs are high, LLMs have demonstrated strong generalization ability and knowledge transfer potential, and can generate structured triples that conform to the format specifications based on a small number of examples. However, current large models still face significant challenges in generating structured information, especially in terms of triple structure accuracy, semantic consistency, and knowledge completeness, which seriously restricts their application in knowledge graph construction tasks with high reliability requirements.

[0003] Traditional knowledge graph construction methods mostly rely on manual annotation or rule-based extraction, such as OpenIE and dependency parsing. While these methods offer some interpretability and accuracy, they suffer from low efficiency and limited coverage when dealing with large-scale unstructured text, and struggle to adapt to complex and diverse semantic expressions. Furthermore, existing relation extraction methods based on embedding models, although achieving high performance in specific domains, generally suffer from dependence on large-scale labeled data, making them unsuitable for zero-sample or low-resource scenarios. To address these issues, researchers have attempted to leverage the language modeling capabilities of LLMs, guiding them to generate structured triples via prompts to construct open-domain knowledge graphs; however, a systematic quality control mechanism and reliable verification methods are still lacking.

[0004] Recent research indicates that single large language models are susceptible to training corpus bias, contextual noise, and semantic ambiguity during knowledge extraction, leading to phenomena such as "structural mismatch," "semantic drift," and even "illusory triples" in the generated results. Although some methods have attempted to improve output quality through post-processing or rule correction, there is a lack of a systematic optimization of the entire graphing process from a closed-loop perspective of generation-validation-feedback. Furthermore, a unified framework for evaluating triple confidence has not yet been established, making it difficult to effectively measure and filter the credibility of automatically generated results.

[0005] In existing technologies, Deepex is a structured knowledge triple extraction method based on language models, modeling the triple generation task as a text generation problem. By incorporating structural cues into the attention mechanism of a pre-trained language model, Deepex can generate entity-relation-entity triples that conform to semantic patterns from input text. For example, faced with the input sentence "This material has excellent thermal conductivity and mechanical strength," Deepex can generate triples ["this material", "has", "thermal conductivity"] and ["this material", "has", "mechanical strength"], improving extraction accuracy. In industrial specification documents, Deepex can automatically extract the relationships between equipment, components, and performance parameters for building an equipment knowledge base, significantly improving the efficiency of structured processing of industrial documents. Although Deepex uses pre-trained language models and models triple extraction as a text generation task, which improves the efficiency of structured information extraction to some extent, it lacks a reliable triple quality control mechanism. Especially when facing texts with complex semantics and multiple meanings in the context, it is prone to problems such as incomplete structure, semantic deviation or redundancy, making it difficult to guarantee the accuracy and consistency of the knowledge graph.

[0006] PIVE (Prompt-based Iterative Verification and Enhancement) employs a multi-turn prompting mechanism to guide large language models in optimizing triple output. Its core idea is to enhance the structural rationality of triples through templated iterative prompts and impose format constraints during the generation phase, improving the model's ability to handle complex semantic relationships. For example, in a technical document describing "this sensor can detect temperature and humidity changes," PIVE, through iterative guidance, can generate ["this sensor", "detects", "temperature changes"] and ["this sensor", "detects", "humidity changes"]. In scientific paper abstracts, PIVE can extract the logical relationships between experimental objects, research methods, and conclusions, providing structured support for scientific literature annotation and knowledge management. However, this method heavily relies on static templates and rule design, lacks dynamic feedback optimization capabilities, and has limited generalization ability when facing different domains or text styles. Furthermore, due to the lack of a clear confidence measurement mechanism, it is difficult to systematically screen and retain high-quality triples, affecting the stability and automation of knowledge construction.

[0007] SAC-KG (Structured Auto-Completion for Knowledge Graphs) proposes the concept of a "language model as a structure generator," directly generating structured triples through a pre-trained language model and filling in missing entity relationships using a graph structure completion mechanism. Its validation module can judge and correct the structural rationality based on the language model. For example, given the input "This system contains multiple independent modules, used for signal processing and data storage respectively," the model can generate ["This system", "Contains", "Signal processing module"] and ["This system", "Contains", "Data storage module"], and can further complete the dependencies between modules. In product specification scenarios, SAC-KG can identify the subordination and dependency relationships between functional components and automatically construct multi-level component function graphs for product recommendation and retrieval systems. However, the structure correction process of this method mainly relies on the language capabilities of the large model itself, lacking an external confidence verification mechanism, making it susceptible to model "illusion" and resulting in uncertainty in the generation logic. Furthermore, without multi-model validation and dynamic evaluation, the system struggles to guarantee the accuracy and consistency of nodes and edges in complex graph structures.

[0008] REBEL (Relation Extraction By End-to-end Language generation), based on the BART architecture, uses sequence generation for end-to-end triple extraction, outputting structured results conforming to a specific grammatical template. The model learns from the sequence of input statements, identifying entity boundaries and relation types, and outputting standard triple forms. For example, in the patent description statement "The power module of this device adopts a replaceable design," REBEL can output ["the device", "contains", "power module"] and ["power module", "design method", "replaceable"]. This method can quickly extract structural information such as device attributes, compliance requirements, and testing indicators from technical standard documents, achieving document structure mapping and rapid retrieval, effectively supporting standard comparison and automatic classification tasks. However, this method lacks structural control capabilities in scenarios with multiple co-occurrences of relations and multiple intersections of entities, especially lacking mechanistic support when facing document-level tasks, making it difficult to perform quality grading, filtering, or semantic feedback, affecting the accuracy and practicality of the final knowledge graph.

[0009] GraphRAG is a graph-enhanced generation method proposed to overcome the shortcomings of traditional RAG methods in connecting complex document information and identifying central nodes. This method transforms text into a graph structure, improves generation quality through entity and relation modeling, and introduces a graph guidance mechanism to optimize context integration capabilities during the generation process. For example, in a scientific text describing "temperature changes causing system pressure fluctuations during this reaction," GraphRAG can identify the causal relationship between "temperature changes" and "pressure fluctuations" and construct a corresponding graph structure for enhanced generation. In knowledge document summarization tasks, GraphRAG can identify the logical relationships between core terms and guide the summarization process with the help of graph structures, generating more compact and thematically focused summary texts, improving the ability to compress and express professional knowledge. However, this method typically relies on external APIs (such as GPT-4-turbo) for generation and inference, resulting in high costs and limiting its scalability and cost-effectiveness in practical engineering deployments. Furthermore, its generated content lacks an explicit alignment mechanism with structured triples, which is detrimental to building highly consistent structured knowledge graphs.

[0010] Therefore, a new knowledge graph construction method is urgently needed that can fully leverage the strong generalization ability of large language models in few-shot learning scenarios, while also introducing multi-dimensional quality control mechanisms at both the structural and semantic levels. Especially in the task of constructing large-scale, multi-domain knowledge graphs with high consistency requirements, an automated technical framework with reusability, scalability, low cost, and high reliability should be developed to lower the construction threshold, improve graph quality, and provide more stable structured knowledge support for downstream tasks such as intelligent question answering and semantic retrieval. Summary of the Invention

[0011] This invention addresses the aforementioned technical problems by designing novel algorithms and processes, proposing a large-model-driven automated knowledge graph construction method based on a confidence feedback mechanism. This method aims to improve the structural accuracy and semantic consistency in the generation of structured triples. It utilizes a few-shot hint mechanism and a cross-validation mechanism of heterogeneous large models to guide structural constraints, strengthening the large language model's control over triple format, thereby reducing format deviations and semantic redundancy during generation. Simultaneously, a multi-dimensional confidence evaluation system is constructed, integrating model consensus judgment, semantic rationality analysis, and knowledge consistency verification to achieve refined quantification and screening of triple quality. To address the issues of insufficient stability and controllability of generated content, this invention introduces a confidence backtracking feedback strategy, constructing a closed-loop process of generation-verification-optimization, enhancing the system's ability to express and correct complex knowledge structures.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] This invention provides a method for large-model-driven automated knowledge graph construction based on a confidence feedback mechanism, comprising the following steps:

[0014] S1. Based on the input text information, the semantic enhancement retrieval module is used to obtain reference text paragraphs that are semantically similar to the input content;

[0015] S2. The retrieved reference text paragraphs are used as part of the prompt content and are sent together with the original input text to the prompt generation module to form a few-sample prompt. The prompt is then sent to the large language model generation module to generate the initial structured triples.

[0016] S3. For each input text, multiple heterogeneous large language model generation modules are called in parallel to generate multiple sets of triples. The cross-validator module is used to perform structural alignment and semantic parsing on the sets of triples generated by different large language model generation modules, and to filter out structural items with cross consensus. For triples with inconsistencies or low confidence, the historical records in the memory database are referenced, and corrections and replacements are made according to the degree of semantic matching to generate a preliminary correction graph.

[0017] S4. A multidimensional confidence evaluator module is used to evaluate the confidence of each triple set. Three evaluation mechanisms are adopted: first, semantic similarity between the triple and the original text is calculated through an embedding model; second, the co-occurrence frequency of each triple in multiple model results is calculated; and third, semantic reasonableness is scored based on language modeling probability and logical consistency. By weighted fusion of the three scores, it is finally determined whether the comprehensive confidence of each triple exceeds the threshold.

[0018] S5. When the overall confidence of a triplet meets the preset requirements, the structural information is written into the memory database as long-term knowledge storage and used as a hint enhancement material in subsequent tasks to participate in a new round of construction process; for triplets with insufficient overall confidence but structural potential, they are sent back to the large language model in step S2 for closed-loop optimization.

[0019] Furthermore, in step S1, the semantic enhancement retrieval module utilizes a locally constructed knowledge database and employs the semantic enhancement embedding technique nomic-embed-text to encode the input text and documents in the database into high-dimensional vector representations. Subsequently, by calculating the cosine similarity between the input and the candidate corpus, it quickly locates and extracts the most relevant knowledge blocks for constructing prompts.

[0020] Furthermore, in step S2, the prompt generation module constructs a complete Few-shot Prompt, including role description, search results, and input text, based on a specific structural template, to guide the large language model in generating structured triples.

[0021] Furthermore, in step S2, the prompt generation module can access multiple heterogeneous large language model generation modules and supports the simultaneous and parallel invocation of multiple heterogeneous large language model generation modules to generate the initial knowledge graph.

[0022] Furthermore, in step S3, the set of triples is represented as:

[0023]

[0024] Where M represents the number of heterogeneous large language models involved in the generation, and T i Let represent the set of triples generated by the i-th large language model.

[0025] Furthermore, in step S4, the overall confidence level of the triples is... j Defined as a weighted linear combination:

[0026] Conf j =α·C j +β·S j +γ·R j

[0027] Where α, β, γ ∈ [0, 1], satisfying α + β + γ = 1, representing the importance weight of each indicator in the overall confidence assessment; C j S is the model consensus score. j R is used for semantic similarity scoring. j Semantic rationality score.

[0028] Furthermore, the model consensus score C j The term used to measure whether a triple appears consistently in the outputs of multiple heterogeneous large language models is defined as follows:

[0029]

[0030] in:

[0031] M: The number of large language models involved in structure generation;

[0032] The set of triples output by the i-th large language model;

[0033] Indicator function, takes the value 1 if the condition is true, otherwise takes the value 0.

[0034] Furthermore, the text semantic similarity S j Used to measure triplet t j The semantic fit between the input text x and the triples is calculated using an embedding model, resulting in a cosine similarity between the triples and the input text, expressed as:

[0035]

[0036] in:

[0037] : The text representation concatenated from triples is generated by an embedding model;

[0038] v x : An embedded representation of the original input text;

[0039] ·: Vector dot product operation;

[0040] ‖·‖: Vector norm.

[0041] Furthermore, semantic logical rationality R j It is used to measure the logical consistency between predicates and entities in triples, and scores are given using structural language modeling or graph relation co-occurrence probability models.

[0042] If a cross-validator is constructed using a language model, the semantic logical rationality score is represented as follows:

[0043] R j =log P(t) j |LM)

[0044] in:

[0045] P(t j |LM): The probability of generating this triplet given the conditional input by the large language model;

[0046] log: The logarithmic function, used for scaling and stabilizing training;

[0047] t j : The natural language representation of the triple;

[0048] If a cross-validator is constructed using a structural knowledge graph, the semantic logical rationality score is represented as follows:

[0049]

[0050] in:

[0051] f valid (s j ,p j ,o j ): The number of times this triple appears in the knowledge base;

[0052] f total The total number of times the entity-predicate combination appears.

[0053] Further, in step S5, the closed-loop optimization process includes the following steps: First, an error signal is constructed to identify the reasons for the failure of the triplet structure due to insufficient comprehensive confidence. By analyzing its gaps in semantic similarity, model consensus, or semantic logical rationality scores, more targeted Prompt content is reconstructed, including adjusting the template structure, adding counterexample constraints, and introducing new supporting text. Subsequently, the correction prompts are re-entered into the large language model generation module to obtain a new candidate structure version. To achieve stable convergence of feedback generation, this closed-loop optimization process can be iterated multiple times, denoted as:

[0054]

[0055] in, Prompt represents the structure generated in the (k+1)th round. (k) This indicates the prompt content for round k, Feedback (k) This is the error signal for the corresponding round;

[0056] If the overall confidence of the structure generated in a certain round of optimization reaches the set threshold, the structure will be included in the memory database as a corrected version, while replacing or supplementing the original low-confidence structure; if the structure still fails to pass the verification after reaching the maximum number of iterations, the structure will be completely removed, and the abnormal behavior will be recorded for future model tuning or to prompt engineering optimization.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0058] 1. Compared with existing technologies, such as the Deepex method, which has the problem of weak structural control in structured information extraction, this invention introduces a few-sample prompting mechanism and structured templates to guide the large language model to generate triples, which effectively improves the consistency and completeness of triple structure output, significantly reduces the dependence of structured information construction on manually labeled data, and shows stronger adaptability and generation accuracy, especially in unsupervised or weakly supervised scenarios.

[0059] 2. Compared to existing technologies, such as the PIVE method, which mainly relies on static templates for iterative correction and is difficult to dynamically adapt to texts in different domains, the multi-model cross-validation mechanism proposed in this invention automatically filters semantically consistent and structurally consistent triples through multi-model consistency judgment and structural comparison. It can identify semantically inconsistent or structurally conflicting triples in real time, improving the robustness and credibility of structured information extraction results. Combined with confidence feedback, it achieves dynamic reconstruction, enhancing the system's generalization ability and robustness under complex corpora in multiple domains.

[0060] 3. Compared with existing technologies, such as the SAC-KG method, which mainly relies on the language model itself for structural completion during the graph construction process and lacks an explicit confidence control mechanism, this invention constructs a confidence evaluator that integrates three-dimensional indicators of model consensus, semantic rationality, and knowledge consistency. This evaluator comprehensively quantifies the quality of triples and comprehensively judges the structural reliability and knowledge correctness of the triple output. It provides a quantitative basis for triple selection and graph construction, and significantly improves the controllability and credibility of the generated structure.

[0061] 4. Compared with existing technologies, such as REBEL, which lacks structural hierarchy control in texts with multiple co-occurring relationships, this invention achieves automatic correction and optimization of triples by designing a structure-aware cross-validator and a closed-loop feedback mechanism. It is particularly suitable for document-level, multi-hop relationship extraction tasks, improving the contextual coherence and knowledge integrity of structured output.

[0062] 5. Compared to existing technologies, such as the GraphRAG method, which relies on external APIs for information generation, resulting in high operating costs and weak controllability, this invention introduces a confidence backtracking and feedback optimization mechanism. It dynamically reconstructs and optimizes based on the quality of triple generation, forming a closed-loop structure of "generation-verification-optimization". This improves the model's graph-building ability under complex text and multi-hop knowledge relationships, supports end-to-end local deployment, replaces the dependence on high-cost external APIs, and significantly reduces system operating costs. At the same time, combined with a lightweight calling strategy and parallel processing mechanism, it significantly improves the efficiency and engineering deployment capabilities of large-scale structured knowledge graph construction tasks. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0064] Figure 1 This is a schematic diagram of the framework of a large model-driven automated knowledge graph construction method based on a confidence feedback mechanism provided in an embodiment of the present invention.

[0065] Figure 2 This is a schematic diagram of the prompting structure of the large model-driven automated knowledge graph construction method based on a confidence feedback mechanism provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] This invention provides a method for large-model-driven automated knowledge graph construction based on a confidence feedback mechanism, comprising the following steps:

[0068] S1. Based on the input text information (such as descriptive documents, scenic spot introductions, scientific and technological explanations, etc.), the semantic enhancement retrieval module obtains reference text paragraphs that are semantically similar to the input content. Specifically, the semantic enhancement retrieval module utilizes a locally built knowledge database and uses the semantic enhancement embedding technology nomic-embed-text to encode the input text and the documents in the database into high-dimensional vector representations. Then, by calculating the cosine similarity between the input and the candidate corpus, it quickly locates and extracts the most relevant knowledge blocks for constructing prompts.

[0069] S2. The retrieved reference text paragraphs are used as part of the prompt content and are sent together with the original input text to the prompt generation module to form a few-shot prompt. This prompt is then sent to the large language model generation module to generate initial structured triples. Specifically, the prompt generation module constructs a complete few-shot prompt, including role description, search results, and input text, based on a specific structure template to guide the large language model in generating structured triples. The prompt generation module can connect to multiple heterogeneous large language model generation modules (such as locally deployed Mistral, Mixtral, or LLaMA series) and supports simultaneous parallel invocation of multiple heterogeneous large language model generation modules to generate the initial knowledge graph.

[0070] S3. For each input text, multiple heterogeneous large language model generation modules are invoked in parallel to generate multiple sets of triples. Subsequently, the system enters the cross-validator module to compare the structural consistency and semantic overlap of the triples generated by the multiple large models. The cross-validator module performs structural alignment and semantic parsing on the sets of triples generated by different large language model generation modules, filters out structural items with cross-consensus, and for triples with inconsistencies or low confidence, it refers to historical records in the memory database and corrects and replaces them according to the degree of semantic matching, generating a preliminary correction graph.

[0071] S4. A multidimensional confidence evaluator module is used to evaluate the confidence of each triple set. Three evaluation mechanisms are adopted: first, semantic similarity between the triple and the original text is calculated through an embedding model; second, the co-occurrence frequency of each triple in multiple model results is calculated; and third, semantic reasonableness is scored based on language modeling probability and logical consistency. By weighted fusion of the three scores, it is finally determined whether the comprehensive confidence of each triple exceeds the threshold.

[0072] S5. When the overall confidence of a triple meets the preset requirements, the structural information is written into the memory database as long-term knowledge storage and used as a prompt enhancement material in subsequent tasks to participate in the new round of construction process. For triples with insufficient overall confidence but structural potential, they are sent back to the large language model in step S2 for closed-loop optimization to improve the robustness and semantic consistency of the generated results.

[0073] Ultimately, the system implements a closed-loop structured knowledge graph construction process that can be deployed locally. It has strong semantic graph building capabilities, high structural consistency control capabilities, and low resource dependency characteristics, and is suitable for automatic knowledge extraction and graph construction tasks in multiple fields and with diverse text types.

[0074] In its implementation, the method of this invention is based on a confidence feedback mechanism and employs a systematic and rigorous model and algorithm process to ensure the efficiency, accuracy, and robustness of automated knowledge graph construction. The following will begin by explaining the principles and implementation details of the core steps, gradually elucidating the specific operation methods and technical points of this invention.

[0075] Step 1: Text Input and Related Knowledge Retrieval

[0076] First, the system receives raw descriptive text input from the user (e.g., geographical descriptions, product specifications, news materials, etc.). This text is then fed into the semantically enhanced retrieval module, where the system uses semantically enhanced vectorized encoding to match and retrieve documents from the local knowledge database.

[0077] The semantically enhanced retrieval system uses the nomic-embed-text embedding model (which can be replaced with general models such as text-embedding-ada-002) to encode the input text into a high-dimensional vector form. The vector representation is denoted as:

[0078] v input =Embed(x)

[0079] Where x represents the original input text, Embed(·) is the embedding function, and v input ∈R d This represents the text representation within the embedded space.

[0080] The system simultaneously retrieves candidate documents {d1, d2, ..., d...} from the knowledge database. k} and compute its embedding representation. Then, the most relevant passage is obtained by calculating the cosine similarity:

[0081]

[0082] The system will return the top k most relevant documents as supplementary contextual material for the construction of subsequent large model hints.

[0083] Step 2: Construct the prompt and generate triples

[0084] Next, the system sends the original input text and the retrieved semantically similar text segments into the prompt generation module to form a few-shot prompt. The prompt structure is as follows: Figure 2 As shown.

[0085] This prompt is fed into a large language model (such as Mixtral, LLaMA 7B, Mistral) to generate initial structured triples (SPO triples), for example:

[0086] ("Huangshan", "Main Peak Elevation", "1864.8 meters")

[0087] ("Huangshan","Province","Anhui Province")

[0088] ("Huangshan", "Vegetation Coverage", "88.9%))

[0089] Step 3: Cross-validation of heterogeneous models

[0090] For each input text, the system can call multiple heterogeneous large language models in parallel (such as...).

[0091] Mistral, GPT-J, Baichuan, etc., generate multiple sets of triples:

[0092]

[0093] Where M represents the number of models involved in the generation, and T i Let represent the set of triples generated by the i-th model.

[0094] Step 4: Multidimensional Confidence Assessment

[0095] To ensure the reliability of the structured triplet results, this invention designs a multidimensional confidence estimator for each triplet t. j =(s j ,p j ,o jThe scoring is based on three independent dimensions: model consensus, text semantic similarity, and semantic logical rationality.

[0096] We represent these three dimensions as follows:

[0097] C j Model consensus score;

[0098] S j Semantic Similarity Score;

[0099] R j Semantic reasonableness score.

[0100] Ultimately, the overall confidence level of the triplet is Conf. j Defined as a weighted linear combination:

[0101] Conf j =α·C j +β·S j +γ·R j

[0102] Where α,β,γ∈[0,1], and α+β+γ=1, it represents the importance weight of each indicator in the overall confidence assessment.

[0103] Model consensus score C j The term used to measure whether a triple appears consistently in the outputs of multiple heterogeneous large language models is defined as follows:

[0104]

[0105] in:

[0106] M: The number of large language models involved in structure generation;

[0107] The set of triples output by the i-th large language model;

[0108] : Indicator function, takes the value 1 if the condition is true, otherwise takes the value 0.

[0109] The higher the score, the better the structure is consistent across models.

[0110] Text semantic similarity S j Used to measure triplet t j The semantic fit between the input text x and the triples is calculated using an embedding model, resulting in a cosine similarity between the triples and the input text, expressed as:

[0111]

[0112] in:

[0113] : The text representation concatenated from triples is generated by an embedding model;

[0114] v x : An embedded representation of the original input text;

[0115] ·: Vector dot product operation;

[0116] ‖·‖: Vector norm.

[0117] This metric measures whether the triples truly reflect the semantic content of the input.

[0118] Semantic logical rationality R j It is used to measure the logical consistency between predicates and entities in triples, and scores are given using structural language modeling or graph relation co-occurrence probability models.

[0119] If a cross-validator is constructed using a language model, the semantic logical rationality score is represented as follows:

[0120] R j =logP(t) j |LM)

[0121] in:

[0122] P(t j |LM): The probability of generating this triplet given the conditional input by the large language model;

[0123] log: The logarithmic function, used for scaling and stabilizing training;

[0124] t j The natural language expression corresponding to the triplet, such as "the highest altitude of a certain scenic spot is 1864.8 meters";

[0125] If a cross-validator is constructed using a structural knowledge graph, the semantic logical rationality score is represented as follows:

[0126]

[0127] in:

[0128] f valid (s j ,p j ,o j ): The number of times this triple appears in the knowledge base;

[0129] f total The total number of times the entity-predicate combination appears.

[0130] The system sets a final confidence threshold τ∈[0,1] and filters those that meet the confidence criteria. j The set of high-confidence triples ≥τ is used as the final structure map output.

[0131] Step 5: Optimize the generation-memory-feedback closed loop

[0132] In the overall architecture of this invention, the generation-verification process goes beyond a one-time structured output. To further improve the system's accuracy, stability, and iterative capabilities, a "memory database" and a "feedback optimization mechanism" are introduced, constructing a continuously enhanced closed-loop system for knowledge graph construction. All triples determined to be highly reliable by the multi-dimensional confidence assessment module are written into the memory database in a standardized format. Each record in the database not only contains the triple ontology (subject, predicate, object) but also includes metadata fields such as its corresponding large language model source, consensus count, generation timestamp, and confidence score. This memory module has long-term storage and high-frequency query capabilities. In subsequent processing of similar texts, the system can prioritize retrieving high-confidence triples with similar structures as prompts, thereby reducing model illusions, improving response speed and consistency to repetitive structures, and supporting the comparison basis of the structural cross-validator in operations such as structural alignment and semantic merging.

[0133] Beyond the role of the memory module, for candidate structures deemed to have semantic potential but below a confidence threshold during the triplet evaluation phase, the system marks them as "pending feedback optimization" and initiates a closed-loop optimization process. This process first constructs an error signal targeting the reasons for the structure's failure. By analyzing its gaps in semantic similarity, structural consensus, or logical consistency scores, it reconstructs more targeted prompt content, such as adjusting the template structure, adding counterexample constraints, or introducing new supporting text. This corrective prompt is then re-entered into the large language model generation module to obtain a new candidate structure. To achieve stable convergence of feedback generation, this closed-loop optimization process can execute multiple iterations, denoted as:

[0134]

[0135] in, Prompt represents the structure generated in the (k+1)th round. (k) This indicates the prompt content for round k, Feedback (k) This is the error signal for the corresponding round;

[0136] If the overall confidence of the structure generated in a certain round of optimization reaches the set threshold, the structure will be included in the memory database as a corrected version, while replacing or supplementing the original low-confidence structure; if the structure still fails to pass the verification after reaching the maximum number of iterations, the structure will be completely removed, and the abnormal behavior will be recorded for future model tuning or to prompt engineering optimization.

[0137] This feedback loop optimization process not only improves the system's tolerance to "boundary samples," but also achieves systematic "knowledge enhancement" through linkage with the memory database. That is, as tasks accumulate, the model's response quality when facing similar types of inputs gradually improves, and the overall system performance evolves and enhances over time, thus forming a structured knowledge construction closed-loop system with memory self-correction capabilities, autonomous knowledge accumulation capabilities, and error repair capabilities.

[0138] Overall, the automated structured knowledge graph construction method designed in this invention consists of a semantic enhancement retrieval unit, a large model generation module, a cross-validator, a confidence evaluator, and a memory database at the input end. These five modules form a data closed loop and are driven by a feedback mechanism for iterative enhancement, constructing a multi-stage control process with the goals of structural integrity, semantic consistency, and logical rationality. Compared with existing purely generative or rule-based structure extraction methods, the system of this invention has the following significant advantages: (1) It integrates few-sample prompts and multi-source information guidance to achieve low-cost and high-generalization structure generation; (2) It improves the consistency and redundancy removal capability of output triples through multi-model cross-validation and structure alignment mechanisms; (3) It adopts a three-dimensional evaluation strategy of embedding similarity, consensus score, and language rationality to effectively control the credibility of the structure output.

[0139] (4) With the help of memory mechanism and feedback optimization, the system has the ability to continuously evolve and self-repair anomalies, and builds an automated knowledge graph construction closed-loop process with stability, scalability and deployment feasibility.

[0140] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0141] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as...).

[0143] The ROM / RAM, magnetic disk, optical disk, etc., include several instructions to cause a service classification device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0144] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for constructing automated knowledge graphs based on a large model driven by a confidence feedback mechanism, characterized in that, Includes the following steps: S1. Based on the input text information, the semantic enhancement retrieval module is used to obtain reference text paragraphs that are semantically similar to the input content; S2. The retrieved reference text paragraphs are used as part of the prompt content and are sent together with the original input text to the prompt generation module to form a few-sample prompt. The prompt is then sent to the large language model generation module to generate the initial structured triples. S3. For each input text, multiple heterogeneous large language model generation modules are called in parallel to generate multiple sets of triples. The cross-validator module is used to perform structural alignment and semantic parsing on the sets of triples generated by different large language model generation modules, and to filter out structural items with cross consensus. For triples with inconsistencies or low confidence, historical records in the memory database are referenced and corrected or replaced according to the degree of semantic matching to generate a preliminary correction map. S4. A multidimensional confidence evaluator module is used to evaluate the confidence of each triple set. Three evaluation mechanisms are adopted: first, semantic similarity between the triple and the original text is calculated through an embedding model; second, the co-occurrence frequency of each triple in multiple model results is calculated; and third, semantic reasonableness is scored based on language modeling probability and logical consistency. By weighted fusion of the three scores, it is finally determined whether the comprehensive confidence of each triple exceeds the threshold. S5. When the overall confidence of a triplet meets the preset requirements, the structural information is written into the memory database as long-term knowledge storage and used as a hint enhancement material in subsequent tasks to participate in a new round of construction process; for triplets with insufficient overall confidence but structural potential, they are sent back to the large language model in step S2 for closed-loop optimization.

2. The method for large-model-driven automated knowledge graph construction based on a confidence feedback mechanism according to claim 1, characterized in that, In step S1, the semantic enhancement retrieval module utilizes a locally built knowledge database and employs the semantic enhancement embedding technique nomic-embed-text to encode the input text and documents in the database into high-dimensional vector representations. Subsequently, by calculating the cosine similarity between the input and the candidate corpus, it quickly locates and extracts the most relevant knowledge blocks for constructing prompts.

3. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 1, characterized in that, In step S2, the prompt generation module constructs a complete Few-shot Prompt, including role description, search results, and input text, based on a specific structural template, to guide the large language model in generating structured triples.

4. The method for large-model-driven automated knowledge graph construction based on a confidence feedback mechanism according to claim 1, characterized in that, In step S2, it is indicated that the generation module can access multiple heterogeneous large language model generation modules and supports the simultaneous parallel invocation of multiple heterogeneous large language model generation modules to generate the initial knowledge graph.

5. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 1, characterized in that, In step S3, the set of triples is represented as: Where M represents the number of heterogeneous large language models involved in the generation, and T i Let represent the set of triples generated by the i-th large language model.

6. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 1, characterized in that, In step S4, the overall confidence level of the triples is Conf. j Defined as a weighted linear combination: Conference j =α·C j +β·S j +γ·R j Where α, β, γ ∈ [0, 1], satisfying α + β + γ = 1, representing the importance weight of each indicator in the overall confidence assessment; C j S is the model consensus score. j R is used for semantic similarity scoring. j Semantic rationality score.

7. The method for large model-driven automated knowledge graph construction based on a confidence feedback mechanism according to claim 6, characterized in that, Model consensus score C j The term used to measure whether a triple appears consistently in the outputs of multiple heterogeneous large language models is defined as follows: in: M: The number of large language models involved in structure generation; The set of triples output by the i-th large language model; : Indicator function, takes the value 1 if the condition is true, otherwise takes the value 0.

8. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 6, characterized in that, Text semantic similarity S j Used to measure triplet t j The semantic fit between the input text x and the triples is calculated using an embedding model, resulting in a cosine similarity between the triples and the input text, expressed as: in: : The text representation concatenated from triples is generated by an embedding model; v x : An embedded representation of the original input text; ·: Vector dot product operation; ‖·‖: Vector norm.

9. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 6, characterized in that, Semantic logical rationality R j It is used to measure the logical consistency between predicates and entities in triples, and scores are given using structural language modeling or graph relation co-occurrence probability models. If a cross-validator is constructed using a language model, the semantic logical rationality score is represented as follows: R j =log P(t j ∣LM) in: P(t j |LM): The probability of generating this triplet given the conditional input by the large language model; log: The logarithmic function, used for scaling and stabilizing training; t j : The natural language representation of the triple; If a cross-validator is constructed using a structural knowledge graph, the semantic logical rationality score is represented as follows: in: f valid (s j ,p j ,o j ): The number of times this triple appears in the knowledge base; f total The total number of times the entity-predicate combination appears.

10. The method for large model-driven automated knowledge graph construction based on confidence feedback mechanism according to claim 1, characterized in that, In step S5, the closed-loop optimization process... The process includes the following steps: First, an error signal is constructed to identify the reasons for the failure of triplet structures due to insufficient overall confidence. By analyzing the gaps in semantic similarity, model consensus, or semantic logical rationality scores, more targeted Prompt content is reconstructed, including adjusting the template structure, adding counterexample constraints, and introducing new supporting text. The corrected prompts are then re-entered into the large language model generation module to obtain new candidate structures. To achieve stable convergence of the feedback generation, this closed-loop optimization process can be iterated multiple times, denoted as: in, Prompt represents the structure generated in the (k+1)th round. (k) This indicates the prompt content for round k, Feedback (k) This is the error signal for the corresponding round; If the overall confidence of the structure generated in a certain round of optimization reaches the set threshold, the structure will be included in the memory database as a corrected version, while replacing or supplementing the original low-confidence structure; if the structure still fails to pass the verification after reaching the maximum number of iterations, the structure will be completely removed, and the abnormal behavior will be recorded for future model tuning or to prompt engineering optimization.

Images