Knowledge base engine construction method and system based on large language model

Abstract

The application relates to a large language model-based knowledge base engine construction method and system, and belongs to the technical field of artificial intelligence. The system is composed of a question and answer hub, notes and a knowledge base three modules to form a triangular closed loop iteration architecture, and realizes the following core functions: an automatic knowledge refining mechanism based on a large language model, which converts unstructured interactive data into standard knowledge units; an enhanced traceable retrieval generation method integrating reflection tokens, which realizes full-link monitoring through retrieval judgment, correlation evaluation and verification of reflection tokens; a self-reflection verification closed loop system, which outputs a certainty score through a knowledge evaluator and executes graded correction measures. The application effectively solves the hallucination problem, low retrieval efficiency and lack of self-reflection mechanism and other defects in the prior art, realizes the efficiency, accuracy and self-evolution ability of the knowledge base engine, and can be widely applied to intelligent question and answer and auxiliary decision systems in various professional fields.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and knowledge engineering technology, and in particular to a method for constructing, iterating and optimizing a knowledge base engine based on a large language model (LLM), specifically including automatic knowledge extraction, traceable retrieval-augmented generation (RAG), and a self-verification closed-loop system. Background Technology

[0002] With the rapid development of artificial intelligence technology, knowledge base systems have become core tools for enterprise knowledge management and intelligent decision-making. While current Retrieval Augmentation Generation (RAG) systems based on large language models can partially alleviate the problems of model illusion and knowledge lag, they still have the following limitations: First, the illusion problem is not completely solved, and the model may ignore or misinterpret retrieval evidence; second, retrieval accuracy and efficiency are insufficient, lacking an intelligent retrieval judgment mechanism; third, knowledge updates rely on manual intervention, resulting in high costs and low efficiency; and fourth, there is a lack of traceability and self-reflection mechanisms, making it impossible to achieve a closed loop of error detection and correction. These shortcomings severely restrict the application of such systems in mission-critical scenarios, necessitating the development of a new generation of knowledge base engines that are efficient, accurate, and possess self-evolving capabilities. Summary of the Invention

[0003] This invention aims to overcome the shortcomings of existing technologies and provide a method and system for constructing a knowledge base engine based on a large language model. The system utilizes a triangular closed-loop iterative architecture consisting of a question-answering central module, a note-taking module, and a knowledge base module to achieve automatic knowledge extraction, accurate retrieval, and intelligent self-reflection, significantly improving the accuracy, real-time performance, and reliability of the knowledge base.

[0004] The core functional modules of this invention include a question-and-answer hub module, a note-taking module, and a knowledge base module. These three modules form a stable closed loop that operates collaboratively and drives each other. The question-and-answer hub module, built on a large language model, serves as the main interface for interaction between the system and the user. It receives inputs from the user's natural language questions and answers, note-taking instructions, and knowledge base construction instructions, processes and analyzes them, and outputs corresponding question-and-answer results, structured notes, and knowledge entries. The note-taking module, as the system's core knowledge entry point and buffer, receives direct records from the user, high-quality question-and-answer records generated by the question-and-answer hub, and iterative optimization inputs for the knowledge base. These records undergo preliminary structuring processing by the question-and-answer hub module, and output standard knowledge units to be confirmed and stored in the knowledge base. The knowledge base module, as the system's structured knowledge storage and application center, receives knowledge units from the note-taking module, original user input (such as file uploads), and queries from the question-and-answer hub. It outputs knowledge sharing and accurate retrieval results, and provides retrieval feedback to the question-and-answer hub to complete answer generation. These three modules form a triangular closed-loop iterative relationship of "question-answering hub - notes - knowledge base": the question-answering hub (model) generates notes, the notes are deposited into the knowledge base, and the knowledge base in turn supports and optimizes the accuracy and interpretability of the question-answering hub, thereby realizing the self-evolution of the system.

[0005] The core innovations of this invention mainly include three parts: an automatic knowledge extraction mechanism, a traceable retrieval enhancement generation method, and a self-reflection verification closed-loop system.

[0006] The automated knowledge extraction mechanism analyzes high-value user Q&A records, user-submitted notes, and data from daily workflows generated by the Q&A hub, either periodically or through event-triggered methods. Leveraging the summarizing and inductive capabilities of the large language model within the Q&A hub module, it automatically extracts and summarizes this unstructured interactive data into clearly structured and factually accurate notes. These notes are then formatted into standard knowledge entries by the Q&A hub module and automatically or manually updated to the knowledge base module, enabling the knowledge base to evolve and continuously optimize.

[0007] A traceable retrieval enhancement generation method integrates a 'ReflectionTokens' mechanism based on existing technology to enhance the traceability and decision-making accuracy of the retrieval process. Specifically, this method achieves fine-grained monitoring through three optimization stages: In the retrieval judgment stage, a custom threshold algorithm is used to generate retrieval necessity reflection tokens, dynamically assessing the query's dependence on the knowledge base, thereby reducing unnecessary retrieval overhead; in the paragraph relevance evaluation stage, a multi-dimensional similarity calculation model is introduced to finely filter retrieval results, improving the accuracy of content matching with query intent; in the generation and verification stage, an evidence chain verification mechanism is combined to ensure that the final answer is strictly based on the retrieval content and maintains its integrity. The entire process is fully traceable through structured log recording, significantly improving the system's interpretability and operational efficiency, and overcoming the retrieval redundancy and insufficient verification problems existing in existing technologies.

[0008] The self-reflection and verification process utilizes a specially designed knowledge evaluator to perform macro-level evaluation and intelligent correction of the entire retrieval-generation output. The system outputs a three-tiered certainty score (sure, uncertain, irrelevant) and takes tiered corrective measures based on the evaluation results: sure results are output directly; uncertain results trigger a knowledge optimization process, including supplementing retrieval, query rewriting, or marking knowledge gaps and incorporating them into the automatic knowledge extraction process; irrelevant results initiate a reprocessing mechanism and are transferred to manual processing. This design forms a self-correcting and continuous optimization mechanism, significantly improving the reliability and stability of the output.

[0009] Through the above-mentioned innovations, this invention solves the problems of illusion, low retrieval efficiency and lack of self-reflection mechanism in the prior art, and realizes the high efficiency, accuracy and self-evolution of the knowledge base engine. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the system's triangular closed-loop iterative architecture, illustrating the interaction relationships between the question-answering central module, the note-taking module, and the knowledge base module.

[0011] Figure 2 A flowchart illustrating the enhanced retrieval process is provided, detailing the steps of retrieval judgment, relevance assessment, and verification based on reflex tokens.

[0012] Figure 3 This is a schematic diagram of the self-introspection and verification process, illustrating the evaluation and corrective action flow. The accompanying drawings help those skilled in the art to understand the embodiments of the present invention. Detailed Implementation

[0013] The following describes in detail the specific steps of the method and system for building a knowledge base engine based on a large language model, in conjunction with the preferred embodiment of the present invention. This embodiment uses a software system as an example, but the present invention is not limited thereto and can be applied to hardware, cloud platforms, or hybrid deployment environments.

[0014] First, during system initialization, a question-answering hub module, a note-taking module, and a knowledge base module are deployed. The question-answering hub module is built based on a pre-trained large language model (such as DeepSeek R1 or a similar model) and integrates with the user interface and other modules through an application programming interface (API) or message queue (such as RocketMQ or Kafka). The note-taking module is designed as a lightweight data processing unit, supporting natural language processing capabilities, and is used for temporary storage and structured data processing. The knowledge base module uses a high-capability vector database (such as ElasticSearch) and can be combined with a relational database (such as PostgreSQL) to achieve multimodal knowledge storage, supporting the retrieval of text, images, and structured data.

[0015] The automated knowledge extraction process involves setting up a scheduled task (e.g., executed daily at midnight) or an event trigger (e.g., when more than 100 new records accumulate in the notes module) to initiate the knowledge extraction process. The question-and-answer hub module analyzes user interaction data, including high-quality question-and-answer records (e.g., answers marked as "helpful" by users) and user-submitted notes. Leveraging the summarizing capabilities of a large language model, it generates structured knowledge entries, such as extracting a dialogue into QA pairs or factual statements. The notes module formats these entries, converting them into standard knowledge units in JSON format, and then automatically or with administrator confirmation, updates the knowledge base module via API calls. For example, in the programming field, the system can automatically extract new standard code components from programmers' questions and answers.

[0016] The system implements a traceable retrieval enhancement generation process: When a user submits a query, the question-answering central module first generates a retrieval judgment reflection token. Specifically, it calls the large language model to generate a binary output (yes / no) to determine whether a knowledge base search is needed. If no, the answer is generated directly based on the model's internal knowledge; if yes, a retrieval request is sent to the knowledge base module. After the knowledge base module returns preliminary retrieval results, it generates a paragraph relevance reflection token for each knowledge paragraph. The large language model evaluates the relevance score (range 0-1), filtering out paragraphs with scores below 0.5. Then, the highly relevant paragraphs are combined with the query input to generate the final answer from the large language model, and a verification reflection token is generated to ensure the answer is based on evidence. The generation and evaluation process of all reflection tokens is recorded in a log database, achieving end-to-end traceability.

[0017] The self-verification process involves a knowledge evaluator (an independent instance of a large language model) that evaluates the "question-retrieved evidence-generated answer" triple and outputs a certainty score (confident, uncertain, irrelevant). If the result is certain, the answer is output directly; if uncertain, a knowledge optimization process is triggered, such as supplementing the retrieval with more paragraphs, rewriting the query (using the large language model to rewrite the user query to improve retrieval), or marking the case as a knowledge gap and sending it into the automated knowledge extraction process; if irrelevant, the result is discarded and transferred to manual processing. The knowledge evaluator is regularly trained and fine-tuned to improve evaluation accuracy.

[0018] The operation of the triangular closed-loop iterative system: In daily user interactions, data generated by the question-and-answer hub module is stored in the knowledge base module through the note-taking module. Updates to the knowledge base module, in turn, optimize the performance of the question-and-answer hub through retrieval feedback. For example, in the financial field, the system automatically learns new market trends from investment advisors' questions and answers, updates the knowledge base, and thus improves the accuracy of subsequent questions and answers.

[0019] The method of this invention supports multimodal data processing; for example, the knowledge base module can store and retrieve text reports, images, charts, and structured tables. Modules communicate via REST APIs or message queues, ensuring loose coupling and scalability. This method can be applied to intelligent question-answering and decision support systems in professional fields such as programming, law, healthcare, and finance, significantly improving the accuracy, real-time performance, and reliability of knowledge management.

[0020] The above embodiments are merely examples. Those skilled in the art can adjust the parameters and components according to actual needs without departing from the core concept of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for constructing a knowledge base engine based on a large language model, characterized in that, Includes the following steps: The system receives and processes user input in natural language questions and answers, note-taking instructions, and knowledge base construction instructions through a question-and-answer central module, outputting corresponding question-and-answer results, structured notes, and knowledge entries. The note-taking module receives output from the question-and-answer central module, direct user input, and knowledge base iterative optimization input, performs structured processing through the central module, and outputs standard knowledge units. The knowledge base module receives standard knowledge units from the note-taking module, raw user input data, and query requests from the central module, performs structured knowledge storage, multimodal retrieval, and application processing, and feeds back the retrieval results to the central module to support answer generation. The question-and-answer central module, note-taking module, and knowledge base module form a triangular closed-loop iterative system, enabling knowledge to be deposited from the question-and-answer central module through the note-taking module to the knowledge base module. Simultaneously, the knowledge base module optimizes the accuracy and interpretability of the central module's output through retrieval enhancement and feedback. This retrieval enhancement and feedback optimization are implemented through traceable retrieval enhancement generation steps, specifically including: Upon receiving a user query, a retrieval judgment reflection token is first generated. The large language model then determines whether a knowledge base retrieval is necessary to generate a reliable answer. If not, the answer is generated directly based on the knowledge within the large language model. If so, a retrieval request is initiated to the knowledge base module. For the preliminary retrieval results returned by the knowledge base module, a paragraph relevance reflection token is generated. The large language model evaluates the relevance score of each paragraph to the query intent, filtering paragraphs below a set threshold. Highly relevant paragraphs are combined with the user query input to the large language model to generate the final answer, and a verification reflection token is generated to ensure the answer is strictly based on retrieval evidence and is complete. The generation and evaluation processes of the retrieval judgment reflection token, paragraph relevance reflection token, and verification reflection token are all recorded in a log database, achieving end-to-end knowledge traceability and system interpretability.

2. The method according to claim 1, characterized in that, It also includes an automatic knowledge extraction step: periodically or based on an event-triggered mechanism, it analyzes high-value user Q&A records, user-submitted notes, and daily workflow data generated by the Q&A hub module, and uses the inductive and summarizing capabilities of the large language model to automatically extract unstructured data into clearly structured and factually accurate knowledge items. After being formatted into standard knowledge units by the note module, the data is automatically or manually updated to the knowledge base module.

3. The method according to claim 1, characterized in that, It also includes a self-verification step: through an independent knowledge evaluator, the triplet of question-retrieval evidence-generated answer output from the retrieval-generation process is macroscopically evaluated, and a certainty score is output, including three levels: certain, uncertain, and irrelevant; based on the certainty score, graded corrective measures are implemented: for certain results, the answer is directly output; for uncertain results, a knowledge optimization process is triggered, including supplementing the retrieval, query rewriting, or marking knowledge gaps and flowing into the automatic knowledge extraction process; for irrelevant results, the generated content is discarded and transferred to manual processing.

4. The method according to claim 3, characterized in that, The knowledge optimization process specifically includes: rewriting the user query semantically using a large language model and then re-retrieving it, or automatically extracting new knowledge from high-quality question and answer records and updating it to the knowledge base module through the notes module.

5. The method according to claim 1, characterized in that, The triangular closed-loop iterative system enables the continuous self-evolution of the knowledge base. Specifically, the data generated by the question-and-answer hub module based on user interaction is precipitated into structured knowledge in the knowledge base module through the note-taking module. The updates of the knowledge base module, in turn, enhance the accuracy and interpretability of the question-and-answer hub module through retrieval feedback.

6. The method according to claim 1, characterized in that, The knowledge base module supports the storage and retrieval of multimodal knowledge, including text, images, and structured data.

7. The method according to claim 1, characterized in that, The interaction between the question-and-answer central module, the note-taking module, and the knowledge base module is achieved through an application programming interface or a message queue, ensuring loose coupling and scalability of the system architecture.

8. A knowledge base engine construction system based on a large language model, used to implement the method of any one of claims 1 to 7, characterized in that, include: The question-and-answer hub module is used to receive and process user input, and output question-and-answer results, structured notes, and knowledge entries; The note-taking module receives and temporarily stores outputs from the question-and-answer hub module, user-generated notes, and optimized inputs from the knowledge base, and outputs standard knowledge units. The knowledge base module receives standard knowledge units from the note-taking module, original user input, and query requests from the question-and-answer hub module, and performs structured storage, multimodal retrieval, and feedback processing. These three modules are interconnected through interfaces or message queues to form a triangular closed-loop iterative architecture, enabling automatic knowledge extraction, traceable retrieval enhancement generation, and introspective verification.

Images