Self-evolutionary RAG knowledge base construction method and system

Abstract

The invention relates to the technical field of knowledge base construction methods, in particular to a self-evolution RAG knowledge base construction method and system. According to the technical scheme, the self-evolution RAG knowledge base construction system comprises a knowledge retrieval and storage module, an information retrieval and preprocessing module and a large model processing and reply generation module; compared with a traditional self-evolutionary RAG knowledge base construction system, the self-evolutionary RAG knowledge base construction system generally needs to manually and continuously track updating of knowledge content and manually update the knowledge base, and needs a large amount of workers to update the content to a vector database, and the self-evolutionary RAG knowledge base construction system can achieve automatic quality evaluation and feedback through an automatic quality evaluation and feedback mechanism. According to the method, high-quality content can be automatically updated to the vector database, the labor cost is reduced, meanwhile, the system combines the advantages of a search engine and a large model, newest knowledge can be obtained through the search engine when no related content exists in the vector database, summarization is conducted, replies are generated, and the consistency and accuracy of data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge base construction methods, and in particular to a self-evolving RAG knowledge base construction method and system. Background Art

[0002] RAG, or Retrieval Augmented Generation, is an artificial intelligence technology that combines information retrieval and text generation. It improves the quality and accuracy of the output by retrieving relevant information from a large document collection and then providing this information as background knowledge to a large language model. RAG technology is particularly suitable for knowledge-intensive scenarios or specific field applications that require continuous updating of knowledge. Its advantage is that there is no need to retrain the large language model for specific tasks.

[0003] The existing self-evolving RAG knowledge base construction system generally requires manual work to continuously track the update of knowledge content and manually update the knowledge base, and requires a large amount of manual work to update the content to the vector database, resulting in low efficiency in knowledge base construction.

[0004] In view of the problem that the existing self-evolving RAG knowledge base construction system generally requires manual continuous tracking of knowledge content updates and manual updating of the knowledge base, resulting in low knowledge base construction efficiency, the self-evolving RAG knowledge base construction system can automatically update high-quality content to the vector database through automatic quality assessment and feedback mechanism, reducing labor costs. At the same time, the system combines the advantages of search engines and large models. When there is no relevant content in the vector database, the system can obtain the latest knowledge through search engines, summarize and generate replies, and improve data consistency and accuracy. Summary of the invention

[0005] In order to overcome the existing self-evolving RAG knowledge base construction system, it is generally necessary to manually track the update of knowledge content and manually update the knowledge base, and a large amount of manual work is required to update the content to the vector database, resulting in low efficiency in knowledge base construction.

[0006] The technical solution of the present invention is: a method for constructing a self-evolving RAG knowledge base, comprising:

[0007] S11: First, prepare and preliminarily organize the knowledge data and preprocess the collected data information;

[0008] S12: Establish a vector database and retrieve relevant information through the similarity between vectors;

[0009] S13: When there is no relevant question in the vector database, the question is uploaded to the search engine;

[0010] S14: The search engine searches for the problem and gives it to the big model for summary;

[0011] S15: Combined with the automated quality assessment system, high-quality answer content is screened and automatically updated to the vector database.

[0012] Preferably, when the vector database is working, the following method is included:

[0013] S21: Use a vector database to store knowledge data, convert each piece of knowledge data into a vector, and store its corresponding metadata;

[0014] S22: Build an efficient vector index to quickly retrieve relevant content, and use an efficient approximate nearest neighbor algorithm for indexing;

[0015] S23: Design an update strategy to perform batch updates after the system receives new high-quality responses, insert them into the vector database, and perform data cleaning and expired content deletion at the same time.

[0016] Preferably, when using the search engine, the following steps are included:

[0017] S31: Use a mature search engine to retrieve the latest knowledge data, convert the user's reasoning request into a query, and execute the query in the search engine;

[0018] S32: Preprocess and parse the content returned by the search engine, including denoising, content extraction and formatting, so as to pass it to the big model.

[0019] Preferably, when processing the user's reasoning request, the following steps are included:

[0020] S41: The user initiates an inference request;

[0021] S42: The system first retrieves relevant content from the vector database;

[0022] S43: If no relevant content is found, search using a search engine;

[0023] S44: input the retrieved content into the big model as background knowledge;

[0024] S45: The large model generates responses and performs quality assessment;

[0025] S46: Automatically update the high-quality responses to the vector database.

[0026] Preferably, when generating a reply in a large model, the following steps are included:

[0027] S51: Select a suitable large model to generate responses. The model has strong natural language understanding and generation capabilities.

[0028] S52: input the knowledge obtained from the vector database and the search engine into the model as background knowledge;

[0029] S53: Use context window technology to integrate this knowledge as the input of the model;

[0030] S54: The model generates response content and performs quality assessment and optimization to ensure the accuracy and completeness of the response.

[0031] Preferably, when constructing the self-evolution system, the following steps are included:

[0032] S61: Establish an automated quality assessment system to score and evaluate the generated responses and screen out high-quality content;

[0033] S62: Collect user feedback on the reply content, and optimize the model generation and vector database update strategy through the feedback data;

[0034] S63: Regularly or based on preset conditions, automatically updating the generated high-quality reply content to the vector database;

[0035] S64: Regularly retrain and adjust the knowledge data in the vector database to improve data quality and coverage.

[0036] A self-evolving RAG knowledge base construction system, including:

[0037] Knowledge retrieval and storage module: used to store and manage knowledge data;

[0038] Information retrieval and preprocessing module: used to retrieve the latest knowledge data;

[0039] Large model processing and response generation module: used to select a suitable large prediction model to generate responses.

[0040] Preferably, the knowledge retrieval and storage module is used to store and manage historically verified knowledge data. The knowledge retrieval and storage module converts the knowledge data into vector form and uses an efficient indexing algorithm to index it in order to quickly retrieve relevant content. At the same time, the knowledge retrieval and storage module is also responsible for the data update mechanism, which batch updates the newly generated high-quality response content to the vector database, and performs data cleaning and deletion of expired content.

[0041] Preferably, the information retrieval and preprocessing module is used to retrieve the latest knowledge data. When a user makes an inference request, the information retrieval and preprocessing module converts the request into a query and executes the query in an external search engine. The query results are then preprocessed and parsed, including denoising, content extraction, and formatting, so that they can be used as background knowledge input to the large model for processing.

[0042] Preferably, the large model processing and reply generation module is used to select a suitable large language model to generate replies. The large model processing and reply generation module inputs knowledge data from the vector database and search engine as background knowledge into the large model, integrates this knowledge using context window technology, and generates reply content. At the same time, the generated replies are quality evaluated and optimized to ensure their accuracy and completeness.

[0043] Preferably, the system can continuously learn from user feedback and new data generated, dynamically adjust and optimize model generation and vector database update strategies, and as the system service time increases, the data quality and coverage in the knowledge base will continue to improve, forming a virtuous self-evolution cycle. At the same time, through the continuously updated self-evolutionary RAG knowledge base construction method, it can quickly adapt to new information and changes in domain knowledge. By retrieving the latest knowledge data through the search engine and inputting it into the model as background knowledge, the system can generate accurate responses related to the current situation in real time.

[0044] Beneficial effects of the present invention:

[0045] 1. Compared with the traditional self-evolving RAG knowledge base construction system, which generally requires manual tracking of knowledge content updates and manual updates of the knowledge base, and requires a large number of manual updates to the vector database, the knowledge base construction efficiency is low. The self-evolving RAG knowledge base construction system can automatically update high-quality content to the vector database through automatic quality assessment and feedback mechanisms, reducing labor costs. At the same time, the system combines the advantages of search engines and large models. When there is no relevant content in the vector database, the system can obtain the latest knowledge through search engines, summarize and generate replies, and improve data consistency and accuracy.

[0046] 2. The system can continuously learn from user feedback and new data generated, dynamically adjust and optimize model generation and vector database update strategies. As the system service time increases, the data quality and coverage in the knowledge base will continue to improve, forming a virtuous self-evolution cycle. At the same time, through the continuously updated self-evolutionary RAG knowledge base construction method, it can quickly adapt to new information and changes in domain knowledge. By retrieving the latest knowledge data through the search engine and inputting it into the model as background knowledge, the system can generate accurate responses related to the current situation in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 What is shown is a schematic diagram of the workflow of a method for constructing a self-evolving RAG knowledge base of the present invention;

[0048] Figure 2 What is shown is a schematic diagram of a vector database workflow of a self-evolving RAG knowledge base construction method of the present invention;

[0049] Figure 3 What is shown is a schematic diagram of the search engine operation flow of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] See also Figure 1-3 The present invention provides an embodiment: a method for constructing a self-evolving RAG knowledge base, comprising:

[0052] S11: First, prepare and preliminarily organize the knowledge data and preprocess the collected data information;

[0053] S12: Establish a vector database and retrieve relevant information through the similarity between vectors;

[0054] S13: When there is no relevant question in the vector database, the question is uploaded to the search engine;

[0055] S14: The search engine searches for the problem and gives it to the big model for summary;

[0056] S15: Combined with the automated quality assessment system, high-quality answer content is screened and automatically updated to the vector database.

[0057] Preferably, when the vector database is working, the following method is included:

[0058] S21: Use a vector database to store knowledge data, convert each piece of knowledge data into a vector, and store its corresponding metadata;

[0059] S22: Build an efficient vector index to quickly retrieve relevant content, and use an efficient approximate nearest neighbor algorithm for indexing;

[0060] S23: Design an update strategy to perform batch updates after the system receives new high-quality responses, insert them into the vector database, and perform data cleaning and expired content deletion at the same time.

[0061] Preferably, when using the search engine, the following steps are included:

[0062] S31: Use a mature search engine to retrieve the latest knowledge data, convert the user's reasoning request into a query, and execute the query in the search engine;

[0063] S32: Preprocess and parse the content returned by the search engine, including denoising, content extraction and formatting, so as to pass it to the big model.

[0064] Preferably, when processing the user's reasoning request, the following steps are included:

[0065] S41: The user initiates an inference request;

[0066] S42: The system first retrieves relevant content from the vector database;

[0067] S43: If no relevant content is found, search using a search engine;

[0068] S44: input the retrieved content into the big model as background knowledge;

[0069] S45: The large model generates responses and performs quality assessment;

[0070] S46: Automatically update the high-quality responses to the vector database.

[0071] Preferably, when generating a reply in a large model, the following steps are included:

[0072] S51: Select a suitable large model to generate responses. The model has strong natural language understanding and generation capabilities.

[0073] S52: input the knowledge obtained from the vector database and the search engine into the model as background knowledge;

[0074] S53: Use context window technology to integrate this knowledge as the input of the model;

[0075] S54: The model generates response content and performs quality assessment and optimization to ensure the accuracy and completeness of the response.

[0076] Preferably, when constructing the self-evolution system, the following steps are included:

[0077] S61: Establish an automated quality assessment system to score and evaluate the generated responses and screen out high-quality content;

[0078] S62: Collect user feedback on the reply content, and optimize the model generation and vector database update strategy through the feedback data;

[0079] S63: Regularly or based on preset conditions, automatically updating the generated high-quality reply content to the vector database;

[0080] S64: Regularly retrain and adjust the knowledge data in the vector database to improve data quality and coverage.

[0081] A self-evolving RAG knowledge base construction system, including:

[0082] Knowledge retrieval and storage module: used to store and manage knowledge data;

[0083] Information retrieval and preprocessing module: used to retrieve the latest knowledge data;

[0084] Large model processing and response generation module: used to select a suitable large prediction model to generate responses.

[0085] Preferably, the knowledge retrieval and storage module is used to store and manage historically verified knowledge data. The knowledge retrieval and storage module converts the knowledge data into vector form and uses an efficient indexing algorithm to index it in order to quickly retrieve relevant content. At the same time, the knowledge retrieval and storage module is also responsible for the data update mechanism, which batch updates the newly generated high-quality response content to the vector database, and performs data cleaning and deletion of expired content.

[0086] Preferably, the information retrieval and preprocessing module is used to retrieve the latest knowledge data. When a user makes an inference request, the information retrieval and preprocessing module converts the request into a query and executes the query in an external search engine. The query results are then preprocessed and parsed, including denoising, content extraction, and formatting, so that they can be used as background knowledge input to the large model for processing.

[0087] Preferably, the large model processing and reply generation module is used to select a suitable large language model to generate replies. The large model processing and reply generation module inputs knowledge data from the vector database and search engine as background knowledge into the large model, integrates this knowledge using context window technology, and generates reply content. At the same time, the generated replies are quality evaluated and optimized to ensure their accuracy and completeness.

[0088] Preferably, the system can continuously learn from user feedback and new data generated, dynamically adjust and optimize model generation and vector database update strategies, and as the system service time increases, the data quality and coverage in the knowledge base will continue to improve, forming a virtuous self-evolution cycle. At the same time, through the continuously updated self-evolutionary RAG knowledge base construction method, it can quickly adapt to new information and changes in domain knowledge. By retrieving the latest knowledge data through the search engine and inputting it into the model as background knowledge, the system can generate accurate responses related to the current situation in real time.

[0089] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A self-evolving RAG knowledge base construction method and system; characterized by: Included are: S11: First, prepare and preliminarily organize the knowledge data and preprocess the collected data information; S12: Establish a vector database and retrieve relevant information through the similarity between vectors; S13: When there is no relevant question in the vector database, the question is uploaded to the search engine; S14: The search engine searches for the problem and gives it to the big model for summary; S15: Combined with the automated quality assessment system, high-quality answer content is screened and automatically updated to the vector database.

2. A self-evolving RAG knowledge base construction method and system according to claim 1, characterized in that: When working with vector databases, the following methods are included: S21: Use a vector database to store knowledge data, convert each piece of knowledge data into a vector, and store its corresponding metadata; S22: Build an efficient vector index to quickly retrieve relevant content, and use an efficient approximate nearest neighbor algorithm for indexing; S23: Design an update strategy to perform batch updates after the system receives new high-quality responses, insert them into the vector database, and perform data cleaning and expired content deletion at the same time.

3. A method and system for constructing a self-evolving RAG knowledge base according to claim 2, characterized in that: When using a search engine, the following steps are included: S31: Use a mature search engine to retrieve the latest knowledge data, convert the user's reasoning request into a query, and execute the query in the search engine; S32: Preprocess and parse the content returned by the search engine, including denoising, content extraction and formatting, so as to pass it to the big model.

4. A self-evolving RAG knowledge base construction method and system according to claim 3, characterized in that: When processing the user's reasoning request, the following steps are included: S41: The user initiates an inference request; S42: The system first retrieves relevant content from the vector database; S43: If no relevant content is found, search using a search engine; S44: input the retrieved content into the big model as background knowledge; S45: The large model generates responses and performs quality assessment; S46: Automatically update the high-quality responses to the vector database.

5. A method and system for constructing a self-evolving RAG knowledge base according to claim 4, characterized in that: When generating responses in a large model, the following steps are included: S51: Select a suitable large model to generate responses. The model has strong natural language understanding and generation capabilities. S52: input the knowledge obtained from the vector database and the search engine into the model as background knowledge; S53: Use context window technology to integrate this knowledge as the input of the model; S54: The model generates response content and performs quality assessment and optimization to ensure the accuracy and completeness of the response.

6. A method and system for constructing a self-evolving RAG knowledge base according to claim 5, characterized in that: When building a self-evolving system, the following steps are included: S61: Establish an automated quality assessment system to score and evaluate the generated responses and screen out high-quality content; S62: Collect user feedback on the reply content, and optimize the model generation and vector database update strategy through the feedback data; S63: Regularly or based on preset conditions, automatically updating the generated high-quality reply content to the vector database; S64: Regularly retrain and adjust the knowledge data in the vector database to improve data quality and coverage.

7. A self-evolving RAG knowledge base construction system, characterized by: Included are: Knowledge retrieval and storage module: used to store and manage knowledge data; Information retrieval and preprocessing module: used to retrieve the latest knowledge data; Large model processing and response generation module: used to select a suitable large prediction model to generate responses.

8. A self-evolving RAG knowledge base construction system according to claim 7, characterized in that: The knowledge retrieval and storage module is used to store and manage historically verified knowledge data. The knowledge retrieval and storage module converts knowledge data into vector form and uses efficient indexing algorithms to index it in order to quickly retrieve relevant content. At the same time, the knowledge retrieval and storage module is also responsible for the data update mechanism, which batch updates the newly generated high-quality response content to the vector database, and performs data cleaning and deletion of expired content.

9. A self-evolving RAG knowledge base construction system according to claim 8, characterized in that: The information retrieval and preprocessing module is used to retrieve the latest knowledge data. When a user makes an inference request, the information retrieval and preprocessing module converts the request into a query and executes the query in an external search engine. The query results are then preprocessed and parsed, including denoising, content extraction, and formatting, so that they can be used as background knowledge input to the large model for processing.

10. A self-evolving RAG knowledge base construction system according to claim 9, characterized in that: The large model processing and reply generation module is used to select a suitable large language model to generate replies. The large model processing and reply generation module inputs knowledge data from the vector database and search engine as background knowledge into the large model, integrates this knowledge using context window technology, and generates reply content. At the same time, the generated replies are quality evaluated and optimized to ensure their accuracy and completeness.

Images