RAG-based intelligent knowledge base management method and system

Abstract

The invention relates to an intelligent knowledge base management method based on RAG, and the method comprises the following steps: S1, question retrieval, S11, question vectorization processing: converting a user question into a numerical vector, coding the question by a system through a pre-trained language model which is the same as text data, and vectorizing the question; s12, carrying out vector similarity retrieval; s2, constructing a knowledge base; S21, performing a data extraction stage, namely extracting knowledge contents from an original data source; s22, segmenting the document, and splitting the text data into word block contents; s23, carrying out vectorization; and S3, answer generation. According to the overall system provided by the embodiment of the invention, the vector database is quickly constructed by utilizing the system, data retrieval is realized, and complete answers conforming to logic are generated in cooperation with the large language model, so that the development cycle of question systems in specific fields is greatly shortened, time for retrieving specific questions by users is greatly facilitated, and the user experience is improved. And question answering of the system is more logical and complete.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent knowledge base management, and in particular to a method and system for intelligent knowledge base management based on RAG. Background Art

[0002] Most of the information processing in existing knowledge base systems is based on searching keyword matching or searching the knowledge base based on document links, which results in poor user experience. At the same time, knowledge base entry relies on labeling of question information, which has high maintenance costs, and it becomes very cumbersome to update and maintain the information in the knowledge base. Retrieval-enhanced generation is a model that combines retrieval and generation technologies. It generates corresponding answers by retrieving external knowledge without repeated training, and can quickly adapt to new knowledge by updating the knowledge base. We hope to provide a method and system for intelligent knowledge base management based on RAG, which simplifies knowledge base maintenance while providing a more friendly and communicative answer presentation, thereby making up for the shortcomings of traditional knowledge bases. Therefore, we propose a method and system for intelligent knowledge base management based on RAG. Summary of the invention

[0003] The present application provides a method and system for intelligent knowledge base management based on RAG to solve the above-mentioned problems.

[0004] The present application provides a method for managing an intelligent knowledge base based on RAG, comprising the following steps:

[0005] S1, question retrieval,

[0006] S11, question vectorization processing, converting user questions into numerical vectors, and the system uses the same pre-trained language model as the text data to encode the questions and vectorize the questions;

[0007] S12, vector similarity retrieval, using multiple queries to expand user questions into more similar questions, and using vector similarity retrieval technology to find the knowledge vector most similar to the question vector in the knowledge base;

[0008] S2. Build a knowledge base.

[0009] S21, data extraction stage, i.e. extracting knowledge content from the original data source;

[0010] S22, document segmentation, splitting text data into word chunks;

[0011] S23, vectorization, refers to converting text into vectors using a pre-trained language embedding model. Fourth, vector data storage, refers to storing vector data into a vector database;

[0012] S3, answer generation,

[0013] S31, first embed the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers;

[0014] S32. The large language model generates answers related to the questions based on the prompt word template, and outputs the answers according to the predetermined output format requirements. The large language model is used to comprehensively analyze the information in the prompt words, combined with the company's internal knowledge and language processing capabilities, to generate logically coherent, accurate and meaningful answers.

[0015] Preferably, the answer generation also combines multiple queries with the retrieved knowledge to generate answers using a large language model.

[0016] Preferably, the building of the knowledge base is an offline process of converting the local data source into a vector, creating an index and storing it in a vector database.

[0017] Preferably, the question retrieval stage vectorizes the question raised by the user and uses an efficient retrieval method to find the knowledge most relevant to the question.

[0018] Preferably, in the answer generation stage, the system first embeds the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers.

[0019] A system for intelligent knowledge base management based on RAG, comprising:

[0020] An external network questioning module, the external network questioning module is connected to the parsing module through data transmission, the parsing module is connected to the matching module through data transmission, the matching module is connected to the text generating module through data transmission, and the text generating module is connected to the display module through data transmission.

[0021] Preferably, the matching module refers to extracting meta information from a natural language query request, performing pre-screening in a vectorized corpus based on the meta information to obtain a pre-screened corpus block set, performing matching in the pre-screened corpus block set based on a matching algorithm, determining the similarity between each corpus block in the pre-screened corpus block set and the natural language query request, and confirming one or more associated corpus blocks based on the similarity between each corpus block and the natural language query request.

[0022] Preferably, the text generation module identifies one or more associated corpora related to the associated corpora in the vectorized corpus based on cosine similarity.

[0023] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0024] The overall system provided in the embodiment of the present application utilizes the system to quickly build a vector database, realize data retrieval, and generate logical and complete answers in conjunction with a large language model, thereby significantly shortening the development cycle of the system for specific field questions, greatly facilitating the time for users to retrieve specific questions, and making the system's answers to questions more logical and complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is a system diagram of the present invention.

[0029] In the figure: 110, external network question module; 120, analysis module; 130, matching module; 140, text generation module; 150, display module. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] Various embodiments of the present application may exist in the form of a range. It should be understood that the description in the form of a range is only for convenience and simplicity and should not be understood as a hard limit to the scope of the present application; therefore, it should be considered that the range description has specifically disclosed all possible sub-ranges and single values ​​within the range. For example, it should be considered that the range description from 1 to 6 has specifically disclosed sub-ranges, such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as single numbers within the range, such as 1, 2, 3, 4, 5 and 6, which are applicable regardless of the range. In addition, whenever a numerical range is indicated in the present application, it is meant to include any quoted numbers (fractions or integers) within the indicated range. Unless otherwise specified, various raw materials, reagents, instruments and equipment used in the present application, etc., can be purchased from the market or can be prepared by existing equipment.

[0032] In the present application, in the absence of any contrary description, the directional words used, such as "upper" and "lower", are specifically the directions of the drawings in the accompanying drawings. In addition, in the present application, the terms "include", "comprise", etc. refer to "including but not limited to". In the present application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. Wherein A, B can be singular or plural. In the present application, "at least one" refers to one or more, and "plural" refers to two or more. "At least one", "at least one of the following" or similar expressions refer to any combination of these items, including any combination of singular items or plural items. For example, "at least one of a, b, or c" or "at least one of a, b and c" can both mean: a, b, c, ab, i.e. a and b, ac, bc or abc, where a, b, c can be single or plural, respectively.

[0033] like Figure 1 and Figure 2 As shown: This embodiment of the application provides a method for managing an intelligent knowledge base based on RAG, comprising the following steps:

[0034] S1, question retrieval,

[0035] S11, question vectorization processing, converting user questions into numerical vectors, and the system uses the same pre-trained language model as the text data to encode the questions and vectorize the questions;

[0036] S12, vector similarity retrieval, using multiple queries to expand user questions into more similar questions, and using vector similarity retrieval technology to find the knowledge vector most similar to the question vector in the knowledge base;

[0037] Specifically: Question retrieval usually involves MMR or similarity scoring to quickly obtain the knowledge closest to the question.

[0038] S2. Build a knowledge base.

[0039] S21, data extraction stage, i.e. extracting knowledge content from the original data source;

[0040] S22, document segmentation, splitting text data into word chunks;

[0041] S23, vectorization, refers to converting text into vectors using a pre-trained language embedding model. Fourth, vector data storage, refers to storing vector data into a vector database;

[0042] S3, answer generation,

[0043] S31, first embed the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers;

[0044] S32. The large language model generates answers related to the questions based on the prompt word template, and outputs the answers according to the predetermined output format requirements. The large language model is used to comprehensively analyze the information in the prompt words, combined with the company's internal knowledge and language processing capabilities, to generate logically coherent, accurate and meaningful answers.

[0045] Specifically: Answer generation in this way not only improves the ability of the overall system to quickly update and expand the knowledge base, but also ensures that the logic and completeness of the generated answers meet user expectations.

[0046] The answer generation also combines multiple queries with retrieved knowledge using a large language model to generate answers.

[0047] The construction of the knowledge base is an offline process of converting the local data source into a vector, creating an index and storing it in a vector database.

[0048] The question retrieval stage vectorizes the questions raised by the users and uses efficient retrieval methods to find the knowledge most relevant to the questions.

[0049] In the answer generation stage, the system first embeds the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers.

[0050] Specifically:

[0051] A system for intelligent knowledge base management based on RAG, comprising:

[0052] The external network questioning module 110 , the external network questioning module 110 data is connected to the parsing module 120 , the parsing module 120 data is connected to the matching module 130 , the matching module 130 data is connected to the text generating module 140 , and the text generating module 140 data is connected to the display module 150 .

[0053] The matching module 130 extracts meta information from a natural language query request, performs pre-screening in a vectorized corpus based on the meta information to obtain a pre-screened corpus set, performs matching in the pre-screened corpus set based on a matching algorithm, determines the similarity between each corpus in the pre-screened corpus set and the natural language query request, and confirms one or more associated corpuses based on the similarity between each corpus and the natural language query request.

[0054] The text generation module 140 identifies one or more associated chunks in the vectorized corpus based on cosine similarity.

[0055] Specifically: Based on the overall system, it can be integrated into the enterprise portal system or some expert knowledge base system to provide users with question-answering services;

[0056] The peripheral question module 110 is an interface or a visual window for accepting user questions, and receives the question content information transmitted by the user to the system;

[0057] The parsing module 120 uses the same vectorization processing method as the internal knowledge file to perform vectorization conversion on the text input by the user;

[0058] The text generation module 140 identifies one or more associated corpus blocks in the vectorized corpus based on cosine similarity; and feeds the associated corpus blocks back to the display module 150;

[0059] The display output module 150 outputs the answer to the question given by the large model according to the prompt word template combined with 140, outputs the answer according to the predetermined output format requirements, and displays the answer result to the user.

[0060] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. It will be apparent to those skilled in the art that various modifications to these embodiments are possible, and the general principles defined in the present application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown in the present application, but will conform to the widest scope consistent with the principles and novel features applied for by the present application.

Claims

1. A method for managing an intelligent knowledge base based on RAG, characterized in that: The steps include: S1, question retrieval, S11, question vectorization processing, converting user questions into numerical vectors, and the system uses the same pre-trained language model as the text data to encode the questions and vectorize the questions; S12, vector similarity retrieval, using multiple queries to expand user questions into more similar questions, and using vector similarity retrieval technology to find the knowledge vector most similar to the question vector in the knowledge base; S2. Build a knowledge base. S21, data extraction stage, i.e. extracting knowledge content from the original data source; S22, document segmentation, splitting text data into word chunks; S23, vectorization, refers to converting text into vectors using a pre-trained language embedding model. Fourth, vector data storage, refers to storing vector data into a vector database; S3, answer generation, S31, first embed the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers; S32. The large language model generates answers related to the questions based on the prompt word template, and outputs the answers according to the predetermined output format requirements. The large language model is used to comprehensively analyze the information in the prompt words, combined with the company's internal knowledge and language processing capabilities, to generate logically coherent, accurate and meaningful answers.

2. The method for managing an intelligent knowledge base based on RAG according to claim 1, characterized in that: The answer generation also combines multiple queries with retrieved knowledge using a large language model to generate answers.

3. The method for managing an intelligent knowledge base based on RAG according to claim 1, characterized in that: The construction of the knowledge base is an offline process of converting the local data source into a vector, creating an index and storing it in a vector database.

4. The method for managing an intelligent knowledge base based on RAG according to claim 1, characterized in that: The question retrieval stage vectorizes the questions raised by the users and uses efficient retrieval methods to find the knowledge most relevant to the questions.

5. The method for managing an intelligent knowledge base based on RAG according to claim 1, characterized in that: In the answer generation stage, the system first embeds the retrieved knowledge into the prompt word template to ensure that the relevant knowledge can be fully utilized when generating relevant answers.

6. A system for intelligent knowledge base management based on RAG, characterized in that: include: An external network questioning module (110), the external network questioning module (110) is connected to a parsing module (120) through data transmission, the parsing module (120) is connected to a matching module (130) through data transmission, the matching module (130) is connected to a text generating module (140) through data transmission, and the text generating module (140) is connected to a display module (150) through data transmission.

7. The system for managing an intelligent knowledge base based on RAG according to claim 6, characterized in that: The matching module (130) is configured to extract meta information from a natural language query request, perform pre-screening in a vectorized corpus based on the meta information to obtain a pre-screened corpus set, perform matching in the pre-screened corpus set based on a matching algorithm, determine the similarity between each corpus in the pre-screened corpus set and the natural language query request, and confirm one or more associated corpus based on the similarity between each corpus and the natural language query request.

8. The system for managing an intelligent knowledge base based on RAG according to claim 6, characterized in that: The text generation module (140) identifies one or more associated chunks in the vectorized corpus based on cosine similarity.

Images