Method for constructing vectorized knowledge base based on dynamic fields for RAG system

Abstract

The invention discloses a method for constructing a vectorized knowledge base based on dynamic fields for an RAG system, and the method comprises the following steps: 1, designing knowledge base fields: designing vectorized knowledge base fields, including static fields and dynamic fields, according to needs; step 2, document analysis and vectorization: analyzing the knowledge documents in various formats to generate a plurality of text blocks, and performing vectorization representation on each text block; step 3, constructing an index: constructing an index for each text block; and 4, determining a recall mechanism, improving the recall accuracy and coverage rate by adopting a text matching or vector similarity mode, and serving the constructed knowledge base for the RAG system. According to the method, a user can be helped to design the knowledge base based on dynamic fields, the construction process of the knowledge base is simplified, the construction efficiency of the vectorized knowledge base is improved, and later data updating and maintenance are more intelligent.

Description

Technical Field

[0001] The present invention relates to the field of computer natural language information processing and artificial intelligence technology, and in particular to a method for constructing a vectorized knowledge base based on dynamic fields for a RAG system. Background Art

[0002] With the rapid progress of RAG (Retrieval-Augmented Generation) technology and its wide application in engineering fields, building a flexible and powerful vectorized knowledge base has become increasingly important.

[0003] However, due to the lack of support for dynamic fields, the system is unable to adapt to changes in data structure, which not only limits the scalability of the system, but also affects its adaptability. Although the traditional predefined fixed field method can ensure that data follows a unified format and data type, thereby maintaining data consistency and accuracy, it is clumsy and cumbersome when dealing with variable data. The addition of new fields or tables often requires complex migration and update operations.

[0004] In order to overcome this limitation, the present invention introduces dynamic field construction technology, which can "dynamically change" according to the data added by the user. The introduction of dynamic fields enables the knowledge base to store and manage data of different structures without changing the database schema. This means that when inserting data, users no longer need to strictly adhere to pre-defined fields, but can flexibly add data in JSON format like operating a NoSQL database. This flexibility is crucial for rapidly changing fields and can easily adapt to new data types and structures.

[0005] Therefore, the present invention provides a method for constructing a vectorized knowledge base based on dynamic fields, proposes a complete set of construction processes, improves the construction efficiency of the vectorized knowledge base, and makes subsequent data updating and maintenance more intelligent. Summary of the invention

[0006] In order to achieve the above objectives, the present invention provides a method for constructing a vectorized knowledge base based on dynamic fields for a RAG system, which helps users to design a vectorized knowledge base based on dynamic fields, simplifies the construction process, improves the construction efficiency of the vectorized knowledge base, and makes it more intelligent.

[0007] The present invention discloses a method for constructing a vectorized knowledge base based on dynamic fields for a RAG system, comprising the following steps:

[0008] Step 1: Design knowledge base fields. Design vectorized knowledge base fields as needed, including static fields and dynamic fields.

[0009] Step 2: document parsing and vectorization: parsing knowledge documents in various formats to generate multiple text blocks, and vectorizing each of the text blocks;

[0010] Step 3: construct an index for each of the text blocks;

[0011] Step 4: Determine the recall mechanism, use text matching or vector similarity to improve the accuracy and coverage of the recall, and use the constructed knowledge base to serve the RAG system.

[0012] Preferably, step 2 is as follows:

[0013] Step 2.1, data preparation, according to the expected knowledge base to be constructed, select representative knowledge documents to form the required data;

[0014] Step 2.2, data parsing, parsing the knowledge documents in various formats, generating structured text from title to content, and collecting metadata of the structured text;

[0015] The metadata includes various information such as page number, file name, language and text type to meet the field design requirements;

[0016] Step 2.3, data segmentation, segmenting the processed structured text into blocks, segmenting the structured text into blocks according to a fixed maximum length to form a plurality of text blocks, and making the contents corresponding to the same title all be segmented into the same text block;

[0017] Step 2.4, data vectorization, using a semantic vector model to vectorize each text block, and storing the vectorized text blocks belonging to the same knowledge base in the same "set".

[0018] More preferably, in step 2.3, the size of each text block is dynamically adjusted according to the contextual constraints of the RAG system.

[0019] More preferably, in step 2.3, an overlap is set between every two consecutive text blocks.

[0020] More preferably, step 3 is as follows:

[0021] Step 3.1, storing the vector representation of each text block and the corresponding data of the text block as an index;

[0022] Step 3.2: Based on the field range required by the design, the collected metadata is used to construct the index of the static field and the dynamic field;

[0023] Step 3.3: Set the update mechanism of the index to ensure that the dynamic field can be updated in real time according to the newly added data to ensure the timeliness of the search results.

[0024] Preferably, the query language should support querying the dynamic fields, allowing users to perform complex searches based on different field types.

[0025] Preferably, the system being tested is able to intelligently recall the most relevant information based on the user's historical behavior and preferences.

[0026] Preferably, for the search results, a summary can be dynamically generated and the query terms can be highlighted.

[0027] Beneficial effects of the present invention:

[0028] The present invention can help users to design knowledge bases based on dynamic fields, simplify the knowledge base construction process, improve the construction efficiency of vectorized knowledge bases, and make subsequent data updating and maintenance more intelligent.

[0029] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The knowledge base construction process in one embodiment of the present invention is shown.

[0031] Figure 2 The document parsing process in one embodiment of the present invention is shown.

[0032] Figure 3 The index building process in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] The present invention involves a question-answering system based on RAG technology and a large language model, which are explained as follows:

[0034] 1. The question-answering system based on RAG technology, referred to as the RAG system, is a natural language processing technology framework that combines retrieval and generation. It first retrieves information related to the input question from a predefined knowledge base, and then inputs this information as additional context into the generative model to generate answers, thereby improving the accuracy of the answers, ensuring data privacy and security, and enhancing explainability.

[0035] 2. The semantic vector model, also known as the embedding model, can convert human language into a vector representation in a high-dimensional space to capture semantic information.

[0036] Example

[0037] like Figures 1 to 3 As shown, the method for constructing a vectorized knowledge base based on dynamic fields for a RAG system includes the following steps:

[0038] Step 1: Knowledge base field design: Design vectorized knowledge base fields as needed, including static fields and dynamic fields.

[0039] Step 2: document parsing and vectorization: parsing knowledge documents in various formats to generate multiple text blocks, and vectorizing each of the text blocks;

[0040] Step 3: construct an index for each of the text blocks;

[0041] Step 4: Determine the recall mechanism, use text matching or vector similarity to improve the accuracy and coverage of the recall, and use the constructed knowledge base to serve the RAG system.

[0042] In some embodiments, in step 1, the static fields include the name, format, and source of the knowledge document; and the dynamic fields include language, page number, date, and location.

[0043] In some embodiments, step 2 is as follows:

[0044] Step 2.1: Data preparation: According to the expected knowledge base to be built, select representative knowledge documents to form the required data; in practical applications, using documents in multiple formats can meet the breadth and depth requirements of the knowledge base collection scope.

[0045] Step 2.2, data parsing, parsing the knowledge documents in various formats, generating structured text from title to content, and collecting metadata of the structured text;

[0046] In actual applications, metadata including page numbers, file names, languages, and text types are collected to meet the field design requirements and facilitate subsequent index construction.

[0047] Step 2.3, data segmentation, segmenting the processed structured text into blocks, segmenting the structured text into blocks according to a fixed maximum length to form a plurality of text blocks, and making the contents corresponding to the same title all be segmented into the same text block;

[0048] Step 2.4: Data vectorization: Use the semantic vector model to vectorize the structured text behind each text block, and store the vectorized text blocks belonging to the same knowledge base in the same "set".

[0049] In some embodiments, in step 2.3, the size of each text block is dynamically adjusted according to the contextual constraints of the RAG system.

[0050] In some embodiments, in step 2.3, an overlap is set between every two consecutive text blocks.

[0051] In practical applications, setting overlap between every two consecutive text blocks can avoid splitting a single context into multiple parts, which will create duplicate text data between blocks.

[0052] In some embodiments, step 3 is as follows:

[0053] Step 3.1, storing the vector representation of each text block and the corresponding data of the text block as an index;

[0054] Step 3.2: Use the collected metadata to build indexes for static and dynamic fields according to the field range required by the design;

[0055] Step 3.3: Set up the index update mechanism to ensure that dynamic fields can be updated in real time based on newly added data to ensure the timeliness of search results.

[0056] In practical applications, the principle of real-time update can ensure the timeliness of index updates.

[0057] In certain embodiments, the query language should support queries on dynamic fields, allowing users to perform complex searches based on different field types.

[0058] In certain embodiments, the piloted system is able to intelligently recall the most relevant information based on the user's historical behavior and preferences.

[0059] In some embodiments, for search results, summaries can be dynamically generated and query terms can be highlighted.

[0060] In practical applications, the joint retrieval of text matching, dense vector and sparse vector similarity can improve the accuracy and coverage of recall.

[0061] The present invention can provide a better user experience through the above means.

[0062] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for constructing a vectorized knowledge base based on dynamic fields for a RAG system; characterized in that: The steps include: Step 1: Design knowledge base fields. Design vectorized knowledge base fields as needed, including static fields and dynamic fields. Step 2: document parsing and vectorization: parsing knowledge documents in various formats to generate multiple text blocks, and vectorizing each of the text blocks; Step 3: Build an index for each text block; Step 4: Determine the recall mechanism, use text matching or vector similarity to improve the accuracy and coverage of the recall, and use the constructed knowledge base to serve the RAG system.

2. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 1, characterized in that: Step 2 is as follows: Step 2.1, data preparation, according to the expected knowledge base to be constructed, select representative knowledge documents to form the required data; Step 2.2, data parsing, parsing the knowledge documents in various formats, generating structured text from title to content, and collecting metadata of the structured text; The metadata includes various information such as page number, file name, language and text type to meet the field design requirements described in step 1 of claim 1; Step 2.3, data segmentation, segmenting the processed structured text into blocks, segmenting the structured text into blocks according to a fixed maximum length to form a plurality of text blocks, and making the contents corresponding to the same title all be segmented into the same text block; Step 2.4: Data vectorization: Use a semantic vector model to vectorize each text block, and store the vectorized text blocks belonging to the same knowledge base in the same "set".

3. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 2, characterized in that: In step 2.3, the size of each text block is dynamically adjusted according to the contextual constraints of the RAG system.

4. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 2, characterized in that: In step 2.3, an overlap is set between every two consecutive text blocks.

5. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 1, characterized in that: Step 3 is as follows: Step 3.1, storing the vector representation of each text block and the corresponding data of the text block as an index; Step 3.2: Based on the field range required by the design, the collected metadata is used to construct the index of the static field and the dynamic field; Step 3.3: Set the update mechanism of the index to ensure that the dynamic field can be updated in real time according to the newly added data to ensure the timeliness of the search results.

6. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 1, characterized in that: The query language should support queries on the dynamic fields, allowing users to perform complex searches based on different field types.

7. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 1, characterized in that: The system being tested is able to intelligently recall the most relevant information based on the user’s historical behavior and preferences.

8. The method for constructing a vectorized knowledge base based on dynamic fields for a RAG system according to claim 1, characterized in that: For search results, summaries can be generated dynamically and query terms can be highlighted.

Images