RAG knowledge base construction method based on LLM dynamic text division

Abstract

The invention provides an LLM dynamic text division-based RAG knowledge base construction method, which comprises the following steps of: adjusting an LLM dynamic text division scheme, carrying out textualization processing on a feasibility research report of a power grid system, then carrying out text block division and vectorization processing based on semantic segmentation, constructing an RAG vector database through an Embedding model, and storing the RAG vector database. And adjusting a set parameter training model to obtain an optimal performance model meeting conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of power information processing and artificial intelligence, and in particular to a RAG knowledge base construction method based on LLM dynamic text partitioning. Background Art

[0002] Dynamic text segmentation refers to dynamically dividing longer texts into smaller paragraphs or fragments so that the large language model (LLM) can process and understand these texts more efficiently. This technology is particularly important when processing long texts or documents. It mainly consists of the following main steps: Before segmenting the text, the text needs to be preprocessed, including removing useless characters, stop words, etc. Then, the text is divided into several paragraphs based on the contextual semantics and logical structure. The segmented paragraphs are input into the LLM for processing to improve processing efficiency and accuracy.

[0003] The level of granularity of document segmentation is critical to retrieval performance, as ineffective segmentation strategies may result in segments that lack context or contain too much irrelevant information, thereby hurting retrieval model performance. In addition to common levels of granularity such as sentences or paragraphs, other advanced approaches can be employed. For example, recursive character segmentation segments text based on a hierarchy of delimiters such as paragraph separators, line breaks, spaces, and single characters. This approach better respects the structure of the document but may lack contextual understanding. To address this issue, semantic-based segmentation uses embeddings to cluster semantically similar text segments. This approach ensures that segments maintain meaningful context and coherence by identifying breakpoints where the embedding distance changes significantly.

[0004] Retrieval-augmented generation (RAG) is the process of optimizing large language model outputs to reference authoritative knowledge bases outside of the training data before generating a response. Large language models (LLMs) are trained with massive amounts of data, using billions of parameters to generate raw output for tasks such as answering questions, translating languages, and completing sentences. Building on the capabilities of LLMs, RAG extends them to access internal knowledge bases for specific domains or organizations, all without retraining the model. This improves the output of LLMs so that they remain relevant, accurate, and useful in a variety of contexts. Summary of the invention

[0005] Utilizing the development achievements of the above-mentioned prior art, the present invention focuses on the auxiliary specification technology of feasibility study reports for power grid project. In order to better construct the RAG knowledge base, a RAG knowledge base construction method based on LLM dynamic text division is proposed. By adjusting the LLM dynamic text division scheme, the current massive feasibility study library is vectorized based on semantic segmentation, and a RAG knowledge base is constructed, which improves the retrieval capability and operation efficiency of the overall system.

[0006] It targets the field of power systems and uses LLM (Large Language Model) dynamic text segmentation technology to build a RAG (Retrieval Enhanced Generator) knowledge base, aiming to improve the analysis, retrieval and optimization capabilities of power grid system feasibility study reports. Through the LLM dynamic text segmentation technology, the feasibility study report related to the power grid system is first processed into text, then the text blocks are divided and vectorized, and then the RAG vector database is built through the Embedding model. Finally, the model is trained by continuously adjusting the set parameters to obtain the optimal performance model that meets the conditions. The present invention focuses on the field of power grids and can provide relevant assistance for the polishing optimization and abstract retrieval of feasibility study technical reports.

[0007] The present invention specifically adopts the following technical solutions: A method for constructing a RAG knowledge base based on LLM dynamic text partitioning: By adjusting the LLM dynamic text partitioning scheme, the feasibility study report of the power grid system is text-processed, and then the text block division and vectorization processing based on semantic segmentation are performed. The RAG vector database is built through the Embedding model for storage, and the model is trained by adjusting the set parameters to obtain the optimal performance model that meets the conditions.

[0008] Furthermore, the text processing of the power grid system feasibility study report includes data cleaning and preprocessing, including removing invalid characters, correcting spelling errors, and unifying the format to obtain structured data.

[0009] Furthermore, on the basis of the structured data, according to the structural characteristics of the feasibility study report in the power grid field, the document_loaders in LLM are modified to pre-process the plain text content of the document according to the structure of the feasibility study report. The specific processing method is: first record the title of the document, then traverse each paragraph in the document, and track the current chapter title, content and path; if the current paragraph is a title, save the current chapter and record the new title level; if the current paragraph is content, append it to the current chapter content; finally, save each chapter according to the document structure, and the internal structure of each chapter is "Title One / nTitle Two / n... / nMain Text Content".

[0010] Furthermore, the text block segmentation is specifically as follows: LumberChunker is combined with LLM to dynamically segment the text to extract the semantic information of the text, and generate multiple text segment groups with semantic independence. G in G i Indicates the first i text fragments to obtain a text block with complete semantic information.

[0011] Furthermore, the specific steps of dynamically dividing the text and extracting the semantic information of the text by using LLM in combination with LumberChunker are as follows: first, each paragraph is uniquely identified by an incremental ID number, and each paragraph is serially connected into a group in sequence. G , until the total number of tags exceeds a predetermined threshold, representing the size of the text block; paragraph group G is provided as input to the language model LLM, through which the current paragraph is obtained G i The content of a specific paragraph in the document differs from the previous text; the document is split into blocks by iterative processing until all paragraphs are processed.

[0012] Furthermore, the vectorization processing based on semantic segmentation builds a RAG vector database through the Embedding model for storage as follows: the text fragments completed by dynamic division are vectorized by using the Embedding model, and then the vectorized text fragments are stored in the RAG vector database; the vectorized text fragments are stored in the vector database of the RAG system; the vector database allows the retrieval of the most relevant power system document fragments by vectorized matching; the vectorization processing of the text fragments completed by dynamic division using the Embedding model is specifically as follows: the text fragments completed by dynamic division are converted into high-dimensional vector representations using the Embedding model, and are achieved by one of word vector retrieval, word vector averaging, and cosine similarity calculation.

[0013] Furthermore, the training model by adjusting the set parameters to obtain the optimal performance model that meets the conditions is to perform retrieval training through a vector database, repeat the process of text block division and vectorization processing based on semantic segmentation to modify the context size θ , and finally obtain the optimal power technology solution retrieval model M θ .

[0014] Furthermore, the process of repeatedly performing text block segmentation and vectorization based on semantic segmentation is repeated to modify the context size. θ , and finally obtain the optimal power technology solution retrieval model Mθ The specific process is: by changing the set threshold parameters of the model to observe whether the model is improved, find the best group of effects, and thus obtain the optimal model; the ability of the model is measured by the discounted cumulative gain DCG@k and the recall rate Recall@k; the discounted cumulative gain evaluates the usefulness or gain of the document based on the position of the document in the result list; the recall rate is used to measure the proportion of relevant items retrieved in the first k results.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0016] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0017] Compared with the prior art, the present invention and its preferred embodiment propose a RAG knowledge base construction method based on LLM (large language model) dynamic text segmentation technology, which is specifically used for document processing and retrieval optimization in the field of power systems. Through LumberChunker technology, the present invention can dynamically divide documents such as feasibility study reports related to power grid systems into semantically independent text blocks to ensure that each text fragment maintains the integrity and independence of the content. Subsequently, these text fragments are vectorized using the Embedding model to construct an efficient RAG vector database. By adjusting the context window size of the model, the present invention optimizes the retrieval accuracy and recall rate, and finally obtains the optimal power technology solution retrieval model. This method significantly improves the efficiency and accuracy of document retrieval in the field of power systems, especially when processing long documents, it can effectively avoid information redundancy and interference from irrelevant content. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: Figure 1 is a flow chart of a method for generating a RAG knowledge base by using LLM to dynamically divide text according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the architecture of the partitioning method according to an embodiment of the present invention; Figure 3 It is a schematic diagram of the experimental results of an embodiment of the present invention when obtaining the optimal text block size.

[0019] Figure 4 It is an internal structure diagram of each chapter of the embodiment of the present invention. DETAILED DESCRIPTION

[0020] Hereinafter, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings. According to these detailed descriptions, those skilled in the art can clearly understand the present application and can implement the present application. Without violating the principles of the present application, the features in the various embodiments can be combined to obtain new implementations, or certain features in certain embodiments can be replaced to obtain other preferred implementations.

[0021] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are specifically described in detail with reference to the accompanying drawings: The embodiment of the present invention uses the LLM dynamic text segmentation technology to first text-process the feasibility study report related to the power grid system, then divide and vectorize the text blocks, and then build a RAG vector database through the Embedding model. Finally, by continuously adjusting the set parameter training model, the optimal performance model that meets the conditions is obtained.

[0022] Among them, the implementation of RAG knowledge base construction based on LLM dynamic text partitioning specifically includes the following steps: Figure 1 As shown; Step 1: Collect text data in the field of power systems, such as feasibility study reports, demand analysis reports, etc. Format these documents, clean the data and ensure that they can be used for model training. Extract the important parts of the documents and convert them into structured data as the basic data set for building the knowledge base.

[0023] First, collect the text data of feasibility study reports in the field of power grid, which usually come from the internal database of power grid enterprises, industry research reports, etc. Then, clean and preprocess these data, including removing invalid characters, correcting spelling errors, unifying the format, etc., to make them structured data suitable for analysis. At the same time, according to the content of the report, divide it into different chapters and paragraphs to facilitate subsequent processing.

[0024] This paper collects text data from feasibility study reports in the power grid field, and cleans and preprocesses them to make them structured data suitable for analysis.

[0025] Step 2: Adjust LumberChunker for power system text processing, deploy LLM locally, and adjust document_loaders in LumberChunker according to the characteristics of power system domain documents.

[0026] Aiming at the complex structure of the feasibility study report in the power grid field, the present invention has specially optimized and customized the document_loaders module in LLM. The optimized document_loaders can intelligently identify and parse the title and text content in the research report, and its workflow is meticulous and efficient: First, it will read and record the title of the entire document, which is the core part of the document information. Next, it will traverse the document paragraph by paragraph and conduct a detailed analysis of each paragraph. In this process, it will continue to track the current chapter title, chapter content, and the path position of the chapter in the document. Once a paragraph is identified as a title paragraph, it will immediately save all the information of the current chapter and record the title level of the new chapter to prepare for the subsequent content processing. If a body paragraph is identified, it will append the content of the paragraph to the content of the current chapter to ensure the integrity and coherence of the chapter content. Finally, according to the original structure of the document, document_loaders will save the detailed information of each chapter one by one. The internal structure of these chapters is saved in a clear and concise form, which lays a solid foundation for subsequent processing and analysis. Finally, each chapter is saved according to the document structure. The internal structure of each chapter is as follows Figure 4 As shown: Therefore, in this step, according to the structural characteristics of the feasibility study report in the power grid field, document_loaders is modified so that it can pre-process the plain text content of the document according to the structure of the feasibility study report. The specific processing method is: first record the title of the document, then traverse each paragraph in the document, and track the current chapter title, content and path. If the current paragraph is a title, save the current chapter and record the new title level; if the current paragraph is content, append it to the current chapter content. Finally, save each chapter according to the document structure, and the internal structure of each chapter is "Title 1 / nTitle 2 / n... / nMain text content" Step 3: Figure 2 As shown in the figure, LLM is used to parse the input data to obtain text content that meets the needs of the power grid field. Then, LLM is used in combination with LumberChunker to dynamically divide the text, extract the semantic information of the text, and generate multiple text fragment groups with semantic independence. G ,in Indicates the first The core purpose of this step is to obtain a text block with complete semantic information.

[0027] In step 2 and step 3, the dynamic text segmentation method in LumberChunker is combined to make it more suitable for data in the power grid field. LumberChunker is a text segmentation method that uses a language model to dynamically segment documents into blocks with complete semantic information. First, the chapters and paragraphs obtained in step 2 are uniquely identified by an incremental ID number, and each paragraph is serialized into a group in sequence. G , until the total number of tags exceeds a predetermined threshold θ , this threshold in this embodiment represents the size of the divided text block. θ Too small a value can result in loss of semantic content when segmenting larger paragraphs; θ If the value is too large, too much context may be provided to the model, which may harm the accuracy of knowledge base reasoning retrieval. G is provided as input to the language model LLM, and the large model will indicate the current paragraph G i The content of a particular paragraph in the block is significantly different from the previous text, that is, the semantics is different. This detection marks the end of a block and defines G i+1 The beginning of a paragraph. The document is processed iteratively and continues to be split into chunks in a loop until all paragraphs have been processed.

[0028] Step 4: Use the Embedding model to vectorize the text segments that have been dynamically divided, and then store the vectorized text segments into the RAG vector database.

[0029] Step 5. Build a vector database. Store the vectorized text fragments in the vector database of the RAG system. The database allows the retrieval of the most relevant power system document fragments by vectorized matching. The semantically independent fragments generated by LumberChunker can significantly improve the accuracy of retrieval and avoid interference from redundant or irrelevant information. Among them, the vector database is used to store the vectorized text vector content and is used for subsequent feasibility study report retrieval and optimization polishing functions.

[0030] Preferably, in steps 4 and 5, the Embedding model is used to convert the dynamically divided text segments into high-dimensional vector representations, mainly through word vector retrieval, word vector averaging, cosine similarity calculation, etc. The vector database is used to store the vectorized text vector content and is used for subsequent feasibility study report retrieval and optimization polishing functions.

[0031] Step 6: Perform retrieval training on the vector database obtained in step 5, repeat steps 3 and 4, and modify the context size θ , and obtain the optimal power technology solution retrieval modelM θ .

[0032] By changing the set threshold parameters of the model, we can observe whether the model has improved, find the best group of results, and get the optimal model. The ability of the model is mainly measured by two parameters: DCG@k and Recall@k. DCG@k (discounted cumulative gain) evaluates the usefulness or gain of a document based on its position in the result list, taking into account that highly relevant documents should be penalized if they appear at a lower position in the list. If relevant documents appear at the top of the ranking list, the DCG value will be higher.

[0033] Recall@k measures the proportion of relevant items retrieved in the top k results, which provides the system's ability to retrieve relevant items in the top k positions. The DCG at rank k is defined as: The recall at rank k is defined as: .

[0034] like Figure 3 The following are the experimental results for obtaining the optimal text block size.

[0035] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0036] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0037] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0038] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0039] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0040] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0041] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0042] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0043] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

[0044] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of RAG knowledge base construction methods based on LLM dynamic text division under the inspiration of the present invention. All equal changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A method for constructing a RAG knowledge base based on LLM dynamic text partitioning, characterized by: By adjusting the LLM dynamic text partitioning scheme, the feasibility study report of the power grid system is processed into text, and then the text block division and vectorization processing based on semantic segmentation are carried out. The RAG vector database is built through the Embedding model for storage. By adjusting the set parameters to train the model, the optimal performance model that meets the conditions is obtained.

2. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 1 is characterized in that: The text processing of the power grid system feasibility study report includes data cleaning and preprocessing, including removing invalid characters, correcting spelling errors, and unifying the format to obtain structured data.

3. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 1 is characterized in that: On the basis of the structured data, according to the structural characteristics of the feasibility study report in the power grid field, the document_loaders in LLM are modified to pre-process the plain text content of the document according to the structure of the feasibility study report. The specific processing method is: first record the title of the document, then traverse each paragraph in the document, and track the current chapter title, content and path; if the current paragraph is a title, save the current chapter and record the new title level; if the current paragraph is content, append it to the current chapter content; finally, save each chapter according to the document structure, and the internal structure of each chapter is "Title One / nTitle Two / n… / nMain Text Content".

4. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 1 is characterized in that: The text block segmentation is specifically as follows: LumberChunker is used to dynamically segment the text using LLM to extract the semantic information of the text, and generate multiple text segment groups with semantic independence. G in G i Indicates the first i text fragments to obtain a text block with complete semantic information.

5. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 4 is characterized in that: The specific steps of using LLM to dynamically divide text and extract semantic information of text in combination with LumberChunker are as follows: first, each paragraph is uniquely identified by an increasing ID number, and each paragraph is serially connected into a group in sequence. G , until the total number of tags exceeds a predetermined threshold, representing the size of the text block; paragraph group G is provided as input to the language model LLM, through which the current paragraph is obtained G i The content of a specific paragraph in the document differs from the previous text; the document is split into blocks by iterative processing until all paragraphs are processed.

6. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 1 is characterized in that: The vectorization processing based on semantic segmentation builds a RAG vector database for storage through the Embedding model as follows: the text fragments completed by dynamic division are vectorized by using the Embedding model, and then the vectorized text fragments are stored in the RAG vector database; the vectorized text fragments are stored in the vector database of the RAG system; the vector database allows the retrieval of the most relevant power system document fragments by vectorized matching; the vectorization processing of the text fragments completed by dynamic division using the Embedding model is specifically as follows: the text fragments completed by dynamic division are converted into high-dimensional vector representations using the Embedding model, and are achieved by one of word vector retrieval, word vector averaging, and cosine similarity calculation.

7. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 5 is characterized in that: The method of adjusting the set parameters to train the model to obtain the optimal performance model that meets the conditions is to perform retrieval training through a vector database, repeat the process of text block division and vectorization processing based on semantic segmentation to modify the context size. θ , and finally obtain the optimal power technology solution retrieval model M θ .

8. The RAG knowledge base construction method based on LLM dynamic text partitioning according to claim 7 is characterized in that: The process of repeatedly performing text block division and vectorization based on semantic segmentation is repeated to modify the context size. θ , and finally obtain the optimal power technology solution retrieval model M θ The specific process is: by changing the set threshold parameters of the model to observe whether the model is improved, find the best group of effects, and thus obtain the optimal model; the ability of the model is measured by the discounted cumulative gain DCG@k and the recall rate Recall@k; the discounted cumulative gain evaluates the usefulness or gain of the document based on the position of the document in the result list; the recall rate is used to measure the proportion of relevant items retrieved in the first k results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the RAG knowledge base construction method based on LLM dynamic text partitioning as described in any one of claims 1 to 8 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a RAG knowledge base based on LLM dynamic text partitioning as described in any one of claims 1 to 8 are implemented.

Images