Rag knowledge base construction method based on llm dynamic text division

Abstract

The application provides an RAG knowledge base construction method based on LLM dynamic text division. The power grid system feasibility study report is textually processed by adjusting an LLM dynamic text division scheme, text block division and vectorization processing based on semantic segmentation are performed, an RAG vector database is constructed by using an Embedding model for storage, a model is trained by adjusting and setting parameters, and an optimal performance model meeting a condition is obtained.

Description

Technical Field

[0001] This invention relates to the fields of power information processing and artificial intelligence technology, and in particular to a method for constructing a RAG knowledge base based on LLM dynamic text segmentation. Background Technology

[0002] Dynamic text segmentation refers to the dynamic division of long texts into smaller paragraphs or fragments so that Large Language Models (LLMs) can process and understand them more efficiently. This technique is particularly important when processing long texts or documents, and mainly involves the following steps: Before segmenting the text, it first needs to be preprocessed, including removing useless characters, stop words, etc. Then, based on contextual semantics and logical structure, the text is divided into several paragraphs. The segmented paragraphs are then input into the LLM for processing to improve processing efficiency and accuracy.

[0003] The granularity of document segmentation is crucial for retrieval performance, as ineffective segmentation strategies can lead to fragments lacking context or containing excessive irrelevant information, thus harming retrieval model performance. Beyond common granularities such as sentences or paragraphs, other advanced methods can be employed. For example, recursive character segmentation divides text based on a hierarchy of delimiters such as paragraph separators, line breaks, spaces, and single characters. This approach better respects document structure but may lack contextual understanding. To address this, semantic-based segmentation uses embeddings to cluster semantically similar text fragments. This method ensures that fragments retain meaningful context and coherence, identifying breakpoints by recognizing significant changes in embedding distance.

[0004] Retrieval Enhancement Generation (RAG) refers to optimizing the output of large language models to reference authoritative knowledge bases beyond the training data sources before generating responses. Large Language Models (LLMs) are trained on massive amounts of data, using billions of parameters to generate raw outputs for tasks such as answering questions, translating languages, and completing sentences. Building on the functionality of LLMs, RAG extends them to access internal knowledge bases of specific domains or organizations, all without requiring model retraining. By improving the methods of LLM output, it ensures relevance, accuracy, and usability across various contexts. Summary of the Invention

[0005] Based on the aforementioned advancements in existing technologies, this invention focuses on auxiliary designation technologies for feasibility study reports in the power grid sector. To better construct the RAG knowledge base, a method for constructing the RAG knowledge base based on LLM dynamic text partitioning is proposed. By adjusting the LLM dynamic text partitioning scheme, the existing massive feasibility study database is processed using semantic segmentation and vectorization, thereby constructing the RAG knowledge base and improving the overall system's retrieval capabilities and operational efficiency.

[0006] This invention targets the power system field, utilizing LLM (Large Language Model) dynamic text segmentation technology to construct a RAG (Retrieval Enhanced Generator) knowledge base, aiming to improve the analysis, retrieval, and optimization capabilities of feasibility study reports for power grid systems. Through LLM dynamic text segmentation technology, feasibility study reports related to power grid systems are first digitized, then divided into text blocks and vectorized. Next, an RAG vector database is constructed using an embedding model. Finally, the model is trained by continuously adjusting set parameters to obtain an optimal performance model that meets the specified conditions. This invention focuses on the power grid field and can provide relevant assistance for the polishing, optimization, and summary retrieval of feasibility study technical reports.

[0007] The present invention specifically adopts the following technical solution:

[0008] A method for constructing a RAG knowledge base based on LLM dynamic text partitioning: By adjusting the LLM dynamic text partitioning scheme, the feasibility study report of the power grid system is textualized, then the text is divided into blocks and vectorized based on semantic segmentation. The RAG vector database is constructed and stored through the Embedding model. The optimal performance model that meets the conditions is obtained by adjusting the set parameters to train the model.

[0009] Furthermore, the textualization of the power grid system feasibility study report includes data cleaning and preprocessing, including removing invalid characters, correcting spelling errors, and standardizing the format to obtain structured data.

[0010] Furthermore, based on the structured data, and according to the structural characteristics of feasibility study reports in the power grid field, the document_loaders in the LLM are modified to preprocess the plain text content of the document according to the structure of the feasibility study report. The specific processing method is as follows: first, record the title of the document; then, traverse each paragraph in the document and track the title, content, and path of the current chapter; if the current paragraph is a title, save the current chapter and record the new title level; if the current paragraph is content, append it to the content of the current chapter; finally, save each chapter according to the document structure, and the internal structure of each chapter is "title 1 / ntitle 2 / n... / nbody content".

[0011] Furthermore, the text block segmentation specifically involves: using LumberChunker with LLM to dynamically segment the text and extract semantic information, generating multiple semantically independent text segment groups. G in G i Represents the LLM dynamic partitioning after the first partition. i From a text fragment, a text block with complete semantic information is obtained.

[0012] Furthermore, the specific steps for dynamically segmenting and extracting semantic information from text using LLM with LumberChunker are as follows: First, each paragraph is uniquely identified by an incrementing ID number, and each paragraph is sequentially concatenated into a group. G The total number of markers continues until it exceeds a predetermined threshold, representing the size of the text block; paragraph groups. G It is fed as input to the language model LLM, and the current paragraph is obtained through the LLM. G i The document is segmented into blocks by iterative processing of the content of a specific paragraph, and the process continues until all paragraphs have been processed.

[0013] Furthermore, the semantic segmentation-based vectorization processing, which uses an embedding model to construct a RAG vector database for storage, involves: using the embedding model to vectorize the dynamically segmented text fragments, and then storing the vectorized text fragments in the RAG vector database; storing the vectorized text fragments in the RAG system's vector database; the vector database allows retrieval of the most relevant power system document fragments through vectorized matching; specifically, the embedding model is used to vectorize the dynamically segmented text fragments, transforming them into high-dimensional vector representations, achieved through one of the following methods: word vector retrieval, word vector averaging, or cosine similarity calculation.

[0014] Furthermore, the step of training the model by adjusting the set parameters to obtain the optimal performance model that meets the conditions involves retrieving and training using a vector database, repeatedly performing text block segmentation and semantic segmentation-based vectorization processing to modify the context size. θ Finally, the optimal power technology solution retrieval model is obtained. M θ .

[0015] Furthermore, the process of repeatedly performing text block segmentation and semantic segmentation-based vectorization is used to modify the context size. θ Finally, the optimal power technology solution retrieval model is obtained. Mθ The specific process is as follows: by changing the set threshold parameters of the model, observe whether the model improves, find the best-performing set, and thus obtain the optimal model; the model's capability is measured by the discounted cumulative gain DCG@k and the recall rate Recall@k; the discounted cumulative gain evaluates the usefulness or gain of a document based on its position in the results list; the recall rate is used to measure the proportion of relevant items retrieved in the first k results.

[0016] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0017] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0018] Compared to existing technologies, this invention and its preferred embodiment propose a method for constructing a RAG knowledge base based on LLM (Large Language Model) dynamic text segmentation technology, specifically designed for document processing and retrieval optimization in the power system field. Using LumberChunker technology, this invention dynamically segments documents such as feasibility study reports related to power grid systems into semantically independent text blocks, ensuring the integrity and independence of each text fragment. Subsequently, these text fragments are vectorized using an embedding model to construct an efficient RAG vector database. By adjusting the model's context window size, this invention optimizes retrieval precision and recall, ultimately obtaining the optimal power technology solution retrieval model. This method significantly improves the efficiency and accuracy of document retrieval in the power system field, especially when processing long documents, effectively avoiding information redundancy and interference from irrelevant content. Attached Figure Description

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0020] Figure 1 This is a flowchart of the method for generating a RAG knowledge base using LLM dynamic text partitioning in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the architecture of the partitioning method according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the experimental results of obtaining the optimal text block size in an embodiment of the present invention.

[0023] Figure 4 This is an internal structure diagram of each chapter in the embodiments of the present invention. Detailed Implementation

[0024] In the following, specific embodiments of this application will be described in detail with reference to the accompanying drawings. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features from various embodiments can be combined to obtain new implementations, or certain features from some embodiments can be substituted to obtain other preferred implementations.

[0025] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings:

[0026] This invention employs LLM dynamic text segmentation technology to first digitize feasibility study reports related to power grid systems, then divides and vectorizes them into text blocks, constructs a RAG vector database using an embedding model, and finally trains the model by continuously adjusting the set parameters to obtain the optimal performance model that meets the conditions.

[0027] The implementation of the RAG knowledge base construction based on LLM dynamic text segmentation specifically includes the following steps, such as: Figure 1 As shown;

[0028] Step 1: Collect textual data in the power system field, such as feasibility study reports and demand analysis reports. Format these documents, clean the data, and ensure they are suitable for model training. Extract the important parts of the documents and transform them into structured data as the foundation dataset for building the knowledge base.

[0029] First, we collect textual data from feasibility study reports in the power grid sector. This data typically comes from the internal databases of power grid companies and industry research reports. Then, we perform data cleaning and preprocessing, including removing invalid characters, correcting spelling errors, and standardizing the format to make it structured data suitable for analysis. Simultaneously, based on the content of the reports, we divide them into different chapters and paragraphs to facilitate subsequent processing.

[0030] By collecting textual data from feasibility study reports in the power grid sector, and performing data cleaning and preprocessing, the data is transformed into structured data suitable for analysis.

[0031] Step 2: Adjust LumberChunker for power system text processing. Deploy LLM locally and adjust document_loaders in LumberChunker according to the characteristics of power system domain documents.

[0032] To address the complex structure of feasibility study reports in the power grid sector, this invention specifically optimizes and customizes the `document_loaders` module in LLM (Limited Language Management). The optimized `document_loaders` intelligently identifies and parses the titles and body text in the study report, with a meticulous and efficient workflow: First, it reads and records the entire document's titles, the core of the document information. Next, it traverses the document segment by segment, performing detailed analysis on each paragraph. During this process, it continuously tracks the current chapter title, chapter content, and the chapter's path position within the document. Once a paragraph is identified as a title paragraph, it immediately saves all information for the current chapter and records the new chapter's title level, preparing for subsequent content processing. If a body text paragraph is identified, it appends the paragraph's content to the current chapter's content, ensuring the completeness and coherence of the chapter content. Finally, following the original document structure, `document_loaders` saves detailed information for each chapter, with a clear and concise internal structure, laying a solid foundation for subsequent processing and analysis. Finally, each chapter is saved according to the document structure, with the internal structure of each chapter as follows: Figure 4 As shown:

[0033] Therefore, in this step, based on the structural characteristics of feasibility study reports in the power grid field, `document_loaders` is modified to preprocess plain text content according to the structure of the feasibility study report. The specific processing method is as follows: First, the document title is recorded. Then, each paragraph in the document is traversed, tracking the current chapter title, content, and path. If the current paragraph is a title, the current chapter is saved, and the new title level is recorded; if the current paragraph is content, it is appended to the current chapter content. Finally, each chapter is saved according to the document structure, with the internal structure of each chapter being "Title 1 / nTitle 2 / n… / nBody Content".

[0034] Step 3, as follows Figure 2 As shown, LLM is used to parse the input data to obtain text content that meets the needs of the power grid field. Then, LumberChunker is used to dynamically segment the text using LLM, extract semantic information, and generate multiple semantically independent text fragment groups. G ,in Represents the LLM dynamic partitioning after the first partition. The core purpose of this step is to obtain a text block with complete semantic information.

[0035] In steps two and three, the dynamic text segmentation method from LumberChunker is incorporated to better adapt it to data in the power grid field. LumberChunker is a text segmentation method that uses a language model to dynamically divide documents into blocks with complete semantic information. First, based on the chapters and paragraphs obtained in step two, each paragraph is uniquely identified by an incrementing ID number and then sequentially concatenated into a group. G until its total number of tags exceeds a predetermined threshold. θ In this embodiment, this threshold represents the size of the text block. θ A value that is too small can lead to the loss of semantic content when segmenting large paragraphs; θ If the value is too large, it may provide too much context to the model, which could impair the accuracy of knowledge base reasoning and retrieval. Paragraph group G The input is provided to the language model LLM, which then points out the current paragraph. G i The content of a specific paragraph differs significantly from the preceding text, indicating a semantic difference. This detection marks the end of a block and defines... G i+1 The beginning of a paragraph. The document is processed iteratively, continuously being divided into blocks in a loop until all paragraphs have been processed.

[0036] Step 4: Use the Embedding model to vectorize the dynamically segmented text fragments, and then store the vectorized text fragments into the RAG vector database.

[0037] Step 5: Construct the Vector Database. The vectorized text fragments are stored in the RAG system's vector database. This database allows for the retrieval of the most relevant power system document fragments through vectorized matching. Utilizing semantically independent fragments generated by LumberChunker significantly improves retrieval accuracy and avoids interference from redundant or irrelevant information. Specifically, the vector database stores the vectorized text content and is used for subsequent feasibility study report retrieval and optimization / polishing functions.

[0038] Preferably, in steps four and five, the dynamically segmented text fragments are converted into high-dimensional vector representations using an embedding model. This is mainly achieved through word vector retrieval, word vector averaging, and cosine similarity calculation. The vector database is used to store the vectorized text content and is used for subsequent feasibility study report retrieval and optimization / polishing functions.

[0039] Step 6: Use the vector database obtained in Step 5 for retrieval and training. Repeat Steps 3 and 4, and modify the context size. θ The optimal power technology solution retrieval model is obtained.M θ .

[0040] By changing the model's set threshold parameters, we can observe whether the model improves and find the best-performing set to obtain the optimal model. The model's capability is primarily measured by two parameters: DCG@k and Recall@k. DCG@k (Converted Cumulative Gain) evaluates a document's usefulness or gain based on its position in the results list, considering that highly relevant documents appearing at lower positions in the list should be penalized. If relevant documents appear at the top of the ranking list, the DCG value will be higher.

[0041] Recall@k measures the proportion of relevant items retrieved from the top k results; it provides information about the system's ability to retrieve relevant items at the top k positions. The DCG at rank k is defined as:

[0042]

[0043] The Recall at rank k is defined as:

[0044] .

[0045] like Figure 3 The results shown are based on the above experiments on obtaining the optimal text block size.

[0046] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0051] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0052] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0053] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0054] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0055] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of RAG knowledge base construction methods based on LLM dynamic text segmentation. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for constructing a RAG knowledge base based on LLM dynamic text segmentation, characterized in that: By adjusting the LLM dynamic text partitioning scheme, the feasibility study report of the power grid system is processed into text to obtain structured data. Then, text blocks are partitioned and vectorized based on semantic segmentation. The RAG vector database is built and stored through the Embedding model. The optimal performance model that meets the conditions is obtained by adjusting the set parameters to train the model. Based on the structured data, and according to the structural characteristics of feasibility study reports in the power grid field, the document_loaders in LLM are modified to preprocess plain text content of the document according to the structure of the feasibility study report; The text block segmentation process specifically involves: using LumberChunker with LLM to dynamically segment the text and extract its semantic information, generating multiple semantically independent text segment groups. G ,in G i Represents the LLM dynamic partitioning after the first partition. i From a text fragment, a text block with complete semantic information is obtained; The semantic segmentation-based vectorization processing, which uses an embedding model to construct a RAG vector database for storage, involves: using the embedding model to vectorize dynamically segmented text fragments, and then storing the vectorized text fragments in the RAG vector database; storing the vectorized text fragments in the RAG system's vector database; the vector database allows retrieval of the most relevant power system document fragments through vectorized matching; specifically, the embedding model is used to vectorize the dynamically segmented text fragments into high-dimensional vector representations, achieved through one of the following methods: word vector retrieval, word vector averaging, or cosine similarity calculation; The process of training the model by adjusting set parameters to obtain the optimal performance model that meets the conditions involves retrieving and training using a vector database, repeatedly performing text block segmentation and semantic segmentation-based vectorization to modify the context size. θ Finally, the optimal power technology solution retrieval model is obtained. M θ .

2. The method for constructing a RAG knowledge base based on LLM dynamic text segmentation according to claim 1, characterized in that: The text processing of the power grid system feasibility study report includes data cleaning and preprocessing, including removing invalid characters, correcting spelling errors, and standardizing the format to obtain structured data.

3. The method for constructing a RAG knowledge base based on LLM dynamic text segmentation according to claim 1, characterized in that: Based on the structured data, and according to the structural characteristics of feasibility study reports in the power grid field, the document_loaders in LLM are modified to preprocess the plain text content of the document according to the structure of the feasibility study report. The specific processing method is as follows: First, record the title of the document; then, traverse each paragraph in the document and track the title, content and path of the current chapter; if the current paragraph is a title, save the current chapter and record the new title level; if the current paragraph is content, append it to the content of the current chapter; finally, save each chapter according to the document structure, and the internal structure of each chapter is "title 1 / ntitle 2 / n… / nbody content".

4. The method for constructing a RAG knowledge base based on LLM dynamic text segmentation according to claim 1, characterized in that: The specific steps for dynamically segmenting and extracting semantic information from text using LumberChunker and LLM are as follows: First, each paragraph is uniquely identified by an incrementing ID number, and each paragraph is sequentially concatenated into a group. G The total number of markers continues until it exceeds a predetermined threshold, representing the size of the text block; paragraph groups. G It is fed as input to the language model LLM, and the current paragraph is obtained through the LLM. G i The document is segmented into blocks by iterative processing of the content of a specific paragraph, and the process continues until all paragraphs have been processed.

5. The method for constructing a RAG knowledge base based on LLM dynamic text segmentation according to claim 1, characterized in that: The process of repeatedly dividing text into blocks and performing semantic segmentation-based vectorization is used to modify the context size. θ Finally, the optimal power technology solution retrieval model is obtained. M θ The specific process is as follows: by changing the set threshold parameters of the model, observe whether the model improves, find the best-performing set, and thus obtain the optimal model; the model's capability is measured by the discounted cumulative gain DCG@k and the recall rate Recall@k; the discounted cumulative gain evaluates the usefulness or gain of a document based on its position in the results list; the recall rate is used to measure the proportion of relevant items retrieved in the first k results.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the RAG knowledge base construction method based on LLM dynamic text segmentation as described in any one of claims 1-5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the RAG knowledge base construction method based on LLM dynamic text segmentation as described in any one of claims 1-5.

Images