Intelligent self-adaptive document segmentation method oriented to RAG system

Abstract

The invention relates to the technical field of natural language processing (NLP), in particular to an intelligent self-adaptive document segmentation method oriented to an RAG system. The method comprises the steps that S1, a deep learning model is adopted to dynamically adjust the size and the step length of a window according to document content density, structure information and context semantics; s2, in combination with a window representation result, calculating the context semantic similarity of the segmentation blocks by adopting a language model, and automatically adjusting the size of an overlapping region based on the context semantic similarity; and S3, based on a context segmentation block representation result, associating the segmentation block with the context by introducing a BERT model, automatically adjusting an overlapping part and a window size, and optimizing a segmentation effect. The invention aims to provide the intelligent self-adaptive document segmentation method oriented to the RAG system so as to improve the document segmentation efficiency and quality and optimize the recall rate and the generation quality of the RAG system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing (NLP), and particularly relates to an intelligent adaptive document segmentation method for a RAG system. BACKGROUND

[0002] With the wide application of large language models (LLM) and retrieval augmented generation (RAG) systems in the field of natural language processing, how to effectively segment documents has become one of the key factors to improve the system effect. The RAG system expands the knowledge boundary of the model and enhances the accuracy of the output by retrieving relevant fragments of an external knowledge base and integrating them into the generation process. However, the quality of document segmentation directly affects the accuracy of retrieval, and thus significantly restricts the quality and accuracy of the generation result.

[0003] Existing document segmentation methods mainly include 1. Rule-based segmentation methods include (1) character-based segmentation: segmentation according to a fixed number of characters or specific characters (such as punctuation marks, line breaks, etc.). This method is simple and efficient, but it can easily damage the semantic structure and is not suitable for complex text structures; (2) token-based segmentation: segmentation by a fixed number of tokens (i.e., words or morphemes). This method is commonly used, usually using a tokenizer consistent with the target language model, but it cannot intelligently adapt to the semantic structure of different texts and may result in fragmented content; (3) content-aware segmentation: using tools such as NLTK and spaCy to segment by sentence segmentation, paragraph recognition, and title processing. This method can better preserve the document structure, but it also lacks flexibility and adaptability.

[0004] 2. Sliding window segmentation method: it is to set a fixed size window and slide segmentation by step, usually setting an overlap area to maintain the context continuity between text blocks. This method can effectively preserve the context information and improve adaptability by adjusting the overlap ratio, but it has the following problems: (1) the overlapping area of adjacent text blocks leads to content duplication, increasing redundant information and affecting processing efficiency; (2) the segmentation effect is highly sensitive to window size (W), step size (S), and overlap ratio (O), and a too large window size easily leads to context redundancy, and a too small window size leads to context loss; (3) when processing long documents, the computational overhead of window-by-window segmentation and processing is relatively large.

[0005] 3. Semantic clustering-based segmentation method: this method segments text by analyzing the semantic structure of the document using clustering algorithms such as K-means and DBSCAN. This method can better preserve the semantic integrity of the text, but it also faces challenges such as high computational complexity and parameter selection issues.

[0006] In summary, these methods mainly rely on preset rules or fixed algorithms, although they can handle some standardized texts, but these methods often have the following shortcomings when dealing with complex documents or diversified texts: 1. Lack of intelligent decision-making mechanism: Most methods rely on manual configuration or fixed rules, and cannot automatically adjust the segmentation method according to the specific content of the document.

[0007] 2. Low-efficiency manual intervention: The need for manual specification of segmentation schemes increases the cost of manual labor and makes it difficult to handle complex or diversified document structures.

[0008] 3. Unable to handle diversified document structures: For documents with different formats, structures or semantic characteristics, existing methods cannot intelligently identify and select the best segmentation method.

[0009] The segmentation strategy of the document usually needs to be flexibly selected according to the content characteristics of different documents (such as document length, structure, syntax, etc.). However, most existing technologies rely on manual judgment to select the segmentation strategy, lacking sufficient intelligence and adaptive ability, which not only increases the manual workload, but also reduces the processing efficiency.

[0010] In view of the deficiencies in the prior art, the present application proposes an innovative scheme combining an adaptive sliding window segmentation method and a deep learning model. This method can automatically adjust the segmentation strategy according to the semantics and structure of the document content, thereby realizing efficient and accurate document segmentation. SUMMARY

[0011] In order to solve the above problems, the purpose of the present application is to provide an intelligent adaptive document segmentation method for RAG system, in order to improve the efficiency and quality of document segmentation, and to optimize the recall rate and generation quality of RAG system.

[0012] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions: Step S1: using a deep learning model to dynamically adjust the window size and step according to the document content density, structure information and context semantics; Step S2: combining the window representation result, using a language model to calculate the context semantic similarity of the segmentation block, and automatically adjusting the overlap area size based on the above context semantic similarity; Step S3: based on the context segmentation block representation result, by introducing the BERT model, the segmentation block is connected with the context, the overlap part and the window size are automatically adjusted, and the segmentation effect is optimized.

[0013] Further, in the traditional sliding window method, the window size and step are statically set. For different types of documents, this static setting strategy may cause unnecessary redundancy or information loss. The step S1 is specifically as follows: Step S11: Use spaCy and nltk natural language processing tools, combined with BERT deep learning model to perform preliminary structured analysis on document content, and identify the hierarchical structure of different paragraphs, sentences and important information; Step S12: Based on the structural information of the text, input the document features into the pre-trained BERT model, and dynamically adjust the window size and step according to the average length and information density of the sentence. For dense text (such as technical documents), the window is small; while for loosely structured and information dispersed documents (such as novels), the window is large.

[0014] Further, the window size calculation formula is as follows:

[0015] Where f is a nonlinear function, the input is the length and semantic density of the sentence, and the output is the size of the window; The step is the amplitude of the window sliding, and dynamically adjusting the step can better adapt to the content changes of the document. For example, if there are multiple paragraphs or semantic units in the document, the step can be set as the distance between paragraphs or sentences, while for documents without obvious paragraph separation, a smaller step can be used. The window step calculation formula is as follows:

[0016] Where g is a function generated according to the document structure analysis.

[0017] Further, in order to retain more semantic information when splitting the document, and avoid redundancy caused by too much overlap, the step S2 is as follows: Step S21: Calculate the semantic embedding of each split block through the BERT model, and compare it with the adjacent split block to calculate the similarity; Step S22: Determine the semantic similarity between adjacent split blocks. If the similarity between adjacent split blocks is higher than 0.8, reduce the overlap between adjacent split blocks to reduce redundancy; if the similarity between adjacent split blocks is lower than 0.5, increase the overlap between adjacent split blocks.

[0018] Further, the similarity calculation formula is as follows:

[0019] Where, And are the semantic embedding vectors of the split blocks And .

[0020] Further, in the step S2, the overlap area size adjustment formula is as follows:

[0021] wherein h is a function for adjusting, the input is the similarity, and the output is the size of the adjusted overlapping part.

[0022] Further, the step S3 is specifically as follows: Step S31: using a pre-trained BERT model to perform semantic analysis on the split blocks, and calculating the context relevance of each split block; Step S32: adjusting the boundary of each split block according to the semantic similarity and context information.

[0023] The present application has the following beneficial effects: 1. The present application uses a deep learning model such as BERT to analyze the semantic density of a document and dynamically adjusts the window size, step size, and overlap ratio of the sliding window. By adaptively adjusting the window size, step size, and overlap ratio, various types of documents can be automatically processed, significantly improving the quality and efficiency of text segmentation, ensuring the coherence of the context and the integrity of the semantics, and being particularly suitable for document segmentation requirements of RAG systems.

[0024] 2. The present application uses a deep learning model to calculate the semantic similarity of adjacent split blocks and dynamically adjusts the size of the overlapping region according to the similarity. By adjusting the overlapping part based on the semantic similarity, it avoids the redundancy caused by excessive overlap, thereby improving the accuracy of subsequent processing tasks (such as text retrieval, generation, etc.), reducing the impact of redundant content, and ensuring the independence of the split blocks and the fluency of the context.

[0025] 3. In the sliding window segmentation process, the present application can automatically select the optimal combination of window size, step size, and overlap ratio, adapt to different types of document content, and improve the recall rate and accuracy of the RAG system.

[0026] 4. The present application ensures that each split block maintains consistency in context and semantics through the understanding of document semantics by a deep learning model, greatly improving the effectiveness of document processing.

[0027] 5. The present application realizes an efficient and intelligent document segmentation system by combining a deep learning model and an adaptive sliding window segmentation algorithm, which is particularly suitable for large-scale document data processing and retrieval recall processes of RAG systems, significantly improving the performance of retrieval and generation. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a schematic diagram of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described in detail below in conjunction with the drawings and specific embodiments: Referring to Figure 1 As shown in the scheme, step S1: using a deep learning model to dynamically adjust the window size and step according to the document content density, structure information and context semantics; Step S2: combining the window representation result, using a language model to calculate the context semantic similarity of the segmentation block, and automatically adjusting the overlap area size based on the above context semantic similarity; Step S3: based on the context segmentation block representation result, by introducing the BERT model, the segmentation block is connected with the context, the overlap part and the window size are automatically adjusted, and the segmentation effect is optimized.

[0030] Further, the step S1 is specifically as follows: Step S11: using spaCy or nltk natural language processing tool, combining BERT deep learning model to preliminarily structure the document content, and identifying the hierarchical structure of different paragraphs, sentences and important information; Step S12: based on the structure information of the text, by inputting the document features to the pre-trained BERT model, the window size and step are dynamically adjusted according to the average length and information density of the sentence.

[0031] Further, the window size calculation formula is specifically as follows:

[0032] Wherein, f is a nonlinear function, the input is the length and semantic density of the sentence, and the output is the size of the window; The window step calculation formula is specifically as follows:

[0033] Wherein, g is a function generated according to the document structure analysis.

[0034] Further, the step S2 is specifically as follows: Step S21: calculating the semantic embedding of each segmentation block by BERT model, and comparing with the adjacent segmentation block to calculate the similarity; Step S22: judging the semantic similarity between adjacent segmentation blocks, if the similarity between adjacent segmentation blocks is higher than 0.8, reducing the overlap part between adjacent segmentation blocks, reducing redundancy; if the similarity between adjacent segmentation blocks is lower than 0.5, increasing the overlap part between adjacent segmentation blocks.

[0035] Further, the similarity calculation formula is as follows:

[0036] Wherein, And Respectively, the semantic embedding vectors of the split blocks And .

[0037] Further, in step S2, the size adjustment formula of the overlapping area is as follows:

[0038] Where h is the adjustment function, the input is the similarity, and the output is the size of the adjusted overlapping part.

[0039] Further, the traditional sliding window method only realizes the context connection through a fixed overlapping part, but this method fails to fully consider the semantic structure of the document. This embodiment can find more suitable connection points between split blocks by introducing BERT and other models, automatically adjust the overlapping part and window size, and thus further improve the splitting effect. Step S3 is as follows: Step S31: Use the pre-trained BERT model to perform semantic analysis on the split blocks, and calculate the context relevance of each split block; for example, if there is high semantic relevance between the contents of two split blocks, the overlap between them can be reduced.

[0040] Step S32: According to the semantic similarity and context information, adjust the boundary of each split block so that the context information between each block can be retained to the greatest extent without redundancy.

[0041] Code example for specific implementation: The following is a code implementation example based on Python and Transformers library, which demonstrates how to perform adaptive adjustment of the sliding window splitting method based on the BERT model.

[0042] import torch from transformers import BertTokenizer, BertModel import numpy as np from sklearn.metrics.pairwise import cosine_similarity # Load BERT model and tokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertModel.from_pretrained('bert-base-uncased') # Compute BERT embeddings for text def get_bert_embeddings(text): inputs = tokenizer(text, return_tensors='pt', padding=True,truncation=True) with torch.no_grad(): outputs = model(**inputs) return outputs.last_hidden_state.mean(dim=1).numpy() # Get the average-pooled embeddings # Calculate similarity between two texts def calculate_similarity(text1, text2): embedding1 = get_bert_embeddings(text1) embedding2 = get_bert_embeddings(text2) return cosine_similarity(embedding1, embedding2)[0][0] # Sliding window cutting function def sliding_window_cut(text, window_size, step_size, overlap_threshold=0.8): tokens = tokenizer.tokenize(text) chunks = [] for i in range(0, len(tokens) - window_size + 1, step_size): window = tokens[i:i + window_size] chunk = tokenizer.convert_tokens_to_string(window) if chunks: # Compare the similarity of adjacent chunks and adjust the overlapping part similarity = calculate_similarity(chunks[-1], chunk) if similarity>overlap_threshold: continue # If the similarity is high, reduce overlap chunks.append(chunk) return chunks # Example text text = "This is a simple example of text to demonstrate sliding window segmentation." # Call the sliding window cut method chunks = sliding_window_cut(text, window_size=5, step_size=3, overlap_threshold=0.7) for chunk in chunks: print(chunk) Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0043] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0044] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0046] The above descriptions are only preferred embodiments of the present application and are not intended to limit the present application to other forms. Any person skilled in the art can make modifications or alterations to the above-mentioned disclosed technical contents into equivalent embodiments. However, any simple modifications, equivalent changes and alterations made to the above embodiments without departing from the technical solutions of the present application, according to the technical essence of the present application, still belong to the protection scope of the present application.

Claims

1. A method for intelligent adaptive document segmentation for RAG system, characterized in that, The method comprises the following steps: Step S1: dynamically adjusting window size and step length according to document content density, structure information and context semantics by using a deep learning model; Step S2: combining window representation results, calculating the context semantic similarity of the segmentation block by using a language model, and automatically adjusting the size of the overlapping area based on the context semantic similarity; Step S3: based on the context segmentation block representation results, by introducing a BERT model, the segmentation block is linked with the context, the overlapping part and the window size are automatically adjusted, and the segmentation effect is optimized.

2. The intelligent adaptive document segmentation method for RAG system according to claim 1, wherein, The step S1 is specifically as follows: Step S11: using spaCy and nltk natural language processing tools, combining a BERT deep learning model to perform preliminary structural analysis on document content, and identifying the hierarchical structure of different paragraphs, sentences and important information; Step S12: based on the structure information of the text, by inputting document features into a pre-trained BERT model, dynamically adjusting the window size and step length according to the average length and information density of the sentence.

3. The intelligent adaptive document segmentation method for RAG system according to claim 2, wherein, The window size calculation formula is specifically as follows: ； Wherein, f is a nonlinear function, the input is the length of the sentence and the semantic density, and the output is the size of the window; The window step length calculation formula is specifically as follows: ； Wherein, g is a function generated according to document structure analysis.

4. The intelligent adaptive document segmentation method for RAG system according to claim 1, wherein, The step S2 is specifically as follows: Step S21: calculating the semantic embedding of each segmentation block by using a BERT model, and comparing with adjacent segmentation blocks to calculate the similarity; Step S22: judging the semantic similarity between adjacent segmentation blocks, if the similarity between adjacent segmentation blocks is higher than 0.8, reducing the overlapping part between adjacent segmentation blocks to reduce redundancy; If the similarity between adjacent segmentation blocks is lower than 0.5, the overlapping part between adjacent segmentation blocks is increased.

5. The intelligent adaptive document segmentation method for RAG system according to claim 4, wherein, The similarity calculation formula is as follows: ； wherein, and are semantic embedding vectors for the split blocks and respectively.

6. The intelligent adaptive document segmentation method for RAG system according to claim 1, wherein, In the step S2, the overlapping area size adjustment formula is as follows: ； Wherein, h is an adjustment function, the input is the similarity, and the output is the size of the adjusted overlapping part.

7. The intelligent adaptive document segmentation method for RAG system according to claim 1, wherein, The step S3 is specifically as follows: Step S31: using a pre-trained BERT model to perform semantic analysis on the segmentation block, and calculating the context correlation of each segmentation block; Step S32: adjusting the boundary of each segmentation block according to the semantic similarity and context information.

Images