Long text reading understanding method based on dynamic partitioning and selection

Abstract

The invention discloses a long text reading understanding method based on dynamic partitioning and selection, which is characterized in that the method adopts dynamic partitioning to dynamically divide long text input into discrete text blocks, selects and screens out irrelevant text segments by using selection partitioning, and splices the remaining text segments according to an original sequence, so as to obtain a long text reading understanding result. In order to conform to context window limitation predefined by a large language model, the method specifically comprises the steps of text preprocessing, dynamic blocking, block selection, large model output and the like. Compared with the prior art, the method has the advantages that the semantic coherence and the understanding accuracy are improved, the processing capability of the model on the super-long text is enhanced, the internal semantic integrity of each block is ensured, the semantic ambiguity caused by the block is reduced, the semantic coherence damage caused by the block with the fixed length is avoided, and the semantic coherence and the understanding accuracy are improved; and the processing capability of the model on the super-long text is enhanced, the data utilization efficiency is high, and the method has an important value and a good application prospect in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a long text reading comprehension method based on dynamic chunking and selection. Technical Background

[0002] Currently in the field of natural language processing, especially for the understanding of long texts and question-answering systems, common technical solutions include directly using pre-trained large language models for processing. These methods generally involve the following steps: Data input: Input the text to be analyzed into the language model. Feature extraction: Encode the input text through the internal mechanism of the language model to generate a feature representation. Question answering: Based on the generated feature representation, the model attempts to understand the question and find the corresponding answer from the text. However, when it comes to processing very long documents, the traditional large language model methods show certain limitations. When processing texts with a context length exceeding a certain limit, the traditional methods will show a significant performance decline. As the text length increases, the difficulty of maintaining semantic coherence and accuracy increases, resulting in a decrease in the reliability and consistency of the final result. There are several main defects in the existing other methods when dealing with long text understanding and question-answering tasks, and these problems together hinder the performance of the model in long text understanding and question-answering tasks. The current methods usually segment the input text into fixed-length chunks, which often leads to sentence truncation, thus destroying the logical structure and semantic coherence of the original text, affecting the overall understanding of the context, and may cause key information loss or meaning distortion. In addition, the pre-trained Transformer decoder architecture has attention mechanism limitations, especially the position encoding limits the input context window to a fixed length, and the quadratic attention calculation complexity limits the input length based on the available computing resources. At the same time, LLMs tend to disproportionately focus on the beginning and end parts of the input, making the key information in the middle may not be effectively integrated. Although sparse attention mechanisms can enhance the ability of large language models to understand long contexts, they rely on predefined methods to reduce the attention calculation cost, which inevitably limits the possibility of performance improvement and cannot fully capture long-distance dependencies. The token pruning strategy understands long inputs by identifying and retaining the "important" tokens that significantly contribute to the attention scores. However, this method fails to fully maintain the semantic structure of the original text, thus restricting potential performance improvements. In contrast, the method centered on chunk-level processing attempts to more effectively solve this limitation by using specialized memory units for efficient retrieval through distributed context storage, but still needs to overcome the aforementioned challenges to achieve better long text processing effects.

[0003] In summary, although existing NLP technologies have made progress in many aspects, they still have obvious deficiencies in processing complex and long text understanding. Large language models (LLMs) have the following main problems when processing long text reading comprehension tasks:

[0004] 1) Semantic coherence destruction caused by fixed-length chunking: Traditional methods divide long texts into fixed-length chunks, which may create breakpoints in the middle of sentences, destroying the semantic integrity of the sentences, and further affecting the model’s understanding of the overall semantics of the text, causing the model to be inaccurate when answering questions based on long texts.

[0005] 2) Unable to accurately filter key information: When faced with extremely long texts, it is difficult to effectively filter out the text blocks that are truly important for answering specific questions, which makes the model easily disturbed by a large amount of irrelevant information when processing long texts, reducing the accuracy and efficiency of the answers.

[0006] 3) Limitations in processing very long texts: As the length of the text increases, the performance of existing methods will significantly decrease when processing very long texts (such as texts with more than a certain number of tokens), and cannot meet the needs of long text processing in practical applications. Summary of the invention

[0007] The purpose of the present invention is to provide a long text reading comprehension method based on dynamic chunking and selection in response to the deficiencies of the prior art. The method adopts a question-aware classifier and introduces a dynamic chunking method to improve the long text reading comprehension performance of the model, greatly improving the ability of large language models in processing complex reasoning tasks, ensuring more accurate answer generation and more stable performance. This method significantly improves the performance of large language models in processing ultra-long text reading comprehension tasks through dynamic chunking and selection methods, improves semantic coherence and understanding accuracy, enhances the model's processing capabilities for ultra-long texts, and improves data utilization efficiency, greatly meeting the needs for long text processing in practical applications, improving user experience and satisfaction, and has good application prospects in various application scenarios.

[0008] The object of the present invention is achieved as follows: a long text reading comprehension method based on dynamic segmentation and selection, which is characterized in that a dynamic segmentation method is used to dynamically divide the long text input into discrete text blocks, a block selection method is used to filter out irrelevant text fragments, and the remaining text fragments are spliced in the original order to meet the context window restrictions predefined by the large language model. The specific steps for achieving long text reading comprehension are as follows:

[0009] 1. Text preprocessing

[0010] Step A1: Read input text

[0011] Read the long technical document uploaded by the user into memory and store it in string format.

[0012] Step A2: Text segmentation

[0013] Split the input text into a sequence of sentences represented as follows by punctuation marks (such as full stops, question marks, etc.):

[0014] [s0, s1, …, s n-1 .

[0015] where S0, S1, …, S n-1 are the segmented sentences; n is the total number of sentences.

[0016] (II) Dynamic chunking

[0017] Step B1: Sentence concatenation

[0018] To preserve the semantic integrity of the sentences, perform neighborhood merging on the sentences. Specifically, for each sentence s i , use the concatenation operation to generate an enhanced context segment s i ’ = [s′0, s′1, …, s′ n-1 :

[0019]

[0020] where represents the concatenation operation.

[0021] Step B2: Sentence embedding

[0022] Use the pre-trained Sentence-BERT model to encode the concatenated sentences to obtain the context embeddings represented as follows:

[0023] [e0, e1, …, e n-1 ∈ R d .

[0024] where e0, e1, …, e n-1 are the embeddings corresponding to all sentences.

[0025] Step B3: Calculate semantic similarity

[0026] B3-1: Calculate the cosine similarity between adjacent sentence embedding pairs by the following formula:

[0027]

[0028] where e i and e i+1 are the embeddings of adjacent sentences.

[0029] B3-2: Calculate the semantic dissimilarity using the following formula. The larger the value, the higher the semantic difference:

[0030] dis(i) = 1 - sim(i, i + 1).

[0031] Step B4: Determine the chunk boundaries

[0032] B4-1: Sort the semantic dissimilarity sequence [dis0, dis1, …, dis n-2 in ascending order to obtain the ordered index [k0, k1, …, k n-2 , and calculate and determine the percentile threshold α ∈ [0, 1] for boundary selection through the quantile using the following formula:

[0033]

[0034] where disi represents the semantic cosine distance sequence for subsequent boundary detection; ki represents the index sequence of the semantic cosine distance arranged in ascending order; α represents the quantile threshold with a value range of [0, 1] for determining the selection of context chunk boundaries; K represents the set of boundary indices calculated according to the quantile, where represents rounding up for selecting boundary indices with larger semantic differences.

[0035] B4-2: Keep the top (1 - α) proportion of the maximum dissimilarity indices as the segmentation boundaries, and divide the original document C into the final document segments represented by the following formula through binary splitting at position K:

[0036]

[0037] where each represents a segmented sub-document.

[0038] Step B5: Chunk refinement

[0039] B5-1: Ensure that the length of each chunk meets the predefined chunk length constraint l through iterative optimization. For each starting chunk c i , find the smallest integer u to obtain the result of chunk merging represented by the following formula:

[0040]

[0041] where c i is the original chunk for each round of iteration.

[0042] B5-2: After merging the original chunks c i for each round of iteration, update the index i to i + u + 1 and continue to process the next unmerged chunk to generate the final chunks and satisfy

[0043] (III) Block Selection

[0044] Step C1: Training Data Preparation

[0045] Extract the true context-question pairs [C, Q] from the question-answering corpus as positive samples, and generate negative samples through negative sampling strategy. For each context-question pair, concatenate the context and question tokens to form a unified sequence represented by the following formula:

[0046]

[0047] where p = |C| and q = |Q| represent the sequence lengths of the context and question respectively.

[0048] Step C2: Feature Extraction

[0049] C2-1: Encode the concatenated sequence through the Transformer layer of the LLM to obtain the layer representation H and the multi-head attention scores A.

[0050] C2-2: Extract the boundary tokens and attention pooling vectors through feature distillation to form the feature matrix (fused context-question representation) H' represented by the following formula:

[0051]

[0052] where and are the representations of the first and last tokens of the first sequence respectively; and are the representations of the first and last tokens of the second sequence; and are the context- and question-specific representations obtained by applying the attention mechanism to the first and second sequences respectively.

[0053] Step C3: Classifier Training

[0054] Train the classifier using a three-layer multi-layer perceptron architecture. Based on the fused context-question representation H', the optimization objective represented by the following formula is to minimize the binary cross-entropy loss to estimate the answer probability p(y|H):

[0055]

[0056] where N represents the total number of training instances; y i ∈ {0, 1} represents the true label of the i-th sample;

[0057] Step C4: Block Selection

[0058] C4-1: For each context block c i Calculate its relevance score T with the question by the following formula i :

[0059] T i = P(y = 1|X i ) = σ(h θ (X i )0).

[0060] Where h θ (X i )0 is the classifier output value, and σ is the softmax function.

[0061] C4-2: Dynamically determine the number of selected blocks according to the compression ratio and retain the top j blocks with the largest T value {c j}, to obtain the compressed context H constructed by the following formula comp :

[0062]

[0063]

[0064] Where m is the total number of blocks.

[0065] (IV) Large model output

[0066] Step D1: Input the compressed context

[0067] Input the compressed context H comp into the original large language model.

[0068] Step D2: Generate an answer

[0069] The LLM generates an answer to the corresponding question based on the input context H comp and returns the answer to the user.

[0070] The specific processing of the dynamic chunking method includes:

[0071] 1) Input text preprocessing

[0072] Receive the input text containing the initial information, context, and question, and decompose the context part into sentences according to punctuation marks to obtain a sentence sequence.

[0073] 2) Sentence embedding and similarity calculation

[0074] Encode the sentence sequence using a pre-trained Sentence-BERT model to obtain sentence embedding representations. Then calculate the semantic similarity between adjacent sentence embeddings, and calculate the similarity scores for adjacent sentence pairs using the cosine similarity formula.

[0075] 3) Determine the chunk boundaries

[0076] Based on the semantic similarity scores, use a percentile-based segmentation threshold to determine the chunk boundaries. Sort the semantic similarity scores from low to high, and select the sentence pairs corresponding to the smallest similarity scores in the top (1 - α) proportion as the chunk boundaries, where α is a preset chunk threshold parameter.

[0077] 4) Chunk optimization

[0078] Optimize the initial chunking results to ensure that the length of each chunk does not exceed the preset maximum chunk length. By iteratively merging adjacent chunks, make the length of each chunk as close as possible to the preset maximum chunk length while maintaining the semantic integrity of the chunks.

[0079] The specific processing of the chunk selection method includes:

[0080] 1) Training data collection and processing

[0081] Extract real question-context pairs from the Q&A dataset as positive samples, and generate negative samples through a negative sampling strategy. Concatenate the tokens of the context and the question into a unified sequence and use it as the input to the training data.

[0082] 2) Feature extraction and classifier training

[0083] Encode the concatenated sequence through the Transformer layer of the LLM to extract the boundary token representations and attention weights of the sequence. Use the attention weights to perform weighted summation on the representations of the context and the question to obtain context-specific and question-specific representations. Concatenate the boundary token representations with the weighted summation result as the input features of the classifier. Train a classifier with a three-layer MLP architecture to learn to estimate the answerability probability based on the context-question representation, and optimize it with the goal of minimizing the binary cross-entropy loss.

[0084] 3) Chunk screening and compression: For each chunk, combine it with the question to form a context-question pair, generate classifier input features through the above method, and use the trained classifier to predict the relevance probability between each chunk and the question. According to the preset compression ratio, select the top k chunks with the highest relevance probability, concatenate these chunks in the original order to form a compressed context input, and input it into the LLM for answer generation.

[0085] In addition to the dynamic chunking method based on semantic similarity, the dynamic chunking method can also adopt a chunking method based on the internal attention mechanism of the language model. For example, by analyzing the attention weight distribution of the language model when processing text, the regions with lower attention weights are used as chunk boundaries, thereby achieving the chunking of text. This method can also avoid the problem of semantic coherence disruption caused by fixed-length chunking, but it may be restricted by the characteristics of the language model itself and is not as versatile as the method based on semantic similarity.

[0086] In the chunk selection stage of the chunk selection method, in addition to using a classifier based on MLP for selection, a method based on reinforcement learning can also be adopted. The chunk selection process is regarded as a reinforcement learning task, and a suitable reward function is defined. For example, the reward is designed according to the relevance between the chunk and the question and the contribution of the chunk to the answer accuracy. An agent is trained through a reinforcement learning algorithm so that it can dynamically select the most relevant chunks according to the question and context. This alternative can more flexibly adapt to different questions and contexts, but more training data and computing resources are required to train the reinforcement learning agent.

[0087] The present invention devises a sparse attention mechanism, enabling the language model to focus only on the local regions relevant to the question when processing long texts and ignoring most of the irrelevant information. This method can avoid the processes of chunking and selection and directly perform efficient processing on the original long text. However, it may require significant modifications to the architecture of the language model, and in dealing with some problems that require global information integration, the effect may not be as good as the dynamic chunking and selection method proposed in the present invention.

[0088] The present invention has the following beneficial technical improvements and remarkable technical effects compared with the prior art:

[0089] 1) Through the dynamic chunking method, the present invention adaptively divides text chunks based on semantic similarity, avoiding the semantic coherence disruption caused by fixed-length chunking and greatly improving semantic coherence and understanding accuracy.

[0090] 2) When the traditional fixed-length chunking method processes long texts, it often generates breakpoints in the middle of sentences, destroying the semantic integrity of the sentences, resulting in inaccurate understanding of the overall semantics of the text by the model and thus affecting the accuracy of answering questions. Through dynamic chunking, the present invention ensures the semantic integrity within each chunk and reduces semantic ambiguity caused by chunking.

[0091] 3) The present invention introduces a question-aware classifier to select the chunks most relevant to the question, effectively screening out the information that is truly important for answering the question and greatly improving the efficiency of key information screening.

[0092] 4) When dealing with extremely long texts, existing methods have difficulty accurately screening out the text chunks that are truly important for answering questions, resulting in the model being interfered by a large amount of irrelevant information, reducing the accuracy and efficiency of the answers. Through the question-aware classifier, the model can accurately identify the chunks related to the question, reducing the interference of irrelevant information.

[0093] 5) Through the dynamic chunking and selection method, the present invention significantly improves the performance of large language models in processing extremely long text reading comprehension tasks, improves semantic coherence and understanding accuracy, enhances the model's processing ability for extremely long texts, and improves data utilization efficiency. These beneficial effects make the present invention of great value in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0095] The present invention proposes a dynamic chunking and selection (DCS) method for enhancing the ability of large language models to process extremely long texts in reading comprehension tasks. This method dynamically divides the long text input into discrete text chunks, then carefully screens out the irrelevant text fragments, and finally concatenates the remaining text fragments in the original order to conform to the predefined context window limit of the large language model. This method significantly enhances the large language model's ability to process context information.

[0096] The following will describe in detail the specific implementation manner of the present invention in conjunction with the accompanying drawings.

[0097] Refer to Figure 1 , the implementation steps of the present invention specifically include:

[0098] 1. Preprocessing of the input text

[0099] Step A1: Read the input text

[0100] The system receives the long technical document uploaded by the user, reads its content into the memory, and stores it in string format.

[0101] Step A2: Text segmentation

[0102] The input text is segmented into a sentence sequence [s0, s1,..., s n-1 according to punctuation marks (such as full stops, question marks, etc.), where n represents the total number of sentences.

[0103] 2. Dynamic Chunking

[0104] Step B1: Sentence concatenation

[0105] To preserve the semantic integrity of sentences, neighborhood merging is performed on the sentences. Specifically, for each sentence s i :

[0106]

[0107] where, represents the concatenation operation. This will generate enhanced context segments [s′0, s′1, …, s′ n-1 .

[0108] Step B2: Sentence embedding

[0109] Use a pre-trained Sentence-BERT model to encode the concatenated sentences to obtain context embeddings [e0, e1, …, e n-1 ∈ R d ..

[0110] Step B3: Calculate semantic similarity

[0111] Calculate the cosine similarity between adjacent sentence embedding pairs:

[0112]

[0113] And calculate the semantic dissimilarity:

[0114] dis(i) = 1 - sim(i, i + 1).

[0115] Step B4: Determine chunk boundaries

[0116] Sort the semantic dissimilarity sequence [dis0, dis1, …, dis n-2 in ascending order to obtain the ordered indices [k0, k1, …, k n-2 . Determine the percentile threshold α ∈ [0, 1] for boundary selection through quantile calculation:

[0117]

[0118] Retain the top (1 - α) proportion of the maximum dissimilarity indices as chunk boundaries, and divide the original document C into final document chunks through binary segmentation at position K:

[0119]

[0120] Step B5: Chunk refinement

[0121] Ensure that the length of each chunk meets the predefined chunk length constraint l through iterative optimization. For each starting chunk c i , find the smallest integer u to obtain the result of chunk merging represented by the following formula:

[0122]

[0123] After merging, update the index i to i + u + 1, and continue to process the next unmerged block to generate the final blocks Meet

[0124] 3. Chunk Selection

[0125] Step C1: Training data preparation

[0126] Extract the true context-question pairs [C, Q] from the question-answering corpus as positive samples, and generate negative samples through a negative sampling strategy. For each context-question pair, concatenate the context and question tokens to form a unified sequence:

[0127]

[0128] where p = |C| and q = |Q| represent the sequence lengths of the context and question respectively.

[0129] Step C2: Feature extraction

[0130] Encode the concatenated sequence through the Transformer layer of the LLM to obtain the final layer representation H' and the multi-head attention scores A. Extract the boundary tokens and the attention pooling vectors through feature distillation to form the feature matrix (the fused context-question representation) H' expressed as follows:

[0131]

[0132] Step C3: Classifier training

[0133] Train a classifier using a three-layer multi-layer perceptron architecture to estimate the answer likelihood p(y|H) based on the fused context-question representation H'. The optimization objective is to minimize the binary cross-entropy loss:

[0134]

[0135] where N represents the total number of training instances, and y i ∈ {0, 1} represents the true label of the i-th sample.

[0136] Step C4: Chunk selection

[0137] For each context chunk c i , calculate its relevance score T i . Dynamically determine the number of selected chunks according to the compression ratio , and retain the top with the largest T j value A block {c j}, and the finally compressed context is constructed as follows:

[0138] H comp = initial: {c j}} j∈top-k ; question]

[0139]

[0140] 4. Large Language Model Outputs

[0141] Step D1: Input the compressed context

[0142] Input the compressed context H comp into the base large language model.

[0143] Step D2: Generate an answer

[0144] The LLM generates an answer to the corresponding question based on the input context and returns the answer to the user.

[0145] The experimental results of the present invention show that after adopting the dynamic chunking method, in the single-hop question-answering task, the average accuracy of the model has increased by 28.62% (taking Llama-3 as an example), and in the multi-hop question-answering task, the average accuracy has increased by 20.02%. This shows that dynamic chunking significantly improves the model's ability to understand long texts, thereby answering questions more accurately. In the multi-hop question-answering task, for example, when processing long texts with a length of 250,000 tokens, the performance of the method of the present invention only decreases by 10%, while the performance of the existing method decreases by more than 30%. This shows that the present invention can more efficiently screen key information when processing ultra-long texts, improve the accuracy and efficiency of answering, and the performance is significantly better than the existing methods.

[0146] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A long text reading comprehension method based on dynamic chunking and selection, characterized in that This method uses dynamic chunking to dynamically divide the long text input into discrete text chunks, uses selective chunking to filter out irrelevant text fragments, and concatenates the remaining text fragments in the original order. The specific steps for long text reading comprehension are as follows: (1) Text preprocessing Step A1: Read the input text Read the long technical document uploaded by the user into memory and store it in string format; Step A2: Text segmentation Segment the input text into a sentence sequence represented as follows according to punctuation marks (such as full stops, question marks, etc.): [s0, s1, …, s n-1 ; Among them, S0, S1, …, S n-1 are the segmented sentences; n is the total number of sentences; (2) Dynamic chunking Step B1: Sentence concatenation For each sentence s i An augmentation operation is adopted to generate an augmented context segment s represented by the following formula i ’ = [s′0, s′1, …, s′ n-1 : Among them, represents a splicing operation; Step B2: Sentence embedding Use the pre-trained Sentence-BERT model to encode the concatenated sentences to obtain the context embeddings represented as follows: [e0, e1, …, e n-1 ∈ R d ; Among them, e0, e1, …, e n-1 are the embeddings corresponding to all sentences; Step B3: Calculate semantic similarity B3-1: Calculate the cosine similarity between adjacent sentence embedding pairs as follows: Among them, e i and e i+1 are the embeddings of adjacent sentences; B3-2: Calculate the semantic dissimilarity as follows: dis(i) = 1 - sim(i, i + 1); Step B4: Determine chunk boundaries B4-1: Sort the semantic dissimilarity sequence [dis0, dis1, …, dis n-2 in ascending order to obtain the ordered index [k0, k1, …, k n-2 , and determine the percentile threshold α ∈ [0, 1] for boundary selection by calculating the following formula with quantiles: where disi is the semantic cosine distance sequence; ki is the index sequence of the semantic cosine distances arranged in ascending order; α is the quantile threshold; K is the set of boundary indices calculated according to the quantile, where represents rounding up; B4-2: Divide the original document C into the final document segments represented as follows by binary segmentation at position K: Among them, each represents a segmented sub-document; Step B5: Chunk refinement B5-1: Ensure that the length of each block meets the predefined block length constraint l through iterative optimization. For each starting block c i , find the smallest integer u to obtain the result of block merging represented by the following formula: Among them, c i is the original block for each round of iteration; B5-2: Combine the original chunks c of each iteration i Merge them, update the index i to i + u + 1, and continue to process the next unmerged chunk to generate the final chunk and satisfy (3) Chunk selection Step C1: Prepare training data Extract the real context-question pairs [C, Q] from the question-answering corpus as positive samples, and generate negative samples through a negative sampling strategy. For each context-question pair, concatenate the context and question tokens to form a unified sequence represented as follows: where p = |C| and q = |Q| are the sequence lengths of the context and question respectively; Step C2: Feature extraction C2-1: Encode the concatenated sequence through the Transformer layer of the LLM to obtain the layer representation H and the multi-head attention scores A; C2-2: Extract the boundary tokens and attention pooling vectors through feature distillation to form the feature matrix represented as follows, that is, the fused context-question representation H'; wherein, and are the representations of the first and last tokens of the first sequence, respectively; and are the representations of the first and last tokens of the second sequence; and are the context and question-specific representations obtained by applying the attention mechanism to the first and second sequences, respectively; Step C3: Classifier training Train the classifier using a three-layer multi-layer perceptron architecture. Based on the fused context-question representation H', the optimization objective represented as follows is to minimize the binary cross-entropy loss to estimate the answer probability p(y|H): where N is the total number of training instances; y i ∈ {0, 1} is the true label of the i-th sample; Step C4: Chunk selection C4-1: For each context block c i Calculate its relevance score T with the question by the following formula i :[[]]END]] T i = P(y = 1|X i ) = σ(h θ (X i )); where h θ (X i )0 is the classifier output value; σ is the softmax function; C4-2: Dynamically determine the number of selected blocks according to the compression ratio, and retain the top blocks with the largest T j value to obtain the compressed context H constructed by the following formula {c j}, where comp : H comp =[initial; {c j} j∈top-k ; question] where m is the total number of chunks; (4) Output of the large model Step D1: Input compressed context Input the compressed context H comp into the original large language model; Step D2: Generate answers The LLM generates the answer to the corresponding question based on the input context H comp and returns the answer to the user.

2. The long text reading comprehension method based on dynamic chunking and selection according to claim 1, wherein The dynamic chunking adopts a dynamic chunking method based on semantic similarity or a chunking method based on the internal attention mechanism of the large language model. The chunking method based on the internal attention mechanism of the large language model takes the region with lower attention weights as the chunk boundary, thereby realizing the chunking of the text.

3. The long text reading comprehension method based on dynamic chunking and selection according to claim 1, wherein, The selective chunking adopts a classifier based on MLP for selective chunking, or a method based on reinforcement learning regards the chunk selection process as a reinforcement learning task, and trains an agent through a reinforcement learning algorithm so that it can dynamically select the most relevant chunks according to the question and context.

4. The long text reading comprehension method based on dynamic chunking and selection according to claim 2, wherein The dynamic chunking method based on semantic similarity specifically includes: 1) Input text preprocessing Receive the input text containing initial information, context, and questions, decompose the context part into sentences according to punctuation marks to obtain a sentence sequence; 2) Sentence Embedding and Similarity Calculation Use the pre-trained Sentence-BERT model to encode the sentence sequence. After obtaining the sentence embedding representation, calculate the semantic similarity between adjacent sentence embeddings, and calculate the similarity score of adjacent sentence pairs through the cosine similarity formula; 3) Determine the Chunk Boundaries According to the semantic similarity scores, use a percentile-based segmentation threshold to determine the chunk boundaries. Sort the semantic similarity scores from low to high, and select the sentence pairs corresponding to the smallest similarity scores in the first (1-α) proportion as the chunk boundaries, where α is a preset chunk threshold parameter; 4) Chunk Optimization Optimize the preliminary chunking results to ensure that the length of each chunk does not exceed the preset maximum chunk length. By iteratively merging adjacent chunks, make the length of each chunk as close as possible to the preset maximum chunk length while maintaining the semantic integrity of the chunks.

5. The long text reading comprehension method based on dynamic chunking and selection according to claim 3, characterized in that The specific processing of selecting chunks by the MLP-based classifier includes: 1) Training Data Collection and Processing Extract real question-context pairs from the Q&A dataset as positive samples, generate negative samples through a negative sampling strategy, concatenate the tokens of the context and the question into a unified sequence, and use it as the input of the training data; 2) Feature Extraction and Classifier Training Encode the concatenated sequence through the Transformer layer of the LLM to extract the boundary token representation and attention weights of the sequence. Use the attention weights to perform weighted summation on the representations of the context and the question to obtain context-specific and question-specific representations; concatenate the boundary token representation with the weighted summation result as the input features of the classifier; train a classifier with a three-layer MLP architecture to learn to estimate the answerability probability based on the context-question representation, and optimize it with the goal of minimizing the binary cross-entropy loss; 3) Chunk Screening and Compression For each chunk, combine it with the question to form a context-question pair, generate classifier input features through the above method, use the trained classifier to predict the correlation probability between each chunk and the question, and according to the preset compression ratio, select the top k chunks with the highest correlation probability, concatenate these chunks in the original order to form the compressed context and then input it into the LLM for answer generation.

Images