Text partitioning result dynamic optimization method and system, medium and product
By dynamically adjusting text blocking boundaries in the search system, and optimizing text blocking using language models and user input parameters, the problem of segmented boundaries in the existing technology cannot be adjusted, and the accuracy and semantic integrity of the search results are improved.
Patent Information
- Application Number
- CN202510539348.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-27
- Publication Date
- 2025-08-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the text chunking method cannot dynamically adjust the segment boundary according to the user's query content, resulting in the search results containing too much irrelevant information, affecting the quality of the generated content.
By searching text in the vector database and splicing it into a single input sequence with the user query, the language model is used to generate the probability distribution of the starting and ending positions, the probability threshold is determined based on the user input parameters, the candidate positions are filtered and sorted, and the segmented boundaries of the search text are dynamically adjusted.
It realizes that the search content is highly correlated with user needs, improves the accuracy and semantic integrity of the search results, and reduces information redundancy.
Smart Images
Figure CN120448518A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of retrieval generation enhancement technology, and in particular to a method, system, medium and product for dynamic optimization of text segmentation results. Background Art
[0002] Retrieval-augmented generation (RAG) systems enhance the model's response quality by retrieving relevant documents before generating answers. However, their performance depends heavily on the effectiveness of the text chunking strategy. Existing text chunking methods only improve the pre-retrieval processing stage. Pre-retrieval chunking methods cannot dynamically adjust segment boundaries based on user queries, and even lack dynamic segment boundary adjustment for individual texts. This can lead to retrieval results containing excessive irrelevant information, thus affecting the quality of the resulting content. Summary of the Invention
[0003] The purpose of this application is to provide a method, system, medium and product for dynamically optimizing text segmentation results to improve the above problems in the prior art.
[0004] To achieve the above objectives, this application provides the following technical solutions:
[0005] The first aspect of the present application provides a method for dynamically optimizing text segmentation results, comprising:
[0006] Searching in a vector database according to a user query to obtain a search text, wherein the search text belongs to the same document;
[0007] The context of the search text and the user query are concatenated into a single input sequence, and the single input sequence and the character position of each word in the search text are input into a language model. The language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text;
[0008] Determining a probability threshold through user input parameters, filtering the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sorting the filtering results in descending order of probability value to obtain a starting position set and an ending position set, wherein the starting position set includes multiple candidate positions and an index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and an index value corresponding to each candidate position;
[0009] The candidate position with the smallest index value in the starting position set is used as the starting position of the search text, the candidate position with the largest index value in the ending position set is used as the ending position of the search text, and the search text is output.
[0010] Furthermore, the index value of the starting position and the index value of the ending position are mapped back to the character positions of the search text respectively through offset mapping, and extended to the nearest sentence boundary to be output as the final search text.
[0011] Preferably, the user input parameter is positively correlated with the probability threshold; the method for determining the probability threshold τ by the user input parameter λ is:
[0012] τ=λ / N
[0013] Wherein, τ is the probability threshold, λ is the user input parameter, N is the number of context tokens of the retrieval text, λ∈(0,N).
[0014] Preferably, the method for determining the number k of candidate positions in the starting position set and the ending position set comprises:
[0015]
[0016] Among them, top k is the maximum value of the number of candidate positions k, λ is a user input parameter, N is the number of context tokens of the retrieval text, λ∈(0,N).
[0017] Preferably, k = min(count(P start >τ),top k ), where P start is the starting position probability, τ is the probability threshold, top k is the maximum value of the number of candidate positions k, where k is the number of candidate positions in the starting position set or the ending position set after screening.
[0018] The second aspect of the present application provides a text segmentation result dynamic optimization system, comprising:
[0019] A probability distribution generation module is configured to search a vector database based on a user query to obtain a search text, wherein the search text belongs to the same document; concatenate the context of the search text and the user query into a single input sequence; input the single input sequence and the character position of each word in the search text into a language model; and the language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text;
[0020] The candidate position determination module is used to determine the probability threshold through user input parameters, filter the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sort the filtering results in descending order of probability value to obtain the starting position set and the ending position set, wherein the starting position set includes multiple candidate positions and the index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and the index value corresponding to each candidate position; the candidate position with the smallest index value in the starting position set is used as the starting position of the search text, and the candidate position with the largest index value in the ending position set is used as the ending position of the search text, and the search text is output.
[0021] The third aspect of the present application provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of a method for dynamically optimizing text segmentation results as described in the first aspect of the present application are implemented.
[0022] The fourth aspect of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the method for dynamic optimization of text segmentation results as described in the first aspect of the present application.
[0023] The present application provides the above-mentioned method for dynamic optimization of text segmentation results, which can at least achieve the following technical effects:
[0024] This application dynamically adjusts the segment boundaries of the retrieved documents based on user input to better align with the query intent, thereby achieving retrieval block refinement, ensuring that the retrieved content is highly relevant to user needs, improving the accuracy of the retrieval results while ensuring the semantic integrity of the text. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0026] Figure 1 A flowchart of a method for dynamically optimizing text segmentation results provided in an embodiment of the present application;
[0027] Figure 2 A schematic diagram of the structure of a text segmentation result dynamic optimization system provided in an embodiment of the present application;
[0028] Figure 3 A schematic diagram of a computer device provided in an embodiment of the present application;
[0029] Figure numerals: 200, a text segmentation result dynamic optimization system; 201, probability distribution generation module; 202, candidate position determination module; 301, memory; 302, processor. DETAILED DESCRIPTION
[0030] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0031] The embodiment of the present application provides a method for dynamically optimizing text segmentation results, based on the RAG retrieval system, by dynamically adjusting the segment boundaries of the retrieved document according to user input to better align with the query intent. Figure 1 As shown, the method for dynamically optimizing text segmentation results provided in this embodiment specifically includes the following steps:
[0032] Step S100: searching in a vector database according to a user query to obtain a search text, wherein the search text belongs to the same document;
[0033] Specifically, the method described in this embodiment includes three steps: initial segmentation, information retrieval, and post-retrieval refinement. First, a conventional segmentation method (such as a fixed-length segmentation method or a semantic segmentation method) is used to preliminarily process the retrieval document library and establish an initial index, thereby adapting the index requirements through structured cutting. Secondly, based on the user query, the initial segmentation related to the user query is obtained from the vector database, and the relevant text blocks are recalled from the entire retrieval document library (multiple documents) through vector similarity calculation or keyword matching, and the returned result is a candidate result containing multiple document fragments. Finally, the retrieval text segmentation of a single document is dynamically adjusted through the post-retrieval refinement method described below in this embodiment, thereby reducing information redundancy.
[0034] Step S200: concatenate the context of the search text and the user query into a single input sequence, input the single input sequence and the character position of each word in the search text into a language model, and the language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text;
[0035] Specifically, by splicing the user query and the context of the retrieved text into a single input sequence, the language model is used to capture the semantic association between the two, and the character position of each token in the retrieved text in the original text is recorded for precise positioning after subsequent retrieval and segmentation.
[0036] Preferably, the language model is a fine-tuned BERT / Transformer language model, that is, a language model obtained by further training and optimization on a specific task dataset based on pre-training. The above language model encodes the input sequence through its encoder, generates the start position probability and end position probability of each word in the retrieval text, and outputs the start position probability distribution set and the end position probability distribution set respectively, as shown in the following formula (1):
[0037] p start ,p end =Model(t1,t2,…,t N ) (1)
[0038] In formula (1), (t1, t2,…, t N ) is a single input sequence consisting of N tokens, p start Represents the probability distribution of each token as the starting position, p end Represents the probability distribution of each token as the terminal position.
[0039] Specifically, the model first performs a series of operations on the input sequence (t1, t2, ..., t N ) is embedded, including word embedding and position embedding, to capture the semantic and position information of each word in the sequence. Then, through multi-layer bidirectional encoding and attention mechanism, the model can effectively capture the deep semantic relationships in the text. Ultimately, the model outputs vector representations of each word. These vectors capture the relationship between words and other words and the semantic meaning of words in the sentence, and then generate the probability distribution of the start and end positions of each word (each word will output the corresponding start position probability and end position probability, even if the probability is 0), and the classification combination is divided into a start position probability distribution set and an end position probability distribution set.
[0040] Step S300: Determine a probability threshold using user input parameters, filter the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sort the filtering results in descending order of probability value to obtain a starting position set and an ending position set, wherein the starting position set includes multiple candidate positions and an index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and an index value corresponding to each candidate position;
[0041] Preferably, in step S300, the user input parameter λ is positively correlated with the probability threshold τ. The relationship is as described in the following formula (2):
[0042] τ = λ / N (2)
[0043] In formula (2), τ is the probability threshold, λ is the user input parameter, N is the number of context tokens of the search text, λ∈(0,N). The number of candidate positions of the starting position and the ending position with high probability / high confidence greater than τ is dynamically controlled by the user input parameter λ, and the total number of starting / ending positions is not greater than the top k , ensure that the number of candidate positions is adaptive to the text size, and obtain the starting position set and the ending position set after sorting. According to formula (3), the maximum number of candidate positions is dynamically adjusted. k :
[0044]
[0045] In formula (3), ceil refers to the integer upward, top k is the maximum value of the number of candidate positions k, λ is a user input parameter, N is the number of context tokens of the retrieval text, λ∈(0,N).
[0046] Preferably, k = min(count(P start >τ),top k ), where Pstart is the starting position probability, τ is the probability threshold, top k is the maximum value of the number of candidate positions k, where k is the number of candidate positions in the starting position set or the ending position set after screening.
[0047] Furthermore, λ, τ and top k The following relationship is satisfied:
[0048] λ increases → τ increases → top k Reduce → finer block division (accuracy first);
[0049] λ decreases → τ decreases → top k Increase → wider chunking (recall priority);
[0050] Specifically, by dynamically adjusting the number of candidate positions, different text segmentation requirements (precision / recall) can be met.
[0051] Step S400: taking the candidate position with the smallest index value in the starting position set as the starting position of the search text, taking the candidate position with the largest index value in the ending position set as the ending position of the search text, and outputting the search text.
[0052] Specifically, in step S400, the index value of the starting position and the index value of the ending position are mapped back to the character position of the search text through offset mapping, and extended to the nearest sentence boundary (such as a period or a line break) to be output as the final search text, thereby ensuring semantic coherence.
[0053] Furthermore, the following table 1 is the pseudo code for the above steps:
[0054] Table 1
[0055]
[0056]
[0057] Based on the same inventive concept, a text segmentation result dynamic optimization system 200 is also provided in the embodiment of the present invention, as described in the following embodiment. Since the principle of solving the problem of a text segmentation result dynamic optimization system is similar to that of a text segmentation result dynamic optimization method, the implementation of a text segmentation result dynamic optimization system can refer to the implementation of a text segmentation result dynamic optimization method, and the repetitions are not repeated here. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0058] Figure 2 This is a structural block diagram of a text segmentation result dynamic optimization system 200 according to an embodiment of the present application. Figure 2 Shown, including:
[0059] The probability distribution generation module 201 is configured to search a vector database based on a user query to obtain a search text, wherein the search text belongs to the same document; concatenate the context of the search text and the user query into a single input sequence; input the single input sequence and the character position of each word in the search text into a language model; and the language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text;
[0060] The candidate position determination module 202 is used to determine the probability threshold through user input parameters, filter the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sort the filtering results in order from large to small according to the probability value to obtain the starting position set and the ending position set, wherein the starting position set includes multiple candidate positions and the index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and the index value corresponding to each candidate position; the candidate position with the smallest index value in the starting position set is used as the starting position of the search text, and the candidate position with the largest index value in the ending position set is used as the ending position of the search text, and the search text is output.
[0061] In this embodiment, a computer device is also provided, such as Figure 3 As shown, it includes a memory 301, a processor 302 and a computer program stored in the memory 301 and capable of running on the processor 302. When the processor 302 executes the computer program, any one of the above-mentioned methods for dynamically optimizing text segmentation results is implemented.
[0062] Specifically, the computer device may be a computer terminal, a server or a similar computing device.
[0063] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any one of the above-mentioned methods for dynamically optimizing text segmentation results.
[0064] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CDROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.
[0065] The embodiments of the present invention achieve the following technical effects:
[0066] 1. This application dynamically adjusts the segment boundaries of retrieved documents based on user input to better align with query intent, thereby achieving refined retrieval blocks and ensuring that the retrieved content is highly relevant to user needs, improving the accuracy of retrieval results while ensuring the semantic integrity of the text;
[0067] 2. Through dynamic threshold control and context-aware boundary expansion, query intent alignment is achieved while maintaining semantic integrity. Benchmark testing shows that, compared to traditional methods, precision is improved by up to 30% and IoU by 13%, without compromising recall. This also reduces computational overhead by 80% compared to semantic segmentation.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for dynamic optimization of text segmentation results, characterized in that: include: Searching in a vector database according to a user query to obtain a search text, wherein the search text belongs to the same document; The context of the search text and the user query are concatenated into a single input sequence, and the single input sequence and the character position of each word in the search text are input into a language model. The language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text; Determining a probability threshold through user input parameters, filtering the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sorting the filtering results in descending order of probability value to obtain a starting position set and an ending position set, wherein the starting position set includes multiple candidate positions and an index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and an index value corresponding to each candidate position; The candidate position with the smallest index value in the starting position set is used as the starting position of the search text, the candidate position with the largest index value in the ending position set is used as the ending position of the search text, and the search text is output.
2. A text segmentation result dynamic optimization method according to claim 1, characterized in that: The user input parameter is positively correlated with the probability threshold.
3. A text segmentation result dynamic optimization method according to claim 2, characterized in that: The method for determining the probability threshold by user input parameters is: τ=λ / N Wherein, τ is the probability threshold, λ is the user input parameter, N is the number of context tokens of the retrieval text, λ∈(0,N).
4. A text segmentation result dynamic optimization method according to claim 1, characterized in that: Determining the number of candidate positions in the starting position set and the ending position set, the method comprising: Among them, top k is the maximum value of the number of candidate positions k, λ is a user input parameter, N is the number of context tokens of the retrieval text, λ∈(0,N).
5. A text segmentation result dynamic optimization method according to claim 4, characterized in that: k=min(count(P start >τ),top k ), where P start is the starting position probability, τ is the probability threshold, top k is the maximum value of the number of candidate positions k, where k is the number of candidate positions in the starting position set or the ending position set after screening.
6. A text segmentation result dynamic optimization method according to claim 1, characterized in that: The output search text includes: The index value of the starting position and the index value of the ending position are respectively mapped back to the character position of the search text through offset mapping, and extended to the nearest sentence boundary to be output as the final search text.
7. A text segmentation result dynamic optimization system, characterized in that: include: A probability distribution generation module is used to search the vector database according to the user query to obtain the search text, wherein the search text belongs to the same document; The context of the search text and the user query are concatenated into a single input sequence, and the single input sequence and the character position of each word in the search text are input into a language model. The language model outputs a start position probability distribution set and an end position probability distribution set through encoding processing, wherein the start position probability distribution set includes the start position probability corresponding to each word in the search text, and the end position probability distribution set includes the start position probability corresponding to each word in the search text; The candidate position determination module is used to determine the probability threshold through user input parameters, filter the values in the starting position probability set and the ending position probability distribution set according to the probability threshold, and sort the filtering results in descending order of probability value to obtain the starting position set and the ending position set, wherein the starting position set includes multiple candidate positions and the index value corresponding to each candidate position, and the ending position set includes multiple candidate positions and the index value corresponding to each candidate position; the candidate position with the smallest index value in the starting position set is used as the starting position of the search text, and the candidate position with the largest index value in the ending position set is used as the ending position of the search text, and the search text is output.
8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text segmentation result dynamic optimization method according to any one of claims 1 to 6 are implemented.
9. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method for dynamic optimization of text segmentation results according to any one of claims 1 to 6 are implemented.