Text processing method, device and program for large language model

Abstract

The invention relates to the technical field of computers, and discloses a text processing method and device for a large language model and a program. The method comprises the steps of obtaining an ultra-long knowledge text in response to a question text input to a target large language model by a user; the super-long knowledge text contains the content of the question answering text; generating a plurality of semantic boundaries corresponding to the super-long knowledge text and a score of each semantic boundary based on a natural language processing algorithm; segmenting the super-long knowledge text into a plurality of text blocks based on the available text budget of the target large language model, the plurality of semantic boundaries, the score of each semantic boundary and a preset coincidence proportion; determining a temporary reply corresponding to each text block based on the question text and the target large language model; generating a summary prompt word according to the question text and the temporary reply corresponding to each text block; and inputting the summary prompt word into the target large language model, and generating an answer text for the question text. According to the method, the compatibility of the large language model to the super-long knowledge text can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a text processing method, device, and program for large language models. Background Technology

[0002] With the widespread application of artificial intelligence in content production, knowledge-based question answering, and human-computer interaction, large language models are being used in scenarios involving long and complex texts, such as legal compliance review, medical literature reading, financial research report analysis, software document retrieval, and enterprise knowledge-based question answering. However, existing large language models are generally limited by the context length window. Once the input sequence exceeds the limit, the model cannot receive and utilize all the information at once. At the same time, due to differences in word segmentation methods and units of measurement among different models, the actual usable effective text space is often further limited by system prompts, dialogue history, and formatting overhead, resulting in a compression of the amount of effective information that can be processed.

[0003] In industrial practice, common approaches include manual simplification, manual summarization, fragmented reading, external retrieval assistance, or multi-round interactive questioning. However, these methods often introduce new problems, such as reliance on human experience for selection criteria, difficulty in balancing content retention and rejection, fragmentation of semantic cues, weakening of causal or referential relationships across paragraphs, and comprehension drift and response delays caused by multi-round interactions. These limitations are particularly pronounced for loosely structured, information-dense, and tightly interconnected long texts, such as contract terms, technical white papers, or cross-disciplinary reviews: on the one hand, key information may be scattered across distant locations, making it difficult for the model to form a complete judgment if it cannot be uniformly incorporated into the model's scope; on the other hand, simply truncating or roughly summarizing can easily alter the original meaning or omit the chain of evidence, thus affecting the reliability and traceability of the output. Furthermore, limited by computing power and cost constraints, attempts to expand the model size or blindly add external preprocessing also face the practical bottlenecks of increased costs, increased latency, and decreased system stability.

[0004] Therefore, improving the usability and response quality of large language models for ultra-long texts while maintaining semantic integrity and contextual consistency has become an urgent technical requirement for knowledge-intensive tasks. Summary of the Invention

[0005] The purpose of this application is to provide a text processing method, device, program, and storage medium for large language models, so as to solve the problem of poor compatibility of large language models with ultra-long knowledge texts in the prior art.

[0006] To achieve the above objectives, the first aspect of this application provides a text processing method for large language models, the method comprising: In response to the user's input of a question text into the target large language model, a very long knowledge text is obtained; the very long knowledge text contains the content of the answer to the question text. Generate multiple semantic boundaries and a score for each semantic boundary based on natural language processing algorithms for ultra-long knowledge texts; Based on the available text budget of the target large language model, multiple semantic boundaries, the score of each semantic boundary, and the preset overlap ratio, the ultra-long knowledge text is divided into multiple text blocks; Based on the question text and the target large language model, determine the temporary response corresponding to each text block; Generate summary prompts based on the question text and the temporary responses corresponding to each text block; The summary prompts are input into the target large language model to generate the answer text for the question text.

[0007] In this embodiment of the application, the step of dividing an ultra-long knowledge text into multiple text blocks based on the available text budget of the target large language model, multiple semantic boundaries, the score of each semantic boundary, and a preset overlap ratio includes: using the available text budget as the maximum allowed length of each text block; determining the actual end position of each of the multiple text blocks one by one based on multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the ultra-long knowledge text; the starting position of the first text block among the multiple text blocks is the same as the starting position of the ultra-long knowledge text; the starting position of all text blocks except the first text block is obtained based on the actual end position of the previous text block and the preset overlap ratio; and dividing the ultra-long knowledge text into multiple text blocks based on the actual end position and the starting position of each text block.

[0008] In this embodiment of the application, the step of determining the actual end position of each of the multiple text blocks based on multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the super-long knowledge text includes: determining the temporary end position of the current text block based on the actual end position of the previous text block or the start position of the super-long knowledge text, the maximum allowed length, and the super-long knowledge text; determining multiple current semantic boundaries and the score of each current semantic boundary in the end portion of the current text block based on multiple semantic boundaries and the score of each semantic boundary; and taking the current semantic boundary with the highest score among the multiple current semantic boundaries as the actual end position of the current text block.

[0009] In this embodiment of the application, the step of determining the temporary response corresponding to each text block based on the question text and the target large language model includes: determining the segmented prompt words corresponding to each text block based on the question text; inputting the segmented prompt words corresponding to each text block into the target large language model to obtain the temporary response corresponding to each text block.

[0010] In this embodiment of the application, the step of generating summary prompts based on the question text and the temporary replies corresponding to each text block includes: generating merged text based on the temporary replies corresponding to each text block; and generating summary prompts based on the question text and the merged text.

[0011] In this embodiment, the step of generating merged text based on the temporary replies corresponding to each text block includes: constructing a list of temporary replies corresponding to multiple text blocks based on the temporary replies corresponding to each text block; taking each temporary reply in the temporary reply list as a baseline reply, and forming multiple first combinations corresponding to the baseline reply with each of the remaining temporary replies; determining the cosine similarity of each first combination corresponding to the baseline reply based on the inverse document frequency method; merging the first combinations with a cosine similarity greater than or equal to a preset similarity threshold into a second combination; taking the temporary reply with the longest number of characters in the second combination as the final reply corresponding to the baseline reply; and generating merged text based on the final replies corresponding to each baseline reply.

[0012] In this embodiment of the application, the step of determining the cosine similarity of each first combination corresponding to the benchmark response based on the inverse document frequency method includes: segmenting each temporary response in each first combination to obtain Chinese word segments corresponding to each temporary response; determining the feature vector corresponding to each temporary response based on the inverse document frequency method and the Chinese word segments corresponding to each temporary response; and determining the cosine similarity of each first combination based on the feature vector corresponding to each temporary response.

[0013] A second aspect of this application provides a computer device, comprising: The memory is configured to store instructions; and The processor is configured to retrieve instructions from memory and, when executing instructions, to implement the aforementioned text processing methods.

[0014] A third aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described text processing method.

[0015] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the text processing method described above.

[0016] The above technical solution enables the accurate acquisition of lengthy knowledge text containing answer clues after receiving the question text. Using semantic boundaries and scores generated by natural language processing algorithms, segmentation based on a preset overlap ratio is achieved without exceeding the available text budget of the target large language model. This ensures that each text block is fully received by the model while maintaining the semantic continuity of adjacent text blocks, thus avoiding information loss and comprehension gaps caused by simple truncation. Subsequently, temporary responses are determined for each text block using the question text as a constraint, ensuring that the model's analysis of local content always focuses on the same question, reducing the introduction of irrelevant information. Then, summary prompts are generated based on the question text and each temporary response, organizing scattered key information into guiding elements that can be comprehensively integrated by the model. Finally, the target large language model outputs the answer text, achieving full coverage, semantic integrity, and logical coherence of the lengthy knowledge text, improving the relevance, accuracy, and interpretability of the answer.

[0017] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The flowchart illustrating a text processing method for a large language model according to an embodiment of this application is shown schematically. Figure 2 The diagram illustrates a structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0021] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0022] Figure 1 A flowchart illustrating a text processing method for a large language model according to an embodiment of this application is shown schematically. Figure 1 As shown in the figure, this application provides a text processing method for a large language model, which may include the following steps.

[0023] Step 101: In response to the user's input of the question text into the target large language model, obtain the ultra-long knowledge text; the ultra-long knowledge text contains the content of the answer to the question text.

[0024] In this embodiment, the user-input question text is natural language text content provided by the user to the target large language model to obtain an answer to the target question. It is typically a question, task instruction, or semantic description of a specific topic. The target large language model is an intelligent model system capable of performing natural language understanding and generation tasks, used to perform semantic parsing and content generation on the input question text. Ultra-long knowledge text refers to text data containing a large amount of information and whose overall length exceeds the contextual limit that the target large language model can process in a single instance. It belongs to the knowledge sources or corpus content that the model can utilize, and this text contains information fragments and semantic clues necessary to answer the question text. Obtaining ultra-long knowledge text means that after receiving the user's question text, the system retrieves long text content semantically related to the question text from a knowledge base, document set, or database through internal retrieval or external data interfaces, using it as the input basis for subsequent segmentation and analysis. This allows for the accurate extraction of ultra-long knowledge text covering the semantic information required for the question after the user asks a question, thus providing a complete and relevant data input basis for subsequent block processing and step-by-step reasoning based on the large language model, ensuring that the model has sufficient knowledge context to answer the user's question.

[0025] Step 102: Generate multiple semantic boundaries and a score for each semantic boundary corresponding to the ultra-long knowledge text based on natural language processing algorithms.

[0026] In this embodiment, the natural language processing algorithm is an algorithmic model used for linguistic feature recognition, semantic understanding, and structural analysis of text content. It can extract grammatical, semantic, and discourse-level structural information from extremely long knowledge texts to identify logical boundaries and semantic continuity. Semantic boundaries refer to the natural demarcation points within extremely long knowledge texts where semantics shifts from completeness to new themes, paragraphs, or logical units. These points can correspond to sentence ends, paragraph ends, or semantic turning points, and are candidate breakpoints for text segmentation. Scoring refers to a numerical index calculated by the natural language processing algorithm based on features such as semantic completeness, syntactic continuity, and contextual coherence at semantic boundaries. This index characterizes the priority or credibility of each semantic boundary as a text segmentation point. Thus, it enables structured semantic analysis of extremely long knowledge texts, generating multiple semantic boundaries covering the entire text and their corresponding scores. This provides a quantitative basis for subsequent text segmentation based on semantic completeness and model context constraints, making the segmentation results more consistent with semantic logic and natural language features.

[0027] Step 103: Based on the available text budget of the target large language model, multiple semantic boundaries, the score of each semantic boundary, and the preset overlap ratio, the ultra-long knowledge text is divided into multiple text blocks.

[0028] In this embodiment, the available text budget of the target large language model refers to the maximum number of characters that can be used to input ultra-long knowledge text after deducting necessary space such as prompts in a single model call, which is used to determine the length range of a single text block that can be completely received and processed by the model. Multiple semantic boundaries and the score of each semantic boundary are structured data generated by natural language processing algorithms in the preceding steps to characterize the semantic breakpoints and their rationality within the ultra-long knowledge text. The semantic boundaries are used to indicate the position of semantic changes in the text, and the scores are used to indicate the completeness and naturalness of the semantic segmentation at the semantic boundary. The preset overlap ratio refers to the proportion of the overlapping part to the length of the previous text block when segmenting adjacent text blocks to ensure the semantic continuity of the context. Multiple text blocks refer to several continuous sub-texts obtained by segmenting the ultra-long knowledge text under the combined effect of the available text budget, semantic boundaries, and overlap ratio. Each text block can be processed independently within the input constraints of the target large language model. This allows for the division of extremely long knowledge texts into several appropriately sized, semantically coherent text blocks without exceeding the limitations of the model context, based on the principle of prioritizing semantic integrity. This provides a structured input foundation for subsequent independent reasoning and intermediate response generation within each text block.

[0029] Step 104: Determine the provisional response for each text block based on the question text and the target large language model.

[0030] Step 105: Generate summary prompts based on the question text and the temporary responses corresponding to each text block.

[0031] In this embodiment, each text block is a text unit that can be independently processed within the model context constraints, obtained by segmenting a very long knowledge text based on semantic boundaries, available text budget, and a preset overlap ratio. A provisional response refers to the local analysis result generated by the target large language model for the text block after receiving the question text and its corresponding text block. This result reflects the information content and potential answers related to the question text within the text block. Summary prompts are used to guide the target large language model in subsequent steps for global summarization and answer generation. This establishes a semantic correspondence between the question text and each text block, enabling the target large language model to generate content-related provisional responses block by block. Based on this, summary prompts that cover all provisional response information are extracted, providing semantically complete and contextually coherent generation guidance for subsequent overall responses to the question text.

[0032] Step 106: Input the summary prompts into the target large language model to generate the answer text for the question text.

[0033] In this embodiment, the summary prompt is a comprehensive prompt generated based on the question text and the temporary responses corresponding to each text block. It guides the target large language model to aggregate key information from multiple text blocks and form a unified answer logic during the generation process. The target large language model is an intelligent algorithm model capable of performing natural language understanding and text generation tasks based on input prompts. After receiving the summary prompt, it can perform comprehensive reasoning and language organization on the semantic elements involved. The answer text is the final output generated by the target large language model in response to the summary prompt, used to directly answer the question content indicated by the question text. Guided by the summary prompt, the target large language model can globally integrate and reason about the local information contained in the previous text blocks, thereby generating a complete, logically coherent answer text that highly matches the semantics of the question text, achieving accurate understanding and concentrated expression of extremely long knowledge text content.

[0034] The above technical solution enables the accurate acquisition of lengthy knowledge text containing answer clues after receiving the question text. Using semantic boundaries and scores generated by natural language processing algorithms, segmentation based on a preset overlap ratio is achieved without exceeding the available text budget of the target large language model. This ensures that each text block is fully received by the model while maintaining the semantic continuity of adjacent text blocks, thus avoiding information loss and comprehension gaps caused by simple truncation. Subsequently, temporary responses are determined for each text block using the question text as a constraint, ensuring that the model's analysis of local content always focuses on the same question, reducing the introduction of irrelevant information. Then, summary prompts are generated based on the question text and each temporary response, organizing scattered key information into guiding elements that can be comprehensively integrated by the model. Finally, the target large language model outputs the answer text, achieving full coverage, semantic integrity, and logical coherence of the lengthy knowledge text, improving the relevance, accuracy, and interpretability of the answer.

[0035] In this embodiment of the application, the step of dividing an ultra-long knowledge text into multiple text blocks based on the available text budget of the target large language model, multiple semantic boundaries, the score of each semantic boundary, and a preset overlap ratio may include: using the available text budget as the maximum allowed length of each text block; determining the actual end position of each of the multiple text blocks one by one based on the multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the ultra-long knowledge text; the starting position of the first text block among the multiple text blocks is the same as the starting position of the ultra-long knowledge text; the starting position of all text blocks except the first text block is obtained based on the actual end position of the previous text block and the preset overlap ratio; and dividing the ultra-long knowledge text into multiple text blocks based on the actual end position and the starting position of each text block.

[0036] In this embodiment, the maximum allowed length refers to the upper limit of the length that a single text block can accommodate in a single model input, which is directly used as the available text budget of the target large language model to hard constrain the segmentation to not exceed the limit; the actual end position refers to the final end position of the current text block determined from the candidate boundaries of the ultra-long knowledge text based on multiple semantic boundaries and the score of each semantic boundary within the range not exceeding the maximum allowed length, in order to ensure that the end of the block falls on a more semantically natural breakpoint; the start position refers to the starting index of the text block in the ultra-long knowledge text, wherein the start position of the first text block is the same as the start position of the ultra-long knowledge text, and the start positions of all text blocks other than the first text block are obtained based on the actual end position of the previous text block and a preset overlap ratio, so as to form an overlapping area between adjacent text blocks and maintain semantic continuity. By using the available text budget as the maximum allowed length, determining the actual end position of each text block one by one with multiple semantic boundaries and their scores, and segmenting ultra-long knowledge texts according to the start position and the actual end position, multiple text blocks with complete coverage, boundaries falling on high-quality semantic breakpoints, and adjacent blocks having preset overlap can be generated without exceeding the model input limit. This significantly reduces the risk of semantic tearing and information omission caused by segmentation and provides structured and coherent input units for subsequent processing.

[0037] In this embodiment of the application, the step of determining the actual end position of each of the multiple text blocks based on multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the super-long knowledge text may include: determining the temporary end position of the current text block based on the actual end position of the previous text block or the start position of the super-long knowledge text, the maximum allowed length, and the super-long knowledge text; determining multiple current semantic boundaries and the score of each current semantic boundary in the end portion of the current text block based on multiple semantic boundaries and the score of each semantic boundary; and taking the current semantic boundary with the highest score among the multiple current semantic boundaries as the actual end position of the current text block.

[0038] In this embodiment, the temporary end position refers to the current candidate block tail position derived by pushing forward from the actual end position of the previous text block or the start position of the ultra-long knowledge text, within the constraint of not exceeding the maximum allowed length, and is used to define the scope of this determination; the block tail portion of the current text block is a local interval defined in the ultra-long knowledge text around the temporary end position, used to search for more natural breakpoints; multiple current semantic boundaries refer to a set of several semantic boundaries identified within the block tail portion, and the score of each current semantic boundary is a quantitative evaluation of these local semantic boundaries in terms of semantic integrity and coherence; the current semantic boundary refers to any candidate boundary in the above set, used to participate in this block tail selection; the actual end position of the current text block refers to the final block tail location determined after selecting the highest score among multiple current semantic boundaries, serving as the termination index of the text block in the ultra-long knowledge text. By first determining a temporary end position to ensure that the length does not exceed the limit, and then comparing the scores of multiple current semantic boundaries in the end part of the block and taking the highest score as the actual end position, the text block boundary can satisfy the length constraint and fall on a more natural semantic breakpoint, thereby reducing the risk of semantic tearing and information fragmentation caused by segmentation and improving the semantic integrity and readability of individual text block content.

[0039] In this embodiment of the application, the step of determining the temporary response corresponding to each text block based on the question text and the target large language model may include: determining the segmented prompt words corresponding to each text block based on the question text; inputting the segmented prompt words corresponding to each text block into the target large language model to obtain the temporary response corresponding to each text block.

[0040] In this embodiment, the question text serves as the task constraint and semantic focus basis. The system generates instructional prompts for each text block to drive the target large language model to perform local analysis. These segmented prompts limit the model's understanding and judgment to the corresponding text block and outputs local results based solely on the question text. The segmented prompts and corresponding text blocks are submitted to the target large language model, which then produces an intermediate response that reflects only the relevant information between that text block and the question text. This step allows the analysis of different text blocks to be performed independently under the same prompt constraints, using the same question text as the standard. This ensures that temporary responses remain consistent in relevance, boundary conditions, and granularity, reducing irrelevant information and cross-block interference. Consequently, each text block obtains a semantically focused, formatted, and easily processed intermediate result.

[0041] In this embodiment of the application, the step of generating summary prompts based on the question text and the temporary replies corresponding to each text block may include: generating merged text based on the temporary replies corresponding to each text block; and generating summary prompts based on the question text and the merged text.

[0042] In this embodiment, the merged text is a unified text result obtained by standardizing the temporary responses corresponding to each text block. Its formation process uses the question text as a semantic benchmark, deduplicating, sorting, and coherently organizing the temporary responses while retaining necessary supporting evidence. This centralizes key information scattered across various text blocks within the same semantic framework. Based on this, the target semantics and scope of expression are determined using the question text, and the key points and evidence provided by the merged text serve as content sources, generating guiding prompts for overall generation. This step ensures strict alignment between the summary prompts and the question text, achieving comprehensive coverage and standardized aggregation of temporary response information, significantly reducing redundancy and offset, and ensuring that subsequent generation processes are based on a unified, complete, and coherent semantic benchmark.

[0043] In this embodiment of the application, the step of generating merged text based on the temporary replies corresponding to each text block may include: constructing a list of temporary replies corresponding to multiple text blocks based on the temporary replies corresponding to each text block; taking each temporary reply in the temporary reply list as a baseline reply, and forming multiple first combinations corresponding to the baseline reply with each of the remaining temporary replies; determining the cosine similarity of each first combination corresponding to the baseline reply based on the term frequency inverse document frequency method; merging the first combinations with a cosine similarity greater than or equal to a preset similarity threshold into a second combination; taking the temporary reply with the longest number of characters in the second combination as the final reply corresponding to the baseline reply; and generating merged text based on the final replies corresponding to each baseline reply.

[0044] In this embodiment, the temporary response list is an ordered set of temporary responses corresponding to each text block, collected in the order of text blocks, and used as the object for deduplication and merging. The benchmark response is a temporary response currently selected for comparison when traversing the temporary response list. The first combination is a set of groups formed by pairing the benchmark response with any other temporary response, used to calculate the similarity of each pair. The preset similarity threshold is a numerical threshold used to determine whether two temporary responses belong to duplicate or highly similar content. The second combination refers to the set of temporary responses corresponding to those pairs with similarity greater than or equal to the preset similarity threshold, which are merged into the benchmark response name after calculating the cosine similarity of the first combination using the word frequency inverse document frequency method. The final response refers to the information representative selected in the set according to the longest character principle, used to replace the other temporary responses in the set. The merged text refers to the result text after collecting and organizing the final responses corresponding to all benchmark responses in a predetermined order. Through the above processing, the deduplication and synonym merging of temporary replies can be completed automatically without changing the original semantics, retaining more representative content with greater information, reducing redundancy and conflicts, and obtaining a merged text that is structurally regular, well-organized, and easy to process later.

[0045] In this embodiment of the application, the step of determining the cosine similarity of each first combination corresponding to the benchmark response based on the inverse document frequency method may include: segmenting each temporary response in each first combination to obtain Chinese word segments corresponding to each temporary response; determining the feature vector corresponding to each temporary response based on the inverse document frequency method and the Chinese word segments corresponding to each temporary response; and determining the cosine similarity of each first combination based on the feature vector corresponding to each temporary response.

[0046] In this embodiment, segmentation is used to convert each temporary response in the first group into a Chinese word segmentation sequence that is easy to statistically analyze and calculate, so as to establish the term space on which the feature vector depends. The feature vector is a numerical representation formed by weighting the terms using the term frequency inverse document frequency method based on Chinese word segmentation, and is used to characterize the information distribution of each temporary response at the term level. Cosine similarity is calculated based on the feature vectors corresponding to each temporary response, and the cosine of the angle between them in the vector space is used to measure the similarity between any two temporary responses in the first group in terms of content and generate comparable similarity scores. Through this step, natural language text can be converted into a computable vector representation and a stable, thresholdable similarity index can be given, thereby providing a quantitative basis for subsequent grouping and deduplication based on similarity thresholds, and ensuring that the merging process is based on a unified and objective measurement standard.

[0047] Through the above technical solution, after receiving the question text, it is possible to obtain ultra-long knowledge text containing answer clues, and use natural language processing algorithms to generate semantic boundaries and scores covering the entire text. This allows the ultra-long knowledge text to be segmented into multiple text blocks whose boundaries fall on high-scoring semantic breakpoints and whose length does not exceed the limit, under the constraints of the available text budget of the target large language model and a preset overlap ratio. Subsequently, based on the question text and the target large language model, temporary responses are determined for each text block, ensuring that local analysis always focuses on the same query objective. When generating the merged text, a temporary response list is first constructed based on the temporary responses corresponding to each text block. For any selected baseline response in the list, it is paired with each of the other temporary responses in the list to form multiple first combinations corresponding to that baseline response. The system performs segmentation and Chinese word segmentation on each first combination, generates feature vectors for each temporary response based on the word frequency inverse document frequency method, and calculates cosine similarity. Pairs with similarity greater than or equal to a preset similarity threshold are merged into a second combination. The temporary response with the longest number of characters in the second combination is then used as the final response corresponding to the baseline response. Based on the final responses corresponding to each baseline response, a structurally well-structured merged text is formed. Finally, summary prompts are generated based on the question text and the merged text and input into the target large language model to obtain the answer text. This achieves full coverage and information aggregation of ultra-long knowledge texts without breaking the context constraints, reduces semantic loss and conflict caused by segmentation and redundancy, and improves the relevance, accuracy, coherence, and reproducibility of the answer text.

[0048] Figure 2 A schematic diagram illustrating the structure of a computer device according to an embodiment of this application is provided. Figure 2 As shown in the illustration, this application provides a computer device that may include: Memory 210 is configured to store instructions; and Processor 220 is configured to retrieve instructions from memory 210 and to implement the aforementioned text processing method when executing instructions.

[0049] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described text processing method.

[0050] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described text processing method.

[0051] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0055] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0056] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0057] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0058] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0059] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A text processing method for large language models, characterized in that, The method includes: In response to user input of question text into a target large language model, an ultra-long knowledge text is obtained; the ultra-long knowledge text contains the content that answers the question text. The algorithm generates multiple semantic boundaries and a score for each semantic boundary corresponding to the ultra-long knowledge text based on natural language processing. Based on the available text budget of the target large language model, the multiple semantic boundaries, the score of each semantic boundary, and the preset overlap ratio, the ultra-long knowledge text is divided into multiple text blocks; Based on the question text and the target large language model, determine the temporary response corresponding to each text block; A summary prompt is generated based on the question text and the temporary response corresponding to each text block; The summary prompts are input into the target large language model to generate an answer text for the question text.

2. The text processing method according to claim 1, characterized in that, The step of segmenting the ultra-long knowledge text into multiple text blocks based on the available text budget of the target large language model, the multiple semantic boundaries, the score of each semantic boundary, and the preset overlap ratio includes: The available text budget is used as the maximum allowed length for each text block; Based on the multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the ultra-long knowledge text, the actual end position of each of the multiple text blocks is determined one by one; the starting position of the first text block among the multiple text blocks is the same as the starting position of the ultra-long knowledge text; the starting position of all text blocks except the first text block is obtained based on the actual end position of the previous text block and the preset overlap ratio; The long knowledge text is divided into multiple text blocks based on the actual end and start positions of each text block.

3. The text processing method according to claim 2, characterized in that, The step of determining the actual end position of each of the multiple text blocks based on the multiple semantic boundaries, the score of each semantic boundary, the maximum allowed length, and the ultra-long knowledge text includes: The temporary end position of the current text block is determined based on the actual end position of the previous text block or the start position of the super-long knowledge text, the maximum allowed length, and the super-long knowledge text. Based on the plurality of semantic boundaries and the score of each semantic boundary, the plurality of current semantic boundaries and the score of each current semantic boundary are determined in the end portion of the current text block. The current semantic boundary with the highest score among the multiple current semantic boundaries is taken as the actual end position of the current text block.

4. The text processing method according to any one of claims 1 to 3, characterized in that, The step of determining the temporary response corresponding to each text block based on the question text and the target large language model includes: Based on the question text, determine the segmentation prompt words corresponding to each text block; The segmented prompt words corresponding to each text block are input into the target large language model to obtain the temporary response corresponding to each text block.

5. The text processing method according to any one of claims 1 to 3, characterized in that, The step of generating summary prompts based on the question text and the temporary responses corresponding to each text block includes: Generate merged text based on the temporary response corresponding to each text block; A summary prompt is generated based on the question text and the merged text.

6. The text processing method according to claim 5, characterized in that, The step of generating merged text based on the temporary response corresponding to each text block includes: Construct a temporary response list corresponding to the multiple text blocks based on the temporary response corresponding to each text block; Each temporary reply in the temporary reply list is used as a baseline reply, and together with each of the remaining temporary replies, they form multiple first combinations corresponding to the baseline reply. The cosine similarity of each first combination corresponding to the benchmark response is determined based on the term frequency inverse document frequency method. The first combination whose cosine similarity is greater than or equal to a preset similarity threshold is merged into the second combination; The temporary response with the longest character count in the second combination will be taken as the final response corresponding to the baseline response. The merged text is generated based on the final response corresponding to each of the aforementioned benchmark responses.

7. The text processing method according to claim 6, characterized in that, The step of determining the cosine similarity of each first combination corresponding to the benchmark response based on the inverse document frequency method includes: Each temporary response in each first combination is segmented to obtain the corresponding Chinese word segmentation for each temporary response; The feature vector corresponding to each temporary reply is determined based on the inverse document frequency method and the Chinese word segmentation corresponding to each temporary reply. The cosine similarity of each first combination is determined based on the feature vectors corresponding to each temporary response.

8. A computer device, characterized in that, include: The memory is configured to store instructions; as well as A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the text processing method according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the text processing method according to any one of claims 1 to 7.

Images