Text segmentation method and related equipment

Abstract

The invention provides a text segmentation method and related equipment. The method comprises the steps of obtaining a to-be-processed text; segmenting the to-be-processed text into ordered statement sequences to obtain an initial statement set of the to-be-processed text; wherein the ordered statement sequence comprises a plurality of statements; the semantic variation, the confusion degree variation and the information entropy variation of a first target text block are calculated when a to-be-decided statement in an initial statement set of the to-be-processed text is added into the first target text block, and the first target text block is a set of multiple statements meeting a merging condition; determining the collaboration degree of the statement to be decided and the first target text block based on the semantic variable quantity, the confusion variable quantity and the information entropy variable quantity; and partitioning the to-be-processed text based on the collaboration degree of the to-be-decided statement and the first target text block to obtain a target partitioned text of the to-be-processed text. The text segmentation quality can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a text segmentation method and related equipment. Background Technology

[0002] In Retrieval-Augmented Generation (RAG) systems, text segmentation is a fundamental step affecting retrieval accuracy and generation quality. Existing technologies aim to segment large-scale text into semantically complete, retrieval-suitable smaller segments. Mainstream methods can be broadly categorized into document structure-based, fixed-size and sliding-window, semantic-based segmentation, and emerging hybrid-granularity and Large Language Model (LLM)-driven segmentation. These methods each prioritize efficiency and semantic fidelity, collectively forming the current technical landscape of the RAG system's preprocessing stage.

[0003] Specifically, existing technologies are evolving from rule-based approaches to incorporating intelligent models. Fixed-size chunking is simple and efficient but prone to semantic fragmentation; sliding windows alleviate this problem by using overlapping intervals but introduce redundant overhead. Document structure-based methods utilize formatting information such as headings and paragraphs, which are effective for formatted documents but difficult to generalize. Semantic clustering and emerging methods such as Mix-of-Granularity (MoG) and LLM-driven perplexity (PPL) chunking can improve semantic coherence, but their high computational complexity and reliance on model inference or labeled data limit their application in real-time or resource-constrained scenarios.

[0004] However, existing methods generally suffer from unresolved core technical problems, particularly in their inability to balance the coherence of text segments with sensitivity to sudden information changes. On the one hand, rule-based methods are prone to causing contextual fragmentation, while complex model methods incur excessive computational overhead; on the other hand, most strategies lack dynamic adaptability and struggle to accurately capture logical shifts within the text.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This disclosure provides a text segmentation method and related equipment that can reduce invalid overlaps and prevent logical breaks, while taking into account both the coherence of the segments and the sensitivity to information abrupt changes, thereby improving the quality of text segmentation.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to one aspect of this disclosure, a text segmentation method is provided, the method comprising: acquiring a text to be processed; segmenting the text to be processed into an ordered sequence of statements to obtain an initial set of statements of the text to be processed; wherein the ordered sequence of statements includes multiple statements; calculating the semantic change, the perplexity change, and the information entropy change of the first target text block when a decision statement in the initial set of statements of the text to be processed is added to the first target text block; determining the coherence degree between the decision statement and the first target text block based on the semantic change, the perplexity change, and the information entropy change; and segmenting the text to be processed into blocks based on the coherence degree between the decision statement and the first target text block to obtain target block text of the text to be processed.

[0009] In some embodiments, calculating the semantic change, perplexity change, and information entropy change of the first target text block when the decision statement in the initial set of statements to be processed is added to the first target text block includes: obtaining the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block; calculating the second average pooled semantic vector, the second average perplexity, and the second average information entropy after the decision statement is added to the first target text block; and calculating the semantic change between the second average pooled semantic vector and the first average pooled semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy, respectively.

[0010] In some embodiments, calculating the semantic change between the second average pooling semantic vector and the first average pooling semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy includes: calculating the semantic similarity gain between the second average pooling semantic vector and the first average pooling semantic vector to obtain the semantic change; calculating the difference between the second average perplexity and the first average perplexity to obtain the perplexity change; and calculating the difference between the second average information entropy and the first average information entropy to obtain the information entropy change.

[0011] In some embodiments, obtaining the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block includes: obtaining the semantic vector, perplexity, and information entropy of each statement in the first target text block; calculating the mean of the semantic vector, perplexity, and information entropy of all statements in the first target text block respectively, to obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block.

[0012] In some embodiments, calculating the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the decision statement to the first target text block includes: calculating the second average pooled semantic vector, the second average perplexity, and the second average information entropy in linear time complexity, provided that the time complexity is linearly related to the vector dimension.

[0013] In some embodiments, the step of calculating the second average pooled semantic vector, the second average perplexity, and the second average information entropy within linear time complexity, where time complexity is linearly related to vector dimension, after adding the statement to be decided to the first target text block, includes: inputting the statement to be decided into a vectorized model and outputting the semantic vector of the statement to be decided; inputting the statement to be decided into a lightweight language model and outputting the conditional probability of each character in the statement to be decided; calculating the perplexity of the statement to be decided based on the perplexity function and the conditional probability of each character in the statement to be decided; calculating the information entropy of the statement to be decided based on the information entropy function and the conditional probability of each character in the statement to be decided; calculating the second average pooled semantic vector based on the first average pooled semantic vector and the semantic vector of the statement to be decided; calculating the second average perplexity based on the first average perplexity and the perplexity of the statement to be decided; and calculating the second average information entropy based on the first average information entropy and the information entropy of the statement to be decided.

[0014] In some embodiments, determining the degree of synergy between the statement to be decided and the first target text block based on the semantic change, the perplexity change, and the information entropy change includes: obtaining the weights of the semantic change, the perplexity change, and the information entropy change; and calculating the degree of synergy between the statement to be decided and the first target text block based on the weights of the semantic change, the perplexity change, and the information entropy change, as well as the semantic change, the perplexity change, and the information entropy change.

[0015] In some embodiments, the step of segmenting the text to be processed based on the degree of synergy between the statement to be decided and the first target text block includes: obtaining a synergy threshold; when the synergy is greater than the synergy threshold, merging the statement to be decided into the first target text block to obtain a second target text block; and when the synergy is less than or equal to the synergy threshold, generating a third target text block based on the statement to be decided.

[0016] In some embodiments, obtaining the synergy threshold includes: obtaining an initial synergy threshold and a target synergy threshold; and determining the synergy threshold based on the initial synergy threshold and the target synergy threshold using a preheating progressive adjustment strategy.

[0017] In some embodiments, determining the synergy threshold based on the initial synergy threshold and the target synergy threshold using a preheating progressive adjustment strategy includes: determining the synergy threshold using a linear preheating progressive adjustment strategy based on the initial synergy threshold and the target synergy threshold; and / or, determining the synergy threshold using an exponential preheating progressive adjustment strategy based on the initial synergy threshold and the target synergy threshold; and / or, determining the synergy threshold using a cosine preheating progressive adjustment strategy based on the initial synergy threshold and the target synergy threshold.

[0018] In some embodiments, merging the statement to be decided into the first target text block to obtain a second target text block includes: determining whether the total length of the number of tokens or characters in the second target text block is greater than or equal to a first preset length; if it is determined that the total length of the number of tokens or characters in the second target text block is less than the first preset length, merging the statement to be decided into the first target text block to obtain a second target text block; if it is determined that the total length of the number of tokens or characters in the second target text block is greater than or equal to the first preset length, generating a fourth target text block based on the statement to be decided.

[0019] In some embodiments, the method further includes: determining whether the total length of the number of tokens or characters of the last target text block in the target text block is less than a second preset length; if the total length of the number of tokens or characters of the last target text block is less than the second preset length, determining the semantic similarity between the last target text block and the second-to-last target text block; and merging the last target text block and the second-to-last target text block when the semantic similarity is greater than a semantic similarity threshold.

[0020] In some embodiments, the method further includes: determining the similarity between candidate statements in the initial set of statements of the text to be processed and the first target text block; and determining candidate statements with a similarity greater than a preset threshold as statements to be decided.

[0021] According to another aspect of this disclosure, a text segmentation apparatus is also provided, the apparatus comprising: an acquisition module for acquiring text to be processed; a segmentation module for segmenting the text to be processed into an ordered sequence of statements to obtain an initial set of statements of the text to be processed; wherein the ordered sequence of statements includes multiple statements; a change calculation module for calculating the semantic change, perplexity change, and information entropy change of the first target text block when the statement to be decided in the initial set of statements of the text to be processed is added to the first target text block; a synergy calculation module for determining the synergy between the statement to be decided and the first target text block based on the semantic change, the perplexity change, and the information entropy change; and a block segmentation module for segmenting the text to be processed into blocks based on the synergy between the statement to be decided and the first target text block to obtain target block text of the text to be processed.

[0022] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the text segmentation method described in any of the preceding claims by executing the executable instructions.

[0023] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the text segmentation method described in any of the preceding claims.

[0024] According to another aspect of this disclosure, a computer program product is also provided, comprising: a computer program or instructions that, when executed by a processor, implement the text segmentation method of any one of the above.

[0025] This disclosure provides a text segmentation method and related apparatus. The method includes: acquiring a text to be processed; segmenting the text to be processed into an ordered sequence of sentences to obtain an initial set of sentences; wherein the ordered sequence of sentences includes multiple sentences; calculating the semantic change, perplexity change, and information entropy change of the first target text block when the sentence to be decided in the initial set of sentences is added to the first target text block, wherein the first target text block is a set of multiple sentences that meet the merging conditions; determining the synergy between the sentence to be decided and the first target text block based on the semantic change, perplexity change, and information entropy change; and segmenting the text to be processed into target blocks based on the synergy between the sentence to be decided and the first target text block to obtain target block text of the text to be processed. This disclosure constructs a unified scoring mechanism by integrating three indicators: semantic similarity gain, language model perplexity change, and information entropy change, to determine whether to merge a new sentence into the current block. Specifically, the semantic similarity change quantifies the semantic coherence between the statement to be decided and the first target text block, ensuring that the overall semantic logic of the text is not disrupted when merging sentences. The perplexity change reflects the language model's adaptation to new sentences. By monitoring fluctuations in perplexity, sentences that do not semantically match the current context or exhibit logical abrupt changes can be identified, thus avoiding their merging into the current block and reducing the possibility of semantic fragmentation. The information entropy change measures the diversity and information density of the text. By analyzing changes in information entropy, sentences that jump between topics or argument chains but have high word similarity can be detected, effectively suppressing logical or informational abrupt changes and ensuring the coherence within the block. Furthermore, the semantic similarity gain, perplexity change, and information entropy change are integrated into a unified score to robustly judge sentence merging decisions across multiple dimensions. This collaborative mechanism reduces invalid overlap while preventing logical breaks and improving the internal consistency of the block. Compared to traditional segmentation techniques, this disclosure uses quantization to suppress logical or informational mutations while maintaining semantic coherence, ensuring that sentences that jump in the topic or argument chain but still have high word similarity can be accurately extracted. This improves the internal consistency of each segment and enhances the overall retrieval recall and generation quality.

[0026] In summary, this disclosure addresses the shortcomings of traditional segmentation techniques by using multiple metrics for collaborative decision-making, balancing global and local consistency, and by employing both perplexity and information entropy metrics to promptly cut off argument jumps, thereby significantly improving the effectiveness of text segmentation.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 This diagram illustrates a system architecture of a text segmentation method according to an embodiment of the present disclosure.

[0030] Figure 2 This diagram illustrates a text segmentation method according to an embodiment of the present disclosure;

[0031] Figure 3 This illustration shows a flowchart of a method for decomposing each tool call sequence according to an embodiment of the present disclosure;

[0032] Figure 4 The flowchart illustrates a method for calculating the semantic change, perplexity change, and information entropy change of a first target text block when a statement to be decided is added to the first target text block, according to an embodiment of the present disclosure.

[0033] Figure 5 This illustration shows a flowchart of a method for calculating the semantic change, perplexity change, and information entropy change of a first target text block when a statement to be decided is added to the first target text block, according to an embodiment of the present disclosure.

[0034] Figure 6 A flowchart illustrating a method for calculating a second average pooling semantic vector, a second average perplexity, and a second average information entropy in an embodiment of this disclosure is shown.

[0035] Figure 7 A flowchart illustrating a method for calculating a second average pooling semantic vector, a second average perplexity, and a second average information entropy in an embodiment of this disclosure is shown.

[0036] Figure 8 This invention discloses a flowchart of a method for filtering statements to be decided in an embodiment of the present disclosure;

[0037] Figure 9 This diagram illustrates a text segmentation method according to an embodiment of the present disclosure;

[0038] Figure 10 This illustration shows one of the specific text segmentation methods in an embodiment of the present disclosure;

[0039] Figure 11 This is a second flowchart illustrating a specific text segmentation method according to an embodiment of the present disclosure;

[0040] Figure 12This is the third flowchart of a specific text segmentation method in an embodiment of the present disclosure;

[0041] Figure 13 This diagram illustrates a text segmentation apparatus according to an embodiment of the present disclosure;

[0042] Figure 14 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0043] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0044] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0045] As mentioned earlier, in traditional text segmentation techniques, fixed-length and sliding window methods are prone to breaking at sentence or paragraph boundaries, resulting in a lack of complete semantic coherence in the retrieved fragments; the block redundancy caused by overlapping sliding windows significantly increases the number of vector storage and retrieval operations, affecting system response efficiency; structured block segmentation methods have high requirements for document format, and their performance drops significantly when dealing with unstructured text such as PDFs and scanned documents; semantic clustering, MoG, MoC and other methods require additional model inference or training costs, making them difficult to deploy in edge or low-computing environments; and most methods cannot dynamically adjust the block segmentation strategy according to different queries at runtime, resulting in limited applicability to diverse scenarios.

[0046] In view of this, this disclosure constructs a unified scoring mechanism by integrating three indicators: semantic similarity gain, language model perplexity change, and information entropy change, to determine whether to merge a new sentence into the current block. Specifically, the semantic similarity change quantifies the semantic coherence between the sentence to be decided and the first target text block, ensuring that the overall semantic logic of the text is not disrupted when merging sentences; the perplexity change reflects the language model's adaptation to new sentences, and by monitoring fluctuations in perplexity, sentences that do not semantically match the current context or have logical abrupt changes can be identified, thus avoiding their merging into the current block and reducing the possibility of semantic fragmentation; the information entropy change measures the diversity and information density of the text, and by analyzing changes in information entropy, sentences that jump in topic or argument chain but have high word similarity can be detected, thus effectively suppressing logical or informational abrupt changes and ensuring the coherence within the block. Furthermore, this invention integrates three metrics—semantic similarity gain, perplexity change, and information entropy change—into a unified score for robustly judging sentence merging decisions across multiple dimensions. This collaborative mechanism reduces invalid overlap while preventing logical breaks, thus improving the internal consistency of segmentation. Compared to traditional segmentation techniques, this disclosure, through quantification, maintains semantic coherence while suppressing logical or informational abrupt changes, ensuring the accurate extraction of sentences that jump between topics or argument chains but still maintain high word similarity. This improves the internal consistency of each segment and enhances overall retrieval recall and generation quality. Ultimately, this disclosure achieves collaborative decision-making through multiple metrics, balancing global and local consistency, and uses both perplexity and information entropy checks to promptly cut off argument jumps, thereby addressing the shortcomings of traditional segmentation techniques and significantly improving text segmentation performance.

[0047] To facilitate understanding, before introducing the embodiments of this disclosure, the following explanations are provided for several terms involved in the embodiments of this disclosure:

[0048] Large Language Model (LLM): A natural language processing model built on deep learning architectures (such as Transformer). By pre-training on large-scale unlabeled text data, it learns the statistical distribution, grammatical rules, semantic associations and cross-domain knowledge of natural language. It has powerful context understanding and text generation capabilities, and can realize diverse natural language tasks such as dialogue interaction, content creation and logical reasoning. It is the core foundational model of tool integration reasoning (TIR) ​​technology.

[0049] Retrieval-Augmented Generation (RAG) is an artificial intelligence framework or technique that combines the generative capabilities of a large language model with the retrieval capabilities of an external knowledge base. Its core idea is to retrieve relevant information from one or more data sources (such as databases, document collections, or real-time information sources) before the LLM generates text. This retrieved information is then provided to the LLM as context, helping the model generate more accurate, timely, factual, and domain-specific answers, while effectively reducing the possibility of the model generating "illusions" (i.e., generating inaccurate or fictitious information).

[0050] Text chunking: In the fields of natural language processing and information processing, this refers to breaking down a long, continuous text into several smaller, more clearly structured, and easier-to-manage and process sub-segments, which are also called "text chunks" or "speech chunks".

[0051] Token: The basic unit of text processing, it can break down continuous text into independent parts. These parts can be represented as words, characters, or even meaningful fragments smaller than characters.

[0052] Perplexity (PPL): A metric commonly used to evaluate the performance of language models, primarily measuring the quality of a model's predictions on a sample sequence. A lower perplexity value indicates a stronger predictive ability of the language model on the test data, meaning a lower level of "uncertainty" or "unexpectedness" in the model's prediction of the text. This metric is derived by calculating the probability assigned to each word by the model on the test set and is generally considered a quantitative measure of the uncertainty exhibited by the model when predicting the next word.

[0053] Information entropy, also known as Shannon entropy, is a concept in information theory used to measure the uncertainty or information content of a random variable. The higher the entropy of a source, the more random and unpredictable its state, and therefore the greater the amount of information received when a message is received from that source. Conversely, if the state of a source is very deterministic, its entropy is low, and the amount of information contained in the received message is also small.

[0054] Transformer is a deep learning model architecture based on the self-attention mechanism. It introduces self-attention, allowing the model to weigh the importance of different parts of the input sequence and capture long-range dependencies when processing sequential data, without having to process data sequentially like a Recurrent Neural Network (RNN). Due to its parallel processing capabilities and efficiency, the Transformer architecture has become the foundation for many state-of-the-art large-scale language models.

[0055] Locality Sensitive Hashing (LSH) is a technique for Approximate Nearest Neighbor Search (ANNS), particularly suitable for high-dimensional data. Its core idea is to map similar data points into the same bucket using a hash function, allowing queries to search for candidate points within only a few buckets instead of searching the entire dataset. It is commonly used for quickly finding similar content.

[0056] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0057] Figure 1 A schematic diagram of an exemplary application system architecture to which the text segmentation method of the embodiments of this disclosure can be applied is shown. For example... Figure 1 As shown, the system architecture may include terminal device 101, network 102 and server 103.

[0058] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.

[0059] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0060] Terminal device 101 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0061] Optionally, the client of the application installed on different terminal devices 101 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.

[0062] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0063] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0064] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0065] Under the above system architecture, this disclosure provides a text segmentation method that can be executed by any electronic device with computing capabilities.

[0066] In some embodiments, the text segmentation method provided in this disclosure can be executed by a terminal device in the system architecture described above; in other embodiments, the text segmentation method provided in this disclosure can be executed by a server in the system architecture described above; in still other embodiments, the text segmentation method provided in this disclosure can be implemented by the terminal device and the server in the system architecture described above through interaction.

[0067] Figure 2 This diagram illustrates a text segmentation method according to an embodiment of the present disclosure, in conjunction with... Figure 2 As shown, the text segmentation method provided in this disclosure embodiment may include the following steps:

[0068] S202, Obtain the text to be processed.

[0069] The text to be processed specifically refers to text data that needs to be segmented. Application scenarios include text structure parsing and content chunking. Specifically, it can be the title, description, or page text of a webpage, such as online news text; or it can be articles, paragraphs, or other text, such as academic papers. There are several ways to obtain the text to be processed. For example, users can submit text files in DOC, TXT, and PDF formats through the system upload interface, retrieve text data by connecting to third-party data interfaces, or use compliant web crawling technology to crawl target text in the public domain. The format of the text to be processed must meet system compatibility requirements, including but not limited to text content extraction results in plain text (TXT), rich text formats (DOC, DOCX), Extensible Markup Language (XML), and Hypertext Markup Language (HTML).

[0070] S204, the text to be processed is divided into an ordered sequence of statements to obtain an initial set of statements in the text to be processed; wherein, the ordered sequence of statements includes multiple statements.

[0071] In this embodiment, sentence segmentation is primarily based on punctuation marks and line breaks. Typically, a complete sentence ends with a period, question mark, or exclamation mark. By identifying these punctuation marks, the text to be processed is segmented into several substrings, each constituting an independent sentence unit. Subsequently, leading and trailing spaces are removed from each segmented sentence.

[0072] In some embodiments, if a sentence that is too short after segmentation is detected, such as having fewer than 5 characters or fewer than 3 tokens, the short sentence is merged with the preceding or following sentence to avoid generating semantically incomplete or meaningless text fragments.

[0073] In another embodiment, when the segmentation result contains an excessively long sentence, such as more than 400 characters or more than 200 tokens, a recursive segmentation strategy is adopted, which performs secondary segmentation by using parallel connectors such as commas and semicolons or logical conjunctions.

[0074] Furthermore, in some embodiments, a lightweight sentence boundary detection (SBD) model can be introduced to perform secondary verification of the initial segmentation results. This model uses 2-3 characters (or tokens) before and after the target position as a context window, and accurately determines whether the position is the true end of a sentence using a pre-trained boundary label classifier. The lightweight sentence boundary detection (SBD) model includes, but is not limited to, implementations based on the BiLSTM+CRF architecture or the Transformer framework; this embodiment does not impose any limitations on these implementations. This embodiment can effectively reduce the probability of missegmentation caused by special cases such as abbreviations, numerical representations, and floating-point number formats.

[0075] Specifically, since all judgments in this method are based on "sentence" units, the accuracy of sentence segmentation directly determines the quality of the segmentation. Therefore, this embodiment can further adopt a multi-stage fusion text segmentation strategy: first, initial segmentation is performed based on punctuation and a proper noun dictionary; then, the segmentation results are optimized and corrected using a pre-trained boundary detection model; finally, abnormal segmentation fragments that are too short or too long are merged or recursively subdivided to generate sentence sequences that are both accurate and stable. Compared to traditional methods that rely solely on regular expression matching or pure model judgment, which are prone to segmentation misjudgments or omissions, this solution effectively solves the segmentation errors caused by special text elements such as abbreviations, numerical expressions, and serial number annotations through a multi-stage collaborative processing mechanism, laying a reliable data foundation for subsequent joint evaluation tasks.

[0076] S206, calculate the semantic change, perplexity change, and information entropy change of the first target text block when the decision statement in the initial set of statements of the text to be processed is added to the first target text block.

[0077] In this embodiment, a sequential block-based strategy is adopted, that is, processing begins with the first sentence of the text to be processed (which has been pre-segmented into an ordered sequence of sentences). Here, the "sentence to be decided" specifically refers to the target sentence that needs to be determined whether to be included in the current text block. Initially, the first target text block is an empty text block; as the sentence-by-sentence processing progresses, several sentences that meet the merging conditions will be aggregated to form the target text block. In the block-based decision-making stage, to determine whether the sentence to be decided should be added to the current text block (here, to facilitate the distinction of potentially new text blocks that may be formed later, the current text block is named the first target text block), three dynamic indicators generated after the sentence to be decided is added to the first target text block need to be calculated: semantic change, confusion change, and information entropy change.

[0078] It should be noted that the first target text block is an empty text block in the initial stage, but in the normal block segmentation process (except for the initial empty state), it is essentially a non-empty set of multiple consecutive statements that meet the merging conditions. Multiple consecutive statements that meet the merging conditions are aggregated into the same text block because they have high semantic relevance and language coherence.

[0079] Specifically, the semantic change measure is used to characterize the degree of change in semantic coherence within a text block; the perplexity change measure is used to characterize the degree of "surprise" in the overall association between the statement to be decided and the first target text block. The higher the value, the more significant the logical abrupt change or information jump between the statement to be decided and the context of the first target text block. It is understandable that the perplexity change measure is based on a comprehensive evaluation of the entire statement to be decided, focusing on the global coherence of the statement to be decided, and is more sensitive to topic shifts or argument jumps in the text; the information entropy change measure is used to characterize the magnitude of change in the information distribution complexity of a text block. The larger the value, the more semantic types and the more dispersed the distribution of the first target text block after the statement to be decided is added. It can effectively capture local polysemy or ambiguity, and indicate text segments with a sudden increase in information density. It is understandable that the information entropy focuses on the local uncertainty of each token in the statement to be decided, focuses on the local feature analysis of the statement to be decided, and can capture fine-grained information changes such as the sudden appearance of new terms and the appearance of polysemous words. By quantitatively analyzing the above three indicators, we can assess the impact of adding the statement to be decided on the quality of the current text block, thus providing a basis for text segmentation decisions.

[0080] S208, determine the degree of synergy between the statement to be decided and the first target text block based on semantic change, perplexity change, and information entropy change.

[0081] In this embodiment, the degree of synergy between the statement to be decided and the first target text block is determined based on semantic change, perplexity change, and information entropy change. Essentially, this involves quantifying the impact of the statement to be decided on the core quality dimensions of the text block, and comprehensively evaluating their matching degree in three aspects: semantic coherence, language model matching, and information distribution rationality. Synergy reflects the level of adaptation between the statement to be decided and the current text block in terms of overall logical coherence, linguistic fluency, and information density coordination. A higher degree of synergy indicates a better fusion effect between the statement to be decided and the text block; conversely, a lower degree of synergy may indicate problems such as logical breaks, information redundancy, or topic deviation.

[0082] In some embodiments, determining the degree of synergy between the statement to be decided and the first target text block based on semantic change, perplexity change, and information entropy change includes: obtaining the weights of semantic change, perplexity change, and information entropy change; and calculating the degree of synergy between the statement to be decided and the first target text block based on the weights of semantic change, perplexity change, and information entropy change, as well as the semantic change, perplexity change, and information entropy change.

[0083] Specifically, this embodiment provides an exemplary collaborative evaluation function scheme that integrates three key indicators through a weighted linear combination:

[0084] Γ=αΔ sim -βΔppl -γΔ h

[0085] Where Γ represents the synergy evaluation value between the statement to be decided and the first target text block, Δ sim Δ represents the semantic change caused by the addition of the statement to be decided. ppl Let Δ be the change in confusion. h Let α be the change in information entropy, and β and γ be the changes in semantics (Δ). sim ), change in confusion (Δ) ppl ), change in information entropy (Δ) h The corresponding weighting coefficient.

[0086] It should be noted that Δ in the formula ppl With Δ h The negative sign is used because an increase in the value usually means a decrease in the degree of matching (logical abrupt change or increased information dispersion), which is contrary to the goal of collaboration.

[0087] In some embodiments, the weight coefficients α, β, and γ are adjustable parameters that can be automatically optimized using a small-scale validation set. Specifically, optimizing the weights based on current scenario data, such as domain text features and segmentation target requirements, enables the collaborative evaluation function to more accurately adapt to actual segmentation needs. For example, if the target text block has extremely high requirements for topic coherence, the α weight can be appropriately increased; if strict control of information density abrupt changes is required, the β weight can be strengthened. This dynamic optimization mechanism based on scenario data improves the evaluation function's generalization ability and effectiveness across different application scenarios.

[0088] It should be noted that, compared to traditional technical solutions that rely solely on a single dimension of semantic similarity for evaluation, this embodiment introduces two key indicators: change in perplexity and change in information entropy. These indicators address the limitations of traditional methods from the perspectives of global logical coherence and local information granularity.

[0089] Specifically, perplexity quantifies the language model's degree of surprise at the target statement as a whole. A higher perplexity value indicates a more significant logical abruptness or information jump between the statement and its existing context. While traditional semantic similarity can capture semantic connections, it cannot identify deeper logical problems such as broken argument chains or abrupt topic flows. By introducing perplexity variation, sentences with similar meanings but logically disjointed structures can be effectively intercepted, such as transitional sentences that suddenly introduce irrelevant arguments, preventing them from blindly merging into the current text block, thereby ensuring the thematic coherence and content generation quality of each block.

[0090] Information entropy measures the average uncertainty of a model when predicting each token. A higher information entropy value indicates a greater variety and more dispersed semantics at that location, enabling it to keenly capture local ambiguity or polysemy and highlight high-risk sections with sudden increases in information density. Traditional methods struggle to identify fine-grained information changes such as the intrusion of new terms and the mixing of polysemous words, while changes in information entropy can prevent contextual congestion or noise buildup caused by explosive information aggregation within fragments at a finer granular level, ensuring that the information density of text blocks remains within a reasonable range.

[0091] This embodiment achieves synergy between changes in perplexity and information entropy in terms of scope, sensitivity, and complementarity. When a sentence's theme is consistent with the preceding text (low perplexity change) but contains numerous ambiguous details (high information entropy change), a pure semantic similarity strategy might misjudge and allow it to pass, but the high information entropy change will trigger an interception. Conversely, if a sentence uses simple vocabulary (low information entropy change) but the topic suddenly shifts (high perplexity change), the high perplexity change will prevent merging, while the low information entropy change might not provide any warning. The combination of these two approaches maintains the overall thematic coherence while controlling the stability of information at the detail level, significantly improving the overall quality of text segmentation.

[0092] In summary, by using a collaborative evaluation function to weight and integrate the three indicators, this embodiment achieves comprehensive quality control from global logic to local information, providing a scientific and accurate decision-making basis for the collaborative adaptation of statements to be decided and text blocks.

[0093] S210, the text to be processed is divided into blocks based on the degree of synergy between the statement to be decided and the first target text block, to obtain the target block text of the text to be processed.

[0094] In this embodiment, the text to be processed is divided into blocks based on the degree of collaboration between the statement to be decided and the first target text block, including: obtaining a collaboration threshold; when the collaboration is greater than the collaboration threshold, merging the statement to be decided into the first target text block to obtain a second target text block; when the collaboration is less than or equal to the collaboration threshold, generating a third target text block based on the statement to be decided.

[0095] Specifically, a synergy threshold τ is set. If Γ > τ, it indicates that the statement S to be decided is... j Adding bad first target text block C k It will not destroy the first target text block C. k The semantic coherence will not lead to the first target text block C. k If the information changes abruptly, the decision statement S will be affected. j Merge into the first target text block C k The second target text block is obtained; when the degree of collaboration is less than or equal to the degree of collaboration threshold, the decision statement S is used. jCreate a new block for the first sentence, which generates the third target text block.

[0096] In some embodiments, to adapt to the text features of different documents, the cooperability threshold τ employs a dynamic adaptive adjustment mechanism. Specifically, it can be optimized in real time based on multi-dimensional document features, such as block number distribution, text block length, and document location information, to ensure optimal matching between the cooperability threshold τ and the text semantic structure.

[0097] For example, the value of τ is adjusted based on the first target text block, i.e., the current text block's sequence number in the document (e.g., the Nth text block). For instance, at the beginning or end of the document, where there is less context information, a stricter τ is used to reduce the merging probability, while in the middle of the document, where the context is richer, τ can be appropriately relaxed to increase the merging probability. This embodiment can avoid mis-segmentation caused by differences in document structure, such as introduction, conclusion, and body text, enhancing the model's adaptability to the global document structure. As another implementation, a dynamic adjustment strategy can also be adopted: allowing larger information jumps at the beginning of the document, thus setting the initial value of τ to a lower level; gradually increasing the value of τ as the document progresses to suppress excessive text block merging.

[0098] For example, the synergy threshold τ can be dynamically adjusted based on the text block length (i.e., local information density). Specifically, when the first target text block is short (e.g., contains only a few sentences), the τ value can be appropriately reduced to increase the probability of merging the sentences to be decided; conversely, if the first target text block is long (e.g., contains multiple complete semantic units), the τ value is increased accordingly to enhance merging flexibility and improve the integrity of the text block. Through the above mechanism, this embodiment can achieve an effective balance between the compactness of the text block and the integrity of the information, avoiding fragmentation problems caused by excessively short blocks or semantic confusion caused by excessively long blocks.

[0099] For example, the synergy threshold τ can also be dynamically adjusted based on document location information (global semantic roles). Specifically, τ is dynamically adjusted according to the position of the statement to be decided in the document (such as whether it is near a paragraph boundary, chapter transition point, etc.). For example, at the first sentence of a paragraph or the beginning of a chapter, a stricter τ is preferred (prioritizing the opening of new blocks), while in the middle of a paragraph, a more lenient merging strategy is allowed. This embodiment can ensure that key semantic boundaries (such as logical transitions, topic transitions) are not incorrectly merged, while optimizing the text coherence of non-critical areas.

[0100] In practice, the optimal τ value is calculated in real time using predefined rules (such as statistical heuristics) or machine learning models (such as lightweight regression / classifiers). For example, the following settings are provided:

[0101] τ = τ0 × f(block number, block length, position weight)

[0102] Where τ0 is the baseline threshold and f(·) is the adjustment function.

[0103] In the specific implementation process, a threshold prediction model can also be trained based on labeled data, inputting document features (such as local semantic similarity, positional encoding, etc.) and outputting a dynamic cooperability threshold τ.

[0104] The dynamic coherence threshold τ provided in this embodiment enables the chunking strategy to adapt to different document types, maximizing the information value of text chunks while ensuring semantic coherence.

[0105] In this embodiment, three indicators—semantic similarity gain, perplexity change, and information entropy change—are integrated into a unified score to robustly judge sentence merging decisions across multiple dimensions. Overcoming the limitations of existing technologies that rely on single-statement vector similarity or simple sliding window strategies, this mechanism, by quantifying the key objective of "suppressing logical or informational abrupt changes while maintaining semantic coherence," can accurately identify and extract sentences that, while maintaining high relevance in surface-level lexical similarity, exhibit significant jumps in thematic continuity or argumentative logical chains. This mechanism significantly enhances the semantic consistency within each text block. Compared to the severe semantic fragmentation or redundant content overlap problems easily caused by traditional block segmentation methods, this solution, by constructing a multi-dimensional robust judgment system, effectively suppresses invalid information overlap between text blocks and avoids discontinuous interruptions in argumentative logic, thereby systematically improving the recall accuracy of subsequent information retrieval and the overall quality of content generation.

[0106] In some embodiments, merging the statement to be decided into a first target text block to obtain a second target text block includes: determining whether the total length of the number of tokens or characters in the second target text block is greater than or equal to a first preset length; if it is determined that the total length of the number of tokens or characters in the second target text block is less than the first preset length, merging the statement to be decided into the first target text block to obtain a second target text block; if it is determined that the total length of the number of tokens or characters in the second target text block is greater than or equal to the first preset length, generating a fourth target text block based on the statement to be decided.

[0107] Specifically, it involves real-time monitoring of the first target text block, i.e., the current block (denoted as C). k The total number of tokens or characters added after the decision statement (denoted as L) k When L k The first preset length, i.e., the maximum length threshold L, is reached. max At that time, the current block C will be forcibly terminated. kThe construction of the block begins with the statement to be decided; it should be noted that if the statement to be decided simultaneously meets the criteria for inclusion (i.e., the synergy assessment passes) with the current block and the total length after merging does not exceed the maximum length threshold, then the statement to be decided is normally merged into the current block C. k middle.

[0108] Figure 3 This diagram illustrates a method for obtaining a synergy threshold according to an embodiment of the present disclosure, in conjunction with... Figure 3 As shown, obtaining the collaboration threshold provided in this embodiment may include the following steps:

[0109] S302, obtain the initial coordination threshold and the target coordination threshold.

[0110] Specifically, an initial collaboration threshold τ is set. init and the target synergy threshold τ max Among them, the initial degree of collaboration threshold τ init Choose a lower value to facilitate a more relaxed initial block merging; target synergy threshold τ max The higher value is selected for rigorous judgment in subsequent block division.

[0111] S304, using a preheating gradual adjustment strategy to determine the synergy threshold based on the initial synergy threshold and the target synergy threshold.

[0112] In this embodiment, a warm-up gradual adjustment strategy is adopted, and the number of blocks is set to K. wu That is, the first K wu The sub-block decision-making process serves as a "warm-up period," during which a gradual adjustment mechanism is used to raise the decision threshold from the initial synergy threshold τ. init Gradually increase to the preset maximum value, i.e., the target synergy threshold τ. max .

[0113] In some embodiments, a linear preheating progressive adjustment strategy is used to determine the synergy threshold based on an initial synergy threshold and a target synergy threshold.

[0114] Specifically, the linear warm-up formula is as follows:

[0115]

[0116] Where, τ init τ represents the initial synergy threshold. max K represents the threshold of target synergy. wu This represents the block number parameter, which is the upper limit of the number of iterations in the warm-up phase, and k represents the sequence number of the current iteration.

[0117] In this embodiment, the linear preheating gradual adjustment strategy ensures that the threshold increases uniformly, with a fixed increase in the threshold for each block decision. This stable change makes the system's changes during the warm-up phase predictable, enabling a smooth transition from the initial synergy threshold to the target synergy threshold, thus avoiding the significant impact of sudden changes in the synergy threshold on the block results.

[0118] In some embodiments, an exponential warm-up progressive adjustment strategy is used to determine the synergy threshold based on an initial synergy threshold and a target synergy threshold.

[0119] Specifically, the exponential warm-up formula is as follows:

[0120]

[0121] Where, τ init τ represents the initial synergy threshold. max K represents the threshold of target synergy. wu ρ represents the number of blocks, which is the upper limit of the number of iterations in the warm-up phase. k represents the sequence number of the current iteration, and ρ>1.

[0122] In this embodiment, since ρ > 1, the collaboration threshold increases relatively slowly in the initial stage of warm-up. This allows for high flexibility at the beginning, fully exploring the relationship between the statement to be decided and the first target text block, avoiding premature and strict merging judgments. As the number of block decisions increases, the threshold growth rate gradually accelerates, becoming steeper in the middle. This allows for a rapid increase in the threshold in the later stages, enabling a faster transition to strict judgment criteria and ensuring the consistency and stability of the final blocks.

[0123] In some embodiments, a cosine preheating progressive adjustment strategy is used to determine the synergy threshold based on an initial synergy threshold and a target synergy threshold.

[0124] Specifically, the cosine warm-up formula is as follows:

[0125]

[0126] Where, τ init τ represents the initial synergy threshold. max K represents the threshold of target synergy. wu This represents the block number parameter, which is the upper limit of the number of iterations in the warm-up phase, and k represents the sequence number of the current iteration.

[0127] In this embodiment, the cosine function has a smoothing characteristic, and using the cosine warm-up formula allows for a very smooth transition of the threshold throughout the warm-up phase. This smooth threshold change avoids abrupt changes in the threshold, making the segmentation results more stable throughout the process and reducing segmentation inconsistencies caused by threshold fluctuations. Furthermore, the smooth threshold change allows the system to finely adjust the segmentation strategy during the warm-up phase, gradually adjusting the merging criteria based on the actual situation of the statement to be decided or the first target text block, thereby optimizing the final segmentation effect and making the segmentation results more in line with actual needs.

[0128] In summary, this embodiment innovatively designs a dynamic adjustment strategy for the segmentation scoring threshold based on the differentiated requirements for content merging tolerance at different stages of a document: a "smooth warm-up" mechanism is implemented in several blocks at the beginning of the document, followed by maintaining a stable threshold. Specifically, for the opening section of the document, such as the document background introduction or research overview, which has significant information jumps, a lower initial merging threshold is used to avoid excessive fragmentation; as the block number increases, the threshold shows a gradient upward trend, gradually increasing the merging standard to ensure the coherence and consistency of subsequent paragraphs. This dynamic scheduling mechanism draws on the learning rate warm-up technique in the fine-tuning process of deep learning models, effectively solving the problem of fluctuating merging standards caused by fixed thresholds, and ultimately achieving a smoother and more controllable document segmentation effect.

[0129] Figure 4 This diagram illustrates a method for calculating the semantic change, perplexity change, and information entropy change of a first target text block when a statement to be decided is added to the first target text block, according to an embodiment of this disclosure. Figure 5 This illustration shows a flowchart of a method for calculating the semantic change, perplexity change, and information entropy change of a first target text block when a statement to be decided is added to the first target text block, according to an embodiment of this disclosure. Figure 4 and Figure 5 As shown in the embodiments of this disclosure, when the decision-making statements in the initial set of statements to be processed are added to the first target text block, the semantic change, perplexity change, and information entropy change of the first target text block may include the following steps:

[0130] S402, obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block.

[0131] In this embodiment, the first target text block C k Maintain three increments, namely the average pooling semantic vector V. k Average perplexity PPL k and average information entropy H kIt should be noted that the average pooling semantic vector V k Average perplexity PPL k and average information entropy H k It is a semantic vector v that is distinct from a single statement. i Confusion level ppl i and information entropy h i The semantic vector, perplexity, and information entropy corresponding to the current block as a whole. Specifically, refer to Figure 5 As shown, it includes several blocks that have been segmented, the current block, the statement to be decided, and the content to be segmented; among them, the current block is the target text block that needs to be segmented with the statement to be decided, also called the first target text block C. k Since the current text block is essentially the same as the first target text block, the current text block can also be denoted as C. k .

[0132] In some embodiments, obtaining the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block includes: obtaining the semantic vector, perplexity, and information entropy of each statement in the first target text block; calculating the mean of the semantic vector, perplexity, and information entropy of all statements in the first target text block respectively, to obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block.

[0133] In this embodiment, the first average pooling semantic vector V k First average perplexity PPL k and the first average information entropy H k It is calculated using the following formula:

[0134]

[0135] Where v(S), ppl(S), and h(S) are the methods for calculating the first target text block C, respectively. k The semantic vector of statement S, and statement S relative to the first target text block C. k The level of confusion and statement S relative to the first target text block C k The information entropy function or method, |C k | Indicates the first target text block C k Total number of statements.

[0136] S404, calculate the second average pooled semantic vector, the second average perplexity, and the second average information entropy after the first target text block is added to the statement to be decided.

[0137] In this embodiment, the semantic vector, perplexity, and information entropy of the statement to be decided are merged with the corresponding values ​​of each statement in the original text block. Then, based on all the merged statements, i.e., the original text block statements + the statement to be decided, the average value is recalculated to obtain the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the statement to be decided.

[0138] S406, calculate the semantic change between the second average pooling semantic vector and the first average pooling semantic vector, the change in perplexity between the second average perplexity and the first average perplexity, and the change in information entropy between the second average information entropy and the first average information entropy, respectively.

[0139] In this embodiment, the increments of semantic similarity, perplexity, and information entropy are calculated respectively. Specifically, the semantic change between the second average pooled semantic vector and the first average pooled semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy are calculated respectively. This includes: calculating the semantic similarity gain between the second average pooled semantic vector and the first average pooled semantic vector to obtain the semantic change; calculating the difference between the second average perplexity and the first average perplexity to obtain the perplexity change; and calculating the difference between the second average information entropy and the first average information entropy to obtain the information entropy change.

[0140] In some embodiments, the semantic change is calculated using the following formula:

[0141] Δ sim =cos(V k ,V′ k )-cos(V k V k )=cos(V k ,V′ k )-1

[0142] Where, Δ sim V represents the semantic change. k ′ is the second average pooling semantic vector, V k Let be the first average pooling semantic vector, where the second average pooling semantic vector represents the average pooling semantic vector of the first target text block after the decision statement is added to the first target text block, and the first average pooling semantic vector represents the average pooling semantic vector of the first target text block before the decision statement is added to the first target text block.

[0143] The change in confusion is calculated using the following formula:

[0144] Δ ppl =PPL′ k -PPL k

[0145] Where, Δ ppl V represents the change in confusion level. k ′ represents the second average perplexity, V k The first average perplexity is defined as follows: the second average perplexity represents the average perplexity of the first target text block after the decision statement is added to the first target text block, and the first average perplexity represents the average perplexity of the first target text block before the decision statement is added to the first target text block.

[0146] The change in information entropy is calculated using the following formula:

[0147] Δ h =H′ k -H k

[0148] Where, Δ h V′ represents the change in information entropy. k V is the second average information entropy. k Let be the first average information entropy, where the second average information entropy represents the average information entropy of the first target text block after the decision statement is added to the first target text block, and the first average information entropy represents the average information entropy of the first target text block before the decision statement is added to the first target text block.

[0149] Continue to refer to Figure 5 As shown, after calculating the increments of semantic similarity, perplexity, and information entropy, the collaborative decision function is then used to determine whether the statement to be decided should be merged into the current block.

[0150] In some embodiments, Figure 6 A flowchart illustrating a method for calculating a second average pooling semantic vector, a second average perplexity, and a second average information entropy, provided in this embodiment of the disclosure, is shown below. Figure 6 As shown, the calculation of the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the decision statement to the first target text block in this embodiment may include the following steps:

[0151] S602, under the condition that the time complexity is linearly related to the vector dimension, calculate the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the statement to be decided to the first target text block within the linear time complexity.

[0152] In this embodiment, the algorithm's running time is proportional to the vector's dimension (i.e., the vector's length or size), specifically O(d), where d is the vector's dimension. Linear time complexity means the required computation is completed in O(d) time.

[0153] Specifically, the first target text block and the statement to be decided have been represented as vectors or statistics of some form.

[0154] Average pooling semantic vector: Assume the semantic vector V of the first target text block k It is the average (dimension d) of all word vectors before the decision statement is added. The first target text block has |C k | word, v j This is the semantic vector of the statement to be decided. The second average pooling semantic vector is calculated using the following formula:

[0155]

[0156] As can be seen, the calculation in this embodiment is for the d-dimensional V k and v j Perform a linear combination, each dimension is O(1), for a total of d dimensions → O(d).

[0157] Average perplexity: Assuming the perplexity PPL of the first target text block k Based on the average of its words when it is not included in the decision statement, the first target text block has |C k | word, ppl j This is the perplexity of the statement to be decided. The second average perplexity is calculated using the following formula:

[0158]

[0159] Among them, PPL k and ppl j It is a known scalar, and the computation is O(1).

[0160] Average information entropy: Assuming the information entropy H of the first target text block k Based on the average of its words when it is not included in the decision statement, the first target text block has |C k | word, h j This is the information entropy of the statement to be decided. The second average information entropy is calculated using the following formula:

[0161]

[0162] Among them, H k and h j It is a known scalar, and the computation is O(1).

[0163] In this embodiment, after the statement to be decided is added to the first target text block, incremental updates are performed on the vector and statistics. As shown in the formula above, when adding a new statement, V′ can be updated in a linear time complexity of O(d) (d is the vector dimension). k ,PPL′ k ,H′k This eliminates the need to recalculate the entire block of historical data. Furthermore, when new statements are not yet incorporated, temporary calculation results can be discarded without affecting the original statistics, thus maintaining real-time performance in long documents and online scenarios.

[0164] In some embodiments, Figure 7 A flowchart illustrating a method for calculating a second average pooling semantic vector, a second average perplexity, and a second average information entropy, provided in this embodiment of the disclosure, is shown below. Figure 7 As shown in this embodiment, under the condition that the time complexity is linearly related to the vector dimension, calculating the second average pooled semantic vector, the second average perplexity, and the second average information entropy of the first target text block after adding the statement to be decided within linear time complexity may include the following steps:

[0165] S702 inputs the statement to be decided into the vectorization model and outputs the semantic vector of the statement to be decided.

[0166] In this embodiment, for semantic vectors, a vectorization model can be used to vectorize each statement, generating a high-dimensional dense vector, i.e., the semantic vector v corresponding to the statement to be decided. i .

[0167] In some embodiments, dimensionality reduction algorithms such as Principal Component Analysis (PCA), Singular Value Decomposition (SVD), UMAP (Uniform Manifold Approximation and Projection), and t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used to reduce the dimensionality of the semantic vector of the decision statement in high dimensions, thereby improving the efficiency of subsequent computation.

[0168] S704 takes the statement to be decided as input into the lightweight language model and outputs the conditional probability of each character in the statement.

[0169] In this embodiment, for the evaluation of perplexity and information entropy, a small-scale autoregressive language model (such as a distilled version of GPT-2Small or TinyLM) can be used to approximate the coherence measure of the original large model. Specifically, the implementation involves processing the statement S to be decided... i Append to its preceding block C k (If there is no preceding block, it is appended to the empty block.) After that, the model outputs the conditional probability P(w) of each token in sequence order. t |w <t C k ).

[0170] Small-scale autoregressive language models can be implemented using small-scale language models optimized through distillation or quantization techniques. For example, TinyLM, obtained by distillation from a large language model (such as GPT-2Medium), contains only 4 to 6 Transformer layers with a parameter size controlled below 50M, achieving millisecond-level response latency in a single forward computation. Furthermore, by implementing 8-bit weight quantization and inter-layer dynamic sparsity techniques, the inference latency can be further optimized to a processing level of 1 to 2 milliseconds per sentence.

[0171] In some embodiments, parallel batch processing is adopted, that is, in a multi-core or multi-GPU environment, a batch of statements to be decided are submitted simultaneously for inference by vectorized models and large language models. Through unified memory scheduling and tensor parallelism, high throughput is achieved, text segmentation efficiency is improved, and real-time performance is guaranteed.

[0172] S706, based on the perplexity function and the conditional probability of each character in the statement to be decided, calculates the perplexity of the statement to be decided.

[0173] In this embodiment, the perplexity is calculated using the following formula:

[0174]

[0175] Among them, T i For statement S i The number of tokens; the logarithmic probability term can be obtained by one forward inference, then averaging, negating and exponentializing.

[0176] S708 calculates the information entropy of the statement to be decided based on the information entropy function and the conditional probability of each character in the statement.

[0177] In this embodiment, the information entropy is calculated using the following formula:

[0178]

[0179] In some embodiments, only the Top-k probability terms (e.g., k=50) can be statistically analyzed to improve computational efficiency while ensuring that the loss of computational accuracy is controllable.

[0180] S710, calculate the second average pooling semantic vector based on the first average pooling semantic vector and the semantic vector of the statement to be decided.

[0181] In this embodiment, the second average pooling semantic vector can be incrementally updated by referring to the calculation method in S602 above, or it can be fully calculated based on the historical data of the entire block. This embodiment does not impose any restrictions on this. Generally speaking, choosing the calculation method in S602 above for incremental updates directly discards the temporary calculation results, does not affect the original statistics, and improves computational efficiency.

[0182] S712, calculate the second average perplexity based on the first average perplexity and the perplexity of the statement to be decided.

[0183] In this embodiment, the second average perplexity is similar to the second average pooling semantic vector, and will not be described in detail here.

[0184] S714, the second average information entropy is calculated based on the first average information entropy and the information entropy of the statement to be decided.

[0185] Similarly, the second average information entropy is similar to the second average pooling semantic vector, and will not be elaborated here.

[0186] In summary, to meet the real-time requirements of multi-indicator joint evaluation, this embodiment adopts an incremental statistical update mechanism and a lightweight language model distillation inference technique in its engineering implementation. Specifically, for core indicators such as vector representation, average perplexity, and entropy value of each text block, incremental calculation based on constant complexity is supported for rapid updates, effectively avoiding redundant calculations of the entire historical data. In the quality evaluation stage, a small autoregressive model that has undergone knowledge distillation compression and quantization is deployed to approximate the calculation of perplexity and entropy values. The time consumption of a single inference is controlled in the millisecond range, achieving second-level completion of text block processing. Its overall performance has significant advantages over existing schemes that rely on full-scale inference or redundant calculations.

[0187] In some embodiments, Figure 8 A flowchart of a method for filtering statements to be decided is provided in this embodiment of the disclosure, combined with Figure 8 As shown, the text segmentation method provided in this embodiment further includes the following steps:

[0188] S802, determine the similarity between candidate statements in the initial set of statements of the text to be processed and the first target text block.

[0189] In this embodiment, for each candidate statement S in the initial statement set j The Locality-Sensitive Hashing (LSH) algorithm can be used to partition the vector V. kCoarse-grained similarity ranking within the space. This method maps high-dimensional vectors to a low-dimensional bucket space using a hash function, ensuring that similar vectors fall into the same bucket with a high probability, thus significantly improving the efficiency of approximate nearest neighbor search for large-scale high-dimensional vector data. This strategy effectively reduces the computational cost and time complexity of subsequent precise similarity calculations while maintaining a high recall rate, thereby optimizing the allocation of computing resources while ensuring retrieval accuracy.

[0190] S804, the candidate statements with a similarity greater than a preset threshold are identified as the statements to be decided.

[0191] This embodiment employs an optimization strategy: only the top T% of highly relevant candidate statements ranked by cosine similarity are subjected to full perplexity and information entropy calculations. This approach effectively reduces the system's computational load, decreasing the frequency of subsequent perplexity and information entropy calculations by 70%–80%.

[0192] In this embodiment, by performing coarse-grained screening of candidate statements, the number of target statements requiring the invocation of high-cost language models is effectively reduced. It should be noted that this embodiment can be implemented in conjunction with steps S702 to S712. This solution can achieve a second-level response time for text block processing, and its overall performance is significantly improved compared to existing solutions that rely on full inference or repeated computation.

[0193] In some embodiments, Figure 9 This is a flowchart of a text segmentation method provided in an embodiment of the present disclosure, combined with... Figure 9 As shown, the text segmentation method provided in this embodiment further includes the following steps:

[0194] S902, determine whether the total length of the token count or character count of the last target text block in the target block is less than the second preset length.

[0195] S904, if the total length of the number of tokens or characters in the last target text block is less than the second preset length, determine the semantic similarity between the last target text block and the second-to-last target text block.

[0196] S906, when the semantic similarity is greater than the semantic similarity threshold, merge the last target text block with the second to last target text block.

[0197] In this embodiment, to address the issue of excessively short-tailed blocks that may occur after block segmentation, an intelligent merging strategy is implemented: when the length of a target text block is lower than a second preset length, i.e., a preset lower limit (denoted as L),... minWhen the semantic similarity with the previous target text block is higher than a set semantic similarity (denoted as δ), a merging operation is automatically triggered, effectively avoiding the generation of invalid small fragments. This strategy utilizes a tail-end residual processing mechanism after global block segmentation. When the length of the tail text block does not reach a preset lower limit and exhibits high semantic similarity with the adjacent preceding block, it is automatically merged into the preceding block, thereby eliminating the interference of meaningless small fragments on subsequent retrieval and generation tasks. Compared to traditional coarse processing that ignores the tail or a general merging approach, this fine-tuning strategy maintains the overall uniformity of the blocks while significantly reducing the additional storage overhead and retrieval call costs caused by extremely small fragments, demonstrating more precise targeting and refined control advantages.

[0198] Figure 10 This is a flowchart illustrating a specific text segmentation method provided in this embodiment of the disclosure, combined with... Figure 10 As shown, the text segmentation method provided in this embodiment includes the following steps:

[0199] S101 first performs basic segmentation on the input text based on punctuation and line breaks, and then performs segmentation checks and discrimination based on the lightweight SDB model.

[0200] S102: For each segmented statement, calculate the semantic vector, perplexity, and information entropy in sequence, and use algorithms such as PCA to reduce the dimensionality of the semantic vector.

[0201] S103, calculate the increments of average pooling semantic similarity, average perplexity and average information entropy before and after adding the statement to be decided to the current block, and then calculate the value of the collaborative evaluation function to determine the block boundary based on the threshold.

[0202] S104: During the block splitting process, it is always checked whether the length of each block exceeds the limit. If it does, the process ends directly and a new block splitting begins from the statement to be decided.

[0203] S105 employs incremental updates, efficient calculation of perplexity and information entropy using a small model, a candidate statement pre-screening strategy, and a parallel batch processing strategy to optimize efficiency.

[0204] S106 performs intelligent merging on the tail block.

[0205] The following section provides an example of a text segmentation method for dynamic segmentation of long online news texts. Figure 11 This is a flowchart illustrating a specific text segmentation method provided in this embodiment of the disclosure, combined with... Figure 11 As shown, the text segmentation method provided in this embodiment includes the following steps:

[0206] S111 first performs basic segmentation based on punctuation and line breaks for the input news text, merges short sentences, segments long sentences, and then uses the BiLSTM+CRF model for segmentation checking and discrimination.

[0207] Specifically, the sentence segmentation process for the original news text is as follows: First, the original news content is initially segmented based on periods (.), question marks (?), exclamation marks (!), and line breaks to generate candidate sentence sequences. Then, the candidate sequences are optimized: sentences shorter than 5 characters or 3 tokens are merged with their adjacent sentences; for excessively long sentences exceeding 200 tokens, recursively segmentation is performed based on commas or coordinating conjunctions (such as "and," "as well as," "or," etc.). Next, a BiLSTM+CRF architecture-based sentence boundary detection (SBD) model is used to perform secondary verification of the initial segmentation boundaries, focusing on correcting missegmentation caused by abbreviations (such as "etc."), numerical expressions (such as "1,000"), or complex symbol combinations, ultimately outputting a semantically verified and accurate sentence sequence.

[0208] S112: For each segmented news sentence, calculate the semantic vector, perplexity, and information level in sequence, and use algorithms such as PCA to reduce the dimensionality of the semantic vector.

[0209] In this embodiment, a pre-calculation operation of metrics is performed on the sentence sequence. The specific process is as follows: First, each sentence is processed sequentially, and its corresponding semantic vector, perplexity of the sentence in its corresponding block, and information entropy are calculated respectively. The semantic vector is generated using the bge-m3 model, and the PCA algorithm is used to reduce the original 1024-dimensional semantic vector to 256 dimensions. The perplexity is calculated using the TinyLM model, which obtains conditional probabilities by performing forward reasoning on the text concatenated with the current sentence and its corresponding block, and then calculates the final perplexity value based on the perplexity formula. The information entropy is calculated by statistically analyzing the Top-100 probability terms and calculating local uncertainty based on the information entropy formula.

[0210] Through the above process, the semantic vector of each statement, as well as its perplexity and information entropy relative to the corresponding block, can be obtained. The application of the PCA dimensionality reduction algorithm effectively improves the efficiency of subsequent semantic similarity calculations; the TinyLM model, as a small-scale model, can efficiently calculate the conditional probabilities required to calculate perplexity and information entropy; and the information entropy only counts the top-100 probability terms, a processing method that significantly improves the computational efficiency of these two metrics.

[0211] S113, calculate the average pooling semantic similarity, average perplexity and average information increment before and after adding the statement to be decided to the current block in sequence, then calculate the collaborative evaluation function value, and combine it with the adaptive threshold calculated based on the warm-up strategy to determine the block boundary.

[0212] In this embodiment, an incremental update strategy is used to dynamically construct text blocks. Specifically, for the current text block and the statement to be decided, the system calculates the following three key indicators in sequence: (1) the average pooled semantic vector after merging the statement into the current block; (2) the average perplexity; and (3) the average information entropy. Subsequently, by comparing and analyzing the changes in indicators before and after merging, the incremental values ​​of the three indicators are calculated respectively, and a comprehensive evaluation function is constructed based on these incremental values ​​to calculate the final collaborative evaluation score.

[0213] During the decision-making phase, the system determines the feasibility of merging a statement by comparing its collaborative evaluation score with a preset threshold. Specifically, the decision logic is as follows: if the collaborative evaluation score exceeds the set threshold and the length of the merged block does not exceed the maximum length limit, the statement to be decided is merged into the current block; otherwise, the statement is treated as the first sentence of a new block and formed as an independent block. This decision-making process is applied iteratively to all combinations of blocks and statements to be decided until all text blocks are constructed.

[0214] As an optimization solution, this embodiment provides two adaptive threshold adjustment mechanisms:

[0215] A dynamic threshold strategy based on block numbering is employed: an adaptive threshold is calculated using a linear or cosine warm-up function, enabling dynamic adjustment of the threshold as the block number increases. This strategy initially uses a more lenient threshold standard to maintain block flexibility, gradually increasing it to the system-set maximum threshold in later stages, thereby enhancing the consistency and stability of later block partitioning while ensuring block quality.

[0216] The parameter optimization mechanism based on reinforcement learning: The core parameters α, β, and γ in the collaborative evaluation function are dynamically updated online through reinforcement learning algorithms, so that the evaluation function can continuously adapt to the current task requirements, thereby constructing an optimized evaluation model that is more in line with the actual application scenario.

[0217] S114: During the block splitting process, it is always checked whether the length of each block exceeds the limit. If it does, the process ends directly and a new block splitting begins from the statement to be decided.

[0218] During the above block segmentation process, it is necessary to monitor in real time whether the current block length exceeds the maximum length limit. If it does, the block addition statement needs to be forcibly terminated and a new block needs to be started.

[0219] S115 employs incremental updates, TinyLM model 8-bit quantization and inter-layer dynamic sparsity processing, LSH algorithm for pre-screening candidate statements, and parallel batch processing strategies to optimize efficiency.

[0220] In this embodiment, based on the aforementioned block-based strategy, several efficiency optimization measures can be introduced to further improve system performance. Specifically, 8-bit quantization and inter-layer dynamic sparsity processing can be applied to the weights of the TinyLM model, thereby effectively reducing inference latency. Simultaneously, a candidate pre-screening mechanism is employed for optimization: for each decision statement S... j First, the Locality Sensitive Hash (LSH) algorithm is applied to the block vector space V. k The system performs coarse-grained similarity ranking, only calculating perplexity and information entropy for the top T% of high-quality candidate statements based on cosine similarity. This optimization strategy reduces the frequency of subsequent, more complex computational operations (perplexity and information entropy evaluation) by 70%-80%. Furthermore, by implementing a parallel batch processing scheme, in a multi-core CPU or GPU computing environment, batches of statements to be tested are simultaneously submitted to the vectorized model and the language model for collaborative inference. Leveraging a unified memory scheduling mechanism and tensor parallelism, the system ultimately achieves high throughput.

[0221] S116 performs intelligent merging for smaller blocks at the tail.

[0222] Finally, for small blocks whose tail length is less than the set minimum block length and whose semantic similarity to the preceding block is greater than a set threshold, an automatic merging strategy should be adopted to merge them into the preceding block. This can eliminate invalid fragments and improve the overall coherence of the blocks.

[0223] This embodiment demonstrates the following significant technical advantages in online news text segmentation applications: By integrating semantic similarity analysis, perplexity level (PPL) assessment, and information entropy calculation into a collaborative optimization mechanism, it effectively improves the semantic consistency of topics within segments, ensuring a tight logical connection of content. Simultaneously, it employs adaptive threshold dynamic adjustment and incremental update strategies to precisely suppress unnecessary overlap between text segments, significantly reducing storage resource consumption and the frequency of retrieval system calls. Furthermore, based on incremental computation technology and quantization model inference optimization, it meets the performance requirements of long text processing and high-concurrency real-time online scenarios while ensuring millisecond-level response latency. In addition, it supports online fine-tuning of weight parameters and thresholds or a dynamic update mechanism based on reinforcement learning, enabling the method to adapt to diverse document styles and downstream application needs, exhibiting high versatility and scenario adaptability.

[0224] The following section provides an example of text segmentation methods for the dynamic segmentation scenario of academic paper texts. Figure 12 This is a flowchart illustrating a specific text segmentation method provided in this embodiment of the disclosure, combined with... Figure 12As shown, the text segmentation method provided in this embodiment includes the following steps:

[0225] S121 first performs basic segmentation based on punctuation and line breaks for the input academic paper, and then performs segmentation checking and discrimination based on the lightweight Transformer-SDB model.

[0226] This embodiment uses a typical scientific research paper (including chapters such as title, abstract, introduction, methods, experiments, discussion, and conclusions) as the research object. First, it performs sentence-level segmentation on the complete paper text. The system uses heuristic rules based on punctuation (including periods, semicolons, question marks, and exclamation marks) and line breaks for initial text segmentation, and uses a regular expression matching mechanism to accurately filter text segments that are prone to misjudgment, such as English abbreviations like "eg" and "ie" and formula numbers. For the segmented text segments, the system executes a dual optimization strategy: text units shorter than 5 Chinese characters or 3 tokens are merged with their adjacent sentences; for extremely long sentences exceeding 150 tokens, recursive segmentation is performed based on grammatical markers such as commas, semicolons, and coordinating conjunctions. Finally, the system uses a trained and optimized lightweight Transformer-SBD (SentenceBoundary Detection) model to perform secondary semantic discrimination on all candidate breakpoints, effectively identifying and correcting abnormal segmentation caused by special text structures such as formula environments, figure / table titles, or bibliographic citations.

[0227] S122: For each segmented sentence in the paper, the semantic vector, perplexity, and information entropy are calculated sequentially, and the semantic vector is reduced in dimensionality using algorithms such as PCA.

[0228] For the segmented sequence of sentences in the paper, this embodiment performs the following calculation process in sequence: first, the semantic vector representation of each sentence is calculated; second, its perplexity in the language model is evaluated; and finally, the information entropy of the text is quantified. Subsequently, dimensionality reduction algorithms such as principal component analysis (PCA) are used to optimize the high-dimensional semantic vectors, thereby improving computational efficiency while ensuring representation accuracy.

[0229] Specifically, an original 768-dimensional semantic vector is generated based on the jina-embeddings-v3 pre-trained model, and then reduced to 256 dimensions using the PCA algorithm, significantly reducing subsequent computational complexity while maintaining the effectiveness of semantic representation. Next, a distilled version of the TinyLM model (fine-tuned based on the GPT-2Small architecture) is used for local coherence analysis: the current sentence is concatenated with its surrounding text block and input into the model; forward inference is used to calculate the perplexity index of the target sentence, quantifying its semantic coherence in the local context. Finally, the word probability distribution predicted by the model is truncated (only the top 80 candidate words with the highest probabilities are retained), and the average information entropy is calculated based on the truncated distribution to accurately capture local information abrupt changes in the sentence (such as disambiguation of polysemous words, abrupt shifts in detail, and other key semantic features). Ultimately, through the synergistic analysis of these three types of indicators, a multi-dimensional characterization of the sentence-level semantic features of academic texts is achieved.

[0230] S123, calculate the increments of average pooling semantic similarity, average perplexity, and average information entropy before and after adding the statement to be decided to the current block, and then calculate the value of the collaborative evaluation function to determine the block boundary based on the threshold.

[0231] In this embodiment, after the system enters the collaborative evaluation and decision-making stage, this method dynamically maintains three key historical indicators for the current text block: semantic representation vector based on average pooling, average perplexity, and average entropy. For each statement to be decided, the new indicator value generated after its inclusion in the current block is first calculated through simulation, thereby accurately calculating the incremental change between the old and new indicators. Subsequently, the collaborative evaluation function, based on pre-set standardized weight parameters, including semantic similarity weight coefficient, perplexity weight coefficient, and entropy weight coefficient, weights and fuses the above increments to generate a comprehensive evaluation score. This score must simultaneously meet two judgment conditions: first, it must exceed the adaptive dynamic threshold corresponding to the current text block (this threshold adopts a linearly increasing or cosine-gradual strategy, increasing stepwise with the block sequence number); second, the total number of tokens in the overall text after merging the statement to be decided must not exceed the system's preset maximum capacity limit. Only when both conditions are met can the target statement be merged into the current block; if neither condition is met, the statement to be decided is marked as the starting statement of the new block.

[0232] S124: During the block splitting process, it is always checked whether the length of each block exceeds the limit. If it does, the process ends directly and a new block splitting begins from the statement to be decided.

[0233] During the above block segmentation process, it is necessary to monitor in real time whether the current block length exceeds the maximum length limit. If it does, it is necessary to forcibly end the block addition statement and start a new block. In some embodiments, parallel constraints can also be applied to the number of characters and formulas to ensure the efficiency of single block loading.

[0234] S125 employs incremental updates, TinyLM model 8-bit quantization and inter-layer dynamic sparsity processing, LSH algorithm for pre-screening candidate statements, and parallel batch processing strategies to optimize efficiency.

[0235] In this embodiment, to improve processing throughput and response speed, the system first uses Locality Sensitive Hash (LSH) to perform coarse sorting of the sentences to be tested in the block vector space, and only performs perplexity and information entropy calculations on the top 30% of candidates with the highest similarity; and in the GPU environment, it calls jina-embeddings-v3 and the TinyLM model after 8-bit quantization and inter-layer dynamic sparsity processing in batches and in parallel; all block statistics adopt an incremental update strategy, and there is no need to backtrack the entire history when a new sentence is added to a block or discarded.

[0236] S126 performs intelligent merging for smaller blocks at the tail.

[0237] After the blocks are divided, this embodiment will automatically merge small blocks with a tail length of less than 50 tokens and a semantic similarity of more than 0.85 with the previous block to eliminate fragmented text blocks.

[0238] In some embodiments, the weight parameters in the collaborative evaluation function can be dynamically fine-tuned using online learning methods such as Bayesian optimization, based on user feedback and downstream retrieval / generation results, to adapt to different disciplinary styles and paper structure requirements.

[0239] This embodiment achieves high topic coherence, low redundancy, and millisecond-level block latency in academic paper segmentation scenarios, while also possessing the advantages of adaptability and adjustability. Through collaborative decision-making using multiple metrics, this method effectively ensures the coherence of topics within blocks, while also considering global and local consistency. By controlling both perplexity and information entropy, it promptly cuts off jumps in the argumentation process and controls information mutations. With the help of incremental updates, pre-screening, and parallel inference techniques, it ensures millisecond-level response to long paper paragraphs, improving processing efficiency. In addition, adaptive thresholds and online fine-tuning mechanisms enable this method to flexibly meet the diverse needs of different disciplines and paper structures, demonstrating stronger adaptability and robustness.

[0240] Based on the same inventive concept, this disclosure also provides a text segmentation device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0241] Figure 13 This diagram illustrates a text segmentation device according to an embodiment of the present disclosure, such as... Figure 13 As shown, the device includes: an acquisition module 131 for acquiring text to be processed; a segmentation module 132 for segmenting the text to be processed into an ordered sequence of statements to obtain an initial set of statements of the text to be processed; wherein the ordered sequence of statements includes multiple statements; a change calculation module 133 for calculating the semantic change, confusion change, and information entropy change of the first target text block when the statement to be decided in the initial set of statements of the text to be processed is added to the first target text block; a synergy calculation module 134 for determining the synergy between the statement to be decided and the first target text block based on the semantic change, the confusion change, and the information entropy change; and a block division module 135 for dividing the text to be processed into blocks based on the synergy between the statement to be decided and the first target text block to obtain target block text of the text to be processed.

[0242] In some embodiments, the change calculation module 133 is specifically used to: obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block; calculate the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the statement to be decided to the first target text block; and respectively calculate the semantic change between the second average pooled semantic vector and the first average pooled semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy.

[0243] In some embodiments, the change calculation module 133 is configured to: calculate the semantic similarity gain value between the second average pooling semantic vector and the first average pooling semantic vector to obtain the semantic change; calculate the difference between the second average perplexity and the first average perplexity to obtain the perplexity change; and calculate the difference between the second average information entropy and the first average information entropy to obtain the information entropy change.

[0244] In some embodiments, the change calculation module 133 is specifically used to: obtain the semantic vector, perplexity, and information entropy of each statement in the first target text block; calculate the mean of the semantic vector, perplexity, and information entropy of all statements in the first target text block respectively, and obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block.

[0245] In some embodiments, the change calculation module 133 is specifically used to: calculate the second average pooled semantic vector, the second average perplexity, and the second average information entropy of the first target text block after adding the statement to be decided within a linear time complexity, provided that the time complexity is linearly related to the vector dimension.

[0246] In some embodiments, the change calculation module 133 is specifically configured to: input the statement to be decided into a vectorized model and output the semantic vector of the statement to be decided; input the statement to be decided into a lightweight language model and output the conditional probability of each character in the statement to be decided; calculate the perplexity of the statement to be decided based on the perplexity function and the conditional probability of each character in the statement to be decided; calculate the information entropy of the statement to be decided based on the information entropy function and the conditional probability of each character in the statement to be decided; calculate the second average pooled semantic vector based on the first average pooled semantic vector and the semantic vector of the statement to be decided; calculate the second average perplexity based on the first average perplexity and the perplexity of the statement to be decided; and calculate the second average information entropy based on the first average information entropy and the information entropy of the statement to be decided.

[0247] In some embodiments, the synergy calculation module 134 is specifically used to: obtain the weights of the semantic change, the confusion change, and the information entropy change; and calculate the synergy between the statement to be decided and the first target text block based on the weights of the semantic change, the confusion change, and the information entropy change.

[0248] In some embodiments, the block segmentation module 135 is specifically used for: obtaining a collaboration degree threshold; when the collaboration degree is greater than the collaboration degree threshold, merging the statement to be decided into the first target text block to obtain a second target text block; when the collaboration degree is less than or equal to the collaboration degree threshold, generating a third target text block based on the statement to be decided.

[0249] In some embodiments, the segmentation module 135 is specifically used to: obtain an initial synergy threshold and a target synergy threshold; and determine a synergy threshold based on the initial synergy threshold and the target synergy threshold using a preheating progressive adjustment strategy.

[0250] In some embodiments, the segmentation module 135 is specifically configured to: determine a synergy threshold based on the initial synergy threshold and the target synergy threshold using a linear preheating progressive adjustment strategy; and / or, determine a synergy threshold based on the initial synergy threshold and the target synergy threshold using an exponential preheating progressive adjustment strategy; and / or, determine a synergy threshold based on the initial synergy threshold and the target synergy threshold using a cosine preheating progressive adjustment strategy.

[0251] In some embodiments, the segmentation module 135 is specifically used to: determine whether the total length of the number of tokens or characters in the second target text block is greater than or equal to a first preset length; if it is determined that the total length of the number of tokens or characters in the second target text block is less than the first preset length, merge the statement to be decided into the first target text block to obtain a second target text block; if it is determined that the total length of the number of tokens or characters in the second target text block is greater than or equal to the first preset length, generate a fourth target text block based on the statement to be decided.

[0252] In some embodiments, the segmentation module 135 is further configured to: determine whether the total length of the number of tokens or characters of the last target text block in the target segmented text is less than a second preset length; if the total length of the number of tokens or characters of the last target text block is less than the second preset length, determine the semantic similarity between the last target text block and the second-to-last target text block; when the semantic similarity is greater than the semantic similarity threshold, merge the last target text block and the second-to-last target text block.

[0253] In some embodiments, the segmentation module 132 is further configured to: determine the similarity between candidate statements in the initial statement set of the text to be processed and the first target text block; and determine the candidate statements with a similarity greater than a preset threshold as the statements to be decided.

[0254] It should be noted that the examples and application scenarios implemented by the modules in the above device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0255] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as a "circuit", "module" or "system".

[0256] Based on the same inventive concept, this disclosure also provides an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the text segmentation method described above by executing the executable instructions. Since the principle by which this electronic device solves the problem is similar to that of the above method embodiments, the implementation of this electronic device embodiment can refer to the implementation of the above method embodiments, and repeated details will not be described again.

[0257] The following reference Figure 14 To describe an electronic device 1400 according to such an embodiment of the present disclosure. Figure 14 The electronic device 1400 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0258] like Figure 14 As shown, the electronic device 1400 is manifested in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, and a bus 1430 connecting different system components (including storage unit 1420 and processing unit 1410).

[0259] The storage unit stores program code that can be executed by the processing unit 1410, causing the processing unit 1410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1410 can perform the following steps of the above method embodiment: obtaining text to be processed; segmenting the text to be processed into an ordered sequence of statements to obtain an initial set of statements of the text to be processed; wherein the ordered sequence of statements includes multiple statements; calculating the semantic change, perplexity change, and information entropy change of the first target text block when the statement to be decided in the initial set of statements of the text to be processed is added to the first target text block; determining the degree of synergy between the statement to be decided and the first target text block based on the semantic change, the perplexity change, and the information entropy change; and segmenting the text to be processed into blocks based on the degree of synergy between the statement to be decided and the first target text block to obtain the target block text of the text to be processed.

[0260] Storage unit 1420 may include readable media in the form of volatile storage units, such as random access memory (RAM) 14201 and / or cache memory 14202, and may further include read-only memory (ROM) 14203.

[0261] Storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0262] Bus 1430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0263] Electronic device 1400 can also communicate with one or more external devices 1440 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1400, and / or with any device that enables electronic device 1400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1450. Furthermore, electronic device 1400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1460. As shown, network adapter 1460 communicates with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0264] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0265] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described text segmentation methods. Since the principle by which this computer-readable storage medium solves the problem is similar to that of the above-described method embodiments, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0266] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0267] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0268] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0269] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0270] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the text segmentation method of any one of the above method embodiments. Since the principle by which this computer program product embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer program product embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0271] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0272] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0273] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0274] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A text segmentation method, characterized in that, The method includes: Get the text to be processed; The text to be processed is segmented into an ordered sequence of statements to obtain an initial set of statements in the text to be processed; wherein, the ordered sequence of statements includes multiple statements; Calculate the semantic change, confusion change, and information entropy change of the first target text block when the decision statement in the initial set of statements of the text to be processed is added to the first target text block. The degree of synergy between the statement to be decided and the first target text block is determined based on the semantic change, the perplexity change, and the information entropy change. The text to be processed is divided into blocks based on the degree of synergy between the statement to be decided and the first target text block, to obtain the target block text of the text to be processed.

2. The text segmentation method according to claim 1, characterized in that, The calculation of the semantic change, perplexity change, and information entropy change of the first target text block when the decision-making statements in the initial set of statements to be processed are added to the first target text block includes: Obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block; Calculate the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the decision statement to the first target text block; Calculate the semantic change between the second average pooling semantic vector and the first average pooling semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy, respectively.

3. The text segmentation method according to claim 2, characterized in that, Calculate the semantic change between the second average pooling semantic vector and the first average pooling semantic vector, the perplexity change between the second average perplexity and the first average perplexity, and the information entropy change between the second average information entropy and the first average information entropy, including: Calculate the semantic similarity gain between the second average pooling semantic vector and the first average pooling semantic vector to obtain the semantic change amount; Calculate the difference between the second average perplexity and the first average perplexity to obtain the change in perplexity; The difference between the second average information entropy and the first average information entropy is calculated to obtain the change in information entropy.

4. The text segmentation method according to claim 2, characterized in that, The step of obtaining the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block includes: Obtain the semantic vector, perplexity, and information entropy of each statement in the first target text block; The average semantic vector, perplexity, and information entropy of all statements in the first target text block are calculated respectively to obtain the first average pooled semantic vector, the first average perplexity, and the first average information entropy of the first target text block.

5. The text segmentation method according to claim 2, characterized in that, The calculation of the second average pooled semantic vector, the second average perplexity, and the second average information entropy after adding the decision statement to the first target text block includes: Under the condition that the time complexity is linearly related to the vector dimension, the second average pooled semantic vector, the second average perplexity, and the second average information entropy are calculated within the linear time complexity after the first target text block is added with the statement to be decided.

6. The text segmentation method according to claim 5, characterized in that, The calculation of the second average pooled semantic vector, second average perplexity, and second average information entropy after adding the decision statement to the first target text block within linear time complexity, under the condition that time complexity is linearly related to vector dimension, includes: The decision statement is input into a vectorized model, and the semantic vector of the decision statement is output. Input the statement to be decided into a lightweight language model and output the conditional probability of each character in the statement to be decided. The perplexity of the statement to be decided is calculated based on the perplexity function and the conditional probability of each character in the statement to be decided. The information entropy of the statement to be decided is calculated based on the information entropy function and the conditional probability of each character in the statement to be decided. The second average pooling semantic vector is calculated based on the first average pooling semantic vector and the semantic vector of the statement to be decided. The second average perplexity is calculated based on the first average perplexity and the perplexity of the statement to be decided. The second average information entropy is calculated based on the first average information entropy and the information entropy of the statement to be decided.

7. The text segmentation method according to claim 1, characterized in that, Determining the degree of synergy between the statement to be decided and the first target text block based on the semantic change, the perplexity change, and the information entropy change includes: Obtain the weights of the semantic change, the confusion change, and the information entropy change; The degree of synergy between the statement to be decided and the first target text block is calculated based on the weights of the semantic change, the perplexity change, and the information entropy change.

8. The text segmentation method according to claim 1, characterized in that, The step of segmenting the text to be processed based on the degree of synergy between the statement to be decided and the first target text block includes: Obtain the synergy threshold; When the degree of collaboration is greater than the degree of collaboration threshold, the statement to be decided is merged into the first target text block to obtain the second target text block; When the degree of collaboration is less than or equal to the degree of collaboration threshold, a third target text block is generated based on the statement to be decided.

9. The text segmentation method according to claim 8, characterized in that, The acquisition of the collaboration threshold includes: Obtain the initial collaboration threshold and the target collaboration threshold; The coordination threshold is determined based on the initial coordination threshold and the target coordination threshold using a preheating gradual adjustment strategy.

10. The text segmentation method according to claim 9, characterized in that, The step of determining the synergy threshold using a preheating gradual adjustment strategy based on the initial synergy threshold and the target synergy threshold includes: A linear preheating progressive adjustment strategy is used to determine the synergy threshold based on the initial synergy threshold and the target synergy threshold; and / or, A gradual adjustment strategy based on an exponential warm-up is used to determine a synergy threshold based on the initial synergy threshold and the target synergy threshold; and / or, The cosine preheating progressive adjustment strategy is used to determine the synergy threshold based on the initial synergy threshold and the target synergy threshold.

11. The text segmentation method according to claim 8, characterized in that, The step of merging the statement to be decided into the first target text block to obtain the second target text block includes: Determine whether the total length of the token count or character count of the second target text block is greater than or equal to the first preset length; If it is determined that the total number of tokens or characters in the second target text block is less than the first preset length, the statement to be decided is merged into the first target text block to obtain the second target text block; If it is determined that the total length of the number of tokens or characters in the second target text block is greater than or equal to the first preset length, a fourth target text block is generated based on the statement to be decided.

12. The text segmentation method according to claim 1, characterized in that, The method further includes: Determine whether the total length of the token count or character count of the last target text block in the target text block is less than the second preset length; If the total length of the number of tokens or characters in the last target text block is less than the second preset length, the semantic similarity between the last target text block and the second-to-last target text block is determined. When the semantic similarity is greater than the semantic similarity threshold, the last target text block is merged with the second-to-last target text block.

13. The text segmentation method according to claim 1, characterized in that, The method further includes: Determine the similarity between candidate statements in the initial set of statements of the text to be processed and the first target text block; Candidate statements with a similarity greater than a preset threshold are identified as statements to be decided.

14. A text segmentation device, characterized in that, The device includes: The acquisition module is used to acquire the text to be processed; The segmentation module is used to segment the text to be processed into an ordered sequence of statements to obtain an initial set of statements in the text to be processed; wherein the ordered sequence of statements includes multiple statements; The change calculation module is used to calculate the semantic change, confusion change, and information entropy change of the first target text block when the decision statement in the initial set of statements of the text to be processed is added to the first target text block. The collaboration degree calculation module is used to determine the collaboration degree between the statement to be decided and the first target text block based on the semantic change amount, the confusion degree change amount, and the information entropy change amount. The segmentation module is used to segment the text to be processed based on the degree of synergy between the statement to be decided and the first target text block, so as to obtain the target segmented text of the text to be processed.

15. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the text segmentation method of any one of claims 1 to 13 by executing the executable instructions.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text segmentation method according to any one of claims 1 to 13.

17. A computer program product comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the text segmentation method according to any one of claims 1 to 13.

Images