Document segmentation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a document segmentation method and device, electronic equipment and storage medium. The method comprises: obtaining text content of a to-be-processed document; performing structured semantic understanding on the text content based on at least one large language model to generate a semantic segmentation strategy, wherein the semantic segmentation strategy comprises a semantic segmentation strategy including a document type and a semantic segmentation rule; performing adaptive semantic segmentation on the text content based on the semantic segmentation strategy through at least one large language model to obtain a primary text block sequence; performing post-processing on the primary text block sequence, and obtaining an optimized text block set meeting preset semantic integrity and length requirements by merging short text blocks and / or splitting long text blocks; and attaching context enhancement information to each text block in the optimized text block set and outputting a target text block set. The present disclosure can improve the semantic integrity and cross-scene adaptability of document segmentation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a document segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In natural language processing tasks such as retrieval enhancement and generation, document segmentation is a crucial preprocessing step, directly impacting the accuracy of subsequent text vectorization and retrieval recall. Existing document segmentation methods mainly include fixed-size segmentation, recursive segmentation, and semantic segmentation. Fixed-size segmentation typically cuts documents into equal-length segments based on a preset fixed number of characters or tags, its core being to control block length through mechanical equalization for uniform processing. Recursive segmentation first relies on natural delimiters within the document, such as paragraphs or chapters, for initial segmentation; if the resulting blocks still exceed a preset size, further recursive splitting is performed. Semantic segmentation converts sentences into vectors using an embedding model, then clusters them based on the similarity between vectors, merging semantically similar sentences into the same block.

[0003] However, the aforementioned traditional methods struggle to achieve a good balance between semantic integrity, structural adaptability, and implementation cost. Fixed-size chunking, by ignoring the inherent semantic and logical structure of the text, is prone to being cut off in the middle of sentences, clauses, or complete semantic units, leading to semantic fragmentation and affecting the accuracy of subsequent retrieval. Recursive chunking schemes heavily rely on the existing regular format and explicit delimiters of documents. For unstructured documents with chaotic formats and ambiguous structures, the selection of splitting points lacks a basis, easily resulting in over-splitting or under-splitting. While semantic chunking schemes focus on semantic coherence, they typically rely on trained, dedicated embedding models. When dealing with domain-specific documents, additional fine-tuning may be required to ensure accuracy, increasing implementation costs and development cycles. Furthermore, the text chunks generated by this method vary significantly in length, hindering subsequent unified vectorization and retrieval processes. Summary of the Invention

[0004] This disclosure provides a document segmentation method, apparatus, electronic device, and storage medium that can improve the semantic integrity and cross-scenario adaptability of document segmentation.

[0005] According to one aspect of this disclosure, a document segmentation method is provided, executed by a dedicated instruction in a processor, comprising: acquiring the text content of a document to be processed; performing structured semantic understanding on the text content based on at least one large language model to generate a semantic segmentation strategy, the semantic segmentation strategy including a semantic segmentation strategy containing document type and semantic segmentation rules; performing adaptive semantic segmentation on the text content according to the semantic segmentation strategy using at least one large language model to obtain a primary text block sequence; performing post-processing on the primary text block sequence by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets preset semantic integrity and length requirements; attaching context enhancement information to each text block in the optimized text block set and outputting a target text block set.

[0006] Optionally, according to the semantic segmentation strategy, adaptive semantic segmentation of the text content through the large language model includes: loading a corresponding semantic breakpoint recognition rule library according to the document type in the semantic segmentation strategy; locating the position of semantic breakpoints in the text according to the semantic breakpoint recognition rule library; wherein, the semantic breakpoint recognition rule library includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

[0007] Optionally, according to the semantic segmentation strategy, adaptive semantic segmentation of the text content through the large language model further includes: when multiple types of semantic breakpoints are identified near the same location, determining the position of the final semantic breakpoint according to a preset priority order; wherein the priority order is dynamically set according to the document type or user configuration.

[0008] Optionally, when segmenting or splitting long text into blocks, a dynamic overlapping region strategy is adopted. The dynamic overlapping region strategy includes: dynamically setting the size of the overlapping region between adjacent text segments according to the document type or structural complexity in the semantic segmentation strategy; or determining the start and end positions of the overlapping region according to the identified semantic unit boundaries to ensure that key contextual information is not fragmented.

[0009] Optionally, a multi-model collaborative approach is used to perform structured semantic understanding or adaptive semantic segmentation on the text content. The multi-model collaborative approach includes at least one of a division-of-labor collaborative approach and a voting decision approach. The division-of-labor collaborative approach involves performing structured semantic understanding and adaptive semantic segmentation separately using different models. The voting decision approach involves obtaining a primary text block sequence through fusion decision based on the initial segmentation strategies output by multiple models.

[0010] Optionally, the fusion decision in the voting decision method includes: selecting the segmentation point adopted by the most models as the selected segmentation point; or, performing weighted voting based on the confidence scores of each model to determine the primary text block sequence.

[0011] Optionally, merging short text blocks includes:

[0012] When multiple short text blocks to be merged belong to the same topic or logical paragraph semantically, they are merged first.

[0013] In addition to splicing the content and summary, the merging process also generates additional metadata describing the reasons for the merging or thematic coherence.

[0014] Optionally, splitting long text blocks also includes: after initial segmentation based on semantic tags, for semantic units that still exceed the length threshold, calling the large language model to perform secondary semantic segmentation.

[0015] Optionally, additional contextual enhancement information includes: extracting a document-level list of keywords or topic tags and associating them with relevant text blocks; and generating query or question pairs for retrieval of the text blocks.

[0016] According to one aspect of this disclosure, a document segmentation system is provided, comprising: a text acquisition module for acquiring the text content of a document to be processed; a structure understanding module for performing structured semantic understanding on the text content based on a large language model to generate a semantic segmentation strategy including document type and segmentation strategy; a semantic segmentation module for adaptively segmenting the text content according to the semantic segmentation strategy using a large language model to form a primary text block sequence; a post-processing module for post-processing the initial text block set by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets preset semantic integrity and length requirements; and an output enhancement module for attaching context enhancement information to each text block in the optimized text block set and outputting a target text block set.

[0017] Optionally, the semantic segmentation module includes a semantic breakpoint recognition unit, which is used to load a corresponding semantic breakpoint recognition rule base according to the document type in the semantic segmentation strategy; and to locate the position of semantic breakpoints in the text according to the semantic breakpoint recognition rule base; wherein, the semantic breakpoint recognition rule base includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

[0018] Optionally, the semantic breakpoint recognition unit is further configured to determine the final semantic breakpoint location according to a preset priority order when multiple types of semantic breakpoints are identified near the same location; wherein the priority order is dynamically set according to the document type or user configuration.

[0019] Optionally, the system further includes a model scheduling module, which manages multiple large language models and schedules corresponding models according to a multi-model collaboration method; wherein the multi-model collaboration method includes at least one of a division-of-labor collaboration method and a voting decision method.

[0020] The division of labor and collaboration involves using different models to perform structured semantic understanding and adaptive semantic segmentation respectively.

[0021] The voting decision-making method is based on the initial segmentation strategy output by multiple models, and the initial text block sequence is obtained through fusion decision.

[0022] According to one aspect of this disclosure, an electronic device is proposed, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the document segmentation method as described above.

[0023] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the document segmentation method described above.

[0024] This disclosure proposes a document segmentation method, apparatus, electronic device, and storage medium. The document segmentation method is executed by a dedicated instruction in a processor and includes: acquiring the text content of a document to be processed; performing structured semantic understanding on the text content based on at least one large language model to generate a semantic segmentation strategy, wherein the semantic segmentation strategy includes a semantic segmentation strategy that includes document type and semantic segmentation rules; performing adaptive semantic segmentation on the text content according to the semantic segmentation strategy using at least one large language model to obtain a primary text block sequence; performing post-processing on the primary text block sequence by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets preset semantic integrity and length requirements; attaching context enhancement information to each text block in the optimized text block set and outputting a target text block set. This disclosure utilizes a large language model to perform structured semantic understanding of documents and generate semantic segmentation strategies. This eliminates the reliance on fixed mechanical rules in the segmentation process, allowing the segmentation method to adaptively determine the segmentation method based on the actual semantic content of the document. This avoids the problems of traditional fixed-size block segmentation methods in the middle stages of sentences or logical units, ensuring the semantic integrity of text blocks. Post-processing optimizes the merging and splitting of text blocks, ensuring a balance between semantic integrity and length regularity in the output text blocks. Contextual enhancement information is added to each text block, so that the segmentation results not only retain the original content but also contain document-level semantic background. This effectively solves the technical problems of traditional segmentation methods, such as semantic fragility, strong dependence on format, and high implementation costs, without the need for additional model training. It provides high-quality text block input for subsequent tasks such as retrieval enhancement and generation.

[0025] Furthermore, by loading a semantic breakpoint recognition rule library corresponding to the document type, it can accurately locate structured markers such as chapter titles, numbered entries, and paragraph separators in the document as candidate splitting points. When multiple types of semantic breakpoints overlap, the final splitting point is determined according to the preset priority, avoiding conflicts and confusion in splitting positions. It can adapt to the structural features of different formats and types of documents, significantly improving the accuracy of splitting points.

[0026] Furthermore, when segmenting or splitting long texts into blocks, the size of the overlapping area between adjacent text segments can be dynamically set according to the document type or structural complexity, or the start and end positions of the overlapping area can be determined according to the semantic unit boundary. This ensures that key contextual information is not fragmented during the segmentation process, effectively solving the semantic coherence problem of cross-segment texts and avoiding information gaps caused by segmentation.

[0027] Furthermore, it supports two modes: collaborative division of labor and voting decision-making. The collaborative division of labor mode completes structured semantic understanding and adaptive semantic segmentation through different models, achieving task decoupling and specialized processing. The voting decision-making mode integrates the segmentation strategies output by multiple models and selects the segmentation point adopted by the most models or determines the segmentation result based on confidence-weighted voting. This can effectively reduce the output bias of a single model and improve the stability and accuracy of the segmentation results.

[0028] Furthermore, when merging short text blocks, multiple short text blocks that semantically belong to the same topic or logical paragraph are merged first. At the same time, additional metadata describing the reason for merging or thematic coherence is generated. This can preserve semantic coherence to the greatest extent while meeting length requirements and avoid semantic confusion caused by excessive merging.

[0029] Furthermore, after initial segmentation based on semantic tags, for semantic units that still exceed the length threshold, a large language model is invoked for secondary semantic segmentation. This ensures that the text block length meets the requirements while maintaining the semantic integrity within the ultra-long semantic units through secondary segmentation, thus avoiding semantic breaks caused by simple mechanical splitting.

[0030] Furthermore, by extracting document-level keyword lists or topic tags and associating them with relevant text blocks, and generating query or question pairs for retrieval of the text blocks, each text block can not only contain its own content, but also carry document-level semantic tags and retrieval enhancement information. This significantly improves the recall performance of text blocks in the retrieval enhancement generation system. It can be widely used in the segmentation and processing of various types of documents such as legal documents, policy documents, academic papers, and general reports, and has significant advantages such as short development cycle, strong compatibility, and easy integration.

[0031] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0032] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0033] Figure 1 This is a main flowchart of a document segmentation method according to an embodiment of the present disclosure;

[0034] Figure 2 This is a sub-flowchart of step S130 in one embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a document segmentation system according to an embodiment of the present disclosure;

[0036] Figure 4 This is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0038] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0039] Large language models (LLMs) are artificial intelligence models trained on large-scale corpora of language data. Through attention mechanisms, LLMs can capture the complex structure and semantics of language, thereby enabling text generation. LLMs are mainly used in scenarios such as text summarization, question answering, and machine translation.

[0040] Retrieval-Augmented Generation (RAG) is a content generation technology that combines search techniques with Large Language Models (LLM). It uses information retrieved from data sources as the basis for generating answers using LLM, thereby enhancing the accuracy and relevance of the generated content.

[0041] In related technologies, existing document segmentation methods mainly include fixed-size segmentation, recursive segmentation, and semantic segmentation. Fixed-size segmentation ignores the semantic structure of the text and is prone to truncation in the middle of sentences or logical units, leading to semantic fragmentation. Recursive segmentation relies heavily on the document's regular format and explicit delimiters, making it difficult to effectively handle unstructured documents with messy formats. Although semantic segmentation focuses on semantic coherence, it requires a dedicated embedding model, which may require additional fine-tuning for specific domain applications. This results in high implementation costs and significant differences in the length of the generated text blocks, which is not conducive to subsequent unified processing.

[0042] Based on this, this disclosure proposes a document segmentation method, apparatus, electronic device, and storage medium. It utilizes a large language model to perform structured semantic understanding of documents and generate semantic segmentation strategies. This eliminates reliance on fixed mechanical rules, allowing the segmentation method to adaptively determine the segmentation method based on the actual semantic content of the document. This avoids the problems of traditional fixed-size block methods in the middle stages of sentences or logical units, ensuring the semantic integrity of text blocks. Post-processing optimizes the merging and splitting of text blocks, ensuring a balance between semantic integrity and length regularity in the output text blocks. Contextual enhancement information is added to each text block, so that the segmentation results not only retain the original content but also contain document-level semantic background. This effectively solves the technical problems of traditional segmentation methods, such as semantic fragility, strong dependence on format, and high implementation costs, without requiring additional model training. It provides high-quality text block input for subsequent tasks such as retrieval enhancement and generation.

[0043] Overall Implementation of the Document Segmentation Method of the Embodiments of this Disclosure

[0044] This disclosure proposes a document segmentation method, applied to a document segmentation device, executed by a dedicated instruction in a processor, as described below. Figure 1 The document segmentation method includes:

[0045] Step S110: Obtain the text content of the document to be processed;

[0046] Step S120: Perform structured semantic understanding on the text content based on at least one large language model to generate a semantic segmentation strategy, wherein the semantic segmentation strategy includes a semantic segmentation strategy that includes document type and semantic segmentation rules;

[0047] Step S130: According to the semantic segmentation strategy, the text content is adaptively segmented using at least one large language model to obtain a primary text block sequence.

[0048] Step S140: Post-process the primary text block sequence by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets the preset semantic integrity and length requirements.

[0049] Step S150: Attach context enhancement information to each text block in the optimized text block set and output the target text block set.

[0050] In step S110, the text content of the document to be processed is obtained.

[0051] In this embodiment, the text content of the document to be processed is obtained through a document parser. Specifically, for original documents in different formats such as PDF, Word, and TXT, the corresponding parser is invoked to extract the plain text content, while removing redundant information such as page numbers, headers, footers, and table borders, and extracting metadata such as the document title and source path. For example, when processing a PDF document of the "Implementing Regulations of the Patent Law," the parsing component will extract the complete text content, including clause numbers, clause content, and chapter titles, and record the document title as "Implementing Regulations of the Patent Law," providing basic information for subsequent context enhancement. After parsing, a plain text string that can be processed by a large language model is obtained.

[0052] In step S120, the text content is subjected to structured semantic understanding based on at least one large language model to generate a semantic segmentation strategy, which includes a semantic segmentation strategy that includes document type and semantic segmentation rules.

[0053] In this embodiment, a Large Language Model (LLM) is invoked to perform structured semantic understanding of the document. Specifically, the first segment of the document (e.g., the first 4000 characters) is extracted as a structural analysis sample. Structured analysis prompts are constructed to guide the LLM in understanding the overall structure and type characteristics of the document. The LLM outputs structured information including document type, chunking strategy, recommended chunk size, chunking reasoning, and document summary. This information collectively constitutes the semantic segmentation strategy.

[0054] For example, by analyzing the structural features of the first 4,000 characters of the "Implementing Regulations of the Patent Law," such as "Chapter 1 General Provisions," "Article 1," and "Article 2," the large language model identifies the document type as "laws and regulations," recommends "clustering by clause theme," suggests a block size of "approximately 500 tokens," and generates a document summary: "This law stipulates the relevant implementing regulations for patent application, examination, authorization, and protection." This semantic segmentation strategy provides clear guidance for subsequent adaptive semantic segmentation.

[0055] In step S130, according to the semantic segmentation strategy, the text content is adaptively segmented using at least one large language model to obtain a primary text block sequence.

[0056] In this embodiment, based on the semantic segmentation strategy generated in the above steps, a large language model is invoked to perform adaptive semantic segmentation on the entire document. For long documents, segmentation is performed first, with a certain overlap between adjacent segments, prioritizing segmentation at semantically complete locations; then, segmentation prompts are constructed for each segment to guide the large language model to output multiple structured text blocks according to the recommended segmentation strategy. These structured text blocks constitute the primary chunk sequence, also known as the initial chunk set.

[0057] For example, the large language model identifies clause markers such as "Article 1" and "Article 2" as semantic breakpoints, dividing each clause and its related content into independent text blocks. Each text block contains an identifier (chunk_id), an abstract, and a content field. The abstract field summarizes the core content of the clause, such as "Article 1: These Regulations are formulated in accordance with the Patent Law, clarifying the legislative basis and scope of application"; the content field contains the complete text of the clause.

[0058] In some embodiments, when segmenting long texts or subsequently splitting them into long text blocks, a dynamic overlapping region strategy is adopted. This strategy includes: dynamically setting the size of the overlapping region between adjacent text segments based on the document type or structural complexity in the semantic segmentation strategy; or determining the start and end positions of the overlapping region based on the identified semantic unit boundaries to ensure that key contextual information is not fragmented. For example, when processing the Implementing Regulations of the Patent Law, due to the high structural complexity of legal documents and the strong logical connections between clauses, the overlapping region between adjacent segments is dynamically set to 300 characters (higher than the conventional 200 characters) to ensure that the contextual information of clauses across segments is fully preserved. Simultaneously, based on the identified semantic unit boundaries (such as the end position of "Article 1" and the start position of "Article 2"), the starting point of the overlapping region is set at the end summary of "Article 1," and the ending point is set at the beginning of "Article 2," so that the overlapping region precisely covers the transition content between the two clauses, thereby avoiding the loss of key information.

[0059] In some embodiments, a multi-model collaborative approach is used to perform structured semantic understanding or adaptive semantic segmentation on the text content. The multi-model collaborative approach includes at least one of a division-of-labor collaborative approach and a voting decision approach. The division-of-labor collaborative approach involves performing structured semantic understanding and adaptive semantic segmentation separately using different models. The voting decision approach involves obtaining a primary text block sequence through fusion decision based on the initial segmentation strategies output by multiple models.

[0060] In this embodiment, under a collaborative division of labor approach, different models perform structured semantic understanding and adaptive semantic segmentation respectively. For example, the first model is responsible for performing structured semantic understanding and generating a semantic segmentation strategy; the second model performs adaptive semantic segmentation based on this strategy, generating a preliminary text block sequence. This division of labor decoupling achieves task decoupling, allowing each model to focus on its strengths, thus improving processing efficiency and output quality.

[0061] In the voting-based decision-making approach, a preliminary text block sequence is obtained through a fusion decision based on the initial segmentation strategies output by multiple models. For example, the same text content is input into multiple large language models, each model outputs its own segmentation scheme, and then the final segmentation result is determined through a fusion decision. For instance, when processing the "Implementing Regulations of the Patent Law," three different models are called to perform segmentation respectively: Model A tends to segment by clause, Model B tends to segment by chapter, and Model C tends to segment by paragraph. After collecting the segmentation schemes from the three models, the final segmentation point position is determined through a voting-based decision-making mechanism. The fusion decision in the voting-based approach includes: selecting the segmentation point adopted by the most models as the selected segmentation point; or, using weighted voting based on the confidence scores of each model to determine the preliminary text block sequence. For example, if two out of the three models use "the first line" as the split point, then that split point is selected. If model A has a confidence score of 0.9, model B has 0.8, and model C has 0.7, the device calculates a weighted score for each candidate split point based on weighted voting and selects the split point with the highest score as the final split point. This multi-model collaborative mechanism effectively reduces the output bias of a single model and improves the stability and accuracy of the splitting results.

[0062] Specifically, see Figure 2 In one embodiment, step S130 includes the following steps.

[0063] Step S131: Load the corresponding semantic breakpoint recognition rule library according to the document type in the semantic segmentation strategy.

[0064] In this embodiment, based on the document type in the semantic segmentation strategy, a corresponding semantic breakpoint recognition rule base is loaded from a pre-set rule base set. The semantic breakpoint recognition rule base includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

[0065] For example, for legal documents such as the Implementing Regulations of the Patent Law, the device loads a rule base in the legal field, which includes regular expressions that match clause markers such as "Article [1234567890 ...

[0066] Step S132: Locate the position of the semantic breakpoint in the text according to the semantic breakpoint recognition rule base.

[0067] In this embodiment, based on the loaded semantic breakpoint recognition rule base, all semantic breakpoint positions that conform to the rules are scanned and located in the document text. For example, for the "Implementing Regulations of the Patent Law", the regular expression "[Article [1234567890 ...

[0068] In an optional embodiment, step S130 further includes the following steps.

[0069] Step S133: When multiple types of semantic breakpoints are identified near the same location, the final semantic breakpoint location is determined according to a preset priority order.

[0070] In this embodiment, when multiple semantic breakpoints of different types exist near the same location in the text, the final selected semantic breakpoint is determined according to a preset priority order. This priority order can be dynamically set based on document type or user configuration. For example, in the Implementing Regulations of the Patent Law, there might be a situation where "Chapter 1 General Provisions" and "Article 1" are simultaneously identified as semantic breakpoints in the same area. In this case, according to the priority settings of legal documents, "Article X," as a finer-grained segmentation unit, has a higher priority than "Chapter X." Therefore, "Article 1" is preferentially selected as the segmentation point, ensuring that each clause is segmented independently, rather than treating the entire chapter as a single large block. This priority mechanism ensures the rationality and consistency of the segmentation granularity.

[0071] In step S140, the primary text block sequence is post-processed by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets the preset semantic integrity and length requirements.

[0072] In this embodiment, the size of the primary text block sequence is optimized. For short text blocks with a length less than a first preset threshold (e.g., 100 tokens), they can be merged with adjacent short text blocks. Merging short text blocks includes: if multiple short text blocks to be merged semantically belong to the same topic or logical paragraph, they are merged first; during merging, in addition to splicing the content and summary, additional metadata describing the reason for merging or thematic coherence is generated. Taking the Implementing Regulations of the Patent Law as an example, if the explanatory description of a clause is divided into multiple excessively short text blocks, and these text blocks semantically describe the applicable circumstances of the clause, the device merges them into a complete text block. During merging, newlines are used to splice the content, semicolons are used to separate the summaries of each block, and additional metadata such as "Reason for merging: Explanation of the applicable circumstances of the same clause" is generated.

[0073] For long text blocks exceeding a second preset threshold (e.g., 800 tokens), the document is segmented based on structured tags, prioritizing segmentation at semantic breakpoints such as chapter titles and clause tags. After initial segmentation based on semantic tags, for semantic units still exceeding the length threshold, the large language model is invoked for secondary semantic segmentation. Taking the Implementing Regulations of the Patent Law as an example, if a clause is too long (e.g., containing multiple sub-items), the device first performs initial segmentation based on semantic tags (e.g., "Item (I)", "Item (II)"), splitting the long text block into multiple sub-units. If a sub-unit still exceeds the length threshold (e.g., 800 tokens), the device invokes the large language model for secondary semantic segmentation of the sub-unit, identifying its internal semantic boundaries (e.g., sentence boundaries, logical transition points) at a finer granular level, further splitting it into multiple sub-text blocks that meet the length requirements, and generating a new summary for each sub-block, in the format of "Original Summary (Part X)", such as "Article 1 (Part 1): Legislative Basis", "Article 1 (Part 2): Scope of Application". This mechanism ensures that each output text block meets the length requirement while maintaining semantic coherence. The set of text blocks obtained after post-processing is called the Optimized Chunk Set.

[0074] Step S150: Attach context enhancement information to each text block in the optimized text block set and output the target text block set.

[0075] In this embodiment, context enhancement information is added to each text block in the optimized text block set to improve its recall performance in the retrieval enhancement generation system. The added context enhancement information includes: extracting a document-level keyword list or topic tags and associating them with relevant text blocks; and generating query or question pairs for retrieval of the text blocks.

[0076] Specifically, a document title, document introduction, and a summary of the text block are added before the text block content, giving each text block document-level background information. Simultaneously, a list of document-level keywords or topic tags is extracted and associated with relevant text blocks. Furthermore, a query or question pair is generated for each text block for retrieval; for example, a clause text block is transformed into a question directly usable for retrieval, such as "What are the provisions regarding patent applications in the Implementing Regulations of the Patent Law?" Finally, the optimized set of target text blocks is output in JSON format for use by the downstream RAG system.

[0077] Taking the Implementing Regulations of the Patent Law as an example, the final output target chunk set contains each chunk with the following content format: "[Document Title] Implementing Regulations of the Patent Law\n[Document Introduction] This law stipulates the implementing regulations related to patent application, examination, authorization, and protection\n[Abstract] Article 1: Legislative Basis and Scope of Application\n[Content] Article 1 These Implementing Regulations are formulated in accordance with the Patent Law of the People's Republic of China..." Simultaneously, the keywords associated with this chunk are "patent application, legislative basis, scope of application," and the generated query is "What is the content of Article 1 of the Implementing Regulations of the Patent Law?"

[0078] The document segmentation method proposed in this disclosure is executed by a dedicated instruction in a processor. The implementation method includes: acquiring the text content of the document to be processed; performing structured semantic understanding on the text content based on at least one large language model to generate a semantic segmentation strategy, wherein the semantic segmentation strategy includes a semantic segmentation strategy that includes document type and semantic segmentation rules; performing adaptive semantic segmentation on the text content according to the semantic segmentation strategy using at least one large language model to obtain a primary text block sequence; performing post-processing on the primary text block sequence by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets preset semantic integrity and length requirements; attaching context enhancement information to each text block in the optimized text block set and outputting a target text block set. This disclosure utilizes a large language model to perform structured semantic understanding of documents and generate semantic segmentation strategies. This eliminates the reliance on fixed mechanical rules in the segmentation process, allowing the segmentation method to adaptively determine the segmentation method based on the actual semantic content of the document. This avoids the problems of traditional fixed-size block segmentation methods in the middle stages of sentences or logical units, ensuring the semantic integrity of text blocks. Post-processing optimizes the merging and splitting of text blocks, ensuring a balance between semantic integrity and length regularity in the output text blocks. Contextual enhancement information is added to each text block, so that the segmentation results not only retain the original content but also contain document-level semantic background. This effectively solves the technical problems of traditional segmentation methods, such as semantic fragility, strong dependence on format, and high implementation costs, without the need for additional model training. It provides high-quality text block input for subsequent tasks such as retrieval enhancement and generation.

[0079] Furthermore, by loading a semantic breakpoint recognition rule library corresponding to the document type, it can accurately locate structured markers such as chapter titles, numbered entries, and paragraph separators in the document as candidate splitting points. When multiple types of semantic breakpoints overlap, the final splitting point is determined according to the preset priority, avoiding conflicts and confusion in splitting positions. It can adapt to the structural features of different formats and types of documents, significantly improving the accuracy of splitting points.

[0080] Furthermore, when segmenting or splitting long texts into blocks, the size of the overlapping area between adjacent text segments can be dynamically set according to the document type or structural complexity, or the start and end positions of the overlapping area can be determined according to the semantic unit boundary. This ensures that key contextual information is not fragmented during the segmentation process, effectively solving the semantic coherence problem of cross-segment texts and avoiding information gaps caused by segmentation.

[0081] Furthermore, it supports two modes: collaborative division of labor and voting decision-making. The collaborative division of labor mode completes structured semantic understanding and adaptive semantic segmentation through different models, achieving task decoupling and specialized processing. The voting decision-making mode integrates the segmentation strategies output by multiple models and selects the segmentation point adopted by the most models or determines the segmentation result based on confidence-weighted voting. This can effectively reduce the output bias of a single model and improve the stability and accuracy of the segmentation results.

[0082] Furthermore, when merging short text blocks, multiple short text blocks that semantically belong to the same topic or logical paragraph are merged first. At the same time, additional metadata describing the reason for merging or thematic coherence is generated. This can preserve semantic coherence to the greatest extent while meeting length requirements and avoid semantic confusion caused by excessive merging.

[0083] Furthermore, after initial segmentation based on semantic tags, for semantic units that still exceed the length threshold, a large language model is invoked for secondary semantic segmentation. This ensures that the text block length meets the requirements while maintaining the semantic integrity within the ultra-long semantic units through secondary segmentation, thus avoiding semantic breaks caused by simple mechanical splitting.

[0084] Furthermore, by extracting document-level keyword lists or topic tags and associating them with relevant text blocks, and generating query or question pairs for retrieval of the text blocks, each text block can not only contain its own content, but also carry document-level semantic tags and retrieval enhancement information. This significantly improves the recall performance of text blocks in the retrieval enhancement generation system. It can be widely used in the segmentation and processing of various types of documents such as legal documents, policy documents, academic papers, and general reports, and has significant advantages such as short development cycle, strong compatibility, and easy integration.

[0085] Description of apparatus and devices according to embodiments of this disclosure

[0086] See Figure 3 This disclosure also provides a document segmentation system 300, including a text acquisition module 310, a structure understanding module 320, a semantic segmentation module 330, a post-processing module 340, and an output enhancement module 350.

[0087] The text acquisition module 310 is used to acquire the text content of the document to be processed.

[0088] The structure understanding module 320 is used to perform structured semantic understanding of the text content based on a large language model, and generate a semantic segmentation strategy that includes document type and segmentation strategy.

[0089] The semantic segmentation module 330 is used to adaptively segment the text content using a large language model according to the semantic segmentation strategy, forming a primary text block sequence.

[0090] The post-processing module 340 is used to perform post-processing on the initial text block set, and obtain an optimized text block set that meets the preset semantic integrity and length requirements by merging short text blocks and / or splitting long text blocks.

[0091] The output enhancement module 350 is used to attach context enhancement information to each text block in the optimized text block set and then output the target text block set.

[0092] In some embodiments, the semantic segmentation module 330 includes a semantic breakpoint recognition unit 331, which is used to load a corresponding semantic breakpoint recognition rule library according to the document type in the semantic segmentation strategy; and to locate the position of semantic breakpoints in the text according to the semantic breakpoint recognition rule library; wherein, the semantic breakpoint recognition rule library includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

[0093] In some embodiments, the semantic breakpoint recognition unit 331 is further configured to determine the position of the final semantic breakpoint according to a preset priority order when multiple types of semantic breakpoints are identified near the same location; wherein the priority order is dynamically set according to the document type or user configuration.

[0094] In some embodiments, the system further includes a model scheduling module 360, which manages multiple large language models and schedules corresponding models according to a multi-model collaboration method; wherein the multi-model collaboration method includes at least one of a division-of-labor collaboration method and a voting decision method.

[0095] The division of labor and collaboration involves using different models to perform structured semantic understanding and adaptive semantic segmentation respectively.

[0096] The voting decision-making method is based on the initial segmentation strategy output by multiple models, and the initial text block sequence is obtained through fusion decision.

[0097] The document segmentation system 300 disclosed herein is used to execute the document segmentation method as described in the above embodiments. Its specific processing procedure is the same as that of the document segmentation method described in the above embodiments, and will not be repeated here.

[0098] This disclosure also provides an electronic device 400, including:

[0099] At least one processor, and,

[0100] A memory that is communicatively connected to at least one processor; wherein,

[0101] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0102] The following is combined Figure 4 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450.

[0103] The processor 410 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0104] The memory 420 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410 using the document segmentation method of the embodiments of this disclosure.

[0105] The input / output interface 430 is used to implement information input and output;

[0106] Communication interface 440 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0107] Bus 450 transmits information between various components of the device (e.g., processor 410, memory 420, input / output interface 430, and communication interface 440);

[0108] The processor 410, memory 420, input / output interface 430 and communication interface 440 are connected to each other within the device via bus 450.

[0109] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the document segmentation method of the above embodiments, which will not be described again here.

[0110] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0111] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0112] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0113] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0117] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A document segmentation method, characterized in that, Executed by a dedicated instruction in the processor, including: Get the text content of the document to be processed; The text content is subjected to structured semantic understanding based on at least one large language model to generate a semantic segmentation strategy, which includes a semantic segmentation strategy that incorporates document type and semantic segmentation rules. Based on the semantic segmentation strategy, the text content is adaptively segmented using at least one large language model to obtain a primary text block sequence. The initial text block sequence is post-processed by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets the preset semantic integrity and length requirements. Attach context enhancement information to each text block in the optimized text block set and output the target text block set.

2. The document segmentation method according to claim 1, characterized in that, According to the semantic segmentation strategy, adaptive semantic segmentation of the text content using the large language model includes: Based on the document type in the semantic segmentation strategy, load the corresponding semantic breakpoint recognition rule library; Based on the semantic breakpoint recognition rule base, locate the position of the semantic breakpoint in the text; The semantic breakpoint recognition rule base includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

3. The document segmentation method according to claim 2, characterized in that, According to the semantic segmentation strategy, adaptive semantic segmentation of the text content using the large language model further includes: When multiple semantic breakpoints of the same type are identified in the vicinity of the same location, the final semantic breakpoint location is determined according to a preset priority order. The priority order is dynamically set based on document type or user configuration.

4. The document segmentation method according to claim 1, characterized in that, When segmenting or splitting long texts into blocks, a dynamic overlapping region strategy is adopted, wherein the dynamic overlapping region strategy includes: Based on the document type or structural complexity in the semantic segmentation strategy, the size of the overlapping area between adjacent text segments is dynamically set; Alternatively, based on the boundaries of the identified semantic units, the start and end positions of the overlapping regions can be determined to ensure that key contextual information is not fragmented.

5. The document segmentation method according to claim 1, characterized in that, The text content is subjected to structured semantic understanding or adaptive semantic segmentation using a multi-model collaborative approach. This multi-model collaborative approach includes at least one of a division-of-labor collaborative approach and a voting decision-making approach. The division of labor and collaboration involves using different models to perform structured semantic understanding and adaptive semantic segmentation respectively. The voting decision-making method is based on the initial segmentation strategy output by multiple models, and the initial text block sequence is obtained through fusion decision.

6. The document segmentation method according to claim 5, characterized in that, The fusion decision-making process in the voting decision-making method includes: Select the split point that is adopted by the most models as the selected split point; Alternatively, a weighted vote can be used based on the confidence scores of each model to determine the initial text block sequence.

7. The document segmentation method according to claim 1, characterized in that, Merging short text blocks includes: When multiple short text blocks to be merged belong to the same topic or logical paragraph semantically, they are merged first. In addition to splicing the content and summary, the merging process also generates additional metadata describing the reasons for the merging or thematic coherence.

8. The document segmentation method according to claim 1, characterized in that, Splitting long text blocks also includes: After initial segmentation based on semantic tags, for semantic units that still exceed the length threshold, the large language model is invoked for secondary semantic segmentation.

9. The document segmentation method according to claim 1, characterized in that, Additional contextual enhancement information includes: Extract a document-level list of keywords or hashtags and associate them with relevant text blocks; And generate query or question pairs for retrieval of text blocks.

10. A document segmentation system, characterized in that, include: The text acquisition module is used to acquire the text content of the document to be processed; The structure understanding module is used to perform structured semantic understanding of the text content based on a large language model, and generate a semantic segmentation strategy that includes document type and segmentation strategy; The semantic segmentation module is used to adaptively segment the text content using a large language model according to the semantic segmentation strategy, forming a primary text block sequence. The post-processing module is used to post-process the initial text block set by merging short text blocks and / or splitting long text blocks to obtain an optimized text block set that meets the preset semantic integrity and length requirements. The output enhancement module is used to attach context enhancement information to each text block in the optimized text block set and then output the target text block set.

11. The document segmentation system according to claim 10, characterized in that, The semantic segmentation module includes a semantic breakpoint recognition unit, which is used to load the corresponding semantic breakpoint recognition rule library according to the document type in the semantic segmentation strategy; and to locate the position of the semantic breakpoint in the text according to the semantic breakpoint recognition rule library. The semantic breakpoint recognition rule base includes rules based on regular expressions or keyword matching, used to identify at least one of chapter titles, numbered entries, paragraph separators, and domain-specific entity tags.

12. The document segmentation system according to claim 11, characterized in that, The semantic breakpoint recognition unit is also used to determine the position of the final semantic breakpoint according to a preset priority order when multiple types of semantic breakpoints are recognized in the vicinity of the same location. The priority order is dynamically set based on document type or user configuration.

13. The document segmentation system according to claim 10, characterized in that, The system also includes a model scheduling module, which manages multiple large language models and schedules the corresponding models according to a multi-model collaboration method; wherein the multi-model collaboration method includes at least one of a division-of-labor collaboration method and a voting decision method. The division of labor and collaboration involves using different models to perform structured semantic understanding and adaptive semantic segmentation respectively. The voting decision-making method is based on the initial segmentation strategy output by multiple models, and the initial text block sequence is obtained through fusion decision.

14. An electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program is executed by the processor to implement the document segmentation method as described in any one of claims 1 to 9.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the document segmentation method as described in any one of claims 1 to 9.

Images