A long document segmentation method and device, computer equipment and medium

Abstract

The application discloses a long document segmentation method and device, computer equipment and medium. The method comprises the following steps: acquiring a long document to be segmented, and dividing the long document to be segmented into a structured document and an unstructured document according to a document type; for the structured document, performing structural segmentation on the structured document according to a preset rule to obtain a plurality of paragraph texts, and for the unstructured document, no structural segmentation is performed; using a text embedding model to perform semantic segmentation on each paragraph text or unstructured document to obtain a plurality of semantic text blocks; and using a large language model to perform model segmentation on each semantic text block according to a pre-set segmentation instruction set to obtain a segmentation result. According to the embodiment, the long document is segmented three times according to the document type, so that each type of document can be segmented. In the segmentation process, the text embedding model and the large language model are used for segmentation, so that the segmentation is more accurate and the semantic integrity of the text is ensured.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, computer device and medium for segmenting long documents. Background Technology

[0002] Natural Language Processing (NLP) is a branch of computer science and artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language, including speech and text data. Long document segmentation is a fundamental task in NLP, referring to the division of long texts into smaller, more easily processed and understood units, such as sentences, paragraphs, or topics.

[0003] In today's digital society, the number of long documents (such as contracts exceeding 10,000 words, research papers, meeting minutes, and multi-chapter reports) generated in areas such as enterprise office work, scientific research and innovation, and government affairs is growing exponentially. These long documents typically contain massive amounts of information, but current mainstream artificial intelligence technologies, such as Large Language Models (LLMs) and text analysis models, are limited by the length of the context window (e.g., most models can only process a maximum of 4,000-16,000 tokens at a time). Therefore, they cannot directly perform subsequent processing such as information extraction, semantic understanding, and question-and-answer interaction on complete long documents. Thus, it is necessary to effectively segment long documents, breaking them down into text units suitable for the model's processing capabilities. This segmentation task is crucial for improving text readability, information retrieval, text summarization, machine translation, and other fields, and has significant implications for downstream tasks such as knowledge base construction, retrieval database construction, and search engine query databases.

[0004] However, current technologies typically employ a uniform segmentation logic (such as using a fixed number of characters or fixed paragraph separators) to process all types of long documents, without distinguishing the structural features of the document. Furthermore, the input length limit for most mainstream large-scale models is generally within 10,000 characters, with only a few models having extremely long input windows. However, because these models are rarely trained on extremely long text outputs, they cannot effectively output all segmented paragraphs. Large-scale models struggle to handle extremely long text content such as books and manuals. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and medium for segmenting long documents, which can segment structured and unstructured long documents and ensure the accuracy of segmentation and the integrity of semantics.

[0006] In a first aspect, embodiments of this application provide a method for segmenting long documents, including:

[0007] Obtain the long document to be segmented, and divide the long document into structured documents and unstructured documents according to document type;

[0008] For the structured document, the structured document is segmented according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed.

[0009] The text embedding model is used to perform semantic segmentation on each paragraph text or unstructured document to obtain multiple semantic text blocks;

[0010] A segmentation instruction set is designed in advance for the large language model, and the large language model is used to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result.

[0011] Secondly, embodiments of this application provide a long document segmentation device, comprising:

[0012] The segmentation unit is used to obtain the long document to be segmented and divide the long document to be segmented into structured documents and unstructured documents according to the document type;

[0013] The structure segmentation unit is used to perform structural segmentation on the structured document according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed.

[0014] Semantic segmentation unit is used to perform semantic segmentation on each paragraph text or unstructured document using a text embedding model to obtain multiple semantic text blocks;

[0015] The model segmentation unit is used to pre-design a segmentation instruction set for the large language model, and use the large language model to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result.

[0016] Thirdly, embodiments of this application provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the program to implement the segmentation method provided in embodiments of this application.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, implements the segmentation method provided in embodiments of this application.

[0018] In this embodiment, a long document to be segmented is obtained and divided into structured and unstructured documents based on document type. For structured documents, structural segmentation is performed according to preset rules to obtain multiple paragraph texts. For unstructured documents, no structural segmentation is performed. A text embedding model is then used to perform semantic segmentation on each paragraph text or unstructured document to obtain multiple semantic text blocks. Finally, a large language model is used to segment each semantic text block according to a preset segmentation instruction set to obtain the segmentation result. Thus, when processing long documents, the long document is segmented according to document type to ensure that each type of document can be segmented. The long document is segmented three times, resulting in shorter text blocks. During the segmentation process, the text embedding model and the large language model are used for segmentation, making the segmentation more accurate and ensuring the semantic integrity of the text. Attached Figure Description

[0019] The technical solution and its beneficial effects will become apparent from the following detailed description of specific embodiments of this application, in conjunction with the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a method for segmenting long documents provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the structure of a long document segmentation device provided in an embodiment of this application.

[0022] Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] It should be noted that the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] This application provides a method, apparatus, computer device, and medium for segmenting long documents. The execution entity of the long document segmentation method can be the long document segmentation apparatus provided in this application, which can be implemented in hardware or software.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for segmenting long documents according to an embodiment of this application. The process may include:

[0027] In step 101, the long document to be segmented is obtained, and the long document to be segmented is divided into structured documents and unstructured documents according to the document type.

[0028] In this embodiment, a document is considered a long document when its length and content complexity make it impossible to read it completely, locate information, or extract data in one go. The form of long documents typically varies depending on the application domain. For example, in corporate office settings, long documents are usually annual reports or project plans; in academic research, they are usually academic papers or journal reviews. These long documents are not only lengthy but also have complex hierarchical structures. To quickly obtain the required information, long documents need to be segmented. However, since documents are mainly divided into structured and unstructured document types, the segmentation method is not entirely universal for different document types. Therefore, it is necessary to first divide long documents according to their document type so that different segmentation processes can be applied to structured and unstructured documents later.

[0029] In an optional embodiment, the method of obtaining the long document to be segmented includes at least one or more of the following: importing local files, pulling from cloud storage, or obtaining through an interface call.

[0030] In this embodiment, long documents are typically stored locally, in the cloud, or on an external system. Therefore, when it is necessary to obtain a long document to be segmented, it can be obtained in the following three ways depending on its storage location: First, if the long document is stored locally, it can be directly imported from the local directory of the computer / server. Second, if the long document is stored in the cloud, it can be pulled from a cloud storage tool or online collaboration tool. Note that if it is an online document, it needs to be converted to a format supported by the local storage. Third, if the long document is stored on an external system, the API interface provided by the external system is called, and authorization information such as API keys or tokens is obtained from the system administrator or platform developer center. An interface request is then sent to the external system to obtain the long document to be segmented. If multiple long documents need to be obtained, the interface can be called repeatedly.

[0031] In the specific implementation process, in order to improve the efficiency of acquisition, three acquisition methods can be combined. For example, if a small number of documents need to be acquired, local file import should be the main method. If there are long documents synchronized from the cloud, they should be pulled from the cloud storage and synchronized to the local machine first, and then imported into the local machine for processing. Alternatively, if long documents need to be acquired in batches, API calls should be the main method, with cloud storage retrieval as a secondary method, thereby reducing manual operations. Or, the three acquisition methods can be directly integrated to download long documents from the cloud and external systems to the local machine, and then imported into the local machine for processing.

[0032] In an optional embodiment, dividing the long document to be segmented into structured and unstructured documents according to document type includes:

[0033] If the long document to be segmented is a table-type document, a form-type document, a web page source code document with HTML tags, or a text document with hierarchical headings, then it is classified as a structured document.

[0034] If the long document to be segmented is a plain text document, it is classified as an unstructured document.

[0035] In this embodiment, the acquired long documents can be automatically classified through format feature recognition and rule matching. For example, for the input long document, format tags are first extracted. If there are explicit format tags, it is determined to be a structured document. Then, it is further determined whether it is a table document, form document, web page source code document, or text document based on the format tags. Specifically, if it is a table document, the format tag is .xlsx table format; if it is a form document, the format tag is fixed fields; if it is a web page source code document, the format tag is containing... HTML tags; if it is a text document, the formatting tags are level headings. If a long document has no formatting tags, such as a plain .txt file (i.e., a plain text document), a scanned OCR text without chapter divisions, and the text is mainly free-narrative (such as colloquial transcription, prose), then it is judged as an unstructured document.

[0036] The segmentation process can incorporate a lightweight rule engine and quickly complete the classification by keyword matching based on a preset format tag library (such as the format tags that the corresponding document type has), thereby dividing long documents into structured documents and unstructured documents.

[0037] In step 102, for the structured document, the structured document is segmented according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed.

[0038] In this embodiment, because structured documents have formatting tags, rules can be pre-set based on these tags to structurally segment the document, resulting in multiple paragraphs. However, unstructured documents, being primarily free text, lack a fixed format and clear hierarchy. Their semantic logic relies on contextual coherence. Forcibly segmenting them according to pre-set rules such as fixed word counts or line breaks can easily lead to sentence breaks or topic fragmentation (e.g., splitting the cause and effect descriptions of the same event into two parts), thus increasing the complexity of subsequent semantic segmentation. Therefore, unstructured documents are not structurally segmented, allowing them to retain their original text content and ensuring that semantic coherence is not compromised.

[0039] In practice, unstructured documents can be cleaned by removing extra spaces, standardizing line breaks, and correcting characters that are incorrectly recognized by OCR, without changing the order and logical relationship of the text content.

[0040] As can be seen from the foregoing embodiments, structured documents include various document types, such as table documents, form documents, web page source code documents, or text documents. Therefore, the formatting tags of different document types are segmented according to different rules.

[0041] In an optional embodiment, the step of structurally segmenting the structured document according to preset rules to obtain multiple paragraph texts includes:

[0042] If the long document to be segmented is a table-type document, the default rule is to segment it according to the rows / columns of the table.

[0043] If the long document to be segmented is a form document, the default rule is to segment it according to the form fields.

[0044] If the document type of the long document to be segmented is a web page source code document with HTML tags, the preset rule is to segment the structure according to the tag nodes.

[0045] If the long document to be segmented is a text document with hierarchical headings, the default rule is to segment the structure according to the heading level and / or paragraph marks.

[0046] In this embodiment, when the long document to be segmented is a table-type document (such as an Excel document), the preset segmentation rules need to ensure the correlation between row / column data. Therefore, the table's rows / columns can be structurally segmented in the following three ways. The first method is segmentation by column grouping, that is, based on the key dimension column in the table (such as "date", "region", etc.), all row data corresponding to the same dimension value are combined with the table header into a paragraph text. For example, segmenting the data table by the "date" column can yield paragraph texts such as "9.1 data", "9.2 data", etc., and each paragraph text contains the complete table header and all row data of that region. The second method is segmentation by the number of rows, that is, if the number of rows in the table is too large (such as more than 1000 rows), in order to avoid the individual paragraph texts being too large, segmentation is performed according to a preset row threshold (such as every 300 rows). The last paragraph text can be less than the threshold number of rows. After segmentation, each paragraph text still retains the complete table header, ensuring that the correspondence between the table header and data is not interrupted. The third method is a mixed row and column splitting. For complex tables containing merged cells or multi-level headers, first group by the top-level header column, and then split within each group by row threshold. For example, in a multi-level header table, if "2024 Data" contains sub-columns such as "First Quarter" and "Second Quarter", then first group by quarter column, and then split within each quarter group by row threshold to ensure a clear data hierarchy.

[0047] When the long document to be segmented is a form-type document (such as a contract), the preset segmentation rules can structurally segment the form fields in the following ways: The first method is segmentation by field module, which identifies the preset field labels in the form (such as "Party A Information", "Party B Information", "Subject Matter" or "Payment Method" in a contract), and treats each field label and its corresponding complete content (such as the name, address, contact information, etc. included in "Party A Information") as a paragraph text. The second method is segmentation by field association. For field groups with logical relationships (such as "Amount", "Payment Time", "Payment Account" included in "Payment Information"), the associated fields are treated as a whole as a paragraph text without splitting the associated fields, ensuring the integrity of the information.

[0048] When the long document to be segmented is a webpage source code document containing HTML tags, the preset segmentation rules can structurally segment the tag nodes in the following ways. The first method is segmentation by specified tag nodes, that is, randomly specifying one or more tag nodes, using the start and end positions of these tags as boundaries, and treating the complete content (including nested child tags and text) within the specified tags as a paragraph text. For example, segmenting each The content within the tag is split into independent paragraph text, retaining the content within the tag. , The second method is to segment by tag level, for multi-level nested tags (such as those containing child elements). It also includes You can specify a certain level tag (such as "section") as the segmentation node to ensure that the paragraph text corresponds to an independent semantic module in the webpage (such as a chapter or a list item).

[0049] When the long document to be segmented is a text document containing hierarchical headings, the preset segmentation rules can structurally segment the heading levels and / or paragraph marks in the following ways: The first method is segmentation by heading level, which sets a target heading level (e.g., a second-level heading). Starting from this heading level, the content up to the next heading of the same level constitutes a paragraph, including the heading text and all its subordinate content (e.g., third-level headings, body paragraphs). The second method is segmentation by paragraph mark, which, for documents without explicit headings but with paragraph breaks (e.g., blank lines), uses paragraph marks as boundaries, treating each independent paragraph as a single paragraph. If a single paragraph is too long (e.g., exceeding 500 words), it can be split into multiple sub-paragraphs using sentence boundaries (e.g., periods, exclamation marks) to avoid excessively long paragraphs. The third method is to split the headings and paragraphs together. For documents that have both hierarchical headings and paragraph marks, the heading level is prioritized for splitting. If the content under a heading contains only a single short paragraph (such as less than 200 words), the heading and paragraph are merged into one paragraph text. If there are multiple paragraphs, the heading and all its subordinate paragraphs are combined into one paragraph text to ensure the relevance between the heading and the content.

[0050] In 103, a text embedding model is used to semantically segment each paragraph text or unstructured document to obtain multiple semantic text blocks.

[0051] In this embodiment, each paragraph text or unstructured document is still a relatively large text block. To ensure that each paragraph text or unstructured document meets the input length requirements of the large language model when subsequently inputting it, each paragraph text or unstructured document needs to be further segmented. However, it is still necessary to ensure the semantic coherence of the segmented text blocks. Therefore, a text embedding model can be used to semantically segment each paragraph text or unstructured document, making the semantic information of each semantic text block coherent. A text embedding model is a machine learning model that converts text data (such as words, phrases, sentences, or paragraphs) into low-dimensional, dense vectors (i.e., "embedded vectors"). These vectors can capture the semantic and syntactic information of the text, making semantically similar texts closer together in the vector space, while semantically different texts are farther apart. By converting text into semantic vectors (the higher the vector similarity, the stronger the semantic association between the texts), the text embedding model can accurately identify semantically coherent units. In specific implementations, the text embedding model can use the BGE (BAAI General Embedding) model or an n-gram model.

[0052] In an optional embodiment, the text embedding model is used to semantically segment each paragraph text or unstructured document to obtain multiple semantic text blocks, including:

[0053] Locate all sentence-level punctuation marks in each paragraph text or unstructured document, and divide the text content in all paragraph texts or unstructured documents into N sentence-level texts based on the sentence-level punctuation marks;

[0054] Each sentence text is transformed into a corresponding text vector using a text embedding model.

[0055] The cosine similarity algorithm is used to calculate the similarity between every two adjacent text vectors, resulting in N-1 similarity values;

[0056] The N-1 similarity values ​​are sorted from high to low, and the lowest similarity value is used as the split point;

[0057] The N sentence texts are segmented according to the segmentation points, and the sentence texts located on the same side after segmentation are merged to obtain multiple semantic text blocks.

[0058] In this embodiment, initial segmentation is performed using sentence-level punctuation marks. Then, semantic similarity values ​​are used to find segmentation points. Text segmented at these points and located on the same side is merged to obtain multiple semantic text blocks. Sentence-level punctuation marks include periods, question marks, and exclamation marks. Using these punctuation marks as boundaries, all text content in each paragraph or unstructured document is split into N independent sentence-like texts. This step ensures that each sentence-like text is fully expressed, avoiding semantic breaks caused by splitting at commas or semicolons.

[0059] Next, a text embedding model is used to convert each sentence text into a corresponding text vector. That is, each sentence text generates one text vector, and N sentence texts generate N text vectors. For example, the text embedding model used is the BGE model. The BGE model performs semantic embedding representation on each sentence text. This step can convert each sentence text into a text vector with a length (dimension) of 512. Then, the similarity value between adjacent text vectors is calculated using cosine similarity. That is, a similarity value is calculated between every two text vectors, resulting in a total of N-1 similarity values. The cosine similarity calculation formula is shown below:

[0060] Here, A and B are two adjacent text vectors, and n is the dimension (length) of the text vector. Then, N-1 similarity values ​​are sorted from highest to lowest, and the position corresponding to the lowest similarity value is selected as the split point. This means that the semantic relationship between the sentence texts on either side of this position is the weakest. The N sentence texts are grouped according to the split point. Sentence texts on the same side are grouped together, and sentence texts within the same group are merged into a single semantic text block due to their close semantic relationship, ultimately resulting in multiple logically complete semantic text blocks.

[0061] In an optional embodiment, the step of segmenting each semantic text block according to the segmentation instruction using the large language model includes:

[0062] Each semantic text block is length checked, and it is determined whether the text length of each semantic text block exceeds the input length of the large language model.

[0063] If the length exceeds the limit, the corresponding semantic text block will continue to be semantically segmented using the text embedding model until the obtained semantic text block meets the input length of the large language model, and then it will be input into the large language model.

[0064] If the input does not exceed the limit, it is directly input into the large language model.

[0065] In this embodiment, to address the input length limitation of the large language model while maintaining semantic integrity, a length check is performed on each semantic text block before inputting it into the large language model. Specifically, the number of characters in each semantic text block is counted and compared with the input length of the large language model. If any semantic text block exceeds the input length, the text embedding model is invoked for further semantic segmentation. After this second semantic segmentation, new semantic text blocks are recombined, and the length check continues until the number of characters in the checked semantic text blocks meets the input length of the large language model. If the semantic text blocks do not exceed the input length of the large language model, they can be directly input into the large language model.

[0066] In step 104, a segmentation instruction set is designed in advance for the large language model, and the large language model is used to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result.

[0067] In this embodiment, since only the segmentation point with the lowest similarity value is selected for segmentation after semantic segmentation, the recombined semantic text blocks are still relatively large. Therefore, a large language model is used to perform more detailed segmentation on each semantic text block. The segmentation instruction is the core rule guiding the large language model on how to split each semantic text block. However, since the content of long documents of different document types varies, multiple segmentation instructions can be designed to form a segmentation instruction set. Each document type can be set with an independent segmentation instruction. This allows the model to identify the document type of the current semantic text block based on the attributes of the text content, and then call the appropriate segmentation instruction to segment the semantic text block using the model, thereby obtaining the segmentation result.

[0068] In an optional embodiment, the step of pre-designing a segmentation instruction set for a large language model and using the large language model to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result includes:

[0069] Design a segmentation instruction set based on text content attributes;

[0070] The large language model is used to determine the text content attributes of the current semantic text block, and a segmentation instruction in the segmentation instruction set is selected based on the text content attributes.

[0071] The current semantic text block is segmented according to the selected segmentation instruction, and the segmentation result is output in JSON format.

[0072] In this embodiment, the text content attribute can identify the text features of the current semantic text block, and then design a suitable segmentation instruction based on the text features. For example, if the text content attribute of the current semantic text block is a formula, then the text feature of the current semantic text block can be identified as an academic formula. The segmentation instruction can be designed as follows: use the formula start symbol as the segmentation starting point and the formula end symbol as the segmentation ending point to ensure that the formula as a whole is not split. If the text content attribute of the current semantic text block is a dialogue, then the text feature of the current semantic text block can be identified as a dialogue interaction. The segmentation instruction can be designed as follows: use the role identifier as the segmentation starting point and segment until the next role identifier to ensure that a single turn of dialogue is complete. If the text content attribute of the current semantic text block is a list (narrative text with serial numbers or symbols), then the text feature of the current semantic text block can be identified as a list structure. The segmentation instruction can be designed as follows: use the list serial number / symbol (such as "1.", "-", "●") as the segmentation starting point. Each list item is treated as a unit, forming an independent segmentation block. If the text content attribute of the current semantic text block is plain text, then the text feature of the current semantic text block can be confirmed as narrative text. The segmentation instruction can then be designed to use sentence-ending punctuation (".", "!", "?") as boundaries to ensure that the segmented blocks after model segmentation are complete sentences, avoiding cross-sentence splitting. Then, all semantic text blocks are segmented, and the segmentation results are output in JSON format.

[0073] In practical implementation, the large language model can use Alibaba's Tongyi Qianwen 2-72b. After the first two segments of a long document, the semantic text blocks typically have narrative text characteristics. Therefore, the segmentation command can call (or select) the segmentation command corresponding to the narrative text class and output the segmentation result in JSON format. For example, suppose the text content of the current semantic text block is:

[0074] Text refers to the written form of language. From a literary perspective, it is usually a sentence or a combination of sentences with a complete and systematic meaning. A text can be a sentence, a paragraph, or a discourse. Text generally includes broad and narrow definitions. In a broad sense, text refers to any discourse that is fixed in writing. In a narrow sense, text refers to the literary entity composed of language, representing a "work," which constitutes an independent and self-sufficient system relative to the author and the world.

[0075] After calling (or selecting) the segmentation command corresponding to the narrative text class for segmentation, the output segmentation result in JSON format is as follows:

[0076] {

[0077] Paragraph 1: "Text refers to the written form of language. From a literary perspective, it is usually a sentence or a combination of sentences that has a complete and systematic meaning."

[0078] Paragraph Two: "A text can be a sentence, a paragraph, or a discourse."

[0079] Paragraph 3: "Text usually contains both broad and narrow texts."

[0080] Paragraph 4: "In a broad sense, a text is any discourse that is fixed by writing."

[0081] Paragraph 5: "For the narrow sense of text, it is a literary entity composed of language and words, referring to the 'work,' which constitutes an independent and self-sufficient system relative to the author and the world."

[0082] }

[0083] At this point, the model segmentation process is complete.

[0084] In one embodiment, for a semantic text block, more than one segmentation instruction is typically used. When it is identified that the current semantic text block has multiple text content attributes, weight values ​​can be pre-set for each segmentation instruction in the segmentation instruction set, and segmentation is preferentially performed according to the segmentation instruction with the higher weight value. For example, the weight value of the segmentation instruction corresponding to the narrative text class can be set as the first weight value, and the weight value of the segmentation instruction corresponding to the academic formula class can be set as the second weight value, with the first weight value being greater than the second weight value. When it is identified that the current semantic text block has two text content attributes, segmentation is preferentially performed according to the segmentation instruction with the first weight value, that is, segmentation is performed according to the segmentation instruction corresponding to the narrative text class. In other implementation scenarios, the weight values ​​can be set independently, and which text content attribute has a higher weight value can also be set independently.

[0085] In another embodiment, the segmentation instruction set can also be set as an editable instruction set, which facilitates the addition, modification or deletion of segmentation instructions in the segmentation instruction set, so as to update the segmentation instruction set according to the usage needs of different users.

[0086] Please see Figure 2 , Figure 2 This is a schematic diagram of a long document segmentation device provided in an embodiment of this application. The long document segmentation device 200 includes: a segmentation unit 201, a structural segmentation unit 202, a semantic segmentation unit 203, and a model segmentation unit 204.

[0087] The segmentation unit 201 is used to obtain the long document to be segmented and divide the long document to be segmented into structured documents and unstructured documents according to the document type.

[0088] The structure segmentation unit 202 is used to perform structure segmentation on the structured document according to preset rules to obtain multiple paragraph texts, and not to perform structure segmentation on the unstructured document.

[0089] Semantic segmentation unit 203 is used to perform semantic segmentation on each paragraph text or unstructured document using a text embedding model to obtain multiple semantic text blocks;

[0090] The model segmentation unit 204 is used to pre-design a segmentation instruction set for the large language model, and use the large language model to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result.

[0091] In an optional embodiment, the segmentation unit 201 can be used to: obtain the long document to be segmented by at least one or more of the following methods: importing local files, pulling from cloud storage, or obtaining through interface calls.

[0092] In an optional embodiment, the segmentation unit 201 can be used to: if the document type of the long document to be segmented is a table-type document, a form-type document, a web page source code document with HTML tags, or a text document with hierarchical headings, then segment it into a structured document; if the document type of the long document to be segmented is a plain text document, then segment it into an unstructured document.

[0093] In an optional embodiment, the structure segmentation unit 202 can be used to: if the document type of the long document to be segmented is a table-type document, then the preset rule is to perform structure segmentation according to the rows / columns of the table; if the document type of the long document to be segmented is a form-type document, then the preset rule is to perform structure segmentation according to the form fields; if the document type of the long document to be segmented is a web page source code document with HTML tags, then the preset rule is to perform structure segmentation according to the tag nodes; if the document type of the long document to be segmented is a text document with hierarchical headings, then the preset rule is to perform structure segmentation according to the heading level and / or paragraph marks.

[0094] In an optional embodiment, the semantic segmentation unit 203 can be used to: locate all sentence-level punctuation marks in each paragraph text or unstructured document; segment the text content in all paragraph texts or unstructured documents into N sentence-type texts based on the sentence-level punctuation marks; convert each sentence-type text into a corresponding text vector using a text embedding model; calculate the similarity between every two adjacent text vectors using a cosine similarity algorithm to obtain N-1 similarity values; sort the N-1 similarity values ​​from high to low and take the lowest similarity value as the segmentation point; segment the N sentence-type texts according to the segmentation point, and merge the segmented sentence-type texts located on the same side to obtain multiple semantic text blocks.

[0095] In an optional embodiment, before the model segmentation unit 204, the following steps can be taken: perform length checks on each semantic text block and determine whether the text length of each semantic text block exceeds the input length of the large language model; if it exceeds, the corresponding semantic text block continues to be semantically segmented using the text embedding model until the obtained semantic text block meets the input length of the large language model, and then inputs it into the large language model; if it does not exceed, it is directly input into the large language model.

[0096] In an optional embodiment, the model segmentation unit 204 can be used to: design a segmentation instruction set for text content attributes; use the large language model to confirm the text content attributes of the current semantic text block, and select a segmentation instruction from the segmentation instruction set according to the text content attributes; segment the current semantic text block according to the selected segmentation instruction, and output the segmentation result in JSON format.

[0097] Please see Figure 3 This application provides a computer device 300, including a processor 320 and a memory, wherein the memory stores a computer program 332, and when the processor executes the program, it calls the computer program 332 stored in the memory to execute the long document segmentation method provided in this embodiment.

[0098] In this embodiment, the computer device 300 includes a processor 320, a memory, and a network interface 350 connected via a system bus 310. The memory may include a non-volatile storage medium 330 and internal memory 340. The non-volatile storage medium 330 stores an operating system 331 and a computer program 332. The computer program 332 includes program instructions that, when executed, cause the processor 320 to perform a long document segmentation method. The processor 320 provides computational and control capabilities to support the operation of the entire computer device 300. The internal memory 340 provides an environment for the operation of the computer program 332 in the non-volatile storage medium 330. When the processor 320 executes the computer program 332, it causes the processor 320 to perform a long document segmentation method.

[0099] This network interface 350 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 300 to which the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program 332 instructing related hardware. The computer program includes program instructions, and the computer program 332 can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0101] Therefore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the long document segmentation method provided in this embodiment.

[0102] In this embodiment, the processor 320 in the computer device loads the executable code corresponding to the processes of one or more applications into the memory according to the following instructions, and the processor 320 runs the applications stored in the memory, thereby achieving:

[0103] Obtain the long document to be segmented, and divide the long document into structured documents and unstructured documents according to document type;

[0104] For the structured document, the structured document is segmented according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed.

[0105] The text embedding model is used to perform semantic segmentation on each paragraph text or unstructured document to obtain multiple semantic text blocks;

[0106] A segmentation instruction set is designed in advance for the large language model. The large language model is then used to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result.

[0107] In an optional embodiment, when the processor 320 executes the process of obtaining the long document to be segmented, it may perform the following: the method of obtaining the long document to be segmented includes at least one or more of the following: importing local files, pulling from cloud storage, or obtaining through interface calls.

[0108] In an optional embodiment, when the processor 320 executes the step of dividing the long document to be segmented into structured and unstructured documents according to document type, it may perform the following: if the document type of the long document to be segmented is a table-type document, a form-type document, a web page source code document with HTML tags, or a text document with hierarchical headings, then it is classified as a structured document; if the document type of the long document to be segmented is a plain text document, then it is classified as an unstructured document.

[0109] In an optional embodiment, when the processor 320 performs the step of structurally segmenting the structured document according to preset rules to obtain multiple paragraph texts, it may perform the following: if the document type of the long document to be segmented is a table-type document, then the preset rule is to perform structural segmentation according to the rows / columns of the table; if the document type of the long document to be segmented is a form-type document, then the preset rule is to perform structural segmentation according to the form fields; if the document type of the long document to be segmented is a web page source code document with HTML tags, then the preset rule is to perform structural segmentation according to the tag nodes; if the document type of the long document to be segmented is a text document with hierarchical headings, then the preset rule is to perform structural segmentation according to the heading level and / or paragraph marks.

[0110] In an optional embodiment, when the processor 320 performs semantic segmentation of each paragraph text or unstructured document using a text embedding model to obtain multiple semantic text blocks, it may perform the following steps: locating all sentence-level punctuation marks in each paragraph text or unstructured document; segmenting the text content of all paragraph texts or unstructured documents into N sentence-type texts based on the sentence-level punctuation marks; converting each sentence-type text into a corresponding text vector using a text embedding model; calculating the similarity between every two adjacent text vectors using a cosine similarity algorithm to obtain N-1 similarity values; sorting the N-1 similarity values ​​from high to low and using the lowest similarity value as the segmentation point; segmenting the N sentence-type texts according to the segmentation point, and merging the segmented sentence-type texts located on the same side to obtain multiple semantic text blocks.

[0111] In an optional embodiment, before the processor 320 performs model segmentation of each semantic text block according to the segmentation instruction using the large language model, it may perform the following: check the length of each semantic text block and determine whether the text length of each semantic text block exceeds the input length of the large language model; if it exceeds, the corresponding semantic text block continues to be semantically segmented using the text embedding model until the obtained semantic text block meets the input length of the large language model, and then inputs it into the large language model; if it does not exceed, it is directly input into the large language model.

[0112] In an optional embodiment, the processor 320 executes the pre-designed segmentation instruction set for the large language model and uses the large language model to segment each semantic text block according to the segmentation instruction set. When obtaining the segmentation result, it can perform the following: design a segmentation instruction set for text content attributes; use the large language model to confirm the text content attributes of the current semantic text block and select a segmentation instruction from the segmentation instruction set according to the text content attributes; segment the current semantic text block according to the selected segmentation instruction and output it in JSON format to obtain the segmentation result.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the long document segmentation method above, which will not be repeated here.

[0114] The long document segmentation device provided in this application embodiment belongs to the same concept as the long document segmentation method in the above embodiment. Any of the methods provided in the long document segmentation method embodiment can be run on the long document segmentation device. For details of its implementation process, please refer to the long document segmentation method embodiment, which will not be repeated here.

[0115] It should be noted that, regarding the long document segmentation method of this application embodiment, those skilled in the art will understand that all or part of the process of implementing the long document segmentation method of this application embodiment can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, it can include the process of the long document segmentation method embodiment. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.

[0116] It is understood that in the specific implementation of this application, user information, such as application usage behavior data, logs and other related data, is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0117] For a long document segmentation device according to an embodiment of this application, its functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer device, such as a processor or memory, or in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0118] The foregoing has provided a detailed description of a method, apparatus, computer device, and medium for segmenting long documents according to embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application. 、

Claims

1. A method for segmenting long documents, characterized in that, include: Obtain the long document to be segmented, and divide the long document into structured documents and unstructured documents according to document type; For the structured document, the structured document is segmented according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed. The text embedding model is used to perform semantic segmentation on each paragraph text or unstructured document to obtain multiple semantic text blocks; The text embedding model is used to semantically segment each paragraph text or unstructured document, resulting in multiple semantic text blocks, including: Locate all sentence-level punctuation marks in each paragraph text or unstructured document, and divide the text content in all paragraph texts or unstructured documents into N sentence-level texts based on the sentence-level punctuation marks; Each sentence text is transformed into a corresponding text vector using a text embedding model. The cosine similarity algorithm is used to calculate the similarity between every two adjacent text vectors, resulting in N-1 similarity values; The N-1 similarity values ​​are sorted from high to low, and the lowest similarity value is used as the split point; The N sentence texts are segmented according to the segmentation points, and the sentence texts located on the same side after segmentation are merged to obtain multiple semantic text blocks; A segmentation instruction set is designed in advance for a large language model. The large language model is then used to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result. The pre-designed segmentation instruction set for the large language model, and the use of the large language model to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result, includes: Design a segmentation instruction set based on text content attributes; The large language model is used to determine the text content attributes of the current semantic text block, and a segmentation instruction in the segmentation instruction set is selected based on the text content attributes. The current semantic text block is segmented according to the selected segmentation instruction, and the segmentation result is output in JSON format.

2. The segmentation method according to claim 1, characterized in that, The method for obtaining the long document to be segmented includes at least one or more of the following: importing local files, pulling from cloud storage, or obtaining through API calls.

3. The segmentation method according to claim 1, characterized in that, The process of dividing the long document to be segmented into structured and unstructured documents based on document type includes: If the long document to be segmented is a table-type document, a form-type document, a web page source code document with HTML tags, or a text document with hierarchical headings, then it is classified as a structured document. If the long document to be segmented is a plain text document, it is classified as an unstructured document.

4. The segmentation method according to claim 3, characterized in that, The structured document is structurally segmented according to preset rules to obtain multiple paragraph texts, including: If the long document to be segmented is a table-type document, the default rule is to segment it according to the rows / columns of the table. If the long document to be segmented is a form document, the default rule is to segment it according to the form fields. If the document type of the long document to be segmented is a web page source code document with HTML tags, the default rule is to segment the structure according to the tag nodes. If the long document to be segmented is a text document with hierarchical headings, the default rule is to segment the structure according to the heading level and / or paragraph marks.

5. The segmentation method according to claim 1, characterized in that, Before using the large language model to segment each semantic text block according to the segmentation instruction set, the process includes: Each semantic text block is length checked, and it is determined whether the text length of each semantic text block exceeds the input length of the large language model. If the length exceeds the limit, the corresponding semantic text block will continue to be semantically segmented using the text embedding model until the obtained semantic text block meets the input length of the large language model, and then it will be input into the large language model. If the input does not exceed the limit, it is directly input into the large language model.

6. A long document segmentation device, characterized in that, include: The segmentation unit is used to obtain the long document to be segmented and divide the long document to be segmented into structured documents and unstructured documents according to the document type; The structure segmentation unit is used to perform structural segmentation on the structured document according to preset rules to obtain multiple paragraph texts; for the unstructured document, no structural segmentation is performed. Semantic segmentation unit is used to perform semantic segmentation on each paragraph text or unstructured document using a text embedding model to obtain multiple semantic text blocks; The semantic segmentation unit includes: locating all sentence-level punctuation marks in each paragraph text or unstructured document, and segmenting the text content in all paragraph texts or unstructured documents into N sentence-type texts based on the sentence-level punctuation marks; Each sentence text is transformed into a corresponding text vector using a text embedding model. The cosine similarity algorithm is used to calculate the similarity between every two adjacent text vectors, resulting in N-1 similarity values; The N-1 similarity values ​​are sorted from high to low, and the lowest similarity value is used as the split point; The N sentence texts are segmented according to the segmentation points, and the sentence texts located on the same side after segmentation are merged to obtain multiple semantic text blocks; The model segmentation unit is used to pre-design a segmentation instruction set for the large language model, and use the large language model to segment each semantic text block according to the segmentation instruction set to obtain the segmentation result; The model segmentation unit includes: a segmentation instruction set designed for text content attributes; The large language model is used to determine the text content attributes of the current semantic text block, and a segmentation instruction in the segmentation instruction set is selected based on the text content attributes. The current semantic text block is segmented according to the selected segmentation instruction, and the segmentation result is output in JSON format.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the segmentation method as described in any one of claims 1 to 5.

Images