Document segmentation method and device based on large language model, equipment and storage medium

Abstract

The invention provides a document segmentation method and device based on a large language model, equipment and a storage medium, and relates to the technical field of text processing. The method comprises the steps of inputting a to-be-segmented target document into a pre-trained large language model, and executing the following operations through the large language model: performing text layout analysis on the target document, and identifying titles and all paragraphs of each level in the target document; for each paragraph, inserting an associated title related to the paragraph in all titles into an initial position of the paragraph to obtain a corresponding target paragraph; and sorting all the target paragraphs based on the semantic similarity among all the target paragraphs, and determining a segmentation result of the target document based on all the sorted target paragraphs. By the adoption of the technical scheme, when document segmentation is carried out, semantic loss in the document segmentation process can be effectively reduced, and therefore the document segmentation effect is improved.

Description

Technical Field

[0001] This application relates to the field of text processing technology, and in particular to a document segmentation method, apparatus, device, and storage medium based on a large language model. Background Technology

[0002] With the surge in digital text information, document segmentation—breaking down long documents into smaller, manageable units (such as paragraphs or sentences)—is a crucial preprocessing step for downstream tasks such as information retrieval, knowledge extraction, and intelligent question answering. The quality of document segmentation directly impacts the accuracy and efficiency of subsequent applications, making it a fundamental link in the text processing chain.

[0003] However, traditional document segmentation methods primarily rely on physical delimiters such as line breaks and periods for mechanical segmentation. While this method is simple and fast, it fails to recognize the document's inherent logic and semantic relationships. For example, it might incorrectly treat a long sentence containing multiple independent clauses as a single unit, or it might abruptly cut off a complete argument. This format-based segmentation method is highly prone to contextual breaks and semantic incompleteness, i.e., the "semantic loss" problem, which severely affects the effectiveness of subsequent processing.

[0004] Therefore, how to reduce semantic loss during document segmentation and thus improve the segmentation effect is a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a document segmentation method, apparatus, device, and storage medium based on a large language model, which can effectively reduce semantic loss during document segmentation, thereby improving the document segmentation effect.

[0006] This application provides a document segmentation method based on a large language model, including: The target document to be segmented is input into a pre-trained large language model, and the following operations are performed through the large language model: Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document; For each paragraph, insert all related headings from the headings into the beginning of the paragraph to obtain the corresponding target paragraph; Based on the semantic similarity among all target paragraphs, all target paragraphs are sorted, and based on all sorted target paragraphs, the segmentation result of the target document is determined.

[0007] According to the document segmentation method based on a large language model provided in this application, the step of inserting all related headings from the paragraph into the beginning position of the paragraph includes: Based on the correspondence between all the headings and all the paragraphs, determine the hierarchical structure relationship between all the headings; According to the hierarchical structure, the associated headings related to the paragraph are inserted at the beginning of the paragraph.

[0008] According to the document segmentation method based on a large language model provided in this application, when there are multiple related headings associated with the paragraph, the step of inserting the related headings into the beginning position of the paragraph according to the hierarchical structure includes: Based on the hierarchical structure, determine the level corresponding to each of the multiple associated headings related to the paragraph; The associated headings are inserted sequentially at the beginning of the paragraph in descending order of hierarchy.

[0009] According to the document segmentation method based on a large language model provided in this application, determining the segmentation result of the target document based on all sorted target paragraphs includes: Determine the document segmentation length; For each of the sorted target paragraphs, the target paragraphs are segmented based on the document segmentation length to obtain the corresponding paragraph segmentation results; The segmentation result of the target document includes the segmentation result of each target paragraph after sorting.

[0010] According to the document segmentation method based on a large language model provided in this application, the step of segmenting the target paragraph based on the document segmentation length to obtain the corresponding paragraph segmentation result includes: The target paragraph is segmented based on the document segmentation length; If the segmentation position does not fall at the end of the target paragraph, then determine whether the segmentation position is the end of a complete sentence; Based on the judgment result, the segmentation of the target paragraph is obtained; The associated headings related to the target paragraph are inserted again into the target paragraph. The starting position of the remaining paragraph after the segmentation is cut off, and the inserted paragraph is used as the new target paragraph. The above steps are repeated until the segmentation position falls at the end position of the new target paragraph. The paragraph segmentation result includes the segmentation obtained from each segmentation.

[0011] According to the document segmentation method based on a large language model provided in this application, the step of obtaining segmented fragments for the target paragraph based on the judgment result includes: If the segmentation position is the end position of a complete sentence, then the segment between the start position of the target paragraph and the end position of the complete sentence is determined as the segmentation segment; If the segmentation position is not the end position of a complete sentence, then the end position of the complete sentence closest to the segmentation position is determined, and the segment between the start position of the target paragraph and the end position of the closest complete sentence is padded based on a preset padding character. The length of the padded segment is the document segmentation length, and the padded segment is determined as the segmentation segment.

[0012] According to the document segmentation method based on a large language model provided in this application, the step of performing text layout analysis on the target document to identify the headings at each level and all paragraphs in the target document includes: The target document is subjected to text layout analysis to identify key content in the target document, including font size, paragraphs, line numbers, and sentences. Identify the headings at each level based on the font size, line number, heading, and sentence. All paragraphs are identified based on the paragraph, the line number, and the sentence.

[0013] This application also provides a document segmentation device based on a large language model, comprising: The input unit is used to input the target document to be segmented into a pre-trained large language model; The processing unit is configured to perform the following operations using the large language model: Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document; For each paragraph, insert all related headings from the headings into the beginning of the paragraph to obtain the corresponding target paragraph; Based on the semantic similarity among all target paragraphs, all target paragraphs are sorted, and based on all sorted target paragraphs, the segmentation result of the target document is determined.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the document segmentation method based on a large language model as described in any of the preceding claims.

[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document segmentation method based on a large language model as described in any of the preceding claims.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the document segmentation method based on a large language model as described in any of the preceding claims.

[0017] The document segmentation method, apparatus, device, and storage medium based on a large language model provided in this application, when performing document segmentation, inputs the target document to be segmented into a large language model, identifies the headings at each level and all paragraphs in the target document through the large language model, inserts the related headings of all headings into the beginning position of the paragraph for each paragraph, and obtains the corresponding target paragraph; and sorts all target paragraphs based on the semantic similarity between all target paragraphs, and then determines the segmentation result of the target document based on all sorted target paragraphs. In this way, the combination of related headings and the semantic similarity between all target paragraphs is used for document segmentation, which fully considers the semantic information between different paragraphs, can effectively reduce semantic loss in the document segmentation process, thereby improving the document segmentation effect. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a document segmentation method based on a large language model, provided as an embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating a method for inserting a paragraph-related heading at the beginning of a paragraph, as provided in an embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating a method for determining the segmentation result of a target document based on all sorted target paragraphs, as provided in an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of a document segmentation device based on a large language model, provided in an embodiment of this application.

[0023] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0026] The technical solution provided in this application can be applied to scenarios such as information retrieval, knowledge extraction, and intelligent question answering. In these scenarios, the quality of document segmentation directly affects the accuracy and efficiency of subsequent applications. Traditional document segmentation methods mainly rely on physical delimiters such as line breaks and periods for mechanical segmentation. However, format-based segmentation methods are prone to causing contextual breaks and semantic incompleteness, i.e., the "semantic loss" problem, which seriously affects the effectiveness of subsequent processing.

[0027] Therefore, in order to effectively reduce semantic loss during document segmentation and improve the segmentation effect, this application provides a document segmentation method based on a large language model. During document segmentation, the target document to be segmented is input into the large language model. The large language model identifies the headings at each level and all paragraphs in the target document. For each paragraph, related headings from all headings are inserted at the beginning of the paragraph to obtain the corresponding target paragraph. Based on the semantic similarity between all target paragraphs, all target paragraphs are sorted. Then, based on the sorted target paragraphs, the segmentation result of the target document is determined. This method combines related headings and the semantic similarity between all target paragraphs for document segmentation, fully considering the semantic information between different paragraphs, effectively reducing semantic loss during the segmentation process, and thus improving the document segmentation effect.

[0028] It is understood that the execution entity of the document segmentation method based on the large language model provided in this application can be a computer, a server, or a specially designed text switching device or other electronic device, or it can be a document segmentation device based on the large language model installed in the electronic device. The document segmentation device based on the large language model can be implemented by software, hardware or a combination of both, and can be set according to actual needs.

[0029] The document segmentation method based on a large language model provided in this application will be described in detail below through several specific embodiments. It is understood that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0030] Figure 1 A flowchart illustrating a document segmentation method based on a large language model provided in this application embodiment is shown below. For example, please refer to... Figure 1 As shown, this document segmentation method based on a large language model may include: S101. Input the target document to be segmented into a pre-trained large language model, and perform the following operations through the large language model.

[0031] For example, the target document can be a Word document, a Portable Document Format (PDF) document, a Hyper Text Markup Language (HTML) document, etc., and the specific settings can be configured according to actual needs.

[0032] For example, when obtaining the target document to be split, it can be obtained directly from the local file system, or it can be obtained through an online document platform. Of course, it can also be obtained through other methods, such as obtaining the target document from a third-party database. The specific settings can be configured according to actual needs.

[0033] Understandably, large language models are natural language processing tools based on deep learning. They possess stronger context awareness capabilities, enabling them to handle complex nested structures, multi-page content, and documents of different styles. Therefore, they can intelligently identify document structure, semantic boundaries, and logical relationships, thereby achieving efficient and accurate document segmentation that better aligns with human reading and comprehension habits.

[0034] By training a large language model with a large corpus, it acquires rich language knowledge and contextual understanding capabilities, enabling it to process documents in various formats, such as Word documents, PDF documents, and HTML documents, and to reasonably divide texts based on structural features such as paragraphs, chapters, and titles, or semantic coherence.

[0035] For example, common large language models, such as the Generative Pre-trained Transformer (GPT) series of models, the Bidirectional Encoder Representations from Transformers (BERT) series of models, and the Large Language Model Meta AI (LLaMA), can all be used well for document segmentation.

[0036] After inputting the target document to be segmented into a pre-trained large language model, the target document can be segmented by the large language model, specifically including the following S102-S104, to obtain the segmentation result of the target document.

[0037] S102. Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document.

[0038] For example, when performing text layout analysis on a target document, it can be achieved in at least four possible ways: One possible approach is to use document structure features to perform text layout analysis on the target document, including: using structural features in the document, such as font size, bolding, indentation, etc., to identify headings and paragraphs.

[0039] In another possible approach, text layout analysis of the target document can be performed based on Optical Character Recognition (OCR) and layout analysis technologies. This includes combining OCR technology with layout analysis methods on the target document. For example, OCR tools can be used to extract document content, and then layout analysis techniques can be used to analyze the physical layout of the document to identify headings and paragraphs in the target document.

[0040] In another possible approach, text layout analysis of the target document can be performed using natural language processing methods. This includes analyzing the document's semantics and context using natural language processing techniques to identify headings and paragraphs. For example, deep analysis software can be used to extract headings and paragraphs from the target document.

[0041] In another possible approach, text layout analysis of the target document can be performed based on machine learning or deep learning methods, including: using machine learning or deep learning models (such as neural network models related to document structure analysis) to perform layout analysis on the document and extract titles and paragraphs from the target document.

[0042] It is understood that when performing text layout analysis on the target document, the embodiments of this application are only illustrated by the above four possible methods, but do not mean that the embodiments of this application are limited to these.

[0043] S103. For each paragraph, insert all related headings from the headings into the beginning of the paragraph to obtain the corresponding target paragraph.

[0044] The number of associated headings related to a paragraph can be one or more, depending on the actual needs.

[0045] When determining the related headings for a paragraph, you can judge their relevance based on the topic's position in the document, its hierarchical structure, and its distance from subsequent paragraphs. Generally, the paragraphs immediately following a heading are considered related to that heading; thus, for each paragraph, you can identify its related headings.

[0046] For any given paragraph, after identifying its related headings, these headings can be inserted as text at the beginning of the paragraph to form a new paragraph, which can be called the target paragraph.

[0047] S104. Based on the semantic similarity between all target paragraphs, sort all target paragraphs, and based on all sorted target paragraphs, determine the segmentation result of the target document.

[0048] For example, in the embodiments of this application, when determining the semantic similarity between different target paragraphs, the semantic similarity between different target paragraphs can be calculated based on the bag-of-words model, or based on word vector representation, or other methods can be used, such as based on a deep learning model, or based on a combination of clustering and ranking, etc. The specific settings can be made according to actual needs.

[0049] After determining the semantic similarity between all target paragraphs, the semantically similar target paragraphs can be sorted adjacently according to the magnitude of semantic similarity. This fully considers the semantic information between different paragraphs and can effectively reduce semantic loss during the document segmentation process. Then, based on all the sorted target paragraphs, the segmentation result of the target document is determined, which can effectively improve the document segmentation effect.

[0050] As can be seen, in this embodiment of the application, when performing document segmentation, the target document to be segmented is input into a large language model. The large language model identifies the headings at each level and all paragraphs in the target document. For each paragraph, the related headings in all headings are inserted into the beginning position of the paragraph to obtain the corresponding target paragraph. Based on the semantic similarity between all target paragraphs, all target paragraphs are sorted. Then, based on all sorted target paragraphs, the segmentation result of the target document is determined. In this way, the semantic similarity between related headings and all target paragraphs is used for document segmentation, which fully considers the semantic information between different paragraphs and can effectively reduce the semantic loss in the document segmentation process, thereby improving the document segmentation effect.

[0051] Based on the above Figure 1 In the illustrated embodiment, for example, in S102 above, when performing text layout analysis on the target document to identify the headings and all paragraphs at each level in the target document, the target document can first be analyzed to identify the key content in the target document, including font size, paragraphs, line numbers, and sentences; and based on font size, line numbers, headings, and sentences, each level can be identified; based on paragraphs, line numbers, and sentences, all paragraphs can be identified, thereby identifying the headings and all paragraphs at each level.

[0052] For example, in addition to font size, paragraphs, line numbers, and sentences, key content can also include other information such as document layout features and style attributes, which can be set according to actual needs. Document layout features include physical layout analysis of the document, such as visual elements like the spacing between headings and body text, indentation, and column division; style attributes include text formatting features such as bold, italics, and color. These styles are usually consistent in headings, helping to identify headings at different levels and all paragraphs.

[0053] For example, in the embodiments of this application, when identifying headings at different levels in a target document based on font size, line number, heading, and sentence, this can be achieved by combining the format features and semantic information of the target document, specifically including: First, extract the parts of the target document whose font size is significantly larger than the body text; these are often candidate titles. Second, determine whether the title is on a separate line or has a large gap between it and the body text by combining line numbers; usually, there will be blank lines or indentation changes before and after the title. Third, analyze whether the target document has typical title characteristics, such as conciseness, concentrated keywords, and lack of complete sentence structure (e.g., missing predicates or punctuation), and exclude non-title content such as list items and footnotes. Finally, a natural language processing model (such as BERT) can be used to perform semantic classification on the candidate titles to further distinguish between main titles, subtitles, or chapter titles, thereby identifying the various levels of headings in the target document.

[0054] For example, when identifying all paragraphs in a target document based on paragraphs, line numbers, and sentences, this can be achieved by analyzing the physical structure and semantic features of the text, specifically including: First, based on the line breaks or paragraph marks in the target document, such as those in Word... <w:p>The line spacing change in the label or PDF divides the text into physical paragraphs; secondly, the paragraph boundary is determined in combination with the line number, for example, the continuous line number change or the appearance of a blank line usually means the beginning of a new paragraph; finally, the internal integrity of the paragraph is further confirmed by using the sentence structure, for example, checking whether a complete subject-predicate-object structure or a punctuation mark end (such as a period, a question mark, etc.) is included, so as to filter out non-paragraph content (such as a title, a list item, etc.), thereby identifying all the paragraphs in the target document.

[0055] After identifying the hierarchical titles and all the paragraphs in the target document respectively, for each paragraph, the associated title related to the paragraph among all the titles can be inserted into the starting position of the paragraph, that is, S103 described above is performed.

[0056] For example, the specific implementation of inserting the associated title related to the paragraph among all the titles into the starting position of the paragraph in S103 described above can be referred to the embodiment shown in the following Figure 2

[0057] Figure 2 For the method of inserting the associated title related to the paragraph into the starting position of the paragraph provided in the embodiments of the present application, for example, please refer to the method flow diagram shown in the following Figure 2 S201, determining the hierarchical architecture relationship between all the titles based on the contrast relationship between all the titles and all the paragraphs.

[0058] The hierarchical architecture relationship refers to the tree structure formed by the upper and lower or parallel relationship between all the titles in the document, for example, the nesting relationship of "chapter-section-subsection", which reflects the logical organization framework of the document.

[0059] For example, in the embodiments of the present application, when the hierarchical architecture relationship between all the titles is determined based on the contrast relationship between all the titles and all the paragraphs, first, the nesting relationship between the titles can be initially inferred by analyzing the position of each title in the document (such as line number, font size, indentation) and the range of its associated paragraph, for example, the title with larger font or located at the beginning of the paragraph can be the upper title; secondly, the content similar titles are judged whether they belong to the same level or the same topic branch in combination with the semantic similarity or keyword matching of the titles; finally, the titles are organized into a hierarchical architecture by constructing a contrast relationship graph between the titles and the paragraphs and using a tree structure algorithm, such as Document Object Model (DOM) tree parsing or hierarchical clustering, wherein the upper title usually covers the content range of multiple lower titles and their associated paragraphs, thereby determining the hierarchical architecture relationship between all the titles.

[0060] S202, inserting the associated title related to the paragraph into the starting position of the paragraph according to the hierarchical architecture relationship.​​

[0061] For example, when there are multiple related headings associated with a paragraph, these multiple related headings can include headings at different levels, such as first-level headings, second-level headings, etc. When inserting the related headings at the beginning of the paragraph according to the hierarchical structure, the level corresponding to each of the multiple related headings can be determined based on the hierarchical structure. Then, the multiple related headings can be inserted into the beginning of the paragraph in descending order of level. This ensures that the document structure is clear and easy to read and understand.

[0062] For example, when determining the level of each associated heading among multiple related headings of a paragraph based on hierarchical relationships, the hierarchical position of each heading in the document structure can be found based on the hierarchical relationships, such as first-level heading, second-level heading, etc., where the level of a first-level heading is higher than that of a second-level heading, and the level of a second-level heading is higher than that of a third-level heading.

[0063] If the related headings associated with a paragraph include first-level headings, second-level headings, and third-level headings, then when multiple related headings are inserted into the beginning of the paragraph in descending order of hierarchy, the insertion result will be: first-level heading, second-level heading, third-level heading, and paragraph, thus obtaining the corresponding target paragraph.

[0064] Based on any of the above embodiments, the specific implementation of determining the segmentation result of the target document based on all sorted target paragraphs in S104 can be found below. Figure 3 The example shown.

[0065] Figure 3 This application provides a flowchart illustrating a method for determining the segmentation result of a target document based on all sorted target paragraphs, as illustrated in this embodiment. For example, please refer to... Figure 3 As shown, the method may include: S301. Determine the document segmentation length.

[0066] For example, when determining the document split length, at least three possible implementations can be included: In one possible implementation, the document segmentation length can be determined based on the file size. For example, a large document can be split into multiple smaller documents according to a specified size (e.g., 1MB each), which is typically suitable for processing binary documents or scenarios where document size control is required.

[0067] In another possible implementation, the document segment length can be determined based on semantic structure. For example, paragraphs, headings, and subheadings can be used to determine the document segment length to ensure that the segmented content is logically clear.

[0068] In another possible implementation, the document segmentation length can be determined based on the model input length limit. For example, in Natural Language Processing (NLP) tasks, models have explicit limitations on input length. For instance, the input window size of the BERT model is 512 tokens. Therefore, when segmenting documents, the document segmentation length can be determined based on the model input length limit to avoid exceeding the limit.

[0069] Based on the above description, after determining the document segmentation length, the following step S302 can be executed: S302. For each target paragraph after sorting, the target paragraph is segmented based on the document segmentation length to obtain the corresponding paragraph segmentation result.

[0070] The segmentation results of the target document include the segmentation results corresponding to each target paragraph after sorting.

[0071] For example, in this embodiment of the application, when segmenting a target paragraph based on the document segmentation length to obtain the corresponding paragraph segmentation result, the target paragraph can be segmented based on the document segmentation length; and it is determined whether the segmentation position falls at the end position of the target paragraph. If the segmentation position falls at the end position of the target paragraph, the segmentation result of the target document can be obtained directly; conversely, if the segmentation position does not fall at the end position of the target paragraph, it is necessary to further determine whether the segmentation position is the end position of a complete sentence, and based on the determination result, the segmented fragments for the target paragraph are obtained; then, the associated title related to the target paragraph is inserted into the target paragraph again, the starting position of the remaining part of the paragraph after the segmentation fragment is removed, and the inserted part of the paragraph is used as the new target paragraph, and the above steps are repeated until the segmentation position falls at the end position of the new target paragraph. The paragraph segmentation result includes the segmented fragments obtained from each segmentation.

[0072] As can be seen, in this embodiment of the application, when segmenting the target paragraph based on the document segmentation length, segmenting the target paragraph by determining whether the segmentation position falls at the end of the target paragraph and whether the segmentation position is the end of a complete sentence can improve the flexibility of text processing, ensure semantic integrity, optimize content display and user experience, enhance the efficiency and accuracy of data processing, and support the structured processing of complex documents.

[0073] For example, in the embodiments of this application, when determining whether the segmentation position falls at the end of the target paragraph, it can be determined based on the format, or based on the semantic content. Alternatively, it can be determined in other ways, such as by using a text processing tool. The specific settings can be made according to actual needs.

[0074] For example, when determining whether a segmentation point falls at the end of a target paragraph based on formatting, one can use first-line indentation or separators as indicators. Typically, Chinese paragraphs begin with two spaces indentation. If there is first-line indentation before the segmentation point, it may indicate the end of a paragraph. In document editing tools such as Word or WPS, paragraph breaks are explicit paragraph end markers. Therefore, one can determine whether a segmentation point falls at the end of a target paragraph by detecting paragraph breaks.

[0075] For example, when determining whether a segmentation point falls at the end of a target paragraph based on semantic content, one can analyze the semantic coherence of the text content to determine whether the segmentation point falls at the end of the target paragraph. If the topic of the content before and after the segmentation point changes significantly, it may be the end of the paragraph. Alternatively, one can analyze the context to determine whether the segmentation point is a natural semantic break, such as a logical relationship or a topic shift.

[0076] For example, when using text processing tools to determine whether the split point falls at the end of the target paragraph, such as string manipulation functions in Python (like endswith()), you can detect paragraph end marks to determine whether the split point falls at the end of the target paragraph.

[0077] For example, in the embodiments of this application, when determining whether the segmentation position is the end position of a complete sentence, it can be based on punctuation marks, or it can be based on grammatical structure. Alternatively, it can be determined in other ways, such as by using a text processing tool. The specific settings can be made according to actual needs.

[0078] For example, when determining whether a sentence segmentation point marks the end of a complete sentence based on punctuation, one can use sentence-end punctuation or special punctuation. Generally, the end of a sentence is accompanied by specific punctuation marks. Sentence-end punctuation: periods, question marks, and exclamation marks are the most common markers of sentence endings; their appearance usually indicates the end of a complete sentence. Special punctuation: some sentences may end with ellipses or dashes; these can also serve as a reference for sentence endings, but further confirmation requires context. Therefore, sentence-end punctuation or special punctuation can be used to determine whether a sentence segmentation point marks the end of a complete sentence.

[0079] For example, when determining whether a segmentation point is the end of a complete sentence based on grammatical structure, a complete sentence usually contains a subject, predicate, and object, and can express a complete meaning. If the content before and after the segmentation point satisfies this structure, it may be the end of the sentence. Usually, there are no incomplete grammatical structures (such as missing predicates or objects) at the end of a sentence, which is also an important basis for judging whether a sentence is complete, thus determining whether the segmentation point is the end of a complete sentence.

[0080] For example, when using text processing tools to determine whether the segmentation position is the end of a complete sentence, such as string manipulation functions in Python (like endswith()), it can be used to detect sentence end markers to determine whether the segmentation position is the end of a complete sentence.

[0081] Based on the above description, we can determine whether the segmentation position is the end of a complete sentence. After obtaining the determination result, we can obtain the segmentation fragments for the target paragraph, thereby achieving the segmentation of the target paragraph.

[0082] For example, in the embodiments of this application, when obtaining the segmented fragments for the target paragraph based on the judgment result, two possible scenarios may be included: In one possible scenario, if the segmentation position is the end position of the complete sentence, the segment between the beginning position of the target paragraph and the end position of the complete sentence can be directly identified as the segmentation segment. This can ensure the integrity of the obtained segmentation segment, which can not only improve information understanding and optimize the performance of NLP tasks, but also improve user experience and enhance content management and retrieval efficiency.

[0083] In another possible scenario, if the segmentation position is not the end of the complete sentence, in order to ensure the integrity of the obtained segment, the end position of the complete sentence closest to the segmentation position can be further determined. Based on the preset padding character, the segment between the start position of the target paragraph and the end position of the closest complete sentence is padded. The length of the padded segment is the document segmentation length. The padded segment is determined as the segmentation segment. This not only ensures the integrity of the obtained segmentation segment, but also makes the length of each segmentation segment consistent.

[0084] For example, the preset padding character can be a symbol-type padding character, a placeholder-type padding character, or a special marker-type padding character, etc., which can be set according to actual needs.

[0085] Among them, symbolic padding characters include underscores, asterisks, and question marks; placeholder padding characters include [to be completed] and <missing>; and special marker padding characters include [MASK] (common in BERT models) or [UNK], which can be set according to actual needs.

[0086] It should be noted that, in this embodiment of the application, when inserting the associated headings related to the target paragraph back into the starting position of the remaining paragraphs after the segmentation is cut off, multiple associated headings can be inserted into the starting position of the remaining paragraphs in descending order of hierarchy. The specific implementation is similar to the above-described implementation of inserting the associated headings related to the paragraphs into the starting position of the paragraphs according to the hierarchical structure. Please refer to the above description. Here, this embodiment of the application will not repeat the details.

[0087] Based on the above description, for any target paragraph, by determining whether the segmentation position falls at the end of the target paragraph and whether the segmentation position is the end of a complete sentence, the target paragraph can be segmented multiple times to obtain multiple segmented pieces. These multiple segmented pieces are the paragraph segmentation results of the target paragraph.

[0088] The document segmentation device based on a large language model provided in this application is described below. The document segmentation device based on a large language model described below can be referred to in correspondence with the document segmentation method based on a large language model described above.

[0089] Figure 4 A schematic diagram of a document segmentation device based on a large language model provided in this application embodiment is shown below. For example, please refer to... Figure 4 As shown, the document segmentation device 40 based on a large language model may include: The input unit 401 is used to input the target document to be segmented into a pre-trained large language model; Processing unit 402 is configured to perform the following operations using the large language model: Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document; For each paragraph, insert the related headings from all the headings into the beginning of the paragraph to obtain the corresponding target paragraph; Based on the semantic similarity among all target paragraphs, all target paragraphs are sorted, and based on all sorted target paragraphs, the segmentation result of the target document is determined.

[0090] For example, in an embodiment of this application, the processing unit 402 is configured to insert related headings from all headings that are relevant to the paragraph into the beginning position of the paragraph, including: Based on the correspondence between all the headings and all the paragraphs, determine the hierarchical structure relationship between all the headings; According to the hierarchical structure, the associated headings related to the paragraph are inserted at the beginning of the paragraph.

[0091] For example, in an embodiment of this application, when there are multiple associated headings related to the paragraph, the processing unit 402 is configured to insert the associated headings related to the paragraph at the beginning of the paragraph according to the hierarchical structure, including: Based on the hierarchical structure, determine the level corresponding to each of the multiple associated headings related to the paragraph; The associated headings are inserted sequentially at the beginning of the paragraph in descending order of hierarchy.

[0092] For example, in an embodiment of this application, the processing unit 402 is used to determine the segmentation result of the target document based on all sorted target paragraphs, including: Determine the document segmentation length; For each of the sorted target paragraphs, the target paragraphs are segmented based on the document segmentation length to obtain the corresponding paragraph segmentation results; The segmentation result of the target document includes the segmentation result of each target paragraph after sorting.

[0093] For example, in an embodiment of this application, the processing unit 402 is used to segment the target paragraph based on the document segmentation length to obtain the corresponding paragraph segmentation result, including: The target paragraph is segmented based on the document segmentation length; If the segmentation position does not fall at the end of the target paragraph, then determine whether the segmentation position is the end of a complete sentence; Based on the judgment result, the segmentation of the target paragraph is obtained; The associated headings related to the target paragraph are inserted again into the target paragraph. The starting position of the remaining paragraph after the segmentation is cut off, and the inserted paragraph is used as the new target paragraph. The above steps are repeated until the segmentation position falls at the end position of the new target paragraph. The paragraph segmentation result includes the segmentation obtained from each segmentation.

[0094] For example, in an embodiment of this application, the processing unit 402 is used to obtain segmented segments for the target paragraph based on the judgment result, including: If the segmentation position is the end position of a complete sentence, then the segment between the start position of the target paragraph and the end position of the complete sentence is determined as the segmentation segment; If the segmentation position is not the end position of a complete sentence, then the end position of the complete sentence closest to the segmentation position is determined, and the segment between the start position of the target paragraph and the end position of the closest complete sentence is padded based on a preset padding character. The length of the padded segment is the document segmentation length, and the padded segment is determined as the segmentation segment.

[0095] For example, in an embodiment of this application, the processing unit 402 is configured to perform text layout analysis on the target document, identifying headings at various levels and all paragraphs in the target document, including: The target document is subjected to text layout analysis to identify key content in the target document, including font size, paragraphs, line numbers, and sentences. Identify the headings at each level based on the font size, line number, heading, and sentence. All paragraphs are identified based on the paragraph, the line number, and the sentence.

[0096] The document segmentation device 40 based on a large language model provided in this application embodiment can execute the technical solution of the document segmentation method based on a large language model in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the document segmentation method based on a large language model. Please refer to the implementation principle and beneficial effects of the document segmentation method based on a large language model, which will not be repeated here.

[0097] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a document segmentation method based on a large language model. This method includes: inputting the target document to be segmented into a pre-trained large language model; performing the following operations through the large language model: performing text layout analysis on the target document to identify the headings at each level and all paragraphs in the target document; for each paragraph, inserting the related headings from all the headings into the beginning position of the paragraph to obtain the corresponding target paragraph; sorting all target paragraphs based on the semantic similarity between all target paragraphs; and determining the segmentation result of the target document based on all the sorted target paragraphs.

[0098] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the document segmentation method based on a large language model provided by the above methods. The method includes: inputting the target document to be segmented into a pre-trained large language model, and performing the following operations through the large language model: performing text layout analysis on the target document to identify the headings and all paragraphs in the target document; for each paragraph, inserting the related headings among all the headings into the beginning position of the paragraph to obtain the corresponding target paragraph; sorting all the target paragraphs based on the semantic similarity between all the target paragraphs, and determining the segmentation result of the target document based on all the sorted target paragraphs.

[0100] In another aspect, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the document segmentation method based on a large language model provided by the above methods. The method includes: inputting a target document to be segmented into a pre-trained large language model, and performing the following operations through the large language model: performing text layout analysis on the target document to identify headings at various levels and all paragraphs in the target document; for each paragraph, inserting related headings from all headings into the beginning position of the paragraph to obtain the corresponding target paragraph; sorting all target paragraphs based on semantic similarity, and determining the segmentation result of the target document based on all sorted target paragraphs.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / w:p>

Claims

1. A document segmentation method based on a large language model, characterized in that, include: The target document to be segmented is input into a pre-trained large language model, and the following operations are performed through the large language model: Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document; For each paragraph, insert all related headings from the headings into the beginning of the paragraph to obtain the corresponding target paragraph; Based on the semantic similarity among all target paragraphs, all target paragraphs are sorted, and based on all sorted target paragraphs, the segmentation result of the target document is determined.

2. The method according to claim 1, characterized in that, The step of inserting all related headings from the paragraph into the beginning of the paragraph includes: Based on the correspondence between all the headings and all the paragraphs, determine the hierarchical structure relationship between all the headings; According to the hierarchical structure, the associated headings related to the paragraph are inserted at the beginning of the paragraph.

3. The method according to claim 2, characterized in that, When there are multiple related headings associated with the paragraph, inserting the related headings at the beginning of the paragraph according to the hierarchical structure includes: Based on the hierarchical structure, determine the level corresponding to each of the multiple associated headings related to the paragraph; The associated headings are inserted sequentially at the beginning of the paragraph in descending order of hierarchy.

4. The method according to any one of claims 1-3, characterized in that, The process of determining the segmentation result of the target document based on all sorted target paragraphs includes: Determine the document segmentation length; For each of the sorted target paragraphs, the target paragraphs are segmented based on the document segmentation length to obtain the corresponding paragraph segmentation results; The segmentation result of the target document includes the segmentation result of each target paragraph after sorting.

5. The method according to claim 4, characterized in that, The step of segmenting the target paragraph based on the document segmentation length to obtain the corresponding paragraph segmentation result includes: The target paragraph is segmented based on the document segmentation length; If the segmentation position does not fall at the end of the target paragraph, then determine whether the segmentation position is the end of a complete sentence; Based on the judgment result, the segmentation of the target paragraph is obtained; The associated headings related to the target paragraph are inserted again into the target paragraph. The starting position of the remaining paragraph after the segmentation is cut off, and the inserted paragraph is used as the new target paragraph. The above steps are repeated until the segmentation position falls at the end position of the new target paragraph. The paragraph segmentation result includes the segmentation obtained from each segmentation.

6. The method according to claim 5, characterized in that, The process of obtaining segmentation for the target paragraph based on the judgment result includes: If the segmentation position is the end position of a complete sentence, then the segment between the start position of the target paragraph and the end position of the complete sentence is determined as the segmentation segment; If the segmentation position is not the end position of a complete sentence, then the end position of the complete sentence closest to the segmentation position is determined, and the segment between the start position of the target paragraph and the end position of the closest complete sentence is padded based on a preset padding character. The length of the padded segment is the document segmentation length, and the padded segment is determined as the segmentation segment.

7. The method according to any one of claims 1-3, characterized in that, The text layout analysis of the target document, identifying headings at each level and all paragraphs in the target document, includes: The target document is subjected to text layout analysis to identify key content in the target document, including font size, paragraphs, line numbers, and sentences. Identify the headings at each level based on the font size, line number, heading, and sentence. All paragraphs are identified based on the paragraph, the line number, and the sentence.

8. A document segmentation device based on a large language model, characterized in that, include: The input unit is used to input the target document to be segmented into a pre-trained large language model; The processing unit is configured to perform the following operations using the large language model: Perform text layout analysis on the target document to identify headings at each level and all paragraphs in the target document; For each paragraph, insert all related headings from the headings into the beginning of the paragraph to obtain the corresponding target paragraph; Based on the semantic similarity among all target paragraphs, all target paragraphs are sorted, and based on all sorted target paragraphs, the segmentation result of the target document is determined.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the document segmentation method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the document segmentation method based on a large language model as described in any one of claims 1 to 7.

Images