A long document outline extraction method, system and computer device

Abstract

This invention discloses a method, system, and computer device for extracting outlines from long documents, belonging to the field of document processing technology. The method includes: S100, reading the document to be processed; S200, processing the document to obtain text blocks and recording the positions of the text blocks within the document; S300, inputting the text blocks into a large language model to generate corresponding result blocks, where each result block is the outline of the current text block; S400, repeating step S300 until all result blocks corresponding to all text blocks are obtained, and then concatenating all result blocks to obtain the document's outline. This invention employs a multi-level processing strategy, segmenting at the paragraph level, and, when encountering extremely long paragraphs, performing precise segmentation at the sentence level. This avoids truncation and information loss due to excessively long text, and also prevents resource waste caused by excessively short text blocks.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a method, system, and computer device for extracting outlines from long documents. Background Technology

[0002] In recent years, with the continuous advancement of deep learning technology, especially the innovation of neural network architectures and the improvement of computing power, large language models have achieved revolutionary results in the field of natural language processing (NLP). These models, such as OpenAI's GPT series, Google's BERT, and Microsoft's Turing NLG, possess hundreds of millions or even trillions of parameters, enabling them to understand and generate complex language expressions, thus demonstrating outstanding performance in various NLP tasks. Large language models possess powerful language understanding and efficient language generation capabilities, able to understand the deep meaning of natural language text, including semantics, sentiment, and logical relationships, and can generate coherent and logical text content based on a given context. Large language models primarily use large amounts of text data for pre-training to obtain strong generalization ability, and then fine-tune them with small amounts of domain-specific data to achieve even stronger capabilities in specific domains, or design excellent prompting engineering to enable the large model to solve specific tasks.

[0003] Document outline extraction is an important research direction in information retrieval and text mining. Its goal is to extract key information from long documents and create a structured summary, including headings at various levels and summaries of the corresponding content. Outline extraction is significant for improving information retrieval efficiency, promoting knowledge management, and supporting academic research.

[0004] Traditional outline extraction primarily uses rule-based methods: identifying structural elements of a document, such as headings and paragraphs, through pre-defined grammatical rules and keyword matching; or extracting corresponding headings and body paragraphs by matching inherent document attributes. Rule-based methods suffer from several problems: rules are typically hard-coded, meaning they lack flexibility for different text formats or styles. These methods often fail to handle text that doesn't conform to pre-defined rules correctly; as the number of rules increases, system management and maintenance become more complex. New text types or domains require rule redesign, increasing the difficulty of expansion; rule development requires domain expert knowledge and experience, often involving significant manual work. Furthermore, rules need constant updating as text types change, further increasing manual costs; due to the complexity and ambiguity of language, rules struggle to cover all possible text scenarios. Even minor rule errors can lead to incorrect extraction results.

[0005] Large language models, pre-trained with a large amount of text, have achieved powerful generalization capabilities and exhibit good performance in both language understanding and generation. Therefore, applying large language models to solve outline extraction problems is highly feasible. Large language models can understand and process various text formats and styles, offering greater flexibility. Furthermore, their training involves data from multiple domains, eliminating concerns about domain adaptability. They also automatically learn patterns through machine learning, reducing the need for manual intervention. Because they are exposed to a large number of language variations and contextual information during training, they can also better understand the complexity and polysemy of language.

[0006] Based on the above problems and reasons, this invention proposes a document outline extraction method based on a large model. It aims to utilize the powerful language understanding, language generation, flexibility and generalization capabilities of a large language model to accurately extract and summarize structured outlines from documents of various domain styles. Summary of the Invention

[0007] This invention provides a method, system, and computer device for extracting outlines from long documents, in order to solve the problem of lack of flexibility in the prior art for different text formats or styles.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] This invention provides a method for extracting outlines from long documents, characterized by comprising:

[0010] S100: Read the document to be processed;

[0011] S200. Process the document to obtain text blocks and record the position of the text blocks in the document;

[0012] S300. Input the text block into the large language model and generate the result block corresponding to the text block. The result block is the outline corresponding to the current text block.

[0013] S400. Repeat step S300 until all text blocks are obtained as result blocks, and then concatenate all result blocks to obtain the document outline.

[0014] Based on this, the present invention can be further improved as follows: In step S200, the method for processing the document to obtain text blocks includes:

[0015] S210, Set the preset maximum number of characters Tmax;

[0016] S220. Determine whether the number of characters Ti in the current segment is greater than the preset maximum number of characters Tmax. If yes, then divide the i-th segment according to the preset maximum number of characters Tmax to generate several text blocks; otherwise, proceed to the next step.

[0017] S230. Determine whether the sum of the number of characters Ti in the current segment and the number of characters T(i,i+1) in the next segment is greater than the preset maximum number of characters Tmax. If yes, the current segment is directly used as a text block. If no, the current segment and the next segment are merged into the latest current segment. Repeat S230 until the sum of the number of characters in the latest current segment and the next segment exceeds the preset maximum number of characters Tmax.

[0018] Based on this, the present invention can be further improved as follows: In step S220, the method of segmenting the current segment according to a preset maximum number of characters includes:

[0019] S231. Traverse the current paragraph, locate all sentence-ending punctuation marks in the current paragraph, divide the current paragraph into several sentences according to the sentence-ending punctuation marks, and record the number of characters in each sentence;

[0020] S232. According to the order of sentences and the number of characters in the current paragraph, combine consecutive sentences that do not exceed the preset maximum number of characters Tmax into a text block.

[0021] Based on this, the present invention can be further improved as follows: In step S300, the method for forming complete prompt information includes:

[0022] S310. Based on the position of the text block in the paragraph text, determine whether the current text block is the first text block. If so, input the current text block into the large language model and generate the result block according to the preset output example. If not, proceed to the next step.

[0023] S320. Based on the content of the current text block, the previous text block, and the result block of the currently generated text block, generate a result block according to a preset output example.

[0024] Based on this, the present invention can be further improved as follows: the method for generating the result block in step S320 includes:

[0025] Choose either JSON or Markdown as the outline format;

[0026] If it is JSON, then based on the content of the current text block, the outline of the currently generated text block is expanded to obtain the outline up to the current text block, and used as the result block of the current text block;

[0027] If it is Markdown, the outline of the current text block is concatenated with the outline of the already generated text blocks to obtain the outline up to the current text block, which is then used as the result block of the current text block.

[0028] This invention also proposes a long document outline extraction system for performing the long document outline extraction method described above, including:

[0029] A document reading module, which reads the document to be processed and converts the document into paragraph text.

[0030] A text fragmentation module is used to set a preset maximum number of characters and to divide the document to be processed into several text blocks according to the preset maximum number of characters.

[0031] The large model outline extraction module is used to generate an outline of the document to be processed based on text blocks.

[0032] Based on this, the present invention can be further improved as follows: the large model outline extraction module includes a large model generation module and a result processing module.

[0033] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0034] The beneficial effects provided by this invention are:

[0035] This invention employs a multi-layered processing strategy, segmenting text at the paragraph level and, when encountering extremely long paragraphs, precisely segmenting them at the sentence level. This avoids truncation and information loss caused by excessively long text, while also preventing resource waste due to excessively short text blocks.

[0036] Meanwhile, by utilizing a pre-trained large language model for document outline extraction, this technical solution significantly reduces the high costs associated with manual annotation, rule design, and maintenance updates required in traditional methods. This plug-and-play solution not only saves manpower and time but also reduces financial investment, making document processing more economical and efficient. Furthermore, due to the powerful capabilities of the large model, frequent rule updates are unnecessary, further reducing long-term operating costs.

[0037] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0038] Figure 1 A schematic diagram of the method flow for extracting the outline of a long document provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the system structure framework of the long document outline extraction system provided in the embodiments of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0041] like Figure 1 As shown, this invention provides a method for extracting outlines from long documents. Based on text fragmentation and large language model prompting engineering, the method extracts the outline of a document, including the following steps:

[0042] S100: Read the document to be processed;

[0043] S200. Process the document to obtain text blocks and record the position of the text blocks in the document;

[0044] S300. Input the text block into the large language model and generate the result block corresponding to the text block. The result block is the outline corresponding to the current text block.

[0045] S400. Repeat step S300 until all text blocks are obtained as result blocks, and then concatenate all result blocks to obtain the document outline.

[0046] The solution provided by this invention has the following effects:

[0047] This invention employs a multi-layered processing strategy, segmenting text at the paragraph level and, when encountering extremely long paragraphs, precisely segmenting them at the sentence level. This avoids truncation and information loss caused by excessively long text, while also preventing resource waste caused by excessively short text blocks.

[0048] Optionally, in some possible implementations, the method for processing the document to obtain text blocks in step S200 includes:

[0049] S210. Set the preset maximum number of characters Tmax, preferably 1000;

[0050] S220. Determine whether the number of characters Ti in the current segment i is greater than the preset maximum number of characters Tmax. If yes, then divide the current segment i into several text blocks according to the preset maximum number of characters Tmax; otherwise, proceed to the next step.

[0051] S230. Determine whether the sum of the number of characters Ti of the current segment i and the number of characters i+1 of the next segment T(i,i+1) is greater than the preset maximum number of characters Tmax. If yes, then the current segment i is directly used as a text block. If no, then the current segment i and the next segment i+1 are merged into the latest current segment inew. Repeat S230 until the sum of the number of characters of the latest current segment inew and the next segment inew+1 exceeds the preset maximum number of characters Tmax.

[0052] Optionally, in some possible implementations, step S220, the method of segmenting the current segment according to a preset maximum number of characters includes:

[0053] S231. Traverse the current segment, locate all sentence-ending punctuation marks in the current segment, including periods, question marks, and exclamation marks in Chinese and English, divide the current segment into several sentences according to the sentence-ending punctuation marks, and record the number of characters in each sentence.

[0054] S232. According to the order of sentences and the number of characters in the current paragraph, combine consecutive sentences that do not exceed the preset maximum number of characters Tmax into a text block.

[0055] Optionally, in some possible implementations, the method for forming a complete prompt message in step S300 includes:

[0056] S310. Based on the position of the text block in the paragraph text, determine whether the current text block is the first text block. If so, input the current text block into the large language model and generate the result block according to the preset output example and prompt instructions. If not, proceed to the next step.

[0057] S320. Based on the content of the current text block, the previous text block, and the result block of the currently generated text block, generate the result block according to the preset output example and prompt instructions.

[0058] Optionally, in some possible implementations, the method for generating the result block in step S320 includes:

[0059] Choose either JSON or Markdown as the outline format;

[0060] If it is JSON, then based on the content of the current text block, the outline of the currently generated text block is expanded to obtain the outline up to the current text block, and used as the result block of the current text block;

[0061] If it is Markdown, the outline of the current text block is concatenated with the outline of the already generated text blocks to obtain the outline up to the current text block, which is then used as the result block of the current text block.

[0062] This invention also proposes a long document outline extraction system for performing the long document outline extraction method described above, such as... Figure 2 As shown, it includes:

[0063] A document reading module, which reads the document to be processed and converts the document into paragraph text.

[0064] A text fragmentation module is used to set a preset maximum number of characters and to divide the document to be processed into several text blocks according to the preset maximum number of characters.

[0065] The large model outline extraction module is used to generate an outline of the document to be processed based on text blocks.

[0066] Optionally, in some possible implementations, the large model outline extraction module includes a large model generation module and a result processing module;

[0067] The large model generation module is used to preset output examples and prompts, and to obtain the result block corresponding to the current text block based on the locally deployed large model.

[0068] The result processing module is used to process the result blocks and generate a document outline.

[0069] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0070] Optionally, some possible implementations may include all or part of the above-described implementations.

[0071] It should be understood that in the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this description, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples described in this specification, as well as some features of different embodiments or examples.

[0072] Of course, those skilled in the art can make various corresponding changes and modifications based on the present invention without departing from its spirit and essence, but such changes and modifications should all fall within the protection scope of the claims of the present invention.

Claims

1. A method for extracting outlines from long documents, characterized in that, include: S100: Read the document to be processed; S200. Process the document to obtain text blocks and record the position of the text blocks in the document; S300. Input the text block into the large language model and generate the result block corresponding to the text block. The result block is the outline corresponding to the current text block. S400. Repeat step S300 until all text blocks are obtained, and the document outline is obtained.

2. The method for extracting outlines from long documents according to claim 1, characterized in that, In step S200, the method for processing the document to obtain text blocks includes: S210, Set the preset maximum number of characters Tmax; S220. Determine whether the number of characters Ti in the current segment is greater than the preset maximum number of characters Tmax. If yes, then divide the i-th segment according to the preset maximum number of characters Tmax to generate several text blocks; otherwise, proceed to the next step. S230. Determine whether the sum of the number of characters Ti in the current segment and the number of characters T(i,i+1) in the next segment is greater than the preset maximum number of characters Tmax. If yes, the current segment is directly used as a text block. If no, the current segment and the next segment are merged into the latest current segment. Repeat S230 until the sum of the number of characters in the latest current segment and the next segment exceeds the preset maximum number of characters Tmax.

3. The method for extracting outlines from long documents according to claim 1, characterized in that, In step S220, the method for segmenting the current segment according to a preset maximum number of characters includes: S231. Traverse the current paragraph, locate all sentence-ending punctuation marks in the current paragraph, divide the current paragraph into several sentences according to the sentence-ending punctuation marks, and record the number of characters in each sentence; S232. According to the order of sentences and the number of characters in the current paragraph, combine consecutive sentences that do not exceed the preset maximum number of characters Tmax into a text block.

4. The method for extracting outlines from long documents according to claim 1, characterized in that, In step S300, the method for forming a complete prompt message includes: S310. Based on the position of the text block in the paragraph text, determine whether the current text block is the first text block. If so, input the current text block into the large language model and generate the result block according to the preset output example. If not, proceed to the next step. S320. Based on the content of the current text block, the previous text block, and the result block of the currently generated text block, generate a result block according to a preset output example.

5. The method for extracting outlines from long documents according to claim 5, characterized in that, In step S320, the method for generating the result block includes: Choose either JSON or Markdown as the outline format; If it is JSON, then based on the content of the current text block, the outline of the currently generated text block is expanded to obtain the outline up to the current text block, and used as the result block of the current text block; If it is Markdown, the outline of the current text block is concatenated with the outline of the already generated text blocks to obtain the outline up to the current text block, which is then used as the result block of the current text block.

6. A long document outline extraction system, characterized in that, The method for extracting the outline of a long document as described in claims 1-5 includes: A document reading module, which reads the document to be processed and converts the document into paragraph text. A text fragmentation module is used to set a preset maximum number of characters and to divide the document to be processed into several text blocks according to the preset maximum number of characters. The outline extraction module is used to generate an outline of the document to be processed based on text blocks.

7. The long document outline extraction system according to claim 6, characterized in that, The outline extraction module includes a large model generation module and a result processing module.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to claims 1-5.

Images