Long document rapid summarization system and method based on semantic extraction

Abstract

The invention relates to the technical field of natural language computing equipment processing, in particular to a long document rapid summarization system and method based on semantic extraction. The system comprises a document loading and analyzing module, a text slicing processing module, a fragment extraction type abstract module, an abstract information fusion module and a generative summary module. Through fusion of an extraction type abstract technology and a generation type abstract technology, the problem of key information loss caused by context length limitation of a large language model is solved, and meanwhile, the fluency, the computing power consumption and the scene adaptability are considered. According to the method, core information can be efficiently extracted, the final abstracts with semantic coherence can be generated, and reliable support is provided for resource-limited scenes.

Description

Technical Field

[0001] This invention relates to the field of natural language computing device processing technology, and in particular to a system and method for rapid summarization of long documents based on semantic extraction. Background Technology

[0002] In current artificial intelligence applications, Large Language Models (LLMs) have been widely used in office work, scientific research, and education, becoming an important tool for improving content generation efficiency. Especially when generating PPTs, report outlines, and other content, it is often necessary to input long documents (such as research reports and academic monographs) ranging from tens of thousands to hundreds of thousands of words to ensure the accuracy and completeness of the generated content. However, the inherent context length limitations of large models mean that directly inputting extremely long documents results in the loss of key content due to information truncation. Users must first extract the core information and organize the logical framework manually or using tools, a process that significantly restricts efficiency. Summarizing information and creating structured outlines for long documents becomes a crucial bridge connecting the original information of long documents with the creative capabilities of large models, further highlighting the importance of methods for quickly summarizing long documents.

[0003] In existing technologies, mainstream long text summarization methods can be divided into two main categories: extractive summarization and generative summarization. Existing patent document CN120805870A provides a document processing and annotation method based on OpenXML, which focuses on paragraph-level segmentation and summarization. In the prior art, patent number CN119415652A in the patent literature relates to a large language model system for document question answering, generation and version comparison, which emphasizes the ability to understand global documents.

[0004] Existing technologies, such as extractive summarization, use algorithms to select key sentences from the original text and directly combine them according to the original text's logical order to form a summary. Its core advantage lies in its simple implementation and lack of complex semantic understanding, but it suffers from insufficient fluency and residual information redundancy in practical applications. Generative summarization relies on a large language model's deep understanding of the original text content, generating entirely new and coherent text through recursive summarization. Its advantages include strong summarization fluency and logical completeness, but it requires significant computing power, has poor real-time performance, and has low adaptability in resource-constrained edge computing scenarios.

[0005] Therefore, it is necessary to provide a solution for rapid summarization of long documents that integrates extraction and generative methods to address the three core issues of fluency, computational consumption, and scenario adaptation, and to provide efficient support for large model contexts and resource-constrained application scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a long document rapid summary system based on semantic extraction, so as to overcome the above-mentioned shortcomings of the existing technology.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A system for rapidly summarizing long documents based on semantic extraction, comprising: The document loading and parsing module is used to load and parse raw documents of different formats into plain text data; The text slicing module is used to divide a document into several document segments based on paragraphs. The segmented extraction and summarization module is used to perform extraction and summarization tasks in parallel for each document segment, using TF-ID; The algorithm evaluates sentence weights and extracts key sentences based on the PageRank algorithm; The summary information fusion module is used to gather all the fragment summaries in the original text order and combine them into a coarse summary that covers the core information of the entire document. The generative summary module is used to input the rough summary as a prompting context into the large language model to generate the final summary.

[0008] Preferably, the document loading and parsing module is implemented based on the LangChain framework, supporting the reading and parsing of PDF, Word, TXT and Markdown format documents. It converts the original document content into standardized text data by calling LangChain's document loader.

[0009] Preferably, the text slicing processing module first performs an initial division based on the natural paragraphs of the document, and then dynamically adjusts the merging strategy of adjacent paragraphs to ensure that the length of each document segment does not exceed the set number of characters.

[0010] Preferably, when extracting key sentences, the segmented extraction and summarization module first divides the document fragment into sentence sequences based on Chinese and English punctuation marks, then uses the TF-IDF algorithm to calculate sentence weights, and combines the PageRank algorithm to rank the importance of sentences. The number of key sentences extracted from a single segment does not exceed the upper limit.

[0011] Preferably, the summary information fusion module avoids information redundancy caused by repeated extraction by introducing a sentence deduplication algorithm, and ensures that the length of the coarse summary meets the context length limit of the large language model by dynamically adjusting the compression ratio.

[0012] Preferably, the generative summary module uses the Qwen3-32B model, which supports both reasoning and non-reasoning modes and improves the fluency and logical integrity of the final summary by optimizing the design of prompt words.

[0013] Preferably, the generative summary module ensures that the generated final summary is concise and accurate by controlling the generation parameters, including temperature value and maximum output length.

[0014] A method for quickly summarizing long documents based on semantic extraction includes the following steps: Step 1, document loading and parsing: The document loader based on the LangChain framework can read and parse documents of various formats. It can seamlessly handle a variety of common document types, including PDF, Word, TXT and Markdown, and convert their content into standardized text data that can be processed in subsequent processes. Step two, text slicing, uses paragraphs as the basic unit and divides the document according to the preset slice length limit. This includes two steps: First, the document is initially divided according to its natural paragraphs; then, adjacent paragraphs are merged to form the final document fragments for processing, provided that the slice length limit is met. Step 3: Segmented Extraction Summarization. For each document segment, it is first divided into a sequence of sentences based on Chinese and English punctuation marks. Then, the TF-IDF algorithm is used to evaluate the sentence weights, and the PageRank algorithm is used to extract the most critical core sentences in each segment. Finally, these sentences are arranged in the order of their appearance in the original text to form a preliminary summary of the segment. Step 4: Summary information fusion. All the summary fragments are collected in their original order and combined into a coarse summary that covers the core information of the entire document and is significantly shortened in length. If the coarse summary still exceeds the context length limit of the large language model, the extraction and compression process of Step 3 and Step 4 is repeated until its length meets the model input requirements. Step 5, LLM summary: The generated rough summary is used as a cue context and, with reference to the cue word template, is input into the large language model to generate the final summary; this summary is characterized by fluent language, natural expression, and highly condensed information, which conforms to human reading habits.

[0015] The beneficial effects of this invention are: by integrating extraction and generative summarization techniques, this technical solution solves the problem of key information loss caused by the context length limitation of large language models in the prior art, while taking into account fluency, computing power consumption and scene adaptability. In the segmented extraction and summarization module, the system's processing efficiency is significantly improved by performing extraction and summarization tasks on document fragments in parallel. This module uses the TF-IDF algorithm to evaluate sentence weights and extracts key sentences based on the PageRank algorithm, ensuring the preservation of core information. In the summary information fusion module, the compression ratio is dynamically adjusted to ensure that the length of the coarse summary conforms to the context length limit of the large language model. This module avoids information redundancy caused by repeated extraction by introducing a sentence deduplication algorithm; In the generative summary module, prompt word templates are introduced to guide the large language model to reorganize, refine, and polish fragmented information points in the rough summary. This module improves the linguistic fluency and logical integrity of the final summary by optimizing the prompt word design. This invention achieves high efficiency and practicality in a rapid summary method for long documents, providing reliable support for application scenarios with large model contexts and limited resources. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a long document rapid summarization system based on semantic extraction according to the present invention; Figure 2 This is a schematic diagram of the workflow of a long document rapid summary system based on semantic extraction according to the present invention; In the diagram: 1. Document loading and parsing module; 2. Text slicing processing module; 3. Sliced ​​extraction and summarization module; 4. Summary information fusion module; 5. Generative summary module. Detailed Implementation

[0017] Reference Figures 1 to 2 A system for rapidly summarizing long documents based on semantic extraction, comprising the following modules: Document loading and parsing module 1 is used to load and parse raw documents of different formats into plain text data; Text slicing module 2 is used to segment the document into several document fragments based on paragraphs as the basic unit. The segmented extraction and summarization module 3 is used to perform extraction and summarization tasks in parallel for each document segment. It uses the TF-IDF algorithm to evaluate sentence weights and extracts key sentences based on the PageRank algorithm. Abstract information fusion module 4 is used to gather all the fragment abstracts in the original text order and combine them into a coarse abstract that covers the core information of the entire document; Generative summary module 5 is used to input the rough summary as a prompting context into the large language model to generate the final summary.

[0018] Five modules are sequentially connected via data flow to form a complete method for rapid summarization of long documents. Module 1 parses the original document into standardized text data; Module 2 segments the text data to suit subsequent processing needs; Module 3 extracts the core information of each segment using extractive summarization techniques; Module 4 merges the segment summaries into a coarse summary while ensuring its length meets model constraints; and Module 5 uses generative techniques to generate the final summary. The close and orderly collaboration between the modules ensures the efficiency and practicality of the entire method.

[0019] The document loading and parsing module 1 is the first step in the entire method, used to load and parse raw documents of different formats into plain text data. This module is implemented based on the LangChain framework and integrates a PDF parser, a Word parser, a TXT parser, and a Markdown parser. When a user inputs a document, module 1 first automatically identifies the document format by calling the document loader in the LangChain framework and selects the corresponding parsing tool based on the format. For example, for PDF documents, module 1 calls the PDF parser to extract the document content and remove irrelevant formatting information; for Word documents, it calls the Word parser to convert it into standardized text data. All parsed text data is stored in a unified encoding format for easy processing by subsequent modules. Module 1 and module 2 are connected via data stream, that is, the parsed plain text data is passed to the text slicing processing module 2.

[0020] The text slicing processing module 2 receives plain text data from module 1 and segments the document into paragraphs. Module 2 first performs an initial division based on the natural paragraphs of the document, and then dynamically adjusts the merging strategy of adjacent paragraphs to ensure that the length of each document segment does not exceed a preset upper limit.

[0021] In this embodiment, the maximum slice length is set to 4096 characters. To achieve this, module 2 introduces a sliding window mechanism, which sets an adjustable window at the paragraph boundary. By counting the number of characters within the window, it determines whether adjacent paragraphs need to be merged or excessively long paragraphs need to be further segmented. Module 2 sequentially numbers the segmented document fragments and passes them to the segmented extraction and summarization module 3, while simultaneously recording the position information of each fragment in the original text for use by subsequent modules.

[0022] The segmented extraction and summarization module 3 performs extraction and summarization tasks in parallel for each document segment. Module 3 first segments the document segment into sentence sequences based on Chinese and English punctuation marks, and then uses the TF-IDF algorithm to evaluate sentence weights. For Chinese documents, module 3 uses the jieba tool for word segmentation to improve the accuracy of the TF-IDF algorithm. After calculating the sentence weights, module 3 ranks the importance of sentences based on the PageRank algorithm and extracts the most critical core sentences from each segment. In this embodiment, the maximum number of key sentences extracted from a single segment is set to 10. Module 3 significantly improves system efficiency through parallel processing, meaning that summarization tasks for multiple document segments can be performed simultaneously, thus meeting real-time requirements. Module 3 arranges the extracted key sentences according to their order of appearance in the original text to form a preliminary summary of the segment, and then passes these preliminary summaries to the summary information fusion module 4.

[0023] The summary information fusion module 4 receives all fragment summaries from module 3 and combines them in the original text order into a coarse summary that covers the core information of the entire document and is significantly shortened in length. Module 4 avoids information redundancy caused by repeated extraction by introducing a sentence deduplication algorithm. Specifically, when combining fragment summaries, module 4 first calculates the similarity of all sentences. If the similarity between two sentences exceeds a preset threshold, only the sentence with the higher weight is retained. In addition, module 4 also dynamically adjusts the compression ratio to ensure that the length of the coarse summary meets the context length limit of the large language model. If the length of the coarse summary exceeds the limit, module 4 re-segments the coarse summary into several sub-fragments and repeats the extraction and compression process of the fragment extraction summary module 3 and the summary information fusion module 4 until the length of the coarse summary meets the model input requirements. Module 4 then passes the finally generated coarse summary to the generative summary module 5.

[0024] The generative summary module 5 uses the rough summary as a prompt context and, referring to a preset prompt word template, inputs it into the large language model to generate the final summary. In this embodiment, module 5 uses the Qwen3-32B model, which supports both inference and non-inference modes, ensuring both generation quality and performance efficiency. Module 5 improves the linguistic fluency and logical integrity of the final summary by optimizing the prompt word design. Specifically, when generating the final summary, module 5 first dynamically adjusts the keywords and semantic structure in the prompt word template based on the content of the rough summary to guide the large language model to reorganize, polish, and refine the fragmented information points in the rough summary. Module 5 also controls generation parameters such as temperature value and maximum output length to ensure that the generated final summary is both concise and accurate. After the final summary is generated, module 5 outputs it to the user interface or stores it to a specified file path.

[0025] In practical applications, the method of this invention can be used for rapid summarization of long documents in various scenarios. For example, in academic research, researchers can use this method to quickly obtain the core content of a long paper; in corporate office settings, employees can use this method to quickly summarize meeting minutes or project reports; and in the news media field, editors can use this method to extract key information from long reports. Regardless of the scenario, this method effectively solves the problem of key information loss caused by the context length limitations of large language models, while also considering fluency, computational power consumption, and scenario adaptability.

[0026] In this implementation case, taking a PDF document as an example, the document loading and parsing module 1 first extracts the paper content into plain text data by calling the PDF parser in the LangChain framework and removes irrelevant formatting information. For example, module 1 automatically identifies and removes non-core content such as headers, footers, and figure / table titles, retaining only the main text. The parsed text data is stored in a unified encoding format and then passed to the text slicing processing module 2.

[0027] Subsequently, the text slicing module 2 performs initial segmentation based on the natural paragraphs of the paper. Since some paragraphs may be too long, module 2 dynamically adjusts the merging strategy of adjacent paragraphs using a sliding window mechanism to ensure that the length of each document segment does not exceed 4096 characters. For example, if a paragraph is 5000 characters long, module 2 splits it into two sub-segments and records the position information of each sub-segment in the original text. After segmentation, module 2 sequentially numbers all document segments and passes them to the segmented extraction and summarization module 3.

[0028] Segmented Extraction and Summarization Module 3 performs extraction and summarization tasks in parallel for each document segment. Module 3 first segments the document segment into sentence sequences based on punctuation and then uses the jieba tool to perform word segmentation on Chinese sentences to improve the accuracy of the TF-IDF algorithm. Subsequently, Module 3 ranks the sentences by importance based on the PageRank algorithm and extracts the most crucial core sentences from each segment. For example, in a certain segment, Module 3 extracts 10 key sentences that cover the main viewpoints and arguments of that segment. Module 3 arranges these sentences in the order of their appearance in the original text to form a preliminary summary of the segment, which is then passed to the summary information fusion module 4.

[0029] The summary information fusion module 4 receives all fragment summaries from module 3 and combines them in the original text order into a coarse summary covering the core information of the entire document. During the combination process, module 4 avoids information redundancy by introducing a sentence deduplication algorithm. For example, if two fragment summaries contain sentences with a similarity exceeding a preset threshold, module 4 retains only the sentence with the higher weight. Furthermore, module 4 dynamically adjusts the compression ratio to ensure that the length of the coarse summary meets the context length limit of the large language model. If the length of the coarse summary exceeds the limit, module 4 re-segments the coarse summary into several sub-fragments and repeats the extraction and compression process of the fragment extraction summary module 3 and the summary information fusion module 4 until the length of the coarse summary meets the model input requirements.

[0030] Finally, the generative summary module 5 uses the rough summary as a contextual prompt and, referring to a preset prompt template, inputs it into the Qwen3-32B model to generate the final summary. Module 5 dynamically adjusts the keywords and semantic structure in the prompt template based on the content of the rough summary to guide the large language model in reorganizing, refining, and polishing the fragmented information points in the rough summary. For example, module 5 controls generation parameters such as temperature values ​​and maximum output length to ensure that the generated final summary is both concise and accurate. After the final summary is generated, module 5 outputs it to the user interface for researchers to review.

[0031] The document loading and parsing module 1's multi-format compatibility eliminates the need for researchers to manually convert document formats; the sliding window mechanism of the text slicing module 2 avoids information truncation or redundancy caused by paragraphs being too long or too short; the parallel processing mechanism of the segmented extraction and summarization module 3 significantly improves system processing efficiency; the sentence deduplication algorithm and dynamic compression ratio adjustment of the summary information fusion module 4 ensure that the quality and length of the rough summary meet the model input requirements; and the prompt word template design and parameter control of the generative summary module 5 enhance the linguistic fluency and logical integrity of the final summary.

[0032] The advantages of this invention are that this technical solution solves the problem of key information loss caused by the context length limitation of large language models in the prior art by integrating extraction and generative summarization technologies, while taking into account fluency, computing power consumption and scene adaptability. In the segmented extraction and summarization module, the system's processing efficiency is significantly improved by performing extraction and summarization tasks on document fragments in parallel. This module uses the TF-IDF algorithm to evaluate sentence weights and extracts key sentences based on the PageRank algorithm, ensuring the preservation of core information. In the summary information fusion module, the compression ratio is dynamically adjusted to ensure that the length of the coarse summary conforms to the context length limit of the large language model. This module avoids information redundancy caused by repeated extraction by introducing a sentence deduplication algorithm; In the generative summary module, prompt word templates are introduced to guide the large language model to reorganize, refine, and polish fragmented information points in the rough summary. This module improves the linguistic fluency and logical integrity of the final summary by optimizing the prompt word design. This invention achieves high efficiency and practicality in a rapid summary method for long documents, providing reliable support for application scenarios with large model contexts and limited resources.

[0033] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A system for rapidly summarizing long documents based on semantic extraction, characterized in that: It includes: The document loading and parsing module (1) is used to load and parse original documents of different formats into plain text data; The text slicing module (2) is used to segment the document into several document fragments based on paragraphs as the basic unit. The segmented extraction and summarization module (3) is used to perform extraction and summarization tasks in parallel for each document segment, using TF-ID; The algorithm evaluates sentence weights and extracts key sentences based on the PageRank algorithm; The summary information fusion module (4) is used to collect all the fragment summaries in the order of the original text and combine them into a coarse summary that covers the core information of the entire document; The generative summary module (5) is used to input the rough summary as a prompt context into the large language model to generate the final summary.

2. The long document rapid summarization system based on semantic extraction according to claim 1, characterized in that: The document loading and parsing module (1) is implemented based on the LangChain framework. It supports reading and parsing PDF, Word, TXT and Markdown format documents. By calling the LangChain document loader, the original document content is uniformly converted into standardized text data.

3. The long document rapid summarization system based on semantic extraction according to claim 1, characterized in that: The text slicing module (2) first performs an initial division based on the natural paragraphs of the document, and then ensures that the length of each document segment does not exceed the set number of characters by dynamically adjusting the merging strategy of adjacent paragraphs.

4. The long document rapid summarization system based on semantic extraction according to claim 1, characterized in that: When extracting key sentences, the segmented extraction summary module (3) first divides the document fragment into a sentence sequence based on Chinese and English punctuation marks, then uses the TF-IDF algorithm to calculate the sentence weights, and combines the PageRank algorithm to rank the importance of the sentences. The number of key sentences extracted from a single segment does not exceed the upper limit.

5. The long document rapid summarization system based on semantic extraction according to claim 1, characterized in that: The summary information fusion module (4) avoids information redundancy caused by repeated extraction by introducing a sentence deduplication algorithm, and ensures that the length of the coarse summary meets the context length limit of the large language model by dynamically adjusting the compression ratio.

6. The long document rapid summarization system based on semantic extraction according to claim 1, characterized in that: The generative summary module (5) uses the Qwen3-32B model, which supports both reasoning and non-reasoning modes. By optimizing the design of prompt words, the language fluency and logical integrity of the final summary are improved.

7. A long document rapid summarization system based on semantic extraction according to claims 1-5, characterized in that: The generative summary module (5) ensures that the generated final summary is concise and accurate by controlling the generation parameters, including temperature value and maximum output length.

8. A method for rapid summarization of long documents based on semantic extraction according to claims 1-7, characterized in that: Includes the following steps: Step 1, document loading and parsing: The document loader based on the LangChain framework can read and parse documents of various formats. It can seamlessly handle a variety of common document types, including PDF, Word, TXT and Markdown, and convert their content into standardized text data that can be processed in subsequent processes. Step two, text slicing, uses paragraphs as the basic unit and divides the document according to the preset slice length limit. This includes two steps: First, the document is initially divided according to its natural paragraphs; then, adjacent paragraphs are merged to form the final document fragments for processing, provided that the slice length limit is met. Step 3: Segmented Extraction Summarization. For each document segment, it is first divided into a sequence of sentences based on Chinese and English punctuation marks. Then, the TF-IDF algorithm is used to evaluate the sentence weights, and the PageRank algorithm is used to extract the most critical core sentences in each segment. Finally, these sentences are arranged in the order of their appearance in the original text to form a preliminary summary of the segment. Step 4: Summary information fusion. All the summary fragments are collected in their original order and combined into a coarse summary that covers the core information of the entire document and is significantly shortened in length. If the coarse summary still exceeds the context length limit of the large language model, the extraction and compression process of Step 3 and Step 4 is repeated until its length meets the model input requirements. Step 5, LLM summary: The generated rough summary is used as a cue context and, with reference to the cue word template, is input into the large language model to generate the final summary. This summary is characterized by fluent language, natural expression, and highly condensed information, which conforms to human reading habits.

Images