Document abstract generation method, computer device, storage medium and program product
By segmenting and filtering document content, determining the word count of the summary based on the intensity of attention, and using a large language model for multiple rounds of summarization, the problem of lack of personalized processing in existing technologies is solved, thereby improving the quality and flexibility of document summarization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FAW JIEFANG AUTOMOTIVE CO
- Filing Date
- 2024-04-26
- Publication Date
- 2026-06-02
AI Technical Summary
Existing document summarization methods lack personalized processing for different document types, resulting in low-quality generated summaries.
By segmenting and filtering the document content, determining the word count of the summary based on the attention intensity of each text block, and using a pre-set large language model for multiple rounds of summarization processing, a summary result of the document is generated.
It improves the flexibility and quality of summary generation, enabling it to better adapt to the information distribution and importance of different documents, and generate accurate and easy-to-understand summaries.
Smart Images
Figure CN118332104B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a document summarization method, computer device, computer-readable storage medium, and computer program product. Background Technology
[0002] Current document summarization methods primarily involve uniform segmentation and content extraction of the entire document. However, different documents have different focuses, and because traditional methods do not offer personalized processing for different document types and lack consideration for specific document types or needs, the quality of the generated summaries is low. Summary of the Invention
[0003] Therefore, it is necessary to provide a document summarization method, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems and improve the quality of the generated summaries.
[0004] Firstly, this application provides a document digest generation method, including:
[0005] Read the text content of a document;
[0006] The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity.
[0007] For each target text block, the number of words in the summary of the target text block is determined based on the intensity of attention it receives.
[0008] By using a pre-defined large language model, multiple rounds of summarization are performed based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate the generation of the summary, and the summary results corresponding to the document are output.
[0009] In one embodiment, the steps of the i-th round of summary processing include:
[0010] Determine the list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1;
[0011] Determine the input text block for the i-th round of summary processing; wherein, the input text block for the first round of summary processing is at least one target text block in the text block list with an attention intensity of no more than i; the input text block for non-first rounds of summary processing is the output text block of the (i-1)-th round and at least one target text block in the text block list with an attention intensity of no more than i.
[0012] Determine the target word count for the input text block;
[0013] The input text is constructed based on the input text block, the target summary word count, and prompts used to indicate the summary generation.
[0014] Input the input text into a preset large language model to obtain the summary text;
[0015] The summary text is divided into blocks according to the input length limit of the preset large language model to obtain at least one summary text block;
[0016] Remove at least one target text block from the text block list whose attention intensity is not greater than i;
[0017] If the sum of the number of summary text blocks and the number of target text blocks in the text block list exceeds a set threshold, at least one summary text block will be used as the output text block for the i-th round.
[0018] In one embodiment, the method further includes:
[0019] If the sum of the number of summary text blocks and the number of target text blocks in the text block list does not exceed the set threshold, the summary text block will be used as the summary result corresponding to the document.
[0020] In one embodiment, the text content is segmented and filtered to obtain multiple target text blocks, including:
[0021] The text volume range is determined based on the input length limit of the preset large language model;
[0022] The text content is divided into blocks based on preset delimiters and text size ranges at multiple levels, resulting in multiple first text blocks;
[0023] The text content in multiple first text blocks is filtered and reorganized to obtain multiple target text blocks.
[0024] In one embodiment, the text content in multiple first text blocks is filtered and reorganized to obtain multiple target text blocks, including:
[0025] For each first text block, select target sentences from the text content of the first text block that have a similarity higher than the preset keyword group, and use the target sentences as the text content of the filtered first text block;
[0026] Based on the range of text volume, the text content of the first filtered text block is reorganized to obtain multiple target text blocks.
[0027] In one embodiment, after reorganizing the text content of the filtered first text block according to the text volume range to obtain multiple target text blocks, the method further includes:
[0028] If the number of target text blocks exceeds the set number, assign a weight to each target text block based on the attention intensity of each target text block;
[0029] Based on the proportion of multiple target text blocks, the multiple target text blocks are reorganized into a second text block with the same number as the set number;
[0030] Use the second text block as the adjusted target text block.
[0031] In one embodiment, the method further includes:
[0032] The number of sentences in a text block is determined based on the input length limit of a pre-defined large language model;
[0033] For each target text block, determine the vector similarity between sentences within the target text block; determine the importance of each sentence within the target text block based on the vector similarity.
[0034] Sort the sentences in the target text block from highest to lowest importance to obtain a sentence list, and select the sentence that corresponds to the number of sentences in the text block from the sentence list;
[0035] The adjusted target text block is constructed based on the selected sentences.
[0036] Secondly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0037] Read the text content of a document;
[0038] The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity.
[0039] For each target text block, the number of words in the summary of the target text block is determined based on the intensity of attention it receives.
[0040] By using a pre-defined large language model, multiple rounds of summarization are performed based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate the generation of the summary, and the summary results corresponding to the document are output.
[0041] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0042] Read the text content of a document;
[0043] The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity.
[0044] For each target text block, the number of words in the summary of the target text block is determined based on the intensity of attention it receives.
[0045] By using a pre-defined large language model, multiple rounds of summarization are performed based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate the generation of the summary, and the summary results corresponding to the document are output.
[0046] Fourthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0047] Read the text content of a document;
[0048] The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity.
[0049] For each target text block, the number of words in the summary of the target text block is determined based on the intensity of attention it receives.
[0050] By using a pre-defined large language model, multiple rounds of summarization are performed based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate the generation of the summary, and the summary results corresponding to the document are output.
[0051] The aforementioned document summarization method, computer equipment, computer-readable storage medium, and computer program product read the text content of a document; segment and filter the text content to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity; for each target text block, the summarization word count is determined based on its attention intensity; and through a pre-set large language model, multiple rounds of summarization processing are performed based on the multiple target text blocks, the summarization word count of each target text block, and prompts used to indicate summarization generation, outputting the document's corresponding summary result. By determining the corresponding summarization word count based on the attention intensity of each text block in the document, more content from highly important text blocks can be preserved. The ability to adapt to the information distribution and importance of different documents based on the attention intensity of text blocks enhances the flexibility of summary generation. Personalized processing of different types of documents based on the attention intensity of text blocks improves the quality of the generated summaries. Multiple rounds of summarization processing through a pre-set large language model fully utilize the large language model's capabilities in understanding natural language, capturing semantic relationships, and generating fluent text, contributing to the generation of accurate and easily understandable summaries. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a flowchart illustrating a document summary generation method in one embodiment;
[0054] Figure 2 This is a flowchart illustrating the text block filtering step in one embodiment;
[0055] Figure 3 This is a schematic diagram illustrating the specific process of a document summary generation method in one embodiment;
[0056] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0058] In one exemplary embodiment, such as Figure 1 As shown, a document summarization method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. This embodiment does not impose any limitations on this. The document summarization method includes:
[0059] Step 102: Read the text content from the document.
[0060] This involves determining the document format and using a reading method that matches that format to read the text content from the document. For example, if the document format is txt or doc, the text content is read directly; if the document format is pdf or pptx, OCR (Optical Character Recognition) technology is used to recognize the text content in the document.
[0061] In one implementation, the document to be identified consists of multiple documents with different formats. The text content of each document is read using different reading methods, and the text content of the multiple documents is then integrated.
[0062] Step 104: The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity.
[0063] By segmenting the text content into blocks, the amount of input required for a single input to the pre-defined large language model can be reduced, thereby improving its processing efficiency. One implementation divides the text into different blocks by finding delimiters. Another implementation segments the text content into multiple equal-length target text blocks based on a pre-defined text length.
[0064] Content filtering is performed on multiple text blocks obtained from the segmentation process to obtain multiple target text blocks. This filtering process achieves content simplification, retaining relatively key text and removing redundant text. On the one hand, it allows the large language model to focus on processing text relevant to summarization, reducing wasted computational resources. On the other hand, it effectively reduces interfering information, improves the performance of the large language model, and thus enhances the quality of the summarization.
[0065] In this context, attention intensity refers to the degree of importance that the text content of the target text block demonstrates to the target topic, event, or viewpoint.
[0066] In one implementation, the attention intensity of each target text block is determined based on a list of attention intensity input by the user.
[0067] In one implementation, for each target text block, the attention intensity of the target text block is determined based on the ratio between the length of the text related to the target topic within that target text block and the total length of the target text block. The higher the ratio, the greater the attention intensity of the target text block. It is understood that multiple pre-defined mapping relationships between ratio ranges and attention intensities are established. By querying these mapping relationships, the ratio range corresponding to a given ratio, and the attention intensity corresponding to that ratio range, can be determined.
[0068] Step 106: For each target text block, determine the number of words in the summary of the target text block based on the attention intensity of the target text block.
[0069] The greater the attention intensity of a target text block, the longer its summary word count. In one implementation, the average summary value of each target text block is determined based on the summary word count limit and the number of target text blocks. The summary word count of each target text block is determined by the following formula: K = A + (sq) × p, where K represents the summary word count of the target text block, A represents the average summary value, s represents the attention intensity of the target text block, q represents the median attention intensity (which can be set according to actual needs), and p represents a preset adjustment coefficient (which can be set according to actual needs).
[0070] In one implementation, a word count for the summary of each target text block is set. If the attention intensity of a target text block exceeds a set intensity threshold, the word count of the summary for that target text block is increased. Optionally, the increase is performed using the following formula: K = k × (s / 10 × 100%), where K represents the adjusted word count of the summary, k represents the original word count of the summary, and s represents the maximum attention intensity among all target text blocks.
[0071] Step 108: Using a pre-set large language model, perform multiple rounds of summarization processing based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate the generation of the summary, and output the summary result corresponding to the document.
[0072] The "Preset Large Language Model" refers to a deep learning model trained on a large amount of text data, capable of generating natural language text or understanding the meaning of language text. The Preset Large Language Model can also be understood as a Large Model, or a Foundation Model, referring to a neural network model containing a very large number of parameters. In this embodiment, the Preset Large Language Model can be any open-source large language model; this embodiment does not impose any limitations on it.
[0073] In each round of summarization, input text is constructed based on the target text block, the number of words in the summary, and prompts used to indicate the summary generation. The input text serves as a prompt to guide the pre-defined large language model to understand the target text block and perform the summary generation task. The input text is provided to the pre-defined large language model, which then clarifies the summary generation task based on the input text, parses the target text block, and outputs the result according to the number of words in the summary.
[0074] In one implementation, during the i-th round of summarization, an input text is constructed based on the i-th target text block, the number of words in the summary of the i-th target text block, and prompts used to indicate summary generation. This input text is provided to a pre-defined large language model, which outputs a summary of the i-th target text block. The summaries of all target text blocks are then integrated to obtain the summary result corresponding to the document. Optionally, in the i-th round of summarization, the input to the pre-defined large language model also includes the summary generated in the (i-1)-th round.
[0075] The document summarization method described above involves reading the text content of a document; segmenting and filtering the text content to obtain multiple target text blocks; assigning a corresponding attention intensity to each target text block; determining the summarization word count for each target text block based on its attention intensity; and performing multiple rounds of summarization processing using a pre-defined large language model, based on the multiple target text blocks, the summarization word count for each target text block, and prompts used to indicate summarization generation, to output the document's corresponding summary result. By determining the corresponding summarization word count based on the attention intensity of each text block in the document, more content from highly important text blocks can be preserved. Adapting to the information distribution and importance of different documents based on the attention intensity of text blocks enhances the flexibility of summarization generation. Personalizing the processing of different types of documents based on the attention intensity of text blocks improves the quality of the generated summaries. Performing multiple rounds of summarization processing using a pre-defined large language model fully utilizes the large language model's capabilities in understanding natural language, capturing semantic relationships, and generating fluent text, contributing to the generation of accurate and easily understandable summaries.
[0076] In an exemplary embodiment, the steps of the i-th round of summarization include: determining a list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1; determining the input text block for the i-th round of summarization; wherein, the input text block for the first round of summarization is at least one target text block in the list of text blocks with an attention intensity not greater than i; the input text block for non-first rounds of summarization is the output text block of the (i-1)-th round and at least one target text block in the list of text blocks with an attention intensity not greater than i; determining the target summarization word count for the input text block; constructing input text based on the input text block, the target summarization word count, and prompt words used to indicate summarization generation; inputting the input text into a preset large language model to obtain summary text; dividing the summary text into blocks according to the input length limit of the preset large language model to obtain at least one summary text block; deleting at least one target text block with an attention intensity not greater than i from the list of text blocks; and if the sum of the number of summary text blocks and the number of target text blocks in the list of text blocks exceeds a set threshold, using at least one summary text block as the output text block for the i-th round.
[0077] Here, the i-th round of summarizing refers to any single summarizing process in the multi-round summarizing process, the (i-1)-th round refers to the round preceding the i-th round, and the first round refers to the first summarizing process in the multi-round summarizing process. It is understandable that because excessively long inputs will lead to a sharp increase in model resource usage and a decrease in model performance, the preset large language model has an input length limit, which represents the maximum text length that the preset large language model can accept and process.
[0078] The input text serves as a prompt to guide the pre-defined large language model in understanding the target text block and performing the summarization task. In one implementation, a summarization prompt template is provided, including prompt words to indicate summarization and multiple placeholders. The input text is obtained by filling the placeholders in the summarization prompt template with the input text block and the target summarization word count.
[0079] Understandably, the text block list stores target text blocks that were not used in the first i-1 rounds of summarization, i.e., target text blocks with an attention intensity greater than i-1. During the i-th round of summarization, processing is primarily focused on at least one target text block with an attention intensity no greater than i. After processing at least one target text block with an attention intensity no greater than i in the text block list based on the preset large language model, the target text block used in the i-th round of summarization, i.e., at least one target text block with an attention intensity no greater than i, is removed from the text block list.
[0080] The word count of the summary for each target text block is determined in advance according to step 106. The word count of the summary for the input text block is the sum of the word count of the target text blocks used in the first i-1 rounds of summary processing and the word count of the summary for at least one target text block in the text block list whose attention intensity is not greater than i.
[0081] In one implementation, the summary text generated by a preset large language model is simply processed to remove irrelevant information such as "The summary of this article is as follows:" and "The main content of this article is:", while retaining the basic content of the summary. The processed summary text is then divided into blocks according to the input length limit of the preset large language model. Specifically, the input length limit of the preset large language model is represented as a token, and the text size of each block after block processing falls within the range of 1 / 3 × token and 2 / 3 × token.
[0082] A threshold of 1 is set. The number of target text blocks in the text block list refers to the target text blocks that were not used in the first i rounds of summarization. If the sum of the number of summary text blocks and the number of target text blocks in the text block list exceeds the set threshold, at least one summary text block is used as the output text block for the i-th round. Based on the output text block of the i-th round and the text block list, the summarization process continues for the (i+1)-th round. This continues until the summary text block is unique and the text block list is empty. The summary text block obtained in the last round of summarization is then used as the summary result for the document.
[0083] In an exemplary embodiment, if the sum of the number of summary text blocks and the number of target text blocks in the text block list does not exceed a set threshold, the summary text block is used as the summary result corresponding to the document.
[0084] Among them, the set threshold is 1. If the sum of the number of abstract text blocks and the number of target text blocks in the text block list does not exceed 1, it indicates that the generated text block is unique, and the abstract text block is used as the abstract result corresponding to the document.
[0085] In an exemplary embodiment, step 104 includes: determining a text volume range according to the input length limit value of a preset large language model; dividing the text content into multiple first text blocks according to multiple levels of preset delimiters and the text volume range; screening and reorganizing the text content in the multiple first text blocks to obtain multiple target text blocks.
[0086] Among them, the input length limit value of the preset large language model is used to represent the maximum text length that the preset large language model can accept and process. The text volume range is the range formed by 1 / 3×token and 2 / 3×token, where token is the input length limit value of the preset large language model.
[0087] By searching for multiple levels of preset delimiters, the text before and after the preset delimiter in the document can be divided into different text blocks. The preset delimiter is a symbol set in advance for distinguishing and marking different text blocks in the document. According to the types or functions of the preset delimiters, multiple levels are set in advance. For example, the first level is a paragraph separator, the second level is a quotation mark, the second level is a period, the third level is a semicolon, and so on. In one implementation, assuming that there are M levels of preset delimiters, the specific chunking method is as follows:
[0088] Step A: Retrieve the preset delimiter of the c-th level (0 < c <= M, the initial value is 1). According to the position of the preset delimiter of the c-th level, divide the text before and after the delimiter into different blocks to obtain multiple text blocks;
[0089] Step B: Detect the text volume of the text block. If the text volume of the text block is greater than the maximum value of the text volume range, then c = c + 1, and return to Step A again;
[0090] Step C: If the text volume of the text block is less than the minimum value of the text volume range, then detect the text volume sizes of this text block and the text blocks before and after it, and merge this text block with the smaller text block among the text blocks before and after it. Repeat Step C until the text volume of the text block meets the requirements of the text volume range.
[0091] Among them, screening the text content in the first text block to achieve content refinement can retain relatively key text and remove redundant text. By reorganizing the screened text, the text volume input to the preset large language model at one time is sufficient, improving the efficiency of abstract generation.
[0092] In an exemplary embodiment, the text content in multiple first text blocks is filtered and reorganized to obtain multiple target text blocks, including: for each first text block, selecting target sentences from the text content of the first text block whose similarity to preset keyword groups is higher than a set similarity, and using the target sentences as the text content of the filtered first text block; and reorganizing the text content of the filtered first text blocks according to the text volume range to obtain multiple target text blocks.
[0093] The preset keyword group is a set of keywords pre-defined based on key domains and can be set according to actual needs. A similarity threshold is set in advance to distinguish the similarity between two objects. Content related to the key domain in the first text block is filtered out using the preset keyword group. In the specific implementation, each keyword in the keyword group and each sentence in the first text block (referring to sentence-level segments) are vectorized into text vectors. The text vector similarity between each keyword and each sentence is calculated, and target sentences with similarity higher than the set similarity threshold are selected and retained as the text content of the filtered first text block. In one implementation, the target sentence and the sentences before and after it are retained as the text content of the filtered first text block. This method ensures the relevance of key content and the relative integrity of the text segments.
[0094] In one implementation, the text content reorganization process can be as follows: if the text size of the first text block is less than the minimum value of the text size range, then the text size of the first text block and the preceding and following text blocks are detected, and the text block is merged with the smaller text block among the preceding and following text blocks, until the text size of all first text blocks meets the text size range requirement.
[0095] In an exemplary embodiment, after reorganizing the text content of the filtered first text block according to the text volume range to obtain multiple target text blocks, the method further includes: when the number of target text blocks is greater than a set number, assigning a weight to each target text block according to the attention intensity of each target text block; reorganizing the multiple target text blocks into a second text block with the same number as the set number according to the weight of the multiple target text blocks; and using the second text block as the adjusted target text block.
[0096] The set quantity is a pre-defined threshold value used to distinguish between large and small target text blocks. If the number of target text blocks exceeds the set quantity, it indicates a larger number of target text blocks. The number of target text blocks refers to the total number of text blocks after filtering and reorganization. Assuming the set quantity is F and the number of target text blocks is R, the R target text blocks need to be processed into F target text blocks. In the specific implementation, based on the proportion of attention intensity of each target text block in the overall text segment, a weight is assigned to the R target text blocks, with a total weight value of F. The R target text blocks are then merged into F second text blocks according to their weights, aiming to make the sum of the weights of each second text block equal to 1. For example, if all target text blocks have the same attention intensity in the overall text segment, then the weight assigned to each target text block is F / R. Another example: if the first 1 / 4 of the overall text segment receives more attention (attention intensity 3), then the weight of the first R / 4 blocks is set to 2F / R, and the weight of the remaining parts is set to 2F / 3R.
[0097] In one exemplary embodiment, such as Figure 2 As shown, the method further includes:
[0098] Step 202: Determine the number of sentences in the text block based on the input length limit of the preset large language model.
[0099] If the average data volume of each sentence-level segment is x, then the number of sentences retained in each text block (i.e. the number of sentences in the text block) is x1 = (1 / 3 × token) / x, where x1 represents the number of sentences in the text block and token is the input length limit value of the preset large language model.
[0100] Step 204: For each target text block, determine the vector similarity between sentences in the target text block; determine the importance of each sentence in the target text block based on the vector similarity.
[0101] Specifically, the vector representation of each sentence in the target text block is determined, and the vector similarity between sentences is calculated based on the vector representation of each sentence in the target text block. For any sentence in the target text block, its importance is the sum of the vector similarities between that sentence and other sentences.
[0102] Step 206: Sort the sentences in the target text block from highest to lowest importance to obtain a sentence list, and select the sentence that corresponds to the number of sentences in the text block from the sentence list.
[0103] The text block contains x1 sentences, and x1 sentences with the highest importance are selected from the sentence list. In one implementation, the sentence selection method follows the TextRank approach.
[0104] In one implementation, the data volume of x1 sentences is detected. If the data volume is greater than the first set data volume (indicating a relatively large data volume), then x1 = x1 - 1, and x1 sentences with a higher importance ranking are reselected according to step 206. If the data volume is less than the second set data volume (indicating a relatively small data volume), then x1 = x1 + 1, and x1 sentences with a higher importance ranking are reselected according to step 206.
[0105] Step 208, form an adjusted target text block according to the selected sentences.
[0106] Among them, x1 sentences with a higher importance in the target text block are used as the adjusted target text block.
[0107] In one implementation, x1 sentences with a higher importance in the target text block and the sentences before and after the x1 sentences are retained as the adjusted target text block. In this way, the relevance of key content and the relative integrity of the text segment are ensured.
[0108] In this embodiment, by selecting sentences with a higher importance to form the target text block, it can ensure that sentences with a higher importance in the text block are retained, retain key text, and remove redundant text, which is beneficial for the large language model to focus on processing text related to abstract generation and reduce waste of computing resources; it can effectively reduce interference information, improve the performance of the large language model, and thus improve the abstract quality.
[0109] In an exemplary embodiment, refer to Figure 3 , Figure 3 shows the specific process of a document abstract generation method. Among them,
[0110] Step 1: Read the text content in the document. Different reading methods are adopted for different types of documents. For example: for txt, doc and other files, the text content is directly read in, and for pdf files, pptx files, etc., the OCR technology is used to identify the text content according to requirements and integrate it with other text content.
[0111] Step 2: Complete the text content chunking operation by searching for delimiters. A type of file may have M levels of delimiters from top to bottom. The specific splitting method is as follows:
[0112] 2.1: Retrieve the delimiter at the c-th level (0 < c <= M, the initial value is 1). According to the position of the delimiter at the c-th level, the text before and after the delimiter is divided into different chunks to obtain multiple text chunks.
[0113] 2.2: Detect the text size of the text block. If the text size of the text block is greater than the maximum value of the text size range (the range formed by 1 / 3×token and 2 / 3×token, where token is the input length limit value of the preset large language model), then c=c+1, and return to step 2.1.
[0114] 2.3: If the text size of a text block is less than the minimum value of the text size range, then check the text size of the text block and the text blocks before and after it, merge the text block with the smaller text block between the preceding and following text blocks, and repeat step 2.3 until the text size of the text block meets the requirements of the text size range.
[0115] Step 3: First step of filtering: Vectorize the text of each keyword in the keyword group and each sentence in the text block (referring to the sentence-level paragraphs in the text block). Calculate the text vector similarity between each keyword and each sentence. Filter out target sentences with similarity higher than the set similarity threshold, retaining the target sentences and the sentences before and after them as the text content of the filtered text block. After completing this filtering step, repeat step 2.3 to ensure the amount of text data in each block.
[0116] Step 4: Second Filtering: If the number of text blocks is large after the first filtering step, proceed to this step. Assuming the number is F, if the number of text blocks R is greater than F, the R text blocks need to be processed into F text blocks. In this step, the attention intensity of each text block in the overall text summary needs to be considered. The specific steps are as follows:
[0117] 4.1 Assign weight values to R text blocks based on the proportion of attention intensity of each text block in the whole text segment, with a total weight value of F.
[0118] 4.2 Merge R text blocks into F text blocks according to their weight, so that the sum of the weights of each text block is 1 as much as possible.
[0119] 4.3 If the average data volume of each sentence-level segment is x, then the number of sentences retained in each text block (i.e. the number of sentences in the text block) is x1 = (1 / 3 × token) / x, where x1 represents the number of sentences in the text block.
[0120] 4.4 Use the textrank method to extract the x1 most important sentences in the text block and their context segments to ensure the relative coherence of the text block content.
[0121] 4.5 Detect the amount of data for x1 sentences. If the amount of data is relatively large, then x1-1 and return to step 4.4; if the amount of data is relatively small, then x1+1 and return to step 4.4.
[0122] Step 5: Summary Generation. The filtered text blocks are processed according to requirements to generate a prompt, adapted to the summary generation task of the large language model. The large language model then generates the summary. The prompt in this step can be modified based on the input text type and the requirements for the summary. The input to the large language model also includes the number of words in the summary of each text block. If the attention intensity of a text block exceeds a set threshold, the number of words in the summary of that text block is increased using the following formula: K = k × (s / 10 × 100%), where K represents the adjusted number of words in the summary, k represents the original number of words in the summary, and s represents the maximum attention intensity among all text blocks.
[0123] Step 6: Processing and Integration of the Generated Summary. The summary text generated by the preset large language model is simply processed to remove irrelevant information such as "The summary of this article is as follows:" and "The main content of this article is:", retaining only the basic summary content. The processed summary text is then divided into blocks according to the input length limit of the preset large language model. Specifically, the input length limit of the preset large language model is represented by a token, and the text size of each block after block processing falls within the range of 1 / 3 × token and 2 / 3 × token. In the i-th round of summary processing (where i is the processing round entering this step), text blocks containing text blocks with a focus intensity greater than i are not included in this round of merging and summary generation.
[0124] Repeat steps 5 and 6 until the generated summary block text is unique. Use the summary text block obtained from the last round of summary processing as the summary result corresponding to the document.
[0125] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0126] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a document summarization generation method.
[0127] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0128] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: reading text content from a document; segmenting and filtering the text content to obtain multiple target text blocks; setting a corresponding attention intensity for each target text block; determining the number of words in the summary of each target text block based on its attention intensity; and performing multiple rounds of summarization processing based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate summary generation using a preset large language model, and outputting a summary result corresponding to the document.
[0129] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1; determining the input text block for the i-th round of summarization processing; wherein, the input text block for the first round of summarization processing is at least one target text block in the list of text blocks with an attention intensity not greater than i; the input text block for non-first rounds of summarization processing is the output text block for the (i-1)-th round and at least one target text block in the list of text blocks with an attention intensity not greater than i; determining the target summarization word count for the input text block; constructing the input text based on the input text block, the target summarization word count, and prompt words used to indicate summarization generation; inputting the input text into a preset large language model to obtain the summary text; dividing the summary text into blocks according to the input length limit of the preset large language model to obtain at least one summary text block; deleting at least one target text block with an attention intensity not greater than i from the list of text blocks; and if the sum of the number of summary text blocks and the number of target text blocks in the list of text blocks exceeds a set threshold, using at least one summary text block as the output text block for the i-th round.
[0130] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the sum of the number of summary text blocks and the number of target text blocks in the text block list does not exceed a set threshold, the summary text block is used as the summary result corresponding to the document.
[0131] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the text volume range based on the input length limit value of a preset large language model; dividing the text content into blocks according to preset delimiters at multiple levels and the text volume range to obtain multiple first text blocks; filtering and reorganizing the text content in the multiple first text blocks to obtain multiple target text blocks.
[0132] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each first text block, select target sentences from the text content of the first text block that have a similarity higher than a preset keyword group, and use the target sentences as the text content of the selected first text block; reorganize the text content of the selected first text block according to the text volume range to obtain multiple target text blocks.
[0133] In one embodiment, when the processor executes the computer program, it further performs the following steps: when the number of target text blocks is greater than a set number, assigning a weight to each target text block according to the attention intensity of each target text block; reorganizing the multiple target text blocks into a second text block with the same number as the set number according to the weight of the multiple target text blocks; and using the second text block as the adjusted target text block.
[0134] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the number of sentences in the text block according to the input length limit of a preset large language model; determining the vector similarity between sentences in each target text block for each target text block; determining the importance of each sentence in the target text block according to the vector similarity; sorting the sentences in the target text block from high to low importance to obtain a sentence list; selecting sentences from the sentence list that are the number of sentences in the text block; and constructing an adjusted target text block based on the selected sentences.
[0135] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: reading text content in a document; segmenting and filtering the text content to obtain multiple target text blocks; setting a corresponding attention intensity for each target text block; determining the number of words in the summary of each target text block based on its attention intensity; and performing multiple rounds of summarization processing based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate summary generation using a preset large language model, and outputting the summary result corresponding to the document.
[0136] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1; determining the input text block for the i-th round of summarization processing; wherein, the input text block for the first round of summarization processing is at least one target text block in the list of text blocks with an attention intensity not greater than i; the input text block for non-first rounds of summarization processing is the output text block for the (i-1)-th round and at least one target text block in the list of text blocks with an attention intensity not greater than i; determining the target summarization word count for the input text block; constructing the input text based on the input text block, the target summarization word count, and prompt words used to indicate summarization generation; inputting the input text into a preset large language model to obtain the summary text; dividing the summary text into blocks according to the input length limit of the preset large language model to obtain at least one summary text block; deleting at least one target text block with an attention intensity not greater than i from the list of text blocks; and if the sum of the number of summary text blocks and the number of target text blocks in the list of text blocks exceeds a set threshold, using at least one summary text block as the output text block for the i-th round.
[0137] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the sum of the number of summary text blocks and the number of target text blocks in the text block list does not exceed a set threshold, the summary text block is used as the summary result corresponding to the document.
[0138] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the text volume range according to the input length limit value of a preset large language model; dividing the text content into blocks according to preset delimiters at multiple levels and the text volume range to obtain multiple first text blocks; filtering and reorganizing the text content in the multiple first text blocks to obtain multiple target text blocks.
[0139] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each first text block, select target sentences from the text content of the first text block that have a similarity higher than a preset keyword group, and use the target sentences as the text content of the selected first text block; reorganize the text content of the selected first text block according to the text volume range to obtain multiple target text blocks.
[0140] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the number of target text blocks is greater than a set number, assigning a weight to each target text block according to the attention intensity of each target text block; reorganizing the multiple target text blocks into a second text block with the same number as the set number according to the weight of the multiple target text blocks; and using the second text block as the adjusted target text block.
[0141] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the number of sentences in the text block according to the input length limit of a preset large language model; determining the vector similarity between sentences in each target text block for each target text block; determining the importance of each sentence in the target text block according to the vector similarity; sorting the sentences in the target text block from high to low importance to obtain a sentence list; selecting sentences from the sentence list that are the number of sentences in the text block; and constructing an adjusted target text block based on the selected sentences.
[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: reading text content from a document; segmenting and filtering the text content to obtain multiple target text blocks; setting a corresponding attention intensity for each target text block; determining the number of words in the summary of each target text block based on its attention intensity; and performing multiple rounds of summarization processing based on multiple target text blocks, the number of words in the summary of each target text block, and prompt words used to indicate summary generation using a preset large language model, and outputting the summary result corresponding to the document.
[0143] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1; determining the input text block for the i-th round of summarization processing; wherein, the input text block for the first round of summarization processing is at least one target text block in the list of text blocks with an attention intensity not greater than i; the input text block for non-first rounds of summarization processing is the output text block for the (i-1)-th round and at least one target text block in the list of text blocks with an attention intensity not greater than i; determining the target summarization word count for the input text block; constructing the input text based on the input text block, the target summarization word count, and prompt words used to indicate summarization generation; inputting the input text into a preset large language model to obtain the summary text; dividing the summary text into blocks according to the input length limit of the preset large language model to obtain at least one summary text block; deleting at least one target text block with an attention intensity not greater than i from the list of text blocks; and if the sum of the number of summary text blocks and the number of target text blocks in the list of text blocks exceeds a set threshold, using at least one summary text block as the output text block for the i-th round.
[0144] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the sum of the number of summary text blocks and the number of target text blocks in the text block list does not exceed a set threshold, the summary text block is used as the summary result corresponding to the document.
[0145] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the text volume range according to the input length limit value of a preset large language model; dividing the text content into blocks according to preset delimiters at multiple levels and the text volume range to obtain multiple first text blocks; filtering and reorganizing the text content in the multiple first text blocks to obtain multiple target text blocks.
[0146] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each first text block, select target sentences from the text content of the first text block that have a similarity higher than a preset keyword group, and use the target sentences as the text content of the selected first text block; reorganize the text content of the selected first text block according to the text volume range to obtain multiple target text blocks.
[0147] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the number of target text blocks is greater than a set number, assigning a weight to each target text block according to the attention intensity of each target text block; reorganizing the multiple target text blocks into a second text block with the same number as the set number according to the weight of the multiple target text blocks; and using the second text block as the adjusted target text block.
[0148] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the number of sentences in the text block according to the input length limit of a preset large language model; determining the vector similarity between sentences in each target text block for each target text block; determining the importance of each sentence in the target text block according to the vector similarity; sorting the sentences in the target text block from high to low importance to obtain a sentence list; selecting sentences from the sentence list that are the number of sentences in the text block; and constructing an adjusted target text block based on the selected sentences.
[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A document summarization method, characterized in that, The method includes: Read the text content of a document; The text content is segmented and filtered to obtain multiple target text blocks; each target text block is assigned a corresponding attention intensity. For each target text block, the number of words in the summary of the target text block is determined based on the attention intensity of the target text block; By using a pre-defined large language model, multiple rounds of summarization processing are performed based on the multiple target text blocks, the number of words in the summary of each target text block, and the prompt words used to indicate the generation of the summary, and the summary result corresponding to the document is output. The steps of the i-th round of abstract processing include: Determine a list of text blocks; the list of text blocks includes target text blocks with an attention intensity greater than i-1; Determine the input text block for the i-th round of summary processing; wherein, the input text block for the first round of summary processing is at least one target text block in the text block list with an attention intensity of no more than i; the input text block for non-first round summary processing is the output text block of the (i-1)-th round and at least one target text block in the text block list with an attention intensity of no more than i. Determine the target word count of the input text block; The input text is constructed based on the input text block, the target summary word count, and prompt words used to indicate summary generation; The input text is input into a preset large language model to obtain the summary text; The summary text is divided into blocks according to the input length limit of the preset large language model to obtain at least one summary text block; Remove at least one target text block from the list of text blocks whose attention intensity is not greater than i; If the sum of the number of the summary text blocks and the number of the target text blocks in the text block list exceeds a set threshold, the at least one summary text block will be used as the output text block for the i-th round.
2. The method according to claim 1, characterized in that, The method further includes: If the sum of the number of the summary text blocks and the number of the target text blocks in the text block list does not exceed a set threshold, the summary text block is used as the summary result corresponding to the document.
3. The method according to claim 1, characterized in that, The text content is segmented and filtered to obtain multiple target text blocks, including: The text volume range is determined based on the input length limit value of the preset large language model; The text content is divided into blocks according to multiple levels of preset delimiters and the text volume range to obtain multiple first text blocks; The text content in the plurality of first text blocks is filtered and reorganized to obtain a plurality of target text blocks.
4. The method according to claim 3, characterized in that, The process of filtering and reorganizing the text content in the plurality of first text blocks to obtain a plurality of target text blocks includes: For each first text block, select target sentences from the text content of the first text block that have a similarity higher than a set similarity with preset keyword groups, and use the target sentences as the text content of the filtered first text block; Based on the specified text volume range, the text content of the first filtered text block is reorganized to obtain multiple target text blocks.
5. The method according to claim 4, characterized in that, After reorganizing the text content of the filtered first text block according to the text volume range to obtain multiple target text blocks, the method further includes: If the number of target text blocks is greater than the set number, a weight is assigned to each target text block according to the attention intensity of each target text block; Based on the weight of the multiple target text blocks, the multiple target text blocks are reorganized into a second text block with the same number as the set number; Use the second text block as the adjusted target text block.
6. The method according to claim 4 or 5, characterized in that, The method further includes: The number of sentences in the text block is determined based on the input length limit value of the preset large language model; For each target text block, determine the vector similarity between sentences in the target text block; determine the importance of each sentence in the target text block based on the vector similarity. The sentences in the target text block are sorted from highest to lowest importance to obtain a sentence list, and sentences of the specified number from the sentence list are selected. The adjusted target text block is constructed based on the selected sentences.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.