An Unsupervised Extractive Summarization Method Based on Word Weight Fusion and Document Comparison

Abstract

This invention relates to an unsupervised extractive summarization method based on word weight fusion and document comparison, comprising: acquiring a set of documents; embedding sentences in the documents using a sentence encoder to obtain vector representations of the sentences; evaluating the importance of words in sentences using a dynamic word weight evaluation method; evaluating the importance of words in sentences using a static word weight evaluation method; fusing the evaluation results of the two methods; extracting sentence importance using a degree centrality-based graph algorithm to obtain a saliency score for sentences in each document, and selecting the sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences to construct positive and negative examples for comparative learning; and using a cosine annealing strategy to dynamically adjust the temperature of the comparative learning. Compared with existing technologies, this invention effectively improves the performance of unsupervised extractive text summarization.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an unsupervised extractive summarization method based on word weight fusion and document comparison. Background Technology

[0002] Unsupervised extractive summarization is an important task in the fields of Natural Language Processing (NLP) and data mining. It is a technique that selects key sentences or fragments directly from the original text to form a summary without relying on labeled data. Its core idea is to automatically evaluate the importance of sentences through algorithms and retain the most representative content from the original text.

[0003] In practical applications, unsupervised extractive summarization can efficiently extract key information from massive amounts of text, which is of great significance for many fields such as news aggregation, enterprise document processing, and customer support automation. For example, in news aggregation platforms, by automatically extracting the core sentences of multiple articles, a concise news overview can be quickly generated; in enterprise knowledge management, extracting key content from lengthy reports or meeting minutes helps employees quickly grasp key information.

[0004] Existing unsupervised extraction summarization methods are mainly based on statistical or graph algorithms, such as calculating sentence weights (e.g., TF-IDF, TextRank) or clustering similar sentences to identify the most representative content in the text. For example, the invention disclosed in CN114996444A discloses an automatic news summarization method and system, which calculates sentence weights by calculating the similarity between sentences and titles, sentence centrality, and the similarity between sentences, in order to filter sentences and obtain summaries.

[0005] However, many of these existing research methods focus primarily on the importance of sentences within a document, potentially overlooking the importance of different keywords within those sentences. Furthermore, many methods' summarization methods are limited to the information extracted from the current document, which may result in the loss of some information crucial for document comprehension. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing unsupervised extraction summarization methods, which mainly focus on the importance of sentences in a document and may ignore the importance of different keywords in a sentence, and to provide an unsupervised extraction summarization generation method based on word weight fusion and document comparison.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] An unsupervised extractive summarization method based on word weight fusion and document comparison includes the following steps: obtaining a set of documents; embedding sentences in the documents using a sentence encoder to obtain vector representations of the sentences; evaluating the importance of words in sentences using a dynamic word weight evaluation method and a static word weight evaluation method; fusing the evaluation results of the dynamic and static word weight evaluation methods; extracting sentence importance using a degree centrality-based graph algorithm to obtain a saliency score for sentences in each document, and selecting the sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences in the candidate sentence matrix to construct positive and negative examples for comparative learning.

[0009] Calculate a cosine similarity score for each sentence in a single document to be predicted; sort the sentences according to their scores, and select the sentence with the highest score as the generated summary.

[0010] Furthermore, the optimization process during training includes:

[0011] The evaluation results of the dynamic word weight evaluation method and the static word weight evaluation method are fused and then optimized using the mean squared error loss function (MSE).

[0012] The obtained candidate sentence matrix is ​​optimized using a binary cross-entropy loss function;

[0013] During the comparative learning process, a cosine annealing strategy is used to dynamically adjust the temperature during comparative learning, and optimization is performed using a cosine-based dynamic comparative loss function.

[0014] The loss function is trained by summing the MSE (mean squared error) loss function, the binary cross-entropy loss function, and the cosine-based dynamic contrast loss function.

[0015] Furthermore, the sentence encoder is a BERT model, used to obtain the internal structure and contextual information of each sentence.

[0016] Furthermore, the dynamic word weight evaluation method uses a word weight learning model to dynamically evaluate word weights, and the word weight learning model includes:

[0017] The GMLP module is used for preprocessing sentence vectors;

[0018] The linear layer is used to calculate the word frequency score of each word in the preprocessed sentence vector;

[0019] Dropout layers, which connect to linear layers, are used to prevent overfitting in word weight learning models.

[0020] The LayerNorm layer is used to normalize all features output by the Dropout layer, resulting in a dynamic evaluation of word weights.

[0021] Furthermore, the static word weight evaluation method performs static evaluation of word weights based on the frequency of word occurrence. The specific process includes:

[0022] The frequency of each word in the document is counted to calculate the word frequency score for each word;

[0023] The number of documents in which each word appears in the set is counted, and the inverse document frequency weight of each word is calculated.

[0024] Based on the word frequency score and the inverse document frequency weight, the word weight score of each word is calculated.

[0025] For sentence vectors in a document, the word weight score of each sentence vector is obtained by averaging the word weight scores of the words in each sentence vector.

[0026] Furthermore, the expression for calculating the word frequency score is as follows:

[0027]

[0028] In the formula, The word frequency score for the i-th word. Let be the frequency of the i-th word in the document. This represents the total number of all words in the document.

[0029] The expression for calculating the inverse document frequency weight is:

[0030]

[0031] In the formula, Let i be the inverse document frequency weight of the i-th word. The total number of documents in the collection. This represents the number of documents containing the word.

[0032] The expression for calculating the word weight score for each word is as follows:

[0033]

[0034] In the formula, The word weight score for the i-th word.

[0035] Furthermore, the calculation process for the saliency score of sentences in each document includes:

[0036] A graph structure is constructed based on the document, in which each sentence is regarded as a node, and the similarity between sentences is used as the weight of the edge between nodes. The similarity between each sentence and other sentences is calculated as the saliency score.

[0037] Furthermore, the similarity between the sentences is calculated by a sentence similarity scorer, which iteratively compares the average semantic content between each sentence in the document to determine the importance of each sentence or its relevance to other sentences, thereby obtaining the similarity calculation results between the sentences.

[0038] Furthermore, the process of constructing the positive and negative examples is as follows:

[0039] The cosine similarity between candidate sentences from the same document is used as a positive example for contrastive learning, while the cosine similarity between candidate sentences from different documents is used as a negative example.

[0040] Furthermore, the cosine annealing strategy uses a higher temperature in the initial training phase of the unsupervised extractive summarization method to encourage broader exploration, and gradually reduces the temperature as training progresses to focus more on capturing specific details and information in the data.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) This invention first uses two different word weight evaluation methods: dynamic word weight evaluation method and static word weight evaluation method to measure the importance of words in sentences and integrates their evaluation results to more scientifically evaluate the weight of words in sentences; in addition, by constructing positive examples of sentences within documents and negative examples of sentences between documents, a cosine annealing strategy is used to achieve dynamic temperature comparison with other documents, thereby expanding the diversity of generated summaries.

[0043] (2) The present invention adopts a cosine annealing strategy to dynamically adjust the temperature of contrastive learning. This strategy uses a higher temperature in the initial stage of training to encourage the model to explore more extensively, and gradually reduces the temperature as training progresses, so that the model gradually focuses more on capturing the specific details and specific information of the data, thereby generating more accurate and targeted summaries. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating an unsupervised extractive summarization method based on word weight fusion and document comparison provided in an embodiment of the present invention.

[0045] Figure 2This is a schematic diagram of the model processing for an unsupervised extractive summarization method based on word weight fusion and document comparison provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown, this embodiment provides an unsupervised extractive summarization method based on word weight fusion and document comparison, including the following steps:

[0051] S1: Obtain a collection of documents, use a sentence encoder to embed sentences in the documents, and obtain vector representations of the sentences;

[0052] S2: The importance of words in a sentence is evaluated using the dynamic evaluation method of word weight. The dynamic evaluation method of word weight uses a word weight learning model to dynamically evaluate word weight.

[0053] S3: The word weight static evaluation method is used to evaluate the importance of words in a sentence. The word weight static evaluation method evaluates the word weight based on the frequency of word occurrence.

[0054] S4: The evaluation results of the dynamic word weight evaluation method and the static word weight evaluation method are fused to obtain the final weight of the word in the sentence, and then optimized by the mean squared error loss function (MSE).

[0055] S5: A degree-centrality-based graph algorithm is used to extract the importance of sentences, obtain the saliency score of sentences in each document, and select the sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix, which is then optimized using the binary cross-entropy loss function;

[0056] S6: Calculate the cosine similarity between each candidate sentence and other candidate sentences to construct positive and negative examples for comparative learning;

[0057] S7: A cosine annealing strategy is used to dynamically adjust the temperature during contrastive learning, and optimization is achieved through cosine-based dynamic contrastive loss;

[0058] S8: The loss functions from the three parts are summed to train the loss function. After training, during the testing phase, the model is used to calculate a corresponding score for each sentence in a single document of the test set. After sorting the sentences according to their scores, the sentence with the highest score is selected as the generated summary.

[0059] In step S1, the sentence encoder is a BERT model, which is used to obtain the internal structure and contextual information of each sentence.

[0060] Specifically, given a set D = {D1, D2, ..., D...} consisting of N documents. N}, where each document D i Given n sentences {s1, s2, ..., s...} n The document is composed of several parts. To capture sentence information, BERT is first used as a sentence encoder to embed each sentence in the document, thus obtaining its vector representation v. i This embedding method can capture the internal structure and contextual information of sentences, providing rich information for subsequent summarization tasks. This is given by formula (1):

[0061] (1)

[0062] In steps S2-S4, two different word weighting evaluation methods are constructed to measure the importance of words in each sentence. The first method, dynamic word weighting, dynamically evaluates the importance of words in a sentence using a word weighting learning model, while the second method, static word weighting, accurately measures the importance of each word in a sentence. By fusing the results of the two weighting evaluation methods, the criticality of various words in the document is evaluated more comprehensively.

[0063] Specifically, in step S2, the dynamic word weight evaluation method mainly uses a word weight learning model for dynamic evaluation. It comprises four components: a GMLP module, a linear layer, Dropout, and a LayerNorm layer. It captures complex nonlinear relationships through a gating mechanism, improving the model's adaptability and representational capabilities.

[0064] The GMLP module is used to preprocess the embedded sentence vectors to extract more word-level information, which helps in the subsequent calculation of word frequency scores.

[0065] The linear layer, consisting of a fully connected layer, is used to calculate the word frequency score for each word, thus reflecting the importance and contribution of the word in the document;

[0066] Dropout layers, which connect to linear layers, are used during training to prevent overfitting by randomly setting some elements to 0, thereby improving the model's generalization ability.

[0067] The LayerNorm layer is used to normalize all features output by the Dropout layer, resulting in a dynamic evaluation of word weights. This normalization of Dropout output accelerates the training process and stabilizes model training and prediction. This structure allows for the simultaneous processing of word-level and sentence-level information, thus better capturing the structure and semantics of a document.

[0068] In step S3, the word weight static evaluation method assigns an importance score to each word in the document using a word weight scorer. The specific process includes:

[0069] The frequency of each word in the document is represented as pc. i Where i represents each word, the word frequency score wpc of each word is obtained according to formula (2). i .

[0070] (2)

[0071] In the formula, The word frequency score for the i-th word. Let be the frequency of the i-th word in the document. This represents the total number of all words in the document.

[0072] Secondly, to account for the adverse effects of common, frequent words on model learning, such as "you," "I," and "is," an Inverse Document Frequency (IDF) weight is introduced. This weight is used to adjust the score weights of common words that appear in most documents. IDF reflects the rarity of a word. By examining whether a word appears in most documents, its IDF value should be low if it appears in most documents, and high if it does. The formula is calculated as follows:

[0073] (3)

[0074] In the formula, Let i be the inverse document frequency weight of the i-th word. The total number of documents in the collection. This represents the number of documents containing the word.

[0075] The calculated word frequency score WPC i Multiplying by the IDF weights yields the final word weight score (wc) for each word. i The formula is calculated as follows.

[0076] (4)

[0077] In the formula, The word weight score for the i-th word.

[0078] Obtain the word weight score wc i Then, the word weight score for each sentence is obtained by averaging the results. The final word weight score is then multiplied by the amplification ratio A and rounded to two decimal places. This reduces the risk of the score being overly biased towards zero.

[0079] In step S4, optimization is performed using the MSE (mean squared error) loss function, combined with the evaluation methods in S2 and S3; the formula is as follows:

[0080] (5)

[0081] In the formula, For the dynamic evaluation method of word weights on S j The resulting word weight score For the word weight static evaluation method for S j The resulting word weight score.

[0082] In steps S5-S7, a candidate sentence list is created by assembling the core sentences of each document in the dataset. Cosine similarity between candidate sentences from the same document is used as positive examples for contrastive learning, while cosine similarity between candidate sentences from different documents is used as negative examples. A cosine annealing strategy is used to dynamically adjust the temperature, combining contrastive learning to enhance the connections between sentences within the same document. This process evaluates the importance of each sentence and its relevance to other sentences.

[0083] In step S5, a graph algorithm based on degree centrality is used to extract sentence importance. Specifically, a graph structure is first constructed based on the document, where each sentence is considered a node. The similarity between sentences determines the weight of the edges between them. In other words, the sentence with the highest similarity to other sentences is the sentence with the highest saliency score, i.e., the most important sentence. This embodiment uses the following degree centrality calculation formula for calculation:

[0084] (6)

[0085] In the formula, The similarity between sentence i and sentence j;

[0086] To further enhance sentence similarity representation, a sentence similarity scorer is introduced. This scorer iteratively compares the average semantic content of each sentence in the document to determine the importance of each sentence or its relevance to other sentences, thereby enhancing sentence similarity.

[0087] Finally, sentences are divided into two categories, important sentences and unimportant sentences, based on saliency score ranking, and optimized using the binary cross-entropy loss function.

[0088] In step S6, sentences from the same document and sentences from different documents are compared and learned. First, assume the text dataset consists of N documents D={D1,D2,...,D...} N Each document D i It contains n sentences, which can be represented as S = {s1, s2, ..., s}. n}. For each document D, the saliency score is calculated by examining the sentences. i The sentences in the document are scored and sorted. Then, the K highest-scoring sentences from each document are selected as candidate sentences, denoted as t. ij , where i represents the document index and j represents the index of the candidate sentence in the document. Finally, an N × K candidate sentence matrix C is obtained, where each row represents K candidate sentences for a document, i.e.:

[0089] (7)

[0090] After obtaining the candidate sentence matrix C and entering the contrastive learning module, the calculation of each selected sentence S is then performed. i The cosine similarity with all other N×K−1 sentences in matrix C. The specific formula is as follows:

[0091] (8)

[0092] Where, if s i With sj If the sentences are from the same document, then cos(s) i , s j The cosine similarity between candidate sentences from the same document is used as a positive example, and the cosine similarity between candidate sentences from different documents is used as a negative example.

[0093] In step S7, a cosine annealing strategy is used to dynamically adjust the temperature during contrastive learning. This further adjusts the model's focus on positive and negative examples during training. Because in the early stages of summarizing, the model tends to easily identify and rely on general patterns in the text—that is, the common structures, information patterns, and basic writing styles in the text data. This reliance on general patterns may lead to a lack of detail and specificity in the summarization. To enhance the model's generalization ability and robustness, and to ensure it can better capture subtle differences and specific content in the text, a cosine annealing strategy is used to dynamically adjust the temperature. This strategy encourages the model to explore more broadly by using a higher temperature in the initial training phase, and gradually decreases the temperature as training progresses, allowing the model to focus more on capturing specific details and information in the data, thereby generating more accurate and targeted summaries. The specific temperature adjustment formula is shown below:

[0094] (9)

[0095] Where tem(min) and tem(max) are the minimum and maximum temperatures, respectively, T current For the training steps at the current moment, T total Let i represent the total training steps, and i represent the current time step.

[0096] The specific formula for the cosine-based dynamic contrast loss is as follows:

[0097] (10)

[0098] In the formula, For cosine-based dynamic contrast loss, Sentences from the same document and sentences cosine similarity, For sentences from different documents and sentences The cosine similarity is calculated, where tem(i) is the temperature adjusted by the cosine annealing strategy at the current time. In step S8, the loss functions of the three parts are summed to train the loss function.

[0099] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An unsupervised extractive summarization method based on word weight fusion and document comparison, characterized in that, The process includes the following steps: obtaining a set of documents; embedding sentences in the documents using a sentence encoder to obtain vector representations of the sentences; evaluating the importance of words in sentences using a dynamic word weighting method and a static word weighting method; fusing the evaluation results of the dynamic and static word weighting methods; extracting sentence importance using a degree-centrality-based graph algorithm to obtain a saliency score for sentences in each document, and selecting the sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences in the candidate sentence matrix to construct positive and negative examples for comparative learning. Calculate a cosine similarity score for each sentence in a single document to be predicted; sort the sentences according to their scores and select the sentence with the highest score as the generated summary. The dynamic word weight evaluation method uses a word weight learning model to dynamically evaluate word weights. The word weight learning model includes: The GMLP module is used for preprocessing sentence vectors; The linear layer is used to calculate the word frequency score of each word in the preprocessed sentence vector; Dropout layers, which connect to linear layers, are used to prevent overfitting in word weight learning models. The LayerNorm layer is used to normalize all features output by the Dropout layer to obtain dynamic evaluation results of word weights. The static word weight evaluation method evaluates word weights based on the frequency of word occurrence. The specific process includes: The frequency of each word in the document is counted to calculate the word frequency score for each word; The number of documents in which each word appears in the set is counted, and the inverse document frequency weight of each word is calculated. Based on the word frequency score and the inverse document frequency weight, the word weight score of each word is calculated. For sentence vectors in a document, the word weight score of each sentence vector is obtained by averaging the word weight scores of the words in each sentence vector.

2. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 1, characterized in that, The optimization process of the method during training includes: The evaluation results of the dynamic word weight evaluation method and the static word weight evaluation method are fused and then optimized using the mean squared error loss function (MSE). The obtained candidate sentence matrix is ​​optimized using a binary cross-entropy loss function; During the comparative learning process, a cosine annealing strategy is used to dynamically adjust the temperature during comparative learning, and optimization is performed using a cosine-based dynamic comparative loss function. The loss function is trained by summing the MSE (mean squared error) loss function, the binary cross-entropy loss function, and the cosine-based dynamic contrast loss function.

3. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 1, characterized in that, The sentence encoder is a BERT model, used to obtain the internal structure and contextual information of each sentence.

4. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 1, characterized in that, The expression for calculating the word frequency score is as follows: In the formula, The word frequency score for the i-th word. Let be the frequency of the i-th word in the document. This represents the total number of all words in the document. The expression for calculating the inverse document frequency weight is: In the formula, Let i be the inverse document frequency weight of the i-th word. The total number of documents in the collection. This represents the number of documents containing the word. The expression for calculating the word weight score for each word is as follows: In the formula, The word weight score for the i-th word.

5. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 1, characterized in that, The calculation process for the saliency score of sentences in each document includes: A graph structure is constructed based on the document, in which each sentence is regarded as a node, and the similarity between sentences is used as the weight of the edge between nodes. The similarity between each sentence and other sentences is calculated as the saliency score.

6. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 5, characterized in that, The similarity between sentences is calculated by a sentence similarity scorer, which iteratively compares the average semantic content of each sentence in the document to determine the importance of each sentence or its relevance to other sentences, thus obtaining the similarity calculation results between sentences.

7. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 1, characterized in that, The process of constructing the positive and negative examples is as follows: The cosine similarity between candidate sentences from the same document is used as a positive example for contrastive learning, while the cosine similarity between candidate sentences from different documents is used as a negative example.

8. The unsupervised extractive summarization method based on word weight fusion and document comparison according to claim 2, characterized in that, The cosine annealing strategy uses a higher temperature in the initial training phase of the unsupervised extractive summarization method to encourage broader exploration, and gradually reduces the temperature as training progresses to focus more on capturing specific details and information in the data.

Images