Unsupervised extraction type abstract generation method based on word weight fusion and document comparison

Abstract

The invention relates to an unsupervised extraction type abstract generation method based on word weight fusion and document comparison, which comprises the following steps: acquiring a set consisting of documents, and embedding sentences in the documents by adopting a sentence encoder to obtain vector representation of the sentences; evaluating the importance of the words in the sentences by adopting a word weight dynamic evaluation method; evaluating the importance of the words in the sentences by adopting a word weight static evaluation method; the evaluation results of the two methods are fused; extracting importance of sentences by adopting a graph algorithm based on degree centrality to obtain significance scores of the sentences in each document, and screening a plurality of sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences to construct a positive example and a counter example for comparative learning; a cosine annealing strategy is adopted to dynamically adjust the temperature of comparative learning. Compared with the prior art, the unsupervised extraction type text abstract performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an unsupervised extractive summary generation method based on word weight fusion and document comparison. Background Art

[0002] Unsupervised extractive summarization is a key task in natural language processing (NLP) and data mining. It involves extracting key sentences or fragments directly from the original text to form a summary, without relying on labeled data. Its core idea is to automatically assess sentence importance through an algorithm, retaining the most representative content from the original text.

[0003] In practical applications, unsupervised extractive summarization can efficiently extract key information from massive amounts of text, which is of great significance in a variety of fields, including news aggregation, enterprise document processing, and customer support automation. For example, on news aggregation platforms, automatically extracting core sentences from multiple articles can quickly generate concise news overviews. In enterprise knowledge management, extracting key points from lengthy reports or meeting minutes helps employees quickly grasp key information.

[0004] Existing unsupervised extractive summarization methods primarily rely on statistical or graph algorithms, such as calculating sentence weights (e.g., TF-IDF, TextRank) or clustering similar sentences to identify the most representative content in a text. For example, patent application CN114996444A discloses a method and system for automatic news summarization. This approach uses sentence weights to filter and generate summaries by calculating sentence similarity to the title, sentence centrality, and inter-sentence similarity.

[0005] However, many of these existing research methods focus primarily on the importance of sentences in documents, potentially overlooking the importance of different keywords within a sentence. Furthermore, many methods generate summaries that are limited to the current document, potentially missing information that is crucial for understanding the document. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the existing unsupervised extractive summarization methods in the above-mentioned prior art, that is, they mainly focus on the importance of sentences in documents and may ignore the importance of different keywords in sentences, and to provide an unsupervised extractive summary generation method based on word weight fusion and document comparison.

[0007] The purpose of the present invention can be achieved by the following technical solutions: An unsupervised extractive summarization method based on word weight fusion and document comparison includes the following steps: obtaining a collection of documents, embedding sentences in the documents using a sentence encoder to obtain a vector representation of the sentences; evaluating the importance of words in sentences using a word weight dynamic evaluation method and a word weight static evaluation method; fusing the evaluation results of the word weight dynamic evaluation method and the word weight static evaluation method; extracting the importance of sentences using a graph algorithm based on degree centrality to obtain a significance score of sentences in each document, and screening multiple sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences in the candidate sentence matrix to construct positive and negative examples for comparative learning; Calculate a corresponding cosine similarity score for each sentence in a single document to be predicted; after sorting the sentences according to their scores, select the sentence with the highest score as the generated summary.

[0008] Furthermore, the optimization process of the method during training includes: After fusing the evaluation results of the word weight dynamic evaluation method and the word weight static evaluation method, the results are optimized using the mean square error (MSE) loss function. The obtained candidate sentence matrix is ​​optimized using the binary cross entropy loss function; When performing the contrastive learning, a cosine annealing strategy is used to dynamically adjust the temperature during contrastive learning, and optimization is performed through a cosine-based dynamic contrast loss function; The loss function is trained by accumulating the MSE mean square error loss function, the binary cross entropy loss function and the cosine-based dynamic contrast loss function.

[0009] Furthermore, the sentence encoder is a BERT model, which is used to obtain internal structure and context information of each sentence.

[0010] Furthermore, the word weight dynamic evaluation method performs dynamic evaluation of word weights through a word weight learning model, and the word weight learning model includes: GMLP module, used to preprocess sentence vectors; A linear layer is used to calculate the word frequency score of each word in the preprocessed sentence vector; Dropout layer, connected to the linear layer, is used to prevent overfitting of the word weight learning model; The LayerNorm layer is used to normalize all features output by the Dropout layer to obtain dynamic evaluation results of word weights.

[0011] Furthermore, the word weight static evaluation method performs a static evaluation of word weights according to the frequency of occurrence of words, and the specific process includes: Count the frequency of each word in the document to calculate the word frequency score of each word; Counting the number of documents in which each word appears in the collection and calculating the inverse document frequency weight of each word; Calculate a word weight score for each word based on the word frequency score and the inverse document frequency weight; For the sentence vectors in the document, the word weight score of each sentence vector is obtained by averaging the word weight scores of the words in each sentence vector.

[0012] Furthermore, the calculation expression of the word frequency score is: Where, is the word frequency score of the i-th word, is the frequency of occurrence of the i-th word in the document, is the total number of all words in the document; The calculation expression of the inverse document frequency weight is: Where, is the inverse document frequency weight of the i-th word, is the total number of documents in the collection, is the number of documents containing the word; The calculation expression of the word weight score of each word is: Where, is the word weight score of the i-th word.

[0013] Furthermore, the calculation process of the significance score of each sentence in each document includes: A graph structure is constructed based on documents. Each sentence in the graph structure is regarded as a node. The similarity between sentences is used as the weight of the edge between nodes. The similarity between each sentence and other sentences is calculated as the significance score result.

[0014] Furthermore, the similarity between the sentences is calculated by a sentence similarity scorer, which iteratively compares the average semantic content between the sentences in the document, and then determines the importance of each sentence or its relevance to other sentences to obtain the similarity calculation result between the sentences.

[0015] Furthermore, the process of constructing the positive and negative examples is as follows: The cosine similarity between candidate sentences from the same document in the candidate sentences is used as the positive example for contrastive learning, and the cosine similarity between candidate sentences from different documents in the candidate sentences is used as the counter example for contrastive learning.

[0016] Furthermore, the cosine annealing strategy uses a higher temperature in the initial stage of training of the unsupervised extractive summary generation method to encourage more extensive exploration, and gradually reduces the temperature as training progresses to focus more on capturing specific details and specific information of the data.

[0017] Compared with the prior art, the present invention has the following advantages: (1) The present invention first uses two different word weight evaluation methods: dynamic word weight evaluation method and static word weight evaluation method to measure the importance of words in sentences, and integrates their evaluation results to more scientifically evaluate the weight of words in sentences; in addition, by constructing positive examples of sentences within documents and negative examples of sentences between documents, the cosine annealing strategy is used to achieve dynamic temperature comparison with other documents, thereby expanding the diversity of generated summaries.

[0018] (2) The present invention adopts a cosine annealing strategy to dynamically adjust the temperature of contrastive learning. This strategy uses a higher temperature in the initial stage of training to encourage the model to conduct more extensive exploration, and gradually reduces the temperature as the training progresses, so that the model gradually focuses more on capturing the specific details and specific information of the data, thereby generating more accurate and targeted summaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of an unsupervised extractive summary generation method based on word weight fusion and document comparison provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a model processing method for an unsupervised extractive summary generation method based on word weight fusion and document comparison provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0023] Example 1 like Figure 1 and Figure 2 As shown, this embodiment provides an unsupervised extractive summary generation method based on word weight fusion and document comparison, including the following steps: S1: Obtain a collection of documents, use a sentence encoder to embed the sentences in the documents, and obtain a vector representation of the sentences; S2: The word weight dynamic evaluation method is used to evaluate the importance of words in a sentence. The word weight dynamic evaluation method uses a word weight learning model to dynamically evaluate word weights. S3: The word weight static evaluation method is used to evaluate the importance of words in sentences. The word weight static evaluation method performs static evaluation of word weights based on the frequency of word occurrence. S4: The evaluation results of the dynamic and static word weight evaluation methods are combined to obtain the final weight of the word in the sentence, which is optimized using the mean square error (MSE) loss function. S5: Use a degree centrality-based graph algorithm to extract the importance of sentences, obtain the significance score of each sentence in each document, and select the sentences with the highest scores as candidate sentences. A candidate sentence matrix is ​​obtained and optimized using a binary cross-entropy loss function. S6: Calculate the cosine similarity between each candidate sentence and other candidate sentences to construct positive and negative examples for comparative learning; S7: Adopting cosine annealing strategy to dynamically adjust the temperature during contrastive learning, and optimizing through cosine-based dynamic contrastive loss; S8: The loss functions of the three parts are accumulated to perform loss function training. After training, in the testing phase, the model is used to calculate a corresponding score for each sentence in a single document in the test set. After sorting the sentences according to their scores, the sentence with the highest score is selected as the generated summary.

[0024] In step S1, the sentence encoder is a BERT model, which is used to obtain the internal structure and context information of each sentence.

[0025] Specifically, given a set of N documents D = {D1, D2, ..., D N}, where each document D i There are n sentences {s1,s2,…,s n In order to capture the sentence information of the sentence, BERT is first used as a sentence encoder to embed each sentence in the document to obtain its vector representation v i , this embedding method can capture the internal structure and context information of the sentence, providing rich information for the subsequent summary task. Given by formula (1): (1) In steps S2-S4, two different word weighting methods were constructed to measure the importance of words in each sentence. The first, dynamic word weighting, uses a word weighting learning model to dynamically assess the importance of words in a sentence, while the second, static word weighting, accurately measures the importance of each word in a sentence. By combining the results of these two weighting methods, we can more comprehensively assess the criticality of various words in a document.

[0026] Specifically, in step S2, the dynamic word weight evaluation method uses a word weight learning model for dynamic evaluation. It consists of four components: a GMLP module, a linear layer, a dropout layer, and a layer norm layer. It uses a gating mechanism to capture complex nonlinear relationships, improving the model's adaptability and representation capabilities.

[0027] The GMLP module is used to preprocess the embedded sentence vectors to extract more word-level information, which helps in the subsequent calculation of word frequency scores; The linear layer, consisting of a fully connected layer, is used to calculate the word frequency score of each word, which reflects the importance and contribution of the word in the document; The Dropout layer is connected to the linear layer and is used to prevent the model from overfitting during training. It improves the generalization ability of the model by randomly setting some elements to 0. The LayerNorm layer is used to normalize all features output by the Dropout layer. The dynamic evaluation results of word weights are used to normalize all features output by Dropout, accelerate the training process, and stabilize the training and prediction of the model. This structure allows the simultaneous processing of word-level and sentence-level information, thereby better capturing the structure and semantics of the document.

[0028] In step S3, the word weight static evaluation method assigns an importance score to each word in the document through a word weight scorer. The specific process includes: Count the frequency of each word in the document, expressed as pc i, where i represents each word, and the word frequency score wpc of each word is obtained according to formula (2) i .

[0029] (2) Where, is the word frequency score of the i-th word, is the frequency of occurrence of the i-th word in the document, is the total number of all words in the document; Secondly, to account for the negative impact of common frequent words, such as "you," "I," and "is," on model learning, we introduced the Inverse Document Frequency (IDF) weighting, which adjusts the score weighting for common words that appear in the majority of documents. IDF reflects the rarity of a word by examining whether the word appears in most documents. If the word appears in most documents, its IDF value should be low, and vice versa. The calculation formula is as follows: (3) Where, is the inverse document frequency weight of the i-th word, is the total number of documents in the collection, is the number of documents containing the word; The calculated word frequency score wpc i Multiply it by the IDF weight to get the final word weight score wc for each word i The calculation formula is as follows.

[0030] (4) Where, is the word weight score of the i-th word.

[0031] Get word weight score wc i Finally, the word weight score of each sentence is obtained by averaging and substituting it into the word weight score. The final word weight score is multiplied by the amplification ratio A and rounded to two decimal places. This reduces the risk of the score being excessively biased towards zero.

[0032] In step S4, the MSE loss function is used to optimize the model by combining the two evaluation methods S2 and S3. The formula is as follows: (5) Where, Dynamic evaluation method for word weights j The word weight score of the composition, The static evaluation method for word weight is used to evaluate S j The word weight score of the composition.

[0033] In steps S5-S7, the core sentences of each document in the dataset are combined into a candidate sentence list. The cosine similarity between candidate sentences from the same document is used as positive examples for contrastive learning, and the cosine similarity between candidate sentences from different documents is used as negative examples for contrastive learning. The temperature is dynamically adjusted using a cosine annealing strategy, combined with contrastive learning to enhance the connection between sentences in the same document. This allows the importance of each sentence or its relevance to other sentences to be assessed.

[0034] In step S5, a graph algorithm based on degree centrality is used to extract the importance of sentences. Specifically, a graph structure is first constructed based on the document, in which each sentence is regarded as a node. The similarity between sentences determines the weight of the edges between them. In other words, the sentence with the highest similarity to other sentences is the sentence with the highest significance score, that is, the most important sentence. This embodiment uses the following degree centrality calculation formula for calculation: (6) Where, is the similarity between sentence i and sentence j; In order to further enhance the similarity representation of sentences, a sentence similarity scorer was introduced. This scorer can iteratively compare the average semantic content between sentences in a document, and then determine the importance of each sentence or its relevance to other sentences, thereby enhancing sentence similarity.

[0035] Finally, the sentences are divided into two categories of samples: important sentences and unimportant sentences by ranking the significance scores, and are optimized using the binary cross entropy loss function.

[0036] In step S6, the sentences from the same document and the sentences from different documents are compared and learned. First, assume that the text dataset consists of N documents D = {D1, D2, ..., D N Each document D i Contains n sentences, which are represented as S={s1,s2,...,s n By calculating the significance score of the sentence, we can i Then, the K sentences with the highest scores are selected from each document as candidate sentences, denoted as t ij , where i represents the document index and j represents the index of the candidate sentence in the document. Finally, we get an N × K candidate sentence matrix C, where each row represents K candidate sentences in a document, namely: (7) After obtaining the candidate sentence matrix C and entering the contrastive learning module, each selected sentence S is calculated.i The cosine similarity with all other N×K−1 sentences in the matrix C. The specific formula is as follows: (8) Among them, if s i With s j are sentences from the same document, then cos(s i , s j ) as positive examples, and vice versa as counter examples, that is, the cosine similarity between candidate sentences from the same document in the candidate sentences is used as the positive example of contrastive learning, and the cosine similarity between candidate sentences from different documents in the candidate sentences is used as the counter example of contrastive learning.

[0037] In step S7, the cosine annealing strategy is used to dynamically adjust the temperature during contrastive learning: this can further adjust the model's focus on positive and negative examples during training. Because in the early stages of summarization, the model tends to easily identify and rely on general patterns of the text, that is, the universal structure, common information patterns, and basic writing style in the text data. This reliance on general patterns may cause the summary to lack details and specificity. In order to enhance the generalization and robustness of the model and ensure that it can better capture the subtle differences and specific content of the text, the cosine annealing strategy is used to dynamically adjust the temperature. This strategy encourages the model to conduct more extensive exploration by using a higher temperature in the initial stage of training, and gradually reduces the temperature as training progresses, so that the model gradually focuses more on capturing the specific details and specific information of the data, thereby generating a more accurate and targeted summary. The specific temperature adjustment formula is as follows: (9) Where tem(min) and tem(max) are the minimum and maximum values ​​of temperature respectively, T current is the training step at the current moment, T total is the total training step, and i is the current moment.

[0038] Among them, the specific formula of cosine-based dynamic contrast loss is as follows: (10) Where, is the cosine-based dynamic contrast loss, For sentences from the same document and sentences The cosine similarity of For sentences from different documents and sentences The cosine similarity of tem(i) is the temperature after the cosine annealing strategy is adjusted at the current moment. In step S8, the loss functions of the three parts are accumulated to perform loss function training.

[0039] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An unsupervised extractive summary generation method based on word weight fusion and document comparison, characterized in that: The method comprises the following steps: obtaining a collection of documents, embedding sentences in the documents using a sentence encoder to obtain a vector representation of the sentences; evaluating the importance of words in sentences using a word weight dynamic evaluation method and a word weight static evaluation method; fusing the evaluation results of the word weight dynamic evaluation method and the word weight static evaluation method; extracting the importance of sentences using a graph algorithm based on degree centrality to obtain a significance score of sentences in each document, and screening multiple sentences with the highest scores as candidate sentences to obtain a candidate sentence matrix; calculating the cosine similarity between each candidate sentence and other candidate sentences in the candidate sentence matrix to construct positive and negative examples for comparative learning; Calculate a corresponding cosine similarity score for each sentence in a single document to be predicted; after sorting the sentences according to their scores, select the sentence with the highest score as the generated summary.

2. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The optimization process of the method during training includes: After fusing the evaluation results of the word weight dynamic evaluation method and the word weight static evaluation method, the results are optimized using the mean square error (MSE) loss function. The obtained candidate sentence matrix is ​​optimized using the binary cross entropy loss function; When performing the contrastive learning, a cosine annealing strategy is used to dynamically adjust the temperature during contrastive learning, and optimization is performed through a cosine-based dynamic contrast loss function; The loss function is trained by accumulating the MSE mean square error loss function, the binary cross entropy loss function and the cosine-based dynamic contrast loss function.

3. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The sentence encoder is a BERT model, which is used to obtain the internal structure and context information of each sentence.

4. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The word weight dynamic evaluation method performs dynamic evaluation of word weights through a word weight learning model, and the word weight learning model includes: GMLP module, used to preprocess sentence vectors; A linear layer is used to calculate the word frequency score of each word in the preprocessed sentence vector; Dropout layer, connected to the linear layer, is used to prevent overfitting of the word weight learning model; The LayerNorm layer is used to normalize all features output by the Dropout layer to obtain dynamic evaluation results of word weights.

5. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The word weight static evaluation method performs a static evaluation of word weights based on the frequency of occurrence of words. The specific process includes: Count the frequency of each word in the document to calculate the word frequency score of each word; Counting the number of documents in which each word appears in the collection and calculating the inverse document frequency weight of each word; Calculate a word weight score for each word based on the word frequency score and the inverse document frequency weight; For the sentence vectors in the document, the word weight score of each sentence vector is obtained by averaging the word weight scores of the words in each sentence vector.

6. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 5 is characterized in that: The calculation expression of the word frequency score is: Where, is the word frequency score of the i-th word, is the frequency of occurrence of the i-th word in the document, is the total number of all words in the document; The calculation expression of the inverse document frequency weight is: Where, is the inverse document frequency weight of the i-th word, is the total number of documents in the collection, is the number of documents containing the word; The calculation expression of the word weight score of each word is: Where, is the word weight score of the i-th word.

7. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The calculation process of the significance score of each sentence in the document includes: A graph structure is constructed based on documents. Each sentence in the graph structure is regarded as a node. The similarity between sentences is used as the weight of the edge between nodes. The similarity between each sentence and other sentences is calculated as the significance score result.

8. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 7 is characterized in that: The similarity between the sentences is calculated by a sentence similarity scorer, which iteratively compares the average semantic content between the sentences in the document, and then determines the importance of each sentence or its relevance to other sentences to obtain the similarity calculation result between the sentences.

9. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 1 is characterized in that: The specific process of constructing the positive and negative examples is as follows: The cosine similarity between candidate sentences from the same document in the candidate sentences is used as the positive example for contrastive learning, and the cosine similarity between candidate sentences from different documents in the candidate sentences is used as the counter example for contrastive learning.

10. The unsupervised extractive summary generation method based on word weight fusion and document comparison according to claim 2, characterized in that: The cosine annealing strategy uses a higher temperature in the initial stage of training of the unsupervised extractive summary generation method to encourage more extensive exploration, and gradually reduces the temperature as training progresses to focus more on capturing specific details and specific information of the data.

Images