Extractive summarization method and system incorporating local topic and hierarchical information

Abstract

The application belongs to the technical field of text abstract extraction, and discloses an extractive abstract method and system combining local theme and hierarchical structure information. First, a given original document is obtained through an encoder to obtain the context representation of the document, which is input into a local theme information extraction module to extract the theme information of the segment to which the sentence belongs, and the local theme information representation and the context representation of the document are fused to obtain the text context representation of the fused local theme information. A text hierarchical structure information embedding module embeds the hierarchical structure information of the text into the text context representation of the fused local theme information. The confidence score of each sentence is calculated through a Sigmoid layer to determine whether the sentence belongs to an abstract sentence. The application pays more attention to important parts of the text and improves the quality of generated abstracts. By modifying TokenPositionEmbeddings of the Longformer, the Longformer can process longer text data.

Description

Technical Field

[0001] This invention belongs to the field of text summarization technology, and particularly relates to an extractive summarization method and system that combines local topic and hierarchical structure information. Background Technology

[0002] Currently, with the rapid development of internet technology and the arrival of the knowledge exploration era, people can use information retrieval engines such as Google, Yahoo, and Baidu to search for data. However, the massive amount of text information and documents on the internet is overwhelming, which has sparked a surge of research into developing technologies that can automatically summarize text. Thus, text summarization technology was born. Because it requires precise text analysis, such as semantic and lexical analysis, to produce good summaries, text summarization is a significant challenge in the field of Natural Language Processing (NLP). The purpose of text summarization is to generate concise, coherent, and important sentences that include all relevant information from the original document. It effectively reduces the information burden on users, enabling them to quickly extract information from redundant data, greatly reducing human and material resources, and playing a crucial role in information retrieval, title generation, and other fields.

[0003] Text summarization tasks can be categorized into different types based on various criteria. According to the number of documents, they are divided into single-document text summarization and multi-document text summarization. Based on implementation method, they can be divided into extractive summarization and abstractive summarization. Extractive summarization methods are often treated as a classification problem, directly selecting sentences from the original text based on their importance to form a summary. Typically, summaries generated in this way perform well in terms of fluency and grammar, but may contain a lot of redundant content and lack coherence between sentences. In contrast, neural network-based generative summarization methods usually employ a sequence-to-sequence (Seq2Seq) architecture, i.e., an encoder-decoder architecture, which is similar to human summarization. First, the encoder encodes the entire text, and then the decoder generates new sentences word by word to form a document summary. This method generates summaries with less redundant information, but because sentences are generated from scratch, their performance in terms of fluency and grammar is relatively poor. Furthermore, generating new words or phrases may produce summaries inconsistent with the original text. Conversely, extractive summarization, which directly selects sentences from the original text and assembles them into a summary, can address this issue. The core task of extractive summarization is to learn long-range sentence context information and model long-range inter-sentence relationships through the encoder, enabling the sentence classifier to extract more valuable sentences. Traditional extractive methods typically use graph-based or clustering-based methods for unsupervised summary extraction. These methods construct relevance relationships between sentences using cosine similarity and then apply ranking methods to calculate sentence importance. With the rapid development of deep learning, many extractive summarization methods employ recurrent neural networks (RNNs) to capture inter-sentence relationships. However, RNN-based methods struggle to handle long-distance dependencies, especially for long document summarization. In recent years, Transformer language models pre-trained on large-scale corpora have achieved excellent results in fine-tuning downstream tasks, and Transformer-based pre-trained language models are widely used in the field of text summarization. Liu et al. proposed the BERTSUM model by improving the BERT embedding layer, and applied the BERT model to the field of text summarization for the first time, achieving state-of-the-art (SOTA) results on the CNN / DailyMail dataset.Zhang et al. designed a hierarchical Transformer to capture long-range inter-sentence relationships, but this method did not achieve significant performance gains for summarizing tasks and suffered from slow training speed and overfitting. Meanwhile, some researchers have introduced Neural Topic Models (NTM) and Graph Neural Networks (GNNs) into text summarizing tasks to capture global semantic information and further guide summary generation. Cui et al. used NTM to capture document topic features and used GNNs to represent documents as graph structures, thereby obtaining inter-sentence relationships.

[0004] With the rapid development of neural networks, extractive summarization tasks have achieved significant results. Currently, extractive summarization methods are mainly considered as sentence ranking tasks and sequence labeling binary classification tasks. In the sentence ranking paradigm, the model needs to score each sentence in the text and place sentences with high scores at the beginning of the summary list and sentences with low scores at the end. This results in an ordered list of sentences, and the first few sentences in the list are then used as the summary. Narayan et al. proposed a topic-aware convolutional neural network model, which first uses a convolutional neural network to extract features from the document, then weights the features according to the topic, and finally uses a selection ranking method to select the most relevant sentences as the summary. Experiments on multiple datasets show that it can produce very concise yet informative text summaries. Li et al. proposed a method for evaluating sentence importance in multi-document summarization using variational autoencoders. Unlike traditional feature engineering-based methods, this method directly learns abstract semantic representations from the raw data and introduces KL divergence to constrain the proximity of the generated sentence representations to the prior distribution, thereby improving the model's generalization ability.

[0005] The second paradigm treats extractive text summarization as a sequence labeling problem. This method extracts and encodes features for each sentence or paragraph, then inputs them into a decoder for label prediction to determine which sentences are selected as summary sentences. Sequence labeling methods are widely used in extractive text summarization and have achieved good results. Nallapati et al. proposed the SummaRuNNer text summarization model, an RNN-based sequence model that generates document summaries by learning the importance of each sentence in the document. This model achieved good summarization performance on multiple text datasets. Zhang et al. proposed a latent variable extraction model that treats sentences as latent variables and uses sentences with activation variables to infer summaries.

[0006] However, most of the above methods are RNN-based extractive summarization methods. Since RNN-based methods struggle to handle long-distance dependencies at the sentence level and may miss some linguistic or structural information due to the input format of the original document, some researchers have begun using Transformer-based pre-trained language models as encoders and representing documents through a more intuitive graph structure. They have also added NTM to extract topic features from the documents to further guide the model to generate high-quality summaries. Jia et al. proposed a deep differential amplifier for summarization, using a differential amplifier to enhance the features of the summarized sentences, thus contrasting them with non-summary sentences. Shi et al. proposed a star-shaped extractive summarization method, modeling sentences in a document as satellite nodes and introducing a virtual central node. The star structure is used to learn the inter-sentence relationships in each document, achieving good results on three public datasets. Ma et al. embedded the topic features extracted by NTM into BERT to generate vector representations with topic features, thereby improving the summarization quality.

[0007] While the above methods have been successful in modeling inter-sentence relationships and extracting global semantics, extractive text summarization methods based on Transformer pre-trained language models still face a problem in text summarization tasks: the input text for text summarization tasks is much longer than that of general natural language processing tasks. Using only Transformer-based encoders cannot adequately handle long texts and suffers from high computational costs. To better understand the original text, researchers have proposed many advanced methods. Xie et al. first preprocessed the document by dividing it into blocks of equal size, then encoded each document block using block encoding. The block encoding results were integrated into an NTM to generate global topic features. Finally, a comparison graph was built between topic features and sentence features to filter the summary sentences. This method achieved good results on both long texts and short news texts, but it is more advantageous for long texts. Beltagy et al. proposed a Longformer model focused on processing long documents. By replacing the Transformer's self-attention mechanism with a sliding window self-attention mechanism, the time complexity was reduced to linear, allowing the model to easily handle long texts. While Longformer handles long texts well, it fails to model local semantic information and document hierarchical structure, thus impacting performance. Therefore, this invention uses Longformer as the encoder, incorporating local contextual information of the current topic segment and hierarchical structure information of the document. This allows the model to pay more attention to local topic information and the overall structural information of the entire text when processing long scientific papers.

[0008] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0009] (1) Traditional Transformer language models usually treat text as a linear sequence, ignoring the inherent hierarchical structure information of the text.

[0010] (2) For long text data, there may be more topics described, because each chapter expresses different topic information. However, the Transformer language model still has limitations in capturing and integrating local contextual information in topic fragments.

[0011] (3) Existing encoders cannot handle long texts well. Summary of the Invention

[0012] To address the problems existing in the prior art, this invention provides an extractive summarization method and system that combines local topic and hierarchical structure information.

[0013] This invention is implemented as follows: an extractive summarization method combining local topic and hierarchical structure information includes:

[0014] Step 1: Given the original document D = {sent1, ...,sent...} n}, where sent n This represents the nth sentence in the original document;

[0015] Step two: Obtain the contextual representation of the document through the encoder, and input it into the local topic information extraction module to extract the topic information of the segment to which the sentence belongs;

[0016] Step 3: Merge the local topic information representation and the document context representation to obtain a text context representation that integrates the local topic information;

[0017] Step four: The text hierarchical structure information embedding module embeds the hierarchical structure information of the text into the text context representation that integrates local topic information. Through two stacked Transformer layers, it learns the document-level hierarchical structure information of sentences, enabling the model to gain a deeper understanding of the text context structure.

[0018] Step 5: Calculate the confidence score of each sentence using the Sigmoid layer to determine whether the sentence belongs to the summary sentence.

[0019] Furthermore, in step one, [BOS] and [EOS] tags are inserted at the beginning and end of each sentence, respectively, with the [BOS] tag representing the entire sentence.

[0020] Furthermore, in step two, a Longformer pre-trained language model is used as a text encoder, and the model embedding layer includes TE: TokenEmbeddings, SE: SegmentEmbeddings, and PE: PositionEmbeddings.

[0021] w i,j = (TE + SE + PE)

[0022] The above formula yields the embedding representation of each word, and the pre-trained Longformer is used to perform context learning on the input sequence;

[0023] {h 1,0 ,h 1,1 ,…,h N,0 ,…,h N,*} = Longformer(w 1,0 ,w 1,1 ,…,w N,0 ,…,w N,* )

[0024] Among them, w i,j w represents the j-th word of the i-th sentence. i,0 and w i,* The [BOS] and [EOS] tags represent the i-th sentence, respectively. i,j This represents the hidden state of the corresponding word; after Longformer encoding, the [BOS] tag is used as the context representation of each sentence, i.e., H. s =(h 1,0 ,…,h N,0 ).

[0025] Furthermore, the specific steps for the local topic information extraction module in step two to extract the topic information of the segment to which the sentence belongs include:

[0026] The hidden vector representation of each sentence is obtained by encoding the sentence context using Bi-LSTM;

[0027] The local context information of the topic segment to which the sentence belongs is represented by the subtraction between the start and end hidden vectors of each topic segment. For the i-th topic segment t i Its specific representation method is as follows:

[0028]

[0029]

[0030] t i =(f i |b i )

[0031] Among them, f i and b i These represent the thematic segments of forward propagation and backward propagation, respectively. i and end iThese represent the start and end positions of the topic segment, respectively, and | represents the vector concatenation symbol;

[0032] Add zero vectors at the beginning and end of both forward and backward propagation to prevent indices from going out of bounds.

[0033] Furthermore, the hierarchical structure information of the text in step four includes sentence hierarchical structure information and chapter title information;

[0034] The sentence hierarchical structure information includes the linear position of the paragraph to which the sentence belongs and the linear position representation of the sentence within the paragraph;

[0035] The position is represented by the numerical index corresponding to the paragraph and sentence. For a given document D = {sent1,…,sent1}, ...,sent2, ...,sent3, ...,sent4, ...,sent5, ...,sent6, ...,sent7, ...,sent8, ...,sent9 ... n}, the i-th sentence sent i The hierarchical information is represented as a two-dimensional vector (s s ,g s This two-dimensional vector represents the position of the sentence at this level, specifically:

[0036] vsent i =(s s ,g s )

[0037] Among them, s s G represents the linear position of the paragraph containing the sentence relative to the entire article. s This indicates the linear position of the sentence within its paragraph.

[0038] The chapter title information is based on multiple preset chapter title categories according to the title features of the PubMed dataset. If the chapter title of the sentence does not belong to one of the preset chapter title categories, the chapter title of the sentence itself is used directly.

[0039] Furthermore, in step four, the text hierarchy information embedding module uses the BERT positional encoding method to encode the vsent vector;

[0040] For the hierarchical structure vector (s) of the i-th sentence s ,g s The expression is as follows:

[0041]

[0042] Where PE represents the BERT positional encoding method, d represents the vector dimension of the sentence, and | represents the vector concatenation symbol;

[0043] The same pre-trained encoder as the document encoder is used to encode the chapter title information to which the sentence belongs. The extracted chapter title is input into the pre-trained encoder to obtain the hidden state corresponding to each character, and the hidden states are added together.

[0044] Furthermore, the sigmoid function used in the sigmoid layer in step five is:

[0045]

[0046] During the training phase, the sigmoid function uses binary cross-entropy as the loss function:

[0047] Loss = {loss1, ..., loss} n}

[0048]

[0049] Where σ represents the sigmoid function, W h Let b represent the learnable parameter matrix. h Indicates bias, loss i This represents the loss incurred when determining whether each sentence belongs to the summary sentence. y represents the probability value predicted for the current sentence. i This represents the actual label value of the sentence.

[0050] Another object of the present invention is to provide an extractive summarization system that combines local topic and hierarchical structure information, the extractive summarization system comprising:

[0051] The document-giving module is used to provide the original document.

[0052] The encoding module is used to obtain a contextual representation of the document using an encoder based on a pre-trained language model;

[0053] The local topic information extraction module is used to extract the topic information of the segment to which a sentence belongs in the context representation of the document;

[0054] The fusion module is used to merge the local topic information representation and the document's context representation to obtain a text context representation that integrates the local topic information;

[0055] The text hierarchy information embedding module is used to embed the text hierarchy information into the text context representation that integrates local topic information;

[0056] The judgment module is used to calculate the confidence score of each sentence through the Sigmoid layer to determine whether the sentence belongs to the summary sentence.

[0057] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0058] First, the specific description of the creative technical effects is as follows:

[0059] (1) By adding text hierarchical structure information to the model, the model can pay more attention to the important parts of the text and improve the quality of the generated summary.

[0060] (2) A local topic information extraction module is proposed to capture the local topic information of the segment to which the sentence belongs, so that the model can have a deeper understanding of long documents and thus produce high-quality summaries.

[0061] (3) By modifying the Token Position Embeddings of Longformer, it is made able to handle longer text data.

[0062] This invention proposes a long document extraction and summarization model that integrates local topic information and document hierarchical structure information from the current topic fragment. The model consists of a text encoder, a local topic information extraction module, and a text hierarchical structure information embedding module, and can effectively generate high-quality summaries.

[0063] Secondly, this invention primarily addresses two shortcomings of traditional methods in long text extraction and summarization tasks. The first is that long text data contains clear internal hierarchical structures and chapter headings. When manually summarizing text, we tend to focus on important parts. For example, in scientific papers, we might pay more attention to "Methodology," "Experimental," and "conclusion," while paying less attention to "Background" or "Relate Work." Furthermore, sentences within a chapter are more closely related to sentences outside that chapter. Understanding the order of sentences and the internal hierarchical structure of the document helps the model better identify important sentences. Traditional Transformer-based text summarization methods often treat text as a sequence structure and cannot handle long documents. The second shortcoming is that the longer the document, the more topics it likely discusses, as each chapter presents different thematic information. The aforementioned methods focus on the overall thematic information, i.e., global information, while ignoring the local thematic information of these chapters. To address these issues, this paper proposes a long document extraction and summarization model that integrates local thematic information from the current topic segment and the document's hierarchical structure. The model consists of a text encoder, a local topic information extraction module, and a text hierarchical structure information embedding module, which can effectively generate high-quality summaries.

[0064] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0065] The technical solution of this invention fills a technical gap in the industry both domestically and internationally: This invention makes up for the problem that traditional long text extraction text summarization methods are insufficient in extracting the hierarchical structure information of the original text and the local topic information of the long text, and proposes a long document extraction and summarization model that integrates the local topic information and document hierarchical structure information in the current topic segment.

[0066] Does the technical solution of this invention solve a long-standing technical problem that people have long desired to solve but have yet to succeed in? The invention primarily addresses two shortcomings of traditional methods in long text extraction and summarization tasks: first, Transformer language models typically treat text as a linear sequence, ignoring the inherent hierarchical structure information of the text; second, for long text data, Transformer language models still have limitations in capturing and integrating local contextual information within topic fragments. To solve these problems, a long text extraction and summarization model that integrates local contextual information from topic fragments and document hierarchical structure information is proposed. Attached Figure Description

[0067] Figure 1 This is a flowchart of an extractive summarization method combining local topic and hierarchical structure information provided in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the extraction and summarization method that combines local topic and hierarchical structure information provided in the embodiments of the present invention.

[0069] Figure 3 This is an architecture diagram of the local topic information extraction module provided in an embodiment of the present invention;

[0070] Figure 4 This is a model structure diagram provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] The key technical points and areas to be protected in this invention are as follows: The overall architecture of the model uses LSTM-Minus to obtain a distributed representation of local topic information and combines it with a text summarization task. Positional encoding is used to represent text hierarchical structure information and embeds it into a text context representation that integrates local topic information.

[0073] like Figure 1 As shown, the extractive summarization method combining local topic and hierarchical structure information provided in this embodiment of the invention includes:

[0074] S101, Step 1: Given the original document D = {sent1, ...,sent...} n}, where sent n This represents the nth sentence in the original document;

[0075] S102, the context representation of the document is obtained through the encoder, and then input into the local topic information extraction module to extract the topic information of the segment to which the sentence belongs;

[0076] S103, the local topic information representation and the document context representation are merged to obtain a text context representation that integrates local topic information;

[0077] S104, the text hierarchical structure information embedding module embeds the hierarchical structure information of the text into the text context representation that integrates local topic information. Through two stacked Transformer layers, it learns the document-level hierarchical structure information of sentences, enabling the model to gain a deeper understanding of the text context structure.

[0078] S105, calculate the confidence score of each sentence through the Sigmoid layer to determine whether the sentence belongs to the summary sentence.

[0079] like Figure 2 As shown, the model as a whole mainly includes three modules: an encoder based on a pre-trained language model, a local topic information extraction module, and a text hierarchical structure information embedding module.

[0080] Since this invention uses long text corpora, the encoder employs a Longformer model improved from the pre-trained Transformer language model to more fully encode long documents. Specifically, for a given original document D = {sent1, ...,sent...} n}, where sent n This represents the nth sentence in the original document.

[0081] To obtain the representation of each sentence, [BOS] and [EOS] tags are inserted at the beginning and end of each sentence, respectively, with the [BOS] tag representing the entire sentence representation. After obtaining the document's contextual representation through the encoder, it is input into the local topic information extraction module to extract the topic information of the segment to which the sentence belongs.

[0082] Subsequently, the local topic information representation and the text context representation are fused to obtain a text context representation that incorporates local topic information. The text hierarchical structure information embedding module embeds the text hierarchical structure information into the text context representation that incorporates local topic information. Through two stacked Transformer layers, the model learns the document-level hierarchical structure information of sentences, enabling it to gain a deeper understanding of the text context structure.

[0083] Finally, the confidence score of each sentence is calculated using a Sigmoid layer to determine whether the sentence belongs to the summary sentence.

[0084] 1. Text hierarchical structure information

[0085] 1.1 Sentence hierarchical structure information

[0086] Since scientific papers contain many chapters, and each chapter contains many paragraphs, with different paragraphs describing different topics, this invention uses paragraphs as the unit for dividing the article into hierarchical units.

[0087] Sentence hierarchical structure information includes the linear position of the sentence within its paragraph and the sentence's linear position within the paragraph. The position is represented by the numerical indices corresponding to the paragraph and the sentence. For a given document D = {sent1, ...,sent...} n}, the i-th sentence sent i The hierarchical information is represented as a two-dimensional vector (s s ,g s The two-dimensional vector represents the position of the sentence at this level, as shown in Formula 1.

[0088] vsent i =(s s ,g s (1)

[0089] Among them, s s G represents the linear position of the paragraph containing the sentence relative to the entire article. s This represents the linear position of the sentence within its paragraph. All sentences within the same paragraph share the same first dimension in `vsent`, indicating a higher correlation between sentences within the same paragraph, while `g`... s Vectors further illustrate the linear relationships between sentences within a paragraph.

[0090] 1.2 Chapter Title Information

[0091] Compared to short news articles, long scientific papers typically contain chapter titles, and the content described within a chapter is usually highly related to its title; in other words, the chapter title is a summary of the section's content. Based on this, this invention uses chapter titles as additional hierarchical structure information when encoding sentences. However, in scientific papers, many similar chapter titles have the same meaning; for example, "Method" and "Methodology" have the same meaning and can both be categorized as "Method." Therefore, for the PubMed dataset used in this invention, eight chapter title categories were defined: "introduction," "background," "case," "method," "result," "discussion," "conclusion," and "additional information." If the chapter title of a sentence does not belong to one of these eight categories, then the chapter title itself is used directly.

[0092] 2. Text Encoding

[0093] Document encoding aims to encode sentences from an input document into fixed-length vector representations. For extractive text summarization tasks, previous methods typically employed RNNs and BERT pre-trained language models as encoders. BERT, a bidirectional Transformer encoder pre-trained on a large corpus, has achieved excellent results in multiple natural language processing tasks. However, for long text data, BERT cannot capture all the data, leading to information loss. Therefore, this invention uses a Longformer pre-trained language model as the text encoder. Longformer improves the traditional Transformer's self-attention mechanism to a sliding window self-attention mechanism, enabling it to easily handle documents of thousands of characters. The traditional Transformer self-attention mechanism first performs a linear transformation on the input word vector matrix, generating a query matrix (Q), a key matrix (K), and a value matrix (V) of dimension d. Multiplying the transpose of each Q matrix and K matrix yields the attention weight matrix, which represents the similarity between words. The attention weight matrix is ​​then multiplied by the V matrix to obtain a matrix containing this similarity relationship. The specific calculation process is shown in Equation 2.

[0094]

[0095] Where (Q,K,V)∈R L×d L represents the sequence length, and d represents the dimension of the word vector matrix. k This represents the dimension of the K matrix. Therefore, QKT ∈R L×L That is, the space complexity of the traditional Transformer self-attention mechanism is O(L). 2 The length of the input sequence is quadratically proportional to the input sequence length, making it unsuitable for handling long input sequences. Compared to the traditional Transformer, the Longformer uses a sliding window self-attention mechanism. If the sliding window size is set to n, then the operation in Equation 2 becomes: the i-th row in Q is only used in conjunction with the row in K. This window contains n rows for dot product operations. Then QK T ∈R L×n Where (n << L), the space complexity of Longformer's sliding window self-attention mechanism is O(L), which is linearly proportional to the length of the input sequence. Therefore, Longformer is more advantageous for encoding long texts.

[0096] To obtain a representation of each sentence, this invention inserts [BOS] and [EOS] tags at the beginning and end of each sentence, respectively. The model embedding layer includes Token Embeddings (TE), Segment Embeddings (SE), and Position Embeddings (PE).

[0097] w i,j =(TE+SE+PE) (3)

[0098] After obtaining the embedding representation of each word using Formula 3, the pre-trained Longformer is used to perform context learning on the input sequence.

[0099] {h 1,0 ,h 1,1 ,…,h N,0 ,…,h N,*} = Longformer(w 1,0 ,w 1,1 ,…,w N,0 ,…,w N,* (4)

[0100] Among them, w i,j This represents the j-th word of the i-th sentence, obtained using Formula 3. i,0 and w i,* The [BOS] and [EOS] tags represent the i-th sentence, respectively. i,j This represents the hidden state of the corresponding word. After Longformer encoding, the [BOS] tag is used as the context representation for each sentence, i.e., H. s =(h 1,0 ,…,hN,0 ).

[0101] 3. Extraction of local theme information

[0102] To capture the local contextual information of the segment to which the sentence belongs, this invention utilizes the LSTM-Minus method to learn text segment embeddings, the detailed structure of which can be found in [link to detailed structure]. Figure 3 As shown.

[0103] The input to the local topic information extraction module is the context representation of each sentence obtained through the encoder. Since LSTM can learn and utilize previous information and store it in memory units through its own gating structure, this invention uses Bi-LSTM encoding to obtain the hidden vector representation of each sentence's context representation. Subsequently, the local context information of the topic segment to which the sentence belongs is represented by the subtraction between the start and end hidden vectors of each topic segment.

[0104] For the i-th topic segment t i For details on its specific representation, please refer to formulas (5) to (7).

[0105]

[0106]

[0107] t i =(f i |b i (7)

[0108] Among them, f i and b i These represent the thematic segments of forward propagation and backward propagation, respectively. i and end i These represent the start and end positions of the topic segment, respectively. | represents the vector concatenation symbol. For example, for... Figure 3 The second topic segment t2 can be represented as [f5-f2, b3-b6], where f5 and f2 represent the hidden states of the 5th and 2nd sentences during forward propagation, and b3 and b6 represent the hidden states of the 3rd and 6th sentences during backward propagation. To prevent subscripts from exceeding boundaries, this invention adds 0 vectors at the beginning and end of both forward and backward propagation. After calculating the local context information of the topic segment to which the sentence belongs, it is concatenated with the document sentence context encoding to further enrich the sentence context representation.

[0109] 4. Text hierarchical structure information encoding

[0110] Currently, there are two main linear positional encoding methods: the fixed value generated by sin / cosine in Transformer and the randomly generated and trainable method used in BERT. The positional encoding method in Transformer can only generate a fixed value using sine or cosine functions, and can only mark the position of the character, but cannot learn the specific function of that position based on the context information of the token. In contrast, BERT's positional encoding randomly initializes an embedding vector with dimensions [seq_length, width]. Here, the first dimension represents the sequence length, and the second dimension represents the vector length corresponding to each character. It is trained together with the entire summarization model, thus it can both mark character positions and learn the function of those positions. Therefore, this invention uses the BERT positional encoding method to encode the vsent vector. For the hierarchical structure vector (s) of the i-th sentence... s ,g s This can be expressed as Formula 8.

[0111]

[0112] Where PE represents the BERT positional encoding method, d represents the vector dimension of the sentence, and | represents the vector concatenation symbol.

[0113] To encode the chapter title information (STE) of a sentence, this invention utilizes a pre-trained encoder identical to the one used for document encoding. The extracted chapter title is input into the pre-trained encoder to obtain the hidden state corresponding to each character, and these hidden states are summed. This allows for better fusion of semantic information from each position within the chapter title, resulting in a more comprehensive representation of the chapter title information.

[0114] 5. Model Training and Inference

[0115] After obtaining the sentence vector through the text hierarchical structure information embedding module, the document-level hierarchical structure information of the sentence is learned through two stacked Transformer layers. Finally, it is input into the sigmoid function to predict whether the sentence belongs to the summary sentence.

[0116]

[0117] During the training phase, this model uses binary cross-entropy as the loss function, and the training objective is to optimize the model by minimizing the binary cross-entropy loss function. See Equations 10-11 for details.

[0118] Loss = {loss1, ..., loss} n} (10)

[0119]

[0120] In Formula 9, σ represents the sigmoid function, and W h Let b represent the learnable parameter matrix. h This represents the bias. The loss in Equation 10 i This represents the loss incurred when determining whether each sentence belongs to the summary sentence. y represents the predicted probability value for the current sentence. i This represents the actual label value of the sentence.

[0121] To better understand the implementation of specific examples, the following deduces the specific process in conjunction with the specific structure of the model.

[0122] (1) Suppose a source document D is used as input, where:

[0123] D = {sent1,sent2,sent3,sent4,sent5,sent6,sent7} contains 7 sentences. The [BOS] and [EOS] tags are inserted at the beginning and end of each sentence, respectively. Figure 4 As shown, the sentence is first processed through an embedding layer to obtain the embedding representation of each token, and then input into the Longformer encoder to obtain the context representation of each sentence.

[0124] (2) The word vector matrix obtained from the Longformer encoder is input into the local topic extraction module. Figure 4 For the specific structure of Topic Segment Representation, see [link / details]. Figure 3 As shown. First, the context representation of each sentence is input into Bi-LSTM encoding to obtain the hidden vector representation of each sentence. Then, the local context information of the topic segment to which the sentence belongs is represented by the subtraction between the start and end hidden vectors of each topic segment. To prevent the subscripts from exceeding the boundaries, this invention adds 0 vectors at the beginning and end of both forward and backward propagation. For example, for Figure 3 The second topic segment t2 can be represented as [f5-f2, b3-b6], where f5 and f2 represent the hidden states of the 5th and 2nd sentences during forward propagation, and b3 and b6 represent the hidden states of the 3rd and 6th sentences during backward propagation.

[0125] (3) The local topic information of the sentence is fused with the contextual representation of the text to obtain a text contextual representation that integrates the local topic information. This is then input into the text hierarchical structure information embedding module, and the text hierarchical structure information is embedded into the text contextual representation that integrates the local topic information using BERT positional encoding.

[0126] (4) After the local topic information extraction module and the text hierarchical structure information embedding module, the sentence document-level hierarchical structure information is learned through two stacked Transformer layers, so that the model can understand the text context structure more deeply. Finally, the sentence classification layer is used, which consists of a sigmoid function. The confidence score of each sentence is calculated through the sigmoid layer to determine whether the sentence belongs to the summary sentence.

[0127] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An extractive summarization method combining local topic and hierarchical structure information, characterized in that, include: Step 1: Given the original document D = {sent1, ...,sent...} n }, where sent n This represents the nth sentence in the original document; Step two: Obtain the contextual representation of the document through the encoder, and input it into the local topic information extraction module to extract the topic information of the segment to which the sentence belongs; Step 3: Merge the local topic information representation and the document context representation to obtain a text context representation that integrates the local topic information; Step four: The text hierarchical structure information embedding module embeds the hierarchical structure information of the text into the text context representation that integrates local topic information. Through two stacked Transformer layers, it learns the document-level hierarchical structure information of sentences, enabling the model to gain a deeper understanding of the text context structure. Step 5: Calculate the confidence score of each sentence using the Sigmoid layer to determine whether the sentence belongs to the summary sentence; In step one, [BOS] and [EOS] tags are inserted at the beginning and end of each sentence, respectively, with the [BOS] tag representing the entire sentence. In step two, the Longformer pre-trained language model is used as the text encoder, and the model embedding layer includes TE: Token Embeddings, SE: Segment Embeddings, and PE: Position Embeddings. w i,j =(TE+SE+PE) The above formula yields the embedding representation of each word, and the pre-trained Longformer is used to perform context learning on the input sequence; {h 1,0 ,h 1,1 ,…,h N,0 ,…,h N,* }=Longformer(w 1,0 ,w 1,1 ,…,w N,0 ,…,w N,* ) Among them, w i,j w represents the j-th word of the i-th sentence. i,0 and w i,* The [BOS] and [EOS] tags represent the i-th sentence, respectively. i,j This represents the hidden state of the corresponding word; after Longformer encoding, the [BOS] tag is used as the context representation of each sentence, i.e., H. s =(h 1,0 ,…,h N,0 ); The specific steps for the local topic information extraction module in step two to extract the topic information of the segment to which the sentence belongs include: The hidden vector representation of each sentence is obtained by encoding the sentence context using Bi-LSTM; The local context information of the topic segment to which the sentence belongs is represented by the subtraction between the start and end hidden vectors of each topic segment. For the i-th topic segment t i Its specific representation method is as follows: t i =(f i |b i ) Among them, f i and b i These represent the thematic segments of forward propagation and backward propagation, respectively. i and end i These represent the start and end positions of the topic segment, respectively, and | represents the vector concatenation symbol; Add zero vectors at the beginning and end of both forward and backward propagation to prevent indices from exceeding the boundaries; The hierarchical structure information of the text in step four includes sentence hierarchical structure information and chapter title information; The sentence hierarchical structure information includes the linear position of the paragraph to which the sentence belongs and the linear position representation of the sentence within the paragraph; The position is represented by the numerical index corresponding to the paragraph and sentence. For a given document D = {sent1,…,sent1}, ...,sent2, ...,sent3, ...,sent4, ...,sent5, ...,sent6, ...,sent7, ...,sent8, ...,sent9 ... n }, the i-th sentence is sent i The hierarchical information is represented as a two-dimensional vector (s s ,g s This two-dimensional vector represents the position of the sentence at this level, specifically: vsent i =(s s ,g s ) Among them, s s G represents the linear position of the paragraph containing the sentence relative to the entire article. s This indicates the linear position of the sentence within its paragraph. The chapter title information uses a PubMed dataset with multiple preset chapter title categories. If the chapter title of the sentence does not belong to one of the preset chapter title categories, the chapter title of the sentence itself is used directly. The text hierarchical structure information embedding module in step four uses the BERT positional encoding method to encode the vsent vector; For the hierarchical structure vector (s) of the i-th sentence s ,g s The expression is as follows: Where PE represents the BERT positional encoding method, d represents the vector dimension of the sentence, and | represents the vector concatenation symbol; The same pre-trained encoder as the document encoder is used to encode the chapter title information to which the sentence belongs. The hidden state corresponding to each character is obtained by inputting the extracted chapter title into the pre-trained encoder, and each hidden state is added together. The sigmoid function used in the Sigmoid layer in step five is: During the training phase, the sigmoid function uses binary cross-entropy as the loss function: Loss={loss1,…,loss n } Where σ represents the sigmoid function, W h Let b represent the learnable parameter matrix. h Indicates bias, loss i This represents the loss incurred when determining whether each sentence belongs to the summary sentence. y represents the probability value predicted for the current sentence. i This represents the actual label value of the sentence.

2. A system for extracting summarizing information by combining local topics and hierarchical structure, used to implement the extractive summarizing method combining local topic and hierarchical structure information as described in claim 1, characterized in that, The extractive summarization system that combines local topic and hierarchical structure information includes: The document-giving module is used to provide the original document. The encoding module is used to obtain a contextual representation of the document using an encoder based on a pre-trained language model; The local topic information extraction module is used to extract the topic information of the segment to which a sentence belongs in the context representation of the document; The fusion module is used to merge the local topic information representation and the document's context representation to obtain a text context representation that integrates the local topic information; The text hierarchy information embedding module is used to embed the text hierarchy information into the text context representation that integrates local topic information; The judgment module is used to calculate the confidence score of each sentence through the Sigmoid layer to determine whether the sentence belongs to the summary sentence.

3. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the extractive summarization method combining local topic and hierarchical information as described in claim 1.

4. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the extractive summarization method combining local topic and hierarchical information as described in claim 1.

Images