Long document summarization generation method based on RNN and sparse self-attention mechanism
By using a prediction model based on RNN and sparse self-attention mechanism, the problems of incomplete fragments, loss of word order information, and high complexity in long document summarization are solved, achieving efficient and accurate text summarization and improving the model's generation efficiency and summarization quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-28
- Publication Date
- 2026-03-17
AI Technical Summary
Existing methods for generating long document summaries suffer from problems such as incomplete fragments, loss of word order information, high time complexity, and high space complexity, especially when processing long documents, where they are inefficient.
A prediction model based on RNN and sparse self-attention mechanism is adopted. The location information is obtained through local recurrent neural network, important query vectors are filtered by sparse self-attention, self-attention distillation reduces complexity, and a caching module is used to preserve semantic integrity. Encoding and decoding modules are constructed to generate text summaries.
It effectively reduces time and space complexity, improves the efficiency and accuracy of text summarization, preserves the semantic integrity between fragments, and enhances the readability and relevance of the summaries.
Smart Images
Figure CN116049384B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing, deep learning, and text summarization, and particularly to a long document summarization method based on RNN and sparse self-attention mechanism. Background Technology
[0002] Text is one of the most important carriers of information. Textual information varies greatly in subject matter and scale, making efficient processing a crucial research question that has driven the rapid development of automatic text classification technology. In recent years, the booming development of internet technology has greatly facilitated daily life, but inevitably, the information on the internet has exploded. Therefore, quickly and effectively obtaining the information needed is extremely important. Automatic text summarization technology can effectively alleviate this problem. As an important research area in natural language processing and artificial intelligence, it uses computers to automatically extract a concise and coherent short text from long texts or text collections that accurately reflects the central content of the source text, significantly reducing the time required for manual screening. This technology was first applied in the meteorological field, using a fixed format to insert predicted data. Later, it was applied in the financial and medical fields, where such tools can greatly help professionals save time.
[0003] Text summarization methods can be broadly categorized into two types: extractive text summarization and generative text summarization. Extractive text summarization directly selects several important sentences from the original text, sorts and reorganizes them to form a summary. Generative text summarization, based on Natural Language Generation (NLG), generates sentences automatically through a model. Currently, generative text summarization primarily utilizes deep learning-based encoder-decoder models. Most long document summarization falls under this category, and is generally based on improvements to Transformer and BART models. While research on long document summarization methods is abundant, several challenges remain:
[0004] First: During the training of a transformer model, each segment is usually processed separately. Therefore, the longest dependency between different tokens depends on the length of the segment. For efficiency reasons, when segmenting, the natural boundaries of the sentence are not considered. Instead, the sequence is divided according to a fixed length, resulting in semantically incomplete segments.
[0005] Second: Since the positional embedding is significantly improved in the word embedding dimension, there is a high probability of losing word order information. Therefore, although the multi-head attention mechanism can ensure that long-distance information is not lost, the positional information of a certain word in this sentence may not be so clear.
[0006] Third: Generative models mostly use attention mechanisms, but their dot product operations make the time complexity and memory usage of each layer O(n^2). At the same time, stacking multiple encoders and decoders will create complex space complexity, which will limit the model from accepting longer sequence inputs. Summary of the Invention
[0007] To address the aforementioned problems in the prior art, this invention proposes a long document summarization method based on RNN and sparse self-attention mechanism. A prediction model based on RNN and sparse self-attention mechanism is constructed, and the prediction specifically includes the following steps:
[0008] S1. The text data is segmented or padded to a fixed length L, and the segments of length L are converted into word vectors by word embedding.
[0009] S2. The word vector fragments are used as input to the encoder, and the encoder encodes the context representation corresponding to the word vector fragments.
[0010] S3. In the decoding stage, the word vector representation and the corresponding context representation are input into the decoder to obtain the final hidden features;
[0011] S4. The hidden features are passed through a Softmax layer to obtain the predicted output.
[0012] Furthermore, during the encoding process, the encoder includes an N-layer cascaded structure. Each layer includes a cascaded local recurrent neural network, a multi-head sparse self-attention feedforward network, and word vector segments as the output of the multi-head sparse self-attention. In the N-layer cascaded structure, the output of the current layer caches the hidden state of the previous segment, undergoes attention distillation, and then serves as the input to the next layer. The processing of the local recurrent neural network output by the multi-head sparse self-attention includes the following steps:
[0013] 101. Construct the query vector, key vector, and value vector in the attention mechanism using the output of a local recurrent neural network;
[0014] 102. Use sparsity measures to select u important query vectors;
[0015] 103. Focus on u important query vectors and value vectors, and then concatenate them with Lu average vectors to calculate self-attention;
[0016] 104. Calculate multi-head attention using the multi-head sparse attention mechanism, and obtain the hidden state representation after passing through the FFN layer.
[0017] Furthermore, the process of selecting u important query vectors using sparsity metrics can be represented as follows:
[0018]
[0019]
[0020]
[0021] Where Q, K, and V are the outputs of the local recurrent neural network, respectively. Constructing the query vector, key vector, and value vector in the attention mechanism; W q W k W v These are the trainable parameters corresponding to Q, K, and V, respectively; M(q) i ,K) represents the sparsity evaluation score of the i-th query; q i To query the i-th element in vector Q; d is the transpose of the j-th element in the key vector K; d is the dimension of the value vector. Let V be a vector set consisting of u important queries; Choice()[0:u-1] represents selecting the u parameters with the largest values; V is a value vector; L K This represents the length of the key vector.
[0022] Furthermore, the process of calculating self-attention in step 103 includes:
[0023]
[0024]
[0025] Here, `concat()` is the concatenation function; `Attention()` represents the attention value obtained from the input query vector, key vector, and value vector; `mean(V)` represents the average of the value vectors along the `q_length` dimension, where the value vector dimensions are `[batch_size, q_length, d_model]`, `batch_size` is the batch size (i.e., the number of data samples captured in one training session), `q_length` is the segment length, and `d_model` is the word vector dimension; `[u:L-1]` represents the vectors from index `u` to `L-1`.
[0026] Furthermore, step 104, which utilizes a multi-head sparse attention mechanism to compute multi-head attention and obtains the hidden state representation through an FFN layer, includes:
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033] Where LayerNorm(x) represents the layer normalization process performed on sample x; This represents the hidden state representation corresponding to the Tth element of the local recurrent neural network in the i-th layer of the encoder's N-layer cascaded structure. Indicates to The result of normalization; This represents the output of the T-th element of the multi-head sparse self-attention layer in the i-th layer of the N-layer cascaded encoder structure; Indicates to The result of normalization; head h Let H be the output of the h-th head in the multi-head sparse self-attention system, where H is the number of heads in the multi-head sparse self-attention system, and W is the output of the h-th head. O For multi-head sparse self-attention, the learnable parameters are: Attention() represents the attention value obtained from the input query vector, key vector, and value vector. Learnable parameters for calculating attention for the h-th head.
[0034] Furthermore, the output of the current layer is cached with the hidden state of the previous segment, then undergoes attention distillation before being used as the input to the next layer, including:
[0035]
[0036]
[0037]
[0038]
[0039] in, This represents the hidden state representation of the (m+1)th segment in the i-th layer of an N-layer cascaded encoder. This represents the key vector corresponding to the (m+1)th segment in the i-th layer of an N-layer cascaded encoder. This represents the key vector corresponding to the (m+1)th segment in the i-th layer of an N-layer cascaded encoder. This represents the value vector corresponding to the (m+1)th segment in the i-th layer of an N-layer cascaded encoder. Indicates to Cache the hidden state of the previous segment; W q W k W vThese are the corresponding trainable parameters; express The gradient value when not updated; Attention_Block() represents the function that calculates the hidden state obtained by multi-head sparse self-attention; This indicates that MaxPool() is the max pooling operation, Convld() is a one-dimensional convolutional filter with a kernel width of 3, ELU() is the activation function, and MaxPool() is the max pooling operation with a stride of 2.
[0040] Furthermore, corresponding to the encoding process, the decoder includes an N-layer cascaded structure. Each layer includes a cascaded local recurrent neural network, a masked multi-head sparse self-attention, a cross-multi-head attention, and a feedforward network. The input of the cross-multi-head attention is the context representation of the encoded output and the output of the masked multi-head sparse self-attention.
[0041] Furthermore, the process of cross-head attention includes:
[0042]
[0043]
[0044] in, Here, H represents the query vector, key vector, and value vector corresponding to the cross-multi-head attention; H is the vector that masks the output of the multi-head sparse self-attention in the decoder; and Context is the context vector output by the encoder. for The transpose of the corresponding trainable matrix; This represents the output of the Mth segment in the decoder, which undergoes cross-head attention in the i-th layer.
[0045] Furthermore, the output of the feedforward network layer above the decoder is normalized and distilled before being used as the output of the current layer, including:
[0046]
[0047]
[0048]
[0049] Where FeedFoward() represents a feedforward network; LayerNorm() represents layer normalization;
[0050] MaxPool() is a max pooling operation; Convld() is a one-dimensional convolutional filter with a kernel width of 3; ELU() is an activation function; MaxPool() is a max pooling operation with a stride of 2. This is the output after distillation.
[0051] Furthermore, when training the prediction model based on RNN and sparse self-attention mechanism, the MSN loss function between predicted and true values is used to backpropagate the model and update the network parameters. The MSN loss function between predicted and true values is expressed as follows:
[0052]
[0053] Among them, y i The actual value; is the predicted value; MSE is the mean squared error loss function; n represents the total amount of data generated for summarization.
[0054] This invention effectively utilizes local RNN operations to extract positional information instead of positional encoding. Each word is processed by a different RNN, avoiding the awkward situation of traditional "one after another" processing, thus enabling parallel processing. It employs sparse self-attention to filter important query vectors, reducing the traditional time complexity from O(n^2) to O(n*logn). Self-attention distillation reduces the number of parameters and dimensionality of the self-attention module. A recursive loop mechanism caches the hidden state results of the previous segment, eliminating the need for repeated calculations and significantly improving efficiency while preserving the semantic integrity between segments. In summary, this invention effectively utilizes a hierarchical framework model to model relevant factors in text summarization, thereby improving the readability, relevance, and accuracy of the model-generated summaries. Attached Figure Description
[0055] Figure 1 This is a schematic diagram of the long document summarization generation model based on RNN and sparse self-attention mechanism of the present invention.
[0056] Figure 2 This is a schematic diagram of the local RNN model of the present invention;
[0057] Figure 3 This is a schematic diagram of the self-attention distillation model of the present invention. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] This invention proposes a long document summarization method based on RNN and sparse self-attention mechanism, constructing a prediction model based on RNN and sparse self-attention mechanism. The prediction model specifically includes the following steps:
[0060] S1. The text data is segmented or padded to a fixed length L, and the segments of length L are converted into word vectors by word embedding.
[0061] S2. The word vector fragments are used as input to the encoder, and the encoder encodes the context representation corresponding to the word vector fragments.
[0062] S3. In the decoding stage, the word vector representation and the corresponding context representation are input into the decoder to obtain the final hidden features;
[0063] S4. The hidden features are passed through a Softmax layer to obtain the predicted output.
[0064] like Figure 1 As shown, this embodiment presents a long document summarization generation model based on RNN and sparse self-attention mechanism. This model is based on the Transformer model and includes a word embedding module, a local RNN module, a sparse self-attention module, a self-attention distillation module, and a caching module; as shown... Figure 2 As shown, after word embedding of the input text, a local RNN module is used instead of traditional positional embedding to obtain positional information; a sparse self-attention module is used to filter important query matrices to reduce the time complexity and memory usage of the dot product operation; for example... Figure 3 As shown, the self-attention distillation module reduces the dimensionality and number of network parameters, thereby reducing the space complexity of long sequence inputs; the caching module caches the hidden state of the previous segment to preserve semantic integrity and establish longer dependency capabilities.
[0065] Specifically, such as Figure 1 This invention proposes a prediction model based on RNN and sparse self-attention mechanism. The model includes an encoding module and a decoding module. The encoder module comprises cascaded N-layer encoders, each layer including a local recurrent neural network (i.e., a local RNN), multi-head sparse self-attention, and a feedforward network. The decoding module comprises cascaded N-layer decoders, each layer including a local recurrent neural network, masked multi-head sparse self-attention, cross-multi-head self-attention, and a feedforward network. The output of the last layer of the decoding module serves as the input to the cross-attention mechanism. The output of the last layer of the decoding module is passed through a Softmax layer to obtain the prediction result. Specific operations include the following:
[0066] (i) By combining word embedding and position embedding, the numerical representation of text is transformed into a vector representation, capturing the semantic and positional relationships between words in a high-dimensional space.
[0067] First, the word embedding module is used to convert the input text into a word vector representation, as follows:
[0068]
[0069] Where Input represents the input text sequence; Embedding() represents word embedding; This represents the word embedding input at time step t in the i-th Transformer layer. When t=0, This represents the word embedding of the input text; t represents the t-th time step, which is the t-th input word vector, and also refers to the t-th word vector.
[0070] Secondly, the local RNN module is used to add positional information to the word embedding representation, thus replacing the positional embedding, to obtain the final vector representation. The final vector representation of the i-th layer structure is:
[0071]
[0072] in, This represents the final vector representation of the input layer at time step t; LocalRNN() represents a local RNN operation, such as... Figure 2 As shown, each solid circle at the bottom represents a word in the sentence, and the dashed circle on the left is added to complete the sentence. In this embodiment, two zero-value vectors and the first word vector are initialized to form three vectors as the input of the first RNN. That is, each RNN corresponds to one RNN. The current word vector and the two word vectors before it are used as the RNN corresponding to that word vector. If there are fewer than three word vectors, they are filled with zero-value vectors. Each RNN runs in only one window with a window size of 3 to avoid gradient vanishing and gradient exploding. When processing each word, the information after the word cannot be obtained. That is, each word to be processed is placed at the last position of the window. In this way, the preceding words are all past and do not include the following information, which needs to be filled in.
[0073] Finally, in recurrent neural networks, the net input of the recurrent neural network generally increases or decreases over time, which can lead to gradient explosion or vanishing. Layer normalization is used to alleviate the gradient explosion or vanishing problem.
[0074]
[0075] in, The normalized final vector representation is represented by LayerNorm(); LayerNorm() represents the layer normalization operation, i∈{1,2,…,N},T=L.
[0076] (ii) Use sparse self-attention mechanism to select important query vectors and value vectors for attention, calculate the similarity between important words and all words, and reduce the time complexity and memory usage of dot product operation.
[0077] First, the sparsity metric is used to select u important query vectors, which can be represented as follows:
[0078]
[0079]
[0080]
[0081]
[0082] Where Q, K, and V are the query vector, value vector, and value vector, respectively; W q W k W v All are trainable parameters; q i This is the i-th query vector; M(q) is the transpose of the j-th key vector; d is the dimension of the value vector; i K) represents the evaluation score of the i-th query; The vector set consisting of the selected u important queries; This is the output corresponding to the RNN. This represents the output of the T-th RNN, since one word corresponds to one RNN network, i.e., T = L.
[0083] Secondly, the selected important query vectors and value vectors are focused on and concatenated with Lu average vectors to calculate self-attention. This process is represented as:
[0084]
[0085]
[0086] Where Z represents an intermediate vector; concat() represents the concatenation function; and Attention() represents the calculation of self-attention.
[0087] Then, multi-head attention is calculated using a multi-head sparse attention mechanism, which is represented as follows:
[0088]
[0089]
[0090] Among them, MultiHeadAttention() represents the calculation of multi-head attention; This represents the output of multi-head attention; Indicates to The normalized result.
[0091] Finally, the hidden state representation of Attention_Block is obtained using the feedback network, which is represented as follows:
[0092]
[0093]
[0094]
[0095] in, This represents the output of the i-th layer feedforward network, i.e., the hidden state representation. This represents the layer normalization result of the hidden state representation; FeedFoward(x) represents the distance of the front network to the data x, and w1, w2, b1, and b2 are all network parameters of the feedforward network.
[0096] (iii) Cache the hidden state of the previous segment, add it to the hidden state of the current segment, and update the query, key, and value vectors.
[0097] When caching the hidden vectors of previous segments, those skilled in the art can select to cache one or more hidden vectors of previous segments based on the memory parameters of the hardware device. For ease of description, this embodiment selects to cache the hidden vectors of one previous segment, and this process is represented as follows:
[0098]
[0099]
[0100]
[0101]
[0102] Here, SG() means that the gradient is not updated and backpropagation is not performed; It is the concatenation of the current hidden state of the i-th segment and the previous hidden state of the segment along the length dimension; These are trainable parameters; Attention_Block() represents the attention module, including sparse self-attention and all operations of the FNN; The calculation does not use the cache of the previous segment; The hidden state of the previous segment has been expanded. This represents the hidden state after the update at time step t+1 of layer i+1.
[0103] (iv) Use self-attention distillation to perform downsampling operations, reduce the number of dimensions and network parameters, and obtain context representation.
[0104] First, such as Figure 3As shown, convolutional pooling operations are added between adjacent Attention Blocks to downsample the features and update the hidden state of the current segment. This process is represented as follows:
[0105]
[0106] The hidden vectors corresponding to each RNN output from the last layer, i.e., the Nth layer, are concatenated together to form the context representation of the encoder output:
[0107]
[0108] Here, Convld() represents a one-dimensional convolutional filter with a kernel width of 3; ELU() represents an activation function; MaxPool() represents a max pooling operation with a stride of 2; Attention_Block() represents an attention module, including sparse self-attention and all operations of FNN, with a total of three attention modules. Each time the input sequence passes through a one-dimensional convolutional filter, it is reduced to half of its original size; Context represents the context representation, which is the set of hidden states in the last Attention_Block.
[0109] (v) The input text undergoes cross-attention with the context representation through the decoder to obtain the predicted output.
[0110] First: The input text is processed by word embedding and a local RNN to obtain the final vector representation. Then, sparse self-attention is input to obtain the hidden state representation. In the decoder, the local RNN and sparse self-attention process the input data in the same way as the decoder. If the size of the input cross-self-attention vector is H, the cross-self-attention process includes:
[0111]
[0112]
[0113] The result of cross self-attention Input feedforward network, i.e.:
[0114]
[0115] Layer normalization is performed on the output of the feedforward network, including:
[0116]
[0117] Attention distillation is performed on the normalized data, including:
[0118]
[0119] The output of the last layer of the decoder, i.e., the Nth layer. The generated summary is obtained by inputting the Softmax layer, including:
[0120]
[0121]
[0122] Here, Softmax() represents the activation function; Output represents the predicted output.
[0123] For ease of understanding, this embodiment provides the data processing flow for both the decoder and encoder. In the N-layer decoder, the text Input is used as input, and the hidden state representation of the Nth layer is used as the context representation as output, including:
[0124]
[0125]
[0126]
[0127]
[0128]
[0129]
[0130]
[0131]
[0132]
[0133]
[0134]
[0135]
[0136]
[0137]
[0138]
[0139]
[0140]
[0141]
[0142]
[0143]
[0144]
[0145]
[0146]
[0147]
[0148] In the decoder, the process of taking the context representation obtained from the encoder and the text input as input to obtain the output includes:
[0149]
[0150]
[0151]
[0152]
[0153]
[0154]
[0155]
[0156]
[0157]
[0158]
[0159]
[0160]
[0161]
[0162]
[0163]
[0164]
[0165]
[0166]
[0167]
[0168]
[0169]
[0170]
[0171]
[0172]
[0173]
[0174]
[0175]
[0176]
[0177]
[0178] When training a prediction model based on RNN and sparse self-attention mechanism, training samples are input into the model to obtain a predicted value. The MSE loss function between this predicted value and the true value of the training samples is calculated. The model parameters are adjusted through backpropagation. When the model's loss function value is less than a set threshold, the model training is complete. The MSE loss function between the predicted value and the true value of the training samples is expressed as:
[0179]
[0180] Among them, y i Represents the actual value; represents the predicted value; MSE represents the mean squared error loss function.
[0181] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A long document summarization generation method based on RNN and sparse self-attention mechanism, characterized in that, A prediction model based on RNN and sparse self-attention mechanism is constructed, and the model specifically comprises the following steps: S1, text data is segmented or padded to a fixed length L, and the L-length segment is converted into a word vector representation through word embedding; S2, the word vector segment is input into the encoder to obtain the context representation corresponding to the word vector segment; in the encoding process, the encoder comprises an N-layer cascade structure, each layer comprising a cascaded local recurrent neural network, a multi-head sparse self-attention, and a feedforward network, the word vector segment being the output of the multi-head sparse self-attention, and the output of the current layer being cached, distilled, and input into the next layer after the hidden state of the previous segment; the multi-head sparse self-attention processing of the output of the local recurrent neural network comprises the following steps: 101, constructing a query vector, a key vector, and a value vector in the attention mechanism using the output of the local recurrent neural network; 102, screening u important query vectors using a sparsity measure; 103, paying attention to the u important query vectors and the value vectors, and calculating self-attention after splicing the L-u average value vectors; 104, calculating multi-head attention using the multi-head sparse attention mechanism, and obtaining a hidden state representation after passing through an FFN layer; S3, inputting the word vector representation and the corresponding context representation into the decoder to obtain a final hidden feature in the decoding stage; S4, obtaining a prediction output through a Softmax layer.
2. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 1, characterized in that, The process of screening u important query vectors using a sparsity measure is represented as: wherein, , , are the outputs of the local recurrent neural network respectively to construct the query vector, key vector and value vector in the attention mechanism; , , are the outputs of the local recurrent neural network respectively , , corresponding trainable parameters; denotes the sparsity evaluation score of the ith query; is the ith element in the query vector Q; is the transpose of the jth element in the key vector K; d is the dimension of the value vector; is a vector set composed of the u important queries; denotes selecting the u parameters with the largest values; V is the value vector; denotes the length of the key vector.
3. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 2, characterized in that, The process of calculating self-attention in step 103 comprises: where concat() is a concatenation function; Attention() represents the attention value obtained by inputting the query vector, the key vector, and the value vector; mean(V) represents taking the average of the value vector in the q_length dimension, the value vector having a dimension of [batch_size, q_length, d_model], batch_size being the batch size, i.e., the number of data samples fetched for training at a time, q_length being the segment length, and d_model being the word vector dimension; [u:L-1] represents a vector with an index from u to L-1.
4. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 2, characterized in that, The process of calculating multi-head attention using the multi-head sparse attention mechanism in step 104, and obtaining a hidden state representation after passing through an FFN layer, comprises: wherein, denotes layer normalization processing on the sample x; denotes the hidden state representation corresponding to the Tth element of the local recurrent neural network in the ith layer in the encoder N-layer cascaded structure; denotes the result of normalization on ; denotes the output of the Tth element of the multi-head sparse self-attention in the ith layer in the encoder N-layer cascaded structure; denotes the result of normalization on ; is the output of the hth head in the multi-head sparse self-attention, H is the number of heads in the multi-head sparse self-attention, is a learnable parameter in the multi-head sparse self-attention; Attention() denotes an attention value obtained according to an input query vector, a key vector and a value vector; , , is a learnable parameter for calculating attention of the hth head; concat() is a concatenation function; denotes the distance of the data x to the previous network.
5. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 1, characterized in that, The output of the current layer is cached after the hidden state of the previous segment, distilled, and input into the next layer, comprising: ， ， wherein, denotes the hidden state representation corresponding to the m+1th segment in the ith layer of the N-layer cascaded structure encoder; denotes the key vector corresponding to the m+1th segment in the ith layer of the N-layer cascaded structure encoder, denotes the key vector corresponding to the m+1th segment in the ith layer of the N-layer cascaded structure encoder, denotes the value vector corresponding to the m+1th segment in the ith layer of the N-layer cascaded structure encoder, denotes the gradient of caches the hidden state of the previous segment; , , are the corresponding trainable parameters; SG( ) denotes the value of when the gradient of is not updated; Attention_Block() denotes a function for calculating multi-head sparse self-attention to obtain a hidden state; Convld() is a one-dimensional convolution filter with a kernel width of 3; ELU() is an activation function; MaxPool() is a maximum pooling operation with a step of 2; and concat() is a concatenation function.
6. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 1, characterized in that, Corresponding to the encoding process, the decoder comprises an N-layer cascade structure, each layer comprising a cascaded local recurrent neural network, a masked multi-head sparse self-attention, a cross multi-head attention, and a feedforward network, the input of the cross multi-head attention being the context representation of the encoding output and the output of the masked multi-head sparse self-attention.
7. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 6, characterized in that, The process of the cross multi-head attention comprises: wherein, , , are the query vector, key vector, value vector corresponding to the cross multi-head attention; is the vector masking the output of the multi-head sparse self-attention in the decoder, is the context vector output by the encoder; , , are , , the transpose of the corresponding trainable matrix; denotes the output of the cross multi-head attention in the i-th layer of the M-th segment in the decoder.
8. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 7, characterized in that, The output of the previous layer of the decoder is normalized and distilled after the feedforward network, and then outputted as the output of the current layer, comprising: wherein, represents a feedforward network; represents layer normalization; Convld() is a one-dimensional convolution filter with a kernel width of 3; ELU() is an activation function; MaxPool() is a maximum pooling operation with a step of 2; is the output after distillation.
9. The long document summarization generation method based on RNN and sparse self-attention mechanism according to claim 1, characterized in that, When the prediction model based on the RNN and the sparse self-attention mechanism is trained, the MSN loss function of the prediction value and the real value is used to update the network parameters of the model through back propagation, and the MSN loss function of the prediction value and the real value is represented as: wherein, is the true value; is the predicted value; MSE is the mean squared error loss function; n represents the total amount of data for which the summary is generated.