A text abstract generation method and system based on BART-BiGRU-LAAM

Abstract

The application discloses a text abstract generation method and system based on BART-BiGRU-LAAM, and belongs to the field of text abstract in natural language processing; the BiGRU feature extraction unit is used to capture semantic information with context timing, and the extraction unit comprises a BiGRU module: context timing information of embedded vectors is captured from positive and negative directions, hidden states with semantic information are obtained through neural network processing of memory, forgetting and updating mechanisms; a convolutional encoder: accepts hidden state vectors, divides the vectors into two parts, and combines the two parts using a gating unit to extract semantic features of the text. Based on the pre-training language model BART, a text abstract generation model with a nested multi-layer Transformer encoder-decoder architecture is built, so that the two problems of too long length of the abstract model, non-precise content and fact consistency of the generated abstract are solved. The application can generate an abstract within a given expected length, and the ROUGE score of the generated abstract is improved compared with other models in the current research field.

Description

Technical Field

[0001] This invention relates to a generative text summarization method and system based on the improved pre-trained language model BART, which integrates semantic enhancement and length awareness, and belongs to the field of text summarization in natural language processing. Background Technology

[0002] With the development of the information age and the widespread use of mobile devices, people can access various types of information anytime, anywhere, including scientific research literature, trending news, social media, and political and economic developments. However, the explosive growth in the volume of this information makes it difficult to quickly and accurately obtain important information. Therefore, an effective technology is needed to help people quickly summarize and understand information, improving information processing efficiency. Text summarization technology is an effective method to solve this problem.

[0003] Early text summarization was primarily rule-based, requiring significant human intervention. This involved carefully designed algorithms to count the frequency of words and sentences and extract features, then generating summaries using pre-defined rules. With the dramatic improvement in computing power and the rapid development of data science, deep learning-based text summarization techniques have gradually become mainstream. Following the advent of the Transformer model, numerous pre-trained language models based on it have enabled models to acquire more general semantic representations during training on large-scale corpora, achieving remarkable success in various downstream tasks of natural language processing.

[0004] The Transformer is an attention-based sequence-to-sequence model. Its encoder and decoder are composed of stacked self-attention modules, employing positional encoding and residual connections, and using layer normalization to improve the training performance of deep networks. The Transformer model possesses strong general feature extraction capabilities and is the main structure of pre-trained language models. In the Transformer architecture's summarization generation model, the source text is transformed into word vector representations through word embeddings and positional encoding during the encoding stage. Then, a multi-head attention mechanism is used to obtain contextual representations, which are input into the decoder. During the decoding stage, the output is generated progressively based on the prediction results and positional information from the previous time step.

[0005] Currently, pre-trained language models transfer prior language knowledge learned from relevant domain datasets to downstream tasks to improve learning performance on different target tasks. This has yielded significant results in text generation and has become a powerful tool for text summarization. Downstream tasks are trained by jointly fine-tuning pre-training parameters and introducing specific parameters for the minimum task, which has become the most common adaptation strategy for most pre-trained language models. With the increasing application scenarios for summarization, the ability for models to control the length of generated summaries and select the correct semantic information within a limited length range has become a new requirement. Summary of the Invention

[0006] This invention, based on the pre-trained language model BART, constructs a text summarization generation model with a nested multi-layer Transformer encoder-decoder architecture. This addresses two major problems of current summarization models: excessively long generated summaries, insufficient conciseness, and inconsistencies in factual accuracy. The proposed text summarization model can generate summaries within a given expected length, and its ROUGE score is improved compared to other models in the current research field, resulting in a more semantically consistent summary with the facts.

[0007] The technical solution adopted in this invention is a text summarization generation method based on BART-BiGRU-LAAM, which includes the following steps:

[0008] Step 1: Collect text data and summaries, construct a text-summary pair dataset, perform text preprocessing operations such as word segmentation and stop word removal on the dataset, and send the preprocessed text information into the word embedding layer of the BART model encoder.

[0009] Step 2: Reconstruct the multi-head self-attention distribution of the BART decoder using LAAM. Simultaneously, while calculating the output at each time step, the remaining number of tokens to be generated is given. t Enhance the top l terms with higher attention scores in the attention distribution between summary terms and source document terms at each time step. t To increase the probability of selecting these terms, the model is given several lexical units, helping it to choose important information from the source document. As decoding progresses, attention to end-of-sequence markers is enhanced, allowing the model to control the length of the output summary.

[0010] Step 3: After encoding with the pre-trained embedding matrix, the initial representation vector of the text, X = (x1,...,x), can be obtained. m ), where m is the number of text terms, this vector is input into the position embedding unit and the BiGRU layer of the BART encoder for feature extraction. A single GRU feature encoding unit extracts the semantic features of the input sequence through update gates and reset gates, and memorizes them to obtain the latent state vector h at the term level. t The BiGRU unit processes the text sequence from both directions, resulting in two hidden state vectors. These two hidden state vectors are combined and input into the convolutional encoder of the GLU unit. The output of each convolutional kernel window in each layer is divided into two parts, and the information from both parts is fused. The output of the last convolutional layer then becomes the semantic representation vector of the source document. Combining the outputs of the two encoders, the semantically enhanced context vector c, used to generate the probabilities of word distributions, can be calculated. t :

[0011]

[0012] Where t is the position of the word to be generated in the abstract; i is the position of the word in the source text; For length-perceived attention distribution; a′ t,i The attention distribution chosen for the input is the product of the remaining length and the cross-attention; m is the number of tokens in the source text; h′ i h is the weighted sum of the outputs of the two encoders. i This refers to the hidden state of the last layer of the encoder in the pre-trained model, i.e., the context representation vector. α is the semantic representation vector obtained from the BiGRU unit; α is a hyperparameter controlling the scaling factor.

[0013] Step 4: The context vector is fed through a linear layer to obtain the Logits of the position to be generated. After deduplication of the source text data, One-Hot encoding is used to encode it into a 0 / 1 vector for dictionary-based semantic enhancement. This vector is then input into a fully connected layer to obtain the word distribution vector of the source text. The Logits of the position to be generated are weighted and averaged with the word distribution vector of the source text to obtain the Logits of the position to be generated after two stages of semantic enhancement.

[0014] Step 5: For the decoder output, the model uses a pointer network to determine whether the terminology to be generated is produced by the dictionary or copied from the source text, and uses an overlay mechanism to prevent the model from generating duplicate summaries. After training the model according to steps 1 to 5, input text data and output a summary of that text.

[0015] Optionally, when training a text summarization model based on BART-BiGRU-LAAM, a loss function is used. As a training objective:

[0016]

[0017] Where T is the total time step; loss t This is a single-step loss function; To optimize the negative logarithm function; ∑ i min(a t,i ,c t,i ) represents the coverage loss; c t,i λ represents the coverage of the attention mechanism received by each word; λ is a hyperparameter that controls the proportion.

[0018] Specifically, the model architecture of this invention consists of:

[0019] A BART-large pre-trained language model with a nested 12-layer Transformer encoder-decoder architecture, where each layer includes:

[0020] BART's word embedding layer transforms preprocessed text data into dynamic embedding vectors using a pre-trained embedding matrix. These dynamic embedding vectors are then input into two modules for feature extraction:

[0021] First, there is the BART encoder, a conventional pre-trained language model feature extraction unit, which includes a position embedding unit: representing the absolute position information of words in the sequence with a uniform distribution in the [0,1] interval and adding it to the dynamic embedding vector; a multi-head attention module: performing multi-head attention calculation on the input sequence and adding it to the dynamic embedding vector of each word; a layer normalization module: performing residual connection and Softmax normalization operations; a feedforward layer: introducing a non-linear activation function to perform two linear transformations on the attention output, first mapping the data to a high-dimensional space and then to a low-dimensional space to extract deeper features; and a layer normalization module: performing normalization in the same way as the aforementioned module.

[0022] Furthermore, this invention utilizes a BiGRU feature extraction unit to capture semantic information with contextual temporal sequence. The extraction unit includes a BiGRU module: capturing contextual temporal sequence information of the embedded vector from both positive and negative directions, and obtaining the hidden state with semantic information through neural network processing with memory, forgetting and update mechanisms; and a convolutional encoder: accepting the hidden state vector, dividing it into two parts, and combining the two using a gating unit to extract the semantic features of the text.

[0023] Furthermore, the semantic feature vector and expected length are input into the BART decoder for summary generation. The BART decoder includes a multi-head attention module with a masking mechanism: masking the attention of words after the generation position, allowing the model to follow the unidirectional generation principle; a layer normalization module: normalizing the attention output; a multi-head attention module: using a cross-attention mechanism to obtain the attention distribution between the word embedding vectors of the source text and the reference summary. In this invention, a LAAM mechanism is introduced into this distribution, using the remaining length of the summary generation as an additional input to the decoder, which is used as the weight for selecting word attention to adjust the summary generation length and select more suitable information; and a feedforward layer and a layer normalization module: functioning the same as the aforementioned modules with the same names.

[0024] Furthermore, after the decoder attention output at each time step, it enters the following two modules for processing:

[0025] Dictionary tagging module: Based on the idea of ​​improving the generation probability of original text words, this invention uses One-Hot encoding to add the original text words contained in the dictionary to the generation probability calculation, thereby improving the model's attention score for original text words; Pointer generation network: When generating unknown characters in the dictionary, the corresponding words in the original text are copied and replaced, solving the inherent OOV problem of generative text summarization.

[0026] Compared to existing technologies, this invention offers the following advantages: It provides a generative text summarization method that integrates semantic enhancement and length control. This method trains a BART-BiGRU-LAAM-based deep learning network model on a training set through text-summarization. Semantic enhancement in both the encoding and decoding stages fully extracts the word-level and contextual features of the source text. Simultaneously, it reconstructs the attention distribution based on the desired length, enabling the model to better learn the semantic information of the source text and possess length control capabilities. This invention improves the fluency, factual consistency, and conciseness of generated summaries, demonstrating strong practicality and broad application prospects. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the BART-BiGRU-LAAM system architecture.

[0028] Figure 2 This is a flowchart illustrating the implementation of the BART-BiGRU-LAAM text summarization method.

[0029] Figure 3 This is a schematic diagram of the LAAM length-aware attention mechanism.

[0030] Figure 4 This is a schematic diagram of the BiGRU lexical feature extraction structure.

[0031] Figure 5 This is a schematic diagram of the GLU unit convolutional encoder structure.

[0032] Figure 6 This is a diagram of the dictionary tag structure.

[0033] Figure 7 The figure shows the comparative experimental results of the examples on the two datasets. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] The technical solution adopted in this invention is a text summarization generation method based on BART-BiGRU-LAAM, and the system structure diagram is shown below. Figure 1 As shown, based on the pre-trained BART model, a BiGRU feature extraction module is introduced to the encoding end, and semantic enhancement is performed in two stages through dictionary tagging at the decoding end. LAAM is also added to improve the model's ability to control the length of the generated summary, optimizing the summary generation model and enabling it to select the correct semantic information within a limited length range, thereby improving the quality of the generated summary. According to... Figure 2 The specific implementation method of the present invention is as follows:

[0036] The first stage involves collecting a dataset and organizing it into training, validation, and test sets in the form of source text-summary pairs. The dataset is then subjected to text preprocessing operations such as word segmentation and stop word removal. The preprocessed text is represented as a sequence of words and input into the pre-trained embedding matrix of the BART model.

[0037] The second stage involves using LAAM to incorporate a length-aware mechanism into the multi-head self-attention distribution of the BART decoder, such as... Figure 3 As shown. Specifically, input selection attention (attn) is introduced. is and sequence end marker attention attn eos Length awareness is used to control the decoding end time and the selection of information during the decoding stage. The calculation formula is as follows:

[0038] a′ t,i =w t,i ×a t,i

[0039]

[0040] Where, a′ t,i Select attention for the input; w t,i Attention weights; a t,i For cross attention; t p is the remaining length; i This is a quantitative metric for the attention score of the abstract's semantic units, where the top l with the highest scores are... t p i The value is 1 for all others and 0 for the rest; a t ′ ,m 'l' represents the end-of-sequence attention marker; 'l' represents the desired length.

[0041] When calculating the output at each time step, the top l terms with higher attention scores in the attention distribution between the augmented summary terms and source document terms. t To increase the probability of selecting these words, the model selects important information from the source document. t This determines the remaining number of generated tokens. Furthermore, the attention score for the sequence end marker is increased as the generated length increases, enabling the model to terminate the decoding phase before the desired length.

[0042] The third stage encodes the pre-trained embedding matrix to obtain the initial text representation vector X = (x1,...,x...). mThe BART encoder's position embedding unit and BiGRU unit are respectively used for feature extraction. The BART encoder works as follows: 1) Generating position embeddings: BART uses position embedding vectors to represent the position of each word in the sentence, allowing the model to learn the context of each word. 2) Generating type embeddings: BART uses type embedding vectors to represent whether the input text is the source text or the next sentence prediction, in order to learn contextual information. 3) Multi-head attention mechanism: BART uses a multi-head attention mechanism to capture different features of lexical units and learn the interactions between lexical units. 4) Pooling: The sentence vector representation is generated by average pooling or max pooling from the vector representations of all lexical units. 5) Fully connected layer: One or more fully connected layers are used to further process the merged vectors and generate the final output vector representation. 6) Output: The output is a vector representing the context. A single GRU feature encoding unit extracts the semantic features of the input sequence through update and reset gates and memorizes them to obtain the hidden state vector h to be passed to the next time step. t The calculation is as follows:

[0043] z t =σ(x t W xz +h t-1 W hz +b z )

[0044] r t =σ(x t W xr +h t-1 W hr +b r )

[0045]

[0046] Among them, z t For the update gate; σ is the sigmoid function; x t Input information for the current time step; r t To reset the door; For candidate hidden state; h t-1 This is the hidden state of the previous time step; W xz W hz W xr W hr W xh and W hh b represents the weight parameters for each unit. z b r and b h This is the bias parameter.

[0047] like Figure 4As shown, the BiGRU unit processes the text sequence from both positive and negative directions to obtain two hidden state vectors, which are denoted as follows: and By combining the hidden state vectors from the two directions and inputting them into the convolutional encoder of the GLU unit, we can obtain the word-level hidden representation after feature extraction. as follows:

[0048]

[0049] Where g is a GLU unit. For example... Figure 5 As shown, the output of each convolutional kernel window in each layer is divided into two parts and the information from the two parts is fused. At this point, the output of the last convolutional layer is the semantic representation vector h of the source document. i c The semantically enhanced context vector, used to generate the probability distribution of word terms, can be obtained through calculation.

[0050] The fourth stage generates the Logits of the position to be generated, which have undergone two stages of semantic enhancement. Specifically, the context vector is fed through a linear layer to obtain the Logits of the position to be generated, as follows:

[0051] l = Wc t +b

[0052] Among them, c t is the context vector; W and b are the learnable weights and bias parameters.

[0053] like Figure 6 As shown, after deduplication of the source text data, One-Hot encoding is used to encode it into a 0 / 1 vector for dictionary-based semantic enhancement. This vector is then input into a fully connected layer to obtain the word distribution vector of the source text, denoted as L. A weighted average of l and L is then performed to obtain the Logits of the position to be generated after two stages of semantic enhancement, as follows:

[0054]

[0055] Wherein, the weight parameter ω t The exponential moving average algorithm is used for updating.

[0056] The fifth stage introduces a pointer generation network, using probability value p gen The pointer determines whether the word to be generated is generated by a dictionary or copied from the source text, calculated as follows:

[0057]

[0058] Where, x t This is the input to the decoder; and b ptrFor weights and bias parameters; s t P(w) represents the hidden state of the last layer of the decoder; P(w) represents the generation probability distribution of the word w to be generated.

[0059] A coverage mechanism is employed, defining the repetition penalty based on the coverage of each word received by the attention mechanism to avoid generating duplicate text. The model uses the optimized negative logarithmic loss function as its core, incorporating coverage loss to obtain a composite loss. After training the model on the dataset through stages one through five, it can output a summary of the input text data.

[0060] This invention proposes a text summarization generation system based on BART-BiGRU-LAAM. Employing the aforementioned BART-BiGRU-LAAM-based text summarization method, it includes: a training set construction module: collecting text information and summaries to construct a text-summary pair dataset; a text preprocessing module: preprocessing each training sample in the training set, including text segmentation and stop word removal of the text and summaries contained in the samples; a feature extraction module: generating word vectors of the source text through a pre-trained embedding matrix, and combining positional information to obtain word-level representation vectors of the text and summaries; and a length-aware module: using length information... The multi-head attention distribution of the reconstructed decoder enables the model to control the length of the generated summary; the network training module: is used to input the text representation vectors of the text and the summary into the BiGRU feature extraction unit and the encoder of the pre-trained language model, respectively, to obtain the context representation vector and semantic representation vector of the source text. The two vectors are combined with the attention mechanism to obtain the context vector, which is input into the decoder. Using the probability distribution vector of the decoder output representation vector and the annotations in the training set, the network is trained with the goal of optimizing the negative log loss function to obtain the summary generation model; the text summarization module: uses the trained network model to analyze and process the input text and output a summary of the text.

[0061] The experiments in this example used Python version 3.9, PyTorch 2.0.1 framework, CUDA 11.8, an RTX 4090 GPU with 24GB of VRAM, and a pre-trained language model based on BART-large. The encoder and decoder layers were set to 12, the multi-head attention layer to 8, and Dropout to 0.1. The vocabulary size was 50,000 words, the word vector dimension was 768, and a linear warmup mechanism was used with an initial learning rate of 3e-5, stopping training when the learning rate was less than 1e-8. The hidden layer had 768 units, the fully connected layer had 2048 hidden layers, the batch size was 32, and the epochs were 40.

[0062] The experiment used the English datasets CNN / DM and Xsum. Dataset details are as follows:

[0063] CNN / DM 287228 13369 11491 789 55 Xsum 204046 11332 11334 431 23

[0064] The variance v between the generated summary length and the expected length is used as an evaluation metric to verify the model's length control capability.

[0065]

[0066] Where l is the desired length; l i n is the length of the generated summary; n is the number of test samples.

[0067] The ROUGE-1, ROUGE-2, and ROUGE-L metrics were used to measure the character repetition rate between the generated and reference summaries to validate the model's information selection ability.

[0068]

[0069] Where S is the set of sentences in the reference summary; n is the length of the n-gram; Count(gram n ) represents the number of n-grams in the reference summary; match(gram n ) represents the number of n-grams that appear simultaneously in both the reference and output summaries; R lcs For recall rate; P lcs β represents the accuracy; β is the hyperparameter controlling the proportion.

[0070] To more intuitively demonstrate the superior performance of the embodiments of the present invention on the two datasets, comparative experimental results are provided here as follows: Figure 7 As shown in the table below, the highest scores of the embodiments of the present invention on the CNN / DM datasets ROUGE-1, ROUGE-2, and ROUGE-L are 44.58, 20.85, and 41.34, respectively; and on the Xsum dataset, they are 45.53, 21.82, and 36.85, respectively; furthermore, the length variances are 0.058 and 0.031, respectively. Among all the comparison models, it exhibits the best performance, thereby verifying the effectiveness of the text summarization generation method based on BART-BiGRU-LAAM proposed in this invention.

[0071]

[0072]

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations of methods, systems, and computer program products according to embodiments of this application. It should be understood that each step in the flowchart and combinations of steps in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to implement the specified functions.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means.

[0076] The above are preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art can make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention, and whose resulting functions do not exceed the scope of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A text summarization method based on BART-BiGRU-LAAM, characterized in that, Includes the following steps: Step 1: Collect text data and summaries, construct a text-summary pair dataset, perform word segmentation and stop word removal on the dataset, and send the preprocessed text information into the word embedding layer of the BART model encoder; Step 2: Reconstruct the multi-head self-attention distribution of the BART decoder using LAAM. Simultaneously, while calculating the output at each time step, the remaining number of tokens to be generated is given. t Enhance the attention distribution between summary terms and source document terms at each time step, focusing on terms with higher attention scores. t Each word; as decoding progresses, increase attention to the end-of-sequence marker to control the length of the output summary; Step 3: After encoding with the pre-trained embedding matrix, the initial representation vector of the text is obtained as X = (x1,...,x...). m ), where m is the number of text terms. This initial representation vector is input into the position embedding unit and BiGRU layer of the BART encoder for feature extraction. A single GRU feature encoding unit extracts the semantic features of the input sequence through update and reset gates and memorizes them to obtain the latent state vector h of the term-level encoding. t The BiGRU unit processes the text sequence from both directions to obtain two hidden state vectors. These two hidden state vectors are combined and input into the convolutional encoder of the GLU unit. The output of each convolutional kernel window in each layer is divided into two parts, and the information from both parts is fused. The output of the last convolutional layer is the semantic representation vector of the source document. The combined outputs of the two encoders are used to calculate the semantically enhanced context vector c, which is used to generate the probability distribution of word terms. t : Where t is the position of the word to be generated in the abstract; i is the position of the word in the source text; For length-perceived attention distribution; a′ t,i The attention distribution chosen for the input is the product of the remaining length and the cross-attention; m is the number of tokens in the source text; h′ i h is the weighted sum of the outputs of the two encoders. i This refers to the hidden state of the last layer of the encoder in the pre-trained model, i.e., the context representation vector. α is the semantic representation vector obtained from the BiGRU unit; α is a hyperparameter controlling the ratio. Step 4: The context vector is fed through a linear layer to obtain the Logits of the position to be generated; after deduplication of the source text data, it is encoded into a 0 / 1 vector using One-Hot encoding for dictionary-based semantic enhancement, and this vector is input into a fully connected layer to obtain the word distribution vector of the source text; the Logits of the position to be generated and the word distribution vector of the source text are weighted and averaged to obtain the Logits of the position to be generated after two stages of semantic enhancement. Step 5: For the decoder output, the pointer network determines whether the positional words to be generated are generated from the dictionary or copied from the source text, and the overlay mechanism avoids duplicate summary generation. After training according to steps 1 to 5, input text data and output text summary.

2. The text summarization method based on BART-BiGRU-LAAM according to claim 1, characterized in that, When training a text summarization model based on BART-BiGRU-LAAM, a loss function is used. As a training objective: Where T is the total time step; loss t This is a single-step loss function; To optimize the negative logarithm function; ∑ i min(a t,i ,c t,i ) represents the coverage loss; c t,i λ represents the coverage of the attention mechanism received by each word; λ is a hyperparameter that controls the proportion.

3. The text summarization method based on BART-BiGRU-LAAM according to claim 1, characterized in that, A text summarization generation model is built based on the pre-trained language model BART, featuring a nested multi-layer Transformer encoder-decoder architecture. The text summarization generation model consists of a BART-large pre-trained language model with a nested 12-layer Transformer encoder-decoder architecture, where each layer includes: BART's word embedding layer: It transforms preprocessed text data into dynamic embedding vectors through a pre-trained embedding matrix. The dynamic embedding vectors are then input into the following two modules for feature extraction. In the pre-training process, the first step is the BART encoder, a conventional pre-trained language model feature extraction unit, which includes a position embedding unit: representing the absolute position information of words in the sequence with a uniform distribution in the [0,1] interval and adding it to the dynamic embedding vector; In the feature extraction process of the two modules, the multi-head attention module performs multi-head attention calculation on the input sequence and adds it to the dynamic embedding vector of each word; the layer normalization module performs residual connection and Softmax normalization operations. Feedforward layer: Introduces a non-linear activation function to perform two linear transformations on the attention output, first mapping the data to a high-dimensional space and then to a low-dimensional space to extract deep features; Layer normalization module: used for normalization; The BiGRU feature extraction unit captures semantic information with contextual temporal sequence. The extraction unit includes a BiGRU module: capturing contextual temporal information of the embedding vector from both positive and negative directions, and obtaining the hidden state with semantic information through neural network processing with memory, forgetting and update mechanisms; and a convolutional encoder: accepting the hidden state vector, dividing it into two parts and combining the two using a gating unit to extract the semantic features of the text.

4. The text summarization method based on BART-BiGRU-LAAM according to claim 3, characterized in that, The semantic feature vector and expected length are input into the BART decoder for summary generation; the BART decoder includes a multi-head attention module with a masking mechanism: masking the attention of words after the generation position, so that the model follows the unidirectional generation principle; and a layer normalization module: normalizing the attention output; Multi-head attention module: It uses a cross-attention mechanism to obtain the attention distribution between the word embedding vectors of the source text and the reference summary. The LAAM mechanism is introduced into this attention distribution. The remaining length of the generated summary is used as an additional input to the decoder. It is used as the weight for selecting word attention to adjust the length of the generated summary and select more suitable information. Feedforward layer and layer normalization module: The function is the same as the aforementioned module with the same name.

5. The text summarization method based on BART-BiGRU-LAAM according to claim 1, characterized in that, After the decoder attention output at each time step, the data is processed in the following two modules: Dictionary tagging module: Based on improving the generation probability of original words, one-hot encoding is used to add the original words contained in the dictionary to the generation probability calculation, thereby improving the attention score for original words; Pointer generation network module: When generating unknown characters for the dictionary, it copies the corresponding tokens from the original text and replaces them.

Images