A text summarization method based on global autoregression

Abstract

This invention proposes a text summarization method based on global autoregression, comprising: S1, inputting text into a first encoder to obtain the semantic representation of the text; S2, inputting the semantic representation H of the text into a decoder to generate a candidate summary; S3, inputting the candidate summary and the text semantic representation H into a second encoder to obtain the perceptual semantic representation H of the candidate summary; S4, inputting the text semantic representation H and the perceptual semantic representation H of the candidate summary into the decoder to generate a new candidate summary; S5, repeating the above steps S3 to S4 until a preset number of repetitions is reached, and finally the candidate summary output by the decoder is the text summary. Through the global autoregression method proposed in this invention, the generated text can be iteratively rewritten repeatedly, continuously improving the content of the output text and enhancing its quality.

Description

Technical Field

[0001] This invention relates to the fields of natural language generation and deep learning technology, and in particular to a text summarization method based on global autoregression. Technical Background

[0002] With the rapid development of information technology, the amount of textual information people have access to has exploded. At the same time, people's demand for various types of information is also gradually increasing, and their requirements for the speed and accuracy of acquiring effective information are correspondingly rising. Among the massive amounts of data generated daily, text data constitutes the majority. How to quickly extract key content from this vast amount of textual information has become an urgent need in people's daily lives. Text summarization technology is a key technology in this field, aiming to summarize lengthy texts into more concise language while retaining key information. Specifically, it refers to the technology of using computers to process input text and output a shorter text that is more concise than the original but still summarizes its meaning.

[0003] Early text summarization techniques were extractive tasks, involving selecting key sentences from the original text to condense the entire long text. In these methods, the extractive summarization problem was modeled as a binary classification problem, meaning the summarization task could be completed by determining whether a sentence was a summary sentence. However, this task paradigm lacked flexibility and had a low upper limit, and was subsequently replaced by generative task paradigms. Generative text summarization techniques are text-to-text natural language generation tasks, where computers, based on understanding the original text, mimic human writing habits to generate a text word by word.

[0004] Generative text summarization technology is mainly based on an autoregressive language model, consisting of an encoder and a decoder. The original text is input into the encoder to obtain a semantic representation of the input text. Then, the decoder combines the output of the encoder to generate the summary text character by character through an autoregressive process. The output of each iteration is used as the input for the next iteration, and the process is repeated until the entire summary text is obtained. The specific process is as follows:

[0005] y i+1 =f(y0,y1,…,y i |X)

[0006] Among them, y i Let f(·) represent the output of the i-th iteration, X represent the original input text, and f(·) represent the summarization model.

[0007] In current text generation, both the encoder and decoder are implemented using pre-trained Transformer models. To improve the speed of model training and inference, the decoder uses attention masks to control the flow of information, enabling parallel computation across multiple iterations. This attention mask manifests as unidirectional attention, where each token in the sequence can only focus on tokens preceding it, meaning each token only considers the preceding context and cannot understand the following context. Furthermore, the model is confident in the decoder's output, assuming each round's output is correct and using it as input for the next round's output. However, if inappropriate content is output in a particular iteration, it will lead to error accumulation in subsequent iterations, ultimately degrading the overall quality of the output summary. Summary of the Invention

[0008] To address the issues of missing context due to unidirectional attention in the model decoder during text generation, and the overconfidence of autoregressive models in the context of the output, this invention provides a global autoregressive method. It designs a Transformer encoder-decoder model with two input data streams and one output data stream, which can perceive the correct content in the candidate summary and rewrite the summary based on the context of the candidate summary.

[0009] This invention proposes a text summarization method based on global autoregression, comprising:

[0010] S1. Input the text into the first encoder to obtain the semantic representation of the text.

[0011] S2, Semantic representation of text Input to decoder to generate candidate summaries;

[0012] S3. Combine candidate summaries and text semantic representations The input is given to the second encoder to obtain the perceptual semantic representation H of the candidate summary;

[0013] S4. Representing the semantics of text The perceptual semantic representation H of the candidate summary is input into the decoder to generate a new candidate summary;

[0014] S5. Repeat steps S3 to S4 until the preset number of repetitions is reached. The final candidate summary output by the decoder is the text summary.

[0015] The first encoder, the second encoder, and the decoder together form a global autoregressive model.

[0016] Furthermore, a cross-attention layer is added to each layer of the Transformer network of the second encoder to receive the output of the first encoder and the output of the normalization layer above it; in each layer of the Transformer network of the decoder, the first cross-attention layer receives the output of the first encoder and the output of the normalization layer above it, and the second cross-attention layer receives the output of the second encoder and the output of the normalization layer above it. The outputs of the two attention layers are vector-summed and input into the feedforward network layer.

[0017] Furthermore, both the encoder and the decoder have N Transformer layers, and the two encoders share the parameters of the self-attention layer and feedforward network layer of the first N-1 Transformer layers.

[0018] Furthermore, the training method for the global autoregressive model includes:

[0019] S10. Train the global autoregressive model using the original text and real summaries in the dataset, optimize the cross-entropy loss, and generate static candidate summaries for all the original texts in the dataset.

[0020] S20. Input the original text from the dataset training into the first encoder to obtain the semantic representation of the original text. The static candidate summary is input into the second encoder along with the static candidate summary to obtain the perceptual semantic representation H of the static candidate summary;

[0021] S30. The two encoders used in the construction of the global autoregressive model are identical offline encoders. The original text and the real summaries from the dataset are sequentially input into the first offline encoder and the second offline encoder to obtain the offline perceptual semantic representation H of the real summaries. * ;

[0022] S40. Calculate the perceptual semantic representation H of the static candidate summary and the offline perceptual semantic representation H of the real summary. * semantic similarity;

[0023] S50. Calculate the loss function of the global autoregressive model and adjust the parameters of the global autoregressive model through the backpropagation algorithm.

[0024] Further, in step S10, the step of generating a static candidate summary includes:

[0025] S101. After segmenting the original text in the dataset into words, convert it into word embedding sequences through the input embedding layer, and then input it into the first encoder to obtain the semantic representation of the original text. The system segments the real summaries in the dataset, converts them into word embedding sequences through the output embedding layer, and then inputs them into the decoder. Simultaneously, it incorporates the semantic representation of the original text. The input decoder uses the cross-attention layer connected to the first encoder in each Transformer network layer. At this time, the cross-attention layer connected to the second encoder in the decoder does not play a role.

[0026] Calculate the cross-entropy loss based on the vocabulary probability distribution output by the decoder, and update the parameters of the first encoder and decoder.

[0027] S102. Input the original text of the dataset into the first encoder of the global autoregressive model trained in step S101 to obtain the semantic representation of the original text. and will The input decoder is connected to the cross-attention layer of the Transformer network in each layer of the input decoder, and the decoder iteratively generates static candidate summaries through autoregression.

[0028] Further, in step S30, the parameters of the offline encoder are estimated using a moving average of the encoder parameters in the global autoregressive model, and the parameter update method is as follows:

[0029] ξ←μξ+(1-μ)θ

[0030] Where ξ and θ are the parameters of the offline encoder and the encoder parameters of the global autoregressive model, respectively, and μ is the weight.

[0031] Further, step S40 includes:

[0032] S401, Using a multi-head attention network, the offline perceptual semantic representation H is... * The semantic features are integrated into the perceptual semantic representation H to obtain a representation that simultaneously possesses H * Semantic representation of H semantic features The calculation method is as follows:

[0033]

[0034] S402, Calculate the true summary Y and the static candidate summary Y * Perceptual semantic similarity between , used to represent the true summary Y and the static candidate summary Y * The semantic feature differences between them are analyzed, and the semantic similarity is calculated as follows:

[0035]

[0036]

[0037] in and The semantic features are those calculated in step S401 above.

[0038] Furthermore, in S50, the similarity loss function is calculated using the following formula:

[0039]

[0040] Minimize similarity loss This is used to train the global autoregressive model to learn to perceive the correct content in the candidate summaries, and to adjust the parameters of the global autoregressive model.

[0041] Furthermore, step S50 also includes training a global autoregressive model through reinforcement learning, including:

[0042] S601, Representing the semantics of text The input decoder connects to the cross-attention layer of the first encoder in each Transformer network layer, and simultaneously inputs the perceptual semantic representation H into the cross-attention layer of the second encoder in each Transformer network layer of the decoder. The decoder uses a sample sampling strategy to generate a sampled summary and a greedy strategy to generate a baseline summary.

[0043] S602. Based on the character matching degree between the sampled summary S and the corresponding real summary in the dataset, the quality of the sampled summary is scored. This score serves as a reward for the model's behavior in generating this summary. The score calculation method is as follows:

[0044]

[0045]

[0046]

[0047] in Y represents the static candidate summary in the dataset, Y represents the true summary in the dataset, and {·} represents the set operation of word sequence extraction.

[0048] S603. To prevent the global autoregressive model from always receiving positive rewards and falling into the zero-reward range during the reinforcement learning training process, the model needs to be penalized when the generated sampling summary S is not good enough.

[0049] S604, The reward obtained by the model in generating the sampled summary S is the baseline summary. The difference between the scores of the sample summary and the score of the sampled summary S;

[0050] S605. Calculate the reinforcement learning loss using the following formula:

[0051]

[0052] Where S represents the sampled summary. Indicates baseline summary, Let X represent the static candidate abstract, l represent the original text to be abstracted, and r(·) represent the sequence length of the sampled abstract S.

[0053] Furthermore, step S50 also includes: training a global autoregressive model using static candidate summaries and real summaries, the steps of which include:

[0054] S701. The static candidate words in the dataset are processed through the output embedding layer to obtain word embedding sequences, which are then input into the decoder of the autoregressive model. At the same time, the text semantic representation is processed. The input is the cross-attention layer in each Transformer network layer of the decoder connected to the first encoder. At this point, the cross-attention layer in each Transformer network layer of the decoder connected to the second encoder is inactive. The cross-entropy loss is calculated based on the vocabulary probability distribution of the encoder's output, as follows:

[0055]

[0056] Where q(v) represents the label smoothing function;

[0057] S702. After segmenting the real summaries in the dataset, the output embedding layer is input to the decoder, and the text semantic representation is simultaneously processed. The perceptual semantic representation H is input into the cross-attention layer connecting the first encoder and the cross-attention layer connecting the second encoder in each layer of the Transformer network of the decoder, respectively. The cross-entropy loss function is calculated based on the vocabulary probability distribution output by the decoder, as follows:

[0058]

[0059] Preferably, the sum of the loss functions in steps S50, S605, S701, and S702 is the final loss function of the global autoregressive model. This loss is minimized by gradient descent, and the autoregressive model parameters are updated.

[0060] The generative summarization method based on global autoregression provided by this invention has the following advantages compared with existing generative summarization methods:

[0061] 1. The global autoregressive model proposed in this invention introduces bidirectional attention to the output text, which can perceive the entire context of the candidate summary.

[0062] 2. The semantic-aware coding method proposed in this invention can detect the correct content in the candidate abstract.

[0063] 3. The abstract rewriting method proposed in this invention can correct errors in candidate abstracts, thus alleviating the problem of error accumulation caused by overconfidence in the output text in previous methods.

[0064] 4. The global autoregressive method proposed in this invention can iteratively rewrite the generated text, continuously improve the content of the output text, and thus continuously improve the quality of the output text. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram illustrating the application process of a global autoregressive model according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the network structure of a global autoregressive model according to an embodiment of the present invention;

[0068] Figure 3 This is a flowchart illustrating a semantic perception method based on contrastive learning according to an embodiment of the present invention.

[0069] Figure 4 This is a schematic diagram of the initial summary generation process of a global autoregressive model according to an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the non-first-time summary generation process of a global autoregressive model according to an embodiment of the present invention;

[0071] Figure 6 This is a schematic flowchart of an autoregressive generation method according to an embodiment of the present invention. Specific implementation methods

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] To make the technical solutions and advantages in the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments in this application can be combined with each other.

[0074] This invention provides a text summarization method based on global autoregression, the method flow is as follows: Figure 1 As shown, the method includes the following steps:

[0075] S1. Input the text into the first encoder, and the first encoder outputs the text semantic representation. ,like Figure 4 As shown.

[0076] S2, Representing the semantics of the text The input is to the decoder, which iteratively decodes using an autoregressive approach to generate candidate summaries, such as... Figure 4 As shown.

[0077] S3. Input the candidate summary and the text semantic representation obtained in S2 into the second encoder to obtain the perceptual semantic representation H of the candidate summary, such as... Figure 5 The process is shown in the solid line diagram.

[0078] S4. Represent the semantic meaning of the text in S1. The candidate summary-aware semantic representation H from S3 is input into the decoder, which then regenerates new candidate summaries via an autoregressive approach, such as... Figure 5 The process is shown in the solid line diagram.

[0079] S5. Repeat steps S3 to S4 above until the preset number of repetitions is reached, such as... Figure 5 The process is shown by the dashed line.

[0080] The global autoregressive model consists of a first encoder, a second encoder, and a decoder. The first encoder is a typical Transformer encoder with N layers, each consisting of a self-attention layer and a feedforward layer. Compared to the first encoder, the second encoder adds a cross-attention layer to each Transformer layer to receive the output of the first encoder and the output of its previous normalization layer. The decoder, based on the typical Transformer decoder, adds a cross-attention layer to each Transformer layer, resulting in two cross-attention layers. The first cross-attention layer receives the output of the first encoder and the output of its previous normalization layer, while the second cross-attention layer receives the output of the second encoder and the output of its previous normalization layer. The outputs of the two attention layers are vector-summed and then input into the feedforward layer. The specific network structure is as follows: Figure 2 As shown.

[0081] In step S1, the input text is segmented and converted into a token sequence using a vocabulary. The vocabulary is the vocabulary of the language corresponding to the text, and each word has a unique number, called a token. The token sequence is converted into a word embedding sequence through an input embedding layer. In each Transformer network layer, it is encoded sequentially through a self-attention layer and a feedforward network layer, finally outputting the semantic representation of the text. ,like Figure 2 The left part shows the "First Encoder" module.

[0082] In step S2, the text semantic representation obtained in step S1 is... The input to the first cross-attention layer of each Transformer layer in the decoder allows the semantics of the text to be incorporated into the decoding process, such as... Figure 2 As shown in the right-hand "decoder" module, the second cross-attention layer connecting the decoder to the second encoder is not active at this point. The decoder iteratively generates tokens using an autoregressive approach, meaning the output of each iteration is used as the input for the next iteration, until the output terminates with a token or the preset maximum number of iterations is reached, as shown below. Figure 6 As shown. Finally, the obtained token sequence is converted into words through a vocabulary and concatenated one by one to form text, resulting in a candidate summary. The decoder generally uses an autoregressive approach, which is a current technology.

[0083] In step S3, the candidate summary obtained in step S2 is re-segmented and mapped to a word list to obtain a token sequence. This token sequence is slightly different from the token sequence output by the decoder in step S2, ignoring redundant whitespace characters in the candidate summary. The re-segmented token sequence is converted into a word embedding sequence by the input embedding layer and input into the second encoder. Simultaneously, the text semantic representation obtained in step S1 is... The input is fed into the cross-attention layer of each Transformer network in the second encoder, enabling the second encoder to perceive the true semantic features in the candidate summary through cross-attention. Finally, the second encoder outputs the perceptual semantic representation H of the candidate summary, such as... Figure 2 The middle section, the "second encoder" module, is shown.

[0084] In step S4, the text semantic representation from step S1 is performed. The perceptual semantic representation H from step S3 above is input into the cross-attention layer connected to the corresponding encoder in each layer of the Transformer network of the decoder, so that the decoder obtains the semantics of the text and candidate summary during the decoding process, such as... Figure 2 The right-hand "decoder" module is shown. Similar to the decoding process in step S2 above, the decoder also generates new candidate summaries through an autoregressive approach. This newly generated candidate summary retains the correct content of the candidate summary obtained in step S3 and ignores its erroneous parts.

[0085] The global autoregressive model proposed in this invention requires training; the encoder and decoder parameters must be determined before it can be used. Specific training methods include:

[0086] S10. Train a global autoregressive model using the original text and real summaries in the dataset, and optimize the cross-entropy loss; after training on the dataset, the model generates static candidate summaries for all the original texts in the dataset.

[0087] S20. Input the original text into the first encoder to obtain the semantic representation of the original text. The static candidate summary is input into the second encoder along with the static candidate summary to obtain the perceptual semantic representation H of the static candidate summary;

[0088] S30. Construct an offline encoder. The offline encoder has the same structure as the two encoders in the global autoregressive model, but its parameters are not updated via gradient descent. Input the original text and the real summary from the dataset into the first offline encoder and the second offline encoder, respectively, to obtain the offline perceptual semantic representation H of the real summary. * ;

[0089] S40. Calculate the perceptual semantic representation H of the static candidate summary obtained in step S20 and the offline perceptual semantic representation H of the real summary obtained in step S30.* semantic similarity;

[0090] S50. Calculate the loss function of the global autoregressive model, and adjust the parameters of the global autoregressive model so that the model can achieve the offline perceived semantic representation H obtained from the real summary. * For reference, the model learns the perceptual semantic representation H of static candidate summaries, thereby enabling the model to perceive the correct content in the candidate summaries.

[0091] In step S10, the specific process for generating static candidate summaries is as follows:

[0092] S101. After segmenting the original text in the dataset into words, convert it into a word embedding sequence through the input embedding layer, and then input it into the first encoder to obtain the semantic representation of the original text. The system extracts real words from the dataset, converts them into word embedding sequences through the output embedding layer, and then inputs them into the decoder. Simultaneously, it incorporates the semantic representation of the original text. The input decoder has a cross-attention layer in each Transformer network connected to the first encoder, such as... Figure 2 The left part represents the "first encoder" and the right part represents the "decoder" module. At this point, the cross-attention layer connecting the second encoder in the decoder is not active. The cross-entropy loss is calculated based on the vocabulary probability distribution output by the decoder, and the parameters of the first encoder and decoder are updated accordingly. The calculation method is as follows:

[0093] q(v)=1-βif v=y i

[0094] q(v)=β / (N-1)if v≠y i

[0095]

[0096] Where Y represents the true summary in the dataset, X represents the original text in the dataset, q(v) represents the label smoothing function, where β is a hyperparameter, V represents the vocabulary, and N is the vocabulary size.

[0097] S102. Input the original text from the dataset into the model trained in step S101 to obtain candidate summaries, that is, input the original text into the first encoder of the model to obtain the semantic representation of the original text. and will The input decoder contains cross-attention layers in each layer of the Transformer network, such as... Figure 5 As shown, the decoder iteratively generates candidate summaries through an autoregressive approach. The candidate summaries corresponding to each original text are added to the dataset as static candidate summaries during training, eliminating the need for the model to generate candidate summaries during training and reducing model training costs.

[0098] In step S20, the original text is first input into the first encoder to obtain the semantic representation of the original text. The static candidate summary is converted into a token sequence through a vocabulary, which is then passed through an input embedding layer to obtain a word embedding sequence, which is then input into the second encoder. At the same time, the semantic representation of the original text is also processed. The input is the cross-attention layer in each Transformer network layer of the second encoder, which is connected to the first encoder. This allows the second encoder to perceive the correct parts of the candidate summary through the semantics of the original text. Finally, the second encoder outputs the perceptual semantic representation H, such as... Figure 2 The middle section, the "second encoder" module, is shown.

[0099] In step S30, the offline perceptual semantic representation H is obtained. * The steps are as follows:

[0100] S301. To prevent the model from overfitting during the subsequent comparative learning process in step S40, ensuring the encoder only outputs a constant result (i.e., the encoder outputs the same result for any input), this invention constructs an offline encoder. This encoder also has two encoders: an offline first encoder and an offline second encoder. The structure is the same as the two encoders in the global autoregressive model proposed in this invention. However, the parameters of the offline first encoder and offline second encoder are not updated via gradient descent. Instead, their parameters are estimated using the moving average of the parameters of the first encoder and the second encoder, respectively. The parameter update method is as follows:

[0101] ξ←μξ+(1-μ)θ

[0102] Where ξ and θ are the parameters of the offline encoder and the encoder parameters of the model, respectively.

[0103] S302. Input the original text into the offline first encoder to obtain the offline original text semantic representation. The system segments the real summaries in the dataset, converts them into word embedding sequences through an input embedding layer, and then inputs them into an offline second encoder. Simultaneously, it inputs the offline original text semantic representation. The input is the cross-attention layer in each layer of the offline second encoder's Transformer network, which is connected to the offline first encoder. Finally, the offline second encoder outputs the offline perceptual semantic representation H. * ,like Figure 3 The lower half is shown.

[0104] In step S40, the steps for calculating semantic similarity are as follows:

[0105] S401. Use a multi-head attention network to process the offline perceptual semantic representation H obtained in step S302 above. * The semantic features are integrated into the perceptual semantic representation H obtained in step S20 above, to obtain a representation that simultaneously possesses H *Semantic representation of H semantic features The calculation process is as follows: Figure 3 The upper part is shown. In step S302 above, the first encoder and the second encoder retain gradients during network computation to update parameters. The offline first encoder and the offline second encoder update parameters through a moving average of the parameters of the first encoder and the second encoder. Therefore, the offline first encoder and the offline second encoder do not need to retain gradients during network computation, thus reducing the model training cost. The specific calculation method is as follows:

[0106]

[0107]

[0108] MultiHeadAttention=concat(head1,…,head h W O

[0109]

[0110]

[0111] Where Y represents the true summary in the dataset, Y * This represents the static candidate summary obtained in step S102 above, where Q, K, and V represent the three inputs to Attention(·,·,·), and W... i Q W i K and W i V These represent the mapping parameters for the three inputs Q, K, and V, respectively, and W... O The parameters represent the mapping parameters for multi-head attention, concat() represents the concatenation operation, h represents the number of attention heads, and softmax() represents the normalization exponential function. The perceptual semantic representation H obtained in step S20 above is fused with the offline perceptual semantic representation H obtained in step S30 above through a multi-head attention mechanism. * The semantic representation obtained after semantic features are obtained has the semantic features of both. Because the semantic features are asymmetric, H and H are swapped. * The order of calculation is obtained If H and H * If their semantic features are consistent, then The difference between H and and H * The difference should be small.

[0112] S402, Calculate the true summary Y and the static candidate summary Y *The perceived semantic similarity between the true summary Y and the static candidate summary Y is represented by the similarity between the two summaries. * The semantic feature differences between them. Because similarity calculation is asymmetric, the similarity loss function is calculated as follows:

[0113]

[0114]

[0115]

[0116] in and The semantic features are those calculated in step S401 above.

[0117] In step S50, the step of calculating the loss function includes:

[0118] Minimize similarity loss This allows the parameters of the first encoder, second encoder, and decoder in the global autoregressive model to be adjusted according to the backpropagation algorithm. This ensures that the perceptual semantic representation obtained after encoding the static candidate summary by the second encoder is as close as possible to the offline semantic representation obtained by the real summary by the offline second encoder in the semantic feature space. This trains the global autoregressive model to learn the correct content in the perceptual candidate summary. At the same time, because the real summary is encoded by the offline encoder, overfitting of the first encoder and the second encoder is avoided.

[0119] In one embodiment, the step of calculating the loss function further includes:

[0120] S60. Train a global autoregressive model using reinforcement learning, and then use the semantic representation of the original text obtained in step S20. The perceptual semantic representation H is simultaneously input into the decoder, which generates a baseline summary S and a sampled summary S according to a greedy strategy and a sampled sampling strategy, respectively. The model is rewarded based on the scores from both parties, and the loss function for reinforcement learning is calculated. Specifically:

[0121] A global autoregressive model is trained using reinforcement learning, where a sampled summary S is used as a random action for the model, and this action is rewarded, allowing the model to explore the action that maximizes the reward. Baseline summary. The action with the highest probability among all current actions of the global autoregressive model is penalized when the reward for a given action is lower than the reward for this highest-probability action. A baseline summary is then generated. The specific process for obtaining the sampling summary S is as follows:

[0122] S601. The semantic representation of the original text obtained in step S20 is... The input to the decoder is the cross-attention layer connected to the first encoder in each Transformer network layer, while the perceptual semantic representation H is input to the cross-attention layer connected to the second encoder in each Transformer network layer. The decoder uses a sample sampling strategy to generate summaries; that is, during the autoregressive decoding process, in each iteration, a token is randomly selected from the vocabulary probability distribution output by the decoder as the output, and this output token is used as the input for the next iteration. This process is repeated until the decoder outputs a termination token or reaches the maximum output length, such as... Figure 6 As shown, the output tokens of each iteration are mapped to words through a vocabulary and concatenated into text to form a sampled summary S.

[0123] S602. Based on the character matching degree between the sampled summary S obtained in step S601 and the corresponding real summary in the dataset, the quality of the sampled summary is scored. This score serves as a reward for the model's behavior of generating this summary, and the calculation method is as follows:

[0124]

[0125]

[0126]

[0127] in Y represents the static candidate summary in the dataset, Y represents the true summary in the dataset, and {·} represents the set operation of word sequence extraction.

[0128] S603. To prevent the global autoregressive model from consistently receiving positive rewards and falling into the zero-reward zone during reinforcement learning training, a penalty needs to be imposed on the model when the generated sampled summary S is insufficient. Repeat the decoding process of step S601, but the decoder employs a greedy strategy: in each iteration, it selects the token with the highest probability from the vocabulary probability distribution to obtain the baseline summary. The reward obtained by the model for generating the sampled summary S is the baseline summary. The difference between the scores of the sample summary and the baseline summary is calculated when the score of the sample summary S is less than that of the baseline summary. When scoring, a negative reward value penalizes the global autoregressive model, while a positive reward value rewards the global autoregressive model.

[0129] The specific method for calculating reinforcement learning loss is as follows:

[0130] S604. Based on the reward obtained by the model in step S603 above for generating the sample summary S, update the parameters of the model, including all parameters of the first encoder, the second encoder, and the decoder, so that the parameters of the model are updated in the direction of maximizing the probability of generating the sample summary. When the reward value is negative, the parameters of the model are updated in the direction of minimizing the probability of generating the sample summary. The magnitude of the parameter update is related to the absolute value of the reward obtained by the model. The larger the absolute value of the reward, the larger the magnitude of the parameter update.

[0131] S605, The calculation method for reinforcement learning loss is as follows:

[0132]

[0133] In one embodiment, calculating the loss function further includes:

[0134] S70. Train a global autoregressive model in a supervised manner using static candidate summaries and real summaries from the dataset, and calculate the cross-entropy loss. Specific instructions are as follows:

[0135] S701. Supervised training of the global autoregressive model using static candidate summaries helps prevent the model from deviating from the parameters used when constructing the static candidate summaries during training. The static candidate word segments from the dataset are processed through an output embedding layer to obtain word embedding sequences, which are then input into the model's decoder. Simultaneously, the semantic representation of the original text obtained in step S20 above is also input. The input is the cross-attention layer in each Transformer network layer of the decoder connected to the first encoder. At this point, the cross-attention layer in each Transformer network layer of the decoder connected to the second encoder is inactive. The cross-entropy loss is calculated based on the vocabulary probability distribution of the encoder output, as follows:

[0136]

[0137] Where q(v) represents the label smoothing function, as described in step S101 above.

[0138] The global autoregressive model is trained in a supervised manner using static candidate summaries, so that the candidate summaries generated by the model for each original text in the dataset are similar to the static candidate summaries. The process of generating candidate summaries is as described in step S102 above.

[0139] S702. Supervised training of the global autoregressive model using real summaries from the dataset to improve stability during reinforcement learning training. The real summaries from the dataset are segmented and passed through the output embedding layer before being input to the decoder. Simultaneously, the semantic representation of the original text obtained in step S20 above is used... The perceptual semantic representation H is input into the cross-attention layer connecting the first encoder and the cross-attention layer connecting the second encoder in each layer of the Transformer network of the decoder, respectively. The overall process is as follows: Figure 2 As shown. The cross-entropy loss function is calculated based on the vocabulary probability distribution output by the decoder, as follows:

[0140]

[0141] The global autoregressive model is trained in a supervised manner using real summaries from the dataset. The model recognizes the real summary content corresponding to each original text in the dataset, which gives the model a definite direction for random behavior during reinforcement learning. This avoids the model from degenerating because it keeps receiving negative rewards during random behavior.

[0142] In one embodiment, the loss functions in steps S50, S60, and S70 are summed to calculate the final loss of the global autoregressive model. This loss is then minimized using gradient descent, and the autoregressive model parameters are updated. Specifically:

[0143] S801. Sum the various loss functions of the global autoregressive model during training to obtain the total loss of the model. The total loss is calculated as follows:

[0144]

[0145] Where γ and δ are hyperparameters. The loss function obtained in step S701 above, The loss function obtained in step S702 above, The loss function obtained in step S402 above, The loss function is obtained in step S604 above.

[0146] S802. Optimize the total loss of the global autoregressive model using gradient descent, and reduce the total loss... Differentiate and backpropagate to obtain the gradients of all model parameters in the direction of minimizing the overall loss, and update the model parameters based on the gradients.

[0147] Experimental verification:

[0148] The global autoregressive model is trained using the CNN / DM dataset. This invention selects three state-of-the-art methods on this dataset for comparison to demonstrate the superior performance of the invention in the goose formation model.

[0149] The CNN / DM dataset is a news dataset collected from Cable News Network (CNN) and the Daily Mail (DM), containing a total of 311,672 news articles. The training set size is 286,817, the validation set size is 13,368, and the test set size is 11,487. The average length of each news article is 766 words, and the average length of the summary is 53 words.

[0150] This invention selects three methods—BART, GSum, and SeqCo—for comparison.

[0151] The BART method, proposed in 2020, is a typical Transformer encoder-decoder structure, using cross-entropy loss to train the model. GSum, proposed in 2021, consists of two Transformer encoders and one Transformer decoder. Unlike the global autoregressive method proposed in this invention, its two encoders are typical Transformer encoders, encoded in parallel. GSum uses extractive summaries as input to the second encoder and trains the model using cross-entropy loss. The SeqCo method, proposed in 2022, is similar to BART, a typical Transformer encoder-decoder structure, using contrastive learning to train the model. However, unlike the contrastive learning method in the global autoregressive model proposed in this invention, SeqCo compares the decoder's output, while the global autoregressive model compares the output of the second encoder. None of these three methods can solve the error accumulation problem that occurs during autoregressive generation. The global autoregressive model proposed in this invention, however, can correct errors that occur during autoregression in the next global iteration through global autoregression.

[0152] Table 1 shows the results of the global autoregressive method and the three methods mentioned above, using three metrics—Rogue-1, Rouge-2, and Rouge-L—on the test set of the CNN / DM dataset.

[0153] Table 1 Comparison of CNN / DM test set performance.

[0154]

[0155] Those skilled in the art will understand that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A text summarization method based on global autoregression, characterized in that, The method includes: S1. Input the text into the first encoder to obtain the semantic representation of the text. S2, Semantic representation of text Input to decoder to generate candidate summaries; S3. Combine candidate summaries and text semantic representations The input is given to the second encoder to obtain the perceptual semantic representation H of the candidate summary; S4. Representing the semantics of text The perceptual semantic representation H of the candidate summary is input into the decoder to generate a new candidate summary; S5. Repeat steps S3 to S4 until the preset number of repetitions is reached. The final candidate summary output by the decoder is the text summary. The first encoder, the second encoder, and the decoder together form a global autoregressive model. The training methods for the global autoregressive model include: S10. Train the global autoregressive model using the original text and real summaries in the dataset, optimize the cross-entropy loss, and generate static candidate summaries for all the original texts in the dataset. S20. Input the original text from the dataset training into the first encoder to obtain the semantic representation of the original text. The static candidate summary is input into the second encoder along with the static candidate summary to obtain the perceptual semantic representation H of the static candidate summary; S30. An offline encoder with the same structure as the two encoders used in the construction of the global autoregressive model is used. The original text and the real summary from the dataset are sequentially input into the offline first encoder and the offline second encoder to obtain the offline perceptual semantic representation H of the real summary. * ; S40. Calculate the perceptual semantic representation H of the static candidate summary and the offline perceptual semantic representation H of the real summary. * semantic similarity; S50. Calculate the loss function of the global autoregressive model and adjust the parameters of the global autoregressive model through the backpropagation algorithm; Step S40 includes: S401, Using a multi-head attention network, the offline perceptual semantic representation H is... * The semantic features are integrated into the perceptual semantic representation H to obtain a representation that simultaneously possesses H * Semantic representation of H semantic features The calculation method is as follows: S402, Calculate the true summary Y and the static candidate summary Y * Perceptual semantic similarity between Used to represent the true summary Y and the static candidate summary Yi * The semantic feature differences between them are analyzed, and the semantic similarity is calculated as follows: in and The semantic features are calculated in step S401 above.

2. The method according to claim 1, characterized in that, In each layer of the Transformer network of the second encoder, a cross-attention layer is added to receive the output of the first encoder and the output of the normalization layer above it. In each Transformer network layer, the decoder receives the output of the first encoder and the output of the normalization layer above it in the first cross-attention layer, and the output of the second encoder and the output of the normalization layer above it in the second cross-attention layer. The outputs of the two attention layers are vector-summed and then input into the feedforward network layer.

3. The method according to claim 2, characterized in that, Both the encoder and decoder have N Transformer layers, and the two encoders share the parameters of the self-attention layer and feedforward network layer of the first N-1 Transformer layers.

4. The method according to claim 1, characterized in that, In step S10, the step of generating a static candidate summary includes: S101. After segmenting the original text in the dataset into words, convert it into word embedding sequences through the input embedding layer, and then input it into the first encoder to obtain the semantic representation of the original text. After segmenting the real summaries in the dataset, the output embedding layer converts them into word embedding sequences, which are then input into the decoder. Simultaneously, the semantic representation of the original text is... The input decoder uses the cross-attention layer connected to the first encoder in each Transformer network layer. At this time, the cross-attention layer connected to the second encoder in the decoder does not play a role. Calculate the cross-entropy loss based on the vocabulary probability distribution output by the decoder, and update the parameters of the first encoder and decoder. S102. Input the original text of the dataset into the first encoder of the global autoregressive model trained in step S101 to obtain the semantic representation of the original text. And The input decoder is connected to the cross-attention layer of the Transformer network in each layer of the input decoder, and the decoder iteratively generates static candidate summaries through autoregression.

5. The method according to claim 1, characterized in that, In step S30, the parameters of the offline encoder are estimated using the moving average of the encoder parameters in the global autoregressive model, and the parameter update method is as follows: ξ←μξ+(1-μ)θ Where ξ and θ are the parameters of the offline encoder and the encoder parameters of the global autoregressive model, respectively, and μ is the weight.

6. The method according to claim 1, characterized in that, In S50, the similarity loss function is calculated using the following formula: Minimize similarity loss This is used to train the global autoregressive model to learn to perceive the correct content in the candidate summaries, and to adjust the parameters of the global autoregressive model.

7. The method according to claim 6, characterized in that, Step S50 further includes training a global autoregressive model through reinforcement learning, including: S601, Representing the semantics of text The input to the decoder is the cross-attention layer connected to the first encoder in each Transformer network layer, while the perceptual semantic representation H is input to the cross-attention layer connected to the second encoder in each Transformer network layer of the decoder. The decoder uses a sample sampling strategy to generate a sampled summary S and a greedy strategy to generate a baseline summary. S602. Based on the character matching degree between the sampled summary S and the corresponding real summary in the dataset, the quality of the sampled summary is scored. This score serves as a reward for the model's behavior in generating this summary. The score calculation method is as follows: in Y represents the static candidate summary in the dataset, Y represents the true summary in the dataset, and {·} represents the set operation of word sequence; S603. To prevent the global autoregressive model from always receiving positive rewards and falling into the zero-reward range during the reinforcement learning training process, the model needs to be penalized when the generated sampling summary S is not good enough. S604, The reward obtained by the model in generating the sampled summary S is the baseline summary. The difference between the scores of the sample summary and the S. S605. Calculate the reinforcement learning loss using the following formula: Where S represents the sampled summary. Indicates baseline summary, Let X represent the static candidate abstract, l represent the original text to be abstracted, and r(·) represent the sequence length of the sampled abstract S. The scoring function described in step S602 above is r(·).

8. The method according to claim 6, characterized in that, Step S50 further includes: training a global autoregressive model using static candidate summaries and real summaries, the steps of which include: S701. The static candidate words in the dataset are processed through the output embedding layer to obtain word embedding sequences, which are then input into the decoder of the autoregressive model. At the same time, the text semantic representation is processed. The input is the cross-attention layer in each Transformer network layer of the decoder connected to the first encoder. At this point, the cross-attention layer in each Transformer network layer of the decoder connected to the second encoder is inactive. The cross-entropy loss is calculated based on the vocabulary probability distribution of the encoder's output, as follows: Where q(v) represents the label smoothing function; S702. After segmenting the real summaries in the dataset, the output embedding layer is input to the decoder, and the text semantic representation is simultaneously processed. The perceptual semantic representation H is input into the cross-attention layer connecting the first encoder and the cross-attention layer connecting the second encoder in each layer of the Transformer network of the decoder, respectively. The cross-entropy loss function is calculated based on the vocabulary probability distribution output by the decoder, as follows:

9. The method according to claim 8, characterized in that, The sum of the loss functions in steps S50, S605, S701, and S702 is the final loss function of the global autoregressive model. This loss is minimized by gradient descent, and the autoregressive model parameters are updated.

Images