A long text summary rapid generation method, model training method and device

Abstract

The present application provides a method for rapidly generating a long text summary, a model training method, and an apparatus. The method for rapidly generating a long text summary provided by the present application includes: obtaining a long text to be summarized and dividing the long text into blocks to obtain multiple text blocks; a summary encoder reading the i-th text block and calculating the forward hidden state and backward hidden state of the i-th text block; predicting a skip position based on the forward hidden state and backward hidden state of the i-th text block; determining the number of the next text block to be read based on the skip position, using the number as the latest i value, and returning to the step where the summary encoder reads the i-th text block; combining the forward hidden state and backward hidden state of all read text blocks to obtain a first hidden representation output by the encoder; adding an embedded representation of the position information of each read text block to reconstruct the first hidden representation to obtain a second hidden embedded representation; and a summary decoder predicting and outputting a summary of the long text based on the second hidden embedded representation.

Description

Technical Field

[0001] This application relates to the technical field of natural language understanding, and in particular, to a method for quickly generating long text summaries, a model training method, and a device. Background Art

[0002] Automatic summarization is an important research direction in the field of natural language processing. Its purpose is to compress, summarize, and refine the original text to provide users with a concise text description that can cover the core content of the original text. With the rapid development of the Internet, the text information resources have shown exponential growth, making it impossible for people to conveniently and quickly obtain important content from a vast amount of information. The text summarization technology can "compress and refine" a large amount of long text information into short text information with refined content, helping people quickly obtain important content, and is widely used in scenarios such as news summary generation, e-commerce platform evaluation summary, and text information retrieval, and has always received extensive research and attention.

[0003] Currently, the commonly used method for generating text summaries is the generative summary method based on neural networks. This method has achieved good results on some short text datasets, but it cannot achieve good results for longer texts, it is difficult to find key information in a large amount of input information, and it requires a large amount of computing power, and the speed of generating summaries is slow. Summary of the Invention

[0004] In view of this, this application provides a method for quickly generating long text summaries, a model training method, and a device to solve the problems of inaccurate long text summary generation, large consumption of computing resources, and slow speed.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] In the first aspect of this application, a method for quickly generating long text summaries is provided. The method is implemented based on a long text summary quick generation model. The long text summary quick generation model at least includes a summary encoder and a summary decoder using skip encoding. The skip encoding encodes discontinuous texts in the long text;

[0007] The method for quickly generating long text summaries specifically includes: obtaining a long text to be summarized, and dividing the long text into blocks to obtain multiple text blocks;

[0008] The summary encoder reads the i-th text block and calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text; <00OO020>Predict the skip reading position based on the forward hidden state and backward hidden state of the i-th text block;

[0010] Determine the next text block number to be read based on the skip position, use it as the latest i value, and return to the step of reading the i-th text block by the summary encoder;

[0011] Combine the forward hidden states and backward hidden states of all read text blocks to obtain the first hidden representation output by the encoder;

[0012] Adding the position information embedding representation of each read text block to reconstruct the first hidden representation to obtain a second hidden embedding representation;

[0013] The summary decoder predicts and outputs a summary of the long text based on the second hidden embedding representation.

[0014] A second aspect of the present application provides a method for training a long text summary fast generation model, for training the long text summary fast generation model, the training method comprising:

[0015] Determining training parameters of the long text summary fast generation model; the training parameters include at least long short-term memory network parameters of the summary encoder and the summary decoder, and forward skipping agent parameters;

[0016] selecting an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, the optimal training method at least comprising optimizing the parameters using a policy gradient algorithm and selecting conditional log probability as a reward function;

[0017] The long text summary fast generation model is comprehensively trained according to the optimal training method corresponding to each parameter, and each parameter of the long text summary fast generation model is determined to complete the training of the long text summary fast generation model.

[0018] A third aspect of the present application provides a device for quickly generating a long text summary, the device being applied to a long text summary quickly generating model, the long text summary quickly generating model comprising at least a summary encoder module and a summary decoder module using skip coding, wherein the skip coding encodes discontinuous text in the long text;

[0019] The summary encoder module is configured to read a long text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder, and adding an embedded representation of the position information of each read text block to reconstruct the first hidden representation to obtain a second hidden embedded representation;

[0020] The abstract decoder module is used to parse the second hidden embedding representation and predict the abstract of the long text.

[0021] In a fourth aspect of the present application, a training device for a long text abstract fast generation model is provided. The device includes a determination module, a processing module, and a training module; wherein,

[0022] The determination module is used to determine the training parameters of the long text abstract fast generation model; the training parameters at least include the long short-term memory network parameters of the abstract encoder and the abstract decoder, and the forward skip-reading agent parameters;

[0023] The processing module is used to select an optimal training method for the training parameters. Among them, the optimal training method for the forward skip-reading agent parameters is different from that of the long short-term memory network parameters. The optimal training method at least includes optimizing the parameters using the policy gradient algorithm and selecting the conditional log probability as the reward function;

[0024] The training module is used to comprehensively train the long text abstract fast generation model according to the optimal training method corresponding to each parameter, determine the various parameters of the long text abstract fast generation model, and complete the training of the long text abstract fast generation model.

[0025] The long text abstract fast generation method, model training method and device provided by the present application first obtain the long text to be abstracted, and divide the long text into multiple text blocks, and then read the i-th text block through the abstract encoder, calculate the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text, so as to predict the skip-reading position based on the forward hidden state and backward hidden state of the i-th text block, and then determine the number of the next text block to be read based on the skip-reading position as the latest i value, and return to the step of the abstract encoder reading the i-th text block. Further, combine the forward hidden states and backward hidden states of all the read text blocks to obtain the first hidden representation output by the encoder, and add the position information embedding representation of each read text block to reconstruct the first hidden representation to obtain the second hidden embedding representation, and finally predict and output the abstract of the long text through the abstract decoder based on the second hidden embedding representation. In this way, the long text is cut by dividing it into multiple text blocks, and each text block is semantically complete, ensuring the integrity of the subsequent abstract generation; predicting the skip-reading position based on the forward hidden state and backward hidden state of each text block can selectively generate the long text abstract from some text blocks, that is, only need to analyze and process some texts, reducing the amount of text that the model needs to process, thereby reducing the consumption of computing resources while ensuring the accuracy of the abstract and improving the abstract generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of Example 1 of the method for quickly generating a long text summary provided by this application;

[0027] Figure 2 Flowchart of Example 1 of the long text summary rapid generation model training method provided in this application;

[0028] Figure 3 This is a schematic diagram of the structure of the first embodiment of the long text summary rapid generation device provided by this application;

[0029] Figure 4 This is a structural diagram of Example 1 of the long text summary rapid generation model training device provided in this application. DETAILED DESCRIPTION

[0030] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0033] This application provides a method for quickly generating long text summaries, a model training method, and a device to solve the problems of inaccurate long text summary generation, high computing resource consumption, and slow speed.

[0034] The long text abstract fast generation method, model training method and device provided by this application first obtain the long text to be abstracted, and divide the long text into blocks to obtain multiple text blocks. Then, the abstract encoder reads the i-th text block and calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks in the long text. Thus, based on the forward hidden state and backward hidden state of the i-th text block, the skip reading position is predicted, and then based on the skip reading position, the number of the next text block to be read is determined as the latest i value, and the step of returning the abstract encoder to read the i-th text block is performed. Further, the forward hidden states and backward hidden states of all the read text blocks are combined to obtain the first hidden representation output by the encoder, and the position information embedding representation of each read text block is added to reconstruct the first hidden representation to obtain the second hidden embedding representation. Finally, the abstract decoder predicts and outputs the abstract of the long text based on the second hidden embedding representation. In this way, a text skip reading technology is designed in the encoder-decoder architecture model, that is, an enhanced lightweight forward skip reading agent is designed. During the process of inputting the long text, it can select the content to be read, focus on the important information of the long text, and discard the unimportant information, resulting in significant improvements in accuracy and speed. In addition, the long text is cut by dividing it into multiple text blocks, and each text block is semantically complete, ensuring the integrity of the subsequent abstract generation. Predicting the skip reading position based on the forward hidden state and backward hidden state of each text block can selectively generate the long text abstract from some text blocks, that is, only need to analyze and process part of the text, reducing the amount of text that the model needs to process, thereby reducing the consumption of computing resources while ensuring the accuracy of the abstract and improving the abstract generation speed.

[0035] Specific embodiments are given below to introduce the technical solution of this application in detail.

[0036] Figure 1 It is a flowchart of the first embodiment of the long text abstract fast generation method provided by this application. Please refer to Figure 1 The method provided in this embodiment is implemented based on the long text abstract fast generation model. The long text abstract fast generation model at least includes an abstract encoder and an abstract decoder using skip encoding. The skip encoding encodes the discontinuous text in the long text. The long text abstract fast generation method may include:

[0037] S101. Obtain the long text to be abstracted, and divide the long text into blocks to obtain multiple text blocks.

[0038] Specifically, the process of dividing the long text into blocks to obtain multiple text blocks may include:

[0039] (1) Obtain the long text to be summarized, determine the complete semantic text blocks according to the semantic content of the long text, and determine the size of each text block.

[0040] It should be noted that text blocks refer to smaller parts that a long text is divided into according to certain rules or standards during the text processing process. In one possible implementation, after obtaining the long text to be summarized, the long text to be summarized is first segmented into multiple paragraphs or sentences. For example, punctuation marks, line breaks, or other separators can be used to segment the long text. Semantic analysis is then performed on the segmented text blocks to determine the semantic content of each text block. For example, in one embodiment, this can be achieved through natural language processing (NLP) technology, such as part-of-speech tagging, named entity recognition, semantic role labeling, etc.

[0041] Furthermore, the size of each text block is determined based on the semantic content. For example, this can be determined based on indicators such as the number of words, characters, and sentences. It should be noted that, generally, a text block should contain a complete semantic unit so that the summarized information can maintain integrity and coherence.

[0042] (2) Determine the maximum proportion of long text blocks based on the size of each text block.

[0043] It should be noted that determining the maximum ratio of long text chunks refers to determining a threshold value for controlling the maximum ratio of chunks into which a long text is segmented. Specifically, this ratio can be determined based on the size of each chunk to ensure that the generated chunks maintain semantic integrity while not being too short or too long. For example, in one embodiment, the maximum ratio can be determined based on the average size of the chunks.

[0044] (3) Segmenting the long text into a plurality of text blocks according to the maximum ratio and the size of each text block.

[0045] It should be noted that multiplying the size of each text block by the maximum ratio gives the maximum length of the text block, which represents the maximum length that each text block can reach during segmentation, so as to segment the long text according to the maximum length of the text block. For example, in one embodiment, starting from the beginning of the long text, the text can be segmented into multiple text blocks according to the maximum length until the long text is completely segmented, resulting in multiple text blocks. As another optional embodiment, the text blocks after initial segmentation can be obtained according to the semantics of the long text, and the text blocks after initial segmentation can be adjusted according to the size of the text blocks after initial segmentation and the maximum ratio, so as to obtain the finally segmented text blocks. For example, if the size of the text block after initial segmentation is less than or equal to the maximum ratio, the text block after initial segmentation remains unchanged; if the size of the text block after initial segmentation is greater than the maximum ratio, the text block is secondarily semantically segmented in units of the maximum ratio. Then, for this text block, the text block after secondary semantic segmentation is the finally segmented text block. For example, if the size of text block A1 is less than the maximum ratio, then A1 is retained; if the size of text block B1 is greater than the maximum ratio, then B1 is further segmented into BB1 and BB2. The text blocks finally obtained from the long text are specifically A1, BB1, and BB2.

[0046] (4) Traverse the boundary semantics of each text block after segmentation, adjust the boundaries with coherent semantics according to the maximum ratio, and obtain the adjusted segmentation result. <​​​​​​​​​​​​​It should be noted that the abstract encoder is an abstract encoder using skip encoding. That is, the text block read at this time is the text block after skipping. That is to say, the skipped text block is not encoded, and only the text block that needs to be read is encoded. This application uses a bidirectional long short-term memory network as the abstract encoder. Among them, skip encoding refers to encoding discontinuous texts in the long text; the bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, BiLSTM) is a deep learning model, which is usually used for modeling and processing of sequence data. BiLSTM combines two long short-term memory networks LSTM in two directions, and processes them from the forward and backward directions of the sequence respectively to capture long-term dependencies in the sequence data.

[0052] Specifically, when obtaining multiple text blocks including a series of tokens from x1 to N, which represent a long document composed of text blocks including a series of tokens from x1 to xN, for the i-th text block, the i-th token can be read through the forward LSTM, and the forward hidden state can be calculated. ; read the i-th token through the backward LSTM, and calculate the backward hidden state. Among them is the word embedding of the input token xi. Among them, i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text.

[0053] S103. Predict the skip position based on the forward hidden state and the backward hidden state of the i-th text block.

[0054] Specifically, predicting the skip position based on the forward hidden state and the backward hidden state of the i-th text block includes:

[0055] (1) Aggregate the forward hidden states and backward hidden states of all blocks read before to predict the skip word.

[0056] It should be noted that for each text block, after obtaining its forward and backward hidden states through the forward and backward LSTMs, these hidden states can be connected to achieve aggregation.

[0057] (2) Determine the target text block where the skip word is located based on the position of the skip word.

[0058] For predicted skipped words, the target text block in which they are located is determined based on their position. Specifically, this can be achieved by recording the starting and ending positions of each text block. For example, if the skipped word is located between the starting and ending positions of the i-th text block, then the skipped word is located in the i-th target text block. Specifically, the similarity between each word and the skipped word can be matched sequentially in the text block following the currently read text block. If the similarity is greater than a preset threshold, the word is determined to be a skipped word.

[0059] (3) The end of the target text block is used as the predicted skip position.

[0060] It should be noted that, in order to simplify the skip coding process, the scope of the skip coding is limited, and the skip coding can only be at the end of the text block, so the end of the target text block is used as the predicted skip position.

[0061] Compared with the text cutting method of the prior art that directly splits long texts, the method provided by the present invention can predict the next skipped word based on the long text itself, that is, based on the characteristics of each long text itself, the previous text predicts the skipping position of the following text, thereby improving the accuracy of the text after skipping, and thus improving the accuracy of summary generation; further, the skipping of the long text of the present invention is predicted based on a bidirectional encoder, while utilizing contextual information, thereby improving the accuracy of the prediction; in addition, based on skipping, part of the text block is directly skipped, so that only part of the text needs to be encoded, parsed and summarized, reducing the amount of data for text processing, and directly reducing the amount of text in the encoding stage, that is, the skipped text paragraphs do not need to be encoded, thereby greatly reducing the processing time of text generation and improving the efficiency of summary generation.

[0062] S104 , determining the number of the next text block to be read based on the skip position, taking it as the latest i value, and returning to the step of the summary encoder reading the i-th text block.

[0063] Specifically, the target text block can be determined based on the skip position. This can be achieved, for example, by comparing the skip position with the start and end positions of each text block. If the skip position is between the start and end positions of the i-th text block, the next text block to be read is numbered i.

[0064] Furthermore, the number of the next text block to be read is determined according to the text block where the skip position is located, and is used as the latest i value. The latest i value is passed to the summary encoder so that it reads the i-th text block and performs subsequent processing.

[0065] S105 , combining the forward hidden states and backward hidden states of all read text blocks to obtain a first hidden representation output by the encoder.

[0066] Combining the forward hidden state and the backward hidden state of the text block read above, for example, the first hidden representation of the encoder output can be obtained by concatenating the vectors .

[0067] It should also be noted that the last hidden state can be used To predict the skipped steps, a lightweight forward network can be used to browse K text blocks so that skip coding does not involve too much computation. Specifically, the first hidden representation output based on the forward hidden state and backward hidden state of the i-th text block is used to predict the skip position distribution, including:

[0068] (1) Perform a pooling operation on the word embedding of the i-th text block to obtain a pool embedding.

[0069] It should be noted that the pooling operation can be set according to actual needs and is not limited in this embodiment. Among them, common pooling operations include maximum pooling, average pooling, etc. In maximum pooling, the maximum value in the word embedding sequence is taken as the pooled embedding; in average pooling, the average value of the word embedding sequence is taken as the pooled embedding. These operations help to convert the word-level representation into a higher-level text block-level representation. For example, in this embodiment, the word embedding of the i-th text block is pooled to obtain the pooled embedding .

[0070] (2) Feed the first hidden representation and pool embedding of the i-th text block into the fully connected layer to obtain the current representation and future representation of the i-th text block respectively.

[0071] Specifically, the first hidden representation of the i-th text block is and pool embedding Feeding into the fully connected layer can obtain the current representation of the i-th text block and the future .

[0072] (3) The skipping position of the i-th text block is obtained by combining the current representation and the future representation of the i-th text block.

[0073] Connect Current Representation and the future The skip position of the i-th text block can be obtained .

[0074] It should be noted that, combined with the above description, the word embeddings of all text blocks are collected to obtain their representations, and t is the sequence number of the skip operation, which can be obtained:

[0075]

[0076] in, Pool embedding for text blocks, For the current representation, For the future, is the skip distribution, represents word embedding, is the first hidden representation.

[0077] Specifically, the model The sample skip value j = [-1, K], where -1 means early stop, 0 means no skipping, and the others indicate the number of blocks to skip. After the skip, the model continues from The next segment is read. This process continues until one of the following occurs:

[0078] (i)-1 is sampled from the skip distribution.

[0079] (ii) The proportion of reading exceeds P.

[0080] (iii) Reaching the last mark.

[0081] S106: Add the position information embedded representation of each read text block, reconstruct the first hidden representation, and obtain a second hidden embedded representation.

[0082] It should be noted that due to the nature of the LSTM encoder, the model is well informed about the location of the informative tokens. However, the encoder with filtering operations discards the input sequence. Therefore, the model only focuses on the relative position relationship of the selected input fragments, and when encoding the input text, the absolute position information of the tokens in the original input is lost. Therefore, in the summary encoder, the input sequence is encoded into the first hidden state Afterwards, you can add absolute positioning The embedding of is returned to the first hidden state to reconstruct the absolute position information and obtain the second hidden embedding representation.

[0083] Specifically, adding the position embedding representations of each read text block and reconstructing the first hidden representation to obtain the second hidden embedding representation includes:

[0084] (1) Obtain a first hidden representation of the text block.

[0085] (2) Obtain the absolute position of the text block in the long text.

[0086] Determine the absolute position of the text block in the long text, that is, the index position of the text block in the long text. This position is usually the starting position or the middle position of the text block in the entire long text sequence.

[0087] (3) Compute the embedding of the absolute position using sinusoidal position encoding.

[0088] Sinusoidal positional encoding is a commonly used method to generate a fixed-dimensional position embedding vector for each position in a sequence to represent the position information.

[0089] (4) Combining the first hidden representation of the text block with the added absolute position embedding to obtain a second hidden embedding representation of the text block.

[0090] Specifically, the first hidden representation of the text block is combined with the added absolute position embedding to obtain the second hidden embedding representation of the text block as .

[0091] S107: The summary decoder predicts and outputs a summary of the long text based on the second hidden embedding representation.

[0092] It should be noted that the summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation, including:

[0093] (1) The summary decoder receives the second hidden embedding representation and the decoder state of the last cycle.

[0094] The purpose of the abstract decoder is to be based on the first hidden representation To generate the summary token. The summary decoder here is a single-layer unidirectional LSTM, which receives the second hidden embedding representation and the last decoder state , and then output the decoder state of the last cycle .

[0095] ,

[0096] in, is the output of the summary decoder at time t-1, that is, a word that generates a summary.

[0097] (2) Compute an attention distribution over the second hidden embedding representation and the decoder state.

[0098] The attention mechanism is used to calculate the context vector , which can be viewed as a fixed-size dense representation of the source text at the current decoding step. At each decoding step, the decoding state and a second hidden embedding representation to compute the attention distribution over the input tokens.

[0099] (3) Perform a weighted summary on the second hidden embedding representation based on the attention distribution to obtain a context vector.

[0100] Specifically, the formula is as follows:

[0101] ,

[0102] in To learn the parameters, is the hidden state representation before the skip operation, is the attention probability distribution, is the context vector, represents the word embedding vector, Decoder status.

[0103] (4) generating a decoder state for the current cycle based on the decoder state and the context vector, updating the decoder state, and returning to the step of receiving the second hidden embedding representation until the update calculation of the preset cycle is completed.

[0104] Update the decoder state based on the decoder state of the current cycle and the generated context vector This updating process can be implemented using recurrent neural networks (such as long short-term memory networks) so that the model can maintain a memory of the context and adjust its state based on the current context information.

[0105] (5) Summarize the decoder states of each cycle to obtain the vocabulary distribution pattern and form a summary of the long text.

[0106] Set the decoder status and context vector Add up and calculate the vocabulary distribution pattern:

[0107] .

[0108] The long text summary fast generation method provided in this embodiment first obtains the long text to be summarized, divides the long text into chunks to obtain multiple text chunks, and then reads the i-th text chunk through the summary encoder, calculates the forward hidden state and backward hidden state of the i-th text chunk, where i is a positive integer, 0 < i ≤ N, and N is the total number of text chunks of the long text. Then, based on the forward hidden state and backward hidden state of the i-th text chunk, the skip reading position is predicted, and further, based on the skip reading position, the number of the next text chunk to be read is determined as the latest i value, and the step of returning the summary encoder to read the i-th text chunk is performed. Further, the forward hidden states and backward hidden states of all the read text chunks are combined to obtain the first hidden representation output by the encoder, and the position information embedding representation of each read text chunk is added to reconstruct the first hidden representation to obtain the second hidden embedding representation. Finally, the summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation. In this way, by using a bidirectional long short-term memory network (bi-LSTM) to encode the input text, an enhanced lightweight forward skip reading agent is designed separately in the encoder module. When the bidirectional long short-term memory network loops to encode the input, the skip reading agent can help the encoder selectively skip secondary input text segments based on current and future information. During the decoding process, these secondary segments are excluded from the attention mechanism. This filtering operation reduces the computational cost and attracts attention to key information to improve the accuracy of the summary, while also improving the speed of summary generation.

[0109] Figure 2 This is the flowchart of the first embodiment of the training method for the long text summary fast generation model provided in this application. Please refer to Figure 2 , and this training method for the long text summary fast generation model is used to train the long text summary fast generation model. The training method includes:

[0110] S201. Determine the training parameters of the long text summary fast generation model; the training parameters at least include the long short-term memory network parameters of the summary encoder and the summary decoder, and the forward skip reading agent parameters.

[0111] It should be noted that there are three parameters to be estimated in the model: . and . Among them, and respectively represent the long short-term memory network LSTM parameters of the summary encoder and the summary decoder; represents the parameter of the forward skip reading agent.

[0112] S202. Selecting an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional log probability as a reward function.

[0113] It should be noted that cannot be estimated directly, but and It can be calculated via backpropagation. Policy gradient algorithms are a class of methods used to optimize parameterized policies, particularly well-suited for reinforcement learning tasks. When training a model, policy gradient algorithms update the model's parameters by maximizing the expected cumulative reward. The core idea of ​​these algorithms is to directly optimize the policy parameters, rather than optimizing a value function or state-value function. Using a policy gradient algorithm to optimize model parameters means that the model parameters are updated via gradient ascent to maximize the expected cumulative reward.

[0114] It should also be noted that the conditional log probability is chosen as the reward function instead of ROUGE because the loss of the model with the ROUGE reward function converges slowly during training and the learning objective of skipping with the ROUGE reward function is too far away from the learning objective of the summary decoder.

[0115] Specific, definition As given The activation function is designed as follows:

[0116] ,

[0117] In practice, the skipping action is reformulated as a reinforcement learning problem, and the high variance problem can be alleviated by using a baseline b, which is the average reward of previous iterations used during training.

[0118] It should be noted that the forward skipping agent usually simply reads the first half of the article. In order to force the forward skipping agent to explore further, we also use the entropy of the probability distribution of skipping actions as the maximum entropy regularization of the forward skipping agent, specifically including:

[0119] (1) Determine the loss function of the training process.

[0120] (2) Training the long text summary fast generation model based on the loss function.

[0121] It should also be noted that during the training process, the goal of the model is to minimize the total loss function, wherein the loss function at least includes Lpg and the entropy of the probability distribution of skipping.

[0122] For the entropy of the probability distribution of skipping actions, negative entropy can be used as a loss term. A larger entropy indicates greater uncertainty in the model and a larger regularization term, encouraging the model to be more exploratory in choosing skipping actions. Specifically, in each training iteration, the loss function is calculated as follows:

[0123]

[0124] in, is the t-th action selected from the sample, is the entropy of the jump distribution at time t, and a is the entropy regularization factor. After combining these two losses, our overall training loss function is as follows:

[0125]

[0126] S203 , comprehensively training the long text summary fast generation model according to the optimal training method corresponding to each parameter, and determining various parameters of the long text summary fast generation model to complete the training of the long text summary fast generation model.

[0127] The long text summary rapid generation model training method provided in this embodiment can improve the accuracy and summary generation speed of the long text summary rapid generation model and reduce the computing cost.

[0128] Corresponding to the aforementioned embodiment of a method for quickly generating a long text summary, the present application also provides an embodiment of a device for quickly generating a long text summary.

[0129] Figure 3 This is a schematic diagram of the structure of the first embodiment of the long text summary rapid generation device provided by this application. Figure 3 The device provided in this embodiment is applied to a long text summary fast generation model, wherein the long text summary fast generation model comprises at least a summary encoder module and a summary decoder module using skip coding, wherein the skip coding encodes discontinuous text in the long text;

[0130] The summary encoder module is configured to read a long text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder, and adding an embedded representation of the position information of each read text block to reconstruct the first hidden representation to obtain a second hidden embedded representation;

[0131] The summary decoder module is configured to parse the second hidden embedding representation to predict and output a summary of the long text.

[0132] Figure 4 This is a schematic diagram of the structure of the first embodiment of the long text summary rapid generation model training device provided by this application. Figure 4 , the device includes a determination module 410, a processing module 420 and a training module 430; wherein,

[0133] The determination module 410 is configured to determine training parameters of the long text summary fast generation model; the training parameters include at least long short-term memory network parameters of the summary encoder and the summary decoder, and forward skipping agent parameters;

[0134] The processing module 420 is configured to select an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional log probability as a reward function;

[0135] The training module 430 is used to comprehensively train the long text summary rapid generation model according to the optimal training method corresponding to each parameter, and determine various parameters of the long text summary rapid generation model to complete the training of the long text summary rapid generation model.

[0136] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0137] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0138] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for quickly generating a long text summary, characterized in that: The method is implemented based on a fast long text summarization generation model, and the fast long text summarization generation model at least includes a summary encoder and a summary decoder using skip encoding, and the skip encoding encodes discontinuous texts in the long text; The specific steps of the long text summarization generation method include: obtaining the long text to be summarized, and dividing the long text into blocks to obtain multiple text blocks; The summary encoder reads the i-th text block, and calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of the text blocks; Predict the skip position based on the forward hidden state and backward hidden state of the i-th text block; Determine the serial number of the next text block to be read based on the skip position as the latest i value, and return to the step where the summary encoder reads the i-th text block; Combine the forward hidden states and backward hidden states of all the read text blocks to obtain the first hidden representation output by the encoder; Add the position information embedding representations of each read text block to reconstruct the first hidden representation to obtain a second hidden embedding representation; The summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation; The predicting the skip position based on the forward hidden state and backward hidden state of the i-th text block further includes: Summarize the forward hidden states and backward hidden states of all the previously read blocks to predict skip words; Determine the target text block where the skip word is located based on the position of the skip word; Take the end of the target text block as the predicted skip position.

2. The method according to claim 1, characterized in that Predicting the skip position based on the forward hidden state and backward hidden state of the i-th text block includes: Perform a pooling operation on the word embedding of the i-th text block to obtain a pooled embedding; Feed the first hidden representation and the pooled embedding of the i-th text block into a fully connected layer to respectively obtain the current representation and future representation of the i-th text block; Synthesize the current representation and future representation of the i-th text block to obtain the skip position of the i-th text block.

3. The method according to claim 1, characterized in that The adding the position embedding representations of each read text block to reconstruct the first hidden representation to obtain a second hidden embedding representation includes: Obtain the first hidden representation of the text block; Obtain the absolute position of the text block in the long text; Calculate the embedding of the absolute position using sine position encoding; Combine the first hidden representation of the text block with the added absolute position embedding to obtain the second hidden embedding representation of the text block.

4. The method according to claim 1, wherein The summary decoder predicting and outputting the summary of the long text based on the second hidden embedding representation includes: The summary decoder receives the second hidden embedding representation and the decoder state of the last cycle; Calculate the attention distribution through the second hidden embedding representation and the decoder state; Perform weighted summarization on the second hidden embedding representation based on the attention distribution to obtain a context vector; Generate the decoder state of the current cycle based on the decoder state and the context vector, update the decoder state, and return to the step of receiving the second hidden embedding representation until the update calculation of the preset cycle is completed; The decoder states of each cycle are summarized to obtain the vocabulary distribution pattern and form a summary of the long text.

5. The method according to claim 1, wherein The long text is divided into blocks to obtain multiple text blocks including: Obtain the long text to be summarized, determine each semantically complete text block based on the semantic content of the long text, and determine the size of each text block; Determine the maximum ratio of long text blocks according to the size of each text block; Segmenting the long text into a plurality of text blocks according to the maximum ratio and the size of each text block; Traversing the boundary semantics of each segmented text block, adjusting the semantically coherent boundary according to the maximum ratio, and obtaining an adjusted segmentation result; Add a marker at the end of each adjusted text block to obtain multiple text blocks.

6. A device for quickly generating a long text summary, characterized in that: The device is applied to the method according to any one of claims 1 to 5, wherein the device is applied to a long text summary fast generation model, wherein the long text summary fast generation model comprises at least a summary encoder module and a summary decoder module using skip coding, wherein the skip coding encodes discontinuous text in the long text; The summary encoder module is configured to read a text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder, and adding an embedded representation of the position information of each read text block to reconstruct the first hidden representation to obtain a second hidden embedded representation; The summary decoder module is configured to parse the second hidden embedding representation to predict and output a summary of the long text.

Images