Long text abstract rapid generation method, model training method and device

Abstract

The invention provides a long text abstract rapid generation method and device and a model training method and device. The long text abstract rapid generation method provided by the invention comprises the following steps: obtaining a long text to be abstracted, and partitioning the long text to obtain a plurality of text blocks; the abstract encoder reads the ith text block and calculates the forward hidden state and the backward hidden state of the ith text block; predicting a skip reading position based on the forward hidden state and the backward hidden state of the ith text block; determining the serial number of the next to-be-read text block based on the skip reading position, taking the serial number as the latest i value, and returning to the step of reading the ith text block by the abstract encoder; combining the forward hidden states and the backward hidden states of all the read text blocks to obtain a first hidden representation output by the encoder; adding the position information embedding representation of each read text block, and reconstructing the first hidden representation to obtain a second hidden embedding representation; the summary decoder predicts to output a summary of the long text based on the second hidden embedding representation.

Description

Technical Field

[0001] This application relates to the technical field of natural language understanding, and particularly to a method for quickly generating long text summaries, a model training method, and a device. Background Art

[0002] Automatic summarization is an important research direction in the field of natural language processing. Its purpose is to compress, summarize, and refine the original text to provide users with a concise and to-the-point text description that covers the core content of the original text. With the rapid development of the Internet, text information resources have shown exponential growth, making it impossible for people to conveniently and quickly obtain important content from a vast amount of information. Text summarization technology can "compress and refine" a large amount of long text information into short text information with refined content, helping people quickly obtain important content, and is widely used in scenarios such as news summary generation, e-commerce platform evaluation summaries, and text information retrieval, and has always received extensive research and attention.

[0003] Currently, the commonly used method for generating text summaries is the generative summary method based on neural networks. This method has achieved good results on some short text datasets, but it cannot achieve good results for longer texts, it is difficult to find key information in a large amount of input information, and it requires a large amount of computing power, resulting in a slow speed of generating summaries. Summary of the Invention

[0004] In view of this, this application provides a method for quickly generating long text summaries, a model training method, and a device, which are used to solve the problems of inaccurate long text summary generation, large consumption of computing resources, and slow speed.

[0005] Specifically, this application is implemented through the following technical solutions: The first aspect of this application provides a method for quickly generating long text summaries. The method is implemented based on a long text summary quick generation model. The long text summary quick generation model at least includes a summary encoder and a summary decoder using skip encoding. The skip encoding encodes discontinuous texts in the long text; The method for quickly generating long text summaries specifically includes: obtaining the long text to be summarized, and dividing the long text into blocks to obtain a plurality of text blocks; The summary encoder reads the i-th text block, calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text; Predict the skip reading position based on the forward hidden state and backward hidden state of the i-th text block; Determine the number of the next text block to be read based on the skip position, take it as the latest i value, and return to the step of reading the i-th text block by the summary encoder; Combine the forward hidden states and backward hidden states of all the read text blocks to obtain the first hidden representation output by the encoder; Adding the position information embedding representation of each read text block, reconstructing the first hidden representation, to obtain a second hidden embedding representation; The summary decoder predicts and outputs a summary of the long text based on the second hidden embedding representation.

[0006] A second aspect of the present application provides a long text summary fast generation model training method for training the long text summary fast generation model, the training method comprising: Determining the training parameters of the long text summary fast generation model; the training parameters at least include long short-term memory network parameters of the summary encoder and the summary decoder, and forward skipping agent parameters; Selecting an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional log probability as a reward function; The long text summary fast generation model is comprehensively trained according to the optimal training method corresponding to each parameter, and each parameter of the long text summary fast generation model is determined to complete the training of the long text summary fast generation model.

[0007] A third aspect of the present application provides a long text summary fast generation device, the device is applied to a long text summary fast generation model, the long text summary fast generation model at least includes a summary encoder module and a summary decoder module using jump coding, the jump coding encodes discontinuous text in the long text; The summary encoder module is used to read a long text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder and add the position information embedded representation of each read text block, reconstruct the first hidden representation, and obtain a second hidden embedded representation; The summary decoder module is used to parse the second hidden embedding representation to predict and output a summary of the long text.

[0008] A fourth aspect of the present application provides a long text summary rapid generation model training device, the device includes a determination module, a processing module and a training module; wherein, The determining module is configured to determine training parameters of the long text summary fast generation model; the training parameters at least include long short-term memory network parameters of a summary encoder and a summary decoder, and a forward skip reading agent parameter; The processing module is configured to select an optimal training method for the training parameters, wherein the optimal training method for the forward skip reading agent parameter is different from that for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters by using a policy gradient algorithm and selecting a conditional log probability as a reward function; The training module is configured to comprehensively train the long text summary fast generation model according to the optimal training method corresponding to each parameter, and determine each parameter of the long text summary fast generation model to complete the training of the long text summary fast generation model.

[0009] The long text summary fast generation method, model training method and device provided by the present application first obtain a long text to be summarized, and divide the long text into multiple text blocks, then read the i-th text block through the summary encoder, calculate the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text, so as to predict a skip reading position based on the forward hidden state and backward hidden state of the i-th text block, and further determine the serial number of the next text block to be read based on the skip reading position as the latest i value, and return to the step of reading the i-th text block by the summary encoder. Further, combine the forward hidden states and backward hidden states of all the read text blocks to obtain a first hidden representation output by the encoder, and add the position information embedding representations of each read text block to reconstruct the first hidden representation to obtain a second hidden embedding representation, and finally predict and output the summary of the long text by the summary decoder based on the second hidden embedding representation. In this way, the long text is cut by dividing it into multiple text blocks, and each text block is semantically complete, ensuring the integrity of the subsequent summary generation; predicting the skip reading position based on the forward hidden state and backward hidden state of each text block can selectively generate a long text summary from some text blocks, that is, only need to analyze and process some texts, reducing the amount of text that the model needs to process, thereby reducing the consumption of computing resources while ensuring the accuracy of the summary and improving the summary generation speed. Description of the Drawings

[0010] Figure 1 It is a flowchart of the first embodiment of the long text summary fast generation method provided by the present application; Figure 2 It is a flowchart of the first embodiment of the long text summary fast generation model training method provided by the present application; Figure 3A schematic diagram of the structure of the first embodiment of the long text summary fast generation device provided by the present application; Figure 4 This is a structural diagram of Example 1 of the long text summary rapid generation model training device provided in this application. DETAILED DESCRIPTION

[0011] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0012] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0013] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0014] The present application provides a method for quickly generating a long text summary, a model training method and a device, which are used to solve the problems of inaccurate generation of long text summaries, large consumption of computing resources and slow speed.

[0015] The method, model training method and device for quickly generating a long text summary provided by this application first obtain the long text to be summarized, and divide the long text into multiple text blocks. Then, the i-th text block is read by the summary encoder to calculate the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer and 0 < i ≤ N, and N is the total number of text blocks in the long text. Thus, the skip reading position is predicted based on the forward hidden state and backward hidden state of the i-th text block, and then the number of the next text block to be read is determined based on the skip reading position as the latest i value, and the step of returning the summary encoder to read the i-th text block is performed. Further, the forward hidden states and backward hidden states of all the read text blocks are combined to obtain the first hidden representation output by the encoder, and the position information embedding representation of each read text block is added to reconstruct the first hidden representation to obtain the second hidden embedding representation. Finally, the summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation. In this way, a text skip reading technology is designed in the encoder-decoder architecture model, that is, an enhanced lightweight forward skip reading agent is designed. During the process of inputting the long text, it can select the content to be read, focus on the important information of the long text, and discard the unimportant information, resulting in significant improvements in accuracy and speed. In addition, the long text is cut by dividing it into multiple text blocks, and each text block has a complete semantics, ensuring the integrity of the subsequent summary generation; predicting the skip reading position based on the forward hidden state and backward hidden state of each text block can selectively generate the long text summary from some text blocks, that is, only need to analyze and process part of the text, reducing the amount of text that the model needs to process, thereby reducing the consumption of computing resources while ensuring the accuracy of the summary and improving the summary generation speed.

[0016] Specific embodiments are given below to introduce the technical solution of this application in detail.

[0017] Figure 1 It is a flowchart of the first embodiment of the method for quickly generating a long text summary provided by this application. Please refer to Figure 1 In this embodiment, the method provided is implemented based on a long text summary quick generation model. The long text summary quick generation model at least includes a summary encoder and a summary decoder that adopt jump coding. The jump coding encodes the discontinuous text in the long text. The long text summary quick generation method may include: S101. Obtain the long text to be summarized, and divide the long text into multiple text blocks.

[0018] Specifically, the process of dividing the long text into multiple text blocks may include: (1) Obtain the long text to be summarized, determine each semantically complete text block based on the semantic content of the long text, and determine the size of each text block.

[0019] It should be noted that a text block refers to a smaller part into which a long text is divided according to certain rules or standards during text processing. In one possible implementation, after obtaining the long text to be summarized, the long text to be summarized is first segmented and divided into multiple paragraphs or sentences. For example, punctuation marks, line breaks or other separators can be used to segment long texts. And a semantic analysis is performed on the segmented text blocks to determine the semantic content of each text block. For example, in one embodiment, this can be achieved through natural language processing (NLP) technology, such as part-of-speech tagging, named entity recognition, semantic role labeling, etc.

[0020] Further, the size of each text block is determined according to the semantic content, for example, this can be determined based on indicators such as the number of words, the number of characters, the number of sentences, etc. It should be noted that, generally, a text block should contain a complete semantic unit so that the information after summary can maintain integrity and coherence.

[0021] (2) Determine the maximum proportion of long text blocks based on the size of each text block.

[0022] It should be noted that determining the maximum ratio of long text blocks refers to determining a threshold value for controlling the maximum ratio of dividing a long text into text blocks. Specifically, this ratio can be determined based on the size of each text block to ensure that the generated text block can maintain semantic integrity without making the text block too short or too long. For example, in one embodiment, the maximum ratio can be determined based on the average size of the text block.

[0023] (3) Segment the long text into a plurality of text blocks according to the maximum ratio and the size of each text block.

[0024] It should be noted that multiplying the size of each text block by the maximum ratio gives the maximum length of the text block, which represents the maximum length that each text block can reach during segmentation, so as to segment the long text according to the maximum length of the text block. For example, in one embodiment, starting from the beginning of the long text, the text can be segmented into multiple text blocks according to the maximum length until the long text is completely segmented, obtaining multiple text blocks. As another alternative embodiment, the text blocks after initial segmentation can be obtained according to the semantics of the long text, and the text blocks after initial segmentation can be adjusted according to the size of the text blocks after initial segmentation and the maximum ratio, so as to obtain the finally segmented text blocks. For example, if the size of the text block after initial segmentation is less than or equal to the maximum ratio, the text block after initial segmentation remains unchanged; if the size of the text block after initial segmentation is greater than the maximum ratio, the text block is semantically segmented again with the maximum ratio as the unit. Then, for this text block, the text blocks after secondary semantic segmentation are the finally segmented text blocks. For example, if the size of text block A1 is less than the maximum ratio, then A1 is retained; if the size of text block B1 is greater than the maximum ratio, then B1 is further segmented into BB1 and BB2. Finally, the text blocks obtained from the long text are specifically A1, BB1, and BB2.

[0025] (4)Traverse the boundary semantics of each text block after segmentation, adjust the boundaries with coherent semantics according to the maximum ratio, and obtain the adjusted segmentation result.

[0026] It should be noted that the boundary semantics can be determined by analyzing the content and context of the text block. Specifically, adjusting the boundaries with coherent semantics according to the maximum ratio may include moving the boundaries inward or expanding them outward so that the adjusted boundaries meet the requirements of the maximum ratio. Further, in order to improve the accuracy of abstract generation and the semantic integrity of the text block, and to avoid poor accuracy of abstract generation caused by overly fine semantic segmentation in the early stage, the semantics at the end of each text block and the coherence degree of the boundary semantics of adjacent text blocks are also identified, specifically referring to the end of the previous text block and the beginning of the next text block.

[0027] (5)Add a marker at the end of each adjusted text block to obtain multiple text blocks.

[0028] It should be noted that adding a marker at the end of each adjusted text block can help identify the end position of the text block and further facilitate subsequent processing. Specifically, this marker can be a special symbol, label, or other symbol that can be used to indicate the end of the text block.

[0029] S102. The abstract encoder reads the i-th text block and calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text.

[0030] It should be noted that the abstract encoder is an abstract encoder using skip encoding. That is, the text block read at this time is the text block after skipping. That is to say, the skipped text block is not encoded, and only the text block that needs to be read is encoded. In this application, a bidirectional long short-term memory network is used as the abstract encoder. Among them, skip encoding refers to encoding discontinuous texts in the long text; the bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, BiLSTM) is a deep learning model, usually used for modeling and processing of sequence data. BiLSTM combines long short-term memory networks LSTM in two directions, processes from the forward and backward directions of the sequence respectively, to capture long-term dependencies in the sequence data.

[0031] Specifically, after obtaining multiple text blocks including a series of tokens from x1:N, which represent a long document composed of text blocks including a series of tokens from x1 to xN, for the i-th text block, the i-th token can be read through the forward LSTM, and the forward hidden state can be calculated. ; Read the i-th token through the backward LSTM, and the backward hidden state can be calculated. Among them is the word embedding of the input token xi. Among them, i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text.

[0032] S103. Predict the skip position based on the forward hidden state and backward hidden state of the i-th text block.

[0033] Specifically, predicting the skip position based on the forward hidden state and backward hidden state of the i-th text block includes: (1) Aggregate the forward hidden states and backward hidden states of all blocks read before to predict the skip word.

[0034] It should be noted that for each text block, after obtaining its forward and backward hidden states through the forward and backward LSTMs, these hidden states can be connected to achieve aggregation.

[0035] (2) Determine the target text block where the skip word is located based on the position of the skip word.

[0036] For the predicted skipped words, the target text block where they are located is determined according to their positions. Specifically, this can be achieved by recording the starting position and the ending position of each text block. For example, if the position where the skipped word is located is between the starting position and the ending position of the ith text block, then the skipped word is located in the ith target text block. Specifically, in the subsequent text blocks of the currently read text block, the similarity between each word and the skipped word can be matched in sequence, and if the similarity is greater than a preset threshold, the word is determined to be a skipped word.

[0037] (3) The end of the target text block is used as the predicted skipping position.

[0038] It should be noted that in order to simplify the skip coding process, the scope of the skip coding is limited, and the skip coding can only be at the end of the text block, so the end of the target text block is used as the predicted skip position.

[0039] Compared with the text cutting method of the prior art that directly divides a long text, the method provided by the present invention can predict the next skipped word based on the long text itself, that is, based on the characteristics of each long text itself, the previous text predicts the skipping position of the following text, thereby improving the accuracy of the text after skipping, and further improving the accuracy of summary generation; further, the skipping of the long text of the present invention is predicted based on a bidirectional encoder, while utilizing contextual information, thereby improving the accuracy of the prediction; in addition, based on skipping, part of the text blocks are directly skipped, so that only part of the text needs to be encoded, parsed and summarized, reducing the amount of data for text processing, and directly reducing the amount of text in the encoding stage, that is, the skipped text paragraphs do not need to be encoded, thereby greatly reducing the processing time of text generation and improving the efficiency of summary generation.

[0040] S104, determining the number of the next text block to be read based on the skip position, taking it as the latest i value, and returning to the step of the summary encoder reading the i-th text block.

[0041] Specifically, the target text block where the skip position is located can be determined according to the skip position. In specific implementation, for example, it can be implemented by comparing the skip position with the start position and the end position of each text block. If the skip position is between the start position and the end position of the i-th text block, the number of the next text block to be read is i.

[0042] Furthermore, the number of the next text block to be read is determined according to the text block where the skip position is located, and is used as the latest i value, and the latest i value is passed to the summary encoder so that it reads the i-th text block and performs subsequent processing.

[0043] S105. Combine the forward hidden states and backward hidden states of all the read text blocks to obtain a first hidden representation output by the encoder.

[0044] Combining the forward hidden state and backward hidden state of the text block read above, for example, the first hidden representation of the encoder output can be obtained by concatenating the vectors .

[0045] It should also be noted that the last hidden state can be used To predict the skipped steps, in order to make the skip coding not involve too much calculation, a lightweight forward network can be used to browse the K text blocks. Specifically, the predicted skip position distribution is obtained by outputting the first hidden representation based on the forward hidden state and the backward hidden state of the i-th text block, including: (1) Perform a pooling operation on the word embeddings of the i-th text block to obtain a pool embedding.

[0046] It should be noted that the pooling operation can be set according to actual needs and is not limited in this embodiment. Among them, common pooling operations include maximum pooling, average pooling, etc. In maximum pooling, the maximum value in the word embedding sequence is taken as the pooling embedding; in average pooling, the average value of the word embedding sequence is taken as the pooling embedding. These operations help to convert the word-level representation into a higher-level text block-level representation. For example, in this embodiment, the word embedding of the i-th text block is pooled to obtain the pool embedding .

[0047] (2) The first hidden representation and pool embedding of the i-th text block are fed to the fully connected layer to obtain the current representation and future representation of the i-th text block respectively.

[0048] Specifically, the first hidden representation of the i-th text block is and pool embedding Feeding into the fully connected layer can obtain the current representation of the i-th text block and future .

[0049] (3) The skipping position of the i-th text block is obtained by combining the current representation and the future representation of the i-th text block.

[0050] Connect current representation and future The skip position of the i-th text block can be obtained .

[0051] It should be noted that, combined with the above description, the word embeddings of all text blocks are collected to obtain their representations, and t is the sequence number of the skip operation, we can get: in, is the pool embedding for the text block, For the current representation, For the future, For skip distribution, represents word embedding, is the first hidden representation.

[0052] Specifically, the model is The sample skip value j = [-1, K], where -1 means early stop, 0 means no skipping, and the others indicate the number of blocks to skip. After the skip, the model continues from The next segment is read. This process continues until one of the following occurs: (i)-1 is sampled from the skip distribution.

[0053] (ii) The proportion of reading exceeds P.

[0054] (iii) Reaching the last mark.

[0055] S106: Add the position information embedded representation of each read text block, reconstruct the first hidden representation, and obtain a second hidden embedded representation.

[0056] It should be noted that due to the nature of the LSTM encoder, the model is well informed about the location of informative tokens. However, the encoder with filtering operations discards the input sequence. Therefore, the model only focuses on the relative position relationship of the selected input fragments, and when encoding the input text, the absolute position information of the tokens in the original input is lost. Therefore, in the summary encoder, the input sequence is encoded as the first hidden state After that, you can add absolute position The embedding of is returned to the first hidden state to reconstruct the absolute position information and obtain the second hidden embedding representation.

[0057] Specifically, adding the position embedding representations of each read text block and reconstructing the first hidden representation to obtain the second hidden embedding representation includes: (1) Obtain a first hidden representation of the text block.

[0058] (2) Obtain the absolute position of the text block in the long text.

[0059] Determine the absolute position of the text block in the long text, that is, the index position of the text block in the long text. This position is usually the starting position or middle position of the text block in the entire long text sequence.

[0060] (3) Compute the embedding of the absolute position using sinusoidal position encoding.

[0061] Sinusoidal positional encoding is a commonly used method to generate a fixed-dimensional position embedding vector for each position in a sequence to represent the position information.

[0062] (4) Combining the first hidden representation of the text block with the added absolute position embedding to obtain a second hidden embedding representation of the text block.

[0063] Specifically, the first hidden representation of the text block is combined with the added absolute position embedding to obtain the second hidden embedding representation of the text block as .

[0064] S107: The summary decoder predicts and outputs a summary of the long text based on the second hidden embedding representation.

[0065] It should be noted that the summary decoder predicts and outputs a summary of the long text based on the second hidden embedding representation, including: (1) The summary decoder receives the second hidden embedded representation and the decoder state of the last cycle.

[0066] The purpose of the abstract decoder is to To generate the summary token. The summary decoder here is a single-layer unidirectional LSTM, and the summary decoder receives the second hidden embedding representation and the last decoder state , and then output the decoder state of the last cycle .

[0067] , in, is the output of the summary decoder at time t-1, that is, a word that generates a summary.

[0068] (2) Calculating an attention distribution via the second hidden embedding representation and the decoder state.

[0069] The attention mechanism is used to calculate the context vector , which can be viewed as a fixed-size dense representation of the source text at the current decoding step. At each decoding step, the decoding state and a second hidden embedding representation to compute the attention distribution over the input tokens.

[0070] (3) Perform a weighted summary on the second hidden embedding representation based on the attention distribution to obtain a context vector.

[0071] Specifically, the formula is as follows: , in To learn the parameters, is the hidden state representation before the skip operation, is the attention probability distribution, is the context vector, represents the word embedding vector, Decoder status.

[0072] (4) generating a decoder state of the current cycle based on the decoder state and the context vector, updating the decoder state, and returning to the step of receiving the second hidden embedding representation until the update calculation of the preset cycle is completed.

[0073] Update the decoder state based on the decoder state of the current cycle and the generated context vector This updating process can be implemented using a recurrent neural network (such as a long short-term memory network) so that the model can maintain a memory of the context and adjust its state based on the current contextual information.

[0074] (5) Summarize the decoder states of each cycle to obtain the vocabulary distribution pattern and form a summary of the long text.

[0075] Set the decoder status and context vector Add together to calculate the vocabulary distribution pattern: .

[0076] The long text summary fast generation method provided in this embodiment first obtains the long text to be summarized, divides the long text into chunks to obtain a plurality of text chunks, and then reads the i-th text chunk through the summary encoder, calculates the forward hidden state and backward hidden state of the i-th text chunk, where i is a positive integer, 0 < i ≤ N, and N is the total number of text chunks of the long text. Thus, based on the forward hidden state and backward hidden state of the i-th text chunk, the skip reading position is predicted, and then based on the skip reading position, the number of the next text chunk to be read is determined as the latest i value, and the step of returning the summary encoder to read the i-th text chunk is performed. Further, the forward hidden states and backward hidden states of all the read text chunks are combined to obtain the first hidden representation output by the encoder, and the position information embedding representation of each read text chunk is added to reconstruct the first hidden representation to obtain the second hidden embedding representation. Finally, the summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation. In this way, by using a bidirectional long short-term memory network (bi-LSTM) to encode the input text, an enhanced lightweight forward skip reading agent is separately designed in the encoder module. When the bidirectional long short-term memory network loops to encode the input, the skip reading agent can help the encoder selectively skip secondary input text segments based on current and future information. During the decoding process, these secondary segments are excluded from the attention mechanism. This filtering operation reduces the computational cost and attracts attention to key information to improve the accuracy of the summary while increasing the speed of summary generation.

[0077] Figure 2 It is a flowchart of the first embodiment of the training method for the long text summary fast generation model provided in this application. Please refer to Figure 2 , and this long text summary fast generation model training method is used to train a long text summary fast generation model, and the training method includes: S201. Determine the training parameters of the long text summary fast generation model; the training parameters at least include the long short-term memory network parameters of the summary encoder and the summary decoder, and the forward skip reading agent parameters.

[0078] It should be noted that there are three parameters in the model that need to be estimated: , and . Among them, and respectively represent the long short-term memory network LSTM parameters of the summary encoder and the summary decoder; represents the parameter of the forward skip reading agent.

[0079] S202: selecting an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional logarithmic probability as a reward function.

[0080] It should be noted that cannot be estimated directly, but and It can be calculated by back propagation; the policy gradient algorithm is a class of methods for optimizing parameterized policies, especially for reinforcement learning tasks. When training a model, the policy gradient algorithm updates the parameters of the model by maximizing the expected cumulative reward. The core idea of ​​these algorithms is to directly optimize the parameters of the policy, rather than optimizing the value function or the state value function. Using the policy gradient algorithm to optimize the model parameters means that the parameters of the model will be updated by gradient ascent to maximize the expected cumulative reward.

[0081] It should also be noted that the conditional log probability is chosen as the reward function instead of ROUGE because the loss of the model with the ROUGE reward function converges slowly during training and the learning objective of skipping with the ROUGE reward function is too far away from the learning objective of the summary decoder.

[0082] Specific, definition As given The activation function is designed as follows: , In practice, the skipping action is reformulated as a reinforcement learning problem, and the high variance problem can be alleviated by using a baseline b, which is the average reward of previous iterations used during training.

[0083] It should be noted that the forward skipping agent usually simply reads the first half of the article. In order to force the forward skipping agent to explore further, we also use the entropy on the probability distribution of skipping actions as the maximum entropy regularization of the forward skipping agent, including: (1) Determine the loss function of the training process.

[0084] (2) Training the long text summary fast generation model based on the loss function.

[0085] It should also be noted that during the training process, the goal of the model is to minimize the total loss function, wherein the loss function at least includes Lpg and the entropy of the probability distribution of skipping.

[0086] For the entropy of the probability distribution of skipping actions, negative entropy can be used as the loss term. The larger the entropy, the greater the uncertainty of the model, and the larger the value of the regularization term, which prompts the model to choose skipping actions more exploratory. Specifically, in each training iteration, the value of the loss function is calculated as follows: in, is the tth action selected from the sample, is the entropy of the jump distribution at time t, and a is the entropy regularization factor. After combining these two losses, our overall training loss function is as follows: S203, comprehensively training the long text summary fast generation model according to the optimal training method corresponding to each parameter, and determining various parameters of the long text summary fast generation model to complete the training of the long text summary fast generation model.

[0087] The long text summary fast generation model training method provided in this embodiment can improve the accuracy and summary generation speed of the long text summary fast generation model and reduce the computing cost.

[0088] Corresponding to the aforementioned embodiment of a method for quickly generating a long text summary, the present application also provides an embodiment of a device for quickly generating a long text summary.

[0089] Figure 3 This is a schematic diagram of the structure of the first embodiment of the long text summary fast generation device provided by this application. Please refer to Figure 3 The device provided in this embodiment is applied to a long text summary fast generation model, and the long text summary fast generation model at least includes a summary encoder module and a summary decoder module using skip coding, and the skip coding encodes discontinuous text in the long text; The summary encoder module is used to read a long text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder and add the position information embedded representation of each read text block, reconstruct the first hidden representation, and obtain a second hidden embedded representation; The summary decoder module is used to parse the second hidden embedding representation to predict and output a summary of the long text.

[0090] Figure 4 This is a schematic diagram of the structure of the first embodiment of the long text summary rapid generation model training device provided by this application. Please refer to Figure 4, the device includes a determination module 410, a processing module 420 and a training module 430; wherein, The determination module 410 is used to determine the training parameters of the long text summary fast generation model; the training parameters at least include the long short-term memory network parameters of the summary encoder and the summary decoder, and the forward skipping proxy parameters; The processing module 420 is used to select an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting a conditional log probability as a reward function; The training module 430 is used to comprehensively train the long text summary fast generation model according to the optimal training method corresponding to each parameter, and determine the various parameters of the long text summary fast generation model to complete the training of the long text summary fast generation model.

[0091] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0092] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0093] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for quickly generating a long text summary, characterized in that: The method is implemented based on a fast long text summarization generation model, and the fast long text summarization generation model at least includes a summary encoder and a summary decoder that adopt skip encoding, and the skip encoding encodes discontinuous texts in the long text; The specific steps of the fast long text summarization generation method include: obtaining the long text to be summarized, and dividing the long text into blocks to obtain a plurality of text blocks; The summary encoder reads the i-th text block, and calculates the forward hidden state and backward hidden state of the i-th text block, where i is a positive integer, 0 < i ≤ N, and N is the total number of text blocks of the long text; Predict the skip reading position based on the forward hidden state and backward hidden state of the i-th text block; Determine the serial number of the next text block to be read based on the skip reading position as the latest i value, and return to the step of the summary encoder reading the i-th text block; Combine the forward hidden states and backward hidden states of all the read text blocks to obtain the first hidden representation output by the encoder; Add the position information embedding representations of the read text blocks, and reconstruct the first hidden representation to obtain the second hidden embedding representation; The summary decoder predicts and outputs the summary of the long text based on the second hidden embedding representation.

2. The method according to claim 1, characterized in that The predicting the skip reading position based on the forward hidden state and backward hidden state of the i-th text block further includes: Summarize the forward hidden states and backward hidden states of all the previously read blocks to predict the skip reading words; Determine the target text block where the skip reading word is located based on the position of the skip reading word; Use the end of the target text block as the predicted skip reading position.

3. The method according to claim 1, characterized in that Predicting the skip reading position based on the forward hidden state and backward hidden state of the i-th text block includes: Performing a pooling operation on the word embeddings of the i-th text block to obtain a pooled embedding; Feed the first hidden representation and the pooled embedding of the i-th text block into a fully connected layer to respectively obtain the current representation and future representation of the i-th text block; Integrate the current representation and future representation of the i-th text block to obtain the skip reading position of the i-th text block.

4. The method according to claim 1, characterized in that: The adding the position embedding representations of the read text blocks and reconstructing the first hidden representation to obtain the second hidden embedding representation includes: Obtain the first hidden representation of the text block; Obtain the absolute position of the text block in the long text; Calculate the embedding of the absolute position using sine position encoding; Combine the first hidden representation of the text block with the added absolute position embedding to obtain the second hidden embedding representation of the text block.

5. The method according to claim 1, characterized in that The summary decoder predicting and outputting the summary of the long text based on the second hidden embedding representation includes: The summary decoder receives the second hidden embedding representation and the decoder state of the last cycle; Calculate the attention distribution through the second hidden embedding representation and the decoder state; Perform weighted summarization on the second hidden embedding representation based on the attention distribution to obtain a context vector; Generate the decoder state of the current cycle based on the decoder state and the context vector, update the decoder state, and return to the step of receiving the second hidden embedding representation until the update calculation of the preset cycle is completed; The decoder states of each cycle are summarized to obtain the vocabulary distribution pattern and form a summary of the long text.

6. The method according to claim 1, characterized in that The long text is divided into blocks to obtain multiple text blocks including: Obtain the long text to be summarized, determine each semantically complete text block according to the semantic content of the long text, and determine the size of each text block; Determine the maximum proportion of long text blocks according to the size of each text block; Segmenting the long text according to the maximum ratio and the size of each text block to obtain a plurality of text blocks; Traversing the boundary semantics of each segmented text block, adjusting the semantically coherent boundary according to the maximum ratio, and obtaining an adjusted segmentation result; Add a marker to the end of each adjusted text block to get multiple text blocks.

7. A long text summary rapid generation model training method, characterized in that: The training method comprises: Determining the training parameters of the long text summary fast generation model; the training parameters at least include long short-term memory network parameters of the summary encoder and the summary decoder, and forward skipping agent parameters; Selecting an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional log probability as a reward function; The long text summary fast generation model is comprehensively trained according to the optimal training method corresponding to each parameter, and each parameter of the long text summary fast generation model is determined to complete the training of the long text summary fast generation model.

8. The method according to claim 7, characterized in that The method further comprises: Determine the loss function of the training process; Training the long text summary fast generation model based on the loss function; The loss function at least includes Lpg and the entropy of the probability distribution of skipping.

9. A device for quickly generating a long text summary, characterized in that: The device is applied to a long text summary fast generation model, the long text summary fast generation model at least includes a summary encoder module and a summary decoder module using skip coding, the skip coding encodes discontinuous text in the long text; The summary encoder module is used to read a long text block and calculate a forward hidden state and a backward hidden state of the text block, thereby predicting a skip position based on the forward hidden state and the backward hidden state of the text block to determine the number of the text block to be read, and then combining the forward hidden state and the backward hidden state of all the read text blocks to obtain a first hidden representation output by the encoder and add the position information embedded representation of each read text block, reconstruct the first hidden representation, and obtain a second hidden embedded representation; The summary decoder module is used to parse the second hidden embedding representation to predict and output a summary of the long text.

10. A long text summary rapid generation model training device, characterized in that: The device comprises a determination module, a processing module and a training module; wherein, The determination module is used to determine the training parameters of the long text summary fast generation model; the training parameters at least include long short-term memory network parameters of the summary encoder and the summary decoder, and forward skipping proxy parameters; The processing module is used to select an optimal training method for the training parameters, wherein the optimal training method for the forward skipping agent parameters is different from the optimal training method for the long short-term memory network parameters, and the optimal training method at least includes optimizing the parameters using a policy gradient algorithm and selecting conditional logarithmic probability as a reward function; The training module is used to comprehensively train the long text summary rapid generation model according to the optimal training method corresponding to each parameter, and determine the various parameters of the long text summary rapid generation model to complete the training of the long text summary rapid generation model.

Images