Deep learning-based text summarization generation method, device, and storage medium

Abstract

The present invention provides a method, device, and storage medium for generating text summaries based on deep learning. The method includes: obtaining a sequence of word vectors, and inputting the sequence of word vectors into a word-level encoder to obtain word-level encodings; inputting the word-level encodings into a context extraction unit to obtain context information; inputting the context information into a context encoder to obtain context information encodings; performing a comprehensive decoding operation on the word-level encodings and the context information encodings to obtain a text summary. The present invention relates to the technical field of automatic text summary generation, and solves the problems of poor sentence coherence and information errors in the existing text summary generation methods.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic text summarization generation, and particularly to a method, device and storage medium for generating text summaries based on deep learning. Background Art

[0002] The task of text summarization is one of the subtasks in the field of natural language processing. Its main purpose is to extract the main content of the text from relatively long texts. Since the 21st century, with the rapid popularization of the Internet, people can use this tool to conveniently and quickly conduct information interaction, thus generating a huge amount of information. However, human energy is limited, and people always hope to obtain the information they need in a short time. The text summarization technology can quickly extract key information from a large amount of text data, effectively helping people save time.

[0003] The generation methods of text summaries are mainly divided into two types: extractive and generative. Traditional text summarization tasks mainly adopt extractive methods. The extractive method is mainly based on sentences. Its main idea is: assign a label of 0 or 1 to all sentences in the original text to indicate whether they do not belong to the summary or belong to the summary, and finally select all sentences labeled as 1 to form the summary. Although it can ensure the correctness of syntax and grammar, since the extracted sentences are often not adjacent in the original text, the coherence between sentences is poor. In the existing technology, the generative summary method generates summaries by understanding the content of the source text, and its sentence coherence is better, but it is prone to generating incorrect information.

[0004] It can be seen that the existing text summary generation methods have problems of poor sentence coherence and incorrect information. Summary of the Invention

[0005] Aiming at the deficiencies in the existing technology, the present invention provides a method for generating text summaries based on deep learning, which solves the problems of poor sentence coherence and incorrect information in the existing text summary generation methods.

[0006] In a first aspect, the present invention provides a method for generating text summaries based on deep learning, the method comprising:

[0007] Obtain a sequence of word vectors, and input the sequence of word vectors into a word-level encoder to obtain word-level encodings;

[0008] Input the word-level encodings into a context extraction unit to obtain context information;

[0009] Input the context information into a context encoder to obtain context information encodings;

[0010] Perform a comprehensive decoding operation on the word-level encodings and the context information encodings to obtain a text summary.

[0011] Further, obtaining a sequence of word vectors includes:

[0012] Obtaining a text sequence \(W=(w_1, w_2, \ldots, w_{}\) n ), where the value of \(w_{}\) i represents the index of the \(i\)-th word in the text sequence in the vocabulary;

[0013] Based on the Word2Vec word embedding model, confirming the vector representation of each word in the text sequence \(W=(w_1, w_2, \ldots, w_{}\) n ), and obtaining a sequence of word vectors \(X=(x_1, x_2, \ldots, x_{}\) n ), where \(x_{}\) i \(\in R^{}\) d .

[0014] Further, the word-level encoder includes a bidirectional gated recurrent unit GRU. Inputting the sequence of word vectors into the word-level encoder to obtain word-level encoding includes:

[0015] Processing the sequence of word vectors \(X=(x_1, x_2, \ldots, x_{}\) n ), where \(x_{}\) i \(\in R^{}\) d in both forward and backward directions through the bidirectional gated recurrent unit GRU to obtain a forward hidden state sequence and a backward hidden state sequence

[0016] Based on the hidden variables in both forward and backward directions at the same moment, confirming the word-level encoding at that moment. The calculation formula is:

[0017]

[0018]

[0019] where is the word-level encoding at time \(t\); \(g(\cdot)\) is the calculation function of the gated linear unit GLU, and \(\sigma(.)\) represents the sigmoid function, represents element-wise multiplication;

[0020] Obtaining the word-level encoding

[0021] Further, the context extraction unit includes two extraction modules, and each extraction module includes a convolutional layer with a gated linear unit GLU. Inputting the word-level encoding into the context extraction unit to obtain context information includes:

[0022] The two extraction modules sequentially calculate the word-level encoding to obtain context information. The calculation formula for each extraction module is:

[0023] O = K i *H W +b i = [AB] ∈ R 2d ;

[0024]

[0025] where * represents the convolution operation, and K i represents the i-th convolution kernel; σ(.) represents the sigmoid function.

[0026] Furthermore, the context information is input into the context encoder to obtain the context information encoding, and the calculation formula is:

[0027]

[0028]

[0029] where f gru is the calculation function of the gated recurrent unit GRU; g(·) is the calculation function of the gated linear unit GLU.

[0030] Furthermore, the decoder performs a comprehensive decoding operation on the word-level encoding and the context information encoding. The decoder includes a gated recurrent unit GRU with an attention mechanism to obtain the text summary, including:

[0031] According to the processed context information encoding and the decoder state, obtain the context variable unique to the time step, and the calculation formula is:

[0032]

[0033] a t = softmax(e t );

[0034]

[0035] where is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, x t represents the decoder input at the current moment, represents the context variable at time t, W h , W c , W s , b atten and v T all represent learnable parameters, and a t represents the attention distribution value;

[0036] Based on the context variables unique to the time step, confirm the probability distribution of the current time step on the target vocabulary. The calculation formula is:

[0037]

[0038] s t represents the hidden state of the decoder at the current moment; represents the context variable at time t, V', V, b, and b' are all learnable parameters, p vocab represents the probability distribution on the target vocabulary;

[0039] The decoder outputs after obtaining the summary sequence based on the probability distribution p vocab on the target vocabulary.

[0040] Furthermore, it also includes: confirming the probability value based on processing the word-level hidden variable and the context hidden variable by the pointer mechanism. The calculation formula is:

[0041]

[0042] Among them, is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, s t represents the hidden state of the decoder at the current moment; x t represents the decoder input at the current moment, and b ptr are all learnable parameters;

[0043] Based on the probability value, confirm the probability distribution of the extended vocabulary. The calculation formula is:

[0044]

[0045] Among them, p vocab represents the probability distribution of word w in the target vocabulary, represents the attention distribution value, and p(w) represents the probability distribution of word w calculated by combining the pointer.

[0046] Furthermore, it also includes: calculating the attention cumulative value based on the coverage mechanism. The calculation formula is:

[0047]

[0048] Among them, a t is the attention distribution value; c t is the attention cumulative value;

[0049] Obtain the calculation formula for updating the attention distribution based on the attention cumulative value:

[0050]

[0051] Among them, is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, x t represents the decoder input at the current moment, represents the context variable at time t, W h W c W s b atten and v T all represent learnable parameters.

[0052] In a second aspect, the present invention provides a computer device, including a processor and a memory. A computer program is stored in the memory. When the computer program is executed by the processor, the processor performs steps of a text summarization generation method based on deep learning, including: obtaining a sequence of word vectors, and inputting the sequence of word vectors into a word-level encoder to obtain word-level encoding; inputting the word-level encoding into a context extraction unit to obtain context information; inputting the context information into a context encoder to obtain context information encoding; performing a comprehensive decoding operation on the word-level encoding and the context information encoding to obtain a text summary..

[0053] In a third aspect, a storage medium stores a computer program. When the computer program is executed by a processor, the processor performs steps of a text summarization generation method based on deep learning; including: obtaining a sequence of word vectors, and inputting the sequence of word vectors into a word-level encoder to obtain word-level encoding; inputting the word-level encoding into a context extraction unit to obtain context information; inputting the context information into a context encoder to obtain context information encoding; performing a comprehensive decoding operation on the word-level encoding and the context information encoding to obtain a text summary.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] In this embodiment, a deep learning model including a word-level encoder, a context extraction unit, a context encoder, and a decoder is constructed to obtain a text summary according to a sequence of word vectors; wherein the word-level encoder is used to obtain word-level encoding according to the sequence of word vectors, the context extraction unit obtains context information through the word-level encoding, and the context encoder is used to obtain context information encoding according to the context information; the decoder simultaneously performs a comprehensive decoding on the word-level encoding and the context information encoding; so that the obtained text summary not only retains the text information of the source text, but also has good coherence. It solves the problems of poor sentence coherence and information errors in the text summarization generation method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the method step diagram of the embodiment of the present invention.

[0057] Figure 2 This is the structural schematic diagram of the deep learning model in another embodiment of the present invention. Specific Embodiments

[0058] The technical solutions in the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] Embodiment 1:

[0060] As Figure 1-2 shown, the first aspect of the present invention is a text summary generation method based on deep learning, and the method includes:

[0061] S1: Obtain a word vector sequence, and input the word vector sequence into a word-level encoder to obtain a word-level encoding;

[0062] The word vector sequence is X = (x1, x2,..., x n ), x i ∈R d ; The word-level encoding is

[0063] The word-level encoder includes a bidirectional gated recurrent unit GRU; a unidirectional working mechanism of the bidirectional gated recurrent unit GRU includes:

[0064] z t = σ(x t W xz + h t-1 W hz + b z );

[0065] r t = σ(x t W xr + h t-1 W hr + b r );

[0066]

[0067]

[0068] Among them, x t represents the output word vector at the current moment, h t-1 represents the hidden state at the previous moment, W xz , W hz , W xr , W hr , W xh , W hh , bz , b r and b h are all learnable parameters, z t represents the update gate; r t represents the reset gate, h t is the hidden state at the current time step.

[0069] S2: Input the word-level encoding into the context extraction unit to obtain context information;

[0070] The context extraction unit includes two extraction modules, and each extraction module includes a convolutional layer with a gated linear unit GLU; the convolutional layer includes 2d convolutional kernels of the same size.

[0071] S3: Input the context information into the context encoder to obtain the context information encoding;

[0072] The context encoder includes a bidirectional gated recurrent unit GRU, and the context encoder converts the context information into the context information encoding.

[0073] S4: Perform a comprehensive decoding operation on the word-level encoding and the context information encoding to obtain the text summary.

[0074] The decoder includes a gated recurrent unit GRU with an attention mechanism. Input the word-level encoding and the context information encoding into the decoder simultaneously to obtain the context variable unique to the time step, then further calculate the probability distribution of the current time step on the target vocabulary, and then obtain the text summary based on the probability distribution.

[0075] The specific implementation process of this embodiment includes:

[0076] In this embodiment, a deep learning model including a word-level encoder, a context extraction unit, a context encoder, and a decoder is constructed to obtain the text summary according to the word vector sequence; the word-level encoder is used to obtain the word-level encoding according to the word vector sequence, the context extraction unit obtains the context information through the word-level encoding, and the context encoder is used to obtain the context information encoding according to the context information; the decoder simultaneously performs comprehensive decoding on the word-level encoding and the context information encoding; so that the obtained text summary not only retains the text information of the source text but also has good coherence.

[0077] Before inputting the word vector sequence into the word-level encoder, it is necessary to obtain the word vector sequence according to the source text, including converting the source text into a text sequence W = (w1, w2,..., w n ), where the value of w i represents the index of the i-th word in the text sequence in the vocabulary; based on the Word2Vec word embedding model, confirm the text sequence W = (w1, w2,..., w n)The vector representation of each word is obtained, and the word vector sequence X = (x1, x2, …, x n ), x i ∈R d ; where R d represents a d-dimensional space.

[0078] In this embodiment, inputting the word vector sequence into a word-level encoder to obtain word-level encoding includes:

[0079] Processing the word vector sequence X = (x1, x2, …, x n ), x i ∈R d in both forward and reverse directions through a bidirectional gated recurrent unit GRU to obtain a forward hidden state sequence and a reverse hidden state sequence

[0080] Among them, the calculation formula for the hidden variable in the forward hidden state sequence is:

[0081]

[0082] The calculation formula for the hidden variable in the reverse hidden state sequence is:

[0083]

[0084] where f gru is the calculation function of the gated recurrent unit GRU.

[0085] Based on the hidden variables in both forward and reverse directions at the same moment, confirm the word-level encoding at that moment. The calculation formula is:

[0086]

[0087]

[0088] where is the word-level encoding at time t; g(·) is the calculation function of the gated linear unit GLU, σ(.) represents the sigmoid function, represents element-wise multiplication;

[0089] Obtain the word-level encoding

[0090] Encode the word vector sequence X = (x1, x2, …, x n ), x i ∈R d in both directions, which can better represent context-related sequences.

[0091] In this embodiment, inputting the word-level encoding into a context extraction unit to obtain context information includes:

[0092] Two extraction modules calculate the word-level encoding in sequence to obtain context information; the calculation formula for each extraction module is:

[0093] O = K i * H W + b i = [AB] ∈ R 2d ;

[0094]

[0095] Among them, * represents the convolution operation, and K i represents the i-th convolution kernel; σ(.) represents the sigmoid function. O represents the output after the convolution unit. A and B are the elements in O divided into two parts.

[0096] In this embodiment, the context information in the word-level encoding is extracted through a convolutional layer with a gated linear unit (GLU). The convolutional layer is used to regard multiple consecutive words as a whole for extraction. The perception range of the convolution kernel is limited to an area of the same size as itself; in this embodiment, two convolutional layers are stacked to increase the receptive field of the convolutional layer, so as to extract more context information for text summary generation.

[0097] In some other embodiments, a residual connection mechanism is also added to assist in the training of the deep learning model. The formula after the residual connection is:

[0098]

[0099] X is the word vector sequence, and Y represents the result after the residual mechanism.

[0100] In this embodiment, the context information is input into the context encoder to obtain the context information encoding. The calculation formula is:

[0101]

[0102]

[0103]

[0104] Among them, f gru is the calculation function of the gated recurrent unit (GRU); g(·) is the calculation function of the gated linear unit (GLU); is the overall output after connecting the output results of the word-level encoder and the context encoder.

[0105] In this embodiment, both the context encoder and the word-level encoder include bidirectional gated recurrent units (GRUs), so their calculation processes are similar.

[0106] Into the input decoder, the decoder performs decoding operations on the word-level encoding and context encoding based on the attention mechanism. It includes:

[0107] According to the processed context information encoding and decoder state, obtain the context variable unique to the time step, and the calculation formula is:

[0108]

[0109] a t = softmax(e t );

[0110]

[0111] Among them, is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, x t represents the decoder input at the current moment, represents the context variable at time t, W h , W c , W s , b atten and v T all represent learnable parameters, a t represents the attention distribution value;

[0112] Based on the context variable unique to the time step, confirm the probability distribution on the target vocabulary at the current time step, and the calculation formula is:

[0113]

[0114] s t represents the hidden state of the decoder at the current moment; represents the context variable at time t, V', V, b and b' are all learnable parameters, p vocab represents the probability distribution on the target vocabulary.

[0115] In this embodiment, the target vocabulary is generated according to the text sequence W = (w1, w2,..., w n ).

[0116] The decoder obtains the summary sequence based on the target vocabulary probability distribution p vocab and outputs it.

[0117] In another embodiment of the present invention, after obtaining the probability distribution p vocab , it further includes:

[0118] Processing the confirmation probability values of word-level hidden variables and context hidden variables based on a pointer mechanism, the calculation formula is:

[0119]

[0120] Among them, is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, x t represents the decoder input at the current moment, s t represents the hidden state of the decoder at the current moment; x t represents the decoder input at the current moment, and b ptr are all learnable parameters;

[0121] Based on the probability value, confirm the probability distribution of the target vocabulary, and the calculation formula is:

[0122]

[0123] Among them, p vocab represents the probability distribution of the word w in the target vocabulary, represents the attention distribution value, and p(w) represents the probability distribution of the word w calculated by combining the pointer.

[0124] Another embodiment of the present invention further includes:

[0125] Calculating the attention cumulative value based on the coverage mechanism, and the calculation formula is:

[0126]

[0127] Among them, a t is the attention distribution value; c t is the attention cumulative value.

[0128] Based on the attention cumulative value, update the calculation formula of the attention distribution, and obtain:

[0129]

[0130] Among them, is the word-level encoding, is the context information encoding, s t represents the hidden state of the decoder at the current moment, x t represents the decoder input at the current moment, represents the context variable at time t, W h 、W c 、W s 、b atten and v TBoth represent learnable parameters.

[0131] In this embodiment, a penalty term coverage is also defined in the deep learning model; its calculation formula is:

[0132]

[0133] where a t is the attention distribution value; c t is the attention accumulation value.

[0134] During the training process of the deep model, when the input text and summary pair (X, Y) are given; then the calculation formula of the loss function at time step t is:

[0135]

[0136] where λ is a fixed coefficient, with a range of [0, 1], and in this embodiment, λ is taken as 1.

[0137] Let the total time step be T, then the calculation formula of the overall loss function is:

[0138]

[0139] where L represents calculating the losses for all time steps.

[0140] It should be noted that in this embodiment, when the decoder generates the summary, the beam search method is adopted. After calculating all the time steps, multiple candidate summary sequences are obtained, and the optimal sequence is selected from the multiple candidate summary sequences for output.

[0141] In a second aspect, the present invention provides a computer device, including a processor and a memory. A computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to execute the steps of the text summary generation method based on deep learning, including: obtaining a word vector sequence, and inputting the word vector sequence into a word-level encoder to obtain a word-level encoding; inputting the word-level encoding into a context extraction unit to obtain context information; inputting the context information into a context encoder to obtain a context information encoding; performing a comprehensive decoding operation on the word-level encoding and the context information encoding to obtain a text summary.

[0142] In a third aspect, the present invention provides a storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of a method for generating a text summary based on deep learning, including: obtaining a sequence of word vectors, and inputting the sequence of word vectors into a word-level encoder to obtain word-level encodings; inputting the word-level encodings into a context extraction unit to obtain context information; inputting the context information into a context encoder to obtain context information encodings; and performing a comprehensive decoding operation on the word-level encodings and the context information encodings to obtain a text summary.

[0143] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0144] It should be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0145] Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements includes not only those elements but also other elements not expressly listed, or further includes elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

Claims

1. A method for generating text summaries based on deep learning, characterized in that: The method includes: Obtain a sequence of word vectors, and input the sequence of word vectors into a word-level encoder to obtain word-level encodings; Input the word-level encodings into a context extraction unit to obtain context information; Input the context information into a context encoder to obtain context information encodings; Perform a comprehensive decoding operation on the word-level encodings and the context information encodings to obtain a text summary; Among them, the comprehensive decoding operation on the word-level encodings and the context information encodings is performed by a decoder, and the decoder includes a gated recurrent unit GRU with an attention mechanism. Obtaining the text summary includes: According to the processed context information encodings and the decoder state, obtain a context variable unique to the time step, and the calculation formula is: ； ； ； Among them, is the word-level encoding, is the context information encoding, represents the hidden state of the decoder at the current moment, represents the decoder input at the current moment, represents the context variable at time t, 、 、 、 and all represent learnable parameters, represents the attention distribution value; Based on the context variable unique to the time step, confirm the probability distribution on the target vocabulary at the current time step, and the calculation formula is: ； represents the hidden state of the decoder at the current moment; represents the context variable at time t, , , and are all learnable parameters, represents the probability distribution over the target vocabulary; The decoder outputs after obtaining the summary sequence based on the probability distribution of the target vocabulary. ​ 2. The method for generating a text summary based on deep learning according to claim 1, wherein: Obtaining the sequence of word vectors includes: Obtain a text sequence , where the value represents the index of the i -th word of the text sequence in the vocabulary; Confirm the text sequence based on the Word2Vec word embedding model Obtain the word vector sequence based on the vector representation of each word 。 3. The method for generating a text summary based on deep learning according to claim 2, wherein: The word-level encoder includes a bidirectional gated recurrent unit GRU. Inputting the sequence of word vectors into the word-level encoder to obtain word-level encodings includes: Process the word vector sequence in both forward and backward directions through a bidirectional gated recurrent unit (GRU). Obtain the forward hidden state sequence and the backward hidden state sequence ; Based on the hidden variables in the forward and reverse directions at the same moment, confirm the word-level encoding at that moment, and the calculation formula is: ； ； Among them is the word-level encoding at time t; is the calculation function of the gated linear unit GLU, represents the sigmoid function, represents element-wise multiplication; Obtain word-level encoding .

4. The method for generating text summaries based on deep learning according to claim 3, wherein: The context extraction unit includes two extraction modules, and each extraction module includes a convolutional layer with a gated linear unit GLU; inputting the word-level encodings into the context extraction unit to obtain context information includes: Two extraction modules perform calculations on the word-level encoding in sequence to obtain context information; the calculation formula for each extraction module is: ； ； where * represents the convolution operation, denotes the i-th convolution kernel.

5. The method for generating text abstract based on deep learning according to claim 4, wherein: Input the context information into the context encoder to obtain context information encodings, and the calculation formula is: ； ； Among them, is the calculation function of the gated recurrent unit GRU; is the calculation function of the gated linear unit GLU.

6. The method for generating text abstract based on deep learning according to claim 1, characterized in that: It also includes: Based on a pointer mechanism, process the word-level hidden variable and the context hidden variable to confirm the probability value, and the calculation formula is: ； Among them, is the word-level encoding, is the context information encoding, represents the hidden state of the decoder at the current moment, represents the decoder input at the current moment, 、 、 、 and are all learnable parameters; Based on the probability value, confirm the probability distribution of the extended vocabulary, and the calculation formula is: ； Among them, represents the probability distribution of word w in the target vocabulary,[ represents the attention distribution value,[ represents the probability distribution of word w calculated by combining pointers.[ 7. The method for generating text summaries based on deep learning according to claim 1, characterized in that: It also includes: Calculate the attention accumulation value based on a coverage mechanism, and the calculation formula is: ； Among them, is the attention distribution value; is the attention accumulation value; Based on the attention accumulation value, update the calculation formula of the attention distribution to obtain: ； Among them, is the word-level encoding, is the context information encoding, represents the hidden state of the decoder at the current moment, represents the decoder input at the current moment, represents the context variable at time t, 、 、 、 and all represent learnable parameters.

8. A computer device, comprising a processor and a memory, wherein a computer program is stored in the memory, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the steps of the deep learning-based text summary generation method according to any one of claims 1-7.

9. A storage medium stores a computer program, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the steps of the deep learning-based text summary generation method according to any one of claims 1-7.

Images