Abstract automatic generation method, device, electronic device and storage medium

Abstract

The invention discloses an automatic abstract generation method, device, electronic device and storage medium. The method calculates the first character vector and the second character vector of each single word based on two trained embedding vector models respectively from the original text and the named entities in the original text, and obtains the word vector of each single word by splicing; the word vector of each single word is encoded and decoded by the trained Transformer encoding and decoding model to obtain the word vector of multiple generated words, which can enhance the feature representation ability of the word vector of the multiple generated words, and each generated word is divided into a first-class generated word or a second-class generated word. The first-class generated words and the second-class generated words are respectively calculated using a trained pointer network and a memory network to obtain first-class output words and second-class output words respectively. The target abstract is composed of multiple first-class output words and / or multiple second-class output words, which can effectively solve the problem that uncommon named entities cannot be generated.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method, device, electronic device and storage medium for automatically generating abstracts. Background Art

[0002] Automatic summarization refers to the process of automatically generating a short text that can summarize a piece of original text through an algorithm. Currently, automatic summarization algorithms are mainly divided into extractive automatic summarization and generative automatic summarization. Extractive automatic summarization refers to extracting one or more sentences with summarization ability from the original text as summaries on a sentence-by-sentence basis, but this method is difficult to adapt to the changing content of the original text. Generative automatic summarization generates an abstract by retelling and summarizing the central idea of ​​the original text by mining deeper semantic information. The content of the abstract is not limited to the sentence vocabulary in the original text, but can be more refined vocabulary and language organization outside the original text, and can be applied to various original texts.

[0003] At present, deep learning models based on Seq2Seq+Attention mechanism are usually used to realize generative automatic summarization, which is difficult to solve the problem that important but relatively rare named entities such as names of people, places, time or institution names in the original text cannot be generated in the abstract. In addition, Seq2Seq units generally use lightweight models such as LSTM and GRU to encode the original text, and the encoding itself is not expressive enough for the original text in the open field. In the automatic generation of abstracts, there is a problem that it is difficult to automatically generate rare and important named entities, and no effective solution has been proposed so far. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, one of the purposes of the present invention is to provide a method for automatic abstract generation, which uses a trained Transformer encoding and decoding model to encode and decode the word vector of each single word to obtain the word vectors of multiple generated words, which can enhance the feature representation ability of the word vectors of multiple generated words, and divide each generated word into a first-category generated word or a second-category generated word. The first-category generated word is calculated using a trained pointer network to obtain the first-category output word, and the second-category generated word is calculated using a trained memory network to obtain the second-category output word. This can effectively solve the problem that uncommon named entities cannot be generated and improve the accuracy of the abstract.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] According to the original text and the named entities in the original text, respectively, calculations are performed based on two trained embedding vector models to obtain a first character vector of each word in the original text and a second character vector of each word in the named entity, and the first character vector and the second character vector of each word are concatenated to obtain a word vector of each word;

[0007] According to the word vectors of each of the single words, encoding and decoding processing is performed based on the trained Transformer encoding and decoding model to obtain word vectors of multiple generated words;

[0008] Based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words;

[0009] A plurality of the first-category output words and / or a plurality of the second-category output words are combined into a target abstract, wherein the first-category output words are calculated based on the trained pointer network according to the word vectors of the first-category generated words, and the second-category output words are calculated based on the trained memory network according to the word vectors of the second-category generated words and the word vectors of each single word in the named entity.

[0010] Furthermore, before respectively calculating the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entity in the original text, the method further includes:

[0011] Get the original text;

[0012] The named entities in the original text are obtained by identifying the original text based on the Stanford NLP tool.

[0013] Furthermore, the trained Transformer codec model includes a position encoding layer, an encoder and a decoder, and the word vectors of the multiple generated words are obtained by performing codec processing based on the trained Transformer codec model according to the word vectors of each of the single words, including:

[0014] Inputting the word vector of each of the single words into the position encoding layer for position encoding to obtain the position vector of each of the single words;

[0015] The word vector and the position vector of each word are added to obtain the embedding vector of each word;

[0016] Inputting the embedding vector of each word into the encoder for encoding to obtain the encoding matrix of the original text;

[0017] According to the encoding matrix, multiple loop decoding is performed based on the decoder to obtain word vectors of multiple generated words, wherein each loop decoding is performed to obtain a word vector of a generated word.

[0018] Furthermore, the trained classification model includes a normalized exponential function, and based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words, including: inputting the word vector of each of the generated words into the following normalized exponential function to calculate and obtain the score vector of each of the generated words:

[0019] Score i =Softmax1((W pm V i +b pm )W c ),

[0020] Among them, V i represents the word vector of the i-th generated word, i=1,...,N, N represents the total number of generated words, W pm , b pm and W c is a trained parameter, and the score vector Score i The word vector of the i-th generated word selects the first score of the pointer network and the second score of the memory network. Softmax1(·) represents the normalized exponential function;

[0021] If the score vector Score i When the first score in is greater than or equal to the second score, the i-th generated word is a first-category generated word;

[0022] If the score vector Score i When the first score in is less than the second score, the i-th generated word is a second-category generated word.

[0023] Furthermore, the trained pointer network includes a linear transformation layer, a normalization layer and a probability distribution calculation function, and the first type of output words is obtained by:

[0024] Processing the word vectors of the first type of generated words through a linear transformation layer and a normalization layer to obtain a first distribution probability sequence, wherein the first distribution probability represents the probability that each word in the dictionary is a first type of generated word;

[0025] A distribution probability sequence of first-category generated words is obtained by calculating the first distribution probability sequence based on the following probability distribution calculation function, wherein the distribution probability represents the probability that each word in an extended dictionary is a first-category generated word, and the extended dictionary includes the dictionary and the original text:

[0026] P(w)=P g P vocab (w)+(1-P g ) Att (w),

[0027] Among them, P g represents the weight of the word vector of the first type of generated word calculated by the linear transformation layer, w represents the word vector of the first type of generated word, P vocab (w) represents the first distribution probability sequence, P Att (w) represents a second distribution probability sequence, wherein the second distribution probability represents the probability that each single word in the original text is a first type of generated word, and P(w) represents the distribution probability sequence;

[0028] According to the distribution probability with the largest value in the distribution probability sequence, the corresponding word in the extended dictionary is queried as the first category of output words.

[0029] Furthermore, the trained memory network includes a normalized exponential function, and the second type of output words are obtained by:

[0030] The word vectors of multiple words in the named entity are averaged to obtain an average vector;

[0031] Concatenating the average vector and the word vector of the second type of generated words to obtain a candidate word vector;

[0032] The candidate word vector is input into the following normalized exponential function to calculate the candidate word score sequence:

[0033] Score m =Softmax2(W m V+b m ),

[0034] Where V represents the candidate word vector, W m and b m is the trained conversion parameter, and the candidate word scoring sequence Score m including taking each single word in the named entity and the second type of generated words as the scores of the second type of output words respectively, and Softmax2(·) represents a normalized exponential function;

[0035] According to the score query with the largest value in the candidate word score sequence, the corresponding candidate word is obtained as the second category output word.

[0036] A second object of the present invention is to provide an automatic abstract generation device, which comprises:

[0037] A word vector extraction module is used to calculate the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entities in the original text, and concatenate the first character vector and the second character vector of each word to obtain the word vector of each word;

[0038] A coding and decoding module, used for performing coding and decoding processing on the word vectors of each of the single words based on a trained Transformer coding and decoding model to obtain word vectors of multiple generated words;

[0039] A classification module, used for classifying the multiple generated words into first-category generated words or second-category generated words based on the word vectors of the multiple generated words and the trained classification model;

[0040] An abstract generation module is used to combine multiple first-category output words and / or multiple second-category output words into a target abstract, wherein the first-category output words are calculated based on the trained pointer network according to the word vectors of the first-category generated words, and the second-category output words are calculated based on the trained memory network according to the word vectors of the second-category generated words and the word vectors of each single word in the named entity.

[0041] The third object of the present invention is to provide an electronic device for performing one of the objects of the invention, which includes a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, and when the computer program is executed by the processor, it is a method for automatically generating abstracts according to one of the objects of the present invention.

[0042] A fourth object of the present invention is to provide a computer-readable storage medium storing one of the objects of the invention, on which a computer program is stored, and when the computer program is executed by a processor, the method for automatically generating abstracts of one of the objects of the present invention is implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention uses two trained embedding vector models to make the word vector of each single word in the original text contain the named entity information and the character information of the original text, and then uses the trained Transformer encoding and decoding model to perform encoding and decoding processing to obtain the word vectors of multiple generated words, which can enhance the feature representation ability of the word vectors of multiple generated words. The word vector of each generated word is calculated based on the trained pointer network to obtain the first type of output word or based on the trained memory network to obtain the second type of output word. Finally, multiple first type output words and / or multiple second type output words are combined into a target abstract, which can effectively solve the problem that uncommon named entities cannot be generated and improve the accuracy of the abstract. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a method for automatically generating abstracts according to Embodiment 1 of the present invention;

[0046] Figure 2 is a structural block diagram of an automatic abstract generation device according to a fifth embodiment of the present invention;

[0047] Figure 3 This is a structural block diagram of an electronic device according to Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in more detail below in conjunction with the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is only illustrative and not restrictive. The various embodiments can be combined with each other to form other embodiments not shown in the following description.

[0049] Embodiment 1

[0050] Example 1 provides a method for automatically generating abstracts. Figure 1 As shown, the following steps are included:

[0051] S110, calculating based on two trained embedding vector models according to the original text and the named entities in the original text to obtain the first character vector of each word in the original text and the second character vector of each word in the named entity, and concatenating the first character vector and the second character vector of each word to obtain the word vector of each word.

[0052] Named entity information is very important. Named entity words such as names of people, places, time or organization names in the original text are very likely to appear in the corresponding abstract. Therefore, by concatenating the first character vector and the second character vector of each word, the word vector of each word in the original text contains the named entity information and the character information of the original text.

[0053] The two trained embedding vector models can be any one of the word2vec model, GloVe (Global Vectors for Word Representation) model, ELMo (Embedding from Language Models) model, and BERT model. The word2vec model obtains the vector representation of a word through the context of the word, but the word2vec model only considers the local information of the word and does not consider the connection between the word and the words outside the local window. The GloVe model uses the co-occurrence matrix to consider the local information and the overall information of the word at the same time, so that the vectors contain as much semantic and grammatical information as possible. The word2vec model and the GloVe model obtain a word vector for each word, which cannot express the different meanings of the same word in different contexts. The ELMo model can learn the complex characteristics of word usage and the changes of complex features in different contexts. The BERT (Bidirectional Encoder Representations from Transformers) model is a deep bidirectional pre-trained language understanding model that uses the Transformer model as a feature extractor. In essence, it learns a feature representation that can have global information for a word by running a self-supervised learning method based on a massive corpus. In this embodiment, the two trained embedding vector models both use the BRET model.

[0054] S120, performing encoding and decoding processing based on the word vector of each single word based on the trained Transformer encoding and decoding model to obtain the word vectors of multiple generated words.

[0055] The Transformer model was proposed in the paper "Attention is All You Need". It uses the self-attention mechanism to achieve fast parallel training. It consists of two parts: an encoder and a decoder. The encoder and decoder contain a preset number of encoder modules and decoder modules respectively.

[0056] The encoder module includes a multi-head attention layer, which is composed of multiple self-attention heads stacked together, and each self-attention head is a self-attention layer. The multi-head attention layer embodies multiple representation subspaces. Through the calculation of the multi-head attention layer, the word vector of each word is projected into different representation subspaces, which can expand the feature annotation space and enhance the feature representation ability. The output of the word vector of each word in each attention head can be obtained by the following self-attention score calculation formula:

[0057]

[0058] head i =Attention(QWi Q ,KW i K ,VW i V ), (2)

[0059] Among them, Q, K and V are the same, which are all matrices composed of word vectors of each word in the original text. Each row in Q, K and V corresponds to the word vector of a single word. k is the dimension of K, QK T The similarity is calculated by performing the inner product between the vectors of the words in the matrix, which reflects the relevance and importance between the words to a certain extent. i Q , W i K and W i V are all parameter matrices, i=1,...,h, where h represents the number of self-attention heads. i represents the output of the i-th self-attention head. For each self-attention head, Q, K, and V are respectively associated with different parameter matrices W i Q , W i K and W i V Multiplying them together will make the word vector of each word calculated based on multiple self-attention heads produce different outputs.

[0060] The output of each attention head in the multi-head attention layer can be calculated by the following formula:

[0061] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W o , (3)

[0062] Among them, W o is a parameter matrix. The output of each attention head is concatenated with W o The multiplication is again transformed into a vector output.

[0063] The output of the multi-head attention layer is added to the matrix consisting of the word vectors of each word in the original text to achieve residual propagation, and normalized using the following batch normalization formula:

[0064]

[0065] BN(x)=αx'+β, (5)

[0066] Where x is the matrix obtained by adding the output of the multi-head attention layer to the matrix composed of the word vectors of each word in the original text, mean(x) and var(x) represent the mean matrix and standard deviation matrix corresponding to the matrix x, respectively, and α and β are trained parameters.

[0067] The matrix obtained by the above normalization process is then forward propagated, residual connected and normalized to obtain the output of the encoder module. That is, the output of the multi-head attention layer is calculated by the activation function, added to the matrix composed of the word vectors of each word in the original text, and then normalized to obtain the output of the encoder module:

[0068] Output=BN(X+Relu(WX+b)), (6)

[0069] Where X represents the matrix obtained by the above normalization process, W and b are both trained parameters, Relu(·) represents the ReLU activation function, and BN(·) is the normalization formula (5).

[0070] Each decoder module in the decoder has the same structure, and each decoder module is similar to the encoder module structure as described above. The only difference between the two is that there is an additional multi-head attention layer connected to the previous encoder module. Considering that the previous word in the original text should not have a relationship with the following word in advance, this multi-head attention layer uses the [mask] symbol to cover half of multiple words in the self-attention process, so that the previous word in the original text cannot be calculated with the following word through the self-attention score calculation formula. After the original text is shifted right by one position, the word vector of each word in the original text shifted right by one position is calculated based on two trained embedding vector models, that is, the input of each decoding of the decoder is the vector of the previous word in the original text or its encoding, that is, Q in the self-attention score calculation formula. At the same time, K and V in the self-attention score calculation formula are not equal to Q, but are matrices processed by a preset number of encoder modules with the above structure. After multiple cycles of decoding, the word vectors of multiple generated words can be obtained.

[0071] As an optional technical solution, the encoder module and the decoder module can both be one of the LSTM (Long Short-Term Memory) model, the RNN (Recurrent Neural Network) model or the GRU (Gated RNN) model.

[0072] S130, classifying the multiple generated words into first-category generated words or second-category generated words based on the word vectors of the multiple generated words and the trained classification model.

[0073] S140. Combining multiple first-category output words and / or multiple second-category output words into a target abstract, wherein the first-category output words are calculated based on the word vectors of the first-category generated words based on the trained pointer network, and the second-category output words are calculated based on the word vectors of the second-category generated words and the word vectors of each single word in the named entity based on the trained memory network.

[0074] Step S130 enables each generated word to be classified during the abstract output process, which is equivalent to using a switch to control the use of a trained pointer network or a trained memory network to perform calculations to obtain corresponding output words.

[0075] Pointer Networks comes from an article published in NIPS 2015, a top machine learning conference. It can overcome the problem of "output heavily dependent on input" in seq2seq models. Based on the word vector of each first-class generated word, a first-class output word is calculated based on the trained pointer network, so that the distribution probability of the first-class generated word can take into account the words in the dictionary and the original text at the same time, thus avoiding the situation where the named entities of uncommon and unpopular words such as names of people and places cannot be generated because they cannot be enumerated through the dictionary.

[0076] Memory Networks was proposed by Facebook AI in 2015. It uses memory components to save scene information to achieve long-term memory. Based on the word vector of each second-category generated word and the word vector of each word in the named entity, a second-category output word is calculated based on the trained memory network. The memory component can store named entities to solve the problem of named entities not being able to be generated, and can also store second-category generated words to be suitable for the generation of medium and long abstracts.

[0077] This method uses two trained embedding vector models to make the word vector of each single word in the original text contain the named entity information and the character information of the original text, and then uses the trained Transformer encoding and decoding model to encode and decode the word vector of each single word to obtain the word vector of multiple generated words, which can enhance the feature representation ability of the word vector of multiple generated words. The word vector of each generated word is calculated based on the trained pointer network to obtain the first type of output word or based on the trained memory network to obtain the second type of output word. Finally, multiple first type output words and / or multiple second type output words are combined into a target abstract, which can effectively solve the problem that uncommon named entities cannot be generated and improve the accuracy of the abstract.

[0078] In some embodiments, before respectively calculating the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entities in the original text, it also includes: obtaining the original text; identifying the named entities in the original text based on the Stanford NLP tool according to the original text, so that the named entities in the original text can be accurately identified.

[0079] As an optional technical solution, the original text can be identified based on a trained entity recognition model to obtain named entities in the original text. The trained entity recognition model is not limited to a convolutional neural network model or a hybrid neural network model.

[0080] Embodiment 2

[0081] Embodiment 2 is an improvement on embodiment 1. The trained Transformer encoding and decoding model includes a position encoding layer, an encoder and a decoder. Since the Transformer model relies entirely on the attention mechanism for encoding and decoding, and does not rely on the sequence model like the traditional method, the temporal feature cannot be reflected from the input perspective of the model. Therefore, adding the position vector obtained by the position encoding layer of each word vector to the embedding vector of each word helps to determine the position of each word, or the distance between each word in the sequence, and thus better express the relationship between words. Specifically comprising the following steps:

[0082] The word vector of each single word is input into the position encoding layer for position encoding to obtain the position vector of each single word; the word vector and the position vector of each single word are added to obtain the embedding vector of each single word; the embedding vector of each single word is input into the encoder for encoding processing to obtain the encoding matrix of the original text; according to the encoding matrix, multiple loop decoding is performed based on the decoder to obtain the word vectors of multiple generated words respectively, and each loop decoding is performed to obtain the word vector of a generated word.

[0083] Specifically, the word vector of each word is input into the following position encoding formula in the position encoding layer to obtain the position vector of each word:

[0084]

[0085]

[0086] Among them, pos represents the position of a word in the original text, pos = 0,...,L-1, L represents the length of the original text, d model Indicates the preset encoding dimension, i = 0,...,d model-1. According to the above position encoding formula, the matrix composed of the word vectors of each single word can be calculated once. Because trigonometric functions (sin function and cos function) are used, it is suitable for position encoding of original text of any length.

[0087] In some embodiments, the trained classification model includes a normalized exponential function, and the word vector of each generated word is input into the following normalized exponential function to calculate the score vector of each generated word:

[0088] Score i =Softmax1((W pm V i +b pm )W c ), (9)

[0089] Among them, V i represents the word vector of the i-th generated word, i=1,...,N, N represents the total number of generated words, W pm , b pm and W c It is a trained parameter used to convert the word vector of each generated word into a score vector of preset dimension (such as 1*2 dimension). i The word vector of the i-th generated word selects the first score of the pointer network and the second score of the memory network. Softmax1(·) represents the normalized exponential function.

[0090] If the score vector Score i When the first score in is greater than or equal to the second score, the i-th generated word is a first-class generated word; if the score vector Score i When the first score in is less than the second score, the i-th generated word is a second-category generated word. Each generated word is divided into a first-category generated word or a second-category generated word, so that each generated word can choose a trained pointer network or a trained memory network for calculation, thereby obtaining a more accurate output word.

[0091] Embodiment 3

[0092] Embodiment 3 is an improvement on embodiment 2. The pointer network uses a probability distribution calculation function to connect the decoder output and the encoder output in the trained Transformer encoding and decoding model. The pointer mechanism solves the problem that named entities that do not appear in the dictionary cannot be generated, and has the ability to effectively handle long-distance dependencies and parallel computing characteristics.

[0093] The trained pointer network includes a linear transformation layer, a normalization layer, and a probability distribution calculation function. The linear transformation layer is a simple fully connected neural network that projects the word vector generated by the decoder into a much larger vector called logits. Assuming that we learn 10,000 different English words in the dictionary from the training set, the log-odds vector is a vector of 10,000 cells in length—each cell corresponds to the score of a certain word. The next normalization layer will convert these scores into probabilities (all positive numbers with an upper limit of 1.0).

[0094] The first type of output words are obtained by calculating based on the word vectors of the first type of generated words based on the trained pointer network, including the following steps: the word vectors of the first type of generated words are processed through a linear transformation layer and a normalization layer to obtain a first distribution probability sequence, and the first distribution probability represents the probability that each word in the dictionary is a first type of generated word. The distribution probability sequence of the first type of generated words is calculated based on the following probability distribution calculation function according to the first distribution probability sequence, and the distribution probability represents the probability that each word in the extended dictionary is a first type of generated word, and the extended dictionary includes the dictionary and the original text.

[0095] P(w)=P g P vocab (w)+(1-P g ) Att (w), (10)

[0096] Among them, P g represents the weight of the word vector of the first type of generated word calculated by the linear transformation layer, w represents the word vector of the first type of generated word, P vocab (w) represents the first distribution probability sequence, P Att (w) represents the second distribution probability sequence, which represents the probability that each word in the original text is a first-class generated word, and plays the role of a pointer "pointing" to the word in the input original text. P(w) represents the distribution probability sequence. g As a soft switch of the pointer network to control whether the source of the first type of output word is the dictionary or the original text, when the word w is a word in the original text but not in the dictionary, P g is 0; when word w does not appear in the original text, P g is 1, making it impossible for the pointer network to find this word from the original text and copy it.

[0097] The corresponding word in the expanded dictionary is queried according to the distribution probability with the largest median value in the distribution probability sequence as the first type of output word. The first type of output word can be a word copied from the dictionary through the first type of generated words, or a word copied from the original text through the pointer mechanism, thereby solving the problem of generating named entities that do not appear in the dictionary.

[0098] Embodiment 4

[0099] Embodiment 4 is an improvement based on one or more of Embodiments 1 to 3. The trained memory network includes a normalized exponential function. The positions corresponding to the word vectors of multiple words in the named entity are averaged to obtain an average vector. The average vector and the word vector of the second type of generated word are concatenated to obtain a candidate word vector. The candidate word vector is input into the following normalized exponential function to calculate a candidate word score sequence:

[0100] Score m =Softmax2(W m V+b m ), (11)

[0101] Where V represents the candidate word vector, V∈[candidate_word_num,hidden_dim], where candidate_word_num represents the number of candidate words, and the candidate words are multiple single words or second-class generated words in the named entity, hidden_dim represents the vector dimension, and W m and b m is the trained conversion parameter, and the candidate word score sequence Score m Including taking each candidate word as the score of the second type of output word, Softmax2(·) represents the normalized exponential function.

[0102] The corresponding candidate word is obtained as the second type of output word by querying the score with the largest median value in the candidate word score sequence. The named entities in the original text are added to the memory network to solve the problem that named entities that do not appear in the dictionary cannot be generated, and the second type of generated words are continuously added to the memory network to be suitable for the generation of medium and long abstracts.

[0103] Embodiment 5

[0104] Embodiment 5 discloses an automatic abstract generation device corresponding to the above embodiment, which is a virtual device structure of the above embodiment. Figure 2 As shown, including:

[0105] A word vector extraction module 210 is used to calculate the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entities in the original text, and concatenate the first character vector and the second character vector of each word to obtain the word vector of each word;

[0106] The encoding and decoding module 220 is used to perform encoding and decoding processing based on the word vectors of each of the single words based on the trained Transformer encoding and decoding model to obtain word vectors of multiple generated words;

[0107] A classification module 230, configured to classify the plurality of generated words into first-category generated words or second-category generated words based on the word vectors of the plurality of generated words and the trained classification model;

[0108] The abstract generation module 240 is used to combine multiple first-category output words and / or multiple second-category output words into a target abstract, wherein the first-category output words are calculated based on the trained pointer network according to the word vectors of the first-category generated words, and the second-category output words are calculated based on the trained memory network according to the word vectors of the second-category generated words and the word vectors of each single word in the named entity.

[0109] The automatic abstract generation device also includes a named entity recognition module 200 for acquiring original text; and identifying the named entities in the original text based on the Stanford NLP tool.

[0110] Embodiment 6

[0111] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention is shown in FIG. Figure 3 As shown, the electronic device includes a processor 310, a memory 320, an input device 330 and an output device 340; the number of processors 310 in the computer device can be one or more. Figure 3 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330 and the output device 340 in the electronic device can be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0112] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the automatic abstract generation method in the embodiment of the present invention (for example, the named entity recognition module 200, the word vector extraction module 210, the encoding and decoding module 220, the classification module 230 and the abstract generation module 240 in the automatic abstract generation device). The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, the automatic abstract generation method of the above-mentioned embodiments 1 to 4 is realized.

[0113] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The input device 330 may be used to receive original text, etc. The output device 340 may include a display device such as a display screen.

[0115] Embodiment 7

[0116] Embodiment 7 of the present invention further provides a storage medium including computer executable instructions, wherein the computer executable instructions are used to execute a method for automatically generating abstracts when executed by a computer processor, the method comprising:

[0117] According to the original text and the named entities in the original text, respectively, calculations are performed based on two trained embedding vector models to obtain a first character vector of each word in the original text and a second character vector of each word in the named entity, and the first character vector and the second character vector of each word are concatenated to obtain a word vector of each word;

[0118] According to the word vectors of each of the single words, encoding and decoding processing is performed based on the trained Transformer encoding and decoding model to obtain word vectors of multiple generated words;

[0119] Based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words;

[0120] A plurality of the first-category output words and / or a plurality of the second-category output words are combined into a target abstract, wherein the first-category output words are calculated based on the trained pointer network according to the word vectors of the first-category generated words, and the second-category output words are calculated based on the trained memory network according to the word vectors of the second-category generated words and the word vectors of each single word in the named entity.

[0121] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the automatic abstract generation method provided by any embodiment of the present invention.

[0122] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a mobile phone, a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0123] It is worth noting that in the above-mentioned embodiment of the automatic abstract generation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0124] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A method for automatically generating abstracts, characterized in that: The following steps are involved: According to the original text and the named entities in the original text, respectively, calculations are performed based on two trained embedding vector models to obtain a first character vector of each word in the original text and a second character vector of each word in the named entity, and the first character vector and the second character vector of each word are concatenated to obtain a word vector of each word; According to the word vectors of each of the single words, encoding and decoding processing is performed based on the trained Transformer encoding and decoding model to obtain word vectors of multiple generated words; Based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words; Combining a plurality of first-category output words and / or a plurality of second-category output words into a target abstract, wherein the first-category output words are calculated based on the word vectors of the first-category generated words based on the trained pointer network, and the second-category output words are calculated based on the word vectors of the second-category generated words and the word vectors of each word in the named entity based on the trained memory network; The trained classification model includes a normalized index function, and based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words, including: inputting the word vector of each of the generated words into the following normalized index function to calculate and obtain the score vector of each of the generated words: Score i =Soft max 1((W pm V i +b pm )W c ), Among them, V i represents the word vector of the i-th generated word, i=1,...,N, N represents the total number of generated words, W pm , b pm and W c is a trained parameter, the score vector Score i The word vector of the i-th generated word selects the first score of the pointer network and the second score of the memory network. Soft max 1(·) represents the normalized exponential function; If the score vector Score i When the first score in is greater than or equal to the second score, the i-th generated word is a first-category generated word; If the score vector Score i When the first score in is less than the second score, the i-th generated word is a second-category generated word; The trained pointer network includes a linear transformation layer, a normalization layer and a probability distribution calculation function, and the first type of output words is obtained by: Processing the word vectors of the first type of generated words through a linear transformation layer and a normalization layer to obtain a first distribution probability sequence, wherein the first distribution probability represents the probability that each word in the dictionary is a first type of generated word; A distribution probability sequence of first-category generated words is obtained by calculating the first distribution probability sequence based on the following probability distribution calculation function, wherein the distribution probability represents the probability that each word in an extended dictionary is a first-category generated word, and the extended dictionary includes the dictionary and the original text: P(w)=P g P vocab (w)+(1-P g )P Att (w), Among them, P g represents the weight of the word vector of the first type of generated word calculated by the linear transformation layer, w represents the word vector of the first type of generated word, P vocab (w) represents the first distribution probability sequence, P Att (w) represents a second distribution probability sequence, wherein the second distribution probability represents the probability that each single word in the original text is a first type of generated word, and P(w) represents the distribution probability sequence; According to the distribution probability with the largest value in the distribution probability sequence, the corresponding word in the extended dictionary is searched as the first type of output word; The trained memory network includes a normalized exponential function, and the second type of output words are obtained by: The word vectors of multiple words in the named entity are averaged to obtain an average vector; Concatenating the average vector and the word vector of the second type of generated words to obtain a candidate word vector; The candidate word vector is input into the following normalized exponential function to calculate the candidate word score sequence: Score m =Soft max 2(W m V+b m ), Where V represents the candidate word vector, W m and b m is the trained conversion parameter, and the candidate word scoring sequence Score m including taking each single word in the named entity and the second type of generated words as the scores of the second type of output words, respectively, and Soft max 2(·) represents a normalized exponential function; According to the score query with the largest value in the candidate word score sequence, the corresponding candidate word is obtained as the second category output word.

2. The method for automatically generating abstracts according to claim 1, characterized in that: Before respectively calculating the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entity in the original text, the method further includes: Get the original text; The named entities in the original text are obtained by identifying the original text based on the Stanford NLP tool.

3. The method for automatically generating abstracts according to claim 1, wherein: The trained Transformer codec model includes a position encoding layer, an encoder and a decoder, and performs encoding and decoding processing based on the trained Transformer codec model according to the word vectors of each word to obtain word vectors of multiple generated words, including: Inputting the word vector of each of the single words into the position encoding layer for position encoding to obtain the position vector of each of the single words; The word vector and the position vector of each word are added to obtain the embedding vector of each word; Inputting the embedding vector of each word into the encoder for encoding to obtain the encoding matrix of the original text; According to the encoding matrix, multiple loop decoding is performed based on the decoder to obtain word vectors of multiple generated words, wherein each loop decoding is performed to obtain a word vector of a generated word.

4. An automatic abstract generation device, characterized in that: It includes: A word vector extraction module is used to calculate the first character vector of each word in the original text and the second character vector of each word in the named entity based on two trained embedding vector models according to the original text and the named entities in the original text, and concatenate the first character vector and the second character vector of each word to obtain the word vector of each word; A coding and decoding module, used for performing coding and decoding processing on the word vectors of each of the single words based on a trained Transformer coding and decoding model to obtain word vectors of multiple generated words; A classification module, used for classifying the multiple generated words into first-category generated words or second-category generated words based on the word vectors of the multiple generated words and the trained classification model; An abstract generation module, used for forming a target abstract from a plurality of first-category output words and / or a plurality of second-category output words, wherein the first-category output words are calculated based on the trained pointer network according to the word vectors of the first-category generated words, and the second-category output words are calculated based on the trained memory network according to the word vectors of the second-category generated words and the word vectors of each single word in the named entity; The trained classification model includes a normalized index function, and based on the word vectors of the multiple generated words and the trained classification model, the multiple generated words are respectively classified into the first category of generated words or the second category of generated words, including: inputting the word vector of each of the generated words into the following normalized index function to calculate and obtain the score vector of each of the generated words: Score i =Soft max 1((W pm V i +b pm )W c ), Among them, V i represents the word vector of the i-th generated word, i=1,...,N, N represents the total number of generated words, W pm , b pm and W c is a trained parameter, the score vector Score i The word vector of the i-th generated word selects the first score of the pointer network and the second score of the memory network. Soft max 1(·) represents the normalized exponential function; If the score vector Score i When the first score in is greater than or equal to the second score, the i-th generated word is a first-category generated word; If the score vector Score i When the first score in is less than the second score, the i-th generated word is a second-category generated word; The trained pointer network includes a linear transformation layer, a normalization layer and a probability distribution calculation function, and the first type of output words is obtained by: Processing the word vectors of the first type of generated words through a linear transformation layer and a normalization layer to obtain a first distribution probability sequence, wherein the first distribution probability represents the probability that each word in the dictionary is a first type of generated word; A distribution probability sequence of first-category generated words is obtained by calculating the first distribution probability sequence based on the following probability distribution calculation function, wherein the distribution probability represents the probability that each word in an extended dictionary is a first-category generated word, and the extended dictionary includes the dictionary and the original text: P(w)=P g P vocab (w)+(1-P g )P Att (w), Among them, P g represents the weight of the word vector of the first type of generated word calculated by the linear transformation layer, w represents the word vector of the first type of generated word, P vocab (w) represents the first distribution probability sequence, P Att (w) represents a second distribution probability sequence, wherein the second distribution probability represents the probability that each single word in the original text is a first type of generated word, and P(w) represents the distribution probability sequence; According to the distribution probability with the largest value in the distribution probability sequence, the corresponding word in the extended dictionary is searched as the first type of output word; The trained memory network includes a normalized exponential function, and the second type of output words are obtained by: The word vectors of multiple words in the named entity are averaged to obtain an average vector; Concatenating the average vector and the word vector of the second type of generated words to obtain a candidate word vector; The candidate word vector is input into the following normalized exponential function to calculate the candidate word score sequence: Score m =Soft max 2(WmV+b m ), Where V represents the candidate word vector, W m and b m is the trained conversion parameter, and the candidate word scoring sequence Score m including taking each single word in the named entity and the second type of generated words as the scores of the second type of output words, respectively, and Soft max 2(·) represents a normalized exponential function; According to the score query with the largest value in the candidate word score sequence, the corresponding candidate word is obtained as the second category output word.

5. An electronic device comprising a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, wherein: When the computer program is executed by a processor, the method for automatically generating abstracts according to any one of claims 1 to 3 is performed.

6. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically generating abstracts according to any one of claims 1 to 3 is implemented.

Images