Abstract generation model training method, device, electronic device and non-transitory computer-readable storage medium

Abstract

The present invention provides a summary generation model training method, device, electronic device and non-transitory computer-readable storage medium. The summary generation model training method comprises the following steps: obtaining a first vector group, wherein the vectors in the first vector group are original encoding vectors that have been trained; generating a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined according to one or more vectors in the first vector group; using the vectors included in the first vector group and the second vector group as input encoding vectors, performing model training to obtain a summary generation model. The embodiment of the present invention can increase the length of the input text, thereby facilitating the generation of summaries of long texts.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a summary generation model training method, device, electronic device and non-transient computer-readable storage medium. Background Art

[0002] Abstract generation refers to the method of generating an abstract based on a text of a certain length. Existing abstract generation methods can be trained through deep learning to obtain the abstract generation model required for generating abstracts. However, the input of existing deep learning models has a certain word limit, which makes it difficult to generate abstracts for long texts. Summary of the invention

[0003] The present invention provides a summary generation model training method, device, electronic device and non-transient computer-readable storage medium to reduce the difficulty of generating summaries of long texts.

[0004] In a first aspect, an embodiment of the present invention provides a summary generation model training method, comprising the following steps:

[0005] Obtaining a first vector group, wherein the vectors in the first vector group are trained original encoding vectors;

[0006] Generate a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined according to one or more vectors in the first vector group;

[0007] The vectors included in the first vector group and the second vector group are used as input coding vectors to perform model training to obtain a summary generation model, wherein the summary generation model is a neural network model that uses the corresponding vector of the text as the input coding vector and uses the summary of the text as the output.

[0008] Optionally, the first vector group includes N vectors, and the second vector group includes N*N vectors, where N is a positive integer.

[0009] Optionally, generating a second vector group based on the first vector group includes:

[0010] The second vector group is determined by the following formula:

[0011] q i×n+j =αp (Imax+1-i) +(1-α)p j ;

[0012] Among them, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p jis a vector in the first vector group, Imax is the maximum position number value of the original encoding vector, α is a preset coefficient, 0<α<1, and α is not equal to 0.5.

[0013] Optionally, the using the vectors included in the first vector group and the second vector group as input encoding vectors to perform model training to obtain a summary generation model includes:

[0014] When the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold, the vectors included in the first vector group are used as input encoding vectors to perform model training;

[0015] When the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold, the vectors included in the first vector group and the vectors included in the second vector group are sequentially arranged as input encoding vectors for model training.

[0016] In a second aspect, an embodiment of the present invention provides a summary generation model training device, comprising:

[0017] An acquisition module, used to acquire a first vector group, wherein the vectors in the first vector group are trained original encoding vectors;

[0018] A generating module, configured to generate a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined based on one or more vectors in the first vector group;

[0019] A training module is used to use the vectors included in the first vector group and the second vector group as input coding vectors to perform model training to obtain a summary generation model, wherein the summary generation model is a neural network model that uses the corresponding vector of the text as the input coding vector and uses the summary of the text as the output.

[0020] Optionally, the first vector group includes N vectors, and the second vector group includes N*N vectors, where N is a positive integer.

[0021] Optionally, the generating module is specifically used for:

[0022] The second vector group is determined by the following formula:

[0023] q i×n+j =αp (Imax+1-i) +(1-α)p j ;

[0024] Among them, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p jis a vector in the first vector group, Imax is the maximum position number value of the original encoding vector, α is a preset coefficient, 0<α<1, and α is not equal to 0.5.

[0025] Optionally, the training module includes:

[0026] A first training submodule is used for performing model training using the vectors included in the first vector group as input encoding vectors when the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold;

[0027] The second training submodule is used to sequentially arrange the vectors included in the first vector group and the vectors included in the second vector group as input coding vectors for model training when the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold.

[0028] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory. When the processor executes a program stored in the memory, the processor implements the steps of any summary generation model training method described in the first aspect.

[0029] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the summary generation model training method as described in any one of the first aspects.

[0030] The summary generation model training method of the embodiment of the present invention includes the following steps: obtaining a first vector group, wherein the vectors in the first vector group are original coding vectors that have been trained; generating a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined according to one or more vectors in the first vector group; using the vectors included in the first vector group and the second vector group as input coding vectors, and performing model training to obtain a summary generation model. The embodiment of the present invention generates the second vector group through the trained original coding vectors. Since the second vector group is generated based on the first vector group, it is compatible with the first vector group. Furthermore, by performing model training through the first vector group and the second vector group, the length of the input text can be increased, thereby facilitating the generation of summaries of long texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0032] Figure 1 is a schematic diagram of word example correspondence in one embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of an intention matrix in one embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of another intention matrix in one embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of another intention matrix in one embodiment of the present invention;

[0036] Figure 5 is a flow chart of a summary generation model training method in one embodiment of the present invention;

[0037] Figure 6 is a coding schematic diagram in an embodiment of the present invention;

[0038] Figure 7 It is a structural diagram of a summary generation model training device in one embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The invention provides a summary generation model training method.

[0041] The technical solution of this embodiment is used to obtain a summary generation model through model training. The summary generation model is a neural network model that uses a text vector as an input encoding vector and a summary of the text as an output. The summary generation model is used to generate a summary of the text, especially a summary of a long text.

[0042] In the technical solution of this embodiment, the architecture of the summary generation model can be summarized as including a feature extraction sub-model and a summary generation sub-model. During implementation, a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model is used as a feature extraction sub-model to extract the feature representation of the original text, and further based on the feature, a summary of the text is generated by a summary generation sub-model such as UNILM (Unified pre-trained Language Model) or Seq2Seq (a neural network model).

[0043] In the technical solution of this embodiment, training data is first obtained and saved in a dictionary format. Exemplarily, in one embodiment, the format is data: {"summarization":"A", "article":"B"}, wherein summarization is the identifier of the summary, A represents the summary of the original text B, article is the identifier of the original text, and B is the original text for which a summary needs to be generated.

[0044] Seq2seq is a one-way language model for text generation, which uses conditional probability distribution to learn and train the expected knowledge. The formula for conditional probability is: p(x1, x2, x3, ..., xn) = p(x1)p(x2|x1)p(x3|x1, x2)...p(xn|x1, ..., xn-1).

[0045] In the formula, xi (i = 1, 2, 3, ... N) represents the prediction results of the i-th word arranged in sequence in the original text. When predicting x1, there is no external input, and when predicting x2, only x1 can be input. When predicting x3, only x1 and x2 can be input, and so on. That is, the model is unidirectional.

[0046] like Figure 1 As shown, in this embodiment, x1 corresponds to heaven, x2 corresponds to air, and so on. <s>As the starting mark, <e>It is the termination sign. When predicting each token, it only depends on the token in front of it.

[0047] UNILM directly treats Seq2Seq as sentence completion. The method is to mask the Attention matrix. As shown in Figure 2, each row of the Attention matrix actually represents the output, and each column represents the input, and the Attention matrix represents the association between the output and the input. Assuming that the white squares all represent 0, then the first row means that x1 is only related to the start tag. <s>The second line indicates that x2 is only related to the start mark <s>It is related to x2, and so on. Therefore, we only need to introduce the lower triangular mask into the Attention matrix of the Bert model and train the input and output by one position to realize the unidirectional language model.

[0048] For example, if the input is "I really want to eat the food from my hometown" and the target sentence is "I want to eat", then UNILM will combine these two sentences into one: [CLS] I really want to eat the food from my hometown [SEP] I want to eat [SEP]. After this transformation, the simplest solution is to train a language model, such as Figure 3 As shown, then enter "[CLS] I especially want to eat the food from my hometown [SEP]" to predict "I want to eat" word by word until "[SEP]" appears.

[0049] like Figure 4 As shown in the figure, in the original example, "I really want to eat the food from my hometown" is also included in the prediction range, which results in the Attention of the original sentence being unidirectional, that is, the Mask matrix of the corresponding part is lower triangular. However, the model only needs to predict "I want to eat", so in this embodiment, the Mask of the original sentence is further removed to obtain a more appropriate Mask.

[0050] like Figure 4 As shown in the figure, the attention of the input part is bidirectional, and the attention of the output part is unidirectional, which meets the requirements of Seq2Seq and achieves simple constraints. Thus, the idea of ​​using a single Bert model to complete the Seq2Seq task is realized. Just add the mask of the above shape without modifying the model architecture, and you can also directly use the Bert MLM (Masked Language Model) pre-trained weights, which converges faster.

[0051] The Bert model used in this embodiment is a NLP (natural language processing) pre-training model proposed by Google, which can be applied to the processing of language understanding models. However, the Bert model uses an absolute position encoding vector trained from random initialization, and the general maximum position number value is set to 512, so it can only process 512 tokens (word examples) at most. Therefore, it is impossible to directly process long texts based on the Bert model, but only through truncation or extraction methods, which may cause part of the text to be lost, resulting in poor results for text summary generation.

[0052] In one embodiment, Figure 5 As shown, the summary generation model training method includes the following steps:

[0053] Step 501: Obtain a first vector group.

[0054] In this embodiment, the vectors in the first vector group are the trained original encoding vectors, that is, the absolute position encoding vectors of the above-mentioned Bert model. In this embodiment, the method for obtaining the first vector group can refer to the relevant technology and is not further limited or described here.

[0055] Step 502: Generate a second vector group based on the first vector group.

[0056] In this embodiment, the first vector group is used as a basis to obtain a second vector group including a larger number of vectors. It can be understood that the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined based on one or more vectors in the first vector group.

[0057] In this way, in this embodiment, the first vector group and the second vector group are obtained based on the original first vector group, so that the number of encoding vectors supported by the Bert model can be expanded. In this way, longer texts can be directly processed, thereby improving the processing effect on the text.

[0058] In some embodiments, the first vector group includes N vectors, where N is a positive integer. Specifically, N is equal to 512 for exemplary illustration. Furthermore, the number of vectors in the second vector group is N*N. In this way, a total of N(N+1) vectors are used as input encoding vectors, thereby increasing the length of the input text.

[0059] In some embodiments, step 502 includes:

[0060] The second vector group is determined by the following formula:

[0061] q i×n+j =αp (Imax+1-i) +(1-α)p j ;

[0062] In the above formula, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p j is a vector in the first vector group. Imax is the maximum position number value of the original coding vector. In this embodiment, the maximum position number value of the absolute position coding vector of the Bert model is set to 512. Therefore, in this embodiment, Imax is equal to 512. Obviously, when the model used changes, or the encoding of the maximum position of the absolute position coding vector corresponding to the model changes, the value of Imax is also adjusted accordingly. α is a preset coefficient, or hyperparameter, 0<α<1, and α is not equal to 0.5. Generally speaking, for ease of calculation, the value range of α is set to 0<α<0.5.

[0063] like Figure 6 As shown, in this embodiment, it can be understood that the position (i-1)×n+j is hierarchically represented as (i, j), and then the position codes corresponding to i and j are αp (513-i) and (1-α)p j , and the final i×n+j encoding vector is the superposition of the two. In this way, the encoding of N*N positions in the second vector group can be represented by the N parameters in the first vector group.

[0064] Furthermore, in order to match the representation of the vectors in the second vector group, in this embodiment, the vectors in the first vector group can be represented as q i , and let q i =p i , i = 1, 2, ..., N, which can be understood as, q i To q N With p i to p N In fact, there is a one-to-one correspondence.

[0065] Step 503: using the vectors included in the first vector group and the second vector group as input encoding vectors, performing model training to obtain a summary generation model.

[0066] Finally, the first vector combination and the second vector group are used as the input coding vector of the Bert model. In this embodiment, the first vector group including N vectors and the second vector group including N*N vectors are actually used as input. Therefore, in this embodiment, an input coding vector including N(N+1) vectors is obtained.

[0067] In some embodiments, step 503 includes:

[0068] When the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold, the vectors included in the first vector group are used as input encoding vectors to perform model training;

[0069] When the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold, the vectors included in the first vector group and the vectors included in the second vector group are sequentially arranged as input encoding vectors for model training.

[0070] In the technical solution of this embodiment, the above-mentioned preset word count threshold is equal to the number of absolute position encoding vectors of the Bert model. In this way, when the number of original texts is less than or equal to the preset word count threshold, its implementation method is basically the same as the existing method. When the number of texts to be summarized is greater than the preset word count threshold, the number of vectors in the obtained first vector combination and the second vector group will increase with the increase in the number of texts, thereby meeting the model training requirements.

[0071] The embodiment of the present invention generates a second vector group through the trained original coding vector. Since the second vector group is generated based on the first vector group, it is compatible with the first vector group. Furthermore, by performing model training through the first vector group and the second vector group, the length of the input text can be increased, thereby facilitating the generation of a summary of a long text.

[0072] An embodiment of the present invention provides a summary generation model training device.

[0073] like Figure 7 As shown, in one embodiment, the summary generation model training device 700 includes:

[0074] An acquisition module 701 is used to acquire a first vector group, wherein the vectors in the first vector group are trained original encoding vectors;

[0075] A generating module 702, configured to generate a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined based on one or more vectors in the first vector group;

[0076] The training module 703 is used to use the vectors included in the first vector group and the second vector group as input coding vectors to perform model training to obtain a summary generation model, wherein the summary generation model is a neural network model that uses the input coding vector of the text as input and uses the summary of the text as output.

[0077] Optionally, the first vector group includes N vectors, and the second vector group includes N*N vectors, where N is a positive integer.

[0078] Optionally, the generating module 702 is specifically configured to:

[0079] The second vector group is determined by the following formula:

[0080] q i×n+j =αp (Imax+1-i) +(1-α)p j ;

[0081] Among them, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p j is a vector in the first vector group, Imax is the maximum position number value of the original encoding vector, α is a preset coefficient, 0<α<1, and α is not equal to 0.5.

[0082] Optionally, the training module 703 includes:

[0083] A first training submodule is used for performing model training using the vectors included in the first vector group as input encoding vectors when the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold;

[0084] The second training submodule is used to sequentially arrange the vectors included in the first vector group and the vectors included in the second vector group as input coding vectors for model training when the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold.

[0085] The summary generation model training device of the present invention can implement each step of the above summary generation model training method embodiment and can achieve basically the same or similar technical effects, which will not be described in detail here.

[0086] An embodiment of the present invention provides an electronic device, including a processor and a memory. When the processor executes a program stored in the memory, the processor implements the steps of any of the above-mentioned summary generation model training methods.

[0087] An embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described summary generation model training methods.

[0088] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.< / s> < / s> < / e> < / s>

Claims

1. A summary generation model training method, characterized in that: The following steps are involved: Obtaining a first vector group, wherein the vectors in the first vector group are trained original encoding vectors; Generate a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined according to one or more vectors in the first vector group; Using the vectors included in the first vector group and the second vector group as input encoding vectors, model training is performed to obtain a summary generation model, wherein the summary generation model is a neural network model that uses the corresponding vectors of the text as input encoding vectors and uses the summary of the text as output; The first vector group includes N vectors, and the second vector group includes N*N vectors, where N is a positive integer; Taking the first vector group as a basis, generating a second vector group includes: The second vector group is determined by the following formula: q i×n+j =αp (Imax+1-i) +(1-α)p j ; Among them, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p j is a vector in the first vector group, Imax is the maximum position number value of the original encoding vector, α is a preset coefficient, 0<α<1, and α is not equal to 0.5; The method of using the vectors included in the first vector group and the second vector group as input encoding vectors and performing model training to obtain a summary generation model includes: When the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold, the vectors included in the first vector group are used as input encoding vectors to perform model training; When the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold, the vectors included in the first vector group and the vectors included in the second vector group are sequentially arranged as input encoding vectors for model training.

2. A summary generation model training device, characterized in that: include: An acquisition module, used to acquire a first vector group, wherein the vectors in the first vector group are trained original encoding vectors; A generating module, configured to generate a second vector group based on the first vector group, wherein the number of vectors in the second vector group is greater than the number of vectors in the first vector group, and each vector in the second vector group is determined based on one or more vectors in the first vector group; A training module, configured to use the vectors included in the first vector group and the second vector group as input encoding vectors to perform model training to obtain a summary generation model, wherein the summary generation model is a neural network model that uses the corresponding vectors of the text as input encoding vectors and uses the summary of the text as output; The first vector group includes N vectors, and the second vector group includes N*N vectors, where N is a positive integer; The generation module is specifically used for: The second vector group is determined by the following formula: q i×n+j =αp (Imax+1-i) +(1-α)p j ; Among them, q i×n+j is a vector in the second vector group, p (Imax+1-i) and p j is a vector in the first vector group, Imax is the maximum position number value of the original encoding vector, α is a preset coefficient, 0<α<1, and α is not equal to 0.5; The training module includes: A first training submodule is used for performing model training using the vectors included in the first vector group as input encoding vectors when the number of words in the text for which a summary needs to be generated is greater than a preset word count threshold; The second training submodule is used to sequentially arrange the vectors included in the first vector group and the vectors included in the second vector group as input coding vectors for model training when the number of words in the text for which a summary needs to be generated is less than or equal to the preset word count threshold.

3. An electronic device, characterized in that: The system comprises a processor and a memory. When the processor executes a program stored in the memory, the processor implements the steps of the summary generation model training method according to claim 1.

4. A non-transitory computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the summary generation model training method described in claim 1 is implemented.

Images