Method and device for training text generation model, and text generation method and device

Abstract

The present disclosure relates to a method and apparatus for training a text generation model, a text generation method and apparatus, an electronic device, and a computer-readable storage medium, and relates to the field of text generation technology. The method comprises: obtaining a first training data set; the training data in the first training data set includes a first word sequence, a first label sequence with the first word sequence as a generation constraint, and the first label sequence includes an insertion class label; obtaining a second training data set; the training data in the second training data set includes a second word sequence, a second label sequence with the second word sequence as a generation constraint, and the second label sequence includes an insertion class label and a non-insertion class label; inputting the first word sequence into a first text generation model for sequence generation task training to obtain a second text generation model; inputting the second word sequence into the second text generation model for sequence generation task training to obtain a third text generation model, thereby improving the text generation effect of the model.

Description

Technical Field

[0001] The present disclosure relates to text generation technology, and more particularly, to a method and apparatus for training a text generation model, a text generation method and apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] Text generation is widely used in daily life and is also a key research direction in machine learning for language processing. Among existing text generation methods, one is constraint-based. Given a word sequence, the generated text must contain all the words in the given word sequence. For example, constraint-based text generation can be considered when generating summaries or search queries from a given word sequence. However, existing constraint-based text generation models have relatively poor performance and cannot meet these requirements. Summary of the Invention

[0003] The embodiments of the present disclosure provide a method for training a text generation model, a text generation method, a text generation device, an electronic device, and a computer-readable storage medium to improve the accuracy and efficiency of text generation.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training a text generation model is provided. The method for training a text generation model comprises:

[0005] Obtain a first training data set; the training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label;

[0006] Obtain a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels;

[0007] Inputting the first word sequence into a first text generation model, and using the first label sequence corresponding to the first word sequence as supervision, the first text generation model is trained on a sequence generation task to obtain a second text generation model;

[0008] The second word sequence is input into the second text generation model, and the second label sequence corresponding to the second word sequence is used as supervision to train the second text generation model on a sequence generation task to obtain a third text generation model.

[0009] Optionally, the method for training a text generation model further includes:

[0010] Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0011] The third word sequence is input into the third text generation model, and the rhetoric type label corresponding to the third word sequence is used as supervision to perform classification task training on the third text generation model to obtain a fourth text generation model.

[0012] Optionally, the method for training a text generation model further includes:

[0013] Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0014] When the first text generation model is trained on the sequence generation task, the first text generation model is simultaneously trained on the classification task using the third training data set;

[0015] Calculate the total loss of the first text generation model, and determine whether to end the training of the first text generation model based on the total loss of the first text generation model; the total loss of the first text generation model is the sum of the loss generated by the first text generation model in the sequence generation task training and the loss generated by the first text generation model in the classification task training.

[0016] Optionally, the method for training a text generation model further includes:

[0017] Acquire a fourth training data set, where training data in the fourth training data set includes a fourth word sequence and a rhetorical type label corresponding to the fourth word sequence;

[0018] When the second text generation model is trained on the sequence generation task, the fourth training data set is used to simultaneously train the second text generation model on the classification task;

[0019] Calculate the total loss of the second text generation model, and determine whether to end the training of the second text generation model based on the total loss of the second text generation model; the total loss of the second text generation model is the sum of the loss generated by the second text generation model in the sequence generation task training and the loss generated by the second text generation model in the classification task training.

[0020] Optionally, before obtaining the first training data set, the method for training a text generation model further includes:

[0021] Obtain a sample sentence and decompose the sample sentence into an initial word sequence;

[0022] Performing data enhancement processing on the initial word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection;

[0023] The training data in the first training dataset are constructed using the initial word sequence and the enhanced word sequence.

[0024] Optionally, before obtaining the second training data set, the method for training the text generation model further includes:

[0025] Obtain a sample sentence and decompose the sample sentence into an initial word sequence;

[0026] Inserting synonyms and / or replacing antonyms into the initial word sequence to obtain at least one intermediate word sequence;

[0027] Performing data enhancement processing on the intermediate word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection;

[0028] The training data in the second training dataset are constructed using the intermediate word sequence and the enhanced word sequence.

[0029] According to a second aspect of an embodiment of the present disclosure, a method for training a text generation model is provided. The method for training a text generation model comprises:

[0030] Obtain a first training data set; the training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label;

[0031] Obtain a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels;

[0032] Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0033] The first text generation model is trained simultaneously using the first training data set, the second training data set, and the third training data set to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

[0034] According to a third aspect of an embodiment of the present disclosure, a text generation method is provided. The text generation method includes:

[0035] Inputting a first target word sequence into a co-occurrence word sampling model to expand the first target word sequence into a second target word sequence containing target co-occurrence words through the co-occurrence word sampling model, wherein the target co-occurrence words are words that co-occur with words in the first target word sequence in a co-occurrence dictionary;

[0036] Inputting the second target word sequence into a text generation model trained by the method for training a text generation model according to the first aspect of the present disclosure or the method for training a text generation model according to the second aspect of the present disclosure to obtain at least one third target word sequence;

[0037] Output a text consisting of the third target word sequence.

[0038] Optionally, when there are multiple third target word sequences, the text generation method further includes:

[0039] Calculating a perplexity index value for each of the third target word sequences;

[0040] Determining the third target word sequence with the smallest perplexity index value as the optimal target word sequence;

[0041] The outputting the text composed of the third target word sequence includes: outputting the text composed of the optimal target word sequence.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a device for training a text generation model is provided. The device for training a text generation model includes:

[0043] A first acquisition module is configured to acquire a first training data set; the training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label;

[0044] A second acquisition module is configured to acquire a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels;

[0045] a first training execution module, configured to input a first word sequence into a first text generation model, and use a first label sequence corresponding to the first word sequence as supervision to perform sequence generation task training on the first text generation model, thereby obtaining a second text generation model;

[0046] The second training execution module is used to input the second word sequence into the second text generation model, and use the second label sequence corresponding to the second word sequence as supervision to perform sequence generation task training on the second text generation model to obtain a third text generation model.

[0047] Optionally, the apparatus for training a text generation model further includes a third acquisition module and a third training execution module.

[0048] A third acquisition module is configured to acquire a third training data set, wherein the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0049] The third training execution module is used to input the third word sequence into the third text generation model, and use the rhetoric type label corresponding to the third word sequence as supervision to perform classification task training on the third text generation model to obtain a fourth text generation model.

[0050] Optionally, the apparatus for training a text generation model further includes a fourth acquisition module and a fourth training execution module.

[0051] The fourth acquisition module is used to acquire a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0052] The fourth training execution module is used to use the third training data set to perform classification task training on the first text generation model while performing sequence generation task training on the first text generation model; and calculate the total loss of the first text generation model, and determine whether to end the training of the first text generation model based on the total loss of the first text generation model; the total loss of the first text generation model is the sum of the loss generated by the first text generation model in the sequence generation task training and the loss generated by the first text generation model in the classification task training.

[0053] Optionally, the apparatus for training a text generation model further includes a fifth acquisition module and a fifth training execution module.

[0054] The fifth acquisition module is used to acquire a fourth training data set, where the training data in the fourth training data set includes a fourth word sequence and a rhetorical type label corresponding to the fourth word sequence;

[0055] The fifth training execution module is used to use the fourth training data set to perform classification task training on the second text generation model while training the second text generation model for the sequence generation task; and calculate the total loss of the second text generation model, and determine whether to end the training of the second text generation model based on the total loss of the second text generation model; the total loss of the second text generation model is the sum of the loss generated by the second text generation model in the sequence generation task training and the loss generated by the second text generation model in the classification task training.

[0056] Optionally, the apparatus for training a text generation model further includes a first decomposition module, a first enhancement processing module and a first construction module.

[0057] The first decomposition module is used to obtain a sample sentence and decompose the sample sentence into an initial word sequence;

[0058] The first enhancement processing module is used to perform data enhancement processing on the initial word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection;

[0059] The first construction module is used to use the initial word sequence and the enhanced word sequence to respectively construct the training data in the first training data set.

[0060] Optionally, the apparatus for training a text generation model further includes a second decomposition module, an insertion and replacement module, a second enhancement processing module and a second construction module.

[0061] The second decomposition module is used to obtain a sample sentence and decompose the sample sentence into an initial word sequence;

[0062] The insertion and replacement module is used to insert synonyms and / or replace antonyms into the initial word sequence to obtain at least one intermediate word sequence;

[0063] The second enhancement processing module is used to perform data enhancement processing on the intermediate word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection;

[0064] The second construction module is used to use the intermediate word sequence and the enhanced word sequence to respectively construct the training data in the second training data set.

[0065] According to a fifth aspect of an embodiment of the present disclosure, a device for training a text generation model is provided. The device for training a text generation model includes:

[0066] A first acquisition module is configured to acquire a first training data set; the training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label;

[0067] A second acquisition module is configured to acquire a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels;

[0068] A third acquisition module is configured to acquire a third training data set, wherein the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0069] A training execution module is used to use the first training data set, the second training data set, and the third training data set to simultaneously train the first text generation model to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

[0070] According to a sixth aspect of an embodiment of the present disclosure, there is provided a text generation device, the text generation device comprising:

[0071] an expansion module, configured to input a first target word sequence into a co-occurrence word sampling model, so as to expand the first target word sequence into a second target word sequence containing a target co-occurrence word through the co-occurrence word sampling model, wherein the target co-occurrence word is a word that co-occurs with a word in the first target word sequence in a co-occurrence dictionary;

[0072] a generation module, configured to input the second target word sequence into a text generation model trained according to the method for training a text generation model of the first aspect of the present disclosure or the method for training a text generation model of the second aspect of the present disclosure, to obtain at least one third target word sequence;

[0073] An output module is used to output a text composed of at least one of the third target word sequences.

[0074] Optionally, the text generation device further includes an evaluation module and a determination module.

[0075] The evaluation module is configured to calculate a perplexity index value for each of the third target word sequences when there are multiple third target word sequences;

[0076] a determination module, configured to determine the third target word sequence having the smallest perplexity index value as the optimal target word sequence;

[0077] The output module is specifically configured to output the text composed of the optimal target word sequence.

[0078] According to a seventh aspect of an embodiment of the present disclosure, an electronic device is provided, which includes a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the method for training a text generation model of the first aspect of the present disclosure, or the method for training a text generation model of the second aspect of the present disclosure, or the text generation method of the third aspect of the present disclosure is implemented.

[0079] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the method for training a text generation model of the first aspect of the present disclosure, or the method for training a text generation model of the second aspect of the present disclosure, or the text generation method of the third aspect of the present disclosure is implemented.

[0080] The embodiments of the present disclosure provide a method and apparatus for training a text generation model, a text generation method and apparatus, an electronic device, and a computer-readable storage medium. The method and apparatus use training data containing insertion labels to perform sequence generation task training on a first text generation model to obtain a second text generation model, so that the second text generation model has the ability to insert words. The method and apparatus use training data containing both insertion labels and non-insertion labels to perform sequence generation task training on the second text generation model to obtain a third text generation model, so that the third text generation model has the ability to insert words and eliminate insertion errors, thereby enhancing the reliability of the text generation model and improving the text generation effect of the text generation model.

[0081] Features and advantages of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0083] Figure 1 This is a schematic structural diagram of an electronic device provided by one embodiment of the present disclosure and can be used to implement the embodiment of the present disclosure;

[0084] Figure 2-3 is a schematic diagram of a method for training a text generation model provided by an embodiment of the present disclosure;

[0085] Figure 4 is a schematic diagram of a method for training a text generation model provided by an embodiment of the present disclosure;

[0086] Figure 5 is a flowchart of a method for training a text generation model provided by an embodiment of the present disclosure;

[0087] Figure 6 is a schematic diagram of a method for training a text generation model provided by an embodiment of the present disclosure;

[0088] Figure 7 is a flowchart of a text generation method provided by an embodiment of the present disclosure;

[0089] Figure 8 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0090] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0091] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present disclosure, its application, or uses.

[0092] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0093] <Hardware Configuration>

[0094] Figure 1 1 is a schematic diagram of an electronic device that can be used to implement the embodiment of the present disclosure. The electronic device can be used to implement the method for training a text generation model or the text generation method of the embodiment of the present disclosure.

[0095] The electronic device 1000 can be a smart phone, a portable computer, a desktop computer, a tablet computer, a server, etc., which is not limited here.

[0096] The electronic device 1000 may include, but is not limited to, a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MCU), or the like, and is configured to execute computer programs, which may be written using an instruction set such as an x86, ARM, RISC, MIPS, or SSE architecture. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a serial interface, or a parallel interface. The communication device 1400 may be capable of wired communication using optical fiber or cable, or wireless communication, specifically, WiFi, Bluetooth, 2G / 3G / 4G / 5G, or the like. The display device 1500 may be, for example, an LCD display or a touchscreen display. The input device 1600 may include, for example, a touchscreen, a keyboard, or somatosensory input. The speaker 1700 is used to output audio signals, and the microphone 1800 is used to collect audio signals.

[0097] As used in the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is used to store a computer program, which is used to control the processor 1100 to operate to implement the method according to the embodiments of the present disclosure. Technicians can design the computer program according to the scheme disclosed in the present disclosure. How the computer program controls the processor to operate is well known in the art and will not be described in detail here. The electronic device 1000 can be installed with an intelligent operating system (such as Windows, Linux, Android, IOS, etc.) and application software.

[0098] It should be understood by those skilled in the art that although Figure 1 , multiple devices of the electronic device 1000 are shown; however, the electronic device 1000 of the embodiment of the present disclosure may only involve some of the devices, for example, only the processor 1100 and the memory 1200.

[0099] Hereinafter, various embodiments and examples of the fundamental disclosure are described with reference to the accompanying drawings.

[0100] <Embodiment of a Method for Training a Text Generation Model>

[0101] First, it is explained how to construct the training data of the embodiment of the present disclosure.

[0102] In one example, original text is obtained, and the original text is filtered according to certain rules to obtain sample text.

[0103] For example, the original text is obtained by crawling the web. For example, the original text is filtered according to the length requirement, and the original text with a length within a certain range is retained. For example, the original text containing Arabic numerals is filtered out, and the original text without Arabic numerals is retained. For example, the original text containing foreign words is filtered out, and the original text in pure Chinese is retained. In the embodiment of the present disclosure, a combination of multiple filtering methods can be used to filter the original text to obtain sample text according to the scenario to which the text generation model is intended to be applied. In the embodiment of the present disclosure, sample text can also be obtained by other methods, which are not limited to this.

[0104] The sample text is divided into sentences to obtain sample sentences. In one example, the sample text can be divided into sentences according to periods to obtain sample sentences.

[0105] The following describes how to construct the first training data set in the embodiment of the present disclosure, including steps S102-S106.

[0106] Step S102: Obtain a sample sentence and decompose the sample sentence into an initial word sequence.

[0107] In one example, word segmentation tools such as Jieba and NLTK can be used to segment each sentence, thereby obtaining an initial word sequence corresponding to the sample sentence.

[0108] Step S104: Perform data enhancement processing on the initial word sequence to obtain at least one enhanced word sequence. The data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection.

[0109] In one example, the enhancement method based on synonym replacement can be implemented by using a common synonym dictionary to search and replace synonyms. The synonym dictionary can be a general Chinese dictionary (such as the Xinhua Dictionary), a general English dictionary (such as the Oxford Dictionary), the WordNet English vocabulary database, a specialized Chinese synonym dictionary, a specialized application synonym dictionary, etc. For example, a word is randomly taken from the initial word sequence, the synonyms of the word are searched using the WordNet English vocabulary database, and then the replacement operation is performed. For example, for the sample sentence "It is awesome", the word "awesome" is selected as the word to be replaced, and the synonyms of "awesome" are found in the WordNet English vocabulary database, such as "amazing", "awe-inspiring", and "awing", and the synonym "amazing" is selected for replacement, resulting in the enhanced word sequence "It is amazing".

[0110] In one example, an enhancement method based on synonym replacement can be to input the initial word sequence into a pre-trained semantic similarity word replacement model, and the semantic similarity word replacement model completes the replacement of similar words. The semantic similarity word replacement model can be obtained by training a word vector model.

[0111] The term frequency–inverse document frequency (TF-IDF) algorithm is a feature vectorization method used in text mining to reflect the importance of terms to documents in a corpus. In one example, the TF-IDF algorithm can be used to identify the more important terms in an initial word sequence, and then replace these important terms with synonyms. This selection and replacement of important terms with synonyms can enhance data augmentation and improve the performance of text generation models.

[0112] Back translation involves translating a word sequence from a first language into a second language to obtain a word sequence in the second language, and then translating the second language word sequence back to the first language. In one example, the initial word sequence is in the first language. A translation model is used to translate the initial word sequence from the first language into a word sequence in the second language. The translation model then uses this translation model to translate the second language word sequence back to the first language to obtain an enhanced word sequence. For example, the initial word sequence of a Chinese sample article is translated into English and then translated back into Chinese to obtain an enhanced word sequence. For example, if the initial word sequence is "Teachers are like engineers of the human soul, imparting a wealth of knowledge," the enhanced word sequence generated through back translation could be "Teachers are like engineers of the human soul, imparting a wealth of experience." or "Teachers are like craftsmen of the human soul, imparting a wealth of experience."

[0113] Noise injection refers to adding noise to the initial word sequence to make the trained model robust to perturbations. In one example, a random noise injection method is used for the initial word sequence. For example, a unigram noise method is used for random noise injection. Based on this method, words that appear frequently in the corpus are inserted as noise into the initial word sequence or replace words in the initial word sequence, thereby obtaining an enhanced word sequence.

[0114] In the disclosed embodiment, considering that the text generation model has a large number of parameters, more training data is needed to fine-tune the parameters. The above-mentioned data enhancement method is used to enrich the training corpus, expand the training data, and make the sentences generated by the model rich and diverse.

[0115] Step S106: Use the initial word sequence and the enhanced word sequence to construct training data in the first training data set respectively.

[0116] The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label.

[0117] The constraints mentioned in the embodiments of this disclosure refer to the requirement that the text ultimately generated by the text generation model completely contain the given vocabulary, and that the order of the given vocabulary cannot be changed. In the embodiments of this disclosure, a label sequence uses a word sequence as a "generation constraint," meaning that if the target text is generated using that word sequence as a constraint, the portion of the target text that differs from the word sequence is used to form the label sequence.

[0118] In the disclosed embodiment, for an initial word sequence, a portion of words can be extracted from the initial word sequence to form a first word sequence, another portion of words can be used to form a first label sequence, and all labels in the first label sequence can be set as insertion class labels to form training data in the first training dataset. In the disclosed embodiment, a dynamic programming method can be used to gradually construct multiple sets of training data in the first training dataset for the initial word sequence or the enhanced word sequence.

[0119] For example, a sample sentence is: "About ten minutes later, they arrived at a courtyard. The decoration inside was magnificent, like an ancient palace." The initial word sequence obtained by decomposing the sample sentence is: ["about", "ten", "about", "minutes", "after", "they", "arrived", "a", "courtyard", "inside", "the", "decoration", "magnificent", "like", "ancient", "palace"].

[0120] Using the initial word sequence, a training data set generated in the first step is:

[0121] {"tokens":["[CLS]","About" ten" minutes" later," they" arrived at" a" courtyard" whose decoration was magnificent and resembled" an" ancient" palace.","[SEP]","[PAD]"],

[0122] "Im_label_tokens": ["[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]", "[NOI]"]}。

[0123] Among them, "tokens" is used as the first word sequence, [CLS] represents the start, and [SEP] represents the end. "Im_label_tokens" is used as the first label sequence with the first word sequence as the generation constraint. The meaning of the "[NOI]" label is that it is inserted as empty here, that is, no insertion is required here. The "[PAD]" symbol is used to complete to ensure that the lengths of the first word sequence and the first label sequence are the same.

[0124] Using the "tokens" in the first step, a training data generated in the second step is:

[0125] {"tokens": ["[CLS]", "Ten", "minutes", ",", "came to", "the courtyard", "inside", "decorated", "resplendent", "like", "a palace", "[SEP]"],

[0126] "Im_label_tokens": ["About", "came", "after", "they", "a", ",", "of", "[NOI]", ",", "in ancient times", ".", "[PAD]"]}。

[0127] Using the "tokens" in the second step, a training data generated in the third step is:

[0128] {"tokens": ["[CLS]", "minutes", "came to", "inside", "decorated", "like", "a palace", "[SEP]"],

[0129] "Im_label_tokens": ["Ten", ",", "the courtyard", "[NOI]", "resplendent", "[NOI]"]}。

[0130] Using the "tokens" in the third step, a training data generated in the fourth step is:

[0131] {"tokens": ["[CLS]", "minutes", "inside", "like", "[SEP]"],

[0132] "Im_label_tokens": ["[NOI]", "Come to", "Decorate", "Palace"]}。

[0133] It can be seen that by extracting 1 / 2 of the tokens from the first word sequence in the training data obtained in the previous step as the first word sequence in this step and constructing the first label sequence with the remaining 1 / 2 of the tokens, a training data can be constructed. In this way, for a single sample sentence, a total of log2 M training data can be constructed, where M is the length of the sample sentence, that is, the number of tokens in the initial word sequence decomposed from the sample sentence. Each piece of training data is added to the first training data set as independent training data.

[0134] The process of constructing training data using the enhanced word sequence and the process of constructing training data using the initial word sequence can be similar, and will not be repeated here.

[0135] The following describes how to construct the second training data set of the present disclosure embodiment, including steps S202 - S208.

[0136] Step S202, obtain a sample sentence and decompose the sample sentence into an initial word sequence.

[0137] Step S204, perform synonym insertion and / or antonym replacement on the initial word sequence to obtain at least one intermediate word sequence.

[0138] In the actual inference process of the text generation model, problems such as error accumulation and repeated generation are likely to occur. Error accumulation means that the text generation model predicts the next sequence based on the prediction result of the previous round sequence. If errors occur in the initial few rounds, it is easy to accumulate and amplify this error in subsequent steps. In the present disclosure embodiment, the text generation model uses a word - character mixed dictionary, which is likely to cause the problem of repeated insertion of words. For example, from the perspective of word - character mixing, "but" and "however" are two different words, and using the word - character mixed model for training is likely to cause the phenomenon of inserting both "but" and "however" at the same time.

[0139] To reduce this problem, intermediate word sequences obtained by performing synonym insertion and / or antonym replacement on the initial word sequence are introduced during the training process of the text generation model.

[0140] For example, the sample sentence is: Care is a look, giving silent blessings to students.

[0141] Initial word sequence: ["Care", "is", "a", "look", ",", "give", "students", "silent", "of", ",, "blessings", "."]

[0142] Use "silence" as a synonym of "silent" for synonym insertion, and get the middle word sequence: ["care", "is", "a", "eyes", ",", "give", "student", "silent", "silent", "blessing", "."].

[0143] Replace the antonym of "blessing" with "curse" and get the middle word sequence: ["care","is","a","eyes",",","give","student","silent","of","curse","."]

[0144] Step S206: Perform data enhancement processing on the intermediate word sequence to obtain at least one enhanced word sequence. The data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection.

[0145] Here, the data enhancement processing method is similar to that in step S104, and will not be described in detail.

[0146] Step S208: Use the intermediate word sequence and the enhanced word sequence to construct training data in the second training data set respectively.

[0147] The training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint. The second label sequence includes an insertion class label and a non-insertion class label. The non-insertion class label may include a deletion class label or a replacement class label.

[0148] In the disclosed embodiment, the process of constructing training data in step S208 is similar to that in step S106 and can also be performed using a dynamic programming method. The difference is that for each token in the second word sequence, if the token is a synonym inserted in step S204, the position corresponding to the token in the second label sequence should be a deletion class label; if the token is an antonym replaced in step S204, the position corresponding to the token in the second label sequence should be a deletion class label or a replacement class label.

[0149] The following are examples:

[0150] For the middle word sequence: ["care","is","a","eyes","","give","student","silent","silent","blessing","."], corresponding to the position of "silence", the deletion tag "[DEL]" can be used to indicate that "silence" should be deleted.

[0151] The generated training data is:

[0152] {"tokens":["[CLS]","Caring","is","a","look","gives","students","silent","blessing",".","[SEP]"],

[0153] "Im_label_tokens":[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[DEL],[NOI],[NOI],[NOI]]}.

[0154] For the middle word sequence: ["care","is","a","eyes",",","give","student","silent","of","curse","."], corresponding to the position of "curse", the deletion label "[DEL]" can be used to indicate that "curse" should be deleted and needs to be deleted in the next round of prediction. Alternatively, the replacement label "[REP:blessing]" can be used to indicate that "blessing" should replace "curse" in the next round of prediction.

[0155] The generated training data is:

[0156] {"tokens":["[CLS]","Caring","is","a","look","gives","students","silent","curse",".","[SEP]"],

[0157] "Im_label_tokens":[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[DEL]]}.

[0158] Alternatively, the generated training data is:

[0159] {"tokens":["[CLS]","Caring","is","a","look","gives","students","silent","curse",".","[SEP]"],

[0160] "Im_label_tokens":[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[NOI],[REP: blessing]]}.

[0161] In one example, the more important words in the initial word sequence can be determined by the TF-IDF (frequency-inverse document frequency) algorithm, and step S204 is performed on the more important words, that is, synonym insertion and / or antonym replacement is performed to obtain an intermediate word sequence. This method is often more effective and can enhance the model's recognition ability.

[0162] The following describes how to construct the third training data set in the embodiment of the present disclosure, including steps S302 - S306 .

[0163] Step S302: Obtain a sample sentence, and label the sample sentence with a corresponding rhetorical type label according to the rhetorical type of the sample sentence.

[0164] For example, the rhetorical type of the sample sentence is determined by manual annotation, and the sample sentence is labeled with the corresponding rhetorical type.

[0165] Step S304: Decompose the sample sentence into word sequences.

[0166] Step S306: Label the word sequence with the rhetoric type label corresponding to the sample sentence as training data, and add it to the third training data set.

[0167] The disclosed embodiment divides the training data into several categories according to rhetorical types, such as metaphor, parallelism, statement, and personification. It is expected that the text generation model can understand and identify different rhetorical types during the classification task training process, which is conducive to the text generation model generating texts with rhetorical type diversity and meeting the user's needs to generate texts of different rhetorical types.

[0168] The following describes how to construct the fourth training data set according to the embodiment of the present disclosure, including steps S402 - S406 .

[0169] Step S402: Obtain a sample sentence, and label the sample sentence with a corresponding rhetorical type label according to the rhetorical type of the sample sentence.

[0170] For example, the rhetorical type of the sample sentence is determined by manual annotation, and the sample sentence is labeled with the corresponding rhetorical type.

[0171] Step S404: Decompose the sample sentence into initial word sequences.

[0172] Step S406: Label the word sequence with the rhetoric type label corresponding to the sample sentence as training data, and add it to the fourth training data set.

[0173] In the embodiment of the present disclosure, the fourth training data set may be constructed in the same manner as the third training data set.

[0174] The original text, sample text, and sample sentences used in the training data of the above-mentioned first to fourth training datasets can be the same or different. The above steps S102, S202, S304, and S404 can adopt similar decomposition methods. Similar methods for constructing training data can be adopted in the above steps S106 and S208. Details are not described herein again.

[0175] The following describes how to train the text generation model.

[0176] In the embodiments of the present disclosure, the dictionary used by the text generation model is a word-character mixed dictionary. Through testing, it is found that in the prior art, text generation based on the insertion method usually has the phenomenon of being unsmooth. After analyzing the bad cases, the reason is that the text generation model based on the conventional dictionary adds the word collocation task to the training task in the training process, resulting in problems such as difficult model convergence and unreasonable parameter allocation. In the embodiments of the present disclosure, the text generation model adopts a word-character mixed dictionary, which can reduce the frequency of unsmooth problems, reduce the training time, improve the training efficiency, and increase the generation accuracy of the text generation model.

[0177] The word-character mixed dictionary is a mixed dictionary of words and characters. In one example, the word-character mixed dictionary is a manually preset dictionary that counts 50,000 common Chinese characters and common Chinese vocabulary, enabling the model to no longer need to pay extra attention to the collocation combinations within words and focus on training the process of inserting words. For example, the dictionary contains "理", "想", and "理想" at the same time, and the text model does not need to learn the relationship between "理" and "想" anymore.

[0178] According to the first aspect of the embodiments of the present disclosure, a method for training a text generation model is provided. Refer to Figure 2 and Figure 3 As shown, the method for training a text generation model provided by the embodiments of the present disclosure includes steps P102-P108.

[0179] Step P102: Obtain a first training dataset. The training data in the first training dataset includes a first word sequence and a first label sequence with the first word sequence as a generation constraint, and the first label sequence includes an insertion class label.

[0180] Step P104: Obtain a second training dataset. The training data in the second training dataset includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, and the second label sequence includes an insertion class label and a non-insertion class label. In the embodiments of the present disclosure, the non-insertion class label includes a deletion class label and / or a replacement class label.

[0181] Step P106: Use the first training dataset to train the first text generation model on a sequence generation task to obtain a second text generation model. Specifically, the first word sequence is input into the first text generation model, and the first label sequence corresponding to the first word sequence is used as supervision. The first text generation model is trained on the sequence generation task to obtain the second text generation model.

[0182] The essence of a text generation model is to model the ability to complete sentences given incomplete input. Therefore, the training goal is to predict the target sequence that should appear in the next iteration based on the intermediate results of the previous iteration. For example, as shown in the following table:

[0183]

[0184] In this table, X 0 The word sequence that is initially input to the text generation model is constrained to be X 0 .X 1 、X 2 , X3, and X4 are the sequences obtained by the text generation model in the first, second, third, and fourth rounds of prediction iterations. The sequence X obtained in the third round of iteration is 3 and the sequence X obtained in the fourth round of iteration 4 The same means that after 3 rounds of iteration, the prediction results tend to be stable and the sequence X can be output. 4 To the user.

[0185] The following uses Equation 1 to illustrate the method and process of using the first training dataset to train the first text generation model for the sequence generation task.

[0186]

[0187] In this formula 1: X 0 The word sequence that is initially input to the first text generation model, that is, the constraint is X 0 The first text generation model iterates K times in total, 1≤k≤K, and the input of each round is the expected prediction sequence of the previous round, X k-1 is the expected prediction sequence of the first text generation model in the k-1 round, X k Generate the expected prediction sequence for the first text model in round k. In each iteration, the insertion position and the corresponding word element are predicted based on the input sequence of the current round, and at most one word element is inserted at each position. k -X k-1 Represents X k and X k-1 The difference between them is the content that needs to be inserted in the kth round. The difference includes one or more words, and x represents any word in the difference. 0) is 1, p(X k ) is the first text generation model predicting the expected prediction sequence X of the kth round in the kth round k The probability of p(X k |X k-1 ) represents the expected prediction sequence X in the k-1th round k-1 As a condition, the expected prediction sequence X of the kth round is predicted in the kth round k The probability of p(x|X k-1 ) represents the expected prediction sequence X in the k-1th round k-1 As a condition, the probability of predicting the word x of the difference part in the kth round. "∏" is the multiplication symbol. The meaning of formula 1 is to use the constraint X 0 As a known condition, from the constraint X 0 Start predicting step by step and accurately predict X in the Kth round k The probability, p(X k ) is larger, the more accurate the prediction of the first text generation model is.

[0188] In step P106, the training is terminated when the loss of the first text generation model drops to a certain level. For example, when the loss of the first text generation model is within a preset range within a preset time period, the first text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated. For example, when the loss of the first text generation model is less than a preset threshold, the first text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated. Alternatively, when the number of training cycles reaches a preset number, the first text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated.

[0189] Step P108: Use the second training dataset to train the second text generation model on the sequence generation task, obtaining a third text generation model. Specifically, the second word sequence is input into the second text generation model, and the second label sequence corresponding to the second word sequence is used as supervision. The second text generation model is trained on the sequence generation task, obtaining the third text generation model.

[0190] The sequence generation task training in step P108 is incremental training. In this disclosed embodiment, incremental training refers to the process of iteratively and incrementally expanding the training data based on the model trained without adding non-insertion class labels. The purpose is to maintain the generalization of the training data while maintaining sensitivity to new training data. In this disclosed embodiment, the deletion class label "[DEL]" and the replacement class flag "[REP:]" are introduced to address the problem of error accumulation in the model and increase the reliability of the results.

[0191] The training process of step P108 is similar to the training process of step P106. Please refer to the content of step P106 and will not be repeated here.

[0192] In step P108, the training is terminated when the loss of the second text generation model drops to a certain level. For example, when the loss of the second text generation model is within a preset range within a preset time period, the second text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated. For example, when the loss of the second text generation model is less than a preset threshold, the second text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated. Alternatively, when the number of training cycles reaches a preset number, the second text generation model is considered to have converged, and the sequence generation task training for this phase can be terminated.

[0193] In an embodiment of the present disclosure, the loss function shown in Equation 2 or Equation 3 can be used to calculate the loss generated by the text generation model in the sequence generation task training, for example, in step P106 and step P108, the loss corresponding to each step is calculated.

[0194]

[0195] In formula 2, L is the loss of the kth round of iteration in the sequence generation task training of the text generation model. The input of each round is the expected prediction sequence of the previous round, X k-1 is the expected prediction sequence of the text generation model in the k-1th round, X k is the expected prediction sequence of the text generation model in the kth round. In the embodiment of the present disclosure, the expected prediction sequence refers to the target sequence predicted by the expected text generation model. + Represents X k and X k-1 The difference between represents the change that needs to be made in the kth iteration, and x is a word in the difference. In each iteration, the text generation model first predicts the position that needs to be changed, and then predicts the specific content at that position. k-1 Represents the expected prediction intermediate vector of the kth round, which represents the expected prediction intermediate vector in order to obtain X k , in X k-1 The position where the change needs to be made. p(X k |X k-1 ) represents the expected prediction sequence X in the k-1th round k-1 As a condition, the expected prediction sequence X of the kth round is predicted in the kth round k The probability of p(x|Φ k-1 ,X k-1 ) represents the k-1 and X k-1 As a condition, predict the probability of word x. p(Φ k-1|X k-1 ) indicates that X k-1 As a condition, predict Φ k-1 probability.

[0196]

[0197] In this formula 3, CE(p,q) represents the loss of the text generation model in this round of iteration in the sequence generation task training, which is measured by cross entropy. In the example corresponding to this formula 3, the vocabulary uses the ONE-HOT encoding method. The dimension of the vocabulary is the total number of ONE-HOT encoding vectors. Each word in the vocabulary has its own unique corresponding ONE-HOT encoding vector. In this formula 3, C is the number of word units in the sequence composed of the changes expected to be made in this round of iteration, that is, the number of word units in the sequence composed of the difference between the expected prediction sequence of this round and the expected prediction sequence of the previous round, 1≤i≤C, p i is the ONE-HOT encoding vector corresponding to the i-th word in the sequence, q i is the probability of predicting the i-th word in the sequence in this round.

[0198] The method for training a text generation model provided by an embodiment of the present disclosure uses training data containing insertion class labels to perform sequence generation task training on a first text generation model to obtain a second text generation model, so that the second text generation model has the ability to insert words, and then uses training data containing both insertion class labels and non-insertion class labels to perform sequence generation task training on the second text generation model to obtain a third text generation model, so that the third text generation model has the ability to insert words and eliminate insertion errors, thereby enhancing the reliability of the text generation model and improving the text generation effect of the text generation model.

[0199] See also Figure 4 In one example, the method for training a text generation model further includes classification task training. Specifically, after step P108, the method for training a text generation model may further include steps P202-SP204.

[0200] Step P202: Obtain a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence.

[0201] Step P204: Use the third training dataset to train the third text generation model for the classification task, thereby obtaining a fourth text generation model. Specifically, the third word sequence is input into the third text generation model, and the rhetorical type label corresponding to the third word sequence is used as supervision. The third text generation model is then trained for the classification task, thereby obtaining a fourth text generation model.

[0202] In step P204, training is terminated when the loss of the third text generation model drops to a certain level. For example, when the loss of the third text generation model is within a preset range within a preset time period, the third text generation model is considered to have converged, and training can be terminated. For example, when the loss of the third text generation model is less than a preset threshold, the third text generation model is considered to have converged, and training can be terminated. Alternatively, when the number of training cycles reaches a preset number, the third text generation model is considered to have converged, and training can be terminated.

[0203] In this embodiment, after training to obtain a third text generation model that has the ability to insert words and eliminate insertion errors, the third text generation model is trained to have rhetoric classification capabilities to obtain a fourth text generation model, so that the parameters learned by the model are more comprehensive, and the prediction of rhetoric types can be completed based on the prediction sequence, thereby enhancing the scalability and robustness of the model to meet the subsequent generation requirements of different rhetoric types.

[0204] See also Figure 5 As shown, the method for training a text generation model may further include performing classification task training on the text generation model.

[0205] In one example, the method for training a text generation model further includes steps P302 - S306 .

[0206] Step P302: Obtain a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence.

[0207] Step P304: While training the first text generation model on the sequence generation task, simultaneously train the first text generation model on the classification task using the third training dataset. Calculate the total loss of the first text generation model, and determine whether to terminate training of the first text generation model based on the total loss of the first text generation model. The total loss of the first text generation model is the sum of the loss incurred by the first text generation model during training for the sequence generation task and the loss incurred by the first text generation model during training for the classification task.

[0208] In step P304, training is terminated when the total loss of the first text generation model drops to a certain level. For example, when the total loss of the first text generation model is within a preset range within a preset time period, the first text generation model is considered to have converged, and training can be terminated. For example, when the total loss of the first text generation model is less than a preset threshold, the first text generation model is considered to have converged, and training can be terminated. Alternatively, when the number of training times reaches a preset number, the first text generation model is considered to have converged, and training can be terminated.

[0209] In this example, the first text generation model is trained using a dual-task joint training method to obtain a second text generation model that is sensitive to both insertion and rhetorical types. That is, a model that has the ability to insert words and identify rhetorical devices at the same time, making the sentences generated by the model rich and diverse.

[0210] In one example, the method for training a text generation model further includes steps P402-P406.

[0211] Step P402: Obtain a fourth training data set, where the training data in the fourth training data set includes a fourth word sequence and a rhetorical type label corresponding to the fourth word sequence.

[0212] Step P404: While training the second text generation model on the sequence generation task, simultaneously train the second text generation model on the classification task using the fourth training dataset. Calculate the total loss of the second text generation model, and determine whether to terminate training of the second text generation model based on the total loss of the second text generation model. The total loss of the second text generation model is the sum of the loss incurred by the second text generation model during training for the sequence generation task and the loss incurred by the second text generation model during training for the classification task.

[0213] In step P404, training is terminated when the total loss of the second text generation model drops to a certain level. For example, when the total loss of the second text generation model is within a preset range within a preset time period, the second text generation model is considered to have converged, and training can be terminated. For example, when the total loss of the second text generation model is less than a preset threshold, the second text generation model is considered to have converged, and training can be terminated. Alternatively, when the number of training times reaches a preset number, the second text generation model is considered to have converged, and training can be terminated.

[0214] In this example, the second text generation model is trained using a dual-task joint training method to obtain a third text generation model that is sensitive to deletion / replacement and rhetorical types. That is, a model that has the ability to insert words, delete words / replace words, and identify rhetorical techniques, making the sentences generated by the model rich and diverse.

[0215] In one example, the total loss of a text generation model is: Loss = Loss_sequence + Loss_classification. Loss is the total loss of the text generation model, Loss_sequence represents the loss incurred by the text generation model during sequence generation training, and Loss_classification represents the loss incurred by the text generation model during classification training. In other words, the total loss of the text generation model is the sum of the loss incurred by the text generation model during sequence generation training and the loss incurred by the text generation model during classification training.

[0216] In the disclosed embodiment, the loss function shown in Equation 4 or Equation 5 can be used to calculate the loss generated by the text generation model during the classification task training. For example, in step P204, step P304, and step P404, the loss generated during the classification task training corresponding to each step is calculated.

[0217]

[0218] Taking step P204 as an example, in formula 4, θ represents the trainable parameter in the third text generation model, l(θ) is the likelihood function of θ, that is, the loss generated by the third text generation model in the classification task training. n is the number of third word sequences input to the third text generation model, 1≤ i≤n. i is the third word sequence of the i-th word, y i is x i The real rhetorical type label. yi (x i ;θ) represents x i and θ as conditions, predict y i probability.

[0219] Taking step P304 as an example, in formula 4, θ represents the trainable parameter in the first text generation model, l(θ) is the likelihood function of θ, that is, the loss generated by the first text generation model in the classification task training. n is the number of third word sequences input to the first text generation model, 1≤ i≤n. i is the third word sequence of the i-th word, y i is x i The real rhetorical type label. yi (x i ;θ) represents x i and θ as conditions, predict y i probability.

[0220] Taking step P404 as an example, in formula 4, θ represents the trainable parameter in the second text generation model, l(θ) is the likelihood function of θ, that is, the loss generated by the second text generation model in the classification task training. n is the number of fourth word sequences input to the second text generation model, 1≤ i≤n. i is the fourth word sequence of the i-th word, y i is x i The real rhetorical type label. yi (x i ;θ) represents x i and θ as conditions, predict y i probability.

[0221]

[0222] In Equation 5, CE(p,q) represents the loss incurred by the text generation model during classification task training, measured by cross-entropy. In the example corresponding to Equation 5, the rhetorical type labels use ONE-HOT encoding. The number of rhetorical type labels is the total number of ONE-HOT encoding vectors, and each rhetorical type label has a unique corresponding ONE-HOT encoding vector.

[0223] Taking step P204 as an example, in formula 4, θ represents the trainable parameter in the third text generation model, l(θ) is the likelihood function of θ, that is, the loss generated by the third text generation model in the classification task training. n is the number of third word sequences input to the third text generation model, 1≤ i≤n. i is the third word sequence of the i-th word, y i is x i The real rhetorical type label. yi (x i ;θ) represents x i and θ as conditions, predict y i probability.

[0224] Taking step P304 as an example, in formula 5, n is the number of third word sequences input into the first text generation model, 1≤i≤n. i is the encoding vector corresponding to the true rhetorical type label of the i-th third word sequence. i For the i-th third word sequence, predict the probability of its true rhetorical type label.

[0225] Taking step P404 as an example, in formula 5, n is the number of the fourth word sequence input into the second text generation model, 1≤i≤n. i is the encoding vector corresponding to the true rhetorical type label of the fourth word sequence i.i For the i-th fourth word sequence, predict the probability of its true rhetorical type label.

[0226] The non-autoregressive, constraint-based text generation model proposed in the embodiment of the present disclosure takes into account the complexity and scalability of the training objectives, and introduces the task of rhetorical device classification for joint training, so that the parameters learned by the model are more comprehensive. On the basis of the predicted sequence, the rhetorical type can also be predicted, thereby enhancing the scalability and robustness of the model to the subsequent generation requirements of different rhetorical types.

[0227] According to a second aspect of an embodiment of the present disclosure, a method for training a text generation model is provided. Figure 6 As shown, the method for training a text generation model includes steps Q102-Q108.

[0228] Step Q102: Obtain a first training data set. The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label.

[0229] Step Q104: Obtain a second training data set. The training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint. The second label sequence includes an insertion class label and a non-insertion class label.

[0230] Step Q106: Obtain a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence.

[0231] Step Q108: Use the first training data set, the second training data set, and the third training data set to simultaneously train the first text generation model to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

[0232] In this embodiment, the first text generation model is synchronously trained using the first training data set, the second training data set, and the third training data set to obtain the second text generation model, so that the second text generation model has the ability to insert words and eliminate insertion errors, and can also complete the prediction of rhetoric types based on the prediction sequence, thereby enhancing the scalability and robustness of the model to meet the subsequent generation requirements of different rhetoric types.

[0233] <Example of Text Generation Method>

[0234] First, the process of constructing a co-occurrence dictionary in the embodiment of the present disclosure is described. The process of constructing a co-occurrence dictionary includes steps S502-S508.

[0235] Step S502: After removing stop words from the segmented overall training corpus, count the high-frequency words that appear, such as words like "ideal" and "white cloud".

[0236] Here, a high-frequency word refers to a word whose appearance frequency exceeds a preset frequency.

[0237] Step S504: For each high-frequency word, maintain a corresponding candidate co-occurrence vocabulary list respectively. For example, traverse the overall training corpus after removing stop words, add the words that have had a co-occurrence relationship with the high-frequency word as candidate co-occurrence words to the candidate co-occurrence vocabulary list of this high-frequency word, and count the number of co-occurrences.

[0238] Step S506: Screen out the co-occurrence candidate words whose number of co-occurrences exceeds a preset threshold from the candidate co-occurrence vocabulary list of this high-frequency word as the target co-occurrence words of this high-frequency word.

[0239] Step S508: Construct a co-occurrence dictionary using the high-frequency word and the target co-occurrence words of the high-frequency word. The form of the co-occurrence dictionary is, for example: High-frequency word A: Target co-occurrence word B of high-frequency word A, Target co-occurrence word C of high-frequency word A, Target co-occurrence word D of high-frequency word A. Among them, the target co-occurrence word B of high-frequency word A, the target co-occurrence word C of high-frequency word A, and the target co-occurrence word D of high-frequency word A are sorted according to the number of co-occurrences with high-frequency word A. For example, they are sorted from high to low according to the number of co-occurrences with high-frequency word A.

[0240] The specific example is as follows:

[0241] Training corpus:

[0242] Sentence 1: The lights shining in the harbor are like colorful fireworks splashing down to the world.

[0243] Sentence 2: The spring breeze is like a colorful pen, outlining the whole world more brightly and colorfully.

[0244] ……

[0245] Sentence n - 1: …

[0246] Sentence n: …

[0247] Screen the high-frequency word groups to obtain ["world"] and ["spring breeze"].

[0248] {

[0249] world: [harbor: 3, fireworks: 2],

[0250] spring breeze: [world: 4, colorful pen: 2],

[0251] }

[0252] That is, the target co-occurring words of the high-frequency word "human world" include "harbor" (with a co-occurrence count of 3) and "fireworks" (with a co-occurrence count of 2). The target co-occurring words of the high-frequency word "spring breeze" include "human world" (with a co-occurrence count of 4) and "colorful pen" (with a co-occurrence count of 2).

[0253] See Figure 7 As shown, the embodiments of the present disclosure provide a text generation method, and this text generation method includes steps L102-L106.

[0254] Step L102: Input the first target word sequence into the co-occurring word sampling model, so as to expand the first target word sequence into a second target word sequence containing target co-occurring words through the co-occurring word sampling model. Among them, the target co-occurring words are the words that co-occur with the words in the first target word sequence in the co-occurrence dictionary.

[0255] The co-occurring word sampling model uses a pre-constructed co-occurrence dictionary to perform co-occurring word sampling, and expands the first target word sequence into a second target word sequence containing target co-occurring words. In one example, methods such as frequency sampling and random sampling can be used to perform co-occurring word sampling.

[0256] In this example, the co-occurrence dictionary can be used to perform co-occurring word sampling and perform random filling of co-occurring words, so as to expand the search path of the input during the inference process of the text generation model.

[0257] Step SL104: Input the second target word sequence into the text generation model trained by the method of any of the foregoing embodiments to obtain at least one third target word sequence.

[0258] The inference process of the text generation model is the reverse engineering of constructing training data. Generally, it is considered that when the results of two consecutive rounds of iteration are the same, the condition for the convergence of the inference result is reached, and the iteration stops.

[0259] Example of the inference process:

[0260] The first target word sequence is [bauhinia], and after co-occurring word filling, the second target word sequence [jasper, bauhinia] is obtained

[0261] The text generation model performs multiple rounds of iteration, and inserts specific words at different positions in each round of iteration. The sequences obtained in each round are:

[0262] (1) [Like jasper hairpin and bauhinia flower]

[0263] (2) [I like jasper hairpin and bauhinia flower.]

[0264] (3) [I like bauhinia flower like jasper most.]

[0265] (4) [But my favorite is the redbud flower that looks like a jade hairpin.]

[0266] (5) [But I like the redbud flower that looks like a jasper hairpin the most.], stop iterating.

[0267] During the fifth round of iteration, the sentence did not change at all, so convergence was completed and the iteration was stopped. The final third target word sequence was [But I like the jasper-like redbud flower the most].

[0268] Step SL106: Output the text consisting of the third target word sequence.

[0269] In one example, when multiple third target word sequences are obtained in step L104, the multiple third target word sequences can be evaluated to filter out the optimal target word sequence from the multiple third target word sequences, and then in step L106, the text consisting of the optimal target word sequence is output to the user.

[0270] In one example, a text evaluation model can be used to evaluate and score multiple third target word sequences. The multiple third target word sequences are sorted from high to low according to their scores, and the top-ranked third target word sequence or several top-ranked third target word sequences are output to the user, allowing the user to obtain a relatively ideal final text.

[0271] In one example, the main evaluation metrics may include fluency, the proportion of deleted class labels during reasoning, etc.

[0272] In one example, after the text generation model generates multiple third target word sequences under constraints, the perplexity index value of each third target word sequence is calculated, and the third target word sequence with the smallest perplexity index value is determined as the optimal target word sequence.

[0273] The perplexity (PPL) metric is used in natural language processing to measure the quality of language models. It estimates the probability of each word appearing in a sentence and normalizes it by sentence length.

[0274]

[0275] In formula 6, S represents the third target word sequence, N is the length of the third target word sequence, that is, the total number of word units in the third target word sequence, 1≤i≤N, W1-W N Represents each word in the third target word sequence. P(w1w2…w N ) represents the total predicted probability of each word in the third target word sequence, P(w i |w1w2…wi-1 ) is based on word W1-W i-1 As a condition, predict the word W i PP(S) is the perplexity index of the third target word sequence. The smaller the value of PP(S), the higher the probability of the third target word sequence appearing, and the more likely it is to provide the text consisting of the third target word sequence to the user.

[0276] <Device for training text generation model>

[0277] According to a fourth aspect of an embodiment of the present disclosure, a device for training a text generation model is provided. The device for training a text generation model includes:

[0278] A first acquisition module is configured to acquire a first training data set; the training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label;

[0279] A second acquisition module is configured to acquire a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels;

[0280] a first training execution module, configured to input the first word sequence into the first text generation model, and use the first label sequence corresponding to the first word sequence as supervision to perform sequence generation task training on the first text generation model, thereby obtaining a second text generation model;

[0281] The second training execution module is used to input the second word sequence into the second text generation model, and use the second label sequence corresponding to the second word sequence as supervision to perform sequence generation task training on the second text generation model to obtain a third text generation model.

[0282] Optionally, the apparatus for training a text generation model further includes a third acquisition module and a third training execution module.

[0283] A third acquisition module is used to acquire a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence;

[0284] The third training execution module is used to input the third word sequence into the third text generation model, and use the rhetoric type label corresponding to the third word sequence as supervision to perform classification task training on the third text generation model to obtain a fourth text generation model.

[0285] Optionally, the apparatus for training a text generation model further includes a fourth acquisition module and a fourth training execution module.

[0286] The fourth acquisition module is used to acquire a third training data set, where the training data in the third training data set includes a third word sequence and a rhetoric type label corresponding to the third word sequence.

[0287] The fourth training execution module is used to use the third training data set to perform classification task training on the first text generation model while performing sequence generation task training on the first text generation model; and calculate the total loss of the first text generation model, and determine whether to end the training of the first text generation model based on the total loss of the first text generation model; the total loss of the first text generation model is the sum of the loss generated by the first text generation model in the sequence generation task training and the loss generated by the first text generation model in the classification task training.

[0288] Optionally, the apparatus for training a text generation model further includes a fifth acquisition module and a fifth training execution module.

[0289] The fifth acquisition module is used to acquire a fourth training data set, where the training data in the fourth training data set includes a fourth word sequence and a rhetoric type label corresponding to the fourth word sequence.

[0290] The fifth training execution module is used to use the fourth training data set to perform classification task training on the second text generation model while performing sequence generation task training on the second text generation model; and calculate the total loss of the second text generation model, and determine whether to end the training of the second text generation model based on the total loss of the second text generation model; the total loss of the second text generation model is the sum of the loss generated by the second text generation model in the sequence generation task training and the loss generated by the second text generation model in the classification task training.

[0291] Optionally, the apparatus for training a text generation model further includes a first decomposition module, a first enhancement processing module and a first construction module.

[0292] The first decomposition module is used to obtain a sample sentence and decompose the sample sentence into an initial word sequence.

[0293] The first enhancement processing module is used to perform data enhancement processing on the initial word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection.

[0294] The first construction module is used to use the initial word sequence and the enhanced word sequence to respectively construct training data in the first training data set.

[0295] Optionally, the apparatus for training a text generation model further includes a second decomposition module, an insertion and replacement module, a second enhancement processing module and a second construction module.

[0296] The second decomposition module is used to obtain a sample sentence and decompose the sample sentence into an initial word sequence.

[0297] The insertion and replacement module is used to insert synonyms and / or replace antonyms into the initial word sequence to obtain at least one intermediate word sequence.

[0298] The second enhancement processing module is used to perform data enhancement processing on the intermediate word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection.

[0299] The second construction module is used to use the intermediate word sequence and the enhanced word sequence to respectively construct training data in the second training data set.

[0300] According to a fifth aspect of an embodiment of the present disclosure, a device for training a text generation model is provided. The device for training a text generation model includes:

[0301] The first acquisition module is used to acquire a first training data set; the training data in the first training data set includes a first word sequence, a first label sequence using the first word sequence as a generation constraint, and the first label sequence includes an insertion class label.

[0302] The second acquisition module is used to acquire a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, and the second label sequence includes an insertion class label and a non-insertion class label.

[0303] The third acquisition module is used to acquire a third training data set, where the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence.

[0304] The training execution module is used to simultaneously train the first text generation model using the first training data set, the second training data set, and the third training data set to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

[0305] <Embodiment of Text Generating Device>

[0306] According to a sixth aspect of an embodiment of the present disclosure, there is provided a text generation device, the text generation device comprising:

[0307] An expansion module is used to input the first target word sequence into a co-occurrence word sampling model to expand the first target word sequence into a second target word sequence containing target co-occurrence words through the co-occurrence word sampling model, wherein the target co-occurrence words are words that co-occur with words in the first target word sequence in the co-occurrence dictionary.

[0308] A generation module is used to input the second target word sequence into the text generation model trained according to the method for training a text generation model of the first aspect of the present disclosure or the method for training a text generation model of the second aspect of the present disclosure to obtain at least one third target word sequence.

[0309] The output module is used to output a text composed of at least one third target word sequence.

[0310] Optionally, the text generation device further includes an evaluation module and a determination module.

[0311] The evaluation module is used to calculate the perplexity index value of each third target word sequence when there are multiple third target word sequences.

[0312] A determination module is configured to determine the third target word sequence with the smallest perplexity index value as the optimal target word sequence.

[0313] The output module is specifically used to output the text composed of the optimal target word sequence.

[0314] <Electronic Equipment Example>

[0315] The present disclosure also provides an electronic device. Figure 8 As shown, the electronic device 30 includes a processor 31 and a memory 32. The memory 32 stores computer instructions. When the computer instructions are executed by the processor 31, the method for training a text generation model or the text generation method disclosed in any of the above embodiments is implemented.

[0316] In a specific example, the electronic device can be an electronic device installed with an intelligent operating system (such as Android, IOS, Windows, Linux, etc.), including but not limited to a portable computer, a desktop computer, a mobile phone, a tablet computer, a server, etc. The electronic device 30 can have a similar Figure 1 The hardware configuration of the electronic device shown.

[0317] <Computer-readable storage medium embodiment>

[0318] The embodiments of the present disclosure also provide a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the method for training a text generation model or the text generation method disclosed in any of the aforementioned embodiments is implemented.

[0319] The embodiments of the present disclosure provide a method and apparatus for training a text generation model, a text generation method and apparatus, an electronic device, and a computer-readable storage medium. The method and apparatus use training data containing insertion labels to perform sequence generation task training on a first text generation model to obtain a second text generation model, so that the second text generation model has the ability to insert words. The method and apparatus use training data containing both insertion labels and non-insertion labels to perform sequence generation task training on the second text generation model to obtain a third text generation model, so that the third text generation model has the ability to insert words and eliminate insertion errors, thereby enhancing the reliability of the text generation model and improving the text generation effect of the text generation model.

[0320] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device and apparatus embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, reference can be made to the descriptions of the method embodiments.

[0321] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0322] The embodiments of the present disclosure may be systems, methods, and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the embodiments of the present disclosure.

[0323] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0324] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0325] The computer program instructions for performing the operation of the embodiments of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing the various aspects of the embodiments of the present disclosure.

[0326] Various aspects of the embodiments of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0327] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0328] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0329] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and a part of the module, program segment or instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0330] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a text generation model, characterized in that: include: Obtain a first training data set; The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label; Obtain a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels; Inputting the first word sequence into a first text generation model, and using the first label sequence corresponding to the first word sequence as supervision, the first text generation model is trained on a sequence generation task to obtain a second text generation model; The second word sequence is input into the second text generation model, and the second label sequence corresponding to the second word sequence is used as supervision to train the second text generation model on a sequence generation task to obtain a third text generation model.

2. The method for training a text generation model according to claim 1, wherein: Also includes: Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence; The third word sequence is input into the third text generation model, and the rhetoric type label corresponding to the third word sequence is used as supervision to perform classification task training on the third text generation model to obtain a fourth text generation model.

3. The method for training a text generation model according to claim 1, wherein: Also includes: Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence; When the first text generation model is trained on the sequence generation task, the first text generation model is simultaneously trained on the classification task using the third training data set; Calculate the total loss of the first text generation model, and determine whether to end the training of the first text generation model based on the total loss of the first text generation model; the total loss of the first text generation model is the sum of the loss generated by the first text generation model in the sequence generation task training and the loss generated by the first text generation model in the classification task training.

4. The method for training a text generation model according to claim 1, wherein: Also includes: Acquire a fourth training data set, where training data in the fourth training data set includes a fourth word sequence and a rhetorical type label corresponding to the fourth word sequence; When the second text generation model is trained on the sequence generation task, the fourth training data set is used to simultaneously train the second text generation model on the classification task; Calculate the total loss of the second text generation model, and determine whether to end the training of the second text generation model based on the total loss of the second text generation model; the total loss of the second text generation model is the sum of the loss generated by the second text generation model in the sequence generation task training and the loss generated by the second text generation model in the classification task training.

5. The method for training a text generation model according to any one of claims 1 to 4, characterized in that: Before obtaining the first training data set, the following steps are also included: Obtain a sample sentence and decompose the sample sentence into an initial word sequence; Performing data enhancement processing on the initial word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection; The training data in the first training dataset are constructed using the initial word sequence and the enhanced word sequence.

6. The method for training a text generation model according to any one of claims 1 to 4, characterized in that: Before obtaining the second training data set, the method further includes: Obtain a sample sentence and decompose the sample sentence into an initial word sequence; Inserting synonyms and / or replacing antonyms into the initial word sequence to obtain at least one intermediate word sequence; Performing data enhancement processing on the intermediate word sequence to obtain at least one enhanced word sequence; the data enhancement processing includes at least one of the following processing methods: synonym replacement, back translation, and noise injection; The training data in the second training dataset are constructed using the intermediate word sequence and the enhanced word sequence.

7. A method for training a text generation model, characterized in that: include: Obtain a first training data set; The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label; Obtain a second training data set; the training data in the second training data set includes a second word sequence and a second label sequence with the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels; Acquire a third training data set, where training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence; The first text generation model is trained simultaneously using the first training data set, the second training data set, and the third training data set to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

8. A text generation method, characterized in that: include: Inputting a first target word sequence into a co-occurrence word sampling model to expand the first target word sequence into a second target word sequence containing target co-occurrence words through the co-occurrence word sampling model, wherein the target co-occurrence words are words that co-occur with words in the first target word sequence in a co-occurrence dictionary; inputting the second target word sequence into a text generation model trained by the method for training a text generation model according to any one of claims 1 to 7 to obtain at least one third target word sequence; Output a text consisting of the third target word sequence.

9. The text generation method according to claim 8, characterized in that In the case where there are multiple third target word sequences, the method further includes: Calculating a perplexity index value for each of the third target word sequences; Determining the third target word sequence with the smallest perplexity index value as the optimal target word sequence; The outputting the text composed of the third target word sequence includes: outputting the text composed of the optimal target word sequence.

10. A device for training a text generation model, characterized in that: include: A first acquisition module, configured to acquire a first training data set; The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label; A second acquisition module is used to acquire a second training data set; The training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels; a first training execution module, configured to input a first word sequence into a first text generation model, and use a first label sequence corresponding to the first word sequence as supervision to perform sequence generation task training on the first text generation model, thereby obtaining a second text generation model; The second training execution module is used to input the second word sequence into the second text generation model, and use the second label sequence corresponding to the second word sequence as supervision to perform sequence generation task training on the second text generation model to obtain a third text generation model.

11. A device for training a text generation model, characterized in that: include: A first acquisition module, configured to acquire a first training data set; The training data in the first training data set includes a first word sequence and a first label sequence using the first word sequence as a generation constraint, wherein the first label sequence includes an insertion class label; A second acquisition module is used to acquire a second training data set; The training data in the second training data set includes a second word sequence and a second label sequence using the second word sequence as a generation constraint, wherein the second label sequence includes insertion class labels and non-insertion class labels; A third acquisition module is configured to acquire a third training data set, wherein the training data in the third training data set includes a third word sequence and a rhetorical type label corresponding to the third word sequence; A training execution module is used to use the first training data set, the second training data set, and the third training data set to simultaneously train the first text generation model to obtain a second text generation model with sequence generation capability and rhetoric type classification capability.

12. A text generation device, characterized in that: include: an expansion module, configured to input a first target word sequence into a co-occurrence word sampling model, so as to expand the first target word sequence into a second target word sequence containing a target co-occurrence word through the co-occurrence word sampling model, wherein the target co-occurrence word is a word that co-occurs with a word in the first target word sequence in a co-occurrence dictionary; a generation module, configured to input the second target word sequence into a text generation model trained by the method for training a text generation model according to any one of claims 1 to 7, to obtain at least one third target word sequence; An output module is used to output a text composed of at least one of the third target word sequences.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer instructions, and is characterized in that when the computer instructions are executed by the processor, the method for training a text generation model according to any one of claims 1 to 7 or the text generation method according to any one of claims 8 to 9 is implemented.

14. A computer-readable storage medium, characterized in that Computer instructions are stored thereon, and when the computer instructions are executed by a processor, they implement the method for training a text generation model described in any one of claims 1-7 or the text generation method described in any one of claims 8-9.

Images