Model pre-training method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a model pre-training method, device, equipment, storage medium and program product, which relate to the field of artificial intelligence technology, specifically the field of natural language processing and deep learning technology. A specific implementation of the method includes: using sample natural language text to perform a first-stage training on an initial generation model to obtain a generation model, wherein sample pseudo-natural language text is generated in the first-stage training; adding an initial discriminant model to the generation model to obtain an initial pre-trained language model; using sample natural language text and sample pseudo-natural language text to perform a second-stage training on the initial pre-trained language model to obtain a pre-trained language model, wherein the pre-trained language model is used to generate pseudo-natural language text and to discriminate the authenticity of natural language text and pseudo-natural language text. This implementation provides a model pre-training method, which improves the quality of model generation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of natural language processing and deep learning technology. Background Art

[0002] In recent years, with the continuous development of natural language processing technology, generative pre-trained language models based on large corpus and large-scale parameters (such as GPT3 (General Pre-trained Transformer-3, third-generation general pre-trained transformer)) have gradually come into people's view. These models can generate rich texts given various prefixes and prompts of input without fine-tuning, and can be widely used in various fields such as question answering, writing, and dialogue.

[0003] However, compared with natural language, the content generated by existing generative pre-training models still has problems such as inappropriate wording, inconsistencies, and language repetition. Summary of the invention

[0004] The embodiments of the present disclosure provide a model pre-training method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a model pre-training method, comprising: performing a first-stage training on an initial generation model using sample natural language text to obtain a generation model, wherein sample pseudo-natural language text is generated in the first-stage training; adding an initial discriminant model to the generation model to obtain an initial pre-trained language model; performing a second-stage training on the initial pre-trained language model using sample natural language text and sample pseudo-natural language text to obtain a pre-trained language model, wherein the pre-trained language model is used for both generating pseudo-natural language text and discriminating the authenticity of natural language text and pseudo-natural language text.

[0006] In a second aspect, an embodiment of the present disclosure proposes a text generation method, comprising: obtaining a natural language text segment; inputting the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and an authenticity score, wherein the pre-trained language model is trained using the method described in the first aspect; and selecting a pseudo natural language text from the plurality of pseudo natural language texts for output according to the authenticity score.

[0007] In the third aspect, an embodiment of the present disclosure proposes a model pre-training device, comprising: a first training module, configured to perform a first-stage training on an initial generation model using sample natural language text to obtain a generation model, wherein sample pseudo-natural language text is generated in the first-stage training; an adding module, configured to add an initial discriminant model to the generation model to obtain an initial pre-trained language model; a second training module, configured to perform a second-stage training on the initial pre-trained language model using sample natural language text and sample pseudo-natural language text to obtain a pre-trained language model, wherein the pre-trained language model is used for both generating pseudo-natural language text and discriminating the authenticity of natural language text and pseudo-natural language text.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a text generation device, comprising: an acquisition module, configured to acquire a natural language text segment; a generation module, configured to input the natural language text segment into a pre-trained language model to obtain a plurality of pseudo-natural language texts and an authenticity score, wherein the pre-trained language model is trained using the device described in the third aspect; and an output module, configured to select a pseudo-natural language text from the plurality of pseudo-natural language texts for output according to the authenticity score.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect or the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute a method as described in any implementation of the first aspect or the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect or the second aspect.

[0012] The model pre-training method provided by the embodiment of the present disclosure introduces the task of distinguishing true and false texts and a large amount of generated pseudo texts for data enhancement during the training process. During the self-supervised adversarial training of true and false texts, the model finds the difference between true and false texts, making the distribution of pseudo texts generated by the model closer to the real text, thereby further accelerating the convergence of the model and improving the quality of model generation.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0015] Figure 1 is a flowchart of an embodiment of a model pre-training method according to the present disclosure;

[0016] Figure 2 It is a schematic diagram of the model training stage;

[0017] Figure 3 is a flowchart of another embodiment of the model pre-training method according to the present disclosure;

[0018] Figure 4 is a flow chart of an embodiment of a text generation method according to the present disclosure;

[0019] Figure 5 It is a schematic diagram of the prediction stage of the model;

[0020] Figure 6 is a flowchart of another embodiment of the text generation method according to the present disclosure;

[0021] Figure 7 is a structural schematic diagram of an embodiment of a model pre-training device according to the present disclosure;

[0022] Figure 8 is a structural schematic diagram of an embodiment of a text generation device according to the present disclosure;

[0023] Fig. 9 It is a block diagram of an electronic device used to implement the model pre-training method or text generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 A process 100 of an embodiment of a model pre-training method according to the present disclosure is shown. The model pre-training method comprises the following steps:

[0027] Step 101, use sample natural language text to perform the first stage training on the initial generation model to obtain the generation model.

[0028] In this embodiment, the training process of the model pre-training method can be divided into two stages. In the first stage, the execution subject of the model pre-training method can use the sample natural language text to perform the first stage training on the initial generation model to obtain the generation model.

[0029] In practice, in the first stage of training, sample pseudo natural language texts can be generated. Usually, the sample natural language texts are input into the initial generation model, and the sample pseudo natural language texts can be output. Based on the difference between the sample pseudo natural language texts and the sample natural language texts, the parameters of the initial generation model can be adjusted so that the distribution of the sample pseudo natural language texts output by the model is closer to the input sample natural language texts. The closer the distribution of the sample pseudo natural language texts is to the sample natural language texts, the higher the quality of the sample pseudo natural language texts is, and the better the performance of the generation model is.

[0030] The sample natural language text may be a large-scale natural language text collected in advance. Generally, the quality of these texts is high, with appropriate wording, coherence, and concise language.

[0031] Wherein, the generative model can be obtained by self-supervising training the initial generative model using a machine learning method and a sample natural language text. Usually, the various parameters of the initial generative model (for example, weight parameters and bias parameters) can be initialized with some different small random numbers. "Small random numbers" are used to ensure that the model will not enter a saturated state due to excessive weights, thereby causing training failure, and "different" is used to ensure that the model can learn normally. The parameters of the initial generative model can be continuously adjusted during the training process until a generative model is trained in which the distribution of the generated pseudo-text is closer to the distribution of the input real text. For example, a BP (Back Propagation) algorithm or an SGD (Stochastic Gradient Descent) algorithm can be used to adjust the parameters of the initial generative model.

[0032] Step 102, adding an initial discriminant model to the generative model to obtain an initial pre-trained language model.

[0033] In this embodiment, the execution subject can add an initial discriminant model to the generative model to obtain an initial pre-trained language model. In this way, the task of distinguishing true from false text is introduced in the training process. Similarly, the various parameters of the initial discriminant model can be initialized with some different small random numbers.

[0034] Step 103 , performing second-stage training on the initial pre-trained language model using the sample natural language text and the sample pseudo-natural language text to obtain a pre-trained language model.

[0035] In this embodiment, the training process of the model pre-training method can be divided into two stages. In the second stage, the execution subject can use the sample natural language text and the sample pseudo natural language text to perform the second stage training on the initial pre-trained language model to obtain the pre-trained language model. The pre-trained language model can be used to generate pseudo natural language text and to determine the authenticity of natural language text and pseudo natural language text.

[0036] In practice, in the second stage of training, the task of distinguishing true and false texts and a large number of high-quality sample pseudo natural language texts can be introduced for data enhancement. Usually, the sample natural language text and the sample pseudo natural language text are input into the initial discriminant model together, and the initial discriminant model can score the sample natural language text and the sample pseudo natural language text respectively, and output a authenticity score between 0 and 1. For the generative model, it is hoped that the distribution of the generated sample pseudo natural language text is infinitely close to the sample natural language text. For the initial discriminant model, it is hoped that no matter how real the generated sample pseudo natural language text is, the initial model can always distinguish it from the sample natural language text. The second stage of training is a game process between the generative model and the initial discriminant model. Based on the difference between the authenticity label and the authenticity score of the sample natural language text and the sample pseudo natural language text, the parameters of the generative model and the initial discriminant model are adjusted together, so that the distribution of the pseudo text generated by the generative model is infinitely close to the input real text, so that the obtained discriminant model can always distinguish the pseudo text generated by the generative model from the real text. At this time, a pre-trained language model including a generative model and a discriminant model is obtained. The sample natural language text is real text, and the value of its authenticity label is 1. The sample pseudo natural language text is pseudo text, and the value of its authenticity label is 0.

[0037] The model pre-training method provided by the embodiment of the present disclosure introduces the task of distinguishing true and false texts and a large amount of generated pseudo texts for data enhancement during the training process. During the self-supervised adversarial training of true and false texts, the model finds the difference between true and false texts, making the distribution of pseudo texts generated by the model closer to the real text, thereby further accelerating the convergence of the model and improving the quality of model generation.

[0038] For ease of understanding, Figure 2 A schematic diagram of the model training stage is shown. The model training stage is divided into two stages. In the first stage STAGE 1, the sample natural language text T is input into the generation model Model_0, and the sample pseudo natural language text T′ is output. Based on the difference between the sample pseudo natural language text T′ and the sample natural language text T, the parameters of the generation model Model_0 are adjusted. In the first stage STAGE 2, a discriminant model is added to the generation model Model_0 to obtain the pre-trained language model Model_1. The sample natural language text T and the sample pseudo natural language text T′ are input into the discriminant model in the pre-trained language model Model_1 together, and the sample natural language text T and the sample pseudo natural language text T′ are judged as true or false. Based on the authenticity judgment result, the parameters of the pre-trained language model Model_1 are adjusted.

[0039] Continue to refer Figure 3 , which shows a process 300 of another embodiment of the model pre-training method according to the present disclosure. The model pre-training method includes the following steps:

[0040] Step 301: Divide the sample natural language text into a sequence of sample natural language text segments.

[0041] In this embodiment, the execution subject of the model pre-training method can divide the sample natural language text into a sequence of sample natural language text segments. For example, for the sample natural language text ABCD, it can be divided into a sequence of sample natural language text segments (A, B, C, D). Among them, A, B, C and D are all sample natural language text segments. The division method includes average division, random division, etc.

[0042] Step 302: input the sample natural language text segments in the sample natural language text segment sequence into the initial generation model in sequence to generate sample pseudo natural language text.

[0043] In this embodiment, the execution subject may sequentially input the sample natural language text segments in the sample natural language text segment sequence into the initial generation model to generate sample pseudo natural language text.

[0044] Typically, the initial generation model can predict subsequent text segments based on the input prefix text segments, and then generate pseudo text. The prediction granularity is finer, thereby improving the quality of the generated pseudo text. For example, after A in the sample natural language text segment sequence is input into the initial generation model, its subsequent B can be predicted based on A. After B in the sample natural language text segment sequence is input into the initial generation model, its subsequent C can be predicted based on B or AB. After C in the sample natural language text segment sequence is input into the initial generation model, its subsequent D can be predicted based on C or ABC. After D in the sample natural language text segment sequence is input into the initial generation model, the end mark can be predicted based on D or ABCD.

[0045] In some embodiments, the sample natural language text is divided into a sequence of sample natural language text segments with a total number of segments being N (N is a positive integer). For the i-th (i is a positive integer, and i≤N) sample natural language text segment in the sequence of sample natural language text segments, it is input into the initial generation model; the initial generation model is used to make a prediction based on the first i sample natural language text segments in the sequence of sample natural language text segments to obtain the i+1-th sample pseudo natural language text segment. The subsequent text segment is predicted in combination with all the previous text segments, so that the subsequent text segment is associated with all the previous text segments, so that the predicted pseudo text is more realistic and natural, and the problems of inconsistency and language repetition are avoided as much as possible.

[0046] Step 303: Based on the difference between the sample pseudo natural language text and the sample natural language text, the parameters of the initial generation model are adjusted to obtain a generation model.

[0047] In this embodiment, the execution subject may adjust the parameters of the initial generation model based on the difference between the sample pseudo natural language text and the sample natural language text to obtain the generation model.

[0048] Generally speaking, the closer the distribution of the sample pseudo-natural language text output by the model is to the sample natural language text, the higher the quality of the sample pseudo-natural language text is and the better the performance of the generation model is.

[0049] Step 304, adding an initial discriminant model to the generative model to obtain an initial pre-trained language model.

[0050] In this embodiment, the specific operation of step 304 is already described in Figure 1 In the illustrated embodiment, step 102 is described in detail and will not be repeated here.

[0051] Step 305: input the sample natural language text and the sample pseudo natural language text into the initial discriminant model to obtain the authenticity scores of the sample natural language text and the sample pseudo natural language text.

[0052] In this embodiment, the execution subject may input the sample natural language text and the sample pseudo natural language text into the initial discriminant model to obtain the authenticity scores of the sample natural language text and the sample pseudo natural language text. The authenticity score has a value between 0 and 1. The larger the authenticity score, the more authentic the text.

[0053] Step 306 , based on the difference between the authenticity labels and the authenticity scores of the sample natural language text and the sample pseudo-natural language text, adjust the parameters of the generative model and the initial discriminant model to obtain a pre-trained language model.

[0054] In this embodiment, the above-mentioned execution subject can adjust the parameters of the generation model and the initial discriminant model based on the difference between the authenticity labels and authenticity scores of the sample natural language text and the sample pseudo-natural language text to obtain a pre-trained language model.

[0055] In practice, for the generative model, it is hoped that the distribution of the generated sample pseudo-natural language text is infinitely close to the sample natural language text. For the initial discriminant model, it is hoped that no matter how real the generated sample pseudo-natural language text is, the initial model can always distinguish it from the sample natural language text. The second stage of training is the game process between the generative model and the initial discriminant model. Based on the difference in the authenticity label and authenticity score of the sample natural language text and the sample pseudo-natural language text, the parameters of the generative model and the initial discriminant model are adjusted together, so that the distribution of the pseudo-text generated by the generative model is infinitely close to the input real text, so that the obtained discriminant model can always distinguish the pseudo-text generated by the generative model from the real text. At this time, a pre-trained language model including a generative model and a discriminant model is obtained. Among them, the sample natural language text is real text, and the value of its authenticity label is 1. The sample pseudo-natural language text is pseudo text, and the value of its authenticity label is 0.

[0056] from Figure 3 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 300 of the model pre-training method in this embodiment highlights the model training step. Therefore, the scheme described in this embodiment divides the sample natural language text into segments and inputs them into the initial generation model in sequence in the first stage, predicts the subsequent text segments based on the input prefix text segments, and then generates pseudo-text. The prediction granularity is finer, thereby improving the quality of the generated pseudo-text. In the second stage, the generation model and the initial discriminant model are allowed to compete, so that the distribution of the pseudo-text generated by the generation model is infinitely close to the input real text, so that the obtained discriminant model can always distinguish the pseudo-text generated by the generation model from the real text.

[0057] Further references Figure 4, which shows a process 400 of an embodiment of a text generation method according to the present disclosure. The text generation method comprises the following steps:

[0058] Step 401, obtaining a natural language text segment.

[0059] In this embodiment, the execution subject of the text generation method can obtain a natural language text segment. The natural language text segment can be a prefix or prompt information. Here, it is necessary to generate a complete natural language text related to the prefix or prompt information according to the prefix or prompt information.

[0060] Step 402 , input the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and authenticity scores.

[0061] In this embodiment, the above-mentioned execution entity can input the natural language text segment into a pre-trained language model to obtain multiple pseudo natural language texts and authenticity scores.

[0062] Among them, the pre-trained language model can be adopted Figure 1 or Figure 3 The pre-trained language model may include a generation model and a discrimination model, which is used to generate pseudo natural language text and to discriminate the authenticity of natural language text and pseudo natural language text.

[0063] Generally, the generative model can generate multiple pseudo-natural language texts, and the discriminative model can score the multiple pseudo-natural language texts respectively, and output a authenticity score between 0 and 1. For the generative model, the closer the distribution of the pseudo-natural language text generated by the generative model is to the real text, the higher the score given by the discriminative model. For example, when a natural language text segment A is input into the generative model, its subsequent text segment can be predicted to be B or B' based on A. Its subsequent text segment can be predicted to be C based on AB, and its subsequent text segment can be predicted to be C' based on AB'. Its subsequent text segment can be predicted to be D based on ABC, and its subsequent text segment can be predicted to be D' based on AB'C'. At this time, two pseudo-natural language text segments can be obtained: ABCD and AB'C'D'. When ABCD and AB'C'D' are input into the discriminative model, the authenticity score of ABCD and the authenticity score of AB'C'D' can be obtained.

[0064] Step 403 , selecting a pseudo natural language text from a plurality of pseudo natural language texts for output according to the authenticity score.

[0065] In this embodiment, the execution subject may select a pseudo natural language text from a plurality of pseudo natural language texts for output according to the authenticity score. Typically, the execution subject selects the pseudo natural language text with the highest authenticity score for output.

[0066] In the text generation method provided by the embodiments of the present disclosure, the pre-trained language model can detect the difference between true and false texts, so that the distribution of the pseudo text generated by the model is closer to the true text, thereby improving the quality of model generation. In addition, the pre-trained language model can score the generated pseudo texts and select the output pseudo texts according to the authenticity score, thereby effectively reducing the fluctuation of sampling stability, so that the authenticity of the output pseudo texts is kept within a certain range, and the pseudo texts with low authenticity scores are filtered out.

[0067] For ease of understanding, Figure 5 A schematic diagram of the model prediction stage is shown. A natural language text segment is input into the pre-trained language model Model_1. The generative model in the pre-trained language model Model_1 outputs multiple pseudo natural language texts s1, s2, s3, s4, ... The discriminative model in the pre-trained language model Model_1 performs authenticity judgment and scoring on the multiple pseudo natural language texts s1, s2, s3, s4, ..., and outputs the authenticity scores of the multiple pseudo natural language texts s1, s2, s3, s4, ... The pseudo natural language text with the highest authenticity score is selected for output.

[0068] Further references Figure 6 , which shows a process 600 of another embodiment of a text generation method according to the present disclosure. The text generation method comprises the following steps:

[0069] Step 601, obtaining a natural language text segment.

[0070] In this embodiment, the specific operation of step 601 is already described in Figure 4 In the illustrated embodiment, step 401 is described in detail and will not be repeated here.

[0071] Step 602: input the natural language text segment into the generation model in the pre-trained language model to obtain a plurality of pseudo natural language texts.

[0072] In this embodiment, the above-mentioned execution entity can input the natural language text fragment into the generation model in the pre-trained language model to obtain multiple pseudo natural language texts.

[0073] Generally, the generative model can generate multiple pseudo natural language texts. For example, when a natural language text segment A is input into the generative model, its subsequent text segment can be predicted to be B or B' based on A. Its subsequent text segment can be predicted to be C based on AB, and its subsequent text segment can be predicted to be C' based on AB'. Its subsequent text segment can be predicted to be D based on ABC, and its subsequent text segment can be predicted to be D' based on AB'C'. At this point, two pseudo natural language text segments can be obtained: ABCD and AB'C'D'.

[0074] Step 603: Input the multiple pseudo natural language texts into the discriminant model in the pre-trained language model to obtain authenticity scores of the multiple pseudo natural language texts.

[0075] In this embodiment, the above-mentioned execution entity can input multiple pseudo natural language texts into the discriminant model in the pre-trained language model to obtain the authenticity scores of the multiple pseudo natural language texts.

[0076] Generally, the discriminant model can score the generated multiple pseudo natural language texts respectively and output a authenticity score between 0 and 1. For the generative model, the closer the distribution of the pseudo natural language text generated by the generative model is to the real text, the higher the score given by the discriminant model. For example, when a natural language text segment A is input into the generative model, its subsequent text segment can be predicted to be B or B' based on A. According to AB, its subsequent text segment can be predicted to be C, and according to AB', its subsequent text segment can be predicted to be C'. According to ABC, its subsequent text segment can be predicted to be D, and according to AB'C', its subsequent text segment can be predicted to be D'. At this time, two pseudo natural language text segments can be obtained: ABCD and AB'C'D'. When ABCD and AB'C'D' are input into the discriminant model, the authenticity score of ABCD and the authenticity score of AB'C'D' can be obtained.

[0077] In some embodiments, the pseudo natural language text to be generated includes M (M is a positive integer) segments. The first segment of the pseudo natural language text is the input natural language text segment. The second segment of the pseudo natural language text is the predicted first pseudo natural language text segment. For the first pseudo natural language text segment, the natural language text segment is input into the generation model, and the first pseudo natural language text segment is output, wherein the first pseudo natural language text segment is the next natural language text segment of the natural language text segment predicted by the generation model. For the j+1th (j is a positive integer, and j≤M-1) pseudo natural language text segment, the j-1 pseudo natural language text segments before the natural language text segment are spliced ​​and input into the generation model, and the jth pseudo natural language text segment is output. Finally, the natural language text segment and the M-1 pseudo natural language text segments are spliced ​​to generate pseudo natural language text. Combine all the previous text segments to predict the subsequent text segment, so that the subsequent text segment is associated with all the previous text segments, so that the predicted pseudo text is more real and natural, and the problems of inconsistency and language repetition are avoided as much as possible.

[0078] Step 604 , selecting a pseudo natural language text from a plurality of pseudo natural language texts for output according to the authenticity score.

[0079] In this embodiment, the specific operation of step 604 is already described in Figure 4 In the illustrated embodiment, step 403 is described in detail and will not be repeated here.

[0080] from Figure 6 It can be seen that Figure 4 Compared with the corresponding embodiment, the process 600 of the text generation method in this embodiment highlights the model prediction step. Therefore, the pre-trained language model in the scheme described in this embodiment includes a generation model and a discriminant model, the generation model generates multiple pseudo texts, the discriminant model performs authenticity judgment and scoring on the generated multiple pseudo texts, and selects the output pseudo text according to the authenticity score, which effectively reduces the fluctuation of sampling stability and improves the quality of the output pseudo text.

[0081] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides a first embodiment of a model pre-training device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0082] like Figure 7As shown, the model pre-training device 700 of this embodiment may include: a first training module 701, an adding module 702 and a second training module 703. The first training module 701 is configured to perform a first-stage training on the initial generation model using sample natural language texts to obtain a generation model, wherein sample pseudo natural language texts are generated in the first-stage training; the adding module 702 is configured to add an initial discriminant model to the generation model to obtain an initial pre-trained language model; the second training module 703 is configured to perform a second-stage training on the initial pre-trained language model using sample natural language texts and sample pseudo natural language texts to obtain a pre-trained language model, wherein the pre-trained language model is used for both generating pseudo natural language texts and discriminating the authenticity of natural language texts and pseudo natural language texts.

[0083] In this embodiment, in the model pre-training device 700, the specific processing of the first training module 701, the adding module 702 and the second training module 703 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps 101 - 103 in the corresponding embodiment are not repeated here.

[0084] In some optional implementations of the present embodiment, the first training module 701 includes: a division submodule, configured to divide the sample natural language text into a sequence of sample natural language text segments; a generation submodule, configured to input the sample natural language text segments in the sequence of sample natural language text segments into the initial generation model in sequence to generate sample pseudo natural language text; and an adjustment submodule, configured to adjust the parameters of the initial generation model based on the difference between the sample pseudo natural language text and the sample natural language text to obtain a generation model.

[0085] In some optional implementations of the present embodiment, the generation submodule is further configured to: input the i-th sample natural language text segment in the sample natural language text segment sequence into the initial generation model, where i is a positive integer, and i≤N, and N is the total number of sample natural language text segments; use the initial generation model to make predictions based on the first i sample natural language text segments in the sample natural language text segment sequence to obtain the i+1-th sample pseudo natural language text segment.

[0086] In some optional implementations of the present embodiment, the second training module 703 is further configured to: input the sample natural language text and the sample pseudo-natural language text into the initial discriminant model to obtain the authenticity scores of the sample natural language text and the sample pseudo-natural language text; based on the difference between the authenticity labels and the authenticity scores of the sample natural language text and the sample pseudo-natural language text, adjust the parameters of the generation model and the initial discriminant model to obtain a pre-trained language model.

[0087] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides a first embodiment of a text generation device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0088] like Figure 8 As shown, the text generation device 800 of this embodiment may include: an acquisition module 801, a generation module 802 and an output module 803. The acquisition module 801 is configured to acquire a natural language text segment; the generation module 802 is configured to input the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and authenticity scores, wherein the pre-trained language model is a pre-trained language model. Figure 7 The embodiment of the device shown is trained; the output module 803 is configured to select a pseudo natural language text from multiple pseudo natural language texts for output according to the authenticity score.

[0089] In the text generation device 800, the specific processing of the acquisition module 801, the generation module 802 and the output module 803 and the technical effects thereof can be referred to in Figure 4 The relevant descriptions of steps 401-403 in the corresponding embodiment are not repeated here.

[0090] In some optional implementations of the present embodiment, the generation module 802 includes: a generation submodule, configured to input a natural language text segment into a generation model in a pre-trained language model to obtain a plurality of pseudo-natural language texts; and a discrimination submodule, configured to input a plurality of pseudo-natural language texts into a discrimination model in the pre-trained language model to obtain authenticity scores of the plurality of pseudo-natural language texts.

[0091] In some optional implementations of the present embodiment, the generation submodule is further configured to: input a natural language text segment into a generation model, and output a first pseudo natural language text segment, wherein the first pseudo natural language text segment is the next natural language text segment of the natural language text segment predicted by the generation model; concatenate the natural language text segment and the first j-1 pseudo natural language text segments predicted by the pre-trained language model and input them into the generation model, and output the jth pseudo natural language text segment, wherein j is a positive integer and j≤M-1, wherein M is the total number of segments of the pseudo sample natural language text; concatenate the natural language text segment and M-1 pseudo natural language text segments to generate pseudo natural language text.

[0092] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0093] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0094] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0095] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0096] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a model pre-training method or a text generation method. For example, in some embodiments, the model pre-training method or the text generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the model pre-training method or the text generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the model pre-training method or the text generation method in any other suitable manner (for example, by means of firmware).

[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model pre-training method, comprising: Performing a first-stage training on an initial generative model using sample natural language text to obtain a generative model, wherein sample pseudo natural language text is generated in the first-stage training; Adding an initial discriminant model to the generative model to obtain an initial pre-trained language model; Performing a second phase of training on the initial pre-trained language model using the sample natural language text and the sample pseudo-natural language text to obtain a pre-trained language model, wherein the pre-trained language model is used for both generating pseudo-natural language text and determining the authenticity of natural language text and pseudo-natural language text; The first stage of training the initial generation model using the sample natural language text to obtain the generation model includes: Dividing the sample natural language text into a sequence of sample natural language text segments; Inputting the i-th sample natural language text segment in the sample natural language text segment sequence into the initial generation model, where i is a positive integer, and i≤N, and N is the total number of segments of the sample natural language text; using the initial generation model to make predictions based on the first i sample natural language text segments in the sample natural language text segment sequence, obtaining the i+1-th sample pseudo natural language text segment, and generating a sample pseudo natural language text; Based on the difference between the sample pseudo natural language text and the sample natural language text, the parameters of the initial generation model are adjusted to obtain the generation model.

2. The method according to claim 1, wherein: The second stage of training the initial pre-trained language model using the sample natural language text and the sample pseudo natural language text to obtain the pre-trained language model includes: Inputting the sample natural language text and the sample pseudo natural language text into the initial discriminant model to obtain authenticity scores of the sample natural language text and the sample pseudo natural language text; Based on the difference between the authenticity labels and authenticity scores of the sample natural language text and the sample pseudo-natural language text, the parameters of the generative model and the initial discriminant model are adjusted to obtain the pre-trained language model.

3. A text generation method, comprising: Obtain natural language text snippets; Inputting the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and authenticity scores, wherein the pre-trained language model is trained using the method according to any one of claims 1 to 2; A pseudo natural language text is selected from the plurality of pseudo natural language texts for output according to the authenticity score.

4. The method according to claim 3, wherein: The step of inputting the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and authenticity scores includes: Inputting the natural language text segment into the generation model in the pre-trained language model to obtain the plurality of pseudo natural language texts; The multiple pseudo natural language texts are input into the discriminant model in the pre-trained language model to obtain authenticity scores of the multiple pseudo natural language texts.

5. The method according to claim 4, wherein: The step of inputting the natural language text segment into the generative model in the pre-trained language model to obtain the plurality of pseudo natural language texts includes: Inputting the natural language text segment into the generation model, and outputting a first pseudo natural language text segment, wherein the first pseudo natural language text segment is the next natural language text segment of the natural language text segment predicted by the generation model; The natural language text segment and the first j-1 pseudo natural language text segments predicted by the pre-trained language model are spliced ​​and input into the generation model, and the j-th pseudo natural language text segment is output, where j is a positive integer and j≤M-1, where M is the total number of segments of the pseudo sample natural language text; The natural language text segment and M-1 pseudo natural language text segments are concatenated to generate the pseudo natural language text.

6. A model pre-training device, comprising: A first training module is configured to perform a first-stage training on an initial generation model using sample natural language text to obtain a generation model, wherein sample pseudo natural language text is generated in the first-stage training; An adding module is configured to add an initial discriminant model to the generative model to obtain an initial pre-trained language model; a second training module configured to perform a second phase of training on the initial pre-trained language model using the sample natural language text and the sample pseudo natural language text to obtain a pre-trained language model, wherein the pre-trained language model is used for both generating pseudo natural language text and determining the authenticity of natural language text and pseudo natural language text; Wherein, the first training module includes: a division submodule, configured to divide the sample natural language text into a sequence of sample natural language text segments; The generation submodule is configured to input the i-th sample natural language text segment in the sample natural language text segment sequence into the initial generation model, wherein i is a positive integer, and i≤N, and N is the total number of segments of the sample natural language text; use the initial generation model to make a prediction based on the first i sample natural language text segments in the sample natural language text segment sequence to obtain the i+1-th sample pseudo natural language text segment, and generate a sample pseudo natural language text; The adjustment submodule is configured to adjust the parameters of the initial generation model based on the difference between the sample pseudo natural language text and the sample natural language text to obtain the generation model.

7. The device according to claim 6, wherein: The second training module is further configured to: Inputting the sample natural language text and the sample pseudo natural language text into the initial discriminant model to obtain authenticity scores of the sample natural language text and the sample pseudo natural language text; Based on the difference between the authenticity labels and authenticity scores of the sample natural language text and the sample pseudo-natural language text, the parameters of the generative model and the initial discriminant model are adjusted to obtain the pre-trained language model.

8. A text generation device, comprising: an acquisition module, configured to acquire a natural language text segment; A generation module, configured to input the natural language text segment into a pre-trained language model to obtain a plurality of pseudo natural language texts and authenticity scores, wherein the pre-trained language model is trained using the device according to any one of claims 6 to 7; The output module is configured to select a pseudo natural language text from the plurality of pseudo natural language texts for output according to the authenticity score.

9. The device according to claim 8, wherein: The generation module comprises: A generation submodule is configured to input the natural language text segment into a generation model in the pre-trained language model to obtain the plurality of pseudo natural language texts; The discriminant submodule is configured to input the multiple pseudo natural language texts into the discriminant model in the pre-trained language model to obtain authenticity scores of the multiple pseudo natural language texts.

10. The device according to claim 9, wherein: The generation submodule is further configured to: Inputting the natural language text segment into the generation model, and outputting a first pseudo natural language text segment, wherein the first pseudo natural language text segment is the next natural language text segment of the natural language text segment predicted by the generation model; The natural language text segment and the first j-1 pseudo natural language text segments predicted by the pre-trained language model are spliced ​​and input into the generation model, and the j-th pseudo natural language text segment is output, where j is a positive integer and j≤M-1, where M is the total number of segments of the pseudo sample natural language text; The natural language text segment and M-1 pseudo natural language text segments are concatenated to generate the pseudo natural language text.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images