Model training and text generation method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a model training and text generation method, device, electronic device and storage medium. The model training method includes: obtaining a first training sample, the first training sample includes a first text and a context mask text of the first text; based on the first text as input and the context mask text as a supervision condition, a text generation model based on a time-series text sequence is trained. In the scheme of the embodiment of the present invention, since the text generation model uses the context mask text of the first text as a supervision condition for training, the text generation model can learn the context information in the first text, so that the trained model can generate more accurate text.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a model training and text generation method, device, electronic device and storage medium. Background Art

[0002] At present, text generation models such as text summarization generation models and machine translation models have been applied to many natural language processing (NLP) tasks. Generally speaking, text generation models are end-to-end models based on neural networks and generate text in a certain time order, for example, generating sentence text word by word from left to right.

[0003] However, there is still room for improvement in the accuracy and reliability of text generated by this model. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a model training and text generation method, device, electronic device and storage medium to at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present invention, a model training method is provided, comprising: obtaining a first training sample, wherein the first training sample comprises a first text and a context mask text of the first text; based on the first text as input and based on the context mask text as a supervision condition, training a text generation model based on a time-series text sequence.

[0006] According to a second aspect of an embodiment of the present invention, a text generation method is provided, comprising: obtaining a target text; inputting the target text into a text generation model based on a temporal text sequence to obtain a generated text, wherein the text generation model is trained according to the model training method described in the first aspect.

[0007] According to a third aspect of an embodiment of the present invention, a model training device is provided, comprising: a sample acquisition module, which acquires a first training sample, wherein the first training sample comprises a first text and a context mask text of the first text; and a model training module, which trains a text generation model based on a temporal text sequence based on the first text as input and the context mask text as a supervision condition.

[0008] According to a fourth aspect of an embodiment of the present invention, a text generation device is provided, comprising: a text acquisition module, which acquires a target text; and a text generation module, which inputs the target text into a text generation model based on a temporal text sequence to obtain a generated text, wherein the text generation model is trained according to the model training method described in the first aspect.

[0009] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0010] According to a sixth aspect of an embodiment of the present invention, there is provided a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0011] In the scheme of the embodiment of the present invention, since the text generation model based on the time-series text sequence adopts the context mask text of the first text as the supervision condition for training, the text generation model can learn the context information in the first text while learning the time-series features of the text, so that the trained model can generate more accurate text. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1A A flowchart of the steps of a model training method according to an embodiment of the present invention;

[0014] Figure 1B for Figure 1A A schematic diagram of an example of a model training method;

[0015] Figure 1C for Figure 1A A schematic diagram of another example of a model training method;

[0016] Figure 2 A flowchart of a text generation method according to another embodiment of the present invention;

[0017] Figure 3 is a structural block diagram of a model training device according to another embodiment of the present invention;

[0018] Figure 4 is a structural block diagram of a text generation device according to another embodiment of the present invention;

[0019] Figure 5 FIG. 4 is a schematic structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0021] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.

[0022] The text generation model based on a temporal text sequence in an embodiment of the present invention can be a sequence-to-sequence (Seq2Seq) framework, specifically, can be implemented based on an encoder-decoder (Encoder-Decoder).

[0023] As some examples, text generation models include but are not limited to Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN), Pointer-Generator-Network (PGN), etc.

[0024] Taking the encoder-decoder structure as an example, the encoder in the text generation model is used to generate a semantic intermediate representation based on the input text, and the output text is obtained through the decoder. For different training tasks, the meaning of the learned semantic intermediate representation is different. Since the text input and text output are highly correlated with the text temporal features in natural language processing, the encoder and decoder in the text generation model each adopt a temporal-related structure, and the training samples usually need to be serialized. For example, the text temporal characteristics need to be reflected in the sentence sequence to reflect which characters are input first and which are input later, or which characters are output first and which are output later. In various text generation models including translation models or summary generation models, based on such text temporal features, on the one hand, implicit semantic information can be effectively learned, which improves the reasoning ability and prediction accuracy of the trained text generation model; on the other hand, such text generation models overemphasize such text temporal features, which makes the reasoning ability of the trained model still have room for further improvement.

[0025] Figure 1AIt is a flowchart of the steps of the model training method according to one embodiment of the present invention. The scheme of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc. For example, in the model training stage, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds. Accordingly, in the inference stage, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations. Figure 1A Model training methods include:

[0026] S110: Acquire a first training sample, where the first training sample includes a first text and a context mask text of the first text.

[0027] It should be understood that the text in the embodiments of the present invention only includes text in the form of characters (including words and Chinese characters, etc.), sentences, paragraphs, chapters, etc. The text in the training sample may not be processed by word embedding (characters as units), and word embedding processing may be performed before participating in training; the text in the training sample may be text processed by word embedding, and the text after word embedding processing may directly participate in model training.

[0028] It should also be understood that the first training sample can be used for the context task, and the context mask text is the text obtained by masking the first text (for example, part of the information in the first text is covered or removed). For the context mask text, the position of the part of the first text that is not covered or removed remains unchanged in the first text, and the covered or removed part participates in the training of the model (adjustment of the model parameters) or the calculation of the objective function of the corresponding training task.

[0029] S120: Based on the first text as input and the context mask text as a supervision condition, a text generation model based on a temporal text sequence is trained.

[0030] It should be understood that the text generation model can be used for text generation prediction, for example, text generation prediction includes predictions such as summary generation, translation generation, etc. For example, in order to enable the trained model to perform summary generation processing, the first text can be a chapter text, and the context mask text can be a context mask text of the chapter text, for example, a part of the chapter obtained by covering or removing part of the text information in the chapter text; the chapter summary can also be used as the context mask text, and the chapter obtained by expanding (expanding) the text information of the chapter summary can be used as the first text. For another example, in order to enable the trained model to perform translation processing to translate the first language into the second language, the text in the first language can be used as the first text, and part of the information in the first text can be covered or removed to obtain the remaining first text as the context mask text, or the text in the second language can be used as the context mask text, and the context mask text can be expanded (expanded) to obtain the first text.

[0031] It is also understood that in order to enable the trained model to perform summary generation processing, the text generation model can be trained based on the first text as input and the context mask text of the summary text of the first text as output. In order to enable the trained model to perform translation processing, the text generation model can be trained based on the text of the first language as input and the context mask text of the text of the second language as output.

[0032] In the scheme of the embodiment of the present invention, since the text generation model based on the time-series text sequence adopts the context mask text of the first text as the supervision condition for training, the text generation model can learn the context information in the first text while learning the time-series features of the text, so that the trained model can generate more accurate text.

[0033] In other words, compared to the way of training text generation models based on word sequences in sentences, humans consider the entire context before writing when dealing with these tasks. Compared with generating words one by one from left to right, the content generated by considering the context is obviously more reasonable. Based on this discovery, this scheme uses a bidirectional language model to propose a training method that incorporates bidirectional context information into the existing generation framework. The method proposed in this scheme can effectively enable the neural network model to simulate the human generation method and improve the quality and accuracy of the generated content.

[0034] The following will be combined Figure 1B and 1C An exemplary implementation of the model training method is described in detail:

[0035] Figure 1B for Figure 1AA schematic diagram of an example of a model training method. In this example, a text generation training task and a context training task are used to train a text generation model based on a time-series text sequence. Each task can be trained in at least one training stage, and the training of different tasks can be spaced apart. The training samples in different training stages of the same training task can be the same or different. Although the training stage of the context training task (training stage ①) and the training stage of the text generation training task (training stage ②) are illustrated, this is merely indicative, and different training tasks can be performed alternately. For example, after training stage ②, the training of the context training task can continue. For another example, Figure 1B The training process shown is performed multiple times. Preferably, the training task before deploying the model for reasoning is a text generation task.

[0036] In this example, in the context training task corresponding to the training stage ①, the first training sample is used to perform training based on the first objective function. The first objective function can be determined by the first text in the first training sample and the context mask text of the first text. Specifically, when the first text and its context mask text are sentences, the context mask text y / y m ={y1, ..., yj, ..., yJ}; first text x ={x1, ..., xi, ..., xI}; first objective function L M = -log P(y m |y / y m , x; θ G ), where y / y m represents the context mask text, x represents the first text, θ G represents the parameters in training on the context task.

[0037] In this example, in the text generation training task corresponding to the training stage ②, the second training sample is used for training based on the second objective function, and the second objective function may include the second text and the label text in the second training sample. Specifically, when the second text and the label text are multi-character texts such as sentences or paragraphs (examples of words in the Germanic language family as characters), the label text y = {y1, ..., y j , …, y J}, J is a positive integer greater than 1, 1≤j≤J; the second text x={x1,…,x i , …, x I}, I is a positive integer greater than 1, 1≤i≤I; the second objective function L G = -log P(y i |y <j , x; θ G ). Among them, y <jRepresents the part before the jth character in the label text; y i is the jth character in the label text; x represents the second text; θ G Represents the parameters in the text generation training task.

[0038] In other words, the above different training tasks use corresponding objective functions to train the text generation model. When a training task includes multiple training stages, different training stages of different tasks use different objective functions, and different training stages of the same training task use the same objective function.

[0039] Figure 1C for Figure 1A Schematic diagram of another example of a model training method. In this example, a first training sample and a second training sample are used to jointly train a text generation model. The objective function of the joint training includes a first function and a second function. The objective function can be determined based on the first function and the second function. For example, the objective function can be a weighted sum of the first function and the second function. More specifically, the objective function can be the sum of the first function and the second function.

[0040] In this example, the text generation model is trained using two training tasks: context training task and text generation task. Different from the above example, the two training tasks do not have their own independent objective functions, but are trained based on the same objective function. This helps the model learn text sequence features and context features as much as possible, avoiding focusing on only one feature, further improving the training effect, and also improving the prediction accuracy of the trained model.

[0041] Specifically, the first training text includes the first text and its context mask text, and the second training text includes the second text and the label text. m ={y1,…,y j , …, y J}, J is a positive integer greater than 1, 1≤j≤J; the first text x(1)={x1,…,x i , …, x I}, I is a positive integer greater than 1, 1≤i≤I; label text y={y1,…,y j , …, y J}; Second text x(2) = {x1, ..., x i , …, x I}; The first function L M = -log P(y m |y / y m , x(1);θ G ); the second function L G = -log P(y i |y<j , x(2);θ G );Y k = argmax P(Y i |Y <j , x; θ G ). Among them, the objective function L = L G +L M .

[0042] Among them, y / y m represents the context mask text, x represents the first text, θ G represents the parameters in the training of the context task. <j Represents the part before the jth character in the label text; y i is the jth character in the label text; x(2) represents the second text; θ G Represents the parameters in the text generation training task.

[0043] When performing joint training based on the objective function, the context training task and the text generation task may each include at least one training stage.

[0044] For example, when calculating the current first function, the calculation can be based on the first text and context mask text in the previous context training task. Specifically, the current first function can be determined based on the first training samples of all training stages of the context training task before calculating the current first function, and then the current first function is summed with the calculated second function to obtain the target function, and then the next parameter adjustment of the context training task is performed.

[0045] For another example, when calculating the current second function, the calculation can be performed based on the second text and label text in the previous text generation training task. Specifically, the current second function can be determined based on the second training samples of all training stages of the text generation training task before calculating the second function, and then the current second function is summed with the calculated first function to obtain the target function, and then the next parameter adjustment of the text generation training task is performed.

[0046] In other examples, the method also includes: obtaining a second training sample, the second training sample includes a second text and a label text of the second text, and the method also includes: training the text generation model based on the second text as input and the label text as a supervision condition, which is conducive to the text generation model learning the temporal characteristics of the second text.

[0047] In other examples, the first training sample is used for a context learning task, the second training sample is used for a text generation task, and the training of a text generation model based on a time-series text sequence includes: using the first training sample and the second training sample to jointly train the text generation model based on the context learning task and the text generation task, so that the model can simultaneously learn the text temporal features and text context features in the training samples.

[0048] In addition, in another implementation of the present invention, the text generation model based on the time-series text sequence is trained, including: based on the second text as input, based on the context mask text of the label text as the output supervision condition, the text generation model is trained. Thus, the model learns the context features and the time sequence features through one training task.

[0049] Specifically, the context mask text of the label text can be determined based on the semantic dependency of the label text. For example, the high-importance arguments of the subject, predicate, and object in the label text can be masked instead of retained, and other arguments such as attributives and complements can be retained instead of masked, so as to obtain the context mask text of the label text. The text generation model includes a structure based on a temporal text sequence, so it can better reflect the relationship between the high-importance arguments of the subject, predicate, and object. Retaining other arguments in the context mask text can improve the effect of context learning, which is conducive to complementing the structure based on the temporal text sequence and improving the prediction accuracy and generalization ability of the model.

[0050] In other examples, the first training sample and the second training sample are used to jointly train the text generation model based on the context learning task and the text generation task, including: according to the objective function, the text generation model is jointly trained based on the context learning task and the text generation task, wherein the objective function includes a first function and a second function, and during the joint training process, the first function is determined by the first text and the context mask sample, and the second function is determined by the second text and the label text, which is conducive to the model learning text sequence features and context features as much as possible, avoiding focusing on learning only one feature, thereby further improving the training effect and the prediction accuracy of the trained model.

[0051] In other examples, the objective function is the sum of the first function and the second function, which improves the computational efficiency of the objective function.

[0052] In other examples, the first function is determined according to the conditional probability of the context mask sample based on the first text, and the second function is determined according to the conditional probability of the label text based on the second text, which improves the loss evaluation of the objective function and is conducive to improving the convergence speed of the model parameters.

[0053] In another implementation of the present invention, the text generation model is jointly trained based on the context learning task and the text generation task, including: based on the context learning task and the text generation task, the text generation model is alternately trained, which is beneficial to improving the training flexibility of the model. For example, if the text generation model is expected to be used for more context prediction tasks, the proportion of the context learning task can be made larger, or the proportion of the first training sample can be made larger, or the context learning task can be used as the last training task; for another example, if the text generation model is expected to be used for more text generation tasks, the proportion of the text generation task can be made larger, or the proportion of the second training sample can be made larger, or the text generation task can be used as the last training task.

[0054] In some other examples, obtaining the first training sample includes: obtaining a first text; and performing mask processing on the first text to obtain a context mask text of the first text.

[0055] In some other examples, the masking of the first text to obtain the context masked text of the first text includes: based on the semantic dependency information in the first text, performing context masking on the first text to obtain the context masked text. The semantic dependency information indicates the relationship between the arguments in the text, and also indicates the importance ranking between the arguments, which is helpful to clarify the goal of the context masking.

[0056] For example, the arguments with high importance of the subject, predicate and object can be masked instead of retained, and other arguments such as attributives and complements can be retained instead of masked, so as to obtain context masked text. The text generation model includes a structure based on a temporal text sequence, so it can better reflect the relationship between the arguments with high importance of the subject, predicate and object. Retaining other arguments in the context masked text can improve the effect of context learning, which is conducive to complementing the structure based on a temporal text sequence and improving the prediction accuracy and generalization ability of the model.

[0057] Figure 2It is a flowchart of the steps of the text generation method according to another embodiment of the present invention. The scheme of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc. For example, in the model training stage, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds. Accordingly, in the inference stage, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations. Figure 2 The text generation method includes:

[0058] S210: Acquire target text.

[0059] S220: Input the target text into a text generation model based on a temporal text sequence to obtain a generated text.

[0060] In the scheme of the embodiment of the present invention, since the text generation model based on the time-series text sequence adopts the context mask text of the first text as the supervision condition for training, the text generation model can learn the context information in the first text while learning the time-series features of the text, so that the trained model can generate more accurate text.

[0061] Figure 3 It is a structural block diagram of a model training device according to another embodiment of the present invention. The solution of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc. For example, in the model training stage, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds. Accordingly, in the inference stage, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations. Figure 3 The text generation device comprises:

[0062] The sample acquisition module 310 acquires a first training sample, where the first training sample includes a first text and a context mask text of the first text.

[0063] The model training module 320 trains a text generation model based on a temporal text sequence based on the first text as input and the context mask text as a supervision condition.

[0064] In the scheme of the embodiment of the present invention, since the text generation model based on the time-series text sequence adopts the context mask text of the first text as the supervision condition for training, the text generation model can learn the context information in the first text while learning the time-series features of the text, so that the trained model can generate more accurate text.

[0065] In another implementation of the present invention, the method further includes: obtaining a second training sample, the second training sample includes a second text and a label text of the second text, and the method further includes: training the text generation model based on the second text as input and based on the label text as a supervision condition.

[0066] In another implementation of the present invention, the first training sample is used for a context learning task, the second training sample is used for a text generation task, and the training of a text generation model based on a temporal text sequence includes: using the first training sample and the second training sample to jointly train the text generation model based on the context learning task and the text generation task.

[0067] In another implementation of the present invention, the first training sample and the second training sample are used to perform joint training on the text generation model based on the context learning task and the text generation task, including: according to the objective function, the text generation model is jointly trained based on the context learning task and the text generation task, wherein the objective function includes a first function and a second function, and during the joint training process, the first function is determined by the first text and the context mask sample, and the second function is determined by the second text and the label text.

[0068] In another implementation of the present invention, the objective function is the sum of the first function and the second function.

[0069] In another implementation of the present invention, the first function is determined according to the conditional probability of the context mask sample based on the first text, and the second function is determined according to the conditional probability of the label text based on the second text.

[0070] In another implementation of the present invention, the text generation model is jointly trained based on the context learning task and the text generation task, including: alternately training the text generation model based on the context learning task and the text generation task.

[0071] In another implementation of the present invention, the acquisition module is specifically used to: acquire a first text; and perform mask processing on the first text to obtain a context mask text of the first text.

[0072] In another implementation of the present invention, the acquisition module is specifically used to: perform context mask processing on the first text based on semantic dependency information in the first text to obtain the context mask text.

[0073] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0074] Figure 4 It is a structural block diagram of a text generation device according to another embodiment of the present invention. The solution of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc. For example, in the model training stage, a computing device (for example, a data center) configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds. Accordingly, in the inference stage, a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations. Figure 4 The text generation device comprises:

[0075] The text acquisition module 410 acquires the target text.

[0076] The text generation module 420 inputs the target text into a text generation model based on a temporal text sequence to obtain a generated text. The text generation model is trained according to the above-mentioned model training method.

[0077] In the scheme of the embodiment of the present invention, since the text generation model based on the time-series text sequence adopts the context mask text of the first text as the supervision condition for training, the text generation model can learn the context information in the first text while learning the time-series features of the text, so that the trained model can generate more accurate text.

[0078] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0079] Reference Figure 5 , shows a schematic structural diagram of an electronic device according to another embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0080] like Figure 5 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0081] in:

[0082] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0083] The communication interface 504 is used to communicate with other electronic devices or servers.

[0084] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above method embodiment.

[0085] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0086] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0087] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0088] The program 510 may be specifically configured to enable the processor 502 to perform the following operations:

[0089] Acquire a first training sample, wherein the first training sample includes a first text and a context mask text of the first text; train a text generation model based on a temporal text sequence based on the first text as input and the context mask text as a supervision condition;

[0090] Alternatively, a target text is obtained; the target text is input into a text generation model based on a temporal text sequence to obtain a generated text.

[0091] In addition, the specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0092] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0093] The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0094] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0095] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations of the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention. The patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A model training method, comprising: Acquire a first training sample, wherein the first training sample includes a first text and a context mask text of the first text, wherein the context mask text is generated based on semantic dependency information in the first text, and the semantic dependency information is used to characterize the relationship between arguments in the first text; Based on the first text as input and the context mask text as a supervision condition, a text generation model based on a temporal text sequence is trained.

2. The method according to claim 1, wherein: The method further comprises: Acquire a second training sample, where the second training sample includes a second text and a label text of the second text. The method further includes: The text generation model is trained based on the second text as input and the label text as a supervision condition.

3. The method according to claim 2, wherein: The first training sample is used for a context learning task, the second training sample is used for a text generation task, and the training of a text generation model based on a temporal text sequence includes: The first training sample and the second training sample are used to perform joint training on the text generation model based on the context learning task and the text generation task.

4. The method according to claim 3, wherein: The adopting the first training sample and the second training sample to perform joint training on the text generation model based on the context learning task and the text generation task includes: According to the objective function, the text generation membrane is jointly trained based on the context learning task and the text generation task, wherein the objective function includes a first function and a second function. During the joint training process, the first function is determined by the first text and the context mask text, and the second function is determined by the second text and the label text.

5. The method according to claim 4, wherein: The objective function is the sum of the first function and the second function.

6. The method according to claim 4, wherein: The first function is determined according to the conditional probability of the context mask text based on the first text, and the second function is determined according to the conditional probability of the label text based on the second text.

7. The method according to claim 3, wherein: The text generation model is jointly trained based on the context learning task and the text generation task, including: Based on the context learning task and the text generation task, the text generation model is alternately trained.

8. The method according to claim 1, wherein: The obtaining of the first training sample comprises: Get the first text; Masking is performed on the first text to obtain context masked text of the first text.

9. The method according to claim 8, wherein: The performing masking processing on the first text to obtain a context masked text of the first text includes: Based on the semantic dependency information in the first text, context masking is performed on the first text to obtain the context masked text.

10. A text generation method, comprising: Get the target text; The target text is input into a text generation model based on a temporal text sequence to obtain a generated text, wherein the text generation model is trained according to the model training method according to any one of claims 1-9.

11. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 10.

12. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images