Method for training generation model, method and device for generating text, equipment and medium

Abstract

The present disclosure provides a training method of a generation model, a text generation method, an apparatus, a device and a medium, relates to the field of artificial intelligence, and in particular to the field of text generation. The specific implementation scheme comprises the following steps: determining a first prediction result by a first decoding unit in combination with a real target of a training sample and encoding information output by an encoding unit; determining a reference input value of a second decoding unit from the real target and the first prediction result; predicting the input reference input value by the second decoding unit to obtain a final prediction result, and adjusting model parameters reversely based on the final prediction result. The present disclosure trains the generation model in a two-stage decoding manner, which can guarantee the diversity of the text generated by the generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of text generation, and specifically to a training method for a generation model, a text generation method, an apparatus, an electronic device, a storage medium, and a computer program product. Background Art

[0002] In Internet marketing, a landing page refers to the web page that is displayed to potential users after they click on an advertisement or search using a search engine. The landing page includes attractive language that not only highlights the selling points of the page and improves user search efficiency, but also helps netizens directly access various components and stimulate user conversion. Summary of the Invention

[0003] The present disclosure provides a training method for generating a model, a copywriting generation method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a training method for a generative model is provided, wherein the generative model includes an encoding unit and two decoding units, and the method includes:

[0005] Determine a first prediction result by combining the real target of the training sample and the encoding information output by the encoding unit through the first decoding unit;

[0006] determining a reference input value of a second decoding unit from the true target and the first prediction result;

[0007] The input reference input value is predicted by the second decoding unit to obtain a final prediction result, so that the model parameters are reversely adjusted based on the final prediction result.

[0008] According to one aspect of the present disclosure, a method for generating a document is provided, comprising:

[0009] The text to be processed is transferred to the encoding unit of the generative model to obtain encoding information; wherein the generative model is obtained by training according to any generative model training method disclosed in the present invention;

[0010] The encoded information is used as the input of the second decoding unit of the generation model, and the generated copy is determined based on the output of the second decoding unit.

[0011] According to one aspect of the present disclosure, a training device for a generative model is provided, wherein the generative model includes an encoding unit and two decoding units, and the device includes:

[0012] a first-stage decoding module, configured to determine a first prediction result by combining the true target of the training sample and the encoding information output by the encoding unit through a first decoding unit;

[0013] a sampling module, configured to determine a reference input value of a second decoding unit from a true target and a first prediction result;

[0014] The two-stage decoding module is used to predict the input reference input value through the second decoding unit to obtain a final prediction result, so as to reversely adjust the model parameters based on the final prediction result.

[0015] According to one aspect of the present disclosure, there is provided a document generation device, comprising:

[0016] An encoding module, configured to transfer the text to be processed to an encoding unit of a generative model to obtain encoded information; wherein the generative model is obtained by training according to any generative model training method disclosed herein;

[0017] The generation module is used to use the encoded information as the input of the second decoding unit of the generation model, and determine the generated text according to the output of the second decoding unit.

[0018] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to at least one processor; wherein,

[0021] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the training method of the generation model or the copy generation method of any embodiment of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to enable a computer to execute the training method for generating a generation model or the text generation method of any embodiment of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the training method for a generation model or the copywriting generation method of any embodiment of the present disclosure.

[0024] According to the technology disclosed in the present invention, the generative model is trained by a two-stage decoding method, which can ensure the diversity of the texts generated by the trained generative model.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a flow chart of a training method for a generative model provided according to an embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of another training method for generating a model provided according to an embodiment of the present disclosure;

[0029] Figure 3 is a flowchart of another training method for generating a model provided according to an embodiment of the present disclosure;

[0030] Figure 4 It is a flowchart of a method for generating a document according to an embodiment of the present disclosure;

[0031] Figure 5 1 is a schematic diagram of a structure of a training device for generating a model according to an embodiment of the present disclosure;

[0032] Figure 6 is a structural diagram of a document generation device provided according to an embodiment of the present disclosure;

[0033] Figure 7 It is a block diagram of an electronic device used to implement the training method of the generative model according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] The generative model in the disclosed embodiments is primarily used to generate attractive language for landing pages, such as titles. The disclosed generative model is built on the Transformer framework and includes one encoding unit and two decoding units. The two decoding units share identical structures and parameters, allowing for model training via a two-stage decoding approach. The specific process is described in the following implementation.

[0036] Figure 1This is a flowchart of a generative model training method according to an embodiment of the present disclosure. This embodiment is applicable to training a generative model using a two-stage decoding approach. The method can be performed by a generative model training device implemented using software and / or hardware and integrated into an electronic device.

[0037] For details, see Figure 1 , the training method of the generative model is as follows:

[0038] S101. Determine a first prediction result by combining a real target of a training sample and encoding information output by an encoding unit through a first decoding unit.

[0039] In the disclosed embodiments, when training a generative model, training sample data is first input into the generative model's encoding unit to obtain corresponding encoding information, which primarily serves as the intermediate input for the first decoding unit. The true target of the training sample refers to the labeled knowledge of the sample. During model training, the true target of the training sample is directly used as the initial input for the first decoding unit.

[0040] In an optional embodiment, the first decoding unit obtains the first prediction result through predictions of multiple time steps. In the specific implementation, the true target corresponding to the sub-prediction result generated by the first decoding unit at any time step is first used as the initial input of the first decoding unit at the next time step; for example, the sub-prediction result generated by the generative model at the first time step is "yesterday", and the true target corresponding to the time step is "today", then "today" is used as the initial input of the first decoding unit at the next time step. In this way, the prediction error only occurs in the first time step and will not accumulate to the subsequent ones. That is, using the true target as the initial input of the first decoding unit can ensure that the error will not accumulate. It should be noted that the traditional generative model mainly uses the seq2seq framework. When training the model, the output of the previous hidden state is used as the input of the next hidden state. In the early stage of training, if an extremely poor result occurs in the previous state, all the subsequent states will be affected, resulting in a completely disordered result in the final generated result.

[0041] Furthermore, the encoded information is used as the intermediate input to the first decoding unit at each time step. This generates a model prediction probability distribution at each time step. Based on the probability distribution of the prediction results obtained by the first decoding unit at each time step, a sub-prediction result can be determined for each time step, and the set of sub-prediction results generated at each time step is used as the first prediction result.

[0042] S102: Determine a reference input value of a second decoding unit from the true target and the first prediction result.

[0043] Optionally, in the early stage of model training, the true target is used as the reference input value of the second decoding unit; in the later stage of model training, the first prediction result generated by step S101 is used as the reference input value of the second decoding unit. The advantage of doing so is that in the early stage of model training, the second decoding unit basically uses the true target as the reference value, which can better and faster have the initial generation capability. As the training time increases, the first prediction result output by the first decoding unit is used as the reference value to prevent the model from correcting the output to the true target too early, solve the over-correction problem, and retain diversity. Because the generation model is different from the classification model, the latter is a complete one-hot problem. The task goal of the generation model is not a complete one-to-one alignment. Synonyms or synonyms can be allowed. Instead, it is better to generate multiple synonyms. If only the true target (i.e., groundtruth) is used as the input of the second decoding unit, the model will continue to correct according to the true target, which will kill diversity in advance.

[0044] S103 . Predict the input reference input value through the second decoding unit to obtain a final prediction result, so as to reversely adjust the model parameters based on the final prediction result.

[0045] After determining the reference input value in step S102, the reference input value is input into the second decoding unit for prediction, obtaining a final prediction result. The model parameters are then adjusted inversely based on the final prediction result, primarily by updating the network parameters of the generative model through gradient backpropagation. It should be noted that the gradient backpropagation occurs from the second decoding unit to the encoding unit, without passing through the first decoding unit.

[0046] In the disclosed embodiment, the generative model is trained by a two-stage decoding method, and the true target is used as the input of the first decoding unit to ensure that prediction errors do not accumulate; depending on the training period, the true target or the first prediction result is used as the input of the second decoding unit to avoid the over-correction problem, thereby ensuring the diversity of the generated copy by the trained generative model.

[0047] Figure 2 This is a flow chart of another method for training a generative model according to an embodiment of the present disclosure. The present disclosure embodiment refines the process of determining the reference input value of the second decoding unit on the basis of the above embodiment. Figure 2 , the training method of the generative model is as follows:

[0048] S201: Determine a first prediction result by combining a real target of a training sample and encoding information output by an encoding unit through a first decoding unit.

[0049] S202: Calculate the probability of the true target and the first prediction result being used as reference input values, respectively, based on the number of training rounds and preset hyperparameters.

[0050] S203: Determine a reference input value of the second decoding unit according to the probability calculation result.

[0051] In the disclosed embodiment, the calculation formula for the probability of using the true target as the reference input value of the second decoding unit is as follows: p = β / (β+exp(epoch / β)); wherein β is a preset hyperparameter, and epoch is the number of training rounds. The probability of the first prediction result being used as the reference input value is 1-p. It can be seen from the probability calculation formula that the probability p will gradually decrease with the increase of the number of training rounds epoch, that is, the probability of using the output of the first decoding unit as the reference input of the second decoding unit is increasing. The advantage of doing so is that in the early stage of model training, the second decoding unit basically uses the true target as the reference value, which can better and faster have the initial generation capability. As the training time increases, the first prediction result output by the first decoding unit is used as the reference value to prevent the model from correcting the output to the true target too early, solve the over-correction problem, and retain diversity.

[0052] S204 . Predict the input reference input value through the second decoding unit to obtain a final prediction result, so as to reversely adjust the model parameters based on the final prediction result.

[0053] In the disclosed embodiment, in the initial stage of model training, the second decoding unit basically completely uses the real target as a reference value, which can provide a better and faster initial generation capability. As the training time increases, the first prediction result output by the first decoding unit is used as a reference value to prevent the model from correcting the output to the real target too early, solve the over-correction problem, and retain diversity.

[0054] Figure 3 This is a flow chart of another method for training a generative model according to an embodiment of the present disclosure. The present disclosure embodiment refines the process of determining the respective sub-prediction results of each time step based on the probability distribution of the prediction results obtained by the first decoding unit at each time step on the basis of the above embodiment. Figure 3 , the method further comprises the steps of:

[0055] S301 . For any time step, take the prediction result with the highest probability in the probability distribution of the prediction result obtained by the first decoding unit at the time step as the sub-prediction result of the time step.

[0056] S302 : For any time step, according to the probability distribution of the prediction results obtained by the first decoding unit at the time step, select one of the top N prediction results in probability order as the sub-prediction result of the time step.

[0057] In the disclosed embodiment, the output of the first decoding unit is converted into a probability distribution through softmax at each time step, and the one with the highest probability is used as the sub-prediction result of the time step, which can ensure the accuracy of the model prediction; and any one of the multiple prediction results with higher probabilities is selected as the sub-prediction result of the time step, which can ensure the diversity of the model prediction when the prediction accuracy does not change much.

[0058] Furthermore, for any time step, the first decoding unit calculates the probability distribution of the prediction result of the first decoding unit at the time step based on the preset noise parameter g and temperature parameter t. Optionally, the probability distribution formula of the first decoding unit at the time step is as follows: Among them, z i is the input to a normal softmax, t is the hyperparameter temperature, g = -log(-log(u)), and u follows the uniform (0, 1) form. It should be noted that some noise is added before calculating the probability distribution to give the direct softmax probability output a certain value. Increasing the temperature t also controls the smoothness of the softmax. Higher temperatures produce smoother probability distributions, while lower temperatures produce sharper distributions, closer to one-hot probability distributions. During actual training, the temperature can be gradually lowered to gradually approach the true discrete distribution.

[0059] Figure 4 This is a flowchart of a method for generating text according to an embodiment of the present disclosure. This embodiment is applicable to generating text using a trained generative model. The method can be performed by a text generation device implemented in software and / or hardware and integrated into an electronic device.

[0060] See also Figure 4 , the copywriting generation method is as follows:

[0061] S401. Transfer the text to be processed to the encoding unit of the generation model to obtain encoding information; wherein the generation model is obtained by training according to any training method of the generation model disclosed in the present invention.

[0062] S402: Using the encoded information as input to a second decoding unit of the generation model, and determining the generated text based on the output of the second decoding unit.

[0063] A generative model can be trained through the above embodiments. For the specific training process, please refer to the above embodiments and will not be described in detail here. On this basis, the generative model can be directly used to predict the text to be processed, wherein the text content in the landing page of the text to be processed can be optionally used, and the purpose of the prediction can optionally be to generate a title based on the text content of the landing page. In the specific implementation, the coding information is first obtained from the coding unit of the generative model of the text to be processed, and the coding information is used as the input of the second decoding unit of the generative model, and the generated text is determined according to the output of the second decoding unit. It should be noted that in the stage of using the trained generative model for prediction, only the second decoding unit needs to be used, that is, there is no need for two-stage decoding during the use of the model.

[0064] In the disclosed embodiment, the trained generation model can be used to achieve the purpose of generating diverse copywriting from the text to be processed.

[0065] Figure 5 Schematic diagram of the structure of the training device of the generative model according to the embodiment of the present disclosure. The generative model includes an encoding unit and two decoding units. This embodiment is applicable to the case of training the generative model by a two-stage decoding method. Figure 5 As shown, the device specifically includes:

[0066] A first-stage decoding module 501 is configured to determine a first prediction result by combining the real target of the training sample and the encoding information output by the encoding unit through a first decoding unit;

[0067] a sampling module 502 for determining a reference input value of a second decoding unit from the true target and the first prediction result;

[0068] The two-stage decoding module 503 is configured to predict the input reference input value through a second decoding unit to obtain a final prediction result, so as to reversely adjust the model parameters based on the final prediction result.

[0069] Based on the above embodiment, optionally, the first-stage decoding module includes:

[0070] An initial input determination submodule, configured to use the true target corresponding to the sub-prediction result generated by the first decoding unit at any time step as the initial input of the first decoding unit at the next time step;

[0071] an intermediate input determination submodule, configured to use the encoded information as the intermediate input of the first decoding unit at each time step;

[0072] The first-stage decoding submodule is used to determine the sub-prediction results of each time step according to the probability distribution of the prediction results obtained by the first decoding unit at each time step, and use the set of sub-prediction results generated at each time step as the first prediction result.

[0073] Based on the above embodiment, optionally, the one-stage decoding submodule is further configured to:

[0074] For any time step, the prediction result obtained by the first decoding unit at that time step with the highest probability in the probability distribution is taken as the sub-prediction result of that time step; or,

[0075] For any time step, according to the probability distribution of the prediction results obtained by the first decoding unit at the time step, any one of the top N prediction results in probability order is selected as the sub-prediction result of the time step.

[0076] Based on the above embodiment, optionally, the method further includes:

[0077] The probability distribution calculation module is used to calculate the probability distribution of the prediction result of the first decoding unit at any time step based on the preset noise parameter and temperature parameter.

[0078] Based on the above embodiment, optionally, the sampling module includes:

[0079] A probability calculation submodule is used to calculate the probability of the true target and the first prediction result being used as reference input values, respectively, based on the number of training rounds and preset hyperparameters;

[0080] The determination submodule is used to determine the reference input value of the second decoding unit according to the probability calculation result.

[0081] The training device for the generative model provided in the embodiment of the present disclosure can execute the training method for the generative model provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method. For any content not described in detail in this embodiment, please refer to the description in any method embodiment of the present disclosure.

[0082] Figure 6 : is a structural diagram of a copywriting generation device according to an embodiment of the present disclosure. The generation model includes an encoding unit and two decoding units. This embodiment is applicable to the case where a copywriting is generated by a trained generation model. Figure 6 As shown, the device specifically includes:

[0083] An encoding module 601 is used to transfer the text to be processed to the encoding unit of the generation model to obtain encoding information; wherein the generation model is obtained by training according to any one of the methods of claims 1-5;

[0084] The generation module 602 is configured to use the encoded information as input to a second decoding unit of the generation model, and determine the generated text based on the output of the second decoding unit.

[0085] The document generation device provided in the embodiment of the present disclosure can execute the document generation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, please refer to the description of any method embodiment of the present disclosure.

[0086] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0087] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0088] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0091] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the training method for generating a model. For example, in some embodiments, the training method for generating a model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method for generating a model described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the training method for generating a model by any other appropriate means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0096] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0097] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0098] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0099] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for a generative model, wherein the generative model includes an encoding unit and two decoding units, the method comprising: Determining a first prediction result by combining a true target of a training sample and encoding information output by the encoding unit through a first decoding unit, wherein the training sample is a training text and the true target of the training sample is annotated knowledge of the training text; determining a reference input value of a second decoding unit from the true target and the first prediction result; The second decoding unit predicts the input reference input value to obtain a final prediction result, so that the model parameters are reversely adjusted based on the final prediction result, wherein the final prediction result is the text corresponding to the training text.

2. The method according to claim 1, wherein Determining a first prediction result by combining the real target of the training sample and the encoding information output by the encoding unit through the first decoding unit includes: Using the true target corresponding to the sub-prediction result generated by the first decoding unit at any time step as the initial input of the first decoding unit at the next time step; Using the encoded information as an intermediate input of the first decoding unit at each time step; According to the probability distribution of the prediction results obtained by the first decoding unit at each time step, the sub-prediction results of each time step are determined, and the set of sub-prediction results generated at each time step is used as the first prediction result.

3. The method according to claim 2, wherein: Determining a sub-prediction result for each time step according to the probability distribution of the prediction result obtained by the first decoding unit at each time step includes: For any time step, the prediction result obtained by the first decoding unit at the time step with the highest probability in the probability distribution is used as the sub-prediction result of the time step; or For any time step, according to the probability distribution of the prediction results obtained by the first decoding unit at the time step, any one of the top N prediction results in probability order is selected as the sub-prediction result of the time step.

4. The method according to claim 2, further comprising: For any time step, the first decoding unit calculates the probability distribution of the prediction result of the first decoding unit at the time step based on the preset noise parameter and temperature parameter.

5. The method according to claim 1, wherein The determining of a reference input value of a second decoding unit from the true target and the first prediction result includes: Calculating, based on the number of training rounds and preset hyperparameters, the probability of the true target and the first prediction result being used as the reference input value; A reference input value of the second decoding unit is determined according to the probability calculation result.

6. A copywriting generation method, comprising: The text to be processed is transferred to the encoding unit of the generative model to obtain encoding information; wherein the generative model is obtained by training according to any one of the methods described in claims 1-5; The encoded information is used as input to a second decoding unit of the generation model, and a generated text is determined based on an output of the second decoding unit.

7. A training device for a generative model, the generative model comprising an encoding unit and two decoding units, the device comprising: a first-stage decoding module, configured to determine a first prediction result by combining a true target of a training sample and encoding information output by the encoding unit through a first decoding unit, wherein the training sample is a training text and the true target of the training sample is annotated knowledge of the training text; a sampling module, configured to determine a reference input value of a second decoding unit from the real target and the first prediction result; A two-stage decoding module is used to predict the input reference input value through the second decoding unit to obtain a final prediction result, so as to reversely adjust the model parameters based on the final prediction result, wherein the final prediction result is the text corresponding to the training text.

8. The device according to claim 7, wherein The first-stage decoding module includes: an initial input determination submodule, configured to use the true target corresponding to the sub-prediction result generated by the first decoding unit at any time step as the initial input of the first decoding unit at the next time step; an intermediate input determination submodule, configured to use the encoded information as an intermediate input of the first decoding unit at each time step; A one-stage decoding submodule is used to determine the sub-prediction results of each time step according to the probability distribution of the prediction results obtained by the first decoding unit at each time step, and to use the set of sub-prediction results generated at each time step as the first prediction result.

9. The device according to claim 8, wherein The one-stage decoding submodule is further configured to: For any time step, the prediction result obtained by the first decoding unit at the time step with the highest probability in the probability distribution is used as the sub-prediction result of the time step; or For any time step, according to the probability distribution of the prediction results obtained by the first decoding unit at the time step, any one of the top N prediction results in probability order is selected as the sub-prediction result of the time step.

10. The apparatus according to claim 8, further comprising: The probability distribution calculation module is used to calculate the probability distribution of the prediction result of the first decoding unit at any time step based on the preset noise parameters and temperature parameters.

11. The device according to claim 7, wherein The sampling module includes: A probability calculation submodule, configured to calculate the probability of the true target and the first prediction result being used as the reference input value, respectively, based on the number of training rounds and preset hyperparameters; A determination submodule is configured to determine a reference input value of the second decoding unit according to a probability calculation result.

12. A copywriting generation device, comprising: An encoding module, configured to transfer the text to be processed to an encoding unit of a generative model to obtain encoding information; wherein the generative model is obtained by training according to the method of any one of claims 1 to 5; A generation module is used to use the encoded information as input to the second decoding unit of the generation model, and determine the generated text according to the output of the second decoding unit.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5 or 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5 or 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5 or 6.

Images