Text generation method and apparatus

Abstract

The embodiment of the application discloses a text generation method and device, and relates to the technical field of artificial intelligence. The main technical scheme comprises the following steps: obtaining a first text; inputting the first text into an outline prediction model to obtain an outline generated by the outline prediction model for the first text as a second text, the outline prediction model being obtained by pre-training based on a pre-training language model; inputting the first text and the second text into a text generation model to obtain a third text generated by the text generation model using the first text and the second text, the third text being continuous to the first text, the text generation model being obtained by pre-training based on a pre-training language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text generation method and apparatus. Background Technology

[0002] Open-ended text generation is a significant research challenge in natural language understanding and generation tasks, with broad application prospects in scenarios such as dialogue, story, and comment generation. Open-ended text generation involves creating a coherent text from a given text (usually a sentence) that allows the sentences to continue.

[0003] Existing open generation methods generally focus on improving the local coherence of the generated text, while neglecting the logic and dependencies between parts of the generated text. This often results in problems such as content repetition and plot stagnation, which seriously affect the quality of the generated text. Summary of the Invention

[0004] In view of this, this application provides a text generation method and apparatus to improve the quality of generated text.

[0005] This application provides the following solution:

[0006] Firstly, a text generation method is provided, the method comprising:

[0007] Get the first text;

[0008] The first text is input into the outline prediction model, and the outline generated by the outline prediction model for the first text is obtained as the second text. The outline prediction model is pre-trained based on a pre-trained language model.

[0009] The first text and the second text are input into the text generation model to obtain a third text that follows the first text, generated by the text generation model using the first text and the second text. The text generation model is pre-trained based on a pre-trained language model.

[0010] According to one feasible embodiment of this application, the outline prediction model is pre-trained in the following manner:

[0011] Obtain first training data, which includes multiple summary texts;

[0012] The first N sentences of the summary text are used as input to the outline prediction model, where N is a preset positive integer. The summary text is used as the target output of the outline prediction model to train the outline prediction model. The outline prediction model includes: a first encoder and a first decoder using a pre-trained language model; the first encoder extracts a first text feature representation from the input text, and the first decoder uses the first text feature representation to predict the outline.

[0013] According to one achievable method in the embodiments of this application, the text generation model is pre-trained in the following manner:

[0014] Obtain second training data, which includes multiple article samples and summary text of each article sample;

[0015] The first N sentences of the article sample and the summary text of the article sample are used as input to the text generation model, where N is a preset positive integer. The article sample is used as the target output of the text generation model to train the text prediction model. The text prediction model includes: a second encoder and a second decoder using a pre-trained language model; the second encoder extracts a second text feature representation from the input text, and the second encoder is used to generate a third text that continues the first N sentences of the article sample using the second text feature representation.

[0016] According to one achievable method in the embodiments of this application, the training objectives used to train the text prediction model include a first training objective and a second training objective;

[0017] The first training objective is to minimize the difference between the third text generated by the text generation model and the corresponding article sample;

[0018] The second training objective is to minimize the similarity between sentences in the third text generated by the text generation model whose distance is less than or equal to a first preset distance, and to maximize the similarity between sentences in the third text output by the text generation model whose distance is greater than or equal to a second preset distance, wherein the first preset distance is less than the second preset distance.

[0019] According to one achievable method in the embodiments of this application, when training the text prediction model, the third text output by the text generation model is further input into the pre-trained coherence evaluation model to obtain the coherence degree of adjacent text pairs output by the coherence evaluation model.

[0020] The training objectives used to train the text prediction model include a first training objective and a third training objective; the first training objective is to minimize the difference between the third text generated by the text generation model and the corresponding article sample; the third training objective is to maximize the coherence of the output of the coherence evaluation model.

[0021] The coherence evaluation model is removed after training.

[0022] According to one achievable method in the embodiments of this application, the coherence evaluation model is pre-trained in the following manner:

[0023] Obtain third training data, which includes positive sample pairs and negative sample pairs. The positive sample pairs are adjacent sentences in the text samples, and the negative sample pairs are non-adjacent sentences in the text samples.

[0024] The coherence evaluation model is trained using the training data. The coherence evaluation model includes a third encoder and a mapping layer. The third encoder is used to extract a third textual feature representation from the input positive or negative sample pairs. The mapping layer is used to obtain the coherence of the positive or negative sample pairs using the third textual feature representation. The training objective is to maximize the coherence of positive sample pairs and minimize the coherence of negative sample pairs.

[0025] According to one achievable method in an embodiment of this application, the first text is a statement input by the user, and the third text includes:

[0026] In intelligent dialogue scenarios, a response sent to the user based on the first text input; or,

[0027] The evaluation text generated after the user enters the first text in the evaluation component is detected; or,

[0028] The story text generated after the user entered the first text in the text publishing component was detected.

[0029] Secondly, a text generation apparatus is provided, the apparatus comprising:

[0030] The text acquisition unit is configured to acquire the first text;

[0031] The outline prediction unit is configured to input the first text into the outline prediction model and obtain the outline generated by the outline prediction model for the first text as the second text. The outline prediction model is pre-trained based on a pre-trained language model.

[0032] The text generation unit is configured to input the first text and the second text into a text generation model, and obtain a third text that follows the first text generated by the text generation model using the first text and the second text. The text generation model is pre-trained based on a pre-trained language model.

[0033] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0034] According to a fourth aspect, an electronic device is provided, characterized in that it comprises:

[0035] One or more processors; and

[0036] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0037] According to the specific embodiments provided in this application, this application can achieve the following technical effects:

[0038] 1) This application first generates an outline for the input first text, and then predicts subsequent texts based on the first text and the outline to generate a third text. By introducing new knowledge and plot information through the outline, it provides guidance on logic and contextual dependencies for the generation of the third text, alleviating problems such as content repetition and plot stagnation that occur due to excessive freedom in the generated third text, and improving the quality of the generated third text.

[0039] 2) Using the summary text as the first training data, the first N sentences of the summary text are used as the input to the outline prediction model, and the summary text is used as the target output to train the outline prediction model. This learning from the summary text can improve the outline prediction model's ability to predict reasonable plot developments based on the beginning.

[0040] 3) During the training process of the text generation model, contrastive learning is further combined with the semantic coherence subtask to improve the readability of the third text generated by the text generation model, thereby further improving the quality of the third text.

[0041] 4) This application only needs to predict the outline based on the first text, and then generate the third text based on the first text and the outline, without introducing any other strong assumptions or constraints, thus ensuring the stability of the model.

[0042] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is an illustrative system architecture diagram applicable to embodiments of this application;

[0045] Figure 2 This is a flowchart of the text generation method provided in the embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the overall model provided in the embodiments of this application;

[0047] Figure 4 This is a schematic diagram illustrating the principle of a training text generation model provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating another training text generation model provided in this application embodiment;

[0049] Figure 6 This is a schematic diagram illustrating the principle of a training coherence evaluation model provided in an embodiment of this application;

[0050] Figure 7 This is a schematic block diagram of the text generation apparatus provided in the embodiments of this application;

[0051] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0053] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0055] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0056] To facilitate understanding of this application, a brief description of the system architecture to which this application applies will be given first. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include a model building device and a text generation device.

[0057] The model building device, after acquiring training data, can train the model using the method provided in this application embodiment to obtain an outline prediction model and a text generation model. The model building device can build the prediction model offline.

[0058] The text generation device uses an outline prediction model and a text generation model to generate long text from the input text. The device can generate long text online or offline.

[0059] The model building device and text generation device can be configured as separate servers, or they can be configured on the same server or server group, or they can be configured on a separate or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability. The model training device and prediction device can also be configured on computer terminals with strong computing capabilities.

[0060] It should be understood that Figure 1 The number of model building devices, text generation devices, outline prediction models, and text generation models shown in the diagram is merely illustrative. Depending on the implementation requirements, any number of model building devices, text generation devices, outline prediction models, and text generation models can be included.

[0061] Figure 2 This is a flowchart of the main method provided in the embodiments of this application. The method can be... Figure 1 The text generation device in the system shown is executed. For example... Figure 2 As shown, the method may include the following steps:

[0062] Step 202: Obtain the first text.

[0063] Step 204: Input the first text into the outline prediction model, and obtain the outline generated by the outline prediction model for the first text as the second text. The outline prediction model is pre-trained based on a pre-trained language model.

[0064] Step 206: Input the first text and the second text into the text generation model, and obtain the third text that follows the first text generated by the text generation model using the first text and the second text, wherein the text generation model is pre-trained based on a pre-trained language model.

[0065] As can be seen from the above process, this application first generates an outline for the input first text, and then predicts subsequent text based on the first text and the outline to generate a third text. By introducing new knowledge and plot information through the outline, problems such as content repetition and plot stagnation that occur in the generated third text are alleviated, thereby improving the quality of the generated third text.

[0066] It should be noted that the terms "first," "second," etc., used in this disclosure do not imply any limitation in size, order, or quantity; they are merely used to distinguish between three texts. For example, "first text," "second text," and "third text" are used to distinguish between three texts. Similarly, "first training data," "second training data," and "third training data" are used to distinguish between three sets of training data, and so on.

[0067] The steps in the above process will be described in detail below with reference to the embodiments.

[0068] In this embodiment, the first text refers to a short text, and the goal is to generate a long text, i.e., the final third text, from this short text. The short text can contain N sentences, where N is a preset positive integer. As a typical approach, the first text can be a single sentence.

[0069] The following describes in detail step 204, namely, "inputting the first text into the outline prediction model and obtaining the outline generated by the outline prediction model for the first text as the second text," with reference to the embodiments.

[0070] The outline prediction model involved in this application embodiment has the ability to reasonably predict the outline based on the opening statement. That is, it can predict the outline (i.e., the second text) related to the first text given the first text as input. The structure of the outline prediction model can be as follows: Figure 3 As shown, it may include a first encoder and a first decoder.

[0071] The first encoder is used to extract a first text feature representation from the input text, and the first decoder is used to predict an outline for the input text using the first text feature representation.

[0072] The input text can be viewed as a text sequence composed of tokens, each of which can include characters, start characters, and delimiters. In the first encoder, each token in the text sequence is first embedded, and then encoded based on the embedding results to obtain the feature representation of each token.

[0073] The token-based embedding process described above includes at least two components: word embedding and position embedding. Word embedding involves encoding each token into a word vector representation. Position embedding involves encoding the position of each token within the text sequence to be predicted, obtaining a positional representation. For example, tokens can be sequentially numbered 0, 1, 2, 3, 4, 5, and 6 based on their positions within the text sequence.

[0074] The first encoder and first decoder can be implemented based on a pre-trained language model, such as BERT (Bidirectional Encoder Representation from Transformers), GPT (Generative Pre-Training), or XLNet. The first encoder and first decoder actually each consist of multiple Transformer layers, each composed of MHA (Multi-head Attention) and FFN (Feed-forward Network).

[0075] like Figure 3 As shown, given the input text X1, the output prediction result is also a text sequence containing sentences S1, S2, ... Figure 3 The outputs S1, S2, and S3 are used as an example.

[0076] When training the outline prediction model, the first training data can be obtained, which may include multiple summary texts. For example, some news databases or websites contain a large number of news and paper summaries, which can be obtained as the first training data.

[0077] The first N sentences of the summary text are then used as input to the outline prediction model, where N is a preset positive integer; for example, when N is 1, the first sentence of the summary text is used as input. The corresponding summary text is then used as the target output of the outline prediction model to train it. In other words, the training objective of the outline prediction model is to minimize the difference between the outline output by the model and the corresponding summary text.

[0078] In this embodiment, a loss function can be constructed based on the aforementioned training objective, such as a cross-entropy loss function. In each iteration, the model parameters are updated using the value of the loss function and methods such as gradient descent, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0079] The following describes in detail step 206, namely, "inputting the first text and the second text into the text generation model and obtaining the third text that follows the first text generated by the text generation model using the first text and the second text", with reference to the embodiments.

[0080] The text generation model is used to generate a third text that follows the first text. Taking the first text as a single sentence as an example, the text generation model generates a long text based on the sentence and an outline generated from it. This long text follows the sentence and learns the basic plot from the outline, significantly improving the plot progression ability of the generated long text. Open-ended text generation can be seen as text continuation, that is, continuing the story of a subsequent long text based on the input sentence. In the embodiments of this application, the long text generated by the text generation model that follows the first text is called the third text. The length of the third text is greater than that of the first text, and it can be a text that begins with the first text.

[0081] The structure of a text generation model can be as follows: Figure 3 As shown on the right, it includes a second encoder and a second decoder. The second encoder is used to extract a second text feature representation from the input text, and the first decoder is used to generate a third text that follows the input text using the second text feature representation.

[0082] Similar to outline generation models, the text input to a text generation model can be viewed as a text sequence composed of tokens. One possible approach is to concatenate the first text with the outline generated from the first text (i.e., the second text), using the concatenated text sequence as input to the text generation model. Each token can include characters, start characters, and separators. In the second encoder, each token in the text sequence can first be embedded, and then encoded based on the embedding results to obtain the feature representation of each token.

[0083] The token-based embedding process described above includes at least two components: word embedding and position embedding. Word embedding involves encoding each token into a word vector representation. Position embedding involves encoding the position of each token within the text sequence to be predicted, obtaining a positional representation. For example, tokens can be sequentially numbered 0, 1, 2, 3, 4, 5, and 6 based on their positions within the text sequence.

[0084] The second encoder and second decoder can be implemented based on a pre-trained language model, such as BERT, GPT, or XLNet. The second encoder and second decoder each consist of multiple Transformer layers, each composed of an MHA and a FFN (feed-forward network).

[0085] like Figure 3 As shown, given the input of the first text X1 and the outlines {S1, S2, and S3}, the generated third text is also a text sequence containing sentences O1, O2, ... O n Where n can be a preset positive integer, or the length of the third text when the text generation model automatically ends the output.

[0086] When training a text generation model, a second training data set can be obtained first. This second training data set includes multiple article samples and the summary text of each article sample. For example, some news, academic paper, and other databases or websites contain a large number of news articles, academic papers, and their summaries. These articles can be obtained as article samples, and their summaries can be used as summary texts to constitute the second training data set.

[0087] Then, the first N sentences of the article sample and its summary text are used as input to the text generation model. Here, N is the same value as in the outline prediction model; for example, N can be 1, representing the first sentence. Specifically, the text sequence formed by concatenating the first N sentences of the article sample and the summary text is used as input to the text generation model. The article sample is then used as the target output of the text generation model to train it.

[0088] One feasible approach is to train the text generation model with the objective of minimizing the difference between the third text generated by the text prediction model and the corresponding article sample. For example, a cross-entropy loss function could be designed accordingly. In each iteration, the model parameters could be updated using methods such as gradient descent based on the value of the loss function, until a preset training termination condition is met. This termination condition could include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0089] As another feasible approach, the text generation model can be further integrated with contrastive learning for the semantic coherence subtask during training. The training objectives for the text generation model can include a first training objective and a second training objective. The first training objective is to minimize the difference between the third text output by the text generation model and the corresponding article sample. The second training objective is used to ensure the semantic coherence of the third text output by the text generation model. In an article, semantic coherence is typically reflected in the higher semantic similarity between closely spaced sentences and the lower semantic similarity between widely spaced sentences. Therefore, the second training objective can be designed as follows: minimizing the similarity between sentences in the third text generated by the text generation model that are less than or equal to a first preset distance, and maximizing the similarity between sentences in the third text output by the text generation model that are greater than or equal to a second preset distance, where the first preset distance is less than the second preset distance.

[0090] The first and second preset distances mentioned above can be expressed in terms of the number of statements. For example, maximizing the similarity between adjacent statements and minimizing the similarity between statements spaced more than 5 statements apart.

[0091] like Figure 4 As shown in the embodiment of this application, loss1 can be designed for the first training objective and loss2 can be designed for the second training objective. In each iteration, loss1 and loss2 are weighted and summed to obtain the value of the total loss function loss. The model parameters are then updated using methods such as gradient descent until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset number threshold.

[0092] As another feasible approach, a coherence evaluation model can be further integrated during the training of the text generation model to improve the coherence of the third-party text generated by the text generation model. For example... Figure 5 As shown, the third text output by the text generation model is input into a pre-trained coherence evaluation model, which then outputs the coherence of adjacent text pairs within the third text. The training objectives used in this approach can include a first training objective and a third training objective. The first training objective is to minimize the difference between the third text output by the text generation model and the corresponding article sample. The third training objective is to maximize the coherence of the output by the coherence evaluation model. The third training objective is used to ensure the semantic coherence of the third text output by the text generation model.

[0093] like Figure 5 As shown in the embodiment of this application, loss1 can be designed for the first training objective and loss3 can be designed for the third training objective. For example, loss3 can be determined by the sum or average of the coherence of all adjacent text pairs in the third text. In each iteration, loss1 and loss3 are weighted and summed to obtain the value of the total loss function loss. The parameters of the text generation model are updated using methods such as gradient descent until a preset training termination condition is met. After training, the coherence evaluation model is removed. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset number threshold.

[0094] The training method for the aforementioned coherence evaluation model is described below. First, a third training dataset is obtained, which includes positive and negative sample pairs. Positive sample pairs can be formed by selecting two adjacent sentences from existing articles, and negative sample pairs can be formed by selecting two non-adjacent sentences. For example, a negative sample pair can be formed by selecting two sentences that are more than five sentences apart.

[0095] Then, the coherence evaluation model is trained using the third training data mentioned above. Positive or negative sample pairs from the third training data are input into the coherence evaluation model, which outputs the coherence score of the positive or negative sample pairs. For example... Figure 6 As shown, the coherence evaluation model may include a third encoder and a mapping layer. The third encoder is used to extract a third textual feature representation from the input positive or negative sample pairs, and the mapping layer is used to obtain the coherence of the positive or negative sample pairs using the third textual feature representation.

[0096] The third encoder can also be implemented based on a pre-trained language model, such as BERT, GPT, or XLNet. The mapping layer can be a regression or classification model, used to map the third text features to one of the values ​​in the coherence space.

[0097] The coherence evaluation model is trained using contrastive learning, with the training objective being to maximize the coherence of positive sample pairs and minimize the coherence of negative sample pairs.

[0098] The methods provided in this application embodiment can be applied to various application scenarios, including but not limited to:

[0099] Application Scenario 1: Intelligent Dialogue Scenario

[0100] After the intelligent chatbot receives the user's input, it uses the input as the first text in the above method embodiment, and then generates a third text as a reply to the user using the method provided in this application embodiment. The user's input can be either user-inputted text or text obtained through speech recognition of the user's input. The third text returned to the user can be in text form or synthesized speech returned to the user.

[0101] Application Scenario 2: Automatically Generate Reviews

[0102] Users often need to evaluate purchased goods, performances, and services when shopping online, watching shows, or ordering services. However, typing out a review can be tedious, especially on devices like smartphones or wearable devices where text input is inconvenient. Using the method provided in this application, if user input is detected in the review component, the user only needs to type a single sentence expressing their opinion, and the system will automatically generate a review text for the user to choose from. This method can be applied to applications such as input methods and e-commerce.

[0103] Application Scenario 3: Intelligent Story Writing

[0104] When using social networking applications, users often need to write a short text to express their feelings, experiences, etc., but typing out a comment can be quite time-consuming. Using the method provided in this application, if a user's input in a text input box is detected, the user only needs to type a single sentence expressing their thoughts, and a story text will be automatically generated and returned to the user. The user can choose to publish this story text or edit it before publishing. This significantly reduces user effort and also makes it more appealing for users with limited writing skills or time to share their thoughts.

[0105] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0106] According to another embodiment, a text generation apparatus is provided. Figure 7 A schematic block diagram of the text generation apparatus according to one embodiment is shown. Figure 7 As shown, the device 700 includes: a text acquisition unit 701, an outline prediction unit 702, and a text generation unit 703, and may further include: a first training unit 704, a second training unit 705, and a third training unit 706. The main functions of each component are as follows:

[0107] The text acquisition unit 701 is configured to acquire the first text.

[0108] The outline prediction unit 702 is configured to input the first text into the outline prediction model and obtain the outline generated by the prediction model for the first text as the second text. The outline prediction model is pre-trained based on a pre-trained language model.

[0109] The text generation unit 703 is configured to input the first text and the second text into the text generation model, and obtain the third text that follows the first text generated by the text generation model using the first text and the second text. The text generation model is pre-trained based on a pre-trained language model.

[0110] The outline prediction model includes a first encoder and a first decoder that employ a pre-trained language model. The first encoder extracts a first text feature representation from the input text, and the first decoder uses the first text feature representation to predict the outline.

[0111] As one possible approach, the first training unit 704 can be pre-trained to obtain the outline prediction model in the following manner:

[0112] Obtain the first training data, which includes multiple summary texts;

[0113] The first N sentences of the summary text are used as input to the outline prediction model, where N is a preset positive integer. The summary text is used as the target output of the outline prediction model to train the outline prediction model.

[0114] As one possible approach, the second training unit 705 can be pre-trained to obtain the text generation model in the following manner:

[0115] Obtain the second training data, which includes multiple article samples and the summary text of each article sample;

[0116] The first N sentences of the article sample and the summary text of the article sample are used as input to the text generation model, where N is a preset positive integer. The article sample is used as the target output of the text generation model to train the text prediction model. The text prediction model includes: a second encoder and a second decoder using a pre-trained language model; the second encoder extracts a second text feature representation from the input text, and the second encoder is used to generate a third text that follows the first N sentences of the article sample using the second text feature representation.

[0117] As one possible approach, the training objectives used by the second training unit 705 to train the text prediction model include a first training objective and a second training objective.

[0118] The primary training objective is to minimize the difference between the third text of the first N sentences of a continuation article sample and the corresponding article sample.

[0119] The second training objective is to minimize the similarity between sentences in the third text generated by the text generation model whose distance is less than or equal to the first preset distance, and to maximize the similarity between sentences in the third text output by the text generation model whose distance is greater than or equal to the second preset distance, wherein the first preset distance is less than the second preset distance.

[0120] As an alternative approach, the second training unit 705, while training the text prediction model, further inputs the third text output by the text generation model into the pre-trained coherence evaluation model to obtain the coherence of adjacent text pairs output by the coherence evaluation model. The training objectives for the text prediction model include a first training objective and a third training objective; the first training objective is to minimize the difference between the text output by the text generation model and the corresponding article sample; the third training objective is to maximize the coherence of the coherence output by the coherence evaluation model. The coherence evaluation model is removed after training.

[0121] As one possible approach, the third training unit 706 can be pre-trained to obtain the coherence evaluation model in the following manner:

[0122] Obtain the third training data, which includes positive sample pairs and negative sample pairs. Positive sample pairs are adjacent sentences in the text samples, and negative sample pairs are non-adjacent sentences in the text samples.

[0123] A coherence evaluation model is trained using training data. The coherence evaluation model includes a third encoder and a mapping layer. The third encoder is used to extract third textual feature representations from the input positive or negative sample pairs. The mapping layer is used to obtain the coherence of positive or negative sample pairs using the third textual feature representations. The training objective is to maximize the coherence of positive sample pairs and minimize the coherence of negative sample pairs.

[0124] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., the user gives explicit consent, the user is properly notified, etc.).

[0125] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0126] And an electronic device, comprising:

[0127] One or more processors; and

[0128] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0129] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0130] in, Figure 8 An exemplary architecture of an electronic device is shown, which may include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820 can communicate with each other via a communication bus 830.

[0131] The processor 810 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0132] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store the operating system 821 for controlling the operation of the electronic device 800, and the basic input / output system (BIOS) 822 for controlling the low-level operations of the electronic device 800. Additionally, it can store a web browser 823, a data storage management system 824, and a text generation device 825, etc. The aforementioned text generation device 825 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0133] The input / output interface 813 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0134] Network interface 814 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0135] Bus 830 includes a pathway for transmitting information between various components of the device, such as processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820.

[0136] It should be noted that although the above-described device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0137] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0138] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A text generation method, characterized in that, The method includes: Get the first text; The first text is input into the outline prediction model, and the outline generated by the outline prediction model for the first text is obtained as the second text. The outline prediction model is pre-trained based on a pre-trained language model. The first text and the second text are input into the text generation model to obtain the third text that follows the first text generated by the text generation model using the first text and the second text. The text generation model is pre-trained based on a pre-trained language model. Specifically, during the training of the text generation model, a pre-trained coherence evaluation model is appended to the text generation model. The text output by the text generation model is input into the coherence evaluation model to obtain the coherence degree of adjacent text pairs in the text output by the coherence evaluation model. After the text generation model is trained, the coherence evaluation model is removed. The third text output by the text generation model has semantic coherence. The method further includes, after obtaining the coherence of adjacent text pairs in the text output by the coherence evaluation model, determining a total loss function value based on the function values ​​of the loss functions corresponding to the first and third training objectives of the text generation model, wherein the first training objective is to minimize the difference between the third text output by the text generation model and the article sample corresponding to the third text, and the third training objective is to maximize the coherence of the output by the coherence evaluation model; updating the parameters of the text generation model based on the total loss function value until the updated total loss function value corresponding to the text generation model is less than or equal to the loss function threshold.

2. The method according to claim 1, characterized in that, The outline prediction model was pre-trained using the following method: Obtain first training data, which includes multiple summary texts; The first N sentences of the summary text are used as input to the outline prediction model, where N is a preset positive integer. The summary text is used as the target output of the outline prediction model to train the outline prediction model. The outline prediction model includes: a first encoder and a first decoder using a pre-trained language model; the first encoder extracts a first text feature representation from the input text, and the first decoder uses the first text feature representation to predict the outline.

3. The method according to claim 1, characterized in that, The text generation model was pre-trained using the following method: Obtain second training data, which includes multiple article samples and summary text of each article sample; The first N sentences of the article sample and the summary text of the article sample are used as input to the text generation model, where N is a preset positive integer. The article sample is used as the target output of the text generation model to train the text generation model. The text generation model includes: a second encoder and a second decoder using a pre-trained language model; the second encoder extracts a second text feature representation from the input text, and the second encoder is used to generate a third text that continues the first N sentences of the article sample using the second text feature representation.

4. The method according to claim 3, characterized in that, The training objectives used to train the text generation model include a first training objective and a second training objective; The first training objective is to minimize the difference between the third text generated by the text generation model and the corresponding article sample; The second training objective is to minimize the similarity between sentences in the third text generated by the text generation model whose distance is less than or equal to a first preset distance, and to maximize the similarity between sentences in the third text output by the text generation model whose distance is greater than or equal to a second preset distance, wherein the first preset distance is less than the second preset distance.

5. The method according to claim 3, characterized in that, The training objectives used to train the text generation model include the first training objective and the third training objective.

6. The method according to claim 5, characterized in that, The coherence evaluation model is pre-trained in the following manner: Obtain third training data, which includes positive sample pairs and negative sample pairs. The positive sample pairs are adjacent sentences in the text samples, and the negative sample pairs are non-adjacent sentences in the text samples. The coherence evaluation model is trained using the training data. The coherence evaluation model includes a third encoder and a mapping layer. The third encoder is used to extract a third textual feature representation from the input positive sample pairs or negative sample pairs. The mapping layer is used to obtain the coherence of the positive sample pairs or negative sample pairs using the third textual feature representation. The training objective is to maximize the coherence of the positive sample pairs and minimize the coherence of the negative sample pairs.

7. The method according to any one of claims 1 to 6, characterized in that, The first text is a statement entered by the user, and the third text includes: In intelligent dialogue scenarios, a response sent to the user based on the first text input; or, The evaluation text generated after the user enters the first text in the evaluation component is detected; or, The story text generated after the user entered the first text in the text publishing component was detected.

8. A text generation device, characterized in that, The device includes: The text acquisition unit is configured to acquire the first text; The outline prediction unit is configured to input the first text into the outline prediction model and obtain the outline generated by the outline prediction model for the first text as the second text. The outline prediction model is pre-trained based on a pre-trained language model. The text generation unit is configured to input the first text and the second text into the text generation model, and obtain a third text that follows the first text generated by the text generation model using the first text and the second text. The text generation model is pre-trained based on a pre-trained language model. The device is further configured to: during the training of the text generation model, connect a pre-trained coherence evaluation model after the text generation model, input the text output by the text generation model into the coherence evaluation model, and obtain the coherence degree of adjacent text pairs in the text output by the coherence evaluation model; after the text generation model is trained, remove the coherence evaluation model, wherein the third text output by the text generation model has semantic coherence; After obtaining the coherence of adjacent text pairs in the text output by the coherence evaluation model, the device is further configured to: determine a total loss function value based on the function values ​​of the loss functions corresponding to the first and third training objectives of the text generation model, wherein the first training objective is to minimize the difference between the third text output by the text generation model and the article sample corresponding to the third text, and the third training objective is to maximize the coherence of the output by the coherence evaluation model; update the parameters of the text generation model based on the total loss function value until the updated total loss function value corresponding to the text generation model is less than or equal to the loss function threshold.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 7.

Images