A text generation model training method and device, medium and electronic equipment
By identifying the original text from a general text dataset and generating training samples using a type recognition model, the problem of poor training performance of text generation models due to insufficient training samples is solved, achieving efficient training and accurate output under conditions of few samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2023-05-08
- Publication Date
- 2026-07-31
AI Technical Summary
When there are few training samples, the text generation model does not train well, resulting in inaccurate output text.
The original text is determined from a general text dataset. Template annotations are determined through a pre-trained type recognition model. Training samples are generated based on the template annotations. The text generation model is trained with the goal of minimizing the difference between the sample annotations and the output text.
With a limited number of training samples, the training performance of the text generation model was improved, and the accuracy of the output text was enhanced.
Smart Images
Figure CN116628198B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of computer technology, and in particular to a training method, apparatus, medium, and electronic device for a text generation model. Background Technology
[0002] With the continuous development of technology, natural language processing has received widespread attention, and natural language models have gradually come into people's view. Among them, text generation models occupy an important position in natural language processing.
[0003] Natural language model training requires a large number of training samples, and text generation models are no exception. However, the number of training samples suitable for a particular scenario may be limited in a general dataset, or the collected data for that scenario may not be labeled. This results in a shortage of training samples for the text generation model, leading to poor training performance and inaccurate output text. Therefore, how to train a text generation model with limited training samples is a challenge.
[0004] Based on this, this specification provides a training method for a text generation model. Summary of the Invention
[0005] This specification provides a training method, apparatus, medium, and electronic device for a text generation model, in order to partially solve the aforementioned problems existing in the prior art.
[0006] The following technical solution is adopted in this specification:
[0007] This manual provides a training apparatus for the text generation model, including:
[0008] Determine the original text from a general text dataset;
[0009] The original text is input into a pre-trained type recognition model to determine the template annotation of the original text;
[0010] Based on the template annotation, determine the target template corresponding to the template annotation;
[0011] Training samples for the training text generation model are generated based on the original text, the template annotation, and the target template, wherein the target template is used to divide at least a portion of the original text into sample annotations, and to concatenate the original text other than the sample annotations and the template annotations into an input part;
[0012] The input portion is input into the text generation model to be trained to obtain the output text. The text generation model to be trained is trained with the goal of minimizing the difference between the sample annotation and the output text.
[0013] Optionally, the original text is determined from a general text dataset, specifically including:
[0014] Determine the parameter dimensions of the text generation model to be trained;
[0015] The number of training samples is determined based on the parameter dimensions.
[0016] The number of original texts is determined from a general text dataset.
[0017] Optionally, based on the template annotation, the target template corresponding to the template annotation is determined, specifically including:
[0018] Based on the template annotation, select the target template corresponding to the template annotation from the preset prompt templates.
[0019] Optionally, the template is labeled as a task type;
[0020] Based on the original text, the template annotation, and the target template, training samples for the text generation model are generated, specifically including:
[0021] Determine the first and second text contained in the original text;
[0022] Based on the target template, the first text and the template annotation are concatenated as the input part of the target template, and the second text is determined as the sample annotation of the target template;
[0023] Training samples for the training text generation model are generated based on the input and the sample annotations.
[0024] Optionally, the template is labeled as a theme type;
[0025] Based on the original text, the template annotation, and the target template, training samples for the text generation model are generated, specifically including:
[0026] Based on the target template, the template annotation is determined as the input part of the target template, and the original text is determined as the sample annotation of the target template;
[0027] Training samples for the training text generation model are generated based on the input and the sample annotations.
[0028] Optionally, the type recognition model is a text classification model;
[0029] The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including:
[0030] The original text is input into a pre-trained text classification model to determine the task type of the original text, and is then used as a template for annotation.
[0031] Optionally, the type recognition model is a keyword extraction model;
[0032] The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including:
[0033] The original text is input into a pre-trained keyword extraction model to determine the topic type of the original text, which is then used as a template for annotation.
[0034] Optionally, the original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including:
[0035] In response to type information input by the user, the target type recognition model of the original text is determined from a pre-trained type recognition model;
[0036] The original text is input into the target type recognition model to determine the template annotation of the original text.
[0037] Optionally, the original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including:
[0038] Identify the pre-trained recognition models for each type;
[0039] For each type recognition model, the original text is input into that type recognition model to obtain the output result;
[0040] The obtained output results are used as template annotations for the original text.
[0041] This specification provides a training apparatus for a text generation model, including:
[0042] The first determination module is used to determine the original text from a general text dataset;
[0043] The second determining module is used to input the original text into a pre-trained type recognition model to determine the template annotation of the original text;
[0044] The third determining module is used to determine the target template corresponding to the template annotation based on the template annotation;
[0045] The generation module is used to generate training samples for the training text generation model based on the original text, the template annotation, and the target template, wherein the target template is used to divide at least a portion of the original text into sample annotations, and to concatenate the original text other than the sample annotations and the template annotations into an input part;
[0046] The training module is used to input the input part into the text generation model to be trained, obtain the output text, and train the text generation model to be trained with the minimum difference between the sample label and the output text as the training objective.
[0047] Optionally, the first determining module is specifically used to: determine the parameter size of the text generation model to be trained; determine the number of training samples based on the parameter size; and determine the number of original texts from a general text dataset.
[0048] Optionally, the third determining module is specifically used to select the target template corresponding to the template annotation from a set of preset prompt templates based on the template annotation.
[0049] Optionally, the template is labeled as a task type;
[0050] The generation module is specifically used to: determine the first text and the second text contained in the original text; determine, according to the target template, to concatenate the first text and the template annotation as the input part of the target template, and determine the second text as the sample annotation of the target template; and generate training samples for training the text generation model according to the input part and the sample annotation.
[0051] Optionally, the template is labeled as a theme type;
[0052] The generation module is specifically used to: determine the template annotation as the input part of the target template and determine the original text as the sample annotation of the target template based on the target template; and generate training samples for training the text generation model based on the input part and the sample annotation.
[0053] Optionally, the type recognition model is a text classification model;
[0054] The second determining module is specifically used to input the original text into a pre-trained text classification model, determine the task type of the original text, and use it as a template for annotation.
[0055] Optionally, the type recognition model is a keyword extraction model;
[0056] The second determining module is specifically used to input the original text into a pre-trained keyword extraction model, determine the topic type of the original text, and use it as a template for annotation.
[0057] Optionally, the second determining module is specifically used to, in response to type information input by the user, determine the target type recognition model of the original text from a pre-trained type recognition model; input the original text into the target type recognition model to determine the template annotation of the original text.
[0058] Optionally, the second determining module is specifically used to: determine the pre-trained recognition models of each type; for each type recognition model, input the original text into the type recognition model to obtain the output result; and use the obtained output results as template annotations for the original text.
[0059] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the text generation model described above.
[0060] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the text generation model described above.
[0061] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:
[0062] The training method for the text generation model provided in this manual first determines the original text from a general text dataset. This original text is then input into a pre-trained type recognition model to determine the template annotations for the original text. Next, based on the template annotations, the corresponding target template is determined. Training samples for the text generation model are generated using the original text, template annotations, and target template. The input is then fed into the text generation model to be trained to obtain the output text. The training objective is to minimize the difference between the sample annotations and the output text.
[0063] As can be seen from the above method, when training the text generation model, this application first inputs the original text determined from the general text dataset into a pre-trained type recognition model to determine the template annotation of the original text. Then, based on the template annotation, the target template corresponding to the template annotation is determined. Next, training samples for training the text generation model are generated based on the original text, template annotation, and target template. The input is then fed into the text generation model to be trained to obtain the output text. The training objective is to minimize the difference between the sample annotation and the output text. By generating training samples for training the text generation model using the original text and template annotation from the general text dataset, and utilizing the target template, the number of training samples for training the text generation model is increased. This allows the text generation model to be trained even with a small number of training samples, resulting in better training performance and improved accuracy of the output text. Attached Figure Description
[0064] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:
[0065] Figure 1 This is a flowchart illustrating a training method for a text generation model provided in this specification.
[0066] Figure 2 This is a schematic diagram of a process for generating training samples provided in this specification;
[0067] Figure 3 This is a schematic diagram of another process for generating training samples provided in this specification;
[0068] Figure 4 A schematic diagram of the training device structure for a text generation model provided in this specification;
[0069] Figure 5 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation
[0070] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.
[0071] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.
[0072] Figure 1 This is a flowchart illustrating a training method for a text generation model provided in this specification, including the following steps:
[0073] S100: Determine the original text from a general text dataset.
[0074] In this specification, the device used to train the text generation model can determine the original text from a general text dataset. This device can be a server or an electronic device such as a desktop computer or laptop. For ease of description, the training method for the text generation model provided in this specification will be described below using a server as the primary execution device.
[0075] The general text dataset can be pre-collected text data or any existing text dataset; this specification does not impose any specific limitations. Since the samples in the general text dataset are pre-collected from various data provision platforms, such as websites or mobile devices, the text in the general text dataset may be in various forms. For example, the text could be "The weather in Beijing is really nice." or "Have you eaten? I have eaten."
[0076] Specifically, the server can randomly select the original text from a general text dataset. Of course, since the sources of the general text dataset may be different, in order to ensure that the distribution of the selected original text is uniform, that is, the number of original texts selected from each source is the same, the server can select the same number of texts as the original text from each source corresponding to the general text dataset.
[0077] Additionally, the server can determine the parameter dimensions of the text generation model to be trained, and based on these parameters, determine the number of training samples. Then, it determines the number of raw text samples from a general text dataset.
[0078] When determining the number of training samples based on parameter dimensions, the following formula can be used for calculation:
[0079]
[0080] Where m is the expected loss of the trained text generation model, and N c D c α N and α D A constant to be set, such as N c It can be 6.4×10 13 D c It can be 1.8 × 1013 α N It can be 0.076, α D It can be 0.103, where N is the parameter size of the text generation model and D is the number of original texts.
[0081] S102: Input the original text into a pre-trained type recognition model to determine the template annotation of the original text.
[0082] The server can input the raw text into a pre-trained type recognition model to determine the original template annotations. The type recognition model can be either a text classification model or a keyword extraction model. It can be a model pre-trained on a text dataset or any existing open-source model; this specification does not impose specific limitations. Template annotations are the annotations obtained after inputting the raw text into the type recognition model.
[0083] Specifically, the server can input the raw text into a pre-trained type recognition model, obtain the output of the type recognition model, and determine the original template annotation based on the output. The template annotation can be one of the following types: task type, topic type, keyword type, genre type, and sentiment type; this specification does not specify a particular type. The task type indicates the task for which the raw text is used. In this specification, the task type includes at least one of the following natural language processing tasks: open question answering, copywriting generation, text matching, story generation, text summarizing, classical poetry generation, programming, entity recognition, sentiment analysis, text correction, prose generation, couplet generation, idiom interpretation, essay generation, translation, commentary generation, reading comprehension, lyrics generation, novel generation, keyword recognition, classical Chinese to modern Chinese translation, and text inference. The topic type refers to the keywords of the raw text; each raw text must have at least one topic. The keyword type refers to the set of keywords in the raw text; each raw text may have one keyword, or multiple keywords, such as three keywords in a given raw text. Genre types include narrative, expository, argumentative, practical, poetry, fiction, drama, and prose, with specific genres available as needed. Emotional types include positive, negative, and neutral.
[0084] In this specification, different template annotations can correspond to different type recognition models. For model annotations with definite discrete or restricted categories, such as task type, genre type, and sentiment type (e.g., sentiment type includes "positive," "negative," and "neutral"), a pre-trained text classification model or any existing open-source text classification model can be used. Therefore, when the type recognition model is a text classification model, taking task type as an example, the server can input the original text into the pre-trained text classification model to determine the task type of the original text and use it as the template annotation. The text classification model can be a model pre-trained based on a text dataset or any existing open-source text classification model. The above explanation only uses task type as an example; however, the server can also execute the above process when the template annotation is genre type or sentiment type, which will not be elaborated further here.
[0085] Furthermore, for template annotations such as keyword types and topic types, which lack defined categories or are not restricted in their classification, a keyword extraction model can be pre-trained or any existing open-source keyword extraction model can be used. Therefore, when the type recognition model is a keyword extraction model, taking topic annotation as an example, the server can input the original text into the pre-trained keyword extraction model to determine the topic type of the original text and use it as the template annotation. The keyword extraction model can be a model pre-trained based on a text dataset or any existing open-source keyword extraction model. The above explanation only uses topic annotation as an example; however, when the template annotation is a keyword type, the server can also follow the same process, which will not be elaborated further here.
[0086] S104: Determine the target template corresponding to the template annotation based on the template annotation.
[0087] The server can determine the target template corresponding to the template annotation based on the template annotation. Specifically, the server can select the target template corresponding to the template annotation from a set of preset prompt templates based on the template annotation of the original text. The prompt template is a template that concatenates the input part of the training sample with the sample annotation. Different template annotations correspond to different templates. When the template annotation is a task type, the target template corresponding to that template annotation can be " <s>Please implement the task of task type 2 based on the original text 1. <e>".in, <s>`<start>` is the start character, `<delimiter>` is the separator. <e>As a terminator, it is used only in this specification in the above-mentioned way. <s>,and <e>Taking the start character, separator, and terminator as examples, other symbols, numbers, or letters can also be used as start characters, separators, and terminators, which will not be elaborated here. The "【】" above represents the slots in the template. "Original Text 1" and "Original Text 2" in the slot are a pair of original texts in the general text dataset, and "Task Type" in the slot is the template annotation corresponding to the original text.
[0088] For example, original text 1 is "In the center of the rock grows a tree, over a hundred feet tall, with branches and trunk of five colors, green leaves like plates, flowers over a foot in diameter, deep blue in color, deep red in stamens, emitting a strange fragrance that lingers like smoke." Original text 2 is "In the center of the large rock grows a tree, over a hundred feet tall, with colorful branches and leaves as large as plates, flowers a foot in diameter, deep blue petals, emitting a strange fragrance that envelops the surroundings like smoke and mist." Original text 1 and original text 2 are a pair of original texts. Through step S102 above, the template corresponding to this original text can be labeled as the task type of translating classical Chinese into modern Chinese. Therefore, the target template corresponding to this original text is "". <s>Please implement the task of task type 2 based on the original text 1. <e>".
[0089] Of course, in this specification, when a template is labeled as a theme type, the target template corresponding to that template label can be " <s>Please generate the following based on the theme: [Original Text] <e>The "topic" in the slot is the topic obtained based on the "original text" in the slot, which is the template annotation corresponding to the "original text".
[0090] When a template is tagged as a keyword, the target template corresponding to that tagged template can be " <s>The generated text needs to contain [Keyword 1], [Keyword 2], ..., [Keyword N]: [Original Text] <e>The "keywords" in the slot are keywords obtained based on the "original text" in the slot, that is, the template annotation corresponding to the "original text". There can be one or more keywords, and the order of the keywords is not restricted. N is a positive integer.
[0091] When a template is labeled as a genre type, the target template corresponding to that template label can be "". <s>[Genre] [Original Text] <e>In this context, "genre" in the slot refers to the genre type obtained based on "original text" in the slot.
[0092] When a template is labeled as a sentiment type, the target template corresponding to that label can be " <s>Please use the emotion from the [Emotion Type] category to generate the text: [Original Text] <e>The “sentiment type” in the slot is a sentiment classification based on the “original text” in the slot, which is the template label corresponding to the “original text”.
[0093] S106: Based on the original text, the template annotation, and the target template, generate training samples for the training text generation model, wherein the target template is used to divide at least a portion of the original text into sample annotations, and to concatenate the original text other than the sample annotations and the template annotations into an input part.
[0094] The server can generate training samples for the text generation model based on the original text, template annotations, and a target template. Specifically, the server can fill the original text and its corresponding template annotations into the target template to generate training samples for the text generation model. The target template is used to divide at least a portion of the original text into sample annotations, and to concatenate the original text and template annotations (excluding those divided into sample annotations) into the input portion. Therefore, the format of the training samples can be " <s>Text Text <e>", of which, the front part" <s>The input section is labeled "Text," and the subsequent "Text" represents the sample annotations. By dividing the original text and template annotations using the target template, training samples for the text generation model are obtained. This allows original text that would otherwise be unsuitable as training samples to be used, thus increasing the training sample pool for the model. Furthermore, in addition to the concatenated content of the original text and template annotations (excluding the sample annotations), the input section of the target template can also include fixed prompts. These prompts are inherent to the target template and contribute to more accurate output text from the subsequent text generation model.
[0095] Continuing with the previous example, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a process for generating training samples provided in this specification. The original text, consisting of original text 1 and original text 2, along with the corresponding template annotation (i.e., classical Chinese translated into modern Chinese), is filled into the target template to generate training samples for the training text generation model. The training samples are thus "". <s>Please complete the task of translating the classical Chinese text into modern Chinese based on the following: "In the center of the rock grew a tree, over a hundred feet tall, with colorful branches and leaves as large as plates. Its flowers were over a foot in diameter, with deep blue petals and deep red stamens. A strange fragrance emanated from the flowers, enveloping the surroundings like smoke and mist." <e>"in," <s>Please implement the task of translating the classical Chinese text into modern Chinese based on the following:
In the center of the rock grew a tree, over a hundred feet tall, with branches and trunk of five colors, emerald leaves as large as plates, flowers over a foot in diameter, deep blue in color, with deep red stamens, emanating a strange fragrance that enveloped everything like smoke and mist.
[0096] For example, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating another method for generating training samples provided in this specification. Assume the original text is "I study at a university in Beijing.", the topic of which is "Beijing," and the target template is... <s>Please generate the following based on the theme: [Original Text] <e>Therefore, the original text and its corresponding template annotation are filled into the target template to generate training samples for the text generation model. The training samples are thus: <s>Please generate the following based on Beijing: "I am studying at a university in Beijing." <e>",in," <s>Please generate the following based on Beijing: "[I am studying at a university in Beijing.]" is the input, and "[I am studying at a university in Beijing.]" is the sample annotation.
[0097] S108: Input the input part into the text generation model to be trained to obtain the output text, and train the text generation model to be trained with the minimum difference between the sample label and the output text as the training objective.
[0098] The server can input the text generation model to be trained into the input part, obtain the output text, and train the text generation model with the goal of minimizing the difference between the sample labels and the output text. The text generation model can be an autoregressive (AR) model, such as the Generative Pre-trained Transformer (GPT) series of models, or a model that integrates autoencoder (AE) technology on top of an AR model, or other existing text generation models; this specification does not specify any particular limitation.
[0099] Specifically, the server can input the input parts into the text generation model to be trained, obtain the output text, and input the sample annotations and output text into the loss function to determine the loss. The training objective is to minimize this loss, and the text generation model is then trained. To avoid the influence of the input parts and identifiers on the output text, and to allow the text generation model to focus more on the quality of the output text, the loss function can be improved. This improved loss function eliminates the calculation of the loss due to the input parts and identifiers when inputting the output text and sample annotations; in other words, it removes the "...". <s>"text" and <e>The loss can be calculated using the following function:
[0100] L1(u)=∑log P(u i |u i-k , ..., u i-1 ;Θ)
[0101] Among them, u={u1,u2,...,u n } represents the vector obtained after word segmentation of the input part, L1(u) represents the loss function, k represents the sliding window, and u i |u i-k , ..., u i-1 Indicates based on k historical words u i-k , ..., u i-1 Predict u i P is the conditional probability, i is a positive integer, and Θ is the overall value space. Therefore, P(u i |u i-k , ..., u i-1 ;Θ) represents the value based on the historical word u in the overall value space. i-k , ..., u i-1 Predict u i The conditional probability.
[0102] The improved loss function is to combine log P(u3|u1, u2), ..., log P(u m |u m-k , ..., u m-1 ) is set to 0, that is, " <s>The loss for the part corresponding to "text" is set to 0, and log P(u) is set to 0. n |u n-k , ..., u n-1 ) is set to 0, that is, " <e>The corresponding partial loss is set to 0, which is u m In the formula, u represents the corresponding part. m The loss of the previous term is set to 0, u n express <e>The corresponding part.
[0103] As can be seen from the above method, when training the text generation model, this application first inputs the original text determined from the general text dataset into a pre-trained type recognition model to determine the template annotation of the original text. Then, based on the template annotation, the target template corresponding to the template annotation is determined. Next, training samples for training the text generation model are generated based on the original text, template annotation, and target template. The input is then fed into the text generation model to be trained to obtain the output text. The training objective is to minimize the difference between the sample annotation and the output text. This method can be used when the training samples for the text generation model are limited. By generating training samples based on the original text and template annotation from the general text dataset, and utilizing the target template, the number of training samples for training the text generation model can be increased. This allows for training the text generation model with sufficient training samples, resulting in a better training effect and improved accuracy of the output text.
[0104] In step S106 above, when the template annotation is a task type, the server can determine the first text and the second text contained in the original text. Based on the target template, it determines that the first text and the template annotation are concatenated as the input part of the target template, and determines the second text as the sample annotation of the target template. Then, based on the input part and the sample annotation, training samples for training the text generation model are generated.
[0105] In addition, when the template is labeled as a topic type, the server can determine the template label as the input part of the target template and the original text as the sample label of the target template based on the target template. Based on the input part and the sample label, the server generates training samples for training the text generation model.
[0106] The above only describes two types of template annotation (i.e., task type and topic type). The process of determining training samples based on the target templates corresponding to other types of template annotations is similar to the above process, and will not be repeated here.
[0107] In this specification, the template annotation for the original text can be one or multiple. In step S102 above, the server can determine the pre-trained recognition models for each type. For each type of recognition model, the original text is input into that type of recognition model to obtain the output result, and the obtained output results are used as the template annotation for the original text.
[0108] Based on this, in step S104 above, when determining the target template corresponding to the template annotation based on the template annotation, the server can determine the target template corresponding to multiple template annotations based on the multiple template annotations of the original text. For example, when the template annotations of the original text include five types: task type, keyword type, theme type, genre type, and sentiment type, the target template can be " <s>Please implement the task of type [Original Text 1], with the following specific requirements: the theme is [Topic Type], the generated text must contain [Keyword 1], [Keyword 2], ..., [Keyword N], the generated text must be of the [Genre Type], and the emotion must be generated according to the [Emotion Type]: [Original Text 2] <e>".
[0109] The server can also select at least one template annotation as the target template annotation from multiple template annotations in the original text, and determine the target template corresponding to the target template annotation. When selecting the target template annotation, the server can randomly select at least one template annotation from multiple template annotations, or it can select at least one template annotation as the target template annotation according to specified rules. The server can also determine the template corresponding to each template annotation in the original text.
[0110] In this specification, to make the text generation model more applicable to a specific scenario or more general, the user can select a type recognition model, ensuring that the template annotations of the original text determined by the type recognition model are unique or numerous. If the template annotation type is singular, the target templates determined based on the template annotations will be relatively singular, and the generated training samples will be more suitable for the text generation model in a specific scenario. If the template annotation type is numerous, multiple target templates can be determined based on the template annotations, and multiple training samples corresponding to the original text can be generated based on multiple target templates, increasing the training samples of the text generation model. Furthermore, different training samples can be generated based on different target templates, making the trained text generation model more general and applicable to different scenarios or tasks. Therefore, in step S102 above, the server can respond to the type information input by the user, determine the target type recognition model for the original text from the pre-trained type recognition model, input the original text into the target type recognition model, and determine the template annotations for the original text.
[0111] In this specification, the text generation model to be trained can also be any existing pre-trained text generation model. The server can fine-tune the pre-trained text generation model based on training samples. Therefore, in step S108 above, the server can input the input part into the pre-trained text generation model to obtain the output text. The training objective is to minimize the difference between the sample annotation and the output text, and then fine-tune the pre-trained text generation model. When fine-tuning the pre-trained text generation model to obtain the final text generation model, the improved loss function can be as follows:
[0112] L2(C)=∑logP(y|x1,…,x m )
[0113] L3(C) = L2(C) + λL1(C)
[0114] Where L1(C) is the loss of the model during pre-training, L3(C) is the improved loss function, and x1,…,x m express" <s>In the "text" section, y represents " <s>Text to <e>"In the middle part, λ is a hyperparameter. P(y|x1,…,x m ) represents based on x1,…,x m Predict the conditional probability of y.
[0115] In this specification, the prompts included in the input portion of the target template can be content that conforms to human language logic, which helps the output text of the subsequent text generation model to be more in line with human logical language and more controllable.
[0116] In this specification, after obtaining the trained text generation model, the server can respond to user input, determine the input text, input the input text into the trained text generation model, and obtain the output text. The input text and the training samples have the same format (i.e., "..."). <s>Input sample annotation <e>Similarly, the input text format is " <s>"Input section". For example, the text entered by the user (i.e., the input text) is " <s>We've assigned an essay titled "Summer in Hangzhou," can you help me write it? The input text is then fed into the trained text generation model to obtain the output text.
[0117] To ensure the quality of the text generation model and the output text, the server can input the input text into the text generation model multiple times, input the output results into the scoring function, obtain the score of each output result, and use the output result with the highest score as the output text.
[0118] The above describes one or more implementations of the methods described in this specification. Based on the same approach, this specification also provides training devices for corresponding text generation models, such as... Figure 4 As shown.
[0119] Figure 4 A schematic diagram of a training apparatus for a text generation model provided in this specification includes:
[0120] The first determining module 200 is used to determine the original text from a general text dataset;
[0121] The second determining module 202 is used to input the original text into a pre-trained type recognition model to determine the template annotation of the original text;
[0122] The third determining module 204 is used to determine the target template corresponding to the template annotation based on the template annotation;
[0123] The generation module 206 is used to generate training samples for the training text generation model based on the original text, the template annotation, and the target template, wherein the target template is used to divide at least a portion of the original text into sample annotations, and to concatenate the original text other than the sample annotations and the template annotations into an input part;
[0124] The training module 208 is used to input the input part into the text generation model to be trained, obtain the output text, and train the text generation model to be trained with the minimum difference between the sample label and the output text as the training objective.
[0125] Optionally, the first determining module 200 is specifically configured to: determine the parameter size of the text generation model to be trained; determine the number of training samples based on the parameter size; and determine the number of original texts from a general text dataset.
[0126] Optionally, the third determining module 204 is specifically used to select the target template corresponding to the template label from a set of preset prompt templates based on the template label.
[0127] Optionally, the template is labeled as a task type;
[0128] The generation module 206 is specifically used to: determine the first text and the second text contained in the original text; determine, according to the target template, to concatenate the first text and the template annotation as the input part of the target template, and determine the second text as the sample annotation of the target template; and generate training samples for training the text generation model according to the input part and the sample annotation.
[0129] Optionally, the template is labeled as a theme type;
[0130] The generation module 206 is specifically used to: determine the template annotation as the input part of the target template and determine the original text as the sample annotation of the target template according to the target template; and generate training samples for training the text generation model according to the input part and the sample annotation.
[0131] Optionally, the type recognition model is a text classification model;
[0132] The second determining module 202 is specifically used to input the original text into a pre-trained text classification model, determine the task type of the original text, and use it as a template annotation.
[0133] Optionally, the type recognition model is a keyword extraction model;
[0134] The second determining module 202 is specifically used to input the original text into a pre-trained keyword extraction model, determine the topic type of the original text, and use it as a template for annotation.
[0135] Optionally, the second determining module 202 is specifically used to, in response to type information input by the user, determine the target type recognition model of the original text from a pre-trained type recognition model; input the original text into the target type recognition model, and determine the template annotation of the original text.
[0136] Optionally, the second determining module 202 is specifically used to: determine the pre-trained recognition models of each type; for each type recognition model, input the original text into the type recognition model to obtain the output result; and use the obtained output results as template annotations for the original text.
[0137] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This paper presents a training method for a text generation model.
[0138] This instruction manual also provides Figure 5 The one shown corresponds to Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 5 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 The training method for the text generation model.
[0139] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0140] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.
[0141] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0142] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0143] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.
[0144] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0145] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0149] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0150] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0151] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0152] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0153] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0155] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.< / s> < / s> < / e> < / s> < / e> < / s> < / s> < / e> < / s> < / e> < / e> < / s> < / e> < / s> < / s> < / e> < / s> < / e> < / s> < / s> < / e> < / s> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s> < / e> < / s>
Claims
1. A method for training a text generation model, the method comprising: include: Determine the original text from a general text dataset; The original text is input into a pre-trained type recognition model to determine the template annotation of the original text; Based on the template annotation, determine the target template corresponding to the template annotation; According to the target template, the original text and the template annotation corresponding to the original text are filled into the target template to generate training samples for training the text generation model. The target template is used to divide at least a part of the original text into sample annotations and to concatenate the original text other than the sample annotations and the template annotations into the input part. The input part is input into the text generation model to be trained to obtain the output text. The text generation model to be trained is trained with the minimum difference between the sample label and the output text as the training objective. When the template is labeled as a task type, the step of filling the original text and the template label corresponding to the original text into the target template according to the target template to generate training samples for the training text generation model includes: Determine the first and second text contained in the original text; Based on the target template, the first text and the template annotation are concatenated as the input part of the target template, and the second text is determined as the sample annotation of the target template; Training samples for the training text generation model are generated based on the input and the sample annotations.
2. The method of claim 1, wherein, Determining the original text from a general text dataset specifically includes: Determine the parameter dimensions of the text generation model to be trained; The number of training samples is determined based on the parameter dimensions. The number of original texts is determined from a general text dataset.
3. The method as described in claim 1, characterized in that, Based on the template annotation, the target template corresponding to the template annotation is determined, specifically including: Based on the template annotation, select the target template corresponding to the template annotation from the preset prompt templates.
4. The method as described in claim 1, characterized in that, The template is labeled as a theme type; Based on the original text, the template annotation, and the target template, training samples for the text generation model are generated, specifically including: Based on the target template, the template annotation is determined as the input part of the target template, and the original text is determined as the sample annotation of the target template; Training samples for the training text generation model are generated based on the input and the sample annotations.
5. The method as described in claim 1, characterized in that, The type recognition model is a text classification model; The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including: The original text is input into a pre-trained text classification model to determine the task type of the original text, and is then used as a template for annotation.
6. The method as described in claim 1, characterized in that, The type identification model is a keyword extraction model; The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including: The original text is input into a pre-trained keyword extraction model to determine the topic type of the original text, which is then used as a template for annotation.
7. The method as described in claim 1, characterized in that, The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including: In response to type information input by the user, the target type recognition model of the original text is determined from a pre-trained type recognition model; The original text is input into the target type recognition model to determine the template annotation of the original text.
8. The method as described in claim 1, characterized in that, The original text is input into a pre-trained type recognition model to determine the template annotation of the original text, specifically including: Identify the pre-trained recognition models for each type; For each type recognition model, the original text is input into that type recognition model to obtain the output result; The obtained output results are used as template annotations for the original text.
9. A training device for a text generation model, characterized in that, include: The first determination module is used to determine the original text from a general text dataset; The second determining module is used to input the original text into a pre-trained type recognition model to determine the template annotation of the original text; The third determining module is used to determine the target template corresponding to the template annotation based on the template annotation; The generation module is used to fill the original text and the template annotation corresponding to the original text into the target template according to the target template, and generate training samples for training the text generation model. The target template is used to divide at least a part of the original text into sample annotations, and to concatenate the original text other than the sample annotations and the template annotations into the input part. Wherein, when the template is labeled as a task type, the step of filling the original text and the template label corresponding to the original text into the target template according to the target template to generate training samples for the training text generation model includes: Determine the first and second text contained in the original text; Based on the target template, the first text and the template annotation are concatenated as the input part of the target template, and the second text is determined as the sample annotation of the target template; Based on the input portion and the sample annotations, training samples for the training text generation model are generated; The training module is used to input the input part into the text generation model to be trained, obtain the output text, and train the text generation model to be trained with the minimum difference between the sample label and the output text as the training objective.
10. The apparatus as claimed in claim 9, characterized in that, The first determining module is specifically used to: determine the parameter size of the text generation model to be trained; determine the number of training samples based on the parameter size; and determine the number of original texts from a general text dataset.
11. The apparatus as claimed in claim 9, characterized in that, The third determining module is specifically used to select the target template corresponding to the template annotation from the preset prompt templates according to the template annotation.
12. The apparatus as claimed in claim 9, characterized in that, The template is labeled as a theme type; The generation module is specifically used to: determine the template annotation as the input part of the target template and determine the original text as the sample annotation of the target template based on the target template; and generate training samples for training the text generation model based on the input part and the sample annotation.
13. The apparatus as claimed in claim 9, characterized in that, The type recognition model is a text classification model; The second determining module is specifically used to input the original text into a pre-trained text classification model, determine the task type of the original text, and use it as a template for annotation.
14. The apparatus as claimed in claim 9, characterized in that, The type identification model is a keyword extraction model; The second determining module is specifically used to input the original text into a pre-trained keyword extraction model, determine the topic type of the original text, and use it as a template for annotation.
15. The apparatus as claimed in claim 9, characterized in that, The second determining module is specifically used to, in response to type information input by the user, determine the target type recognition model of the original text from a pre-trained type recognition model; input the original text into the target type recognition model, and determine the template annotation of the original text.
16. The apparatus as claimed in claim 9, characterized in that, The second determining module is specifically used to: determine the pre-trained recognition models of each type; for each type recognition model, input the original text into the type recognition model to obtain the output result; and use the obtained output results as template annotations for the original text.
17. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 8.
18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 8.