Training method of text generation model, method of generating text, and respective apparatuses

Abstract

Embodiments of the present specification provide a text generation model training method, a text generation method, and respective devices, wherein the text generation model training method comprises: obtaining a text generation model and a plurality of respective samples of tasks having text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a label accuracy level; for each training stage, obtaining a sample set of the training stage according to the samples of the respective samples of the plurality of tasks that match the training target of the training stage in terms of the label accuracy level; for a first training stage of the plurality of training stages, training the text generation model using the sample set of the first training stage to obtain an updated text generation model; and for any training stage after the first training stage, continuing to train the text generation model using the sample set of the any training stage to obtain an updated text generation model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a text generation model training method, a text generation method, and respective devices. BACKGROUND

[0002] In natural language processing, an important field is automatic generation of data to text. For example, automatic generation of phrases, sentences, paragraphs, articles, etc. from some data all belong to the application of this field. It can be from more to less generation, or from less to more generation. In the generation from data to text, different tasks are usually divided according to different data sources. For example, table to text generation task, SQL statement to text generation task, and various text generation tasks.

[0003] At present, different tasks are modeled, trained respectively under the condition that the data sources are different, which consumes a lot of time and leads to low model training efficiency. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a text generation model training method and a text generation method. One or more embodiments of the present specification also relate to a text generation model training device and a text generation device, a computing device, a computer readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a text generation model training method is provided, comprising: obtaining a text generation model and a plurality of tasks each having samples related to text, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a labeled accuracy level; for each training stage, obtaining a sample set of the training stage according to the samples of the plurality of tasks each having a labeled accuracy level matching the training target of the training stage, wherein the sample sets of different training stages are different; for a first training stage of the plurality of training stages, training the text generation model using the sample set of the first training stage to obtain an updated text generation model; for any training stage after the first training stage, training the text generation model obtained by the previous training stage of the any training stage using the sample set of the any training stage to obtain an updated text generation model.

[0006] According to a second aspect of the embodiments of the present specification, a training device of a text generation model is provided, comprising: an initialization module configured to obtain a text generation model and a plurality of task-specific samples with text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a labeled accuracy level. A sample determination module is configured to obtain, for each training stage, a sample set of the training stage from the samples of the plurality of task-specific samples with a labeled accuracy level matching the training target of the training stage, wherein the sample sets of different training stages are different. A first updating module is configured to train the text generation model using the sample set of a first training stage in the plurality of training stages to obtain an updated text generation model. A second updating module is configured to train the text generation model obtained by the previous training stage of any training stage using the sample set of the any training stage to obtain an updated text generation model.

[0007] According to a third aspect of the embodiments of the present specification, a method for generating text is provided, comprising: obtaining one or more task-specific structured data; parsing the one or more task-specific structured data to obtain a plurality of nodes and relationships between the nodes of the one or more tasks; converting the plurality of nodes and relationships between the nodes of the one or more tasks to obtain a graph of the one or more tasks and obtaining a vector representation of each node; inputting the graph of the one or more tasks and the vector representation of each node into a text generation model trained by a training method of the text generation model as described in any embodiment of the present specification to obtain text output by the text generation model.

[0008] According to a fourth aspect of the embodiments of the present specification, a device for generating text is provided, comprising: a data acquisition module configured to obtain one or more task-specific structured data. A structure parsing module is configured to parse the one or more task-specific structured data to obtain a plurality of nodes and relationships between the nodes of the one or more tasks. A structure conversion module is configured to convert the plurality of nodes and relationships between the nodes of the one or more tasks to obtain a graph of the one or more tasks and obtain a vector representation of each node. A text output module is configured to input the graph of the one or more tasks and the vector representation of each node into a text generation model trained by a training method of the text generation model as described in any embodiment of the present specification to obtain text output by the text generation model.

[0009] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the training method of the text generation model or the steps of the method of generating text.

[0010] According to a sixth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which, when executed by a processor, implement the steps of the training method of the text generation model or the steps of the method of generating text.

[0011] According to a seventh aspect of the embodiments of the present specification, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the training method of the text generation model or the steps of the method of generating text.

[0012] One embodiment of the present specification provides a training method of a text generation model, which obtains a text generation model and samples of a plurality of tasks respectively having text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponds to a different training target, and the samples have a labeled accuracy level; for each training stage, a sample set of the training stage is obtained according to the samples of the plurality of tasks respectively having text relevance, which have a labeled accuracy level matching the training target of the training stage, wherein the sample sets of different training stages are different; for a first training stage of the plurality of training stages, the text generation model is trained using the sample set of the first training stage to obtain an updated text generation model; for any training stage after the first training stage, the text generation model updated by the previous training stage of the any training stage is trained using the sample set of the any training stage to obtain an updated text generation model. It can be seen that the method jointly models a plurality of tasks having text relevance, one model is applied to multiple tasks, and the samples are classified according to different labeled accuracy levels and trained in stages, the samples of the accuracy level matching the training target are used for training in each training stage, and the accuracy of the text generation model is gradually improved through multiple stages of training. Moreover, because the plurality of tasks have text relevance, the samples of the plurality of tasks interact with each other during the training process to promote the convergence of the model, effectively reducing the training time, fully improving the model training efficiency, and finally quickly obtaining a text generation model applied to multiple tasks.

[0013] Another embodiment of the present specification provides a method for generating text, wherein the method obtains structured data of one or more tasks respectively, parses the structured data of the one or more tasks respectively to obtain a plurality of nodes and relationships between the nodes of the one or more tasks respectively, converts the plurality of nodes and relationships between the nodes of the one or more tasks respectively to obtain a graph of the one or more tasks respectively, and obtains a vector representation of each node, so as to convert input data of different structures into a unified graph structure, and then the graph of the one or more tasks respectively and the vector representation of the nodes can be input into a text generation model trained by a training method of the text generation model as described in any embodiment of the present specification, so as to obtain text output by the text generation model, and therefore, one text generation model can be used to generate text for a plurality of tasks of different data sources, thereby improving the text generation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of a training method of a text generation model provided by an embodiment of the present specification;

[0015] Figure 2 is a schematic diagram of table data converted into a graph provided by an embodiment of the present specification;

[0016] Figure 3 is a schematic diagram of SQL data converted into a graph provided by an embodiment of the present specification;

[0017] Figure 4 is a schematic diagram of a graph structure provided by another embodiment of the present specification;

[0018] Figure 5 is a structural schematic diagram of a text generation model provided by an embodiment of the present specification;

[0019] Figure 6 is a schematic diagram of a sample data proportion in a plurality of training stages of a multi-task sample provided by an embodiment of the present specification;

[0020] Figure 7 is a structural schematic diagram of a training device of a text generation model provided by an embodiment of the present specification;

[0021] Figure 8 is a flowchart of a method for generating text provided by an embodiment of the present specification;

[0022] Figure 9 is a structural schematic diagram of a device for generating text provided by an embodiment of the present specification;

[0023] Figure 10 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0024] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details, and in other instances, well-known methods have not been described in detail in order not to obscure typically overlapping or inherently

[0025] The terminology used in one or more embodiments of the present description is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present description. As used in one or more embodiments of the present description and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present description, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first can be termed a second, and, similarly, a second can be termed a first, without departing from the scope of one or more embodiments of the present description. As used herein, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" depending on the context.

[0027] In the present description, a method for training a text generation model and a method for generating text are provided, and the present description also relates to an apparatus for training a text generation model and an apparatus for generating text, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0028] To make the training method of the text generation model provided by the embodiments of the present specification easier to understand, first, taking the generation of structured data to text as an example, the application scenario of the training method of the text generation model is exemplarily described. For example, structured data to text generation includes Table-to-Text, SQL-to-Text, ResponseNLG and other tasks. Among them, Table-to-text aims to generate corresponding textual description from given structured table data, which can help people quickly obtain key information in structured data; SQL-to-Text task generates natural language questions from SQL statements, which can be used as data augmentation for semantic parsing tasks; ResponseNLG is a reply generation task based on table and SQL statement, which can generate corresponding replies based on SQL and table. More specifically, for example, the ResponseNLG task can be a task of generating natural language replies based on the current SQL query and the query result table. The data sources of the three tasks are different, and the data quality under different annotation specifications is uneven, resulting in low training efficiency of the text generation model. According to the method provided in the embodiments of the present specification, multiple tasks with text relevance such as Table-to-Text generation, SQL-to-Text, ResponseNLG can be jointly pre-trained, different training stages are divided according to the annotation accuracy level, and finally applied to Table-to-Text task, SQL-to-Text task and ResponseNLG task, thereby improving the training efficiency of the model.

[0029] Specifically, referring to Figure 1 , Figure 1 a flowchart of a training method of a text generation model according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0030] Step 102: obtaining a text generation model and samples of multiple tasks with text relevance respectively, wherein the text generation model includes multiple training stages, each training stage corresponds to a different training target, and the samples have annotation accuracy levels.

[0031] The machine learning method adopted by the text generation model is not limited. For example, the text generation model can adopt a Transformer model for construction. The Transformer is a deep learning model based on an encoder-decoder structure and a self-attention mechanism. Specifically, the Transformer can include a stack of multiple Encoders and a stack of multiple Decoders. Each Encoder can have two sub-layers: a multi-head self-attention layer and a fully connected feed-forward neural network layer. The connection of the sub-layers can use LayerNorm and residual connection, which can avoid gradient vanishing and explosion. Each Decoder can have multiple sub-layers, such as a multi-head self-attention layer and a fully connected feed-forward neural network layer.

[0032] The multiple tasks have text relevance, which can be understood as the input data of the multiple tasks being different in origin, but the input data of two tasks having text relevance. In the case where the multiple tasks have text relevance, after the text generation model is jointly constructed, the samples of the multiple tasks can interact with each other to promote model convergence based on a multi-stage training process.

[0033] The samples of the multiple tasks can be obtained in any manner. For example, the data source of the samples can be artificially collected data, network crawled data, platform processed data, and the like. Each task has a plurality of samples, and each sample has a labeling accuracy level. The labeling accuracy level is used to represent the labeling quality of the sample, for example, it can be no labeling, coarse labeling, and fine labeling. For example, some samples are network crawled data, and the labeling accuracy level of the sample can be no labeling. For another example, some labeling personnel lack experience, and the labeling accuracy level of the sample can be coarse labeling. For yet another example, some labeling personnel have rich experience, and the labeling accuracy level of the sample can be fine labeling.

[0034] The number of the multiple training stages and the training targets can be set according to the needs of the implementation scene, and the method provided by the embodiments of the present disclosure does not limit this. It can be understood that the training target can be a target for improving the model accuracy of any one or more tasks. According to the needs of the implementation scene, the corresponding training target can be set to improve the model accuracy of different tasks in different training stages.

[0035] It should be noted that the multiple tasks can include two or more than two tasks. The multiple training stages can include two or more than two stages.

[0036] Step 104: For each training stage, a sample set of the training stage is obtained according to the samples of the respective sample set of the plurality of tasks that match the annotation precision level of the training target of the training stage, wherein the sample sets of different training stages are different.

[0037] It can be understood that under different training targets, the samples participating in training can be selected as needed, as long as the training target is reached. Therefore, when selecting samples as needed, the training target can be matched with the annotation precision level of the task to determine the samples that meet the needs of the training target for training.

[0038] The sample sets of different training stages are different because the training targets of each training stage are different, and the selection of samples will naturally change. For example, the training target of the first training stage is full-quantity training of the plurality of tasks, at this time, all samples of the annotation precision level match the training target, and all samples can be added to the sample set of the first training stage. For another example, the training target of a certain training stage is to improve the precision of a certain task, at this time, the samples of the high annotation precision level of the task match the training target, and the samples of the high annotation precision level of the task and the samples of other tasks that do not affect or can improve the training precision of the task are added to the sample set of the training stage.

[0039] Step 106: For the first training stage of the plurality of training stages, the sample set of the first training stage is used to train the text generation model to obtain an updated text generation model.

[0040] The specific implementation of the training process of the text generation model is not limited, and can be set according to the machine learning algorithm used. For example, taking a text generation model based on Transformer as an example, the training of Transformer is parallel, that is, the vectors of all nodes of the input sequence of a sample are trained at the same time, which greatly increases the calculation efficiency. Transformer uses Positional Encoding to understand the order of language, and uses self-attention mechanism and full connection layer for calculation. The Transformer model mainly includes two parts, namely, Encoder and Decoder. The Encoder is responsible for mapping the input (language sequence) to the hidden layer, and then the Decoder decoder maps the hidden layer to the natural language sequence, so as to output the text. After outputting the text, the parameters of the text generation model can be adjusted for further training until the training stopping condition (such as the number of iterations reaching the preset number, or the loss value reaching the preset threshold) is reached, and an updated text generation model can be obtained.

[0041] Step 108: for any training stage after the first training stage, using the sample set of the any training stage to train the text generation model obtained by the previous training stage of the any training stage, to obtain an updated text generation model.

[0042] The plurality of training stages means that text model training is performed stage by stage, and the text generation model used in a later training stage is the text generation model trained in the previous training stage, so that the text generation model is trained in multiple stages and the accuracy is continuously improved. Moreover, each task in the plurality of tasks can be at least correspondingly set in the training target of at least one training stage in the plurality of training stages, so that for the plurality of tasks, after multiple training stages, the final text generation model can be applied to the plurality of tasks to realize text generation.

[0043] It can be seen that the method jointly models the plurality of tasks with text correlation, one model is applied to the plurality of tasks, the samples are divided into levels according to different annotation accuracies, and then divided into stages for training, samples with an accuracy level matched with the training target are used for training in each training stage, and the accuracy of the text generation model is gradually improved through multiple stages of training. Moreover, because the plurality of tasks have text correlation, the samples of the plurality of tasks interact with each other during the training process to promote the convergence of the model, effectively reducing the training time and fully improving the model training efficiency, and finally the text generation model applied to the plurality of tasks can be quickly obtained.

[0044] For example, the plurality of tasks includes a first task and a second task, the samples of the first task include samples of a first annotation accuracy level and samples of a second annotation accuracy level, the samples of the second task include samples of the first annotation accuracy level and samples of the second annotation accuracy level, and the annotation accuracy of the first annotation accuracy level is greater than the annotation accuracy of the second annotation accuracy level. Accordingly, the obtaining, for each training stage, a sample set of the training stage according to samples of an annotation accuracy level matched with a training target of the training stage in the respective samples of the plurality of tasks can include:

[0045] For the first training stage, according to the training target of the training stage being full-amount training, using the samples of the first annotation accuracy level and the second annotation accuracy level of the first task, and the samples of the first annotation accuracy level and the second annotation accuracy level of the second task, to obtain the sample set of the training stage;

[0046] For the second training stage, according to the training target of the training stage being training to improve the accuracy of the first task, using the samples of the first annotation accuracy level of the first task and the samples of the first annotation accuracy level and the second annotation accuracy level of the second task, to obtain the sample set of the training stage;

[0047] For the third training stage, according to the training target of the training stage, the training for improving the accuracy of the second task, the sample set of the training stage is obtained by using the sample of the first annotation accuracy level of the second task.

[0048] In the above embodiment, the first training stage is equivalent to pre-training of mixed data of various accuracies of multiple tasks, and the trained text generation model has a preliminary ability to generate text for multiple tasks. The second training stage is equivalent to removing the coarse annotation sample of the first task, and using the fine annotation sample of the first task and the sample of the second task for continuous training. The third training stage is equivalent to removing the fine annotation sample of the first task and the coarse annotation sample of the second task, and using the fine annotation sample of the first task for continuous training. The text generation model obtained after training has the ability to accurately output text for the first task and the second task.

[0049] In combination with the above embodiment, in one or more embodiments of the present specification, the multiple tasks can further include a third task having text relevance with the first task and the second task. Accordingly, the sample set of the first training stage further includes samples of the third task, the samples of the third task are samples of a third annotation accuracy level, and the annotation accuracy of the third annotation accuracy level is less than the annotation accuracy of the second annotation accuracy level. In this embodiment, only the first training stage uses the samples of the third task, thereby improving the generalization by means of the samples of the third task, and avoiding the influence of low-precision annotation of the third task on the model accuracy in the subsequent stage.

[0050] In the above embodiment, the model gradually converges for the first task and the second task in multiple stages during the training process, and finally a text generation model applicable to the first task and the second task is trained, which effectively reduces the training time and fully improves the model training efficiency.

[0051] The application scenarios of the method provided by the embodiments of the present specification are not limited, and the following will be described by way of example in combination with multiple tasks, each of which is a task of generating text from structured data of different structures. In order to unify and jointly model multiple tasks with heterogeneous data, the method provided by the embodiments of the present specification first needs to convert the structured data of multiple tasks into a unified graph structure. Therefore, the method can further include:

[0052] Obtaining structured data of the multiple tasks respectively;

[0053] Parsing the structured data of the multiple tasks respectively to obtain a plurality of nodes and relationships between the nodes of the multiple tasks respectively;

[0054] obtaining a graph of each of the plurality of tasks by using the plurality of nodes and the relationship between the nodes of each of the plurality of tasks, and obtaining a vector representation of each of the nodes;

[0055] obtaining a sample of each of the plurality of tasks by using the graph and the vector representation of the nodes.

[0056] For example, in one or more embodiments of the present specification, one task is a task of generating text from SQL data, another task is a task of generating text from SQL data and table data, and another task is a task of generating text from table data. For the three tasks, the input heterogeneous data is first uniformly converted into a graph structure. Specifically:

[0057] For example, for table data, the cells of the table can be regarded as nodes, and the edges between the nodes can be established according to the relationship between the cells. For example, a rule for establishing edges can be set in advance, such as identifying nodes in the same row, regarding nodes in the same row as nodes having a relationship to construct edges; such as identifying nodes in the same column, regarding nodes in the same column as nodes having a relationship to construct edges; identifying nodes in the same table, regarding nodes in the same table as nodes having a relationship to construct edges, and the like, and finally converting into a graph structure. For example, as shown in the schematic diagram of the table data conversion graph shown in Figure 2 , the table is converted to obtain the corresponding graph structure.

[0058] For example, for SQL data, the nodes and the edges between the nodes can be determined according to the syntax constraint rules. For example, as shown in the schematic diagram of the SQL data conversion graph shown in Figure 3 , the SQL data is converted to obtain the corresponding graph structure.

[0059] For example, for data containing both table and SQL, the above examples can be followed to first convert the SQL data and the table data into graph structures, and then connect the associated nodes in the graph structures of the two to form a complete graph structure. For example, as shown in the graph structure shown in Figure 2 , after connecting the associated nodes, the graph structure shown in Figure 4 is obtained. For example, as shown in the graph structure shown in Figure 2 , the graph structure shown in Figure 2 , Figure 3 , the "ROOT" in the graph structure represents the root node of the tree-shaped graph structure, and the intermediate nodes or leaf nodes are connected according to the association relationship.

[0060] In the application scenario of structured data generating text, the text generation model can be a language model introduced with a structured adapter. For example, a pre-trained language model introduced with a structured adapter can be used as an encoder (i.e., SA-GAT). The structured adapter is used to extract features from the structure of the graph. The structured adapter can be understood as a module including a graph attention network.

[0061] For example, the language model introduced with the structured adapter can include a transformer encoding module (Transformer Encoder Block) and a transformer decoding module (Transformer Decoder Block), for example, as shown in the text generation model structure diagram Figure 5 The structured adapter is located after the feedforward network sublayer of the transformer encoding module. As shown in the text generation model structure diagram Figure 5 The structured adapter (L-NE) shown in Figure 5 may include a Layer Normalization module and a Graph Attention Network.

[0062] Where "SQL", "…", "low", "risk", "…", and "%" represent input sequences.

[0063] The Layer Normalization module is used to calculate the expectation and label difference of each layer of the neural network, standardize the node vector, and input the processed node vector to the graph attention network after activation function processing.

[0064] The graph attention network is used to extract relevant features from the input graph and add the extracted relevant features to the node vector.

[0065] For example, the model expression of the graph attention network is as follows:

[0066]

[0067]

[0068]

[0069] h r = Concat(z 1 , …, z H )

[0070] where 1≤h≤H, v, h are used to represent nodes; W, Q, K represent parameters used in the model; and N(v) represents the direct neighbor nodes of node v in the graph, represents the correlation score of h node and v node, n represents the number of nodes, h n is all the neighbor nodes of v; represents the correlation of all nodes is normalized; z h represents the weighted sum of the node vector by the correlation coefficient, which contains the information of all the neighbor nodes of the current node; h r represents the parallel H times calculation results are linked together to obtain the representation of the node vector.

[0071] For example, the plurality of tasks include a first task, a second task, and a third task. The first task is a task of generating text from SQL data (which can be referred to as a SQL-to-Text task or SQL2Text), the second task is a task of generating text from SQL data and table data (which can be referred to as a ResponseNLG task), and the third task is a task of generating text from table data (which can be referred to as a Table-to-Text task or Table2Text). The text correlation of the SQL-to-Text task and the ResponseNLG task is specifically manifested in that both tasks are related to SQL data, and the text correlation of the ResponseNLG task and the Table-to-Text task is specifically manifested in that both tasks are related to table data. The text correlation of the SQL-to-Text task and the Table-to-Text task is specifically manifested in that the input data of both tasks is related to the input data of the ResponseNLG task. Since the data sources of the three tasks are different, and the data quality under different annotation specifications is uneven, the SQL-to-Text task has samples of two annotation accuracy levels of rough and fine, the ResponseNLG task also has samples of two annotation accuracy levels of rough and fine, and the annotation accuracy level of the Table-to-Text task sample is no annotation. Therefore, according to the method provided in the embodiment of the present specification, the sample data is divided into different annotation accuracy levels according to quality, and then trained in multiple training stages. Specifically:

[0072] First, the SQL data of the SQL-to-Text task is converted to obtain corresponding graph structure data, the table data of the Table-to-Text task is converted to obtain corresponding graph structure data, and the SQL data and the table data of the ResponseNLG task are converted to obtain corresponding graph structure data. The graph structure data and the node vector of each node in the graph structure are used to obtain the samples of the SQL-to-Text task, the Table-to-Text task and the ResponseNLG task.

[0073] Next, as Figure 6 The multi-task samples are shown in the sample data proportion diagram in the multi-training stage. The text generation model is trained in three training stages:

[0074] In the first training stage stag1, the training target is the full data pre-training of the SQL-to-Text task, the Table-to-Text task and the ResponseNLG task. The samples of the three tasks are put into a sample set, and the pre-training text generation model is trained using the sample set.

[0075] In the second training stage stag2, the training target is the labeled data pre-training of the SQL-to-Text task and the ResponseNLG task. The samples of the Table-to-Text task, the coarse-labeled samples of the SQL-to-Text task, the fine-labeled samples of the SQL-to-Text task and the samples of the ResponseNLG task are not used, and the text generation model is continued to be pre-trained. The text generation model trained in this training stage has the ability to accurately output text for SQL data.

[0076] In the third training stage stag3, the training target is the fine-labeled data fine-tuning of the ResponseNLG task. The fine-labeled samples of the SQL-to-Text task and the coarse-labeled samples of the ResponseNLG task are not used, and the text generation model is continued to be trained using the fine-labeled samples of the ResponseNLG task. The text generation model trained in this training stage has the ability to accurately output text for ResponseNLG data.

[0077] Corresponding to the method embodiments described above, the present specification also provides a training device embodiment of a text generation model, Figure 7 A structural diagram of a training device of a text generation model is shown in an embodiment of the present specification. As Figure 7 indicated, the device comprises:

[0078] The initialization module 702 can be configured to obtain a text generation model and a plurality of samples of respective tasks having text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a labeled accuracy level.

[0079] The sample determination module 704 can be configured to, for each training stage, obtain a sample set of the training stage according to the samples of the respective tasks having a labeled accuracy level matching the training target of the training stage, wherein the sample sets of different training stages are different.

[0080] The first update module 706 can be configured to, for a first training stage of the plurality of training stages, train the text generation model using the sample set of the first training stage to obtain an updated text generation model.

[0081] The second update module 708 can be configured to, for any training stage after the first training stage, train the text generation model obtained by the previous training stage of the any training stage using the sample set of the any training stage to obtain an updated text generation model.

[0082] The device jointly models a plurality of tasks having text relevance, one model is applied to multiple tasks, and the samples are classified according to different labeled accuracy levels and trained in stages, samples of an accuracy level matching the training target are used for training in each training stage, and the accuracy of the text generation model is gradually improved through multiple stages of training. Moreover, because the multiple tasks have text relevance, the samples of the multiple tasks interact with each other during the training process to promote model convergence, effectively reducing the training time, fully improving the model training efficiency, and ultimately quickly obtaining a text generation model applied to multiple tasks.

[0083] In one or more embodiments of the specification, the plurality of tasks are respectively tasks of generating text from structured data of different structures; and the text generation model is a language model introducing a structured adapter, wherein the structured adapter is used for feature extraction on the structure of a graph. Accordingly, the device can further comprise:

[0084] The data input module is configured to obtain structured data of the respective tasks;

[0085] The data analysis module is configured to analyze the structured data of the respective tasks to obtain a plurality of nodes and relationships between the nodes of the respective tasks;

[0086] a graph conversion module configured to convert the plurality of tasks into respective graphs using the plurality of tasks and the relationships between the nodes;

[0087] a sample generation module configured to generate samples for the plurality of tasks using the graphs and the vector representations of the nodes.

[0088] In one or more embodiments of the present specification, the language model of the introduced structured adapter includes a transformer encoding module and a transformer decoding module, wherein the structured adapter is located after a feedforward network sublayer of the transformer encoding module.

[0089] In one or more embodiments of the present specification, the plurality of tasks includes a first task and a second task. The samples of the first task include samples of a first annotation accuracy level and samples of a second annotation accuracy level, and the samples of the second task include samples of the first annotation accuracy level and samples of the second annotation accuracy level. The annotation accuracy of the first annotation accuracy level is higher than the annotation accuracy of the second annotation accuracy level. Accordingly, the sample determination module can include:

[0090] A one-stage sample determination module can be configured to, for a first training stage, use the samples of the first annotation accuracy level and the second annotation accuracy level of the first task and the samples of the first annotation accuracy level and the second annotation accuracy level of the second task to obtain a sample set for the training stage according to a training target of the training stage being full-amount training.

[0091] A two-stage sample determination module can be configured to, for a second training stage, use the samples of the first annotation accuracy level of the first task and the samples of the first annotation accuracy level and the second annotation accuracy level of the second task to obtain a sample set for the training stage according to a training target of the training stage being training to improve the accuracy of the first task.

[0092] A third-stage sample determination module can be configured to, for a third training stage, use the samples of the first annotation accuracy level of the second task to obtain a sample set for the training stage according to a training target of the training stage being training to improve the accuracy of the second task.

[0093] In one or more embodiments of the present specification, the first task can be a task of generating text from SQL data, and the second task can be a task of generating text from SQL data and table data.

[0094] In one or more embodiments of the present specification, the plurality of tasks can further include a third task having text relevance with the first task and the second task. The sample set of the first training stage further includes a sample of the third task, and the sample of the third task is a sample of a third annotation precision level, and the annotation precision of the third annotation precision level is less than the annotation precision of the second annotation precision level.

[0095] For example, the third task is a task of generating text from table data.

[0096] The above is a schematic scheme of the training device of the text generation model according to the present embodiment. It should be noted that the technical scheme of the training device of the text generation model belongs to the same concept as the technical scheme of the training method of the text generation model described above, and the details of the technical scheme of the training device of the text generation model which are not described in detail can be referred to the description of the technical scheme of the training method of the text generation model.

[0097] Referring to Figure 8 , Figure 8 A flowchart of a method for generating text according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0098] Step 802: Obtain the structured data of each of one or more tasks.

[0099] Step 804: Analyze the structured data of each of the one or more tasks to obtain a plurality of nodes and relationships between the nodes of each of the one or more tasks.

[0100] Step 806: Convert the plurality of nodes and relationships between the nodes of each of the one or more tasks to obtain a graph of each of the one or more tasks, and obtain a vector representation of each node.

[0101] Step 808: Input the graph of each of the one or more tasks and the vector representation of each node into a text generation model trained by applying the training method of the text generation model described in any embodiment of the present specification to obtain text output by the text generation model.

[0102] The above is a schematic scheme of the method for generating text according to the present embodiment. It should be noted that the technical scheme of the method for generating text belongs to the same concept as the technical scheme of the training method of the text generation model described above, and the details of the technical scheme of the method for generating text which are not described in detail can be referred to the description of the technical scheme of the training method of the text generation model.

[0103] Corresponding to the above method for generating text, the present specification also provides a device for generating text, Figure 9A structural diagram of an apparatus for generating text is shown. As shown in Figure 9 The apparatus comprises:

[0104] The data obtaining module 902 can be configured to obtain structured data of one or more tasks respectively.

[0105] The structure analyzing module 904 can be configured to analyze the structured data of the one or more tasks respectively to obtain a plurality of nodes and relationships between the nodes of the one or more tasks respectively.

[0106] The structure converting module 906 can be configured to convert the plurality of nodes and relationships between the nodes of the one or more tasks respectively to obtain a graph of the one or more tasks respectively and obtain a vector representation of each node.

[0107] The text output module 908 can be configured to input the graph of the one or more tasks respectively and the vector representation of each node into a text generation model trained by the training method of the text generation model described in any embodiment of the present specification to obtain text output by the text generation model.

[0108] The above is a schematic scheme of the apparatus for generating text of the present embodiment. It should be noted that the technical scheme of the apparatus for generating text and the technical scheme of the method for generating text described above belong to the same concept, and the details of the technical scheme of the apparatus for generating text which are not described in detail can be referred to the description of the technical scheme of the method for generating text.

[0109] Figure 10 A structural block diagram of a computing device 1000 is shown according to an embodiment of the present specification. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 through a bus 1030, and a database 1050 is used to save data.

[0110] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 1040 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a global system for mobile communications (GSM) interface, a code division multiple access (CDMA) interface, a Bluetooth interface, a near field communication (NFC) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a Wi-MAX interface, an Ethernet interface, or the like.

[0111] In one embodiment of the present specification, the above-mentioned components of the computing device 1000 and other components not shown in the Figure 10 may be connected to each other, for example, through a bus. It should be understood that Figure 10 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0112] The computing device 1000 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 1000 can also be a mobile or stationary server.

[0113] The processor 1020 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the training method of the text generation model, for example, including:

[0114] obtaining a text generation model and a plurality of samples of respective tasks having text relevance, wherein the text generation model includes a plurality of training stages, each training stage corresponding to a different training target, and the samples have a labeled accuracy level;

[0115] For each training stage, a sample set of the training stage is obtained according to the samples of the respective tasks having the labeled accuracy level matching the training target of the training stage in the plurality of samples of the respective tasks, wherein the sample sets of different training stages are different;

[0116] For a first training stage of the plurality of training stages, the text generation model is trained using the sample set of the first training stage to obtain an updated text generation model;

[0117] For any training stage after the first training stage, the text generation model obtained by the previous training stage of the any training stage is trained using the sample set of the any training stage to obtain an updated text generation model.

[0118] Alternatively, the processor 1020 is configured to execute computer executable instructions, which, when executed by the processor, implement the steps of the above method for generating text, for example comprising:

[0119] Obtaining structured data of each of the one or more tasks;

[0120] Parsing the structured data of each of the one or more tasks to obtain a plurality of nodes and relationships between the nodes of each of the one or more tasks;

[0121] Converting the plurality of nodes and relationships between the nodes of each of the one or more tasks to obtain a graph of each of the one or more tasks, and obtaining a vector representation of each node;

[0122] Inputting the graph of each of the one or more tasks and the vector representation of each node into the text generation model trained according to the above method for training a text generation model to obtain text output by the text generation model.

[0123] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical scheme of the above method for training a text generation model, or the technical scheme of the above method for generating text. Details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the above method for training a text generation model, or the technical scheme of the above method for generating text.

[0124] An embodiment of the present specification also provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the above method for training a text generation model, for example comprising:

[0125] Obtaining a text generation model, and samples of a plurality of tasks each having text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a labeled accuracy level;

[0126] For each training stage, a sample set of the training stage is obtained according to the samples of the plurality of tasks that match the label precision level of the training target of the training stage, wherein the sample sets of different training stages are different;

[0127] For a first training stage of the plurality of training stages, the text generation model is trained using the sample set of the first training stage to obtain an updated text generation model;

[0128] For any training stage after the first training stage, the text generation model obtained by the previous training stage of the any training stage is trained using the sample set of the any training stage to obtain an updated text generation model.

[0129] Alternatively, the computer executable instructions are executed by the processor to implement the steps of the above method for generating text, for example, comprising:

[0130] Obtaining structured data of one or more tasks respectively;

[0131] Parsing the structured data of the one or more tasks respectively to obtain a plurality of nodes and relationships between the nodes of the one or more tasks respectively;

[0132] Converting the plurality of nodes and relationships between the nodes of the one or more tasks respectively to obtain a graph of the one or more tasks respectively, and obtaining a vector representation of each node;

[0133] Inputting the graph and the vector representation of each node of the one or more tasks respectively into the text generation model trained by the training method of the text generation model to obtain the text output by the text generation model.

[0134] The above is a schematic scheme of a computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium belongs to the same concept as the technical scheme of the training method of the text generation model or the technical scheme of the method for generating text, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the training method of the text generation model or the technical scheme of the method for generating text.

[0135] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer executes the steps of the above training method of the text generation model. For example, comprising:

[0136] obtaining a text generation model, and samples of a plurality of tasks respectively having text relevance, wherein the text generation model comprises a plurality of training stages, each training stage corresponding to a different training target, and the samples have a label accuracy level;

[0137] For each training stage, a sample set of the training stage is obtained according to the samples of the plurality of tasks respectively having the label accuracy level matching the training target of the training stage, wherein the sample sets of different training stages are different;

[0138] For a first training stage of the plurality of training stages, the text generation model is trained using the sample set of the first training stage to obtain an updated text generation model;

[0139] For any training stage after the first training stage, the text generation model obtained by the previous training stage of the any training stage is trained using the sample set of the any training stage to obtain an updated text generation model.

[0140] Or, when the computer program is executed in the computer, the computer is caused to perform the steps of the above method for generating text.

[0141] For example, it includes:

[0142] Obtaining structured data of one or more tasks respectively;

[0143] Parsing the structured data of the one or more tasks respectively to obtain a plurality of nodes and relationships between the nodes of the one or more tasks respectively;

[0144] Converting the plurality of nodes and relationships between the nodes of the one or more tasks respectively to obtain a graph of the one or more tasks respectively, and obtaining a vector representation of each node;

[0145] Inputting the graph of the one or more tasks respectively and the vector representation of the node into the text generation model trained by the training method of the text generation model as described in any embodiment of the present specification to obtain text output by the text generation model.

[0146] The above is a schematic scheme of a computer program of the present embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above training method of the text generation model, or the technical scheme of the above method for generating text belong to the same concept. The technical scheme of the computer program which is not described in detail can be referred to the description of the technical scheme of the training method of the text generation model, or the technical scheme of the above method for generating text.

[0147] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0148] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0149] It should be noted that for the foregoing method embodiments, the acts described therein can be performed in a different order from the order described, and that certain acts can be performed in parallel or concurrently. In addition, certain acts can be omitted. Furthermore, the described acts and the order described therein are not exclusive and additional or fewer acts can be performed by, or in conjunction with, the described embodiments, depending on the implementation. It should also be noted that the described embodiments are not limited to the specific implementations described herein but include any alternatives, modifications, equivalents, and improvements conforming to the principles of the described embodiments.

[0150] In the above-described embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0151] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for a text generation model, comprising: Obtain a text generation model and samples for multiple tasks with text relevance. The text generation model includes multiple training stages, each corresponding to a different training objective. The samples have labeled accuracy levels. The input data for the multiple tasks with text relevance come from different sources, but the input data for any two tasks are text-related. For each training phase, a sample set for that training phase is obtained based on the samples from the samples of each of the multiple tasks that match the training objective of that training phase with the labeled accuracy level. The sample sets for different training phases are different. For the first training stage among the multiple training stages, the text generation model is trained using the sample set of the first training stage to obtain an updated text generation model. For any training stage after the first training stage, the text generation model obtained in the previous training stage is trained using the sample set of that training stage to obtain an updated text generation model.

2. The method according to claim 1, wherein the plurality of tasks are tasks for generating text from structured data of different structures; The text generation model is a language model that incorporates a structured adapter, wherein... The structured adapter is used to extract features of the graph's structure; The method further includes: Obtain the structured data for each of the multiple tasks; The structured data of each of the multiple tasks is parsed to obtain several nodes of each task and the relationships between the nodes; The graphs of the multiple tasks are obtained by transforming the nodes of each task and the relationships between the nodes, and the vector representations of the nodes are obtained. By using the graph and the vector representation of the nodes, samples for each of the multiple tasks are obtained.

3. The method according to claim 2, wherein the language model for introducing the structured adapter comprises: The transformer encoding module and the transformer decoding module are provided, wherein the structured adapter is located after the feedforward network sublayer of the transformer encoding module.

4. The method according to claim 1, wherein the plurality of tasks includes a first task and a second task; in, The samples for the first task include samples of a first annotation accuracy level and samples of a second annotation accuracy level. The samples for the second task include samples of a first annotation accuracy level and samples of a second annotation accuracy level. The annotation accuracy of the first annotation accuracy level is greater than the annotation accuracy of the second annotation accuracy level. For each training phase, the sample set for that training phase is obtained by selecting samples from the samples of each of the multiple tasks that match the labeled accuracy level of the training objective for that training phase. This includes: For the first training phase, since the training objective of this training phase is full training, the sample set for this training phase is obtained by using the samples of the first and second annotation accuracy levels of the first task, and the samples of the first and second annotation accuracy levels of the second task. For the second training phase, since the training objective of this phase is to improve the accuracy of the first task, the sample set for this training phase is obtained by using the samples of the first annotation accuracy level of the first task and the samples of the first annotation accuracy level and the second annotation accuracy level of the second task. For the third training phase, since the training objective of this phase is to improve the accuracy of the second task, the sample set for this training phase is obtained by using the samples of the first labeled accuracy level of the second task.

5. The method according to claim 4, wherein the first task is a task of generating text from SQL data, and the second task is a task of generating text from SQL data and tabular data.

6. The method according to claim 4 or 5, wherein the plurality of tasks further comprises: A third task that is text-related to the first and second tasks; The sample set of the first training phase also includes samples of the third task, which are samples of the third annotation accuracy level, and the annotation accuracy of the third annotation accuracy level is less than that of the second annotation accuracy level.

7. The method according to claim 6, wherein the third task is the task of generating text from tabular data.

8. A method for generating text, comprising: Obtain structured data for one or more tasks; The structured data of each of the one or more tasks is parsed to obtain several nodes of each of the one or more tasks and the relationships between the nodes; By utilizing the nodes of each of the one or more tasks and the relationships between the nodes, the graphs of the one or more tasks are obtained, and the vector representations of the nodes are obtained. The graphs and node vector representations of the one or more tasks are input into the text generation model trained by the method described in claim 1 to obtain the text output by the text generation model.

9. An apparatus for generating text, comprising: The data acquisition module is configured to acquire structured data for one or more tasks. The structure parsing module is configured to parse the structured data of the one or more tasks respectively, and obtain a number of nodes of the one or more tasks and the relationships between the nodes. The structure transformation module is configured to transform the graph of each of the one or more tasks by utilizing several nodes of each task and the relationships between the nodes, and to obtain the vector representation of each node. The text output module is configured to input the graphs and node vector representations of the one or more tasks into the text generation model trained by the method described in claim 1, and obtain the text output by the text generation model.

10. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the training method of the text generation model according to any one of claims 1 to 7, or implement the steps of the text generation method according to claim 8.

11. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the training method of the text generation model according to any one of claims 1 to 7, or the steps of the text generation method according to claim 8.

Images