Performing multiple tasks with continuous adaptation

Abstract

Methods and apparatuses for performing multiple tasks are presented. A text input can be obtained. A set of shared representations of the text input in multiple layers can be generated. Multiple task-specific representations of the text input can be generated based on the set of shared representations. The multiple tasks can be respectively performed with the multiple task-specific representations.

Description

BACKGROUND

[0001] Natural Language Processing (NLP) is a technology for communicating with a computer using a natural language, which aims to enable a computer to understand and use a natural language to realize communication between a human and a machine, so as to replace a human to perform various tasks related to the natural language, such as a Query Understanding task, a Machine Reading Comprehension task, a Question Answering task, etc. An NLP task can be performed through a neural network model. For example, various NLP tasks can be performed through a Bidirectional Encoder Representations from Transformers (BERT) model, a Generative Pre-trained Transformer (GPT) model, a Robustly optimized BERT approach (RoBERTa) model, etc. from a transformer. These models are usually complex models that rely on deep networks with a huge number of parameters, and thus have excellent performance in performing NLP tasks. SUMMARY

[0002] This summary is provided to introduce a set of concepts that are further described below in the DETAILED DESCRIPTION. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure propose methods and apparatuses for performing multiple tasks. A text input can be obtained. A set of shared representations of the text input in multiple layers can be generated. Multiple task-specific representations of the text input can be generated based on the set of shared representations. The multiple task-specific representations can be utilized to perform the multiple tasks, respectively.

[0004] It should be noted that the above one or more aspects include the features recited in the following detailed description and the claims. The following detailed description and the accompanying drawings are detailed in the aspects of some illustrative features. These features only indicate a variety of ways in which the principles of the various aspects can be implemented, and the disclosure is intended to include all aspects and equivalent transformations thereof. BRIEF DESCRIPTION OF DRAWINGS

[0005] The disclosed aspects will be described below with reference to the accompanying drawings. These drawings are provided to illustrate and not to limit the disclosed aspects.

[0006] Figure 1 An exemplary process for performing multiple tasks with continuous adaptation is shown in accordance with an embodiment of the present disclosure.

[0007] Figure 2 An exemplary process for generating task-specific representations is shown in accordance with an embodiment of the present disclosure.

[0008] Figure 3 An exemplary process for training a multi-task model with multiple single-task reference models is shown in accordance with an embodiment of the present disclosure.

[0009] Figure 4 An exemplary process for training a multi-task model with a multi-task reference model is shown in accordance with an embodiment of the present disclosure.

[0010] Figure 5 is a flowchart of an exemplary method for performing multiple tasks in accordance with an embodiment of the present disclosure.

[0011] Figure 6 An exemplary apparatus for performing multiple tasks is shown in accordance with an embodiment of the present disclosure.

[0012] Figure 7 An exemplary apparatus for performing multiple tasks is shown in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The present disclosure will now be discussed with respect to several exemplary embodiments. It is to be understood that the discussion of these embodiments is merely meant to provide a better understanding of the present disclosure and is not meant to teach any limitation of the scope of the present disclosure.

[0014] It is desirable to use a neural network model such as a BERT model, a GPT model, a RoBERTa model, etc. to perform multiple tasks. There are some existing ways to use a neural network model to perform multiple tasks. In one way, taking the BERT model as an example, given multiple tasks, the BERT model can be trained separately for each task, thereby obtaining multiple BERT models for the multiple tasks. This way requires hosting multiple BERT models, which requires occupying a large amount of storage resources and computing resources. In addition, when there are additional tasks, the BERT model needs to be retrained, which can affect the performance of the model in performing existing tasks. In another way, a multi-task model can be constructed by adding multiple task-specific output layers for a fixed number of tasks on top of a shared feature extractor. In this paper, a model capable of performing multiple tasks at the same time can be referred to as a multi-task model. The multi-task model can be used to perform the fixed number of tasks. However, when the multi-task model is used to perform additional tasks, all parameters in the multi-task model need to be updated for the additional tasks, which will affect its performance in performing existing tasks.

[0015] Embodiments of the present disclosure propose an improved method for performing multiple tasks. The multiple tasks can be based on the same text input. A set of representations of the text input in multiple layers can be generated. The set of representations can be referred to as a set of shared representations, which can be used to generate multiple representations of the text input for multiple tasks. A representation for a particular task can be referred to as a task-specific representation. Multiple task-specific representations for multiple tasks can be further used to perform the multiple tasks, respectively.

[0016] In one aspect, embodiments of the present disclosure propose a multi-task model with novel structure for performing multiple tasks. The multi-task model can include, for example, a shared encoder, multiple task-specific encoders for multiple tasks, and multiple task-specific linear layers, etc. The shared encoder can include a set of shared encoder layers, which can generate a set of shared representations of a text input in multiple layers. During training of the multi-task model, or during deployment of the multi-task model for performing multiple tasks, the parameters of the shared encoder can be fixed. The task-specific encoders, which can also be referred to as adapters, can be adapted for a target task. In this context, the task for which a task-specific encoder is adapted can be referred to as the target task of the task-specific encoder. The task-specific encoder can capture task-specific semantics of its target task from the set of shared representations provided by the shared encoder, and generate a task-specific representation of the target task. The task-specific linear layers can utilize the task-specific representation provided by the task-specific encoder to perform the target task corresponding to the task-specific representation. The task-specific encoders can be connected to the shared encoder like plug-ins. This way of connection does not affect the parameters of the shared encoder. Moreover, when there is an additional task, the multi-task model can be made to perform the additional task by adding an additional task-specific encoder and an additional task-specific linear layer for the additional task. This way does not affect the structures and parameters in the multi-task model related to performing existing tasks, e.g., does not change the parameters of the shared encoder and existing task-specific encoders and task-specific linear layers, and thus does not affect the performance of the multi-task model in performing existing tasks. Therefore, the multi-task model according to embodiments of the present disclosure can perform multiple tasks with continuous adaptation.

[0017] In another aspect, embodiments of the present disclosure propose to train a multi-task model through a teacher-student architecture. The multi-task model can be trained with multiple sets of training data for multiple tasks. In this document, a set of data used to train the multi-task model can be referred to as a set of supervised data. Embodiments of the present disclosure propose to obtain multiple sets of training data for training the multi-task model through a teacher-student architecture. For example, one or more reference models can be trained with a small amount of sets of supervised data, and then the multiple sets of training data can be generated through the trained one or more reference models. In this document, a set of supervised data can refer to a set of data used to train a reference model, and a reference model refers to a model that can be used to assist in training a multi-task model, which can also be referred to as a teacher model. In this mode, the multi-task model learns knowledge from the teacher model, which can also be referred to as a student model. In one implementation, the multiple sets of training data can be generated by multiple single-task reference models, and the multiple single-task reference models can be pre-trained with multiple sets of supervised data, respectively. In this document, a single-task reference model can refer to a reference model that can perform a single task. In another implementation, the multiple sets of training data can be generated by a multi-task reference model, and the multi-task reference model can be pre-trained with multiple sets of supervised data. In this document, a multi-task reference model can refer to a reference model that can perform multiple tasks simultaneously.

[0018] In yet another aspect, the multi-task model according to embodiments of the present disclosure can be used in a multilingual or cross-lingual scenario. A neural network model with multilingual or cross-lingual capability can be employed as the shared encoder. The task-specific encoder can be adapted to generate a task-specific representation for a text input in any language, which can be, for example, any of the multiple languages supported by the shared encoder. For example, the task-specific encoder can be trained for a target task based on a text input in any language. The trained task-specific encoder can be capable of generating a task-specific representation for a text input in any language for the target task. Preferably, the task-specific encoder can be specialized to generate a task-specific representation for a text input in a target language. In this document, the language for which the task-specific encoder is specialized can be referred to as the target language of the task-specific encoder. For example, the task-specific encoder can be trained for a target task based on a text input in a target language. The trained task-specific encoder can be capable of generating a more accurate task-specific representation for a text input in the target language for the target task. In this way, task-specific representations for text inputs in different languages for the same target task can be generated independently of each other through different task-specific encoders. Further, a multi-task model comprising such task-specific encoders can be flexible to perform the same task based on different languages.

[0019] Figure 1 An exemplary process 100 for performing multiple tasks with continuous adaptation is shown according to embodiments of the present disclosure. The process 100 can be performed by a multi-task model 110. A text input 102 can be obtained. The text input 102 can be used to perform multiple tasks, such as task 1 to task M, where M ≥ 1 is the number of tasks. Multiple task results corresponding to the multiple tasks can be obtained for the text input 102 by the multi-task model 110, such as task results 152-1 to 152-M.

[0020] The text input 102 can be denoted as x = (x1, …, xn), where n is the number of words included in the text input x, and xi is the ith word in the text input x. An embedding layer 120 can generate an initial representation of the text input x. i n i

[0021] The shared encoder 130 can be a known neural network model, such as a neural network model based on a structure of a Fully Connected Layer, a neural network model based on a structure of a Transformer Layer, and the like. Taking the neural network model based on the structure of the Transformer Layer as an example, it can include, for example, a BERT model, a GPT model, a RoBERTa model, and the like. During training of the multi-task model 110, or during deployment of the multi-task model 110 for performing the multiple tasks, the parameters of the shared encoder 130 can be fixed. For example, the parameters of the shared encoder 130 can be fixed during performance of the multiple tasks, which can generate a set of shared representations of the text input x in multiple layers based on the initial representation of the text input x provided by the embedding layer 120. The shared encoder 130 can include a set of shared encoder layers, such as shared encoder layers 130-1 to 130-L, where L ≥ 1 is the number of shared encoder layers. A set of shared representations of the text input x in L layers, such as shared representations 132-1 to 132-L, can be generated by the shared encoder layers 130-1 to 130-L. Taking the shared encoder 130 as the BERT model as an example, it can include a set of Transformer layers. The set of Transformer layers can generate a set of context-shared representations where D is the dimension of the shared representation. The shared representation 132-1 of the text input x output by the shared encoder layer 130-1 in the shared encoder 130 can be denoted as hi(l) = f (xl, xl+1, …, xi, xi+1, …, xn) (l ∈ [1, L]), where hi(l) represents the shared representation of the ith word xi in the text input x output by the shared encoder layer 130-l. l i，l i

[0022] ​​​​​​The shared representations 132-1 to 132-L can be provided to multiple task-specific encoders, such as task-specific encoders 140-1 to 140-M. The task-specific encoders 140-1 to 140-M can be for multiple tasks, such as task 1 to task M, respectively. Each task-specific encoder 140-m (m e [1, M]) can capture task-specific semantics of its target task from the shared representations 132-1 to 132-L provided by the shared encoder 130, and generate a task-specific representation of the target task. The task-specific encoders 140-1 to 140-M can generate multiple task-specific representations, such as task-specific representations 142-1 to 142-M, of the text input x based on the shared representations 132-1 to 132-L, respectively. Each task-specific encoder 140-m can be connected to the shared encoder 130 like a plug-in. This connection manner does not affect the parameters of the shared encoder 130. Exemplary structures of the task-specific encoders and exemplary procedures of generating the task-specific representations will be illustrated later in connection with Figure 2

[0023] The task-specific representations 142-1 to 142-M can be used to perform multiple tasks, respectively. For example, the task-specific representations 142-1 to 142-M can be used to perform multiple tasks by task-specific linear layers 150-1 to 150-M, respectively. The multiple task-specific linear layers 150-1 to 150-M can output multiple task results, such as task results 152-1 to 152-M, for the multiple tasks, respectively. Taking an example where task m is a domain classification task in a query understanding task, the task result 152-m for this task m can be a binary classification result indicating whether the text input 102 belongs to a specific domain.

[0024] According to an embodiment of the present disclosure, in a case where multiple tasks are performed based on a same text input, only a set of shared representations, such as the shared representations 132-1 to 132-L in Figure 1 , of the text input can be generated. The set of shared representations can be provided to multiple task-specific encoders for the multiple tasks, and in turn used to generate multiple task-specific representations to perform the multiple tasks. That is, the shared encoder 130 only needs to be executed once, which significantly saves the computing resources and improves the efficiency of the model. However, it should be understood that the multi-task model 110 can also be used to perform multiple tasks based on different text inputs. A text input for a current task can be provided to the multi-task model. The multi-task model can perform the current task at least by a task-specific encoder and a task-specific linear layer for the current task.

[0025] ​The multi-task model 110 according to the embodiments of the present disclosure can efficiently and flexibly support various NLP tasks. For example, the multi-task model 110 with the task-specific encoders 140-1 to 140-M and the corresponding task-specific linear layers 150-1 to 150-M can perform the tasks 1 to M. There is a demand to use the multi-task model 110 to perform an additional task, e.g., an additional task M+1. According to the embodiments of the present disclosure, the multi-task model 110 can be made to perform the additional task by adding an additional task-specific encoder (not shown) and an additional task-specific linear layer (not shown) for the additional task to the multi-task model 110.

[0026] It is assumed that the additional task M+1 can be performed based on the text input 102. In an implementation, an additional task-specific representation of the text input 102 for the additional task M+1 can be generated based on the shared representations 132-1 to 132-L of the text input 102, and the additional task M+1 can be performed using the additional task-specific representation. For example, the additional task-specific representation for the additional task M+1 can be generated based on the shared representations 132-1 to 132-L by an additional task-specific encoder. The additional task M+1 can be performed using the additional task-specific representation by an additional task-specific linear layer.

[0027] Adding the additional task-specific encoder and the additional task-specific linear layer for the additional task to the multi-task model 110 can make the multi-task model flexibly support various NLP tasks. Moreover, since the task-specific encoders can be connected to the shared encoder like plug-ins, when the task-specific encoders and the task-specific linear layers are added, the structures and parameters in the multi-task model related to performing the existing tasks will not be affected, so that the performance of the multi-task model in performing the existing tasks will not be affected. For example, the parameters of the shared encoder 130, the task-specific encoders 140-1 to 140-M, and the task-specific linear layers 150-1 to 150-M will not change due to the addition of the additional task-specific encoder and the additional task-specific linear layer. Therefore, the multi-task model 110 according to the embodiments of the present disclosure can perform multiple tasks using continuous adaptation. In addition, one or more task-specific encoders and the corresponding task-specific linear layers can be flexibly removed from the multi-task model 110 as needed.

[0028] It should be understood that, Figure 1 The process 100 in is merely an example of a process for performing multiple tasks using continuous adaptation. The process for performing multiple tasks can include any other steps, and can include more or fewer steps, according to actual application requirements. In addition, the process 100 in can be performed by one or more of the devices 200-1 to 200-N. Figure 1The multi-task model 110 in FIG. 1 is merely an example of a multi-task model. According to actual application requirements, a multi-task model can have any other structure, and can include more or fewer layers. Moreover, it should be understood that although the foregoing discussion and the following discussion can involve an example in which a neural network model based on a transformer layer structure is employed as the shared encoder, embodiments of the present disclosure are not limited thereto, but can employ a neural network model based on other structures, e.g., based on a fully connected layer structure, as the shared encoder in a similar manner.

[0029] According to embodiments of the present disclosure, a multi-task model can be used in a multi-lingual or cross-lingual scenario. A neural network model with multi-lingual or cross-lingual capability can be employed as the shared encoder. The neural network model with multi-lingual or cross-lingual capability can be, for example, a Cross-lingual Language Model (XLM). The task-specific encoder can be adapted to generate a task-specific representation of a text input in any language, which can be, for example, any of a plurality of languages supported by the shared encoder. For example, the task-specific encoder can be trained for a target task based on a text input in any language. The trained task-specific encoder is capable of generating a task-specific representation of a text input in any language for the target task. Preferably, the task-specific encoder can be specifically used to generate a task-specific representation of a text input in a target language. For example, the task-specific encoder can be trained for a target task based on a text input in the target language. For example, the task-specific encoder can be trained with a training data set in the target language for the target task. The trained task-specific encoder is capable of generating a more accurate task-specific representation of a text input in the target language for the target task. In this way, task-specific representations of text inputs in different languages for the same target task can be generated independently of each other by different task-specific encoders. For example, there is a text input in English for a classification task and a text input in French for the classification task. Task-specific representations of the text input in English for the classification task and the text input in French for the classification task can be generated by two task-specific encoders, respectively. Further, a multi-task model including such task-specific encoders can flexibly perform the same task based on different languages.

[0030] Figure 2 An exemplary process 200 for generating a task-specific representation according to embodiments of the present disclosure is shown. The process 200 can be performed by a task-specific encoder 210. The task-specific encoder 210 can correspond to the task-specific encoder 110 in FIG. 1. Figure 1Any one of the task-specific encoders 140-1 to 140-M in the shared representations 132-1 to 132-L in the shared encoder 130. The task-specific encoder 210 can be, for example, based on a transformer structure, which can include, for example, a set of task-specific feature extraction units 220-1 to 220-L, a set of scaled self-attention units 230-1 to 230-L, a concatenation unit 240, a Layer Normalization 250, a Feed-Forward Layer 260, a Layer Normalization 270, a concatenation unit 280, and so on.

[0031] The shared representations 202-1 to 202-L can correspond to the shared representations 132-1 to 132-L in the shared encoder 130. Figure 1 The task-specific encoder 210 can capture the task-specific semantics of its target task from the shared representations 202-1 to 202-L and generate a task-specific representation of the target task. For example, the task-specific encoder 210 can first extract a task-specific feature set for the target task from each of the shared representations 202-1 to 202-L and encode the task-specific feature set into a task-specific sub-representation. With the shared representation h l 202-l, the task-specific feature extraction unit 220-l can extract a task-specific feature set for the target task from the shared representation h l For example, the task-specific feature extraction unit 220-l can extract a task-specific feature set for the target task from the shared representation h l by applying a linear transformation (W l k , W l q , W l v ) to the shared representation h l , where W l k , W l q and are trainable model parameters, and d is the dimension of the word embeddings inside the task-specific encoder 210. The dimension d can be much smaller than the dimension D of the shared representation h l The extracted task-specific feature set can be, for example, a triple (k l , q l , v l ), where k l is the key, q l is the query, and v l is the value.

[0032] Subsequently, the extracted task-specific feature set (k l , ql , v l ) into task-specific sub-representations. For example, the scaling self-attention unit 230-1 can perform a scaling attention operation on the task-specific feature set (k l , q l , v l ) to obtain task-specific sub-representations layer l , as shown in the following equation:

[0033] layer l = SA(k l , q l , v l ) (1)

[0034] A set of task-specific sub-representations corresponding to the shared representations 202-1 to 202-L, e.g., task-specific sub-representations layer1 to layer L , can be combined into a task-specific intermediate representation. For example, the task-specific sub-representations layer1 to layer L may be concatenated into a task-specific intermediate representation attention by the concatenation unit 240, as shown in the following equation:

[0035] attention = concat(layer1, …, layer L ) (2)

[0036] Next, a task-specific representation can be generated based at least on the task-specific intermediate representation attention. For example, the task-specific intermediate representation attention can be normalized by the normalization layer with residual connection 250 to obtain an output att_output, as shown in the following equation:

[0037] att_output = LN(v + W * attention) (3)

[0038] wherein and W is a trainable model parameter of dimension (L x d, L x d).

[0039] Preferably, the forward feedback layer 260 and another normalization layer with residual connection 270 can further process the output att_output to obtain an output ada_output. For the i-th word x i in the text input x, the word embedding in the output ada_output corresponding to the i-th word x i in the text input x can have a dimension of L x d.

[0040] The top-level shared representation, i.e., shared representation 202-L, can be concatenated with the output ada_output through the cascading unit 280 to form a task-specific representation 282.

[0041] It should be understood that Figure 2 Process 200 in the document is merely an example of a process for generating a task-specific representation. Depending on the specific application requirements, the process for generating the task-specific representation may include any other steps, and may include more or fewer steps. Furthermore, Figure 2 The task-specific encoder 210 described herein is merely one example of a task-specific encoder. Depending on the specific application requirements, the task-specific encoder can have any other structure and can include more or fewer layers. For example, the feedforward layer 260 and the normalization layer 270 can be removed from the task-specific encoder 210, thereby obtaining the task-specific representation 282 by directly concatenating the output att_output provided by the normalization layer 250 with the shared representation 202-L. Furthermore, it should be understood that although the foregoing and following discussion may involve using a neural network model based on a transformer layer structure as an example of a task-specific encoder, the embodiments of this disclosure are not limited thereto, but can employ neural network models based on other structures, such as fully connected layer structures, as task-specific encoders in a similar manner.

[0042] As previously described, a multi-tasking model according to embodiments of this disclosure can be used, for example... Figure 1 The multi-task model 110 in the example executes multiple tasks. The multi-task model can be trained for these multiple tasks. When the trained multi-task model is deployed, it can generate multiple task-specific representations for these multiple tasks and use these generated task-specific representations to execute the multiple tasks respectively. The multi-task model may include, for example, a shared encoder, multiple task-specific encoders for multiple tasks, and multiple task-specific linear layers. The multi-task model can be trained by pre-training the shared encoder and optimizing the parameter sets of the multiple task-specific encoders and / or the parameter sets of the multiple linear layers of the multiple task-specific linear layers while keeping the parameters of the pre-trained shared encoder fixed.

[0043] Shared encoders can be pre-trained using known methods. Taking the BERT model as an example, the shared encoder can be pre-trained using methods such as masked language model (MLM) and next sentence prediction (NSP).

[0044] After the shared encoder is pre-trained, multiple sets of task-specific encoder parameters and / or multiple sets of linear layer parameters can be optimized while fixing the parameters of the pre-trained shared encoder. Each set of task-specific encoder parameters and each set of linear layer parameters is much smaller in number compared to the parameters in the shared encoder, and thus the optimization of the sets of task-specific encoder parameters and / or linear layer parameters does not consume excessive computational resources and storage resources.

[0045] The multiple sets of task-specific encoder parameters and / or the multiple sets of linear layer parameters can be optimized in a distributed manner. For example, the multiple sets of task-specific encoder parameters can be optimized independently of each other. The optimization of each set of task-specific encoder parameters does not affect the other sets of task-specific encoder parameters. Similarly, the multiple sets of linear layer parameters can also be optimized independently of each other. The optimization of each set of linear layer parameters also does not affect the other sets of linear layer parameters. In addition, optimizing the various sets of parameters independently of each other can also ensure that the training for additional tasks does not affect the sets of parameters for existing tasks in the multi-task model, and thus can ensure the performance of the multi-task model in performing the existing tasks.

[0046] The multiple sets of task-specific encoder parameters and / or the multiple sets of linear layer parameters can be optimized respectively using multiple sets of training data for the multiple tasks. The optimization can be based on a standard supervised loss function. The multiple sets of training data used to optimize the multiple sets of task-specific encoder parameters and / or the multiple sets of linear layer parameters can be obtained in various ways. For example, the sets of training data can be sets of supervised data, such as sets of human-labeled data.

[0047] In addition, embodiments of the present disclosure propose obtaining the multiple sets of training data used to optimize the multiple sets of task-specific encoder parameters and / or the multiple sets of linear layer parameters based on a teacher-student architecture. For example, one or more reference models can be trained using a small amount of sets of supervised data, and then the multiple sets of training data can be generated by the trained one or more reference models. In one implementation, the multiple sets of training data can be generated by multiple single-task reference models, and the multiple single-task reference models can be pre-trained respectively using multiple sets of supervised data. An example process of training a multi-task model by the multiple single-task reference models will be described later in connection with Figure 3 In another implementation, the multiple sets of training data can be generated by a multi-task reference model, and the multi-task reference model can be pre-trained using multiple sets of supervised data. An example process of training a multi-task model by the multi-task reference model will be described later in connection with Figure 4

[0048] Figure 3 ​An exemplary process 300 for training a multi-task model by multiple single-task reference models is shown according to an embodiment of the present disclosure. In the process 300, a multi-task model 320 can be trained by multiple single-task reference models, e.g., single-task reference models 310-1 to 310-M. The single-task reference models can be models with higher complexity than the multi-task model 320. For example, when the shared encoder adopted by the multi-task model 320 is a 12-layer BERT model, the single-task reference models can be 24-layer BERT models. The multi-task model 320 can correspond to Figure 1 the multi-task model 110 in FIG. 1.

[0049] The single-task reference models 310-1 to 310-M can be trained respectively in advance by multiple supervised data sets, e.g., supervised data sets 302-1 to 302-M, to obtain trained single-task reference models 312-1 to 312-M. The trained single-task reference models 312-1 to 312-M can generate training data sets 316-1 to 316-M based on non-supervised data sets 314-1 to 314-M respectively. For example, the non-supervised data set 314-m for task m can be provided to the trained single-task reference model 312-m. The non-supervised data set 314-m can include, for example, a set of unannotated texts. For each text in the set of unannotated texts, the trained single-task reference model 312-m can generate a soft or pseudo annotation for the text. In this paper, the soft or pseudo annotation can refer to the annotation generated by the reference model. The text and its corresponding soft or pseudo annotation can be combined into a training sample. Then, the set of training samples corresponding to the non-supervised data set 314-m can be combined into the training data set 316-m.

[0050] The training data sets 316-1 to 316-M can be provided to the multi-task model 320. The multi-task model 320 can include, for example, an embedding layer 330, a shared encoder 340, task-specific encoders 350-1 to 350-M, task-specific linear layers 360-1 to 360-M, etc. These modules can respectively correspond to Figure 1 the embedding layer 120, the shared encoder 130, the task-specific encoders 140-1 to 140-M, the task-specific linear layers 150-1 to 150-M in FIG. 1. The training data sets 316-1 to 316-M can be used to optimize multiple sets of task-specific encoder parameters corresponding to the task-specific encoders 340-1 to 340-M and / or multiple sets of linear layer parameters corresponding to the task-specific linear layers 350-1 to 350-M respectively with at least the parameters of the shared encoder 340 fixed, so as to realize the training of the multi-task model 320.

[0051] It should be appreciated that Figure 3 The process 300 in FIG. 3 is merely an example of a process for training a multi-task model from a plurality of single-task reference models. Depending on actual application requirements, the process for training a multi-task model from a plurality of single-task reference models can include any other steps, and can include more or fewer steps. For example, in addition to the training data sets 316-1 to 316-M, the multi-task model 320 can be trained with the supervised data sets 302-1 to 302-M used to train the single-task reference models 310-1 to 310-M.

[0052] Figure 4 An example process 400 for training a multi-task model from a multi-task reference model is shown, according to an embodiment of the present disclosure. In the process 400, a multi-task model 420 can be trained from a multi-task reference model 410. The multi-task reference model 410 can be a known multi-task model, such as a Multi-Task Deep Neural Network (MT-DNN) model. The multi-task model 420 can correspond to the multi-task model 110 in FIG. 1. Figure 1

[0053] The multi-task reference model 410 can be trained from a plurality of supervised data sets, such as the supervised data sets 402-1 to 402-M, respectively, to obtain a trained multi-task reference model 412. The trained multi-task reference model 412 can generate training data sets 416-1 to 416-M based on a non-supervised data set 414-1 to 414-M, respectively. For example, the non-supervised data set 414-m for task m can be provided to the trained multi-task reference model 412. The non-supervised data set 414-m can include, for example, a set of unlabeled texts. For each text in the set of unlabeled texts, the trained multi-task reference model 412 can generate a soft label or pseudo label for the text. The text and its corresponding soft label or pseudo label can be combined into a training sample. Then, the set of training samples corresponding to the non-supervised data set 414-m can be combined into the training data set 416-m.

[0054] The training data sets 416-1 to 416-M can be provided to the multi-task model 420. The multi-task model 420 can include, for example, an embedding layer 430, a shared encoder 440, task-specific encoders 450-1 to 450-M, task-specific linear layers 460-1 to 460-M, etc. These modules can correspond to the embedding layer 130, the shared encoder 140, the task-specific encoders 150-1 to 150-M, the task-specific linear layers 160-1 to 160-M, etc. in FIG. 1, respectively. Figure 1 ​the embedding layer 120, the shared encoder 130, the task-specific encoders 140-1 to 140-M, and the task-specific linear layers 150-1 to 150-M in the multi-task reference model 410. The training of the multi-task model 420 can be implemented by optimizing a plurality of sets of task-specific encoder parameters corresponding to the task-specific encoders 440-1 to 440-M and / or a plurality of sets of linear layer parameters corresponding to the task-specific linear layers 450-1 to 450-M, respectively, with the training data sets 416-1 to 416-M, while at least fixing the parameters of the shared encoder 440.

[0055] It should be appreciated that, Figure 4 The process 400 in FIG. 4 is merely an example of a process for training a multi-task model by a multi-task reference model. Depending on actual application requirements, the process for training a multi-task model by a multi-task reference model can include any other steps, and can include more or fewer steps. For example, in addition to the training data sets 416-1 to 416-M, the multi-task model 420 can also be trained with the supervised data sets 402-1 to 402-M used to train the multi-task reference model 410.

[0056] Figure 5 FIG. 5 is a flowchart of an example method 500 for performing a plurality of tasks according to embodiments of the present disclosure.

[0057] At 510, a text input can be obtained.

[0058] At 520, a set of shared representations of the text input in a plurality of layers can be generated.

[0059] At 530, a plurality of task-specific representations of the text input can be generated based on the set of shared representations.

[0060] At 540, the plurality of tasks can be performed respectively with the plurality of task-specific representations.

[0061] In an implementation, the generating a set of shared representations can include generating the set of shared representations by a set of shared encoder layers in a shared encoder.

[0062] The parameters of the shared encoder can be fixed during the performance of the plurality of tasks.

[0063] In an implementation, the generating a plurality of task-specific representations can include generating the plurality of task-specific representations based on the set of shared representations by a plurality of task-specific encoders, respectively.

[0064] Each of the plurality of task-specific encoders can generate a task-specific representation for a target task by: extracting a task-specific feature set for the target task from each of the set of shared representations, and encoding the task-specific feature set into a task-specific sub-representation; combining a set of task-specific sub-representations corresponding to the set of shared representations into a task-specific intermediate representation; and generating the task-specific representation based at least on the task-specific intermediate representation.

[0065] Each of the plurality of task-specific encoders can be adapted to generate a task-specific representation of a text input in an arbitrary language or a task-specific representation of a text input in a target language.

[0066] In an implementation, the performing the plurality of tasks can include performing the plurality of tasks respectively using the plurality of task-specific representations through a plurality of task-specific linear layers.

[0067] In an implementation, the method 500 can be implemented through a multi-task model. The multi-task model can include at least a shared encoder and a plurality of task-specific encoders and a plurality of task-specific linear layers for the plurality of tasks.

[0068] The training of the multi-task model can include pre-training the shared encoder, and optimizing a plurality of sets of task-specific encoder parameters of the plurality of task-specific encoders and / or a plurality of sets of linear layer parameters of the plurality of task-specific linear layers with parameters of the pre-trained shared encoder fixed.

[0069] The plurality of sets of task-specific encoder parameters can be optimized independently of each other. The plurality of sets of linear layer parameters can be optimized independently of each other.

[0070] The optimizing the plurality of sets of task-specific encoder parameters and / or the plurality of sets of linear layer parameters can include optimizing the plurality of sets of task-specific encoder parameters and / or the plurality of sets of linear layer parameters respectively using a plurality of sets of training data for the plurality of tasks.

[0071] The plurality of sets of training data can be generated by a plurality of single-task reference models. The plurality of single-task reference models can be pre-trained respectively using a plurality of sets of supervised data.

[0072] The plurality of sets of training data can be generated by a multi-task reference model. The multi-task reference model can be pre-trained using a plurality of sets of supervised data.

[0073] In an embodiment, the method 500 can further include generating an additional task-specific representation for an additional task of the text input based on the set of shared representations, and performing the additional task using the additional task-specific representation.

[0074] The generating the additional task-specific representation can include generating, by an additional task-specific encoder, the additional task-specific representation based on the set of shared representations. The performing the additional task can include performing, by an additional task-specific linear layer, the additional task using the additional task-specific representation.

[0075] It is to be understood that the method 500 can further include any of the steps / processes for performing multiple tasks according to the embodiments of the present disclosure described above.

[0076] Figure 6 An exemplary apparatus 600 for performing multiple tasks according to an embodiment of the present disclosure is shown.

[0077] The apparatus 600 can include a text input obtaining module 610 configured to obtain a text input, a shared representation generating module 620 configured to generate a set of shared representations of the text input in multiple layers, a task-specific representation generating module 630 configured to generate multiple task-specific representations of the text input based on the set of shared representations, and a task performing module 640 configured to perform the multiple tasks using the multiple task-specific representations, respectively. Further, the apparatus 600 can also include any other modules configured for performing multiple tasks according to the embodiments of the present disclosure described above.

[0078] Figure 7 An exemplary apparatus 700 for performing multiple tasks according to an embodiment of the present disclosure is shown.

[0079] The apparatus 700 can include at least one processor 710 and a memory 720 storing computer executable instructions. The computer executable instructions, when executed, can cause the at least one processor 710 to obtain a text input, generate a set of shared representations of the text input in multiple layers, generate multiple task-specific representations of the text input based on the set of shared representations, and perform the multiple tasks using the multiple task-specific representations, respectively.

[0080] In an embodiment, the generating the set of shared representations can include generating the set of shared representations by a set of shared encoder layers of a shared encoder.

[0081] In an embodiment, the generating the multiple task-specific representations can include generating the multiple task-specific representations based on the set of shared representations by multiple task-specific encoders, respectively.

[0082] In an implementation, the computer-executable instructions, when executed, can further cause the at least one processor 710 to: generate, based on the set of shared representations, an additional task-specific representation of the text input for an additional task; and perform the additional task with the additional task-specific representation.

[0083] It is to be understood that the processor 710 can also perform any other steps / processes of the method for performing multiple tasks according to the above-described embodiments of the present disclosure.

[0084] Embodiments of the present disclosure propose a computer program product for performing multiple tasks, comprising a computer program which, when executed by at least one processor, is configured to: obtain a text input; generate a set of shared representations of the text input in multiple layers; generate multiple task-specific representations of the text input based on the set of shared representations; and perform the multiple tasks respectively with the multiple task-specific representations. In addition, the computer program can also be executed for implementing any other steps / processes of the method for performing multiple tasks according to the above-described embodiments of the present disclosure.

[0085] Embodiments of the present disclosure can be embodied in a non-transitory computer readable medium. The non-transitory computer readable medium can include instructions which, when executed, cause one or more processors to perform any of the operations of the method for performing multiple tasks according to the above-described embodiments of the present disclosure.

[0086] It should be appreciated that all operations in the above-described method are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of the operations, but should cover all other equivalent transformations under the same or similar concepts. In addition, unless otherwise specified or clearly understood from the context, the articles "a" and "an" as used in the specification and the appended claims should be interpreted to mean "one" or "one or more".

[0087] It should also be appreciated that all modules in the above-described apparatus can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any of these modules can be further divided functionally into sub-modules or combined together.

[0088] Processors have been described in connection with various apparatus and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software depends upon the particular applications and general design constraints imposed on the system. As an example, the processors given in the disclosure, any portion of the processors, or any combination of the processors can be implemented with a microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in the disclosure. The functions of the processors given in the disclosure, any portion of the processors, or any combination of the processors can be implemented with software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0089] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, execution threads, procedures, functions, etc. The software can reside on a computer-readable medium. The computer-readable medium can include, for example, memory such as a magnetic storage device (e.g., hard disk), optical storage device, memory on processor, programmable logic devices, smart cards, flash devices, random access memories (RAMs), read-only memories (ROMs), programmable ROMs (PROMs), erasable PROMs (EPROMs), electrically erasable PROMs (EEPROMs), registers, or the like. Although memory is shown separate from processor in various aspects given in the disclosure, memory can be internal to the processor, e.g., cache or register.

[0090] The above description is provided as is to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are or can be become known in this art will be encompassed by the scope of the claims.

Claims

1. A method for performing a plurality of tasks, comprising: obtaining a text input; generating a set of shared representations of the text input in a plurality of layers; generating a plurality of task-specific representations of the text input based on the set of shared representations by a plurality of task-specific encoders for the plurality of tasks, respectively; and performing the plurality of tasks by a plurality of task-specific linear layers utilizing the plurality of task-specific representations, respectively, wherein the method further comprises: generating an additional task-specific representation of the text input for an additional task based on the set of shared representations by an additional task-specific encoder; and performing the additional task by an additional task-specific linear layer utilizing the additional task-specific representation. the generating a set of shared representations comprises:

2. The method of claim 1, wherein, generating the set of shared representations by a set of shared encoder layers in a shared encoder. parameters of the shared encoder are fixed during performance of the plurality of tasks.

3. The method of claim 2, wherein, each of the plurality of task-specific encoders generates a task-specific representation for a target task by:

4. The method of claim 1, wherein, extracting a set of task-specific features for the target task from each of the set of shared representations and encoding the set of task-specific features into a task-specific sub-representation; combining a set of task-specific sub-representations corresponding to the set of shared representations into a task-specific intermediate representation; and generating the task-specific representation based on at least the task-specific intermediate representation. each of the plurality of task-specific encoders is adapted to generate a task-specific representation of a text input in an arbitrary language or a task-specific representation of a text input in a target language.

5. The method of claim 1, wherein, the method is implemented by a multi-task model, and the multi-task model comprises at least a shared encoder and a plurality of task-specific encoders and a plurality of task-specific linear layers for the plurality of tasks.

6. The method of claim 1, wherein, training of the multi-task model comprises:

7. The method of claim 6, wherein, pre-training the shared encoder; and optimizing a plurality of sets of task-specific encoder parameters of the plurality of task-specific encoders and / or a plurality of sets of linear layer parameters of the plurality of task-specific linear layers with parameters of the pre-trained shared encoder fixed.

8. The method of claim 7, wherein, the plurality of sets of task-specific encoder parameters are optimized independently of each other, and / or the plurality of sets of linear layer parameters are optimized independently of each other. the optimizing a plurality of sets of task-specific encoder parameters and / or a plurality of sets of linear layer parameters comprises:

9. The method of claim 7, wherein, optimizing the plurality of sets of task-specific encoder parameters and / or the plurality of sets of linear layer parameters by a plurality of sets of training data for the plurality of tasks, respectively. the plurality of sets of training data are generated by a plurality of single-task reference models, and the plurality of single-task reference models are pre-trained by a plurality of sets of supervised data, respectively.

10. The method of claim 9, wherein, the plurality of sets of training data are generated by a multi-task reference model, and the multi-task reference model is pre-trained by a plurality of sets of supervised data.

11. The method of claim 9, wherein, 12. An apparatus for performing a plurality of tasks, comprising: at least one processor; ​ and a memory storing computer-executable instructions that, when executed, cause the at least one processor to: obtain a text input, generate a set of shared representations of the text input in a plurality of layers, generate, by a plurality of task-specific encoders for the plurality of tasks, a plurality of task-specific representations of the text input based on the set of shared representations, respectively, and perform, by a plurality of task-specific linear layers for the plurality of tasks, the plurality of tasks using the plurality of task-specific representations, respectively, wherein the computer-executable instructions, when executed, further cause the at least one processor to: generate, by an additional task-specific encoder, an additional task-specific representation of the text input for an additional task based on the set of shared representations, and perform, by an additional task-specific linear layer, the additional task using the additional task-specific representation.

13. The apparatus of claim 12, wherein, The generating a set of shared representations comprises: generating the set of shared representations by a set of shared encoder layers of a shared encoder.

14. A computer program product for performing a plurality of tasks, comprising a computer program executed by at least one processor for: obtaining a text input; generating a set of shared representations of the text input in a plurality of layers; generating, by a plurality of task-specific encoders for the plurality of tasks, a plurality of task-specific representations of the text input based on the set of shared representations, respectively; and performing, by a plurality of task-specific linear layers for the plurality of tasks, the plurality of tasks using the plurality of task-specific representations, respectively, wherein the computer program is further executed by the at least one processor for: generating, by an additional task-specific encoder, an additional task-specific representation of the text input for an additional task based on the set of shared representations; and performing, by an additional task-specific linear layer, the additional task using the additional task-specific representation. ​

Images