Implicit bridging of machine learning tasks

Abstract

The invention relates to implicit bridging of machine learning tasks. Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning tasks. A method includes receiving (i) a model input and (ii) data identifying a first machine learning task to be performed on the model input to generate a model output of a first type for the model input; augmenting the model input with an identifier of the first machine learning task to generate an augmented model input; and processing the augmented model input using a machine learning model. An exemplary system applying implicit bridging for machine learning tasks as described in the specification trains a machine learning model to perform certain types of machine learning tasks without requiring the use of explicit training data for the certain types of machine learning tasks during training.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201780068195.5, filed on November 2, 2017. Background Art

[0003] Machine translation systems require large amounts of parallel training data to achieve a high level of accuracy. Generally, it is difficult to obtain large amounts of parallel data for languages ​​spoken by a small number of people compared to languages ​​spoken by many people. For example, most text found on the Internet is in English, whereas the amount of text found in languages ​​such as Japanese or Korean is relatively small. This makes it challenging to obtain parallel data for smaller languages.

[0004] Traditional machine translation systems overcome this problem by bridging translations between smaller languages ​​via a third language, i.e. translating a portion of the text in the first language into the third language and then from the third language into the second language. This bridging process suffers from many problems, including the propagation of errors, increased latency, and increased system complexity. Summary of the invention

[0005] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages:

[0006] A system that applies implicit bridging to machine learning tasks as described in this specification trains a machine learning model to perform certain types of machine learning tasks without requiring explicit training data for certain types of machine learning tasks to be used during training. For example, the system can perform zero-shot translation to train a machine learning model to translate Japanese text segments into corresponding Korean text segments even if training data for this type of task is not used during training. Thus, explicit bridging for translation is avoided, thereby avoiding the propagation of errors, reducing latency for machine learning tasks, and reducing system complexity. For example, decoding speed can be twice as fast because explicit bridging through a third language is unnecessary when translating from Japanese to Korean.

[0007] Furthermore, once the machine learning model has been trained, a system that performs zero-shot translation as described in this specification can achieve a high level of accuracy that is comparable to (if not better than) systems that apply explicit bridges for translation.

[0008] A system that performs zero-shot translation as described in this specification can use a single machine learning model to perform machine translation between N languages, rather than having N^2 separate models, thereby reducing the required time and complexity of model parameter tuning, as well as reducing the computing resources consumed by the machine learning model. In addition, the reduction in the number of models can enable more language pairs to be used within a single device, as service machines typically have limited memory. In addition, the reduction in the number of models can greatly simplify the system architecture, thereby improving the production / setup time associated with the system.

[0009] A system that performs zero-shot translation as described in this specification can allow scaling to additional languages. For example, new data can be added to an existing model, possibly with oversampling or undersampling, so that all languages ​​are appropriately represented and used with new pre-tokens in the event that the target language changes. No changes to the architecture of the existing model are required.

[0010] A system that performs zero-shot translation as described in this specification can allow for low-resource language improvements. All parameters of the system are implicitly shared by all language pairs that are modeled. This forces the system to generalize across language boundaries during training. When language pairs with little available data and language pairs with abundant data are mixed into a single system, translation accuracy on low-resource language pairs can be improved.

[0011] Various example embodiments described herein relate to neural networks. A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from a received input according to the current values ​​of a set of corresponding parameters. The neural network can be trained on a machine learning task using training data to determine training values ​​for the layer parameters, and the neural network can be used to perform the machine learning task on the neural network input.

[0012] In general, an innovative aspect of the subject matter described in this specification can be specifically implemented in a method for training a machine learning model on training data, wherein the machine learning model is configured to: receive an augmented model input, the augmented model input comprising a model input having an identifier of a machine learning task to be performed on the model input; and perform the machine learning task on the received augmented model input to generate a corresponding type of model output for the model input, and wherein the method comprises: obtaining training data comprising multiple paired data sets, wherein each of the paired data sets comprises (i) an input data set and (ii) an output data set; and training the machine learning model on the training data to perform multiple machine learning tasks, wherein the multiple machine learning tasks include the machine learning tasks to be performed on the model input.

[0013] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. The system of one or more computers may be configured to perform specific operations or actions by means of software, firmware, hardware, or any combination thereof installed on the system, which software, firmware, hardware, or any combination thereof in operation can cause the system to perform the actions. One or more computer programs may be configured to perform specific operations or actions by means of including instructions, which, when executed by a data processing device, cause the device to perform the actions.

[0014] The foregoing and other embodiments may each optionally include one or more of the following features, alone or in combination: In some implementations, the augmented model input includes a model input having a pre-pended token identifier of a machine learning task.

[0015] In some embodiments, the model inputs include model inputs of a first input type and the model outputs include model outputs of a first output type, and the plurality of paired data sets do not include an input data set of the first input type paired with an output data set of the first output type.

[0016] In some implementations, data sets in the plurality of paired data sets include text segments in different languages.

[0017] In some embodiments, the method further includes generating training data, comprising: generating a vocabulary of fixed size V in each of the different languages; and merging the generated vocabularies to generate a new vocabulary by sequentially selecting the highest-appearing words in each generated vocabulary until the size of the new vocabulary reaches V.

[0018] In some implementations, each paired data set includes input text segments in an input language paired with text segments in a target language different from the input language.

[0019] In some implementations, the plurality of machine learning tasks includes, for each paired dataset, translating an input text segment into a text segment in a target language.

[0020] In some implementations, augmenting the model input includes the model input having a preceding token indicative of at least a target language.

[0021] In general, an innovative aspect of the subject matter described in this specification can be specifically implemented in a method for the following steps: receiving (i) a model input and (ii) data identifying a first machine learning task to be performed on the model input to generate a first type of model output for the model input; augmenting the model input with an identifier of the first machine learning task to generate an augmented model input; and processing the augmented model input using a machine learning model, wherein the machine learning model has been trained on training data to perform multiple machine learning tasks including the first machine learning task, and wherein the machine learning model has been configured through training to: process the augmented model input to generate a first machine learning model output of the first type for the model input.

[0022] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. The system of one or more computers may be configured to perform specific operations or actions by means of software, firmware, hardware, or any combination thereof installed on the system, which software, firmware, hardware, or any combination thereof in operation can cause the system to perform the actions. One or more computer programs may be configured to perform specific operations or actions by means of including instructions, which, when executed by a data processing device, cause the device to perform the actions.

[0023] The foregoing and other embodiments may each optionally include one or more of the following features, alone or in combination: In some implementations, augmenting the model input with the identifier includes prepending a token identifier of the first machine learning task to the model input.

[0024] In some embodiments, the training data includes multiple paired data sets, wherein each of the paired data sets includes an input data set paired with an output data set, and the model input has a first type of model input, and the multiple paired data sets do not include a pairing of data sets including an input data set of the first type of model input paired with an output data set of the first type of model output.

[0025] In some implementations, datasets in the plurality of paired datasets include text segments in different languages, and each paired dataset includes input text segments in an input language paired with text segments in a target language different from the input language.

[0026] In some implementations, the plurality of machine learning tasks includes, for each paired dataset, translating an input text segment into a text segment in a target language.

[0027] In some implementations, augmenting the model input with an identifier of the machine learning task to generate the augmented model input includes appending a token indicative of at least the target language to the model input.

[0028] The details of one or more embodiments of the subject matter of this specification are set forth in the drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An example implicit bridging system for performing machine learning tasks is shown.

[0030] Figure 2 is a flow chart of an example process for performing implicit bridging for machine learning tasks.

[0031] Figure 3 is a flow chart of an example process for implicit bridging of training a machine learning system to perform a machine learning task.

[0032] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0033] This specification describes methods and systems for performing machine learning tasks using implicit bridging. For example, a zero-shot translation system uses training data from a set of language pairs, such as English-Japanese, Japanese-English, English-Korean, Korean-English, to train a machine translation model to translate text in a source language into text in a target language. Through training, the zero-shot translation system learns to translate unseen language pairs, such as Korean-Japanese and Japanese-Korean, even if this type of explicit data has not been used in training. As another example, the system learns to construct a parse tree representation of the text and uses training data from a set of (sentence, parse tree) and (parse tree, sentiment) pairs to train a machine learning model to predict the sentiment of a given parse tree. Through training, the system learns to directly predict the sentiment of a given sentence, even if this type of explicit data has not been used in training.

[0034] Figure 1An example implicit bridging system 100 for performing machine learning tasks is shown. For example, system 100 can be a system for performing zero-shot translation as described in more detail below. Other machine learning tasks include sentiment analysis or other natural language processing tasks. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0035] The implicit bridging system 100 receives a model input 102a and data 102b identifying a machine learning task to be performed on the model input. The implicit bridging system 100 uses an augmentation module 104 and a machine learning model 106 to process the received model input 102a and the data 102b identifying the machine learning task to be performed on the model input to generate a model output 108. The generated model output 108 is a specific type of model output. For example, in some cases the model input can be a text segment in a source language, such as "Hello, how are you?", and the machine learning task to be performed on the text segment can be to translate from the text segment from the source language to a target language, for example, translating "Hello, how are you?" from English to Spanish. In this example, the target language can represent the type of model output generated.

[0036] The augmentation module 104 receives the model input 102a and data 102b identifying a machine learning task to be performed on the model input, and augments the model input 102a with an identifier of a first machine learning task to generate an augmented model input 108. In some embodiments, the augmentation module 104 augments the model input 102a by prepending a token identifier of the first machine learning task to the model input. For example, as described above, in some cases the model input 102a may be a text segment in a source language and the machine learning task to be performed on the text segment may be translating the text segment from the source language to a target language. In this example, the augmentation module 104 may prepend a token indicating at least the target language to the model input. For example, the augmentation module 104 may prepend a token <2xx>, where "xx" represents a target language code, such as EN for English or JP for Japanese. Continuing with the above example, the augmented model input 108 may be <2ES> Hello, how are you?

[0037] In some embodiments, the augmentation module 104 can also replace the standard token <2xx> by prepending the token <2xx> to the output text segment, for example <s>To augment the output text segment in the target language. For example, prepending the text segment in the target language can be beneficial in some cases because the input text segment is encoded in the source language and then independent of the target language. This can allow one text segment in the source language to be translated into many languages ​​with only one encoding.

[0038] In some implementations, the augmentation module may prepend " <xx>"Symbol and prefix the corresponding text in the target language" <xx>” token. For example, in some cases this type of prepending can enable the system to add monolingual data to the machine learning model, for example, to obtain a low-resource language. In these cases, the augmentation module 104 can be configured to receive the model input 102a and data 102b identifying the machine learning task to be performed on the model input and the output from the machine learning model 106.

[0039] The machine learning model 104 receives the generated augmented model input 108. The machine learning model 104 has been configured through training to process the augmented model input 108 to generate a machine learning model output 110 for the model input 102a. The type of machine learning model output generated by the machine learning model depends on the augmented model input received, i.e., the machine learning task identifier, and the type of task that the machine learning model has been trained to perform. For example, continuing the above example, the machine learning model output 110 may be "Hola, como estas?". In the following reference Figure 3 Describes in more detail the implicit bridge for training machine learning models to perform machine learning tasks.

[0040] The machine learning model 104 includes an encoder component 112 and a decoder component 114. In some embodiments, the encoder component 112 and the decoder component 114 are both recurrent neural networks. In some embodiments, the decoder neural network can include an attention mechanism and can include a softmax output layer. An example neural machine translation model is described in more detail in "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation," Wu, Yonghui et al., arXiv:1609.08144 (2016).

[0041] As described above, in some cases the model input 102a can be a text segment in a source language, and the machine learning task to be performed on the text segment can be to translate the text segment from the source language to a target language. The source language can be one of many possible source languages, and the target language can be one of many possible target languages. In these cases the machine learning model 106 can include an encoder shared by different source languages, a decoder shared by different target languages, and a shared vocabulary of size V across different languages. In some embodiments, the vocabulary can be a shared chunk vocabulary, i.e., a vocabulary containing subword units that can be assembled into words.

[0042] In some embodiments, the implicit bridging system 100 can be configured to generate a shared vocabulary. For example, the system 100 can generate multiple vocabularies of a fixed size V in each of multiple languages ​​and merge the generated vocabularies to generate a new single vocabulary of size V. For example, the system 100 can sequentially select the highest-occurring words in each generated vocabulary until the size of the new vocabulary reaches V. Optionally, the system can remove duplicate words from the generated vocabularies until the size of the new vocabulary reaches V. For example, when generating a vocabulary that merges English words with German words, the system can deduplicate the English word "die" with the German article "die".

[0043] In some cases, the vocabulary may include a probability distribution of words across different languages, e.g., a vocabulary generated for n different languages ​​includes a uniform distribution of V / n words for each language. In other cases, the vocabulary may include a data-driven distribution of words across different languages, e.g., a generated vocabulary may include a different number of words for each language.

[0044] In some cases, the implicit bridging system 100 can be configured to generate a model output that is a translation of the received model input into a single target language. In these cases, the machine learning model 106 can include an encoder 112 shared across different source languages, a decoder for a single target language, and a vocabulary shared across different source languages. In other cases, the implicit bridging system 100 can be configured to generate a model output that is a translation of the received model input in a single source language into multiple target languages.

[0045] Figure 2 is a flow chart of an example process for performing implicit bridging of machine learning tasks. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system including a machine learning model appropriately programmed in accordance with the present specification, such as Figure 1 The implicit bridging system 100 may perform process 200 .

[0046] The system receives (i) a model input and (ii) data identifying a first machine learning task to be performed on the model input to generate a first type of model output for the model input (step 202). For example, in some embodiments the received model input may include text segments in a source language, such as Japanese. The received data identifying the first machine learning task to be performed on the model input to generate a first type of model output for the model input may include data identifying a task of translating text segments in a source language to generate corresponding text segments in a target language, such as Korean. In other embodiments, the received model input may include text segments, and the received data identifying the first machine learning task to be performed on the model input to generate a first type of model output for the model input may include data identifying a task of predicting the sentiment of the text segments.

[0047] The system augments the model input with the identifier of the first machine learning task to generate an augmented model input (step 204). In some embodiments, the system can augment the model input with the identifier of the first machine learning task by prepending a token identifier of the first machine learning task to the model input. For example, where the model input is a text segment in a source language and the machine learning task includes translating the text segment into a text segment in a target language, the system can prepend a "<2xx>" token to the text segment in the source language, where xx represents a target language code, such as EN for English or DE for German. As another example, where the model input is a text segment and the machine learning task includes predicting a sentiment of the text segment, the system can prepend a "<2sentiment" token to the text segment.

[0048] As another example, the system may prepend the text segments in the source language with an additional token that also indicates the source language, e.g. <s> <en> <de> How are you>< / de> < / en> < / s> <s> How do I get one?< / s> This approach can be beneficial in some cases, such as when translating homophones from two different languages ​​that have different meanings, such as "die" in English and "die" in German. When text segments are prepended with tokens indicating the source and target languages, the order of the tokens must be maintained in each example. The system can then learn that the first token indicates the source language and the second token indicates the target language, or vice versa.

[0049] Alternatively, in some cases the system may be able to bypass the standard token by prepending the token <2xx> to the output text segment (e.g., replacing the standard token <s>) to augment the output text segment in the target language. In some cases, prepending the text segment in the target language can be beneficial because the input text segment is encoded in the source language and then independent of the target language. This can allow one text segment in the source language to be translated into many languages ​​with only one encoding.

[0050] As another alternative, for example, to enable multilingual and monolingual training, the system may in some cases prepend the source language text segments with " <xx>"Symbol and prefix the corresponding text in the target language" <xx>" token. In some cases, this prepending method can enable the system to add monolingual data to the machine learning model, for example, for low-resource languages. For example, when training an English, Hindi, Punjabi machine learning model with Punjabi as a low-resource language, the system can include monolingual Punjabi data to the model via the following:

[0051] <pa> Punjabi Sentences< / pa> < / xx> < / xx> < / s> <pa> Punjabi Sentences< / pa> < / xx> < / xx> < / s> .

[0052] In these examples, even if the system does not directly translate the text segments from or into Punjabi, the system can be exposed to an increased amount of Punjabi text and can learn Punjabi vocabulary and rare words. In this way, the ability of the machine learning model to translate text segments into or from Punjabi can be increased.

[0053] The system processes the augmented model input using a machine learning model (step 206). The machine learning model is a machine learning model that has been trained on training data to perform a set of machine learning tasks including a first machine learning task and has been configured through training to process the augmented model input to generate a first machine learning model output of a first type for the model input.

[0054] For example, the first machine learning task may be a task of translating a model input into a specific language, such as Korean, and the machine learning model may have been trained on the training data to perform the task of translating a given model input into one or more languages ​​including the specific language, for example, the machine learning model may have been trained to translate a given model input into English, Korean, and Japanese. As another example, the first machine learning task may be a task of predicting the sentiment of a text segment, and the machine learning model may have been trained on the training data to perform two subtasks—generating a parse tree representation of a given model input and predicting the sentiment of the parse tree representation.

[0055] In some cases, the training data for training the machine learning model may include a set of paired data sets, where each of the paired data sets includes an input data set paired with an output data set. In this case, the model input may be a first type of model input and the set of paired data sets may not include an input data set of the first type of model input paired with an output data set of the first type of model output.

[0056] For example, the model input may be a text segment in a source language, such as Japanese, which is to be translated into a text segment in a target language, such as Korean. In this case, the training data used to train the machine learning model may include a pair set of text segments in different languages, such as (Japanese, English), (English, Japanese), (Korean, English), (English, Korean), wherein the pair set of text segments in different languages ​​does not include a pairing of text in the source language with text in the target language, such as (Japanese, Korean). However, because the machine learning model has been trained to receive a model input containing text segments in Japanese, for example, when translating a text segment in Japanese into a text segment in English, and has been trained to translate a given text segment into Korean, for example, by processing the training data pair (English, Korean), the machine learning model has been trained to perform zero-shot translation, for example, through text segments in English, and is able to directly translate a text segment in Japanese into a corresponding text segment in Korean.

[0057] As another example, the model input may be a text segment for which the sentiment is to be predicted. In this case, the training data used to train the machine learning model may include pairs of sentences and corresponding parse tree representations, and pairs of parse tree representations and corresponding sentiments. Using this training data, the system can be trained to directly predict the sentiment of a given text segment. Figure 3 Describes in more detail the implicit bridge for training machine learning models to perform machine learning tasks.

[0058] Figure 3 is a flow chart of an example process 300 for implicitly bridging a machine learning model for training a machine learning model to perform a machine learning task. For example, process 300 may be used to train Figure 1 The machine learning model 104. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system for performing implicit bridging appropriately programmed according to the present specification, such as Figure 1 The system 100 may perform the process 300 .

[0059] The system obtains training data for training a machine learning model (step 302). The training data for training the machine learning model includes a set of paired data sets, each of which includes an input data set paired with an output data set. In some embodiments, the model input received by the machine learning model at runtime may be a model input of a first input type, for example, input1, and the model output generated by the machine learning model may be a model output of a first output type, for example, output1. In some cases, the set of paired data sets used to train the machine learning model may not include an input data set of the first input type paired with an output data set of the first output type, for example, the set of paired data sets may not include a pair (input1, output1). For example, the set of paired data sets may include one or more of the pairs (input1, output2), (input2, output2), (input2, output3), (input3, output1), or (input3, output3), where the index indicates the type of input or output.

[0060] In some embodiments, the datasets in the set of paired datasets may be text segments in different languages. In this case, each paired dataset may then include an input text segment of an input language paired with an output text segment of an output language different from the input language. In these embodiments, the model input received by the machine learning model at runtime may be an input text segment of a first language, for example, a text segment of Japanese, and the model output generated by the machine learning model may be a text segment of a second language, such as Korean. The set of paired datasets may not include a text segment of the first language paired with a text segment of the second language, for example, a (Japanese, Korean) pair. For example, the set of paired datasets may include pairs such as (English, Korean), (Korean, English), (English, Japanese), (Japanese, English).

[0061] Each of the paired datasets also includes a language identifier that specifies a machine learning task associated with the paired dataset, for example, the paired dataset (English, Korean) may include an identifier that specifies that the English dataset is translated into a Korean dataset.

[0062] In some embodiments, the datasets in the set of paired datasets can be text segments, parse tree representations, and sentiments in one or more languages. In this case, the paired datasets can include text segments paired with corresponding parse tree representations and parse tree representations paired with corresponding sentiments. The model input received by the machine learning model at runtime can be an input text segment, for example, "I hate flowers", and the model output generated by the machine learning model can be a sentiment, for example, "negative". Each of the paired datasets also includes an identifier that specifies a task associated with the paired dataset, for example, an identifier that specifies "generate a parse tree representation" or "predict sentiment".

[0063] The system trains the machine learning model on the training data to perform a set of machine learning tasks (step 304). The system uses standard machine learning techniques to train the machine learning model. For example, where the machine learning model is a neural network, the system can train the neural network on the training data by processing training inputs, such as text segments in a source language, to generate training outputs, such as text segments in a target language, according to a given machine learning task, such as translating a given input into text segments in a target language. The system can then compare the training outputs to the known outputs by calculating a loss function and back-propagating the loss function gradients with respect to the current neural network weights to determine a set of updated neural network weights that minimize the loss function, as described above with reference to Figure 1 As described.

[0064] The set of machine learning tasks includes machine learning tasks to be performed on the model input at runtime, as described above with reference to step 304. In other words, the machine learning model is trained to perform machine learning tasks that can be provided to the system at runtime. For example, continuing the example given above in step 302, in some cases the data sets in the set of paired data sets include text segments in different languages, such as Japanese, English, or Korean. In this example, each paired data set includes an input text segment of the input language paired with a text segment of a target language different from the input language, such as (Japanese, English), (English, Japanese), (English, Korean), (Korean, English). The set of machine learning tasks can then include, for each paired data set, translating the input text segment into a text segment of the target language, for example, translating the input text segment into an English text segment, translating the input text segment into a Japanese text segment, and translating the input text segment into a Korean text segment.

[0065] Through the training process 300, the machine learning model learns to process a given model input according to a given machine learning task, even if the model has not been explicitly trained to perform the machine learning task on a particular type of model input. For example, as described above, even if the machine learning model is only trained to translate Japanese text into a third "bridge" language and the third language into Korean, the machine learning model can learn to translate a text segment in Japanese directly into a corresponding text segment in Korean.

[0066] In some implementations, the system can augment the training process 300 to take advantage of available parallel data to improve the effectiveness of the process 300. As described above, in this context, the parallel data describes training data for a machine learning task to be performed on a model input at runtime. In other words, the parallel data can describe training data for the machine learning task that is not included in the training data obtained at step 302. Continuing with the above example, the parallel data can include training data that can be used to train a machine learning model to translate a text segment in Japanese into a corresponding text segment in Korean, e.g., a paired dataset (Japanese, Korean).

[0067] The system can perform steps 302 and 304 described above to train the machine learning model to perform implicit bridging of machine learning tasks. For example, as described above, even if the training data obtained in step 302 includes training data for training the machine learning model to translate Japanese text into a "bridge" language and translate a third language into Korean, the machine learning model can learn to translate a text segment in Japanese directly into a corresponding text segment in Korean.

[0068] The system obtains additional parallel training data to train the machine learning model to perform a machine learning task not represented by the training data obtained at step 302. Similar to the training data described above with reference to step 302, the obtained parallel training data may include a set of paired data sets, wherein each of the paired data sets includes an input data set paired with an output data set. Continuing with the above example, the parallel training data may include a set of text segments in Japanese paired with corresponding text segments in Korean. In some embodiments, the obtained parallel training data may be a small amount of training data. For example, the size of the parallel training data may be smaller than the training data set described above with reference to step 302, and / or the parallel training data may include fewer training examples than the training data set obtained at step 302.

[0069] The system uses the obtained parallel training data to train the machine learning model to perform the machine learning task not represented by the training data obtained at step 302. As described above, this can include applying standard machine learning techniques. For example, where the machine learning model is a neural network, the system can train the neural network on the parallel training data by processing the parallel data training input, such as text segments in the source language Japanese, to generate parallel data training outputs, such as text segments in the target language Korean. As described above with reference to Figure 1 As described, the system can then compare the parallel data training outputs with the known parallel data outputs by calculating a loss function and back-propagating the loss function gradients relative to the current neural network weights to determine a set of updated neural network weights that minimize the loss function.

[0070] As and when additional parallel data becomes available to the system, the system can repeat this additional training process for the additional parallel data.

[0071] By incrementally training the multi-task machine learning model on additional parallel data for zero-shot directions, i.e., for machine learning tasks not represented by the originally obtained training data, the system can further refine the multi-task machine learning model and improve the accuracy of the results obtained from using the machine learning model at runtime. In some embodiments, performing an augmented training process as described above can enable the machine learning model to perform some machine learning tasks with higher accuracy compared to a model trained using a single training process to utilize a mixture of available training data, i.e., training data including parallel data.

[0072] For illustrative purposes, the systems and methods described in this specification have used machine translation as the primary example use case. However, the described systems and methods can be applied to a variety of other settings, including other natural language tasks such as parsing or sentiment analysis. For example, the systems and methods can be used to predict the sentiment from a given sentence, for example, to predict the sentiment of the phrase "I hate flowers". In this example, the system can be trained to perform two machine learning subtasks, namely (1) translating a given text segment into a parse tree, and (2) predicting the sentiment of the parse tree. For example:

[0073] <2parsetree> I love flowers (ROOT (S (NP (PRP I)) (VP (VBP love)(NP (NNS flowers))) (. .)))

[0074] …

[0075] <2sentiment> (ROOT (S (NP (PRP I)) (VP (VBP love) (NP (NNS flowers)))(. .))) positive .

[0076] The system can then directly learn to bridge from sentence to sentiment:

[0077] <2sentiment> I hate flowers “negative”.

[0078] Embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagation signal, however.

[0079] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. An apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may include, in addition to hardware, code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0080] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as an independent program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may but need not correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordination files (e.g., files storing a portion of one or more modules, subroutines or codes). A computer program may be deployed to execute on a computer or on multiple computers located at a site or distributed across multiple sites.

[0081] As used in this specification, "engine" or "any engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be a functional coding block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device including one or more processors and a computer-readable medium, such as a server, a smart phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. Additionally, two or more of these engines can be implemented on the same computing device or on different computing devices.

[0082] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and the apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0083] Computers suitable for executing computer programs include, as examples, central processing units that can be based on general-purpose microprocessors or special-purpose microprocessors or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The necessary elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operationally coupled to receive data from one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or transfer data to one or more large-capacity storage devices for storing data, or both. However, a computer does not necessarily have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0085] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0086] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser that a user can use to interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0087] The computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the user device, for example, for the purpose of displaying data to a user interacting with the user device as a client and receiving user input from the user interacting with the user device as a client. Data generated at the user device, for example, the result of the user interaction, can be received from the user device at the server.

[0088] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases.

[0089] Similarly, although operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired effect. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0090] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method comprising: receiving (i) a model input representing a language input in a source language and (ii) data identifying a target task to be performed on the model input; as well as Processing (i) the model input and (ii) a task identifier identifying the target task using a machine learning model to generate a language output, wherein the machine learning model has been trained on a set of training samples, and wherein the machine learning model includes: an encoder neural network configured to encode the model input independently of the target task, and a decoder neural network shared between a plurality of different languages ​​and configured to generate an output from a shared vocabulary, the shared vocabulary including outputs from all of the plurality of different languages, and wherein processing the model input comprises prepending the task identifier to an output of the decoder neural network.

2. The method according to claim 1, wherein: The set of training samples includes samples across multiple different languages.

3. The method according to claim 1, wherein: The target task requires generating output text in a target language; and The task identifier includes a token identifying the target language.

4. The method according to claim 3, wherein: The set of training samples includes a plurality of paired data sets, wherein each of the plurality of paired data sets includes a text input data set in a corresponding input language paired with an output data set in a corresponding output language.

5. The method according to claim 4, wherein: The plurality of paired data sets do not include a paired data set including an input data set in the source language paired with an output data set in the target language.

6. The method according to claim 1, wherein: The task identifier includes a token that identifies the target task.

7. The method according to claim 1, wherein: The decoder neural network has an attention mechanism.

8. The method according to claim 1, wherein: The shared vocabulary is a shared chunk vocabulary including sub-word units shared between the plurality of different languages.

9. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable to, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving (i) a model input representing a language input in a source language and (ii) data identifying a target task to be performed on the model input; as well as Processing (i) the model input and (ii) a task identifier identifying the target task using a machine learning model to generate a language output, wherein the machine learning model has been trained on a set of training samples, and wherein the machine learning model includes: an encoder neural network configured to encode the model input independently of the target task, and a decoder neural network shared between a plurality of different languages ​​and configured to generate an output from a shared vocabulary, the shared vocabulary including outputs from all of the plurality of different languages, and wherein processing the model input comprises prepending the task identifier to an output of the decoder neural network.

10. The system according to claim 9, wherein: The set of training samples includes samples across multiple different languages.

11. The system of claim 9, wherein: The target task requires generating output text in a target language; and The task identifier includes a token identifying the target language.

12. The system according to claim 11, wherein: The set of training samples includes a plurality of paired data sets, wherein each of the plurality of paired data sets includes a text input data set in a corresponding input language paired with an output data set in a corresponding output language.

13. The system according to claim 12, wherein: The plurality of paired data sets do not include a paired data set including an input data set in the source language paired with an output data set in the target language.

14. The system according to claim 9, wherein: The task identifier includes a token that identifies the target task.

15. The system according to claim 9, wherein: The decoder neural network has an attention mechanism.

16. The system of claim 9, wherein: The shared vocabulary is a shared chunk vocabulary including sub-word units shared between the plurality of different languages.

17. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving (i) a model input representing a language input in a source language and (ii) data identifying a target task to be performed on the model input; as well as Processing (i) the model input and (ii) a task identifier identifying the target task using a machine learning model to generate a language output, wherein the machine learning model has been trained on a set of training samples, and wherein the machine learning model includes: an encoder neural network configured to encode the model input independently of the target task, and a decoder neural network shared between a plurality of different languages ​​and configured to generate an output from a shared vocabulary, the shared vocabulary including outputs from all of the plurality of different languages, and wherein processing the model input comprises prepending the task identifier to an output of the decoder neural network.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein: The set of training samples includes samples across multiple different languages.

19. The one or more non-transitory computer-readable storage media of claim 17, wherein: The target task requires generating output text in a target language; and The task identifier includes a token identifying the target language.

20. The one or more non-transitory computer-readable storage media of claim 19, wherein: The set of training samples includes a plurality of paired data sets, wherein each of the plurality of paired data sets includes a text input data set in a corresponding input language paired with an output data set in a corresponding output language.

21. A computer-implemented method comprising: receiving (i) a model input comprising text in a source language and (ii) data identifying a target language into which the text in the source language is to be translated by a machine learning model; augmenting the model input with an identifier identifying at least the target language to generate an augmented model input; as well as The augmented model input is processed using a machine learning model to generate a model output, the model output being a translation of the model input into the target language, wherein the machine learning model has been trained on training data to translate the model input into a plurality of different languages ​​including the target language, and wherein the model output is a sequence including outputs from a shared vocabulary, the shared vocabulary including outputs from all of the plurality of different languages.

22. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable to, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving (i) a source language text segment and (ii) data identifying a target language into which the source language text segment is to be translated by a machine learning model, wherein: The machine learning model has been trained on multiple different paired datasets. Each paired data set includes (iii) a first language text segment in a corresponding first language and (iv) a second language text segment in a corresponding second language, the second language text segment being a translation of the first language text segment into the second language, and wherein the machine learning model has not been trained on any paired dataset comprising translations of text segments from the source language to the target language; augmenting the source language text segment with an identifier identifying at least the target language to generate an augmentation model input; and The augmented model input is processed using the machine learning model to generate a target language text segment that is a translation of the source language text segment into the target language even if the machine learning model has not been trained on any paired dataset that includes translations of text segments from the source language to the target language.

Images