Cross-lingual classification using multilingual neural machine translation

Abstract

A multilingual classification neural network model is trained and / or used to perform a natural language processing classification task, wherein the model reuses the encoder portion of a multilingual neural machine translation model. In various embodiments, a client device can generate a natural language data stream from a verbal input from a user. The natural language data stream can be used as an input to an encoder portion of the multilingual classification model. The output generated by the encoder portion can be used as an input to a classifier portion of the multilingual classification model. The classifier portion can generate a predicted classification label for the natural language data stream. In many embodiments, an output can be generated based on the predicted classification label, and the client device can present the output.

Description

Technical Field

[0001] The present application relates generally to techniques for performing natural language processing ("NLP") classification tasks, and more specifically, to cross-lingual classification using multilingual neural machine translation. Background Art

[0002] Machine translation ("MT") can translate text, speech, etc. from a first language into a second language using one or more computing devices. For example, an MT process can replace words in a first language with corresponding words in a second language. However, simple replacement of words can produce inaccurate translations because a phrase can have a specific meaning that is different from the literal translation of each word within the phrase. For example, the English phrase "leap year" has a specific meaning (e.g., the year that includes February 28), but a word replacement may translate "leap year" into the second language as a word indicating a year that leaps.

[0003] Neural machine translation ("NMT") models can use artificial neural networks to predict the likelihood of a sequence of words. In many embodiments, the NMT model can translate entire phrases and / or sentences at once. Rather than translating individual tokens (i.e., individual characters, words, etc.), translating entire sentences and / or phrases can leverage the overall context of the text and / or speech input. The NMT model can utilize a variety of neural network learning techniques to perform the translation, including: feature learning techniques, deep learning techniques, etc. Summary of the invention

[0004] This article describes techniques for enabling a multilingual classification neural network model to perform natural language processing ("NLP") classification tasks by reusing the encoder portion of a multilingual NMT model to utilize a language-agnostic representation parameterized by the encoder portion through the learned weights of the encoder portion generated by training the NMT model. NLP tasks generally focus on how computing devices can understand human language (e.g., text, speech, hand movements, etc.) and derive meaning from human language in a useful way.

[0005] In various embodiments, the multilingual classification network model may include an encoder part and a classifier part. The encoder part of the multilingual classification model may be a reused encoder part of a trained multilingual NMT model.

[0006] According to various embodiments, a multilingual NMT model can be trained to translate between multiple languages. For example, a multilingual NMT neural network model can include an encoder portion, one or more decoder portions, and one or more attention mechanisms (i.e., one decoder portion and one corresponding attention mechanism for each language). For example, when training an NMT model to translate between four languages, the NMT model can share one encoder portion between all four languages ​​and have a separate decoder portion and corresponding attention mechanism for each of the four languages ​​(i.e., four separate decoder portions and four corresponding attention mechanisms).

[0007] The encoder portion of the trained multilingual NMT model can be reused, while the decoder portion(s) and attention mechanism(s) can be discarded. Reusing the encoder portion of the trained multilingual NMT network model can enable other network models (such as a multilingual classification model according to many embodiments) to utilize the language-agnostic representation of the encoder portion parameterized by the learned weights of the encoder portion, wherein the learned weights are generated by training the NMT model to translate various languages.

[0008] In many embodiments, the encoder portion of the multilingual classification model may include one or more recurrent network layers. As an illustrative example, the encoder portion may include a stack of bidirectional recurrent network layers and unidirectional recurrent network layers (e.g., three unidirectional recurrent network layers). The hidden state of the memory unit in the last recurrent network layer may form the multilingual classification model encoder output. The recurrent network layers may each include one or more memory units, to which inputs may be applied in sequence, and in each iteration of the applied inputs, a new hidden state may be calculated using (one or more) memory units based on the inputs of that iteration and based on the current hidden state (which may be based on (one or more) inputs of (one or more) previous iterations). In some embodiments, the memory unit may be a long short-term memory ("LSTM") unit. In some embodiments, additional or alternative (one or more) memory units, such as a gated recurrent unit ("GRU"), may be utilized.

[0009] Additionally or alternatively, the classifier portion of the multilingual classification model may include a network layer to perform pooling (e.g., pooling over time). The classifier portion of the multilingual classification model may process the encoder output and generate a classification label for the NLP task. For example, the classifier portion may include a pre-pooling layer having one or more feed-forward network layers, a pooling layer, and a post-pooling layer having one or more feed-forward network layers. In some embodiments, a soft maximum function may process the post-pooling layer output to generate an NLP classification label for the multilingual classification model.

[0010] In various embodiments, by reusing the encoder portion of a trained NMT model as the encoder portion of a multilingual classification model, the multilingual classification model can be used to classify received inputs in a particular language, even though no training examples in the particular language have been used to train the multilingual classification model (e.g., at least the learned weights of the classifier portion may not be based on any training examples including training example inputs in the particular language). For example, the encoder portion of the multilingual classification model can be initialized by reusing the encoder portion of an NMT model that is trained to translate English into French and French into English (i.e., trained to translate two languages). After initialization of the encoder portion, the multilingual classification model can be trained using only English training examples. In various embodiments, the multilingual classification model trained using only English training examples can still be used to generate accurate classification predictions for various French inputs, even though no training examples are based on labeled classification predictions for French training example inputs. According to embodiments disclosed herein, this surprising result may be based on the language-agnostic representations generated using the reused encoder portion of the NMT model, and the combination of the encoder portion with the classification portion.

[0011] In some examples, reusing the encoder portion of a trained NMT model as the encoder portion of a multilingual classification model can allow the multilingual classification model to be trained to classify received inputs in a particular language using only a small number of training examples that include training example inputs in the particular language. For example, a multilingual classification model initialized with an NMT encoder as described above (i.e., trained to translate English into French and French into English) can be trained using a majority of English training examples (i.e., training examples with English training example inputs) and a small number of French training examples (i.e., training examples with French training example inputs). In various embodiments, the number of French training examples utilized can be less than fifty percent of the English training examples utilized, less than ten percent of the English training examples utilized, less than five percent of the English training examples utilized, and so on. In various embodiments, a multilingual classification model trained using only a small number of French training examples can still be used to generate accurate classification predictions for a variety of French inputs. According to embodiments disclosed herein, this surprising result may be based on language-agnostic representations generated using the reused encoder portion of the NMT model, and the combination of the encoder portion with the classification portion.

[0012] In various embodiments, zero-shot learning or few-shot learning can therefore be used to train a multilingual classification model for one or more languages. By using the zero-shot and / or few-shot techniques disclosed herein, various computational efficiencies can be achieved. As an example, the computational resources used in training a multilingual classification model can be minimized because a small number of training examples (few samples) and / or no training examples (zero samples) can be used for various languages ​​during training, and the trained multilingual classification model can still be used to accurately and robustly classify natural language inputs of these languages. For example, a large number of training examples for each of N (e.g., N=3 or more, 5 or more, 10 or more, etc.) languages ​​can be used to train a multilingual classification model, while a small number (or no) of training examples for each of M (e.g., M=3 or more, 5 or more, 10 or more, etc.) languages ​​can be used to train a multilingual classification model. However, using the trained multilingual classification model, accurate and robust classification can still be achieved for natural language inputs of each of M languages. As another example, various computational resources used to generate supervised training examples for various languages ​​can be minimized. For example, it may not be necessary to expend resources to generate labels for training examples in certain languages ​​(zero-shot) and / or it may only be necessary to expend resources to generate labels for a small number of training examples in certain (one or more) languages ​​(few-shot).

[0013] In addition, various learning, (one or more) model architectures, and / or other techniques disclosed herein enable training and use of multilingual classification models that generate accurate classifications for a large number of languages. As described above, such training of models can achieve various computational efficiencies. For example, such training of models can utilize a smaller number of training examples than, for example, training a separate model for each different language. In addition, once the model is trained, it can be stored in less memory than, for example, a different classification model for each language. Such memory savings, while still enabling classification predictions across multiple languages, may also be beneficial in many scenarios, such as scenarios where a client device stores a multilingual classification model locally and uses the model to locally process natural language input.

[0014] The above description is provided as an overview of various embodiments. These and other embodiments are described in more detail herein.

[0015] A multilingual classification neural network model is trained and / or used to perform a natural language processing classification task, wherein the model reuses the encoder portion of a multilingual neural machine translation model. In various embodiments, a client device can generate a natural language data stream from a verbal input from a user. The natural language data stream can be used as an input to an encoder portion of the multilingual classification model. The output generated by the encoder portion can be used as an input to a classifier portion of the multilingual classification model. The classifier portion can generate a predicted classification label for the natural language data stream. In many embodiments, an output can be generated based on the predicted classification label, and the client device can present the output.

[0016] In some embodiments, a method implemented by one or more processors is provided, the method comprising receiving a natural language data stream in a first language, the natural language data stream being generated based on a user input via a user interface of a client device. In various embodiments, the method comprises generating a classification label prediction for the natural language data stream in the first language, wherein generating the classification label prediction comprises processing the natural language data stream using an encoder portion of a multilingual classification model to generate an encoder output. In many embodiments, the method further comprises processing the encoder output using a classifier portion of a multilingual classification model to generate a classification label prediction for the natural language data stream in the first language, wherein the multilingual classification model is a trained neural network model, wherein the encoder portion is a reusable multilingual neural machine translation encoder trained using multiple languages ​​including a first language and at least a second language, and wherein the multilingual classification model is trained for a classification task using supervised training examples in at least the second language, the supervised training examples in the second language each having a corresponding second language training example input in the second language. In many embodiments, the method further comprises generating an output using the classification label prediction for the natural language data stream in the first language. In many embodiments, the method further comprises causing the client device to render the output for presentation to the user.

[0017] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0018] In various implementations, a multilingual classification model is trained for a classification task without using any supervised training examples in the first language having first language training example input in the first language.

[0019] In various embodiments, a multilingual classification model is trained for a classification task using a first number of supervised training examples in a second language, each of the first number of supervised training examples in a second language having a corresponding second language training example input in the second language, wherein the multilingual classification model is trained for the classification task using a second number of supervised training examples in a first language, each of the second number of supervised training examples in a first language having a corresponding first language training example input in the first language, and wherein the second number is less than fifty percent of the first number. In some versions of these embodiments, the second number is less than ten percent of the first number, or less than five percent of the first number.

[0020] In various implementations, a multilingual neural machine translation encoder is trained to translate from at least a first language to at least a second language and / or from at least a second language to at least a first language.

[0021] In various embodiments, training a multilingual classification model using supervised training examples in a second language includes processing a second language training example input using an encoder portion and a classifier portion to generate a predicted output, and updating a weight of the classifier portion based on an error without updating any weights of a reused encoder portion, wherein the error is determined based on the predicted output and based on labels of the supervised training examples in the second language.

[0022] In various embodiments, processing the natural language data stream using the encoder portion to generate the encoder output includes: processing the natural language data stream using a bidirectional recurrent network layer of the encoder portion to generate the bidirectional recurrent output. In many embodiments, the method also includes processing the bidirectional recurrent output using one or more unidirectional recurrent network layers of the encoder portion, wherein the encoder output includes one or more hidden states of a last unidirectional recurrent network layer of the one or more unidirectional recurrent network layers.

[0023] In various embodiments, processing the encoder output using the classifier portion to generate a classification label prediction includes: processing the encoder output using a pooling layer of the classifier portion to generate a pooled output. In many embodiments, the method also includes processing the pooled output using one or more soft maximum layers of the classifier portion to generate a classification label prediction. In some versions of these embodiments, processing the encoder output using a pooling layer to generate a classification label prediction includes: processing the encoder output using a pre-pooling feed-forward layer in the pooling layer to generate a pre-pooling output. In some versions of these embodiments, the method also includes processing the pre-pooling output using a pooling layer in the pooling layer to generate a pooling output. In some versions of these embodiments, the method also includes processing the pooling output using a post-pooling feed-forward layer in the pooling layer to generate a post-pooling output. In some versions of these embodiments, the method also includes processing the post-pooling output using one or more soft maximum layers to generate a classification label prediction for a natural language data stream in a first language.

[0024] In various implementations, the plurality of languages ​​includes at least a first language, a second language, and a third language.

[0025] In various implementations, the user interface input is a spoken input, and the method further comprises generating a natural language data stream by converting the spoken input to text using a speech-to-text processor.

[0026] In various embodiments, the classification task is a natural language inference classification task, and wherein the classification label prediction indicates whether the natural language data stream is an entailment, is neutral, or is a contradiction. In some versions of these embodiments, the classification label prediction indicates that the natural language data stream is an entailment, and generating an output using the classification label prediction includes: in response to the classification label prediction indicating that the natural language data stream is an entailment: using the immediately preceding natural language data stream in conjunction with the natural language data stream when generating the output. In some versions of these embodiments, the natural language data stream and the preceding natural language data stream are each generated based on a corresponding spoken input. In some versions of these embodiments, the method further includes: using the classification label prediction to determine that the natural language data stream includes a request to modify the state of a networked hardware device. In some versions of these embodiments, in response to determining that the natural language data stream includes a request to modify the state of a networked hardware device, the method further includes causing a control signal to be provided to the networked hardware device, wherein the control signal causes the state of the networked hardware device to be modified. In some versions of these embodiments, the output perceptibly indicates a modification to the state of the networked hardware device. In some versions of these embodiments, the networked hardware device is selected from the group consisting of a light, a thermostat, and a networked camera.

[0027] Additionally, some embodiments include one or more processors (e.g., (one or more) central processing units (CPUs), (one or more) graphics processing units (GPUs), and / or (one or more) tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the performance of any of the methods described herein. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described herein.

[0028] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0030] Figure 2 An example of training a multilingual neural machine translation model according to an embodiment disclosed herein is shown.

[0031] Figure 3 An example of training a multilingual classification model according to an embodiment disclosed herein is shown.

[0032] Figure 4 An example of the encoder portion of a multilingual classification model according to embodiments disclosed herein is shown.

[0033] Figure 5 An example of a classifier portion of a multilingual classification model according to embodiments disclosed herein is shown.

[0034] Figure 6 is a flowchart illustrating a process of training a multilingual neural translation model according to an embodiment disclosed herein.

[0035] Figure 7 is a flowchart illustrating a process of training a multilingual classification model according to an embodiment disclosed herein.

[0036] Figure 8 is a flow chart illustrating a process of utilizing a multilingual classification model according to an embodiment disclosed herein.

[0037] Fig. 9 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0038] Figure 1 An example environment is shown in which implementations disclosed herein may be implemented. Figure 1 The example environment 100 in includes a client device 102, a multilingual NMT engine 108 (also referred to herein as the “NMT engine”), a multilingual NMT training engine 114 (also referred to herein as the “NMT training engine”), a multilingual NMT training instance engine 118 (also referred to herein as the “NMT training instance engine”), a multilingual classification engine 110 (also referred to herein as the “classification engine”), a multilingual classification training engine 122 (also referred to herein as the “classification training engine”), and a multilingual classification training instance engine 126 (also referred to herein as the “classification training instance engine”). The client device 102 may be, for example, a standalone voice-enabled speaker device, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), and / or a wearable device of a user including a computing device (e.g., a watch with a computing device, glasses with a computing device, a virtual reality or augmented reality computing device, etc.). Additional and / or alternative client devices may be provided.

[0039] Multilingual NMT engine 108, multilingual NMT training engine 114, multilingual NMT training instance engine 118, multilingual classification engine 110, multilingual classification training engine 122, and multilingual classification training instance engine 126 are example components in which the techniques described herein may be implemented and / or example components with which the systems, components, and techniques described herein may interface. Figure 1 The operations performed by one or more engines 108, 110, 114, 118, 122, 126 of the present invention may be distributed across multiple computing systems. In some embodiments, one or more aspects of engines 108, 110, 114, 118, 122, 126 may be combined in a single system and / or one or more aspects may be implemented on client device 102. For example, in some of these embodiments, aspects of multilingual NMT engine 108 may be combined with aspects of multilingual classification engine 110. Engines according to many embodiments may each be implemented in one or more computing devices that communicate, for example, via a communication network. The communication network may include a wide area network such as the Internet, one or more local area networks ("LANs") such as Wi-Fi LANs, a mesh network, etc., and / or one or more bus subsystems. The communication network may optionally utilize one or more standard communication technologies, protocols, and / or inter-process communication technologies.

[0040] In many implementations, the client device 102 can include a speech capture / speech to text (“STT”) / text to speech (“TTS”) module 104. In other implementations, the speech capture / STT / TTS module can be implemented separately from the client device 102. In many implementations, the speech capture / STT / TTS module 104 can generate speech recognition output (e.g., a natural language (“NL”) data stream) based on spoken input to the client device. Additionally or alternatively, the speech capture / STT / TTS module 104 can be configured to perform one or more functions: for example, capturing a user's voice via a microphone; converting the captured audio to text (and / or other representations or embeddings); and / or converting text to speech.

[0041] In various embodiments, the NL data stream can be generated from spoken input from a user via the client device 102 (e.g., by processing the spoken input through the voice capture / STT / TTS module 104). The NL data stream can be generated from various forms of user interface input, including spoken input captured by (one or more) microphones (and optionally converted to text as described herein), typed input (e.g., captured by a physical and / or virtual keyboard), gestures, etc. In some embodiments, the NL data stream includes a sequence of tokens. The tokens in the sequence can each be one or more characters, such as one or more words (e.g., "chicago" or "new york") or (one or more) characters that form part of a word (e.g., tokens "c", "h", "i", "a", "g", "o", rather than a single token "chicago"). In many embodiments, the tokens of the NL data stream can be used as input to a multilingual classification model 120 (also referred to herein as a "classification model") to generate a predicted output for an NLP task using the multilingual classification engine 110. In some embodiments, the speech capture / STT / TTS module 104 can convert the predicted classification label into a spoken representation, which can be presented to the user using an audio device (such as a speaker in an independent voice-enabled speaker) of the client device 102. Additionally or alternatively, the predicted classification label generated by the multi-language classification engine 110 can be used by the client device 102 to provide control signals for one or more linked devices associated with the client device (such as networked hardware devices), third-party applications, applications stored locally on the client device, etc. In various embodiments, the networked hardware devices can include: smart thermostats, smart refrigerators, networked cameras, smart lights, smart light switches, etc.

[0042] In many embodiments, using the multilingual NMT engine 108, the client device 102 can train the multilingual NMT model 112 (also referred to herein as the “NMT model”) before reusing the encoder portion of the multilingual NMT model 112 to initialize the encoder portion of the multilingual classification model 120 (such as Figure 3 and Figure 4 The following is about the multi-language classification encoder part 121 shown in FIG. Figure 2 The multilingual NMT model 112 is described as being trained using, for example, the multilingual NMT engine 108, the multilingual NMT training engine 114, and the multilingual NMT training instance engine 118 according to many of the embodiments described herein. Additionally or alternatively, the multilingual NMT engine 108 may be omitted. For example, in some such embodiments, the client device 102 may access a trained encoder portion of the multilingual NMT model 112 (such as Figure 3 The multi-language NMT encoder portion 113 is shown) (stored locally on the client device and / or accessed remotely via a communication network).

[0043] Additionally or alternatively, in various implementations, using the multilingual classification engine 110, the client device 102 can train the multilingual classification model 120. Figure 3 For example, the multilingual classification model 120 is trained using the multilingual classification engine 110 , the multilingual classification training engine 122 , and the multilingual classification training instance engine 126 .

[0044] The multilingual classification model 120 can be trained to solve various NLP tasks, including: semantic analysis, identification of textual implications, word sense disambiguation, topic identification, sentiment analysis, etc. For example, identification of textual implications determines the relationship between sentences and / or phrases. In various embodiments, the multilingual classification model 120 trained for the NLP task of identifying textual implications can generate at least three classification labels: implications, contradictions, and neutrality. The implications classification label indicates that the second phrase and / or sentence is implied by the first phrase and / or sentence (i.e., it is the consequence of the first phrase and / or sentence). The contradiction classification label can indicate that the second phrase and / or sentence contradicts the first phrase and / or sentence. Similarly, the neutral classification label can indicate that the second sentence and / or phrase is neither implied by the first sentence and / or phrase nor contradicts the first sentence and / or phrase. In various embodiments, the client device 102 can generate a first NL data stream and a second NL data stream from a spoken input from a user via a voice capture / TTS / STT module 104. The first NL data stream and the second NL data stream may be provided to the multilingual classification model 120 to provide an entailment, contradiction, and / or neutral classification label. The classification label may be used to determine whether the first NL data stream and the second NL data stream are related, and, for example, whether the second phrase and / or sentence provided by the user is an additional command for the networked hardware device. For example, the user may say "turn on the light" as a first phrase and say "make it blue" as a second phrase. The speech capture / TTS / STT module 104 may convert the two phrases into a first NL data stream and a second NL data stream. Using the multilingual classification engine 110, the multilingual classification model 120 may determine a textual entailment classification label indicating a relationship between the phrases "turn on the light" and "make it blue". For example, the multilingual classification model 120 may determine that "make it blue" is entailed by "turn on the light". The entailed classification label may indicate that the phrases are related, and the client device 102 may send a command to the smart light to turn on the light and change the color of the light to blue. Additionally or alternatively, the contradictory and / or neutral classification labels may indicate that the phrases are not related. For example, if the smart light cannot be changed to the color blue, the multilingual classification model 120 may generate contradictory and / or neutral predicted classification labels. In some such implementations, the client device 102 may turn on the smart light and request additional information about the spoken phrase "make it blue".

[0045] As another example, semantic analysis can approximate the concepts of sentences and / or phrases. The multilingual classification model 120 trained for the NLP task of semantic analysis can generate predicted classification tags indicating the concepts of sentences and / or phrases. For example, a user can provide the phrase "play me some music" as a verbal input to the client device 102. The speech recognition / STT / TTS module 104 can generate an NL data stream from the verbal input and provide the NL data stream as an input to the multilingual classification model 120 using the multilingual classification engine 110. The multilingual classification model 120 trained for semantic analysis can provide a classification tag indicating a classification tag related to playing music for the verbal input "play me some music". After receiving the classification tag indicating playing music, the client device 102 (e.g., using an additional music engine not shown) can determine the music to be played for the user and play music for the user via one or more speaker devices associated with the client device 102.

[0046] Figure 2 An example image 200 for training the multilingual NMT model 112 according to various embodiments is shown. The NMT training instances are retrieved from the multilingual NMT training instances 116. The NMT training instances may be retrieved by the multilingual NMT training instance engine 118 (in Figure 1 ) is generated, and may include, for example, a pair of information such as NMT training input and NMT known output. In many embodiments, the pair of information may be a phrase and / or sentence in a first language (i.e., NMT training input) and a phrase and / or sentence in the first language translated into a second language (i.e., NMT known output). For example, the NMT training input may be an English phrase, and the corresponding NMT known output may be a known translation of the English phrase as a German phrase. Additionally or alternatively, the phrase and / or sentence in the first language may be translated into a variety of other languages ​​(i.e., translated into a second language, translated into a third language, translated into a fourth language, etc.). As another example, the NMT training input may be an English phrase, and the corresponding NMT known output may be a translation of an English phrase (i.e., NMT training input) into a German phrase, a French phrase, a Spanish phrase, etc. (i.e., NMT known output).

[0047] The multilingual NMT training engine 114 applies the NMT training examples as input to train the multilingual NMT model 112. In various embodiments, the multilingual NMT model 112 may be trained using supervised learning. For example, the multilingual NMT training engine 114 may generate an NMT model output by applying the NMT training input to the multilingual NMT model 112. Additionally or alternatively, the multilingual NMT training engine 114 may compare the NMT model output with the NMT known output, and using the comparison, update one or more weights in the multilingual NMT model 112 (e.g., back propagate the difference across the multilingual NMT model 112). Figure 6 Process 600 shown in describes a process for training a multilingual NMT model according to many embodiments disclosed herein.

[0048] In many embodiments, the multilingual NMT model 112 may include various neural network layers, including a multilingual NMT encoder portion 113 (also referred to herein as an "NMT encoder") and one or more multilingual NMT decoder portions 115D1-115DN (also referred to herein as "NMT decoders"). In some such embodiments, each multilingual NMT decoder portion 115D1-115DN (one or more) may have a corresponding attention mechanism 117A1-117AN (one or more). In many embodiments, the multilingual NMT encoder portion 113 may include various recurrent network layers. For example, the multilingual NMT encoder portion 113 may include a stack of a first bidirectional recurrent network layer and three unidirectional recurrent network layers. In various embodiments, the multilingual classification encoder portion 121 of the multilingual classification model 120 may mirror the layers of the multilingual NMT encoder portion 113. For example, the recurrent network layers 406-412 (which may include a combination of bidirectional and / or unidirectional LSTM layers) Figure 4 ) can mirror the combination of recurrent network layers utilized in the multilingual NMT encoder portion 113. In various embodiments, utilizing the same combination of recurrent network layers in the NMT encoder portion 113 and the classification encoder portion 121 can facilitate the reuse of the multilingual NMT encoder portion 113 to initialize the multilingual classification encoder portion 121.

[0049] The multilingual NMT model 112 can share a single multilingual NMT encoder 113 with one or more multilingual NMT decoder parts 115D1-115DN (one or more) with corresponding attention mechanisms 117A1-117AN (one or more). For example, the multilingual NMT model 112 can be trained to translate between N languages. Using the shared multilingual NMT encoder part 113, a separate number N of multilingual NMT decoder parts 115 with corresponding attention mechanisms 117 can be utilized, wherein each multilingual NMT decoder part 115 and corresponding attention mechanism 117 translates a separate language. As another example, the multilingual NMT decoder part 115D1 and the corresponding attention mechanism 117A1 can translate the input from the multilingual NMT encoder part 113 into a first language, the multilingual NMT decoder part 115D2 and the corresponding attention mechanism 117A2 can translate the input from the multilingual NMT encoder part 113 into a second language, and so on, until the final Nth language can be translated using the multilingual NMT decoder part 115DN and the corresponding attention mechanism 117DN.

[0050] In many embodiments, the (one or more) multilingual NMT decoder parts 115D1-115DN may include one or more recurrent network layers, such as bidirectional recurrent network layers and / or unidirectional recurrent network layers. For example, the (one or more) multilingual NMT decoder parts according to various embodiments may include a stack of four unidirectional recurrent network layers utilizing LSTM units. In many embodiments, the first recurrent network layer may be interwoven with an attention mechanism. According to many embodiments, the attention mechanism may allow the decoder part to refer back to the entire input sequence, rather than forcing the neural network model to encode all information into a fixed-length vector. In other words, the attention mechanism may enable a neural network model (such as the multilingual NMT model 112) to access its internal memory (i.e., the hidden states of a network part such as the multilingual NMT encoder part 113), and may learn through training what to obtain from the encoder part to use in the decoder part of the neural network model.

[0051] Figure 3 An example image 300 of training a multilingual classification model 120 according to various embodiments is shown. The multilingual classification training engine 122 may initialize the multilingual classification encoder portion 121 by reusing the trained multilingual NMT encoder portion 113 of the multilingual NMT model 112. Additionally or alternatively, the multilingual classification training instance engine 126 ( Figure 1) can generate classification training instances from the multilingual classification training instances 124. In some implementations, the classification training instances can include a pair of information for an NLP classification task, such as a classification training input and a classification known output. In many implementations, the pair can be a phrase and / or sentence for the NLP classification task (i.e., classification training input) and a known classification label for the phrase and / or sentence (i.e., classification known output).

[0052] In various implementations, the multilingual classification model 120 may include a multilingual classification encoder portion 121 (also referred to herein as a “classification encoder”) and a multilingual classification classifier portion 123 (also referred to herein as a “classification classifier”). The multilingual classification encoder portion 121 may include various recurrent network layers. Figure 4 The multilingual classification encoder part 121 according to various embodiments is shown in FIG. Additionally or alternatively, the multilingual classification classifier part 123 may include various neural network layers, including: a feedforward layer, a pooling layer, a softmax layer, etc. Figure 5 , a multilingual classification classifier portion 123 is shown in accordance with various embodiments.

[0053] In various embodiments, the multilingual classification training instance engine 126 can generate training instances to utilize zero-shot learning. Utilizing zero-shot learning, the multilingual classification model 120 can be trained to generate classification labels for input in a first language from a user via a client device without using any training instances in the first language. For example, the multilingual classification model 120 can utilize a language-agnostic representation generated using a trained multilingual NMT encoder portion to enable a neural network model to generate predicted classification labels for input in a language other than the language(s) of the training instances used to train the neural network model.

[0054] Additionally or alternatively, the multilingual classification training instance engine 126 can generate training instances to utilize few-shot learning. Utilizing few-shot learning can train the multilingual classification model 120 to generate classification labels for input in the first language from the user via the client device using only a few training instances in the first language. In many implementations, using a small number of training examples in the input language can improve classification label prediction accuracy.

[0055] The multilingual classification training engine 122 may use classification training examples according to many embodiments as input to train the multilingual classification model 120. In various embodiments, the multilingual classification model 120 may be trained using supervised learning. For example, the multilingual classification training engine 122 may generate a classification model output by applying the classification training input to the multilingual classification model 120. Additionally or alternatively, the multilingual classification training engine 122 may compare the classification model output with the classification known output and, using the comparison, update one or more weights in the multilingual classification model 120 (e.g., backpropagate the difference across the entire multilingual classification model 120). In some embodiments, the multilingual classification training engine 122 may backpropagate the difference across the entire multilingual classification model 120 including the multilingual classification encoder portion 121 and the multilingual classification classifier portion 123. In other embodiments, the multilingual classification training engine 122 may backpropagate the difference only across the multilingual classification classifier portion 123, while keeping the multilingual classification encoder portion 121 unchanged (i.e., freezing the multilingual classification encoder portion after initializing the multilingual classification encoder portion with the trained multilingual NMT encoder portion). Figure 7 Process 700 is shown describing a process for training a multilingual classification model according to many embodiments disclosed herein.

[0056] Figure 4 1 shows a multilingual classification encoder portion 121 of a multilingual classification model according to various embodiments described herein. In many embodiments, the multilingual classification encoder portion 121 of a multilingual classification model can be reused by reusing a trained multilingual NMT encoder portion 113 (e.g. Figure 24 (shown) to initialize the multilingual classification encoder portion 121 of the multilingual classification model. In some such embodiments, the multilingual classification encoder portion 121 has the same combination of neural network layers as the multilingual NMT encoder portion 113, so that the trained weights in the multilingual NMT encoder portion 113 can be directly reused as weights in the multilingual classification encoder portion 121. For example, the weights learned during the training of the NMT encoder portion 113 can be reused in one or more recurrent network layers of the multilingual classification encoder portion 121, such as the recurrent network layers 406-412. In some embodiments, the initialized weights in the classification encoder portion 121 can be updated during the training of the multilingual classification model 120. In other embodiments, the initialized weights in the classification encoder portion 121 can remain unchanged during training (e.g., the weights in the classification encoder portion 121 can remain frozen with the weights reused from the NMT encoder portion 113, and only the weights in the multilingual classification classifier portion 123 are updated during training). The classification model input 404 may include I1-I4. In various embodiments, I1-IN may include the training inputs of the multilingual classification model 120. In other implementations, I1-IN may include an NL data stream for classification using the multilingual classification model 120 .

[0057] The classification encoder portion 121 may include various combinations of recurrent network layers including (one or more) bidirectional memory layers and / or (one or more) unidirectional memory layers. In many embodiments, the recurrent network layer may utilize memory cells including LSTM units and / or GRUs. For example, the first layer of the classification encoder portion 121 may include a bidirectional recurrent network layer (such as a bidirectional LSTM layer) 406, which includes first direction memory units M1D1N1-M1D1NN and second direction memory units M1D2N1-M1D2NN. In some such embodiments, the classification encoder portion 121 may additionally include a stack of three unidirectional recurrent network layers (such as unidirectional LSTM layers) 408, 410, 412.

[0058] Inputs I1-IN may be used as inputs to first direction memory cells M1D1N1-M1D1NN. For example, I1 may be used as input to M1D1N1. The output generated based on applying I1 to M1D1N1 may be used together with I2 as input to M1D1N2. The output generated based on applying I2 and the output of M1D1N1 to M1D1N2 may be used together with I3 as input to M1D1N3. This continues until M1D1NN is reached.

[0059] The second direction memory units M1D2N1-M1D2NN similarly operate on the inputs I1-IN in a second direction opposite to their actual order. For example, IN is used as an input to M1D2NN. The output based on applying IN to M1D2NN is applied to M1D2NN-1 (also not shown) together with the input IN-1 (not shown). This continues until M1D2N1 is reached.

[0060] The first unidirectional recurrent layer 408 includes a unidirectional recurrent network layer having memory units M2N1-M2NN. The output generated based on the first bidirectional recurrent memory layer 406 is used as an input to the first unidirectional recurrent network layer 408. For example, the output generated based on M1D1N1 and M1D2N1 (e.g., the concatenation of the two inputs) is used as an input to M2N1. The output generated based on applying M1D1N1 and M1D2N1 to M2N1 is applied to M2N2 together with the output generated based on M1D1N2 and M1D2N2. This continues in one direction until M2NN is reached.

[0061] Additionally or alternatively, the classification encoder portion 121 can utilize a stack of unidirectional recurrent memory layers 408, 410, 412. Similarly, the output generated by M2N1 can be used as an input to M3N1. The output generated by M3N1 can be used as an input to M3N2 together with the output generated based on the output of M2N2. This can continue until M3NN is reached. Additionally or alternatively, the output generated by M3N1 can be used as an input to M4N1. The output generated by M4N1 and the output generated by M3N2 can be applied as input to M4N2. This can continue until M4NN is reached.

[0062] In various embodiments, the output 414 of the classification encoder portion may include OE1-OEN. OE1 may include the output generated by M4N1, OE2 may include the output generated by M4N2, and so on until OEN is reached. In various embodiments, OE1-OEN may be used as input to the multilingual classification classifier portion 123.

[0063] Figure 5 The multilingual classification classifier portion 123 is illustrated. In many embodiments, the classification classifier can include various feed-forward layers and / or pooling layers. For example, the classification classifier can utilize a pre-pooling feed-forward layer 502, a pooling layer 504, and a post-pooling feed-forward layer 506. Additionally or alternatively, a soft maximum layer 508 can combine the outputs of the post-pooling feed-forward layer 506 to generate a classification label 510.

[0064] In various embodiments, the pre-pooling feed-forward layer 502 may include a plurality of feed-forward units F1N1-F1NN. For example, F1N1-F1NN may receive an output 414 generated by a classification encoder, which may include OE1-OEN. F1N1 may receive OE1 as input, F1N2 may receive OE2, and so on, until F1NN receives input OEN. The output generated by F1N1-F1NN may be provided as input to a pooling layer 504, which may include units P1-PN. For example, the output generated by F1N1 may be provided to P1 as input, the output generated by F1N2 may be provided to P2 as input, and so on, until the output generated by F1NN is provided to PN as input. Additionally or alternatively, the output generated by P1-PN may be applied to a post-pooling feed-forward layer 506. For example, the output generated by P1 can be applied as input to F2N1, the output generated by P2 can be applied as input to F2N2, and so on, until the output generated by PN is provided to F2NN as input. In addition, the outputs of F2N1-F2NN can be applied to the soft maximum layer 508. The soft maximum layer 508 can generate a classification label 510.

[0065] Figure 6 is a flowchart illustrating an example process 600 for training a multilingual NMT model according to various embodiments disclosed herein. For convenience, the system performing the operations is described with reference to Figure 6 The system may include various components of various computer systems, such as one or more components of client device 102. In addition, although the operations of process 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0066] At box 602, the system generates a multilingual NMT training instance in at least a first language and a second language. In many embodiments, the multilingual NMT training instance may include at least one pair of phrases and / or sentences, including a phrase and / or sentence in a first language and a known translation of the phrase and / or sentence in a second language. Additionally or alternatively, the multilingual NMT training instance may include at least one pair of phrases and / or sentences, including a phrase and / or sentence in a second language and a known translation of the phrase and / or sentence in the first language. In various embodiments, the multilingual training instance may be generated in other languages.

[0067] At block 604 , the system selects the multilingual NMT training instance generated at block 602 .

[0068] At block 606, the system applies the multilingual NMT training examples as input to the multilingual NMT model to generate multilingual NMT outputs. In many implementations, the multilingual NMT model may include, for example, a shared NMT encoder portion and separate NMT decoder portions with corresponding attention mechanisms for the languages ​​into which the multilingual NMT model can translate.

[0069] At block 608, the system updates the multilingual NMT model based on the output generated by the multilingual NMT training instance and the known output of the multilingual NMT training instance. For example, the system can determine an error based on the generated multilingual NMT output and the known output of the NMT training instance, and back-propagate the error on the neural network model to update the weights of the multilingual NMT model.

[0070] At block 610, the system determines whether there are one or more additional unprocessed multilingual NMT training instances. If so, the system proceeds back to block 604, selects additional multilingual NMT training instances, and then performs blocks 606 and 608 based on the additional unprocessed multilingual NMT training instances. In some embodiments, at block 610, if one or more training criteria have been met (e.g., a threshold number of epochs have occurred, a threshold training duration has occurred, and / or all multilingual NMT training instances have been processed), the system may determine not to process any additional unprocessed multilingual NMT training instances. Although process 600 is described with respect to non-batch learning techniques, batch learning may be utilized additionally and / or alternatively.

[0071] The encoder portion of a multilingual NMT neural network model trained according to process 600 may thereafter be reused to initialize the encoder portion of an additional neural network model, such as a multilingual classification model.

[0072] Figure 7 is a flowchart illustrating an example process 700 for training a multilingual classification model according to various embodiments disclosed herein. For convenience, the system described with reference to performing operations Figure 7 The system may include various components of various computer systems, such as one or more components of client device 102. In addition, although the operations of process 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0073] At block 702, the system generates multilingual classification training examples. The multilingual classification training examples in various embodiments may be generated in various ways including subblock 704 and / or subblock 706. At block 704, the system uses a first number of training examples in a second language and a second number of training examples in a first language, wherein input to a trained multilingual classification model is generated in the first language. For example, the system may learn with a small number of samples and increase classification label accuracy by using a small number of training examples in the first language compared to the number of training examples in the second language used.

[0074] At block 706, the system uses a number of training examples in the second language and no training examples in the first language, wherein the input to the trained multilingual classification model is in the first language. For example, the system can utilize zero-shot learning and can use the multilingual classification model trained only in the second language to classify input received in the first language by utilizing the language agnostic representations in the reused NMT encoder portion.

[0075] At block 708 , the system initializes an encoder portion of a multilingual classification model by reusing an encoder portion of a multilingual NMT model, wherein the multilingual NMT model is trained using at least a first language and a second language.

[0076] At block 710 , the system selects one of the multilingual classification training instances generated at block 702 .

[0077] At block 712 , the system applies the multilingual classification training examples as input to the multilingual classification model to generate a predicted classification output.

[0078] At block 714, the system updates the multilingual classification model based on the predicted classification output and the known output of the multilingual classification training instance. For example, the system can determine an error based on the predicted classification output and the known output, and back-propagate the error on the neural network model to update the weights of the multilingual classification model. In various embodiments, the multilingual classification model updates only the weights in the classification classifier portion, while leaving the weights in the classification encoder portion unchanged (i.e., leaving the weights in the classification encoder portion frozen after initializing them by reusing the weights from the trained NMT encoder portion).

[0079] At box 716, the system determines whether there are one or more other unprocessed multilingual classification training instances. If so, the system proceeds back to box 710, selects another multilingual classification training instance, and then performs boxes 712 and 714 based on the other unprocessed multilingual classification training instances. In some embodiments, at box 716, if one or more training criteria have been met (e.g., a threshold number of cycles have occurred, a threshold training duration has occurred, and / or all multilingual classification training instances have been processed), the system may determine not to process any other unprocessed multilingual classification training instances. Although process 700 is described with respect to non-batch learning techniques, batch learning may be utilized additionally and / or alternatively. The multilingual classification model trained according to process 700 may thereafter be reused to predict classification labels for future NL data streams for the trained NLP task.

[0080] Figure 8 is a flowchart illustrating an example process 800 for performing an NLP classification task using a trained multilingual classification model according to various embodiments disclosed herein. For convenience, the system performing the operation is described with reference to Figure 8 The system may include various components of various computer systems, such as one or more components of client device 102. In addition, although the operations of process 800 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0081] At block 802, the system receives an NL data stream in a first language. For example, a user may provide voice input to a client device, and the client device may use a voice capture / STT / TTS module to capture the spoken input and convert it to a text form, such as an NL data stream.

[0082] At block 804, the system processes the NL data stream using an encoder portion of a multilingual classification model trained using a second language to generate an encoder output, wherein the encoder portion is initialized by reusing an encoder portion of an NMT model trained using at least the first language and the second language. In many implementations, the encoder portion of the NMT model can have the same layers as the encoder portion of the classification model, which can facilitate reusing weights in the NMT encoder portion to initialize weights in the classification encoder portion. Figure 4 FIG. 4 shows a classification encoder portion utilizing a recurrent network layer according to many embodiments.

[0083] At block 806, the system generates a classification label prediction by processing the encoder output using a classifier portion of the multilingual classification model. The classification classifier portion may include various network layers, including feed-forward layers, pooling layers, soft-max layers, etc., to generate a classification label for the input provided by the classification encoder portion. Figure 5 A multilingual classification classifier portion according to many embodiments described herein is shown in FIG.

[0084] At block 808, the system presents the output generated using the classification label prediction to the user. For example, the system can convert the classification label prediction into a verbal output using a voice capture / STT / TTS module and provide the verbal output to the user via a speaker associated with the client device. Additionally or alternatively, the client device can utilize the classification label prediction to send a control signal to one or more devices connected to the client device, for example, combining multiple control commands to control a smart TV by determining that the second command has a classification label implied by the first command.

[0085] Fig. 9 910, which may optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device and / or (one or more) other components may include one or more components of the example computing device 910.

[0086] The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 including, for example, a memory subsystem 925 and a file storage subsystem 926; a user interface output device 920; a user interface input device 922, and a network interface subsystem 916. The input and output devices allow interaction with a user of the computing device 910. The network interface subsystem 916 provides an interface to an external network and is connected to corresponding interface devices in other computing devices.

[0087] User interface input device 922 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into a display; an audio input device such as a voice recognition system, a microphone; and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 910 or over a communication network.

[0088] User interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube ("CRT"); a flat panel device such as a liquid crystal display ("LCD"); a projection device or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from computing device 910 to a user or another machine or computing device.

[0089] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include an executable Figures 6 to 8 Selected aspects of one or more processes in Figure 1 The logic of the various components is depicted in .

[0090] These software modules are typically executed by the processor 914 alone or in conjunction with other processors. The memory 925 used in the storage subsystem 924 may include multiple memories, including a main random access memory ("RAM") 930 for storing instructions and data during program execution and a read-only memory ("ROM") 932 in which fixed instructions are stored. The file storage subsystem 926 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 926 in the storage subsystem 924, or in other machines accessible to the processor(s) 914.

[0091] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0092] Computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Fig. 9 The description of computing device 910 depicted in FIG. 1 is intended only as a specific example for the purpose of illustrating some embodiments. Fig. 9 Many other configurations of computing device 910 are possible with more or fewer components than the computing device depicted.

[0093] Although several embodiments have been described and shown herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or the application for which the teachings are used. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from the embodiments specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors for performing a classification task, comprising: receiving a natural language data stream in a first language, the natural language data stream generated based on user input via a user interface of a client device; Generating a classification label prediction for the natural language data stream in the first language, wherein generating the classification label prediction comprises: processing the natural language data stream using an encoder portion of a multilingual classification model to generate an encoder output, and processing the encoder output using a classifier portion of the multilingual classification model to generate classification label predictions for a natural language data stream in the first language; wherein the multilingual classification model is a trained neural network model, wherein the encoder portion is a reused multilingual neural machine translation encoder trained using a plurality of languages ​​including the first language and at least a second language, and wherein the multilingual classification model is trained for a classification task using supervised training examples in at least the second language, the supervised training examples in the second language each having a corresponding second language training example input in the second language; generating an output using classification label prediction of the stream of natural language data in the first language; and The client device is caused to render the output for presentation to a user.

2. The method according to claim 1, wherein: The multilingual classification model is trained for the classification task without using any supervised training examples in the first language having first language training example input in the first language.

3. The method according to claim 1, in, training the multilingual classification model for the classification task using a first number of supervised training examples in a second language, the first number of supervised training examples in a second language each having a corresponding second language training example input in the second language; wherein the multilingual classification model is trained for the classification task using a second number of supervised training examples in the first language, the second number of supervised training examples in the first language each having a corresponding first language training example input in the first language; and Wherein, the second number is less than fifty percent of the first number.

4. The method according to claim 3, wherein: The second number is smaller than ten percent of the first number, or smaller than five percent of the first number.

5. The method according to claim 1, wherein: The multilingual neural machine translation encoder is trained to translate from at least the first language to at least the second language and / or from at least the second language to at least the first language.

6. The method according to claim 1, wherein: Training the multilingual classification model using supervised training examples in the second language includes: processing second language training example inputs using the encoder portion and the classifier portion to generate predicted outputs, and updating weights of the classifier portion based on an error without updating any weights of the encoder portion that are reused, wherein the error is determined based on the predicted output and based on labels of the supervised training examples in the second language.

7. The method according to claim 1, wherein: Processing the natural language data stream using the encoder portion to generate the encoder output comprises: Processing the natural language data stream using a bidirectional recurrent network layer of the encoder portion to generate a bidirectional recurrent output; and The bidirectional recurrent output is processed using one or more unidirectional recurrent network layers of the encoder portion, wherein the encoder output includes one or more hidden states of a last unidirectional recurrent network layer of the one or more unidirectional recurrent network layers.

8. The method according to claim 1, wherein: Processing the encoder output using the classifier portion to generate the classification label prediction comprises: processing the encoder output using a pooling layer of the classifier portion to generate a pooled output; and The pooled output is processed using one or more soft max layers of the classifier portion to generate the class label prediction.

9. The method according to claim 8, wherein: Processing the encoder output using the pooling layer to generate the classification label prediction includes: Processing the encoder output using a pre-pooled feed-forward layer in the pooling layer to generate a pre-pooled output; Processing the pre-pooled output using a pooling layer in the pooling layer to generate a pooled output; Processing the pooled output using a post-pooled feed-forward layer in the pooling layer to generate a post-pooled output; and The post-pooling output is processed using the one or more soft max layers to generate the classification label prediction for the natural language data stream in the first language.

10. The method according to claim 1, wherein: The multiple languages ​​include at least the first language, the second language and a third language.

11. The method according to claim 1, wherein: The user interface input is a verbal input, and the method further comprises: The natural language data stream is generated by converting the spoken input into text using a speech-to-text processor.

12. The method according to claim 1, wherein: The classification task is a natural language inference classification task, and wherein the classification label prediction indicates whether the natural language data stream is an entailment, is neutral, or is a contradiction.

13. The method according to claim 12, wherein: The classification label prediction indicates that the natural language data stream is an entailment, and wherein generating the output using the classification label prediction comprises: In response to indicating that the natural language data stream is entailing a classification label prediction: In generating the output, an immediately preceding natural language data stream is used in conjunction with the natural language data stream.

14. The method according to claim 13, wherein: The natural language data stream and the preceding natural language data stream are each generated based on a corresponding spoken input.

15. The method according to claim 14, further comprising: using the classification label prediction to determine that the natural language data stream includes a request to modify a state of a networked hardware device; In response to determining that the natural language data stream includes a request to modify a state of the networked hardware device: causing a control signal to be provided to the networked hardware device, wherein the control signal causes a state of the networked hardware device to be modified; Wherein the output perceptibly indicates a modification to a state of the networked hardware device.

16. The method according to claim 15, wherein: The networked hardware device is selected from the group consisting of a light, a thermostat, and a networked camera.

17. The method according to claim 7, wherein: The bidirectional recurrent network layer is a bidirectional long short-term memory LSTM layer or a bidirectional gated recurrent unit GRU layer.

18. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the following method: receiving a natural language data stream in a first language, the natural language data stream generated based on user input via a user interface of a client device; Generating classification label predictions for the natural language data stream in the first language, wherein Generating the classification label prediction includes: processing the natural language data stream using an encoder portion of a multilingual classification model to generate an encoder output, and processing the encoder output using a classifier portion of the multilingual classification model to generate classification label predictions for a natural language data stream in the first language; wherein the multilingual classification model is a trained neural network model, wherein the encoder portion is a reused multilingual neural machine translation encoder trained using a plurality of languages ​​including the first language and at least a second language, and wherein the multilingual classification model is trained for a classification task using supervised training examples in at least the second language, the supervised training examples in the second language each having a corresponding second language training example input in the second language; generating an output using classification label prediction of the stream of natural language data in the first language; and The client device is caused to render the output for presentation to a user.

19. A system for performing a classification task, comprising one or more processors, wherein the one or more processors are configured to perform the following method: receiving a natural language data stream in a first language, the natural language data stream generated based on user input via a user interface of a client device; Generating classification label predictions for the natural language data stream in the first language, wherein Generating the classification label prediction includes: processing the natural language data stream using an encoder portion of a multilingual classification model to generate an encoder output, and processing the encoder output using a classifier portion of the multilingual classification model to generate classification label predictions for a natural language data stream in the first language; wherein the multilingual classification model is a trained neural network model, wherein the encoder portion is a reused multilingual neural machine translation encoder trained using a plurality of languages ​​including the first language and at least a second language, and wherein the multilingual classification model is trained for a classification task using supervised training examples in at least the second language, the supervised training examples in the second language each having a corresponding second language training example input in the second language; generating an output using classification label prediction of the stream of natural language data in the first language; and The client device is caused to render the output for presentation to a user.

Images