Contrastive pre-training for language tasks

Abstract

Systems and methods are provided for training a machine learning language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task in which an encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example, the proposed method masks out a subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a "generator" (which can be a small masked language model, for example), and then trains the encoder to predict whether each token is from the original data or a replacement produced by the generator.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 905,602, filed September 25, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to natural language processing (NLP). More specifically, this disclosure relates to techniques for pre-training machine learning language encoding models. Background Technology

[0004] Early work on pre-trained text encoders used language model objectives. A drawback of these methods is that the resulting model is unidirectional—the model cannot see future tokens when producing the current representation. Therefore, current state-of-the-art pre-training methods primarily rely on masked language models (MLMs). These methods select a small subset of the input (typically around 15%), mask the token identities or attention to these tokens, and then train the model to reconstruct the original input. While this produces a bidirectional model, these objectives incur significant computational costs. As an example, the substantial computational cost can be partly attributed to the fact that the model only learns from 15% of each example token.

[0005] Therefore, while self-supervised pre-training has yielded strong results for many NLP tasks, these methods also require significant computation to be effective, raising concerns about their cost and accessibility. Since more computational pre-training almost always leads to better accuracy, this disclosure recognizes that a crucial consideration for pre-training methods should be computational efficiency rather than absolute downstream accuracy. From this perspective, it is desirable for pre-training algorithms to be significantly more computationally and parametrically efficient. Summary of the Invention

[0006] Various aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0007] One example aspect of the present disclosure relates to a computer-implemented method for training a machine-learned language encoder model. The method can be performed for each of one or more training iterations. The computer-implemented method includes obtaining, by a computing system comprising one or more computing devices, raw language input comprising a plurality of raw input tokens. The computer-implemented method also includes selecting, by the computing system, one or more of the plurality of raw input tokens to serve as one or more masked tokens. The computer-implemented method also includes generating, by the computing system, one or more replacement tokens. The computer-implemented method also includes replacing, by the computing system, the one or more masked tokens in the raw language input with the one or more replacement tokens, respectively, to form a noisy language input comprising a plurality of updated input tokens. For example, the plurality of updated input tokens can comprise a mixture of the one or more replacement tokens and the plurality of raw input tokens that were not selected to serve as masked tokens. The computer-implemented method also includes processing, by the computing system, the noisy language input with the machine-learned language encoder model to produce a plurality of predictions for the plurality of updated input tokens, respectively, where the prediction produced by the machine-learned language encoder model for each updated input token predicts whether such updated input token is one of the raw input tokens or one of the replacement input tokens. The computer-implemented method also includes training, by the computing system, the machine-learned language encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the machine-learned language encoder model. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0008] Example implementations can include one or more of the following features.

[0009] In some implementations, generating, by the computing system, the one or more replacement tokens includes generating, by the computing system, the one or more replacement tokens using a machine-learned language generator model.

[0010] In some implementations, the machine-learned language generator model comprises a masked language model that has been trained to predict the one or more masked tokens.

[0011] In some implementations, the computer-implemented method further includes training, by the computing system, the machine-learned language generator model based at least in part on a second loss function that evaluates a difference between the one or more replacement tokens and the one or more masked tokens.

[0012] In some implementations, the second loss function comprises a maximum likelihood estimation function.

[0013] In some implementations, the method further includes training, by the computing system, the machine-learned language generator model in a reinforcement learning regime based on a second objective function that evaluates predictions made by the machine-learned language encoder model for replacement tokens generated by the machine-learned language generator model.

[0014] In some implementations, the method includes jointly training, by the computing system, the machine-learned language generator model and the machine-learned language encoder model based on a combined loss function that includes a combination of the loss function and a second loss function.

[0015] In some implementations, one or more weights are shared between the machine-learned language generator model and the machine-learned language encoder model. In some implementations, the machine-learned language encoder model includes a Transformer network text encoder.

[0016] In some implementations, the loss function evaluates a replacement token as it is included in the original input tokens when the replacement token is equal to the original token it replaces.

[0017] In some implementations: the one or more training iterations include one or more pre-training iterations; and the method further includes, after the one or more pre-training iterations: performing one or more fine-tuning training iterations in which the machine-learned language encoder model is trained to perform a language task.

[0018] In some implementations, the plurality of original input tokens includes a plurality of original words.

[0019] In some implementations, the method further includes, prior to the one or more training iterations: training, by the computing system, the machine-learned language generator model separately according to a second loss function; and after training, by the computing system, the machine-learned language generator model separately: initializing, by the computing system, the machine-learned language encoder model with weight values based on the machine-learned language generator model.

[0020] In some implementations, generating, by the computing system, the one or more replacement tokens includes sampling, by the computing system, the one or more replacement tokens from a noise distribution.

[0021] Another general aspect includes a computing system including one or more processors. The computing system further includes one or more non-transitory computer-readable media that store a machine-learned language encoder model produced by execution of the method of any of the claims. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods and / or store models produced by the actions of the methods. Other aspects of the disclosure relate to various systems, apparatus, non-transitory computer readable media, user interfaces, and electronic devices.

[0022] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and serve to explain the related principles. BRIEF DESCRIPTION OF DRAWINGS

[0023] A detailed discussion of embodiments of the application in relation to the drawings is presented in the following description with reference to the figures. The discussion and description are not meant to limit the scope of the application.

[0024] Figure 1 Data flow in an example pre-training process according to example embodiments of the present disclosure is depicted.

[0025] Figure 2A A block diagram of an example computing system according to example embodiments of the present disclosure is depicted.

[0026] Figure 2B A block diagram of an example computing device according to example embodiments of the present disclosure is depicted.

[0027] Figure 2C A block diagram of an example computing device according to example embodiments of the present disclosure is depicted.

[0028] Reference numerals repeated in multiple figures are intended to identify identical features in the various implementations.

[0029] DETAILED DESCRIPTION

[0030] SUMMARY

[0031] In general, the present disclosure relates to systems and methods for training machine learning language encoding models through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task in which an encoder learns to distinguish input tokens from plausible substitutes. In some implementations, on each training example, the proposed method masks a subset of the original input tokens (e.g., 15%), replaces the masked tokens with samples from a “generator” (which can be a small masked language model, for example), and then trains the encoder to predict whether each token is from the original data or a replacement produced by the generator. Example experiments contained in U.S. Provisional Patent Application No. 62 / 905,602 show that this task is more sample efficient than a masked language model because the loss comes from all input tokens, not just the masked subset. The proposed method is also more parameter efficient, producing better results when training converges.

[0032] As shown by the example experimental data contained in U.S. Provisional Patent Application No. 62 / 905,602, example models trained by example implementations of the proposed method significantly outperform methods such as BERT and XLNet given the same model size, data, and computation. While the method is particularly beneficial for small models, it can also be used at scale as shown by the example experimental results in U.S. Provisional Patent Application No. 62 / 905,602, which show that example models according to the present disclosure match the performance of the current state-of-the-art pre-trained Transformer (RoBERTa) using less than ¼ of the computation.

[0033] The systems and methods of the present disclosure provide some technical effects and benefits. As one example technical effect and benefit, the systems and methods of the present disclosure enable more efficient training of language encoder models. In particular, compared to existing masked language model techniques, the primary representation learning task is posed on all word pieces, not just a masked subset, which makes it more computationally efficient. Thus, for each training example, the encoder model can learn from 100% of the input word pieces, not just a smaller masked percentage (e.g., ~15%). This enables the model to learn (e.g., converge) faster and through fewer training iterations. Training the model using fewer training iterations conserves computational resources, such as process usage, memory usage, network bandwidth, etc.

[0034] As another example technical effect and benefit, the proposed techniques result in improved model performance. In particular, the proposed techniques address a misalignment introduced in existing masked language model techniques, where the model sees artificial [MASK] word pieces during pre-training but not during fine-tuning / testing. Mitigating this misalignment results in improved model performance (e.g., accuracy).

[0035] Example embodiments of the present disclosure will now be discussed in greater detail.

[0036] Example training process

[0037] Figure 1 Data flow in an example pre-training process of a machine learning language encoder model 12 according to example embodiments of the present disclosure is depicted.

[0038] The process can include obtaining, by a computing system comprising one or more computing devices, an original language input comprising a plurality of original input word pieces 18 (“the,” “artist,” “sold,” “the,” “painting”).

[0039] The process also includes selecting, by the computing system, one or more of the plurality of original input tokens 18 to use as one or more masked tokens. In the illustrated example, the original tokens “the” and “painting” have been selected to use as masked tokens, as indicated by 20a and 20b.

[0040] The process also includes generating, by the computing system, one or more replacement tokens 23a and 23b. The process also includes replacing, by the computing system, the one or more masked tokens 20a and 20b in the original language input with the one or more replacement tokens 23a and 23b, respectively, to form a noisy language input comprising a plurality of updated input tokens 22, where the plurality of updated input tokens 22 includes the one or more replacement tokens 23a and 23b and the plurality of original input tokens 18 that were not selected to use as masked tokens.

[0041] The process also includes processing, by the computing system, the noisy language input with the machine-learned language encoder model 12 to produce a plurality of predictions 24 for the plurality of updated input tokens 22, where the prediction 24 produced by the machine-learned language encoder model for each updated input token 22 predicts whether the updated input token is one of the original input tokens 18 or one of the replacement input tokens 23a and 23b.

[0042] The process also includes training, by the computing system, the machine-learned language encoder model 12 based at least in part on a loss function 26 that evaluates the plurality of predictions 24 produced by the machine-learned language encoder model 12.

[0043] In some implementations, the machine-learned language generator model 14 includes a masked language model that has been trained to predict the one or more masked tokens 20a and 20b.

[0044] In some implementations, the process also includes training, by the computing system, the machine-learned language generator model 14 based at least in part on a second loss function 28 that evaluates a difference between the one or more replacement tokens 23a and 23b and the one or more tokens that were selected to use as masked tokens (e.g., “the” and “painting”).

[0045] In some implementations, the second loss function 28 includes a maximum likelihood estimation function.

[0046] In some other implementations (not shown), the process further includes training a machine learning language generator model 14 by a computational system using a reinforcement learning scheme based on a second objective function 28, which evaluates predictions 24 generated by the machine learning language encoder model 12 for the replacement lexical units 23a and 23b generated by the machine learning language generator model 14. For example, the generator model 14 could be rewarded for "fooling" the encoder model 12.

[0047] In some implementations, the process includes the computational system jointly training the machine learning language generator model 14 and the machine learning language encoder model 12 based on a combined loss function that includes a combination of loss function 26 and a second loss function 28.

[0048] In some implementations, one or more weights are shared between the machine learning language generator model 14 and the machine learning language encoder model 12. In some implementations, the machine learning language encoder model 12 includes a Transformer network text encoder as described by Vaswani et al. in 2017.

[0049] In some implementations, the loss function 26 evaluates a replacement word 23a as if it were included in the original input word 18, when one of the replacement words (e.g., 23a “the”) is equal to the original word it replaces (e.g., “the” from 18). For example, the “true” prediction 24 for “the” 23a is considered correct.

[0050] In some implementations, Figure 1 Following the training process shown, the machine learning language encoder model 12 can be fine-tuned to perform language processing tasks. For example, language processing tasks may include question answering; next word or sentence completion or prediction, translation, entity recognition, language classification, and other language tasks.

[0051] Therefore, the exemplary aspect of this disclosure relates to systems and methods for comparative pre-training. For example... Figure 1 As shown, an example method trains two models (e.g., neural networks), a generator G14 and a discriminator D12. Each can be, or include, for example, an encoder that takes input tokens x = [x1; x2; ...; x...]. n The sequence on [h1; h2; ...; h] is converted into a contextualized vector representation of the sequence h(x) = [h1; h2; ...; h...]. n As an example, each encoder can be or include a Transformer network or other networks that include self-attention.

[0052] In some implementations, for a given position t (e.g., where x...), t= [MASK] position), generator 14 output (e.g., using a softmax layer) is used to generate a specific word x t The probability of:

[0053]

[0054] Where e represents the lexical embedding. For a given position t, discriminator 12 predicts the lexical x. t Whether it is "true" means that it comes from the data distribution rather than the generator distribution (e.g., a noise distribution). An example discriminator is as follows:

[0055] D(x,t)=sigmoid(w T h D (x) t )

[0056] Where w corresponds to the learning weights of the discriminator.

[0057] Generator 14 can be trained to perform a masked language model. Given an input x = [x1; x2; ...; x...] n The masking language model first selects a random set of positions (integers between 1 and n) to mask m = [m1; ...; m...]. k The terminology at the selected position is replaced with the [MASK] terminology: which can be represented as x. masked =REPLACE(x; m; [MASK]). Generator 14 can then learn to maximize the likelihood of the masked lexical. Discriminator 12 can be trained to distinguish lexicals in the data from those sampled from generator 14. More specifically, a "noisy" example x can be created by replacing the masked lexicals 20a and 20b with generator samples. noised 22. Then, discriminator 12 can be trained to predict x. noised Which terms in equation 22 do not match the original input x 18? Formally, example expressions for the input (the first three equations) and the generator and discriminator losses (the last two equations) are:

[0058] m i ~unif{1,n}for i=1 to k

[0059] x masked =REPLACE(x, m, [MASK])

[0060]

[0061]

[0062]

[0063]

[0064] While similar to the training objective of a GAN, there are several key differences. First, if the generator 14 happens to generate a correct token, that token is considered “real” rather than “fake.” This conceit has been found to improve results on downstream tasks. More importantly, in some implementations, the generator 14 is trained with maximum likelihood rather than adversarially to fool the discriminator 12. Training the generator 12 adversarially is challenging because it is not possible to backpropagate through sampling from the generator 14. Finally, the generator 14 takes only context as input, while GANs typically also provide a noise vector to the GAN generator.

[0065] Thus, one example learning objective is to minimize the combined loss on a large corpus of raw text X:

[0066]

[0067] In some implementations, the expected value of the loss can be approximated with a single sample.

[0068] An additional example implementation is as follows.

[0069] Weight sharing: Some example implementations can improve the efficiency of pretraining by sharing weights between the generator and discriminator. If the generator and discriminator are the same size, then the weights of all encoders can be tied, in part or in whole. However, it has been found that having a small generator is particularly effective, in which case, for example, only the token embeddings of the generator and discriminator are shared. In this case, one can use the token embeddings, which are the size of the hidden states of the discriminator, and then one can add a linear projection layer to reduce them to the size of the hidden states of the generator. The “input” and “output” embeddings of the generator can be tied as in BERT. Tying the token embeddings can be beneficial because masked language models are particularly effective at learning token representations: while the discriminator only updates the tokens that are present in the input or sampled by the generator, the generator’s softmax densely updates all token embeddings on the vocabulary.

[0070] Smaller generator: If the generator and discriminator are the same size, training the model typically takes about twice as much computation per step as training with a masked language model alone. Thus, some example implementations use a smaller generator to reduce the computational cost of training the generator. One can make the model smaller by reducing the hidden layer / embedding size while keeping all other hyperparameters the same. Interestingly, even ignoring the computational differences, using a smaller generator is better than a larger one. One possible reason for this behavior is that having a generator that is too strong can make the task too challenging for the discriminator, preventing it from learning effectively.

[0071] Training algorithm: Some example training objectives train the generator and discriminator jointly. Other example training algorithms have a two-stage training process as follows: 1. Train the generator for n steps using L MLM generator. 2. Initialize the weights of the discriminator with the weights of the generator. Then, train the discriminator for n steps using L Disc generator. 2. Initialize the weights of the discriminator with the weights of the generator. Then, train the discriminator for n steps using L

[0072] Note that the weight initialization in this procedure requires that the generator and discriminator have the same size. It has been found that without weight initialization, the discriminator will sometimes fail to learn anything beyond the majority class, possibly because the generator produces such convincing fakes. On the other hand, joint training naturally provides the discriminator with a curriculum in which the generator starts weak but gets better throughout training. Another possible approach is to use reinforcement learning to adversarially train the generator as in GANs to help discrete operations sampled from the generator.

[0073] After training, the proposed model can be used for a number of different tasks. As an example, a classification task such as sentiment analysis can be performed (e.g., by adding a classification layer on top of the model output). Another example task is question answering, in which a system including the model receives a question about a text sequence and is asked to mark the answer in the sequence. In one example, a Q&A model can be trained by learning two additional vectors that mark the beginning and end of the answer. In named entity recognition (NER), a system including the model can receive a text sequence and mark the various types of entities (people, organizations, dates, etc.) that appear in the text. In one example, a NER model can be trained by feeding the output vector for each word token into a classification layer that predicts the NER label. Natural language generation is another example task that can be performed (e.g., to facilitate generating suggested search queries or next word prediction).

[0074] Accordingly, the output of the trained language encoder model can be input to one or more neural network layers to perform a natural language processing task, such as classification, question answering, or natural language generation. The one or more neural network layers can then output a result of the natural language task (e.g., a classification). A natural language model for a particular natural language task can be trained by fine-tuning a pre-trained language encoder model. The parameters of the pre-trained language encoder model can be input to an untrained natural language model (e.g., a classification model) at initialization. The natural language model can then be trained (e.g., using supervised or unsupervised learning) for its particular (downstream) natural language processing task. Accordingly, the pre-trained language encoder model can be used to more easily and more efficiently train a natural language model (e.g., reducing training computation and amount of training data required, and increasing accuracy). Training of the natural language model can train all or some of the parameters of the natural language model. For example, the training can update all parameters, including the parameters of the trained language encoder model, or can only train additional parameters added on top of the trained language encoder model.

[0075] Example devices and systems

[0076] Figure 2A A block diagram of an example computing system 100 in accordance with example embodiments of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled over a network 180.

[0077] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0078] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as, for example, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0079] In some implementations, the user computing device 102 can store or include one or more machine learning models 120. For example, the machine learning models 120 can be or can otherwise include a variety of machine learning models such as neural networks (e.g., deep neural networks) or other types of machine learning models (including non-linear models and / or linear models). The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. See Figure 1 Example machine learning models 120 are discussed.

[0080] In some implementations, the one or more machine learning models 120 can be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine learning model 120 (e.g., multiple instances of a cross-lingual tokenizer perform parallel language encoding / processing).

[0081] Additionally or alternatively, one or more machine learning models 140 can be included in, or stored and implemented by, the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning models 140 can be implemented by the server computing system 140 as part of a web service (e.g., a language processing service service). Thus, one or more models 120 can be stored and implemented at the user computing device 102, and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0082] The user computing device 102 can also include one or more user input components 122 that receive user input. For example, the user input components 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or trackpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can function as a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other means by which a user can provide user input.

[0083] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor cores, microprocessors, ASICS, FPGAs, controllers, microcontrollers, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0084] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0085] As noted above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or can otherwise include a variety of machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. See, e.g., Goodfellow, et al., Deep Learning, MIT Press, 2016, which is incorporated by reference herein in its entirety. Figure 1 Example models 140 are discussed.

[0086] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with a training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130, or can be part of the server computing system 130.

[0087] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor cores, microprocessors, ASICS, FPGAs, controllers, microcontrollers, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0088] The training computing system 150 can include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using a variety of training or learning techniques, such as, for example, error backpropagation. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). A variety of loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or a variety of other loss functions. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations.

[0089] In some implementations, performing backpropagation of errors can include performing truncated backpropagation over time. The model trainer 160 can perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization capabilities of the trained models.

[0090] In particular, the model trainer 160 can train the machine learning models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, a set of example input tokens.

[0091] In some implementations, the training examples can be provided by the user computing device 102 if the user has provided consent. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0092] The model trainer 160 includes computer logic in the form of software or firmware instructions. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device that are loaded into memory and run by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-readable instructions stored in tangible computer-readable storage medium, such as RAM hard disks, or optical or magnetic media.

[0093] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communications over the network 180 can be carried using a variety of communications protocols, such as TCP / IP, HTTP, SMTP, FTP, and the like, encoding or formatting schemes, such as HTML, XML, and the like, and / or protection schemes, such as VPN, secure HTTP, SSL, and the like, via any type of wired and / or wireless connection.

[0094] Figure 2A An example computing system that can be used to implement this disclosure is shown. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include a model trainer 160 and a training dataset 162. In such an implementation, model 120 can be trained and used locally on user computing device 102. In some such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0095] Figure 2B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0096] Computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and (multiple) machine learning models. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0097] like Figure 2B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0098] Figure 2C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0099] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the multiple models stored therein) using an API (e.g., a common API across all applications).

[0100] The central intelligence layer includes several machine learning models. For example, such as... Figure 2CAs shown, a respective machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within, or otherwise implemented by, the operating system of the computing device 50.

[0101] The central intelligence layer can be in communication with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As Figure 2C As shown, the central device data layer can be in communication with some other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can be in communication with each device component using an API (e.g., a private API).

[0102] Additional disclosure

[0103] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0104] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or otherwise causes the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. Implementations of the technology can include hardware, methods or processes, or computer software on a computer-accessible medium.

[0105] While the subject matter has been described in detail with respect to various specific embodiments of it, it should be understood that the embodiments are presented by way of example and for purposes of illustration only. Those skilled in the art will readily recognize variations and modifications of the embodiments as they become apparent in light of the foregoing description. Therefore, it is to be understood that this subject disclosure is not intended to be limited to the specific embodiments described. Rather, it is intended to cover any and all modifications and / or equivalents within the scope of the subject disclosure. For example, features shown or described as part of one embodiment can be used with another embodiment to yield still a further embodiment. It is therefore intended that this subject disclosure be construed as including all such variations and modifications.

Claims

1. A computer-implemented method for training a machine learning language encoder model, the method comprising: For each of one or more training iterations: The raw language input, which includes multiple raw input lexical units, is obtained by a computing system comprising one or more computing devices; The computing system selects one or more of the plurality of original input lexical units to be used as one or more masking lexical units; The computing system generates one or more replacement lexical units, wherein the one or more replacement lexical units include alternative natural language lexical units; The computing system replaces the one or more masked lexical units in the original language input with the one or more replacement lexical units respectively to form a noisy language input including a mixture of the one or more replacement lexical units and the original input lexical units that were not selected as masking lexical units; The computing system processes the noisy language input using the machine learning language encoder model to generate a corresponding prediction for each updated input lexical unit, comprising multiple predictions for the plurality of updated input lexical units, wherein the prediction generated by the machine learning language encoder model for each updated input lexical unit predicts whether such updated input lexical unit is one of the original input lexical units or one of the replacement lexical units; and The machine learning language encoder model is trained by the computing system based at least in part on a loss function that evaluates a plurality of corresponding predictions generated by the machine learning language encoder model.

2. The computer-implemented method according to claim 1, wherein, Generating the one or more replacement lexical units by the computing system includes generating the one or more replacement lexical units by the computing system using a machine learning language generator model.

3. The computer-implemented method according to claim 2, wherein, The machine learning language generator model includes a masked language model that has been trained to predict the one or more masked lexical units.

4. The computer-implemented method according to claim 2 or 3, further comprising: The machine learning language generator model is trained by the computing system based at least in part on a second loss function, which evaluates the difference between the one or more replacement lexical units and one or more original lexical units selected as masking lexical units.

5. The computer-implemented method according to claim 4, wherein, The second loss function includes the maximum likelihood estimation function.

6. The computer-implemented method according to claim 2 or 3, further comprising: The computing system trains the machine learning language generator model using a reinforcement learning scheme based on a second objective function, wherein the second objective function evaluates the predictions generated by the machine learning language encoder model for the replacement lexical units generated by the machine learning language generator model.

7. The computer-implemented method according to claim 4, wherein, The method includes the computing system jointly training the machine learning language generator model and the machine learning language encoder model based on a combined loss function that includes a combination of the loss function and the second loss function.

8. The computer-implemented method according to claim 4, wherein, The method further includes, prior to the one or more training iterations: The machine learning language generator model is trained independently by the computing system based on the second loss function; and After the machine learning language generator model is trained independently by the computing system: The computing system initializes the machine learning language encoder model with weight values ​​based on the machine learning language generator model.

9. The computer-implemented method according to claim 2 or 3, wherein, One or more weights are shared between the machine learning language generator model and the machine learning language encoder model.

10. The computer-implemented method according to claim 1, wherein, Generating the one or more replacement morphemes by the computing system includes sampling the one or more replacement morphemes from a noise distribution by the computing system.

11. The computer-implemented method according to any one of claims 1-3, wherein, The machine learning language encoder model includes the Transformer network text encoder.

12. The computer-implemented method according to any one of claims 1-3, wherein, The loss function evaluates a replacement word as if it were included in the original input word when one of the replacement words is equal to the original word it replaces.

13. The computer-implemented method according to any one of claims 1-3, wherein: The one or more training iterations include one or more pre-training iterations; and The method further includes, after the one or more pre-training iterations: One or more fine-tuning training iterations are performed, wherein the machine learning language encoder model is trained to perform a language task.

14. The computer-implemented method according to any one of claims 1-3, wherein, The multiple original input lexical units include multiple original words.

15. A computing system, comprising: One or more processors; as well as One or more non-transitory computer-readable media storing instructions for performing the method of any one of claims 1-14.

16. The computing system according to claim 15, wherein, The one or more non-transitory computer-readable media also store the machine learning language encoder model.

17. The computing system according to claim 15 or 16, wherein, The one or more non-transitory computer-readable media also store the machine learning language generator model.

18. One or more non-transitory computer-readable media storing instructions for performing the method of any one of claims 1-14.

19. A computing system, comprising: One or more processors; as well as One or more non-transitory computer-readable media storing a machine learning language encoder model generated by performing the method of any one of claims 1-14.

20. One or more non-transitory computer-readable media storing a machine learning language encoder model generated by performing the method of any one of claims 1-14.

Images