Contrasted pre-training for language tasks

Abstract

To provide a technique for prior training of a machine-learned language code model.SOLUTION: In particular, the present disclosure describes a contrast learning task in which the encoder learns to distinguish input tokens from an alternative that seems reasonable. In some implementation aspects, in each training example, the proposed method masks out a subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a "generator" (which may be a small masked language model, for example), then trains the encoder to predict whether each token came from the original data or is a replacement generated by the generator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 905,602, filed September 25, 2019, which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to natural language processing (NLP). More particularly, the present disclosure relates to techniques for pre-training machine-learned language encoding models. [Background technology]

[0003] Early work on pre-training text encoders used language modeling objectives. A drawback of these methods is that the resulting model is unidirectional, meaning the model does not look at future tokens when generating a representation of the current token. Therefore, current state-of-the-art pre-training methods primarily rely on masked language modeling (MLM). These approaches select a small subset of the input (typically around 15%), mask the token identification or attention to those tokens, and then train the model to recover the original input. While resulting in a bidirectional model, these objectives are computationally expensive. As an example, the significant computational cost can be partially attributed to the fact that the model only learns from 15% of the tokens per example.

[0004] Thus, while self-supervised pre-training produces strong results for many NLP tasks, these methods also require significant computation to be effective, raising concerns about cost and accessibility. This disclosure recognizes that because pre-training with more computation almost always results in better accuracy, the key consideration for pre-training methods should be computational efficiency, rather than absolute downstream accuracy. From this perspective, it is desirable for pre-training algorithms to be substantially more computationally efficient and parameter-efficient. Summary of the Invention [Means for solving the problem]

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0006] One exemplary aspect of the present disclosure is directed to a computer-implemented method for training a machine-learned language encoder model. The method can be performed for each of one or more training iterations. The computer-implemented method includes obtaining, by a computing system including one or more computing devices, an original language input including a plurality of original input tokens. The computer-implemented method also includes selecting, by the computing system, one or more of the plurality of original input tokens to serve as the one or more masked tokens. The computer-implemented method also includes generating, by the computing system, one or more replacement tokens. The computer-implemented method also includes replacing, by the computing system, one or more masked tokens in the original language input with the one or more replacement tokens, respectively, to form a noisy language input including a plurality of updated input tokens. For example, the plurality of updated input tokens can include a mixture of one or more replacement tokens and a plurality of original input tokens that were not selected to serve as masked tokens. The computer-implemented method also includes processing, by the computing system, the noisy language input through the machine-learned language encoder model to generate a plurality of predictions for a plurality of updated input tokens, respectively, wherein the prediction generated by the machine-learned language encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens. The computer-implemented method also includes training, by the computing system, the machine-learned language encoder model based at least in part on a loss function that evaluates the plurality of predictions generated by the machine-learned language encoder model.Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] Example implementations may include one or more of the following features.

[0008] In some implementations, generating, by the computing system, one or more replacement tokens includes generating, by the computing system, the one or more replacement tokens using a machine-learned language generator model.

[0009] In some implementations, the machine-learned language generator model includes a masked language model trained to predict one or more masked tokens.

[0010] In some implementations, the computer-implemented method further includes training, by the computing system, a machine-learned language generator model based at least in part on a second loss function that evaluates the difference between the one or more replacement tokens and the one or more masked tokens.

[0011] In some implementations, the second loss function includes a maximum likelihood estimation function.

[0012] In some implementations, the method further includes training, by the computing system, the machine-learned language generator model in a reinforcement learning scheme based on a second objective function that evaluates predictions generated by the machine-learned language encoder model for replacement tokens generated by the machine-learned language generator model.

[0013] In some implementations, the method includes jointly training, by a computing system, the machine-learned language generator model and the machine-learned language encoder model based on a combined loss function that includes a combination of the loss function and a second loss function.

[0014] In some implementations, one or more weights are shared between the machine-learned language generator model and the machine-learned language encoder model. In some implementations, the machine-learned language encoder model includes a Transformer Network Text Encoder.

[0015] In some implementations, when one of the replacement tokens is equal to the original token it replaces, the loss function evaluates such replacement token as if it were included in the original input token.

[0016] In some implementations, the one or more training iterations include one or more pre-training iterations, and the method further includes performing one or more fine-tuning training iterations after the one or more pre-training iterations, in which the machine-learned language encoder model is trained to perform a language task.

[0017] In some implementations, the multiple original input tokens include multiple original words.

[0018] In some implementations, the method further includes, before the one or more training iterations, separately training, by the computing system, the machine-learned language generator model on the second loss function, and after separately training, by the computing system, initializing, by the computing system, the machine-learned language encoder model with weight values ​​based on the machine-learned language generator model.

[0019] In some implementations, generating, by the computing system, the one or more replacement tokens includes sampling, by the computing system, the one or more replacement tokens from a noise distribution.

[0020] Another general aspect includes a computing system including one or more processors. The computing system also includes one or more non-transitory computer-readable media that store the machine-learned language encoder model generated via execution of the method of any one of the claims. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method and / or store the model generated via the actions of the method. Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0021] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0022] A detailed discussion of embodiments, directed to those skilled in the art, is set forth herein, and this specification refers to the accompanying drawings. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a data flow in an exemplary pre-training process according to an exemplary embodiment of the present disclosure. [Figure 2A] FIG. 1 is a block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 2B]FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 2C] FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference numbers that are repeated among the drawings are intended to identify like features in various implementations.

[0025] overview In general, the present disclosure is directed to systems and methods for training machine-learned language encoding models using contrastive learning tasks. In particular, the present disclosure describes a contrastive learning task in which an encoder learns to distinguish input tokens from plausible alternatives. In some implementations, in each training example, the proposed method masks out a small subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a “generator” (which may be, for example, a small masked language model), and then trains the encoder to predict whether each token came from the original data or a replacement generated by the generator. Example experiments included in U.S. Provisional Patent Application No. 62 / 905,602 demonstrate that this task is more sample efficient than masked language modeling because losses arise from all input tokens, not just the masked-out subset. The proposed approach is also more parameter-efficient and produces better results when trained to convergence.

[0026] As shown by the exemplary experimental data contained in U.S. Provisional Patent Application No. 62 / 905,602, exemplary models trained by exemplary implementations of the proposed approach substantially outperform methods such as BERT and XLNet given the same model size, data, and computation. This approach is particularly beneficial for small models, but also works at large scales, as shown by the exemplary experimental results in U.S. Provisional Patent Application No. 62 / 905,602, which show that exemplary models according to the present disclosure match the performance of RoBERTa, the current state-of-the-art pre-trained transformer, while using less than one-quarter of the computation.

[0027] The disclosed system and method provide several technical effects and benefits. As one exemplary technical effect and advantage, the disclosed system and method enable more efficient training of a language encoder model. In particular, compared to existing masked language modeling techniques, the main representation learning task is presented across all tokens, not just a masked-out subset, making it more computationally efficient. Thus, for each training example, the encoder model can learn from 100% of the input tokens, not just a small masked-out percentage (e.g., ~15%). This allows the model to learn (e.g., converge) faster and with fewer training iterations. Using fewer training iterations to train the model saves computing resources, such as process usage, memory usage, and network bandwidth.

[0028] As another exemplary technical effect and advantage, the proposed technique results in improved model performance. In particular, the proposed technique resolves a mismatch introduced in existing masked language modeling techniques, where the model sees the artificial [MASK] token during pre-training, but not during fine-tuning / testing. By mitigating this mismatch, model performance (e.g., accuracy) is improved.

[0029] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in more detail.

[0030] Exemplary Training Process FIG. 1 illustrates data flow in an exemplary pre-training process for a machine-learned language encoder model 12 according to an exemplary embodiment of the present disclosure.

[0031] The process may include obtaining, by a computing system including one or more computing devices, original linguistic input including a plurality of original input tokens 18 (“the,” “artist,” “sold,” “the,” “painting”).

[0032] The process also includes selecting, by the computing system, one or more of the plurality of original input tokens 18 to serve as one or more masked tokens. In the illustrated example, the original tokens "the" and "painting" have been selected to serve as the masked tokens, as shown at 20a and 20b.

[0033] The process also includes generating, by the computing system, one or more replacement tokens 23 a and 23 b. The process also includes replacing, by the computing system, one or more masked tokens 20 a and 20 b in the original linguistic input with the one or more replacement tokens 23 a and 23 b, respectively, to form a noisy linguistic input including a plurality of updated input tokens 22, the plurality of updated input tokens 22 including the one or more replacement tokens 23 a and 23 b and a plurality of original input tokens 18 that were not selected to serve as masked tokens.

[0034] The process also includes processing the noisy language input through the machine-learned language encoder model 12 by the computing system to generate a plurality of predictions 24 for a plurality of updated input tokens 22, respectively, wherein the predictions 24 generated by the machine-learned language encoder model for each updated input token 22 predict whether such updated input token is one of the original input tokens 18 or one of the replacement input tokens 23a and 23b.

[0035] The process also includes training, by the computing system, the machine-learned language encoder model 12 based at least in part on a loss function 26 that evaluates multiple predictions 24 generated by the machine-learned language encoder model 12.

[0036] In some implementations, the machine-learned language generator model 14 includes a masked language model trained to predict one or more masked tokens 20a and 20b.

[0037] In some implementations, the process further includes training, by the computing system, the machine-learned language generator model 14 based at least in part on a second loss function 28 that evaluates the difference between one or more replacement tokens 23a and 23b and one or more tokens selected to serve as masked tokens (e.g., "the" and "painting").

[0038] In some implementations, the second loss function 28 includes a maximum likelihood estimation function.

[0039] In some other implementations (not shown), the process further includes training, by the computing system, the machine-learned language generator model 14 in a reinforcement learning scheme based on a second objective function 28 that evaluates the predictions 24 generated by the machine-learned language encoder model 12 for the replacement tokens 23 a and 23 b generated by the machine-learned language generator model 14. For example, the generator model 14 may be rewarded for “fooling” the encoder model 12.

[0040] In some implementations, the process includes jointly training, by a computing system, the machine-learned language generator model 14 and the machine-learned language encoder model 12 based on a combined loss function that includes a combination of the loss function 26 and the second loss function 28.

[0041] In some implementations, one or more weights are shared between the machine-learned language generator model 14 and the machine-learned language encoder model 12. In some implementations, the machine-learned language encoder model 12 includes a Transformer Network Text Encoder such as that described in Vaswani et al., 2017.

[0042] In some implementations, when one of the replacement tokens (e.g., 23a "the") is equal to the original token it replaces (e.g., "the" from 18), the loss function 26 evaluates such replacement token 23a as if it were included in the original input token 18. For example, the prediction 24 of "real" for "the" 23a is considered correct.

[0043] 1, the machine learning language encoder model 12 can be fine-tuned to perform language processing tasks. By way of example, language processing tasks can include question answering, next word or sentence completion or prediction, translation, entity recognition, language classification, and other language tasks.

[0044] Accordingly, exemplary aspects of the present disclosure are directed to systems and methods for contrastive pre-training. As shown in FIG. 1, one exemplary approach trains two models (e.g., neural networks): a generator G14 and a discriminator D12. Each trains a discriminator D12 based on, for example, input tokens x=[x1;x2;...;x n ]The sequence above can be expressed as a contextualized vector representation h(x)=[h1;h2;...;h n ]. As an example, each encoder can be or include a Transformer network or other network that includes self-attention.

[0045] In some implementations, given a position t (e.g., x t =[MASK]), generator 14 finds the specific token x t output the probability of generating (e.g., using a softmax layer).

number

[0046] The generator 14 can be trained to perform masked language modeling. n ], masked language modeling is first performed by k Select a random set of positions (integers between 1 and n) to mask out the [MASK] token. The tokens in the selected positions are replaced with the [MASK] token, which is masked =REPLACE(x;m;[MASK]). The generator 14 can then learn to maximize the likelihood of the masked-out tokens. The discriminator 12 can be trained to distinguish tokens in the data from tokens sampled from the generator 14. More specifically, by replacing the masked-out tokens 20a and 20b with generator samples, the "noisy" exemplary x noised 22 can be created. Then, the discriminator 12 noised It can be trained to predict which tokens in 22 do not match the original input x18. Formally, example formulas for the input (first three equations) and the generator and discriminator losses (last two equations) are: For i=1~k, m i ~unif{1,n} x masked =REPLACE(x,m,[MASK])

[0047] For i∈m

number

number

[0048] Thus, one example training objective is to minimize the combined loss over a large corpus X of raw text.

number

[0049] A further exemplary implementation is as follows:

[0050] Weight Sharing: Some example implementations can improve the efficiency of pre-training by sharing weights between the generator and the discriminator. If the generator and discriminator are the same size, some or all of the encoder weights can be tied. However, it has been found to be particularly efficient to have a small generator, in which case, for example, only the token embeddings of the generator and the discriminator are shared. In this case, a token embedding that is the size of the discriminator's hidden state can be used, and then a linear projection layer can be added to reduce it to the hidden state size of the generator. The embeddings of the generator's "input" and "output" can be tied, as in BERT. Tied token embeddings can be beneficial because masked language modeling is particularly effective at learning token representations; the discriminator only updates tokens that are present in the input or sampled by the generator, while the generator's softmax over the vocabulary densely updates all token embeddings.

[0051] Small Generator: When the generator and discriminator are the same size, training a model typically requires approximately twice as much computation per step as training with masked language modeling alone. Therefore, some example implementations use a smaller generator to reduce the computational cost of training the generator. A model can be made smaller by reducing the hidden layer / embedding size while keeping all other hyperparameters constant. Interestingly, using a smaller generator can perform better than a larger generator, even ignoring the computational difference. One possible reason for this behavior is that a generator that is too strong can pose a very difficult task for the discriminator, preventing it from learning effectively.

[0052] Training Algorithm: Some exemplary training objectives involve joint training of the generator and discriminator. Another exemplary training algorithm has the following two-stage training process: 1. Train only the generator as L MLM 2. Initialize the discriminator weights with the generator weights. Then, while keeping the generator weights frozen, train L Disc The discriminator is trained for n steps by

[0053] Note that the weight initialization in this procedure requires the generator and discriminator to have the same size. Without weight initialization, we know that the discriminator may not be able to learn beyond the majority of classes, likely because the generator produced such convincing fakes. Joint training, on the other hand, naturally provides a curriculum for the discriminator, where the generator is initially weak and improves through training. Another possible approach is to train the generator adversarially, using reinforcement learning to adapt to the discrete sampling behavior from the generator, as in GANs.

[0054] After training, the proposed model can be used for a number of different tasks. For example, classification tasks such as sentiment analysis can be performed (e.g., by adding a classification layer on top of the model's output). Another exemplary task is question answering, in which a system including the model receives a question about a text sequence and is required to mark answers within the sequence. In one example, a Q&A model can be trained by learning two extra vectors that mark the beginning and end of an answer. In named entity recognition (NER), a system including the model can receive a text sequence and mark various types of entities (e.g., people, organizations, dates) that appear in the text. In one example, an NER model can be trained by feeding the output vector for each token into a classification layer that predicts a NER label. Natural language generation is another exemplary task that can be performed (e.g., to facilitate the generation of suggested search queries or next word predictions).

[0055] Thus, to perform natural language processing tasks such as classification, question answering, or natural language generation, the output of the trained language encoder model can be input to one or more neural network layers. The one or more neural network layers can then output the results of the natural language task (e.g., classification). A natural language model for a specific natural language task can be trained by fine-tuning a pre-trained language encoder model. Parameters of the pre-trained language encoder model can be input to an untrained natural language model (e.g., a classification model) at initialization. The natural language model can then be trained (e.g., using supervised or unsupervised learning) for that specific (downstream) natural language processing task. Thus, the pre-trained language encoder model can be leveraged to more easily and efficiently train the natural language model (e.g., reduce training computations and the amount of training data required, and improve accuracy). Training the natural language model can train all or some of the parameters of the natural language model. For example, the training can update all parameters, including those of the trained language encoder model, or can train only additional parameters that are added on top of the trained language encoder model.

[0056] Exemplary Devices and Systems 2A illustrates a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0057] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0058] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0059] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. An exemplary machine-learned model 120 is discussed with reference to FIG. 1.

[0060] In some implementations, one or more machine-learned models 120 may be received from server computing system 130 over network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel linguistic encoding / processing across multiple instances of linguistic tokens).

[0061] Additionally or alternatively, one or more machine-learned models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a language processing service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0062] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0063] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 134 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0064] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0065] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to FIG. 1.

[0066] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0067] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 154 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0068] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters for several training iterations.

[0069] In some implementations, performing backpropagation may include performing abbreviated temporal backpropagation. The model trainer 160 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0070] In particular, model trainer 160 can train machine-learned models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, a set of example input tokens.

[0071] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0072] Model trainer 160 includes computer logic used to provide the desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0073] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communications over network 180 may be carried over any type of wired and / or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0074] 2A illustrates one exemplary computing system that may be used to implement the present disclosure. Other computing systems may be used as well. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0075] 2B illustrates a block diagram of an exemplary computing device 10 for performing according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0076] The computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned model. For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0077] 2B , each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0078] 2C illustrates a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0079] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0080] The central intelligence layer includes several machine-learned models. For example, as shown in FIG. 2C , a respective machine-learned model (e.g., model) may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., single model) to all of the applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0081] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 2C , the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0082] Additional Disclosures The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0083] One or more computer systems may be configured to perform particular operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, during operation, causes the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0084] While the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation, not limitation, of the disclosure. Those skilled in the art, upon understanding the foregoing, may readily conceive of modifications, variations, and equivalents to such embodiments. Accordingly, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet a further embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0085] 10. Computing Devices 12 Machine-learned language encoder models 14 Machine-learned language generator models 18 Original Input Token 20a Masked Tokens 20b Masked Tokens 22 Updated Input Token 23a Replacement Tokens 23b Replacement Tokens 24 Predictions 26 Loss Function 28 Second loss function 50 computing devices 100 Computing Systems 102 User Computing Devices 112 processors 114 memory 116 Data 118 Command 120 machine learning models 122 User Input Components 130 Server Computing System 132 processors 134 memory 136 Data 138 Command 140 machine learning models 150 Training Computing System 152 processors 154 memory 156 Data 158 Command 160 Model Trainer 162 training data sets 180 Network

Claims

1. 1. A computer-implemented method for performing inference with a machine-learned language encoder model, comprising: processing an input with the machine-learned language encoder model to generate an output; The machine-learned language encoder model is pre-trained for one or more training iterations, each of the one or more training iterations comprising: obtaining an original linguistic input comprising a plurality of original input tokens; selecting one or more of the plurality of original input tokens to act as one or more masked tokens; generating one or more replacement tokens, the one or more replacement tokens comprising alternative natural language tokens; replacing the one or more masked tokens in the original language input with the one or more replacement tokens, respectively, to form a noisy language input comprising an updated plurality of input tokens, the updated plurality of input tokens comprising a mixture of the one or more replacement tokens and the original input tokens that were not selected to act as masked tokens; processing the noisy language input with the machine-learned language encoder model without the original language input replaced by the one or more masked tokens to generate a respective prediction for each updated input token in the updated plurality of input tokens, wherein for each updated input token, the prediction generated by the machine-learned language encoder model predicts whether such updated input token is one of the original input tokens or one of the replaced tokens; training a machine-learned language encoder model based at least in part on a loss function that evaluates a plurality of the predictions produced by the machine-learned language encoder model; 4. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein generating the one or more replacement tokens comprises using a machine-learned language generator model to generate the one or more replacement tokens.

3. 3. The computer-implemented method of claim 2, wherein the machine-learned language generator model comprises a masked language model trained to predict the one or more masked tokens.

4. Each training iteration training the machine-learned language generator model based at least in part on a second loss function that evaluates a difference between the one or more replaced tokens and the one or more original tokens selected to serve as masked tokens. The computer-implemented method of claim 2 , further comprising:

5. 5. The computer-implemented method of claim 4, wherein the second loss function comprises a maximum likelihood estimation function.

6. 5. The computer-implemented method of claim 4, wherein the machine-learned language generator model and the machine-learned language encoder model are jointly trained based on a combined loss function that includes a combination of the loss function and the second loss function.

7. prior to the one or more training iterations, the machine-learned language generator model is trained with the second loss function; 5. The computer-implemented method of claim 4, wherein after separately training the machine-learned language generator model, the machine-learned language encoder model is initialized with weight values ​​that are based on the machine-learned language generator model.

8. 3. The computer-implemented method of claim 2, wherein the machine-learned language generator model is trained in a reinforcement learning scheme based on a second objective function that evaluates the predictions generated by the machine-learned language encoder model on the replacement tokens generated by the machine-learned language generator model.

9. 3. The computer-implemented method of claim 2, wherein one or more weights are shared between the machine-learned language generator model and the machine-learned language encoder model.

10. 2. The computer-implemented method of claim 1, wherein generating the one or more replacement tokens comprises sampling the one or more replacement tokens from a noise distribution.

11. 2. The computer-implemented method of claim 1, wherein the machine-learned language encoder model comprises a Transformer Network Text Encoder.

12. 2. The computer-implemented method of claim 1, wherein when one of the substitution tokens is equal to the original token it replaces, the loss function evaluates such substitution token as if it were contained in the original input token.

13. the one or more training iterations include one or more pre-training iterations; 2. The computer-implemented method of claim 1, further comprising performing one or more fine-tuning training iterations after the one or more pre-training iterations, wherein the one or more fine-tuning training iterations train the machine-learned language encoder model to perform a specific language task.

14. The computer-implemented method of claim 1 , wherein the plurality of original input tokens comprises a plurality of original words.

15. A computing system comprising: one or more processors; one or more non-transitory computer-readable media recording a machine-learned language encoder model and instructions that, when executed by the one or more processors, cause the computing system to perform one or more fine-tuning training iterations; Equipped with The machine-learned language encoder model is pre-trained by performing a pre-training operation, the pre-training operation comprising: obtaining an original linguistic input comprising a plurality of original input tokens; selecting one or more of the plurality of original input tokens to act as one or more masked tokens; generating one or more replacement tokens, the one or more replacement tokens comprising alternative natural language tokens; replacing the one or more masked tokens in the original language input with the one or more replacement tokens, respectively, to form a noisy language input comprising an updated plurality of input tokens, the updated plurality of input tokens comprising a mixture of the one or more replacement tokens and the original input tokens that were not selected to act as masked tokens; processing the noisy language input with the machine-learned language encoder model without the original language input replaced by the one or more masked tokens to generate a respective prediction for each updated input token in the updated plurality of input tokens, wherein for each updated input token, the prediction generated by the machine-learned language encoder model predicts whether such updated input token is one of the original input tokens or one of the replaced tokens; training a machine-learned language encoder model based at least in part on a loss function that evaluates a plurality of the predictions produced by the machine-learned language encoder model; Including, A computing system, wherein in the one or more fine-tuning training iterations, the machine-learned language encoder model is trained to perform a specific language task.

16. 20. The computing system of claim 15, wherein the one or more non-transitory computer-readable media further store instructions for performing the pre-training operation.

17. 20. The computing system of claim 15, wherein the one or more non-transitory computer readable media further record the machine-learned language generator model.

18. One or more non-transitory computer-readable media storing a machine-learned language encoder model, The machine-learned language encoder model is trained by performing operations, the operations comprising: obtaining an original linguistic input comprising a plurality of original input tokens; selecting one or more of the plurality of original input tokens to act as one or more masked tokens; generating one or more replacement tokens, the one or more replacement tokens comprising alternative natural language tokens; replacing the one or more masked tokens in the original language input with the one or more replacement tokens, respectively, to form a noisy language input comprising an updated plurality of input tokens, the updated plurality of input tokens comprising the one or more replacement tokens and the plurality of original input tokens that were not selected to act as masked tokens; processing the noisy language input with the machine-learned language encoder model without the original language input replaced by the one or more masked tokens to generate a respective prediction for each updated input token in the updated plurality of input tokens, wherein for each updated input token, the prediction generated by the machine-learned language encoder model predicts whether such updated input token is one of the original input tokens or one of the replaced tokens; training a machine-learned language encoder model based at least in part on a loss function that evaluates a plurality of the predictions produced by the machine-learned language encoder model; [0023] In one or more non-transitory computer readable media,

19. 20. The one or more non-transitory computer-readable media of claim 18, wherein generating the one or more replacement tokens comprises using a machine-learned language generator model to generate the one or more replacement tokens.

20. 20. The one or more non-transitory computer-readable media of claim 18, wherein the machine-learned language generator model comprises a masked language model trained to predict the one or more masked tokens.

Images