Adversarial pretraining of machine learning models
Patent Information
- Application Number
- IN202217057400
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-04-17
- Filing Date
- 2022-10-07
- Publication Date
- 2026-10-01
- Estimated Expiration
- 2041-02-19
AI Technical Summary
Conventional machine learning models are susceptible to adversarial attacks and tend to compromise generalization when using adversarial training, which hinders their ability to perform well on new datasets.
The implementation of virtual adversarial pretraining for mapping layers of machine learning models, followed by task-specific tuning using supervised learning, to enhance robustness and generalization by introducing noise-adjusted representations during the pretraining stage.
This approach results in models that are both robust to adversarial attacks and capable of generalizing well on new data sets, while reducing computational expenses associated with adversarial pretraining.
Abstract
Description
BACKGROUND
[0001] Machine learning can be used to perform a broad range of tasks, such as naturallanguage processing, financial analysis, and image processing. Machine learning modelscan be trained using several approaches, such as supervised learning, semi-supervisedlearning, unsupervised learning, reinforcement learning, etc. In approaches such assupervised or semi-supervised learning, labeled training examples can be used to train amodel to map inputs to outputs.SUMMARY
[0002] This Summary is provided to introduce a selection of concepts in a simplifiedform that are further described below in the Detailed Description. This Summary is notintended to identify key features or essential features of the claimed subject matter, nor is itintended to be used to limit the scope of the claimed subject matter.
[0003] The description generally relates to techniques for training of machine learningmodels. One example includes a method or technique that can be performed on a computingdevice. The method or technique can include providing a machine learning model havingone or more mapping layers, including at least a first mapping layer configured to mapcomponents of pretraining examples into first representations in a space. The method ortechnique can also include performing a pretraining stage on the one or more mapping morelayers using the pretraining examples. The pretraining stage can include adding noise to thefirst representations of the components of the pretraining examples to obtain noise-adjustedfirst representations. The pretraining stage can also include performing a self-supervisedlearning process to pretrain the one or more mapping layers using at least the firstrepresentations and the noise-adjusted first representations of the components of the trainingdata items.
[0004] Another example includes a system having a hardware processing unit and astorage resource storing computer-readable instructions. When executed by the hardwareprocessing unit, the computer-readable instructions can cause the hardware processing unitto receive input data and process the input data using a machine learning model having afirst layer and a second layer to obtain a result. The first layer can have been pretrained ina pretraining stage using virtual adversarial training for a self-supervised learning task. Thecomputer-readable instructions can also cause the hardware processing unit to output theresult.
[0005] Another example includes a computer-readable storage medium storinginstructions which, when executed by one or more processing devices, cause the one or moreprocessing devices to perform acts. The acts can include providing a machine learningmodel having one or more mapping layers, including at least a first mapping layerconfigured to map components of data items into first representations in a space. The actscan also include performing one or more initial pretraining iterations of a learning processto train the machine learning model with the first representations. The acts can also includeperforming noise adjustment on the first representations to obtain noise-adjusted firstrepresentations. The acts can also include performing one or more subsequent pretrainingiterations of the learning process to train the machine learning model with the noise-adjustedfirst representations.BRIEF DESCRIPTION OF THE DRAWINGS
[0006] The Detailed Description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference number identifies the figure inwhich the reference number first appears. The use of similar reference numbers in differentinstances in the description and the figures may indicate similar or identical items.
[0007] FIG. 1 illustrates an example machine learning model for pretraining, consistentwith some implementations of the present concepts.
[0008] FIG. 2 illustrates an example machine learning model for tuning, consistent withsome implementations of the present concepts.
[0009] FIG. 3 illustrates an example training workflow for training a machine learningmodel, consistent with some implementations of the present concepts.
[0010] FIG. 4 illustrates an example machine learning model for performing predictionsat runtime, consistent with some implementations of the present concepts.
[0011] FIGS. 5A-5G illustrate examples of experimental results obtained by machinelearning models produced using the disclosed implementations.
[0012] FIG. 6 illustrates an example system, consistent with some implementations ofthe present concepts.
[0013] FIG. 7 illustrates an example method or technique for pretraining, tuning, and / oremploying a machine learning model, consistent with some implementations of the presentconcepts.
[0014] FIGS. 8, 9, 10, and 11 illustrate example user experiences and user interfaces,consistent with some implementations of the present concepts.DETAILED DESCRIPTION
[0015] There are various types of machine learning frameworks that can be trained toperform a given task. Support vector machines, decision trees, and neural networks are justa few examples of machine learning frameworks that have been used in a wide variety ofapplications, such as image processing and natural language processing. Some machinelearning frameworks, such as neural networks, use layers of nodes that perform specificoperations.
[0016] In a neural network, nodes are connected to one another via one or more edges.A neural network can include an input layer, an output layer, and one or more intermediatelayers. Individual nodes can process their respective inputs according to a predefinedfunction, and provide an output to a subsequent layer, or, in some cases, a previous layer.The inputs to a given node can be multiplied by a corresponding weight value for an edgebetween the input and the node. In addition, nodes can have individual bias values that arealso used to produce outputs. Various training procedures can be applied to learn the edgeweights and / or bias values. The term "parameters" when used without a modifier is usedherein to refer to learnable values such as edge weights and bias values that can be learnedby training a machine learning model, such as a neural network.
[0017] A neural network structure can have different layers that perform differentspecific functions. For example, one or more layers of nodes can collectively perform aspecific operation, such as pooling, encoding, or convolution operations. For the purposesof this document, the term "layer" refers to a group of nodes that share inputs and outputs,e.g., to or from external sources or other layers in the network. The term "operation" refersto a function that can be performed by one or more layers of nodes. The term "modelstructure" refers to an overall architecture of a layered model, including the number oflayers, the connectivity of the layers, and the type of operations performed by individuallayers. The term "neural network structure" refers to the model structure of a neuralnetwork. The term "trained model" and / or "tuned model" refers to a model structuretogether with parameters for the model structure that have been trained or tuned. Note thattwo trained models can share the same model structure and yet have different values for theparameters, e.g, if the two models are trained on different training data or if there areunderlying stochastic processes in the training process.
[0018] There are many machine learning tasks for which there is a relative lack oftraining data. One broad approach to training a model with limited task-specific trainingdata for a particular task involves "transfer learning." In transfer learning, a model is firstpretrained on another task for which significant training data is available, and then the modelis tuned to the particular task using the task-specific training data.
[0019] The term "pretraining," as used herein, refers to model training on a set ofpretraining data to adjust model parameters in a manner that allows for subsequent tuningof those model parameters for one or more specific tasks. In some cases, the pretrainingcan involve a self-supervised learning process on unlabeled training data, where a "self-supervised" learning process involves learning from the structure of pretraining examples,potentially in the absence of explicit (e.g, manually-provided) labels. Subsequentmodification of model parameters obtained by pretraining is referred to herein as "tuning."Tuning can be performed for one or more tasks using supervised learning from explicitly-labeled training data, in some cases using a different task for tuning than for pretraining.
[0020] Conventional techniques for training and tuning of model parameters can besusceptible to certain attacks. For instance, a carefully-constructed or "adversarial"sentence or image may "fool" a model into outputting a clearly incorrect classification forthat sentence or image, even when the correct classification is readily apparent to a humanuser. One broad approach to developing models that are robust to such attacks is known as"adversarial" training. However, conventional adversarial training tends to hurt the abilityof models to generalize, as discussed more below.
[0021] The disclosed implementations offer a mechanism for virtual adversarialpretraining of one or more mapping layers of a model. After pretraining using the disclosedtechniques, the pretrained mapping layers can be tuned with a task-specific layer to performa specific task using supervised learning. By pretraining a model the disclosed techniques,the final model may tend to both generalize well and also be robust to adversarial attacks,as discussed more below.
[0022] The following discussion provides some specific examples in a natural languageprocessing context. However, as also discussed further below, the disclosed techniques canalso be employed for other tasks besides natural language processing, such as imageprocessing, computer vision, radar, sonar, medical scenarios, analysis of genetic markers,etc.PRETRAINING MODEL EXAMPLE
[0023] FIG. 1 illustrates a first natural language processing model 100 that shows howlayers of a machine learning model can be pretrained using the disclosed implementations.Natural language processing model 100 is an example of a machine learning model that canbe used to perform one or more natural language processing tasks, as discussed more below.For the purposes of this document, the term "natural language" means language that isnormally used by human beings for writing or conversation.
[0024] Natural language processing model 100 can receive pretraining examples 102,which can include documents, sentences, phrases, or other representations of languagehaving various components, such as words and / or tokens. The components of thepretraining examples can be processed by embedding layers 104, which include a lexiconencoder 104(1) and a transformer encoder 104(2). Generally, both the lexicon andtransformer encoders operate to produce representations (e.g., vectors) that representindividual words or tokens in a vector space where semantically-similar and / orsyntactically-similar words, tokens, sentences, phrases, documents, queries, etc., arerelatively close to one another, and less semantically-similar or syntactically-similar words,sentences, tokens, phrases, documents, queries, etc., are relatively further apart. Thesevectors are also referred to herein as "embeddings."
[0025] Lexicon encoder 104(1) can produce first embeddings 106, e.g., a sequence ofembedding vectors for each word or token in the pretraining examples 102. An input to thelexicon encoder can be a sequence of tokens of length m, X = {x;,..., xm}. Specific tokenscan be used to delineate the beginning of each sequence, and to separate individual sentencesin a given sequence. The lexicon encoder can map X into a sequence of one embeddingvector for each token. In some implementations, these token embedding vectors areconstructed by summing corresponding word, segment, and positional embeddings for eachtoken in the pretraining examples 102.
[0026] Transformer encoder 104(2) can obtain contextual information for each word ortoken, e.g, via self-attention, and generate second embeddings 108, e.g., a sequence ofcontext embedding vectors. Self-attention is a mechanism relating positions of tokenswithin a sentence, paragraph, or document to compute the similarities between those tokens.In some implementations, the transformer encoder is a multilayer bidirectional transformerencoder that is configured to map the first embeddings 106 into the second embeddings 108.As discussed more below, the second embeddings, or context embedding vectors, can beused as a shared representation of phrases or sentences across different tasks. The contextembedding vectors represent the words or tokens as well as the context within which eachword or token appears in an underlying document, query, or other input.
[0027] The second embeddings 108 can be input to pretraining task layer 110 to performtask-specific processing, as discussed more below. Pretraining task layer 110 can evaluatethe second embeddings 108 to produce a pretraining prediction output 112, such as maskedtoken / word prediction and / or next sentence prediction for a self-supervised learningprocess, as discussed more below. Errors computed during pretraining can be used to adjustthe parameters of the pretraining task layer, transformer encoder 104(2), and / or lexiconencoder 104(1), as also discussed more below.TUNING MODEL EXAMPLE
[0028] FIG. 2 illustrates an example an exemplary natural language processing model200 that can be tuned using the disclosed implementations. After pretraining of the lexiconencoder 104(1) and transformer encoder 104(2), these layers can be connected to a targettask layer 202 for tuning purposes. As discussed elsewhere herein, the tuning can employa supervised learning process that adjusts parameters of the lexicon encoder, transformerencoder, and / or the target task layer based on error determined using tuning examples 204.The error can be based on a difference between labels of the tuning examples and tuningprediction output 206 of the target task layer. The following are but a few examples of thetypes of layers that can be employed for the target task layer.
[0029] In some implementations, the target task layer 202 can be a single-sentenceclassification layer that can label a sentence using class labels. As one specific example, asingle-sentence classification layer can predict whether an English sentence isgrammatically plausible. Another example classification task is to determine whether thesentiment of a sentence extracted from movie reviews is positive or negative. As discussedmore below, the output of the tuning task layer for a classification task can include acorresponding probability for each potential classification. The tuning examples 204 caninclude labeled examples where the labels identify the correct classification for examplesentences.
[0030] In other implementations, the target task layer 202 can be a pairwise textsimilarity layer that performs a regression task on a pair of input sentences. The regressiontask can involve outputting a real-valued similarity score indicating the semantic similarityof the two sentences. The tuning examples can include labeled sentence pairs where thelabels identify semantic similarity between the sentence pairs.
[0031] In other implementations, the target task layer 202 can be a pairwise textclassification layer that determines a relationship between a pair of input sentences, wherethe relationship is selected from a set of pre-defined labels. For example, the labels canindicate whether one input sentence has an entailment relationship, a contradictionrelationship, or a neutral relationship with respect to the other input sentence. The tuningtask layer can output a corresponding probability for each relationship given a pair ofsentence examples. The tuning examples can include labeled sentence pairs where the labelsidentify the correct relationship between the sentence pairs.
[0032] In other implementations, the target task layer 202 can be a relevance rankinglayer. In such implementations, the model can receive two inputs, e.g., a web search queryand a list of candidate documents, and output relevance scores that reflect how relevant eachcandidate document is to the query. The relevance scores can be used in a subsequentnatural language processing operation to rank the candidate answers in the order ofrelevance to the query.EXAMPLE TRAINING WORKFLOW
[0033] FIG. 3 illustrates an example training workflow 300 for training of a machinelearning model, consistent with some implementations. The training workflow is describedusing the natural language processing model 100 shown in FIG. 1 as an example model forpretraining and natural language processing model 200 shown in FIG. 2 as an examplemodel for tuning, but the training workflow can be performed on many different types ofmachine learning models.
[0034] Training workflow 300 can include a pretraining stage 302 and a tuning stage304. As discussed more below, the pretraining stage can be used to determine pretrainedparameters for one or more layers of a machine learning model, and the tuning stage can beused to adapt those pretrained parameters to a particular target task. As also described morebelow, in some cases the tuning stage is applied to one or more target task layers that arenot pretrained during the pretraining stage.
[0035] In some implementations, the pretraining stage 302 can utilize unlabeled trainingdata 306. For instance, the unlabeled training data can provide an unlabeled corpus ofdocuments in a given natural language. The embedding layers 104 can be pretrained byunsupervised learning to predict tokens in the corpus as discussed more below.
[0036] First, selected pretraining examples 308 are selected from the unlabeled trainingdata 306. Next, a masker 310 can randomly mask off individual components (e.g., wordsor tokens) in the selected pretraining examples to obtain masked pretraining examples 312.The masked pretraining examples can be input to lexicon encoder 104(1), which can outputcorresponding first embeddings 106 for each unmasked component (e.g., word or token) inthe selected pretraining examples.
[0037] Next, the first embeddings 106 are input to a noise adjustment process 314,which can adjust the first embeddings using a virtual adversarial technique discussed morebelow to obtain noise-adjusted first embeddings 316. First embeddings 106 and noise-adjusted first embeddings 316 can be input to a pretraining parameter adjustment process318. In the pretraining parameter adjustment process 318, parameters of the embeddinglayers and the pretraining task layer 110 are adjusted to obtain an updated model 320, whichis then output to a pretraining model history 322. The next pretraining iteration can proceedby retrieving the previous model 324 from the pretraining model history 322 and continuingwith pretraining iterations until a stopping condition is reached, e.g., the model converges,achieves a threshold accuracy on a test data set, a training budget is exhausted, and / or allthe unlabeled training data 306 has been exhausted.
[0038] After the pretraining stage 302, the embedding layers and the target task layer202 (FIG. 2) can be tuned together in tuning stage 304. Initially, the pretrained parametersof the embedding layers can be provided in tuning model history 326, along with initialized(e.g., random) parameters for the target task layer 202. Collectively, the pretrainedparameters of the embedding layers and the initialized parameters of the task-specific layerform a first iteration of the natural language processing model 200. As discussed morebelow, subsequent tuning iterations can be used to revise these initial parameters to obtainsubsequent iterations of the machine learning model.
[0039] One specific tuning approach is to use a stochastic gradient descent technique totrain the parameters of the embedding layers 104 and the target task layer 202 using selectedtuning examples 328 from labeled task-specific examples 330. In each training iteration orepoch, a batch of labeled training task-specific examples can be selected and input to atuning parameter adjustment process 332. Tn the tuning parameter adjustment process, themodel parameters are tuned to obtain an updated model 336, which is then output to thetuning model history 326. The next tuning iteration can proceed by retrieving the previousmodel 338 from the tuning model history and continuing with tuning iterations until astopping condition is reached, e.g., the model converges, achieves a threshold accuracy ona test data set, a training budget is exhausted, and / or all the labeled task-specific examples330 have been exhausted.RUNTIME MODEL EXAMPLE
[0040] After pretraining and tuning, the natural language processing model 200 can beemployed at runtime as shown in FIG. 4. First, input data 402 (e.g., a sentence, query,paragraph, etc.) can be input to lexicon encoder 104(1). The lexicon encoder can encodecomponents (words or tokens) of the input data to obtain first embeddings 106 and providethese to the transformer encoder 104(2). The transformer encoder 104(2) can producesecond embeddings 108 and provide these to the target task layer 202. The target task layer202 can produce a runtime prediction output 404. For instance, the runtime predictionoutput can provide a distribution of probabilities for a set of defined classifications, asdiscussed elsewhere herein.
[0041] Note that FIG. 4 and FIG. 2 both illustrate natural language processing model200. In FIG. 2, natural language processing model is shown processing tuning examples,and in FIG. 4, natural language processing model is shown processing input data. Also,note that first embeddings 106 and second embeddings 108 are shown in each of FIGS. 1,2, and 4 generically, In practice, these embeddings will change as a function of the databeing processed during pretraining, tuning, and at runtime.ADDITIONAL DETAILS
[0042] As noted, natural language processing models 100 and 200 can be neuralnetworks with multiple layers. The layers can include one or more mapping layers thatproduce representations of input data. For example, as discussed above, the one or moremapping layers can include a lexicon encoder (e.g., a first mapping layer) that maps asequence of tokens into a sequence of input embedding vectors. The one or more mappinglayers can also include a transformer encoder (e.g., a second mapping layer) that maps theinput embedding vectors into a sequence of contextual embedding vectors.
[0043] In some implementations, noise is added to the output of the first mapping layer,e.g., the lexicon encoder for certain natural language models. In other implementations,noise can be added at other layers of the machine learning model, e.g., to the output ofsubsequent mapping layers, such as the transformer encoder or other layers that map inputdata to corresponding representations.
[0044] The following provides some more implementation details for natural languagemodels having lexicon and transformer encoders as the mapping layers. In someimplementations, the one or more mapping layers can be pretrained by self-supervisedlearning. One approach involves masking tokens and pretraining the one or more mappinglayers with a pretraining task layer to predict the masked tokens bidirectionally, e.g., left-to-right and right-to-left. Another approach involves next sentence prediction. Additionaldetails can be found in Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova, "BERT: Pre-training of deep bidirectional transformers for languageunderstanding," 2018, arXiv preprint, arXiv:1810.04805, which is incorporated herein byreference in its entirety.
[0045] In some implementations, one or more initial iterations of pretraining areperformed without noise adjustment. Subsequently, pretraining can be continued for one ormore subsequent iterations using noise adjustment. The noise adjustment can be employedfor virtual adversarial training, e.g., as discussed below. Virtual adversarial training caninvolve using a training objective that encourages the one or more mapping layers toproduce a smooth output distribution for pairs of embeddings and corresponding noise-adjusted embeddings. As also discussed below, the noise can be added randomly in anadversarial direction, subject to a perturbation bound.
[0046] After the one or more mapping layers are pretrained, the one or more mappinglayers and one or more classification layers can be tuned together, e.g., using a supervisedlearning task. This can be performed using virtual adversarial training and / or using standardsupervised learning techniques that do not employ adversarial mechanisms.
[0047] Generally, a trained model can be used to perform one or more tasks. In naturallanguage processing scenarios, the trained model can be employed in a search engine, e.g.,to match queries to documents or to determine the sentiment of a given sentence. As anotherexample, the trained model can be used in a digital assistant or bot to rank candidateresponses to user utterances.
[0048] In addition, while natural language models are used herein as examples, thedisclosed techniques can be employed for other applications. Consider computer vision, forinstance. One or more image mapping layers (¢.g., a convolutional layer) can be pretrainedusing a self-supervised approach, e.g, trying to recover a part of an image that has beenremoved. Subsequently, a task-specific classification layer can be trained with the imagemapping layers to predict object classifications, assign semantic labels to images, etc.TECHNICAL EFFECT
[0049] As noted previously, adversarial training has been shown to improve robustnesswhile tending to hurt generalization. Here, robustness means susceptibility to an attack, andgeneralization means the ability of a trained model to perform well on new datasets. Forinstance, consider an image classifier that is trained using adversarial training. Typically,adversarial training encourages the model to duplicate hard labels for training examplesafter noise is added to the training examples. Thus, for instance, given a labeled image of acat and the same image of the cat with added noise, adversarial training encourages themodel to classify both images as a cat.
[0050] This approach has been shown to produce robust models that withstand attacks,e.g., an attacker that adds noise to another image of a cat is much less likely to "fool" amodel that has been trained using adversarial training. However, adversarial training hasbeen demonstrated to hurt generalization. Thus, for instance, a model trained usingconventional adversarial training may not perform as well on a new dataset with images ofcats that have not previously been seen by the model, e.g., such a model may exhibit loweraccuracy than a model trained using without adversarial examples.
[0051] As discussed more below, however, models pretrained using the disclosedvirtual adversarial techniques exhibit both improved robustness and generalization relativeto previous techniques. Here, virtual adversarial training encourages the model to producea similar output distribution for noise-adjusted inputs relative to the output distribution forunadjusted inputs. A transformer pretrained using the virtual adversarial techniquesdisclosed herein can be subsequently tuned with one or more classification layers, andclassifications output by such a model are likely to not only resist attacks better than thosewith conventionally-pretrained transformers, but also tend to generalize better on new datasets than those with conventionally-pretrained transformers. This is true whether thesubsequent training of the transformer and classifier(s) is performed with adversarialtraining (virtual or otherwise) or using non-adversarial learning techniques.
[0052] Furthermore, note that adversarial techniques can be computationally expensive.Specifically, computing the adversarial direction for a given pretraining example can becomputationally burdensome. Thus, some implementations may adopt a curriculumapproach by performing one or more initial pretraining iterations without adding adversarialnoise, and performing subsequent pretraining iterations with added adversarial noise. Asdiscussed more below, this approach can offer the generalization and robustness benefits ofadversarial pretraining while reducing the computational expense of performing allpretraining iterations in an adversarial manner.ALGORITHMIC DETAILS AND EXPERIMENTAL RESULTS
[0053] The following section provides some algorithmic details on the pretraining stage302 and the tuning stage 304 shown in FIG. 3. In addition, the following section reportssome experimental results on models pretrained and tuned using the disclosedimplementations. Specific examples of the disclosed training techniques are referred tobelow using the term "ALUM." ALUM provides a unifying adversarial training algorithmapplicable to pretraining and fine-tuning of machine learning models.
[0054] In the following description, MLNI refers to Multi-genre Natural LanguageInference, and is a data set with sentence pairs annotated for textual entailment relationships.QQP refers to Quora Question Pairs, which is a dataset of question pairs annotated toindicate whether the questions duplicate the meaning of each other. RTE refers toRecognizing Textual Entailment, which is a dataset of sentences labeled to indicate whetherthe meaning of one sentence can be inferred from the other. QNLI refers to a question-answering dataset with questions and corresponding answers to the questions that areidentified in spans of text provided in the dataset. MPRC refers to Microsoft ResearchParaphrase Corpus, a dataset of sentences labeled with paraphrase relationships indicatingwhether one sentence accurately paraphrases another. CoLA refers to Corpus of LinguisticAcceptability, which is a dataset of sentences labeled for grammatical acceptability. SSTrefers to Stanford Sentiment Treebank, which is a dataset of examples labeled with positiveor negative sentiments expressed by the examples. STS-B refers to Semantic TextualSimilarity Benchmark, which is a dataset with labels indicating similarity between twosentences.Training Objectives
[0055] Both pretraining and fine-tuning can be viewed as minimizing or reducing thestandard error on training data, with the training objectives derived from self-supervision(without manual labels from, e.g, a human) and direct supervision (labeled examples intask-specific fine-tuning), respectively.
[0056] For example, the following training algorithms can seek to learn a classificationfunction f(x; 0): x - C, parametrized by 6. In masked language modeling (MLM), C isthe vocabulary, and f(x; @) tries to predict the masked token y. In fine-tuning, C is thetask-specific label set, and f(x; 8) is the classifier. Given a training dataset D of input-output pairs (x,y), the classifier f(x; 8) can be trained to minimize the empirical risk as:MINE yy) [L(F (6; 6), Y)] (1)where I(f (x; 8),y) is the loss function (e.g. cross entropy).Adversarial Training
[0057] Pretraining a large neural language model such as BERT, mentioned above, canimprove generalization performance in task-specific fine-tuning. However, such modelscan still suffer from catastrophic loss in adversarial scenarios, with attacks as simple asreplacing a few words in input sentences that preserve the semantics.
[0058] One approach to improve model robustness and withstand adversarial attacks isadversarial training. Adversarial training can augment the training objective by applyingsmall perturbation to input images that maximizes the adversarial loss:Mink y).p [maxi (f(x + 6:0), y)] 2)where the inner maximization can be solved by running a number of projected gradientdescent steps. However, adversarial training has a tendency to reduce generalizationperformance.The ALUM Algorithm
[0059] The following presents an overall training algorithm, ALUM:Input: 7: the total number of iterations, = {(x1,v1),.-., (Xn. ¥o)}: the dataset,f(x; 8): the machine learning model parametrized by 8, 0: the variance of therandom initialization of perturbation &, euro symbol perturbation bound, K: the number ofiterations for perturbation estimation, 7: the step size for updating perturbation, 7:the global learning rate, a: the smoothing proportion of adversarial training in theaugmented learning objective.Equation
[0060] ALUM is applicable to both pretraining and fine-tuning. As noted above, theembedding space can be perturbed by adding noise for virtual adversarial training;EquationEffectively, this introduces an adversarial term that favors label smoothness in theembedding neighborhood, and «a is a hyperparameter that controls the tradeoff betweenstandard errors and robust errors.
[0061] Experiments conducted with ALUM demonstrated that virtual adversarialtraining is superior to conventional adversarial training, especially when labels might benoisy. For instance, BERT pretraining uses the masked words as self-supervised labels, butin many cases, they could be replaced by other words to generate completely legitimate newtext. This was verified empirically by demonstrating that pretraining benefits from large a.In some cases, « is set to a relatively large number, e.g. « = 10 for pretraining, and a = 1for fine-tuning in the following reported experiments.
[0062] Compared to standard training, adversarial training can introduce additionalcomputational expense due to the inner maximization. ALUM adopts a curriculum learningapproach: first pretrain the model using the standard objective (1); and then continue thetraining with virtual adversarial training (3).
[0063] Referring back to the ALUM algorithm set forth above, lines 4-6 run K projectedgradient steps to find the perturbation § that maximizes the adversarial loss (violation oflocal smoothness). Note that a larger K leads to a better approximation, but it is moreexpensive. To have a reasaonble trade-off between speed and performance, K = 1 was usedin the following reported experiments.Generalization vs. Robustness
[0064] The experiments discussed below established that applying adversarialpretraining using ALUM improved both generalization and robustness for a wide range ofnatural language processing (NLP) tasks. As previously noted, adversarial training isgenerally thought to hurt generalization. Based on the following results, adversarialpretraining may help overcome the generalization disadvantages previously associated withadversarial training. For instance, by perturbing the embedding space rather than the inputspace, adversarial training as discussed herein may bias toward on-manifold perturbationthan regular perturbation, which helps generalization.Experiments
[0065] The following section reports results for model generalization and modelrobustness of ALUM compared with other models, including BERT and RoBERTa (Liu, etal., "RoBERTa: A Robustly Optimized BERT Pretraining Approach," arXiv preprintarXiv:1907.11692, 2019). Subsequently, results are reported using ALUM with adversarialfine-tuning.Datasets and Benchmarks
[0066] The following experiments were conducted using three publicly available large-scale raw corpora in total of 82G words:• Wikipedia is an English Wikipedia dump which contains 13G raw text aftercleaning.• OPENWEBTEXT (Gokaslan and Cohen) (Gokaslan et al., "Openwebtextcorpus,") is a large text corpus of 38G words extracted from Reddit.• STORIES is a subset of CommonCrawl data developed by Trinh and Le(Trinh et al , "A Simple Method for Commonsense Reasoning," arXivpreprint, arXiv:1806.02847, 2018) for commonsense reasoning(Winograd schemas). It contains 31G words.
[0067] To evaluate model generalization, the GLUE (Wang et al, "Glue: A Multi-taskBenchmark and Analysis Platform for Natural Language Understanding," arXiv preprint,arXiv:1804.07461, 2018) and SQuAD (Rajpurkar et al, "SQuAD: 100,000+ Questions forMachine Comprehension of Text," Proceedings of the 2016 Conference on EmpiricalMethods in Natural Language Processing, pp. 2383-2392, 2016) (V1.1 and V2.0)benchmarks were used, as well as three named entity recognition (NER) tasks in biomedicaldomain. To evaluate robustness to adversarial attack, the ANLI (Nie et al. "AdversarialNLI: A New Benchmark for Natural Language Understanding," arXiv preprint, arXiv:1910-14599, 2019), adversarial SQUAD (Jia and Liang, "Adversarial Examples for EvaluatingReading Comprehension Systems," arXiv preprint, arXiv:1707-07658, 2017), andHELLASWAG (Hampel, "7he Influence Curve and its Role in Robust Estimation," Journalof the American Statistical Association, 69(346):383-393, 1974) datasets were employed.Two representative benchmarks were used, MNLI (in GLUE) and ANLI, in the adversarialfine-tuning experiments, combined with three additional datasets: SWAG (Zellers et al."SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference,"arXiv preprint, arXiv:1808.05326, 2018), SNLI (Bowman et al, "4 Large AnnotatedCorpus for Learning Natural Langage Inference," Proceedings of the 2015 Conference onEmpirical Methods in Natural Language Processing (EMNLP), Association forComputation Linguistics, 2015) and SciTail (Khot, et al., "Sci7ail: A Textual FntailmentDataset from Science Question Answering," AAAL 2018). These benchmarks cover a widerange of NLP tasks including classification, ranking, regression, machine readingcomprehension, and named entity recognition.Implementation Details
[0068] Three types of training were performed for the following experiments:pretraining from scratch, continual pretraining on a well-trained model, and task-specificfine-tuning. The following describes specific, nonlimiting implementations of models thatcan be built using the concepts disclosed herein.
[0069] BERT models were pretrained from scratch using the Wikipedia corpus. ADAMwas used as the optimizer with a learning rate of 1 x 10~* which firstly firstly schedules1% steps warmup and then decays linearly. Diederik Kingma and Jimmy Ba., "ADAM: 4Method for Stochastic Optimization," in arXiv preprint arXiv:1412.6980v9. Subsequently,the model was updated for 1M steps with a batch size of 256. The perturbation size was setaseuro symbol = 1 x 1075, the step size p = 1 x 1073, and the variance of normal distribution o =1 x 107°. Since the loss is dominated by the first term in Eq 3, a value @ = 10 was set, andK = 1 for training efficiency. The training took 10 days on one DGX-2 machine with 16V100-32G GPUs.
[0070] For continual pretraining on RoBER Ta, RoBERTa's default training parameterswere used, except a smaller learning rate (4 X 107%), and run for 100K training steps witha batch size of 256 on the corpora of 82G text. The training took 7 days on two DGX-2machines.
[0071] For fine-tuning with and without adversarial training, ADAM and RADAM (Liuet al., "On the Variance of the Adaptive Learning Rate and Beyond," arXiv preprint, arXiv:1908.03265, 2019) were used as optimizers with learning rates of {5 x 107%,8 x 107%, 1 x1075,2 x 107°}, A linear leaning rate decaying schedule with a warm-up of 0.1 wasapplied to all the tasks, unless stated otherwise. The batch size was set to 16, 32 or 64. Thedropout rate was set to 0.1 for all the task-specific layers, except 0.3 for MNLI (Williamset al., "A Broad-Coverage Challenge Corpus for Sentence Understanding throughInference," Proceedings of the 2018 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human Language Technologies, Vol. 1 (LongPapers), pp. 1112-1122, 2018) and 0.05 for CoLA. To avoid gradient exploding, thegradient norm was clipped within 1. All the texts were tokenized using WordPiece andchopped to spans up to 512 tokens. The model was fine-tuned for up to 10 epochs with theprovided task-specific training set and the most accurate task-specific model was pickedbased on its performance on the dev set.Model Generalization
[0072] The following section reports generalization results for pre-trained models viafine-tuning for downstream tasks. In the pretraining-from-scratch setting, the followingthree models are compared:• BERTBASE is a BERT base model trained 1M steps with a batch size of256.• BERT+BASE is similar to BERTBASE, except that it is trained with 1.6Msteps, which takes the same amount of time as that ofALUM grpr_gase. as detailed below.• ALUM prpr_pase 1s @ BERT model trained using ALUM with the sametraining setting as that of BERTBASE except that during the courseof the 1M-step training, standard training was employed in the first500K steps, and adversarial training in the second 500K steps. Sinceeach adversarial training step takes approximately 1.4 times longerthan a step in standard training, ALUM prrr_pasg takesapproximately the same amount of time to train as BERT+BASE.
[0073] FIG. 5A shows the results of these models on three benchmark datasets: SQuADv1.1 and v2.0, and MNLI in GLUE. ALUM ggrr-pgask consistently outperforms BERT andBERT+ across all the datasets, ¢.g., on SQUAD v1.1, ALUM gppr_pasg Obtains 2.3% pointsabsolute improvement in F1 score over BERTBASE and 1.2% points over BERT+BASE.An analysis was performed on accuracy of a fine-tuned ALUM pprr_pasg on MNLI as afunction of the number of pretraining steps. When adversarial training was applied after the500K-th step, ALUM starts outperforming BERT and the gap grew larger with the numberof pretraining steps.
[0074] To further validate the generalization of ALUM, pre-trained models were fine-tuned and evaluated on three NER tasks in the biomedical domain, which differsdramatically from the Wikipedia corpus used for pretraining. As shown in FIG. 5B, ALUMoutperforms BERTBASE on all three tasks, including BC2GM (Smith et al., "Overview ofBiocreative II Gene Mention Recognition," Genome Biology, 9 Suppl. 2:52, 2008), NCBI(Dogan et al., "NCBI Disease Corpus: A Resource for Disease Name Recognition andConcept Normalization," Journal of Biomedical Informatics, 47, 2014) and JNLPBA(Collier and Kim, "Introduction fo the Bio-Entity Recognition Task at JNLPBA,"Proceedings of the International Joint Workshop on Natural Language Processing inBiomedicine and its Applications (NLPBA / BioNLP), pp. 73-78, 2004).
[0075] The following experiments substantiate the generalization merit of ALUM in thecontinual pretraining setting. The public RoOBERTa models were taken as the initial models,and trained them for 100K steps using the 82G text dataset. The RoBERTa models use thesame network architectures as BERT, but are pre-trained on much larger amounts of texts.They outperform BERT models in many NLP tasks.
[0076] Standard continual pretraining was performed on RoBERTa. The results in FIG.5C show that this failed to produce any significant gains. Potentially, the existing RoOBERTahas already been well-trained and it is challenging to achieve further improvements on topof it using standard pretraining approaches.
[0077] Next, pretraining was continued with ROBERTalLARGE and RoBERTaBASEusing ALUM, and the resultant models are denoted as ALUM gopprra-14rce andALUM goprrra-RASE > Fespectively. These were then fine-tuned on eight downstream tasksof GLUE. The results in FIG. 5D show that both ALUM gp, pepra-pase andALUM g,prrra-iarck outperform their ROBERTa counterparts on seven out of eight tasks.For instance, consider the results on MNLI-m as an example. ALUM poprrTa-BASEoutperforms RoBERTaBASE by +0.5% accuracy. ALUM gopprra-rarce OutperformsROBERT j4r.4, by +0.7% accuracy. These results show that even with a manageablecomputational cost (i.e., 100K training steps on 82G texts), continual adversarial pretrainingcan improve model generalization over a model such as RoBERTa, and achieve significantimprovements on a wide range of downstream NLP tasks.Model Robustness
[0078] The following section describes experiments conducted on three benchmarksincluding ANLI, HELLASWAG and adversarial SQuAD, to evaluate the robustness of pre-trained models. Four datasets were employed as the final ANLI training data, includingANLI, MNLI, SNLI, and FEVER (Thorne et al, "Fever: 4 Large-scale Dataset for FactExtraction and Verification," arXiv preprint, arXiv:1803.05355, 2018). Models were testedon three different rounds having respective difficulty level. The same fine-tuningapproaches were performed for both the baseline models and their ALUM counterparts.
[0079] FIG. SE summarizes the results on ANLI. Note that both ALUM ggpr_gass andALUM poprrra-1arce Significantly outperform their counterparts including BERTBASEand ROBERTA j4,g with large margins. For example, ALUM gopprra-rarce Obtains57.0% (vs 49.7%) in terms of accuracy which is 7.3% absolute improvement overROBERTA jg. ALUM gggrr-pasg not only outperforms counterpart BERTBASE, butalso beats BERTLARGE. ALUM pg pprra-1arce also outperforms the XILNet by 5.0%(57.0% vs 52.0%) on the ANLI task.
[0080] FIG. 5F reports the results on Adversarial SQUAD and HELLASWAG. Notethat that adversarial training ALUM improves results on both of these adversarial datasets.For instance, on the setting of AddOneSent in Adversarial SQUAD, ALUM grrr-paskobtains 4.2% (63.2% vs 59.0%) and 5% (69.8% vs 64.8%) absolute gains comparing withBERTBASE in terms of F1 score and EM, respectively. The gains overALUM goprra-iarce 1S slightly smaller than the BASE counterpart, but it is stillsignificant. For example, it achieves 3.4 % (69.4% vs. 66.0%) and 2.1% (75.0% vs. 72.9%)in terms of F1 metrics in the Adversarial SQUAD dataset. Similar observations are foundon HELLASWAG. This clearly demonstrates that the pretrained models trained viaadversarial training can bring significant improvement on model robustness, as comparedwith its counterpart without adversarial training. From all the experiments above, this leadsto the observations that ALUM can achieve both excellent generalization and exceptionalrobustness to downstream tasks.Continuous Adversarial Fine-tuning
[0081] The following section explores whether adversarial pretraining techniques arecomplementary to adversarial fine-tuning. Two representative benchmarks are employed:MNLI and ANLI in this study, using ROBERTaLARGE and ALUM g,grrra-LarcE as thebaseline. The ALUM g,perTa-1.4RcE Model fine-tuned with adversarial training is referredto as ALUM gogrrra-rarce-smarr- The adversarial fine-tuning procedure is described inJiang et al., "Robust and Efficient Fine-Tuning for Pre-trained Natural Language Modelsthrough Principled Regularized Optimization," arXiv preprint, arXiv:1911.03437, 2019.
[0082] Figure 5G shows the results on these two downstream tasks. The improvementsare significant and consistent. For example, the adversarial fine-tuning technique brings a0.5% (91.4% vs. 90.9%) additional gain on the MNLI task and a 1.1% (58.8% vs 57.7%)gain on the ANLI task respectively. These empirical results that model performance can beimproved with adversarial fine-tuning, even when the pre-trained model has alreadybenefited from adversarial pretraining.EXAMPLE SYSTEM
[0083] The present implementations can be performed in various scenarios on variousdevices. FIG. 6 shows an example system 600 in which the present implementations can beemployed, as discussed more below.
[0084] As shown in FIG. 6, system 600 includes a client device 610, a server 620, aserver 630, and a client device 640, connected by one or more network(s) 650. Note thatthe client devices can be embodied both as mobile devices such as smart phones or tablets,as well as stationary devices such as desktops, server devices, etc. Likewise, the servers canbe implemented using various types of computing devices. In some cases, any of the devicesshown in FIG. 6, but particularly the servers, can be implemented in data centers, serverfarms, etc.
[0085] Certain components of the devices shown in FIG. 6 may be referred to herein byparenthetical reference numbers. For the purposes of the following description, theparenthetical (1) indicates an occurrence of a given component on client device 610, (2)indicates an occurrence of a given component on server 620, (3) indicates an occurrence onserver 630, and (4) indicates an occurrence on client device 640. Unless identifying aspecific instance of a given component, this document will refer generally to the componentswithout the parenthetical.
[0086] Generally, the devices 610, 620, 630, and / or 640 may have respective processingresources 601 and storage resources 602, which are discussed in more detail below. Thedevices may also have various modules that function using the processing and storageresources to perform the techniques discussed herein. The storage resources can includeboth persistent storage resources, such as magnetic or solid-state drives, and volatile storage,such as one or more random-access memory devices. In some cases, the modules areprovided as executable instructions that are stored on persistent storage devices, loaded intothe random-access memory devices, and read from the random-access memory by theprocessing resources for execution.
[0087] Client device 610 can include a configuration module 611 that can interact witha model training module 621 on server 620. Generally speaking, the configuration modulecan provide certain configuration parameters to the model training module. Theconfiguration parameters can include architecture parameters and training parameters (e.g.,hyperparameters). The architecture parameters can specify the structure of a machinelearning model, e.g., the number of nodes, arrangement of the nodes in layers, connectivityof the nodes and / or layers, etc. The architecture parameters can also specify input data andoutput data for each node and / or layer.
[0088] The configuration parameters can also include training parameters that specifyinformation such as learning rates or other hyperparameters (perturbation bounds, numberof noise-free and noise-induced pretraining iterations, etc), unsupervised or self-supervisedlearning parameters and / or data sources, and / or supervised learning parameters and / or datasources. The model training module 621 uses these training configuration parameters toperform model training functionality on a model specified by the architecture parameters.In particular, the model training module can perform training workflow 300 (FIG. 3) basedon the training configuration parameters. As just one example, the unsupervised learningdata sources can include one or more repositories of sentences, where pretraining stage 302can involve performing masked language modeling and / or next sentence prediction. Asanother example, the supervised learning data sources can include task-specific labeledtraining examples, and / or loss / objective functions for individual tasks.
[0089] The model training module 621 can output a trained, final model to server 630.Model execution module 631 can execute the final model in response to received inputs.For example, the interface module 641 on client device 640 can provide input data to anapplication module 632 executing on server 630. The application module can invoke themodel execution module to evaluate the input data on any of the tasks provided in the task-specific layers. The model execution module can process the uploaded input data using thefinal model and provide output of the model to the application module. The applicationmodule can provide the output directly to the client device 640 over network(s) 650, orperform further processing of the output before sending a response to the client device. Forinstance, the application module might select a review to include in the response based ona sentiment of the review output by the model, or rank documents relative to an input querybased on a similarity of the documents to the query as estimated by the model, etc.EXAMPLE METHOD
[0090] FIG. 7 illustrates an example method 700, consistent with the present concepts.As discussed more below, method 700 can be implemented on many different types ofdevices, e.g., by one or more cloud servers, by a client device such as a laptop, tablet, orsmartphone, or by combinations of one or more servers, client devices, etc.
[0091] Method 700 begins at block 702, where a machine learning model is provided.The machine learning model can have one or more mapping layers, including a firstmapping layer that is configured to map components of pretraining examples into firstrepresentations in a space. In natural language scenarios, the pretraining examples caninclude sequences of words or tokens, such as documents, paragraphs, or sentences, and thecomponents can include the words or tokens themselves. In image processing scenarios,the pretraining examples can include images and the components can include portions ofthe images, other images in a sequence, etc.
[0092] The machine learning model can be a neural network, such as discussed in theattached appendices or shown in FIG. 1. Note that block 702 can involve generating anarchitecture (e.g., node structure, connections between nodes, etc.) of the machine learningmodel manually or using automated techniques. Alternatively, block 702 can involvereceiving the machine learning architecture from a remote device or via user input.
[0093] Method 700 continues at blocks 704 and 706, which collectively can correspondto a pretraining stage.
[0094] At block 704, the pretraining stage includes adding noise to the firstrepresentations to obtain noise-adjusted first representations of the training data items. Asnoted previously, the noise can be added in an adversarial direction with a randommagnitude, subject to a perturbation bound.
[0095] At block 706, the pretraining stage includes performing a self-supervisedlearning task on the one or more mapping layers, using the first representations and thenoise-adjusted first representations. In some cases, the self-supervised learning taskinvolves learning from the structure of the pretraining examples. For example, the structureof a document, paragraph, or sentence can be used to learn the meanings of words or tokens.Likewise, a sequence of images (e.g., a video) can convey information about individualobjects that are present in the images, or one object in an image can convey informationabout another (e.g., masked) object in the image.
[0096] Method 700 continues at block 708, where a supervised learning task isperformed on a classification or regression layer and the one or more mapping layers. Asnoted previously, the supervised learning task can be performed using labeled training data.The supervised learning task can be performed using conventional, adversarial, and / orvirtual adversarial techniques.
[0097] Method 700 continues at block 710, where input data is processed using theclassification layer and the one or more mapping layers to obtain a result. For instance, theinput data can include a query and a document, and the result can reflect a similarity of thequery to the document. As another example, the input data can include a sentence, and theresult can characterize a sentiment of the sentence. As another example, the input data caninclude an image and the result can identify a classification of an object in the image.
[0098] Method 700 continues at block 712, where the result is output. For instance, theresult can be output to a local application, sent over a network, displayed via a user interface,etc.EXAMPLE USER EXPERIENCES
[0099] The following describes several user experiences that can be provided usingmachine learning models that have been trained and / or tuned using the disclosed techniques.For instance, application module 632 on server 630 can generate user interfaces such asthose shown below based on output of a machine learning model.
[00100] FIG. 8 illustrates an example review page 800 for a laptop 802, where the reviewpage is one example of a user interface that can be used to convey reviews to users. Here,a user has selected to filter for negative reviews via a drop-down menu 804, and in responsethe review page shows three reviews, review 806, review 808, and review 810. Review 806and review 810 are clearly negative reviews and thus are appropriate to display to the userin response to their request for negative reviews. However, review 808 is a very positivereview of the laptop, but includes the terms "stupendous" and "discriminating." A modelthat not robust could mischaracterize this review as negative, for instance, if these are low-frequency terms that happened to be used in one or more negative example review and themodel overfits to those few examples.
[00101] FIG. 9 illustrates an example review page 900 for laptop 802 that could beproduced using the disclosed techniques. Here, review 808 has been replaced by review902, which is actually a negative review and thus correctly displayed to a user who wishesto see negative reviews for the laptop. Because the disclosed implementations providemachine learning models that generalize well to unseen or low-frequency terms, thedisclosed implementations may correctly learn that the terms "stupendous" and"discriminating" are not strongly associated with negative reviews and accordingly do notmisclassify review 808 as a negative review. More generally, a machine learning modeltrained using the disclosed implementations can be used to determine whether to outputindividual reviews in response to a request for negative and / or positive reviews based onsentiments for the reviews as predicted by the machine learning model.
[00102] FIG. 10 illustrates an example scenario 1000 where a user 1002 interacts withclient device 640. As noted above, the client device can have a local application and / orinteract with a server application that can provide search functionality. In this scenario, theuser requests that the application determine whether the user has any email complaints aboutcustomer service. The application responds by asking the user if he would like to search hisemails for the string "customer service." This search may only reveal results that includethis specific string, and may omit other relevant results.
[00103] FIG. 11 illustrates an example scenario 1100 where the application has access toa model with a relevance ranking layer that can rank the semantic similarity of a query tomultiple answers in a semantic space. This allows the application to provide a flexiblesearch functionality. Here, the user asks for emails about customer service, and theapplication is able to identify an email from a customer named Sara about a delivery that islate. Here, the email may not include the terms "customer" or "service" or alternativeformulations of these terms, and thus may not have been identified by a conventional textsearch. However, while there may be little textual similarity between Sara's email and theseterms, the relevance ranking layer may nevertheless indicate that the query "customerservice" 1s semantically similar to an email complaint about a late delivery. As aconsequence, the application is able to identify Sara's email as being responsive to the user'squery. More generally, a machine learning model trained using the disclosedimplementations can be used to rank documents relative to a query based at least onsimilarities of the documents to the query as output by the machine learning model.DEVICE IMPLEMENTATIONS
[00104] As noted above with respect to FIG. 6, system 600 includes several devices,including a client device 610, a server 620, a server 630, and a client device 640. As alsonoted, not all device implementations can be illustrated, and other device implementationsshould be apparent to the skilled artisan from the description above and below.
[00105] The term "device", "computer," "computing device," "client device," and / or"server device" as used herein can mean any type of device that has some amount ofhardware processing capability and / or hardware storage / memory capability. Processingcapability can be provided by one or more hardware processors (e.g., hardware processingunits / cores) that can execute computer-readable instructions to provide functionality.Computer-readable instructions and / or data can be stored on storage, such asstorage / memory and / or the datastore. The term "system" as used herein can refer to a singledevice, multiple devices, etc.
[00106] Storage resources can be internal or external to the respective devices with whichthey are associated. The storage resources can include any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and / or optical storage devices (e.g,CDs, DVDs, etc.), among others. As used herein, the term "computer-readable media" caninclude signals. In contrast, the term "computer-readable storage media" excludes signals.Computer-readable storage media includes "computer-readable storage devices." Examplesof computer-readable storage devices include volatile storage media, such as RAM, andnon-volatile storage media, such as hard drives, optical discs, and flash memory, amongothers.
[00107] In some cases, the devices are configured with a general purpose hardwareprocessor and storage resources. In other cases, a device can include a system on a chip(SOC) type design. In SOC design implementations, functionality provided by the devicecan be integrated on a single SOC or multiple coupled SOCs. One or more associatedprocessors can be configured to coordinate with shared resources, such as memory, storage,etc., and / or one or more dedicated resources, such as hardware blocks configured to performcertain specific functionality. Thus, the term "processor," "hardware processor" or"hardware processing unit" as used herein can also refer to central processing units (CPUs),graphical processing units (GPUs), controllers, microcontrollers, processor cores, or othertypes of processing devices suitable for implementation both in conventional computingarchitectures as well as SOC designs.
[00108] Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components. For example, andwithout limitation, illustrative types of hardware logic components that can be used includeField-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits(ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc.
[00109] In some configurations, any of the modules / code discussed herein can beimplemented in software, hardware, and / or firmware. In any case, the modules / code can beprovided during manufacture of the device or by an intermediary that prepares the devicefor sale to the end user. In other instances, the end user may install these modules / codelater, such as by downloading executable code and installing the executable code on thecorresponding device.
[00110] Also note that devices generally can have input and / or output functionality. Forexample, computing devices can have various input mechanisms such as keyboards, mice,touchpads, voice recognition, gesture recognition (e.g., using depth cameras such asstereoscopic or time-of-flight camera systems, infrared camera systems, RGB camerasystems or using accelerometers / gyroscopes, facial recognition, etc.). Devices can also havevarious output mechanisms such as printers, monitors, etc.
[00111] Also note that the devices described herein can function in a stand-alone orcooperative manner to implement the described techniques. For example, the methods andfunctionality described herein can be performed on a single computing device and / ordistributed across multiple computing devices that communicate over network(s) 650.Without limitation, network(s) 650 can include one or more local area networks (LANs),wide area networks (WANs), the Internet, and the like.
[00112] Various examples are described above. Additional examples are describedbelow. One example includes a method performed on a computing device, the methodcomprising providing a machine learning model having one or more mapping layers,including at least a first mapping layer configured to map components of pretrainingexamples into first representations in a space and performing a pretraining stage on the oneor more mapping layers using the pretraining examples, wherein the pretraining stagecomprises. The method also comprises adding noise to the first representations of thecomponents of the pretraining examples to obtain noise-adjusted first representations andperforming a self-supervised learning process to pretrain the one or more mapping layersusing at least the first representations and the noise-adjusted first representations of thecomponents of the pretraining examples.
[00113] Another example can include any of the above and / or below examples where theone or more mapping layers comprise a second mapping layer configured to map the firstrepresentations and the noise-adjusted first representations into second representations, thesecond representations reflecting context in which individual components appear with othercomponents in the pretraining examples.
[00114] Another example can include any of the above and / or below examples where thefirst mapping layer comprises a lexicon encoder and the second mapping layer comprises atransformer encoder.
[00115] Another example can include any of the above and / or below examples where thepretraining examples comprise sentences, the components comprise words or tokens in thesentences, and the self-supervised learning process comprises at least one of masked wordprediction, masked token prediction, or next sentence prediction.
[00116] Another example can include any of the above and / or below examples where themethod further comprises, after the pretraining stage, performing a supervised learningprocess on a classification layer and the one or more mapping layers.
[00117] Another example can include any of the above and / or below examples where thesupervised learning process is performed using adversarial training or virtual adversarialtraining.
[00118] Another example can include any of the above and / or below examples where theclassification layer is selected from a group comprising a single-sentence classificationlayer, a pairwise text similarity layer, and a pairwise text classification layer.
[00119] Another example can include any of the above and / or below examples where thepretraining examples comprise images or video, and the one or more mapping layers includea convolutional layer.
[00120] Another example can include any of the above and / or below examples where thesupervised learning process trains the classification layer to predict classifications of objectsin the images or video.
[00121] Another example can include any of the above and / or below examples where themethod further comprises performing one or more initial training iterations of the self-supervised learning process without the noise-adjusted first representations and performingone or more subsequent training iterations of the self-supervised learning process with thenoise-adjusted first representations.
[00122] Another example can include any of the above and / or below examples where theadding noise comprises regularizing a training objective using virtual adversarial training.
[00123] Another example can include any of the above and / or below examples where thetraining objective encourages a smooth output distribution of the machine learning modelfor pairs of first representations and corresponding noise-adjusted first representations ofthe components of the pretraining examples.
[00124] Another example includes a system comprising a hardware processing unit anda storage resource storing computer-readable instructions which, when executed by thehardware processing unit, cause the hardware processing unit to: receive input data, processthe input data using a machine learning model having a first layer and a second layer toobtain a result, the first layer having been pretrained in a pretraining stage using virtualadversarial training for a self-supervised learning task, and output the result.
[00125] Another example can include any of the above and / or below examples where thevirtual adversarial training used in the pretraining stage involves adding noise torepresentations of components of pretraining examples that are used to adjust parameters ofthe first layer.
[00126] Another example can include any of the above and / or below examples where theinput data comprises reviews, the result characterizes sentiments associated with the reviewsas predicted by the machine learning model, and the computer-readable instructions, whenexecuted by the hardware processing unit, cause the hardware processing unit to determinewhether to output individual reviews in response to a request for negative reviews based atleast on the sentiments predicted by the machine learning model.
[00127] Another example can include any of the above and / or below examples where theinput data comprises a query, the result reflects similarities of the query to a plurality ofdocuments as output by the machine learning model, and the computer-readableinstructions, when executed by the hardware processing unit, cause the hardware processingunit to rank the plurality of documents relative to the query based at least on the similaritiesoutput by the machine learning model.
[00128] Another example includes A computer-readable storage medium storinginstructions which, when executed by one or more processing devices, cause the one or moreprocessing devices to perform acts comprising: providing a machine learning model havingone or more mapping layers, including at least a first mapping layer configured to mapcomponents of data items into first representations in a space, performing one or more initialpretraining iterations of a learning process to train the machine learning model with the firstrepresentations, performing noise adjustment on the first representations to obtain noise-adjusted first representations, and performing one or more subsequent pretraining iterationsof the learning process to train the machine learning model with the noise-adjusted firstrepresentations.
[00129] Another example can include any of the above and / or below examples where thefirst representations comprising embedding vectors, the noise-adjusted first representationscomprising noise-adjusted embedding vectors.
[00130] Another example can include any of the above and / or below examples where theperforming noise adjustment comprises determining an adversarial direction in which toperform the noise adjustment.
[00131] Another example can include any of the above and / or below examples where theone or more subsequent pretraining iterations encourage the machine learning model toproduce a smooth output distribution for predictions made using the embedding vectors andthe noise-adjusted embedding vectors.CONCLUSION
[00132] Although the subject matter has been described in language specific to structuralfeatures and / or methodological acts, it is to be understood that the subject matter defined inthe appended claims is not necessarily limited to the specific features or acts describedabove. Rather, the specific features and acts described above are disclosed as example formsof implementing the claims and other features and acts that would be recognized by oneskilled in the art are intended to be within the scope of the claims.
Claims
1. A method performed on a computing device, the method comprising: providing a machine learning model having one or more mapping layers, including at least a first mapping layer configured to map components of pretraining examples into first representations in a space; and performing a pretraining stage on the one or more mapping layers using the pretraining examples, wherein the pretraining stage comprises: adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations; and performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations and the noise-adjusted first representations of the components of the pretraining examples.
2. The method of claim 1, wherein the one or more mapping layers comprise a second mapping layer configured to map the first representations and the noise-adjusted first representations into second representations, the second representations reflecting context in which individual components appear with other components in the pretraining examples.
3. The method of claim 2, wherein the first mapping layer comprises a lexicon encoder and the second mapping layer comprises a transformer encoder.
4. The method of claim 3, wherein the pretraining examples comprise sentences, the components comprise words or tokens in the sentences, and the self-supervised learning process comprises at least one of masked word prediction, masked token prediction, or next sentence prediction.
5. The method of claim 1, further comprising: after the pretraining stage, performing a supervised learning process on a classification layer and the one or more mapping layers.
6. The method of claim 5, wherein the supervised learning process is performed using adversarial training or virtual adversarial training,7. The method of claim 5, wherein the classification layer is selected from a group comprising a single-sentence classification layer, a pairwise text similarity layer, and a pairwise text classification layer.
8. The method of claim 5, wherein the pretraining examples comprise images or video, and the one or more mapping layers include a convolutional layer.
9. The method of claim 8, wherein the supervised learning process trains the classification layer to predict classifications of objects in the images or video.
10. The method of claim 1, further comprising; performing one or more initial training iterations of the self-supervised learning process without the noise-adjusted first representations; and performing one or more subsequent training iterations of the self-supervised learning process with the noise-adjusted first representations.
11. The method of claim 1, wherein the adding noise comprises regularizing a training objective using virtual adversarial training,12. The method of claim 11, wherein the training objective encourages a smooth output distribution of the machine learning model for pairs of first representations and corresponding noise-adjusted first representations of the components of the pretraining examples.
13. A system comprising: a hardware processing unit; and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to: receive input data; process the input data using a machine learning model having a first layer and a second layer to obtain a result, the first layer having been pretrained in a pretraining stage using virtual adversarial training for a self-supervised learning task; and output the result.
14. The system of claim 13, wherein the virtual adversarial training used in the pretraining stage involves adding noise to representations of components of pretraining examples that are used to adjust parameters of the first layer.
15. The system of claim 14, wherein the input data comprises a query, the result reflects similarities of the query to a plurality of documents as output by the machine learning model, and the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: rank the plurality of documents relative to the query based at least on the similarities output by the machine learning model.
