Adversarial pre-training of machine learning models

Abstract

This document relates to training of machine learning models. An example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pre-training examples to first representations in a space. The example method also includes performing a pre-training phase on the one or more mapping layers using the pre-training examples. The pre-training phase can include adding noise to the first representations of the components of the pre-training examples to obtain noise-adjusted first representations. The pre-training phase can also include performing a self-supervised learning process to pre-train the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Description

Background Technology

[0001] Machine learning can be used to perform a wide range of tasks, such as natural language processing, financial analysis, and image processing. Machine learning models can be trained using various methods, such as supervised learning, semi-supervised learning, unsupervised learning, and reinforcement learning. In methods such as supervised or semi-supervised learning, labeled training examples are used to train the model to map inputs to outputs. Summary of the Invention

[0002] This summary is provided to introduce the selection of concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] The description generally relates to techniques for training machine learning models. One example includes a method or technique that can be executed on a computing device. This method or technique may include providing a machine learning model having one or more mapping layers, the one or more mapping layers including at least a first mapping layer configured to map components of pre-trained examples to a first representation in a space. The method or technique may also include performing a pre-training phase on multiple layers of the one or more mappings using pre-trained examples. The pre-training phase may include adding noise to the first representation of the components of the pre-trained examples to obtain a noise-adjusted first representation. The pre-training phase may also include performing a self-supervised learning process to pre-train the one or more mapping layers using at least the first representation of the components of training data items and the noise-adjusted first representation.

[0004] Another example includes a system with a hardware processing unit and storage resources for storing computer-readable instructions. When executed by the hardware processing unit, the computer-readable instructions can cause the hardware processing unit to receive input data and process the input data using a machine learning model with a first layer and a second layer to obtain a result. The first layer can be pre-trained during a pre-training phase using virtual adversarial training for a self-supervised learning task. The computer-readable instructions can also cause the hardware processing unit to output the result.

[0005] Another example includes a computer-readable storage medium storing instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform actions. The actions may include providing a machine learning model having one or more mapping layers, the one or more mapping layers including at least a first mapping layer configured to map components of data items to a first representation in a space. These actions may also include performing one or more initial pre-training iterations of a learning process to train the machine learning model with the first representation. These actions may also include performing noise adjustment on the first representation to obtain a noise-adjusted first representation. These actions may further include performing one or more subsequent pre-training iterations of the learning process to train the machine learning model with the noise-adjusted first representation. Attached Figure Description

[0006] Specific embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost digit of the reference number(s) identifies the first figure in which the reference number appears. The use of similar reference numerals in different instances in the description and drawings may indicate similar or identical items.

[0007] Figure 1 The illustration shows an example machine learning model for pre-training that is consistent with some implementations of this concept.

[0008] Figure 2 The illustration shows an example machine learning model for tuning that is consistent with some implementations of this concept.

[0009] Figure 3 The diagram illustrates an example training workflow for training machine learning models that is consistent with some implementations of this concept.

[0010] Figure 4 The illustration shows an example machine learning model for performing predictions at runtime, consistent with some implementations of this concept.

[0011] Figures 5A-5G The illustration shows an example of experimental results obtained by using a machine learning model generated through the disclosed implementation.

[0012] Figure 6 The illustration shows some example systems that are consistent with some implementations of this concept.

[0013] Figure 7 The illustrations depict example methods or techniques for pre-training, tuning, and / or employing machine learning models that are consistent with some implementations of this concept.

[0014] Figure 8 , Figure 9 , Figure 10 and Figure 11 The illustrations show example user experiences and user interfaces consistent with some implementations of this concept. Detailed Implementation

[0015] Various types of machine learning frameworks can be trained to perform a given task. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks that have been widely used in applications such as image processing and natural language processing. Some machine learning frameworks, such as neural networks, use layers of nodes that perform specific operations.

[0016] In a neural network, nodes are interconnected via one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Individual nodes can process their respective inputs according to a predefined function and provide their outputs to subsequent layers, or in some cases, to previous layers. The input to a given node can be multiplied by the corresponding weight value of the edge between the input and the node. Furthermore, nodes may have individual bias values, which are also used to produce the output. Various training procedures can be applied to learn edge weights and / or bias values. The term "parameters" in this document, without modification, refers to learnable values, such as edge weights and bias values, which can be learned by training machine learning models (such as neural networks).

[0017] Neural network architectures can have different layers that perform different specific functions. For example, one or more layers of nodes can work together to perform specific operations such as pooling, encoding, or convolution. For the purposes of this document, the term "layer" refers to a set of nodes that share input and output (e.g., share input and output with external sources or other layers in the network). The term "operation" refers to a function that can be performed by one or more layers of nodes. The term "model architecture" refers to the overall architecture of a layered model, including the number of layers, the connectivity of the layers, and the types of operations performed by each layer. The term "neural network architecture" refers to the model structure of a neural network. The terms "trained model" and / or "tuned model" refer to the model structure and the parameters of a model structure that has been trained or tuned. Note that, for example, if two models are trained on different training data, or if there are potential stochastic processes during training, two trained models can share the same model structure but have different parameter values.

[0018] Many machine learning tasks suffer from a relative lack of training data. A widely used approach to train models using limited task-specific training data involves "transfer learning." In transfer learning, a model is first pre-trained on another task with a large amount of training data available, and then tuned to the specific task using task-specific training data.

[0019] As used herein, the term "pre-training" refers to training a model on a pre-training dataset in a way that allows for subsequent tuning of those model parameters for one or more specific tasks. In some cases, pre-training can involve a self-supervised learning process on unlabeled training data, where the "self-supervised" learning process involves learning from the structure of pre-trained examples, possibly without explicit (e.g., manually provided) labels. Subsequent modifications to the model parameters obtained through pre-training are referred to herein as "tuning." Tuning can be performed on one or more tasks using supervised learning from explicitly labeled training data, and in some cases, on tasks different from those used for pre-training.

[0020] Traditional techniques used to train and tune model parameters are vulnerable to specific attacks. For example, even if the correct classification is obvious to human users, a carefully crafted or "adversarial" sentence or image can "trick" the model into outputting an obviously incorrect classification for that sentence or image. A widely used approach to develop models robust to such attacks is called "adversarial" training. However, as described below, traditional adversarial training often impairs the model's generalization ability.

[0021] The disclosed implementation provides a mechanism for virtual adversarial pre-training of one or more mapping layers of a model. After pre-training using the disclosed technique, the pre-trained mapping layers can be fine-tuned using task-specific layers to perform specific tasks using supervised learning. As discussed below, by pre-training a model using the disclosed technique, the final model can tend to both generalize well and be robust to adversarial attacks.

[0022] The following discussion provides some specific examples within the context of natural language processing. However, as discussed further below, the disclosed techniques can also be used for tasks beyond natural language processing, such as image processing, computer vision, radar, sonar, medical applications, genetic marker analysis, and more.

[0023] Pre-trained model examples

[0024] Figure 1 The illustration shows a first natural language processing model 100, illustrating how the layers of a machine learning model can be pre-trained using the disclosed implementation. As discussed further below, natural language processing model 100 is an example of a machine learning model that can be used to perform one or more natural language processing tasks. For the purposes of this document, the term "natural language" refers to the language that humans typically use for writing or conversation.

[0025] Natural Language Processing Model 100 may receive pre-trained examples 102, which may include documents, sentences, phrases, or other representations of language with various components such as words and / or tokens. The components of the pre-trained examples may be processed by an embedding layer 104, which includes a dictionary encoder 104 (1) and a transformer encoder 104 (2). Typically, both the dictionary encoder and the transformer encoder are used to generate representations (e.g., vectors) that represent individual words or tokens in a vector space, where semantically and / or syntactically similar words, tokens, sentences, phrases, documents, queries, etc., are relatively close to each other, while semantically less similar or syntactically similar words, sentences, tokens, phrases, documents, queries, etc., are relatively far apart. These vectors are also referred to as “embeds” in this paper.

[0026] The dictionary encoder 104(1) can produce a first embedding 106, for example, a sequence of embedding vectors for each word or token in the pre-training example 102. The input to the dictionary encoder can be a sequence of tokens of length m, X = {x1, ..., x2}. m}. Specific tokens can be used to delineate the beginning of each sequence and to separate individual sentences within a given sequence. A dictionary encoder can map X to a sequence of embedding vectors for each token. In some implementations, these token embedding vectors are constructed by summing the corresponding word, segment, and position embeddings for each token in pre-trained example 102.

[0027] Transformer encoder 104(2) can obtain contextual information for each word or token, for example, via self-attention, and generate a second embedding 108, such as a sequence of context embedding vectors. Self-attention is a mechanism that associates the positions of tokens in a sentence, paragraph, or document to compute similarity between those tokens. In some implementations, the transformer encoder is a multi-layer bidirectional transformer encoder configured to map the first embedding 106 to the second embedding 108. As discussed more below, the second embedding, or context embedding vector, can be used as a phrase or sentence across different tasks. The context embedding vector represents a word or token and the context in which each word or token appears in the underlying document, query, or other input.

[0028] As discussed further below, the second embedding 108 can be fed into the pre-trained task layer 110 to perform task-specific processing. The pre-trained task layer 110 can evaluate the second embedding 108 to produce a pre-trained prediction output 112, such as masked token / word prediction and / or next sentence prediction for a self-supervised learning process. Errors computed during pre-training can be used to adjust the parameters of the pre-trained task layer, the transformer encoder 104(2), and / or the dictionary encoder 104(1), as discussed in more detail below.

[0029] Model tuning example

[0030] Figure 2 An example of an exemplary natural language processing model 200 is shown, which can be tuned using the disclosed implementation. After pre-training on a dictionary encoder 104(1) and a transformer encoder 104(2), these layers can be connected to a target task layer 202 for tuning purposes. As discussed elsewhere in this document, tuning can employ a supervised learning process that adjusts the parameters of the dictionary encoder, transformer encoder, and / or target task layer based on errors determined using tuning example 204. The error can be based on the difference between the tuning prediction output 206 of the target task layer and the label of the tuning example. The following are just a few examples of layer types that can be used for the target task layer.

[0031] In some implementations, the target task layer 202 can be a single-sentence classification layer, which can use class tags to label sentences. As a concrete example, a single-sentence classification layer can predict whether an English sentence is grammatically correct. Another example classification task is to determine whether the sentiment of a sentence extracted from a movie review is positive or negative. As discussed further below, the output of the tuning task layer for the classification task can include the corresponding probability for each potential class. Tuning example 204 can include labeled examples, where the labels identify, for example, the correct classification of a sentence.

[0032] In other implementations, the target task layer 202 can be a pairwise text similarity layer that performs a regression task on the input sentence pairs. The regression task may involve outputting real-valued similarity scores representing the semantic similarity between two sentences. Tuning examples may include labeled sentence pairs, where the labels identify the semantic similarity between the sentence pairs.

[0033] In other implementations, the target task layer 202 can be a pairwise text classification layer that determines a relationship between a pair of input sentences, where the relationship is selected from a predefined set of tags. For example, the tags could indicate whether one input sentence has an implication, contradiction, or neutral relationship relative to another input sentence. The tuning task layer can output a probability corresponding to each relationship given a pair of sentence examples. The tuning examples can include labeled sentence pairs, where the tags identify the correct relationship between the sentence pairs.

[0034] In other implementations, the target task layer 202 can be a relevance ranking layer. In such an implementation, the model can receive two inputs, such as a web search query and a list of candidate documents, and output a relevance score reflecting how relevant each candidate document is to the query. The relevance score can be used for subsequent natural language processing operations to rank the candidate answers according to their relevance to the query.

[0035] Example training workflow

[0036] Figure 3 The diagram illustrates an example training workflow 300 for training machine learning models, consistent with some implementations. The training workflow uses... Figure 1 The natural language processing model 100 shown is used as a pre-trained example model. Figure 2 The natural language processing model 200 shown is described as an example model for tuning, but training workflows can be performed on many different types of machine learning models.

[0037] The training workflow 300 may include a pre-training phase 302 and an optimization phase 304. As discussed further below, the pre-training phase may be used to determine the pre-trained parameters of one or more layers of a machine learning model, and the optimization phase may be used to adapt those pre-trained parameters to a specific target task. As described in more detail below, in some cases, the optimization phase is applied to one or more target task layers that were not pre-trained in the pre-training phase.

[0038] In some implementations, the pre-training phase 302 can utilize unlabeled training data 306. For example, unlabeled training data can be provided as an unlabeled corpus of documents in a given natural language. As discussed further below, the embedding layer 104 can be pre-trained via unsupervised learning to predict tokens in the corpus.

[0039] First, a selected pre-training example 308 is chosen from the unlabeled training data 306. Next, a masker 310 can randomly mask individual components (e.g., words or tokens) in the selected pre-training example to obtain a masked pre-training example 312. The masked pre-training example can be fed into a dictionary encoder 104 (1), which can output a corresponding first embedding 106 for each unmasked component (e.g., word or token) in the selected pre-training example.

[0040] Next, the first embedding 106 is input into the noise adjustment process 314, which uses virtual adversarial techniques discussed below to adjust the first embedding to obtain a noise-adjusted first embedding 316. The first embedding 106 and the noise-adjusted first embedding 316 can then be input into the pre-training parameter adjustment process 318. In the pre-training parameter adjustment process 318, the parameters of the embedding layer and the pre-training task layer 110 are adjusted to obtain an updated model 320, which is then output to the pre-training model history 322. The next pre-training iteration can be performed by taking the previous model 324 from the pre-training model history 322 and continuing the pre-training iteration until a stopping condition is met (e.g., model convergence, reaching a threshold accuracy on the test dataset, exhaustion of the training budget, and / or exhaustion of all unlabeled training data 306).

[0041] After the pre-training phase 302, the embedding layer and the target task layer 202 ( Figure 2These can be tuned together in tuning phase 304. Initially, the pre-trained parameters of the embedding layer and the initialization (e.g., random) parameters of the target task layer 202 can be provided in the tuning model history 326. The pre-trained parameters of the embedding layer and the initialization parameters of the task-specific layer together form the first iteration of the natural language processing model 200. As discussed more below, subsequent tuning iterations can be used to modify these initial parameters to obtain subsequent iterations of the machine learning model.

[0042] One specific tuning method uses stochastic gradient descent to train the parameters of the embedding layer 104 and the target task layer 202 using tuning examples 328 selected from labeled task-specific examples 330. In each training iteration or epoch, a batch of labeled training task-specific examples can be selected and fed into the tuning parameter adjustment process 332. During the tuning parameter adjustment process, the model parameters are tuned to obtain an updated model 336, which is then output to the tuning model history 326. The next tuning iteration can be performed by retrieving previous models 338 from the tuning model history and continuing the tuning iteration until a stopping condition is met (e.g., model convergence, reaching a threshold accuracy on the test dataset, exhaustion of the training budget, and / or exhaustion of all labeled task-specific examples 330).

[0043] Runtime model example

[0044] like Figure 4 As shown, after pre-training and tuning, the natural language processing model 200 can be used at runtime. First, input data 402 (e.g., sentences, queries, paragraphs, etc.) can be fed into a dictionary encoder 104 (1). The dictionary encoder can encode the components (words or tokens) of the input data to obtain first embeddings 106 and provide these to a transformer encoder 104 (2). The transformer encoder 104 (2) can produce second embeddings 108 and provide them to a target task layer 202. The target task layer 202 can produce a runtime prediction output 404. For example, the runtime prediction output can provide a set of defined probability distributions for classification, as discussed elsewhere in this paper.

[0045] Please note, Figure 4 and Figure 2 Both diagrams illustrate Natural Language Processing Model 200. Figure 2 The example shown is an example of natural language processing model optimization, and... Figure 4 The diagram illustrates a natural language processing model processing input data. Furthermore, note that, generally, the first embedding 106 and the second embedding 108 are... Figure 1 , Figure 2 and Figure 4 Each of the embeddings is shown in the diagram. In practice, these embeddings will vary depending on the data processed during pre-training, tuning, and runtime.

[0046] Additional details

[0047] As noted, natural language processing models 100 and 200 can be neural networks with multiple layers. These layers may include one or more mapping layers that generate representations of the input data. For example, as described above, one or more mapping layers may include a dictionary encoder (e.g., a first mapping layer) that maps a sequence of tokens to a sequence of input embedding vectors. One or more mapping layers may also include a transformer encoder (e.g., a second mapping layer) that maps the input embedding vectors to a sequence of context embedding vectors.

[0048] In some implementations, noise is added to the output of the first mapping layer, such as in a dictionary encoder for a specific natural language model. In other implementations, noise can be added to other layers of the machine learning model, such as to the output of subsequent mapping layers, like a transformer encoder or other layers that map input data to corresponding representations.

[0049] Below are some implementation details of natural language models with a dictionary and transformer encoder as mapping layers. In some implementations, one or more mapping layers can be pre-trained via self-supervised learning. One approach involves masking tokens and pre-training one or more mapping layers using pre-trained task layers to predict masked tokens bidirectionally, such as from left to right and from right to left. Another approach involves next-sentence prediction. For more details, see Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018, arXivpreprint, arXiv: 1810.04805, the entire contents of which are incorporated herein by reference.

[0050] In some implementations, one or more initial iterations of pre-training are performed without noise adjustment. Subsequently, noise adjustment can be used to continue pre-training on one or more subsequent iterations. For example, noise adjustment can be used for virtual adversarial training, as described below. Virtual adversarial training can involve using a training objective that promotes one or more mapping layers to produce a smooth output distribution for embeddings and their corresponding noise-adjusted embedding pairs. As discussed below, noise can be randomly added in the adversarial direction, subject to perturbation bounds.

[0051] After one or more mapping layers are pre-trained, these mapping layers and one or more classification layers can be tuned together, for example, using a supervised learning task. This can be done using virtual adversarial training and / or using standard supervised learning techniques that do not employ adversarial mechanisms.

[0052] Typically, a trained model can be used to perform one or more tasks. In natural language processing scenarios, a trained model can be used in search engines, for example, to match queries with documents or determine the sentiment of a given sentence. As another example, a trained model can be used in digital assistants or robots to rank candidate responses to user utterances.

[0053] Furthermore, although this paper uses natural language models as an example, the techniques disclosed can be used in other applications. For example, consider computer vision. One or more image mapping layers (e.g., convolutional layers) can be pre-trained using self-supervised methods, for example, attempting to recover a portion of an image that has been deleted. Subsequently, task-specific classification layers can be trained using the image mapping layers to predict object classifications, assign semantic labels to images, and so on.

[0054] Technical effect

[0055] As mentioned earlier, adversarial training has been shown to improve robustness, but often impairs generalization. Here, robustness means sensitivity to attacks, and generalization means the ability to train a model to perform well on new datasets. For example, consider an image classifier trained using adversarial training. Typically, adversarial training encourages the model to replicate the hard-labeled training examples after noise has been added. Thus, for example, given a labeled image of a cat and the same image of a cat with noise added, adversarial training encourages the model to classify both images as cats.

[0056] This approach has been shown to generate robust models that can withstand attacks; for example, an attacker adding noise to another image of a cat is unlikely to "fool" a model that has already been trained using adversarial training. However, adversarial training has been shown to impair generalization. Therefore, for example, a model trained using conventional adversarial training may perform poorly on new datasets with images of cats that the model has not seen before; for instance, the accuracy of such a model may be lower than that of a model trained using no adversarial examples.

[0057] However, as discussed further below, models pre-trained using the disclosed virtual adversarial techniques exhibit improved robustness and generalization compared to previous techniques. Here, virtual adversarial training facilitates the model to produce an output distribution for noise-adjusted inputs that is similar to the output distribution for unadjusted inputs. Transformers pre-trained using the disclosed virtual adversarial techniques can be subsequently tuned with one or more classification layers, and the classifications output by such models not only resist attacks better than those using conventionally pre-trained transformers, but also tend to generalize better on new datasets than those using conventionally pre-trained transformers. This is true regardless of whether subsequent training of the transformers and classifiers is performed through adversarial training (virtual or otherwise) or using non-adversarial learning techniques.

[0058] Furthermore, it's important to note that adversarial techniques can be computationally expensive. Specifically, computing the adversarial orientation for a given pre-trained example can be computationally intensive. Therefore, some implementations employ a curriculum approach, performing one or more initial pre-training iterations without adding adversarial noise, and subsequent iterations with added adversarial noise. As described below, this approach can provide the generalization and robustness advantages of adversarial pre-training while reducing the computational cost of performing all pre-training iterations adversarially.

[0059] The following sections provide details about the algorithm and experimental results. Figure 3 Some algorithmic details of the pre-training phase 302 and tuning phase 304 are shown below. Furthermore, the following sections report some experimental results using pre-trained and tuned models with the publicly disclosed implementation. The term "ALUM" is used below to refer to specific examples of the disclosed training technique. ALUM provides a unified adversarial training algorithm suitable for the pre-training and fine-tuning of machine learning models.

[0060] In the following descriptions, MLNI refers to Multi-Type Natural Language Inference, a dataset of sentence pairs annotated with textual entailment relations. QQP refers to Quora Question Pairs, a dataset of question pairs annotated to indicate whether questions are repetitive. RTE refers to Identifying Textual Entailment, a dataset of sentences labeled to indicate whether the meaning of one sentence can be inferred from another. QNLI refers to Question Answering, a dataset containing questions and corresponding answers identified within the text range provided in the dataset. MPRC refers to the Microsoft Research Paraphrasing Corpus, a dataset of sentences labeled with paraphrasing relations indicating whether one sentence accurately paraphrases another. CoLA refers to the Linguistic Acceptability Corpus, a dataset of sentences labeled with grammatical acceptability. SST refers to the Stanford Sentiment Treebank, a sample dataset labeled with positive or negative sentiment expressed by examples. STS-B refers to the Semantic Textual Similarity Benchmark, a labeled dataset representing the similarity between two sentences.

[0061] Training objectives

[0062] Both pre-training and fine-tuning can be viewed as minimizing or reducing the standard error of the training data, with training objectives derived from self-supervision (without manual labels from sources such as humans) and direct supervision (labeled examples in task-specific fine-tuning), respectively.

[0063] For example, the following training algorithm seeks to learn a classification function f(x; θ): x → C, parameterized by θ. In Masked Language Modeling (MLM), C is the vocabulary, and f(x; θ) attempts to predict masked tokens y. In fine-tuning, C is the task-specific set of tags, and f(x; θ) is the classifier. Given a training dataset D of input-output pairs (x, y), the classifier f(x; θ) can be trained to minimize the empirical risk as follows:

[0064]

[0065] Where l(f(x; θ), y) is the loss function (e.g., cross-entropy).

[0066] Combat training

[0067] Pre-training large neural language models (such as BERT mentioned above) can improve generalization performance during task-specific fine-tuning. However, such models still suffer catastrophic losses in adversarial scenarios, where attacks can be as simple as replacing a few semantically preserving words in the input sentence.

[0068] One approach to improve model robustness and resistance to adversarial attacks is adversarial training. Adversarial training maximizes the adversarial loss by applying small perturbations to the input image to increase the training objective.

[0069]

[0070] The internal maximization can be achieved by running multiple projective gradient descent steps. However, adversarial training tends to reduce generalization performance.

[0071] ALUM algorithm

[0072] The following introduces a comprehensive training algorithm, ALUM:

[0073] Input: T: Total number of iterations = {(x1, y1), ..., (x n y n )}:Dataset f(x;θ): Machine learning model parameterized by θ, σ 2 : Variance of random initialization of perturbation δ, ε: Perturbation bound, K: Number of iterations for perturbation estimation, η: Step size for updating perturbation, τ: Global learning rate, α: Smoothing ratio of adversarial training in the reinforcement learning objective.

[0074]

[0075] Output: θ

[0076] ALUM is applicable to both pre-training and fine-tuning. As mentioned above, the embedding space can be perturbed by adding noise to the virtual adversarial training:

[0077]

[0078] In fact, this introduces an adversarial term that favors label smoothness in the embedding neighborhood, and α is a hyperparameter that controls the tradeoff between standard error and robust error.

[0079] Experiments using ALUM demonstrate that virtual adversarial training outperforms traditional adversarial training, especially when the tokens may be noisy. For example, BERT pre-training uses masked words as self-supervised tokens, but in many cases, these can be replaced with other words to generate perfectly legitimate new text. This is demonstrated by showing that pre-training benefits from a large α. In some cases, α is set to a relatively large number; for example, in the experiments reported below, α = 10 was used for pre-training, and α = 1 was used for fine-tuning.

[0080] Compared to standard training, adversarial training introduces additional computational overhead due to internal maximization. ALUM employs a course learning approach: first, the model is pre-trained using a standard objective (1); then, training continues through virtual adversarial training (3).

[0081] Returning to the ALUM algorithm presented above, lines 4-6 run K projected gradient steps to find the perturbation δ that maximizes the adversarial loss (violation of local smoothness). Note that a larger K leads to a better approximation, but at a higher cost. To strike a reasonable trade-off between speed and performance, K=1 was used in the experiments reported below.

[0082] Generalization and robustness

[0083] The experiments discussed below demonstrate that adversarial pre-training using ALUM improves generalization and robustness across a wide range of Natural Language Processing (NLP) tasks. As previously mentioned, adversarial training is generally considered to impair generalization. Based on the following results, adversarial pre-training can help overcome the generalization drawbacks previously associated with adversarial training. For example, by perturbing the embedding space instead of the input space, the adversarial training discussed in this paper may favor perturbations on the manifold rather than regular perturbations, which contributes to generalization.

[0084] experiment

[0085] The following sections report the model generalization and robustness results of ALUM compared to other models, including BERT and RoBERTa (Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, arXiv preprint arXiv:1907.11692, 2019). Results are then reported using ALUM with adversarial fine-tuning.

[0086] Datasets and benchmarks

[0087] The following experiments were conducted using three publicly available large-scale raw corpora, totaling 82G of words:

[0088] • Wikipedia is an English Wikipedia dump containing 13GB of cleaned-up original text.

[0089] ·OPENWEBTEXT (Gokaslan and Cohen) (Gokaslan et al., “Openwebtext corpus”) is a large text corpus of 38G words extracted from Reddit.

[0090] • STORIES is a subset of the CommonCrawl data developed by Trinh and Le (Trinh et al., “A Simple Method for Commonsense Reasoning”, arXiv preprint, arXiv: 1806.02847, 2018) for commonsense reasoning (Winograd pattern). It contains 31G of words.

[0091] To evaluate model generalization, the GLUE (Wang et al., “Glue: A Multi-task Benchmark and Analysis Platform for Natural Language Understanding,” arXiv preprint, arXiv: 1804.07461, 2018) and SQuAD (Rajpurkar et al., “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016) (VI.1 and V2.0) benchmarks were used, along with three Named Entity Recognition (NER) tasks in the biomedical field. To assess the robustness against adversarial attacks, the ANLI (Nie et al., “Adversarial NLI: A New Benchmark for Natural Language Understanding,” arXiv preprint, arXiv: 1910-14599, 2019), adversarial SQuAD (Jia and Liang, “Adversarial Examples for Evaluating Reading Comprehension Systems,” arXiv preprint, arXiv: 1707-07658, 2017), and HELL ASWAG (Hampel, “The Influence Curve and its Role in Robust Estimation,” Journal of the American Statistical Association, 69(346): 383-393, 1974) datasets were used.Two representative benchmarks, MNLI (in GLUE) and ANLI, were used in the adversarial fine-tuning experiments, combined with three additional datasets: SWAG (Zellers et al., “SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference,” arXiv preprint, arXiv: 1808.05326, 2018), SNLI (Bowman et al., “A Large Annotated Corpus for Learning Natural Language Inference,” Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computation Linguistics, 2015), and SciTail (Khot et al., “SciTail: A Textual Entailment Dataset from Science Question Answering,” AAAI, 2018). These benchmarks cover a wide range of NLP tasks, including classification, ranking, regression, machine reading comprehension, and named entity recognition.

[0092] Implementation details

[0093] Three types of training were performed on the following experiments: pre-training from scratch, continuous pre-training on a well-trained model, and task-specific fine-tuning. The following describes specific, non-restrictive implementations of models that can be built using the concepts disclosed in this paper.

[0094] The BERT model was pre-trained from scratch using the Wikipedia corpus. ADAM was used as the optimizer with a learning rate of 1×10⁻⁶. -4 It first schedules a 1% warm-up step, followed by linear decay. (Diederik Kingma and Jimmy Ba, “ADAM: A Method for Stochastic Optimization,” in arXiv preprint arXiv:1412.6980v9). Subsequently, the model is updated in 1M steps with a batch size of 256. The perturbation size is set to ε = 1 × 10⁻⁶. -5 Step size η = 1 × 10 -3 And the variance of the normal distribution is σ = 1 × 10 -5Since the loss is primarily controlled by the first term in Equation 3, a value of α = 10 and K = 1 were set for training efficiency. Training was conducted for 10 days on a DGX-2 machine equipped with 16 V100-32G GPUs.

[0095] For continuous pre-training of RoBERTa, use RoBERTa's default training parameters, except for a small learning rate (4×10). -5 The training was performed on an 82G corpus of text, with 100K training steps and a batch size of 256. Training was conducted over 7 days on two DGX-2 machines.

[0096] For fine-tuning with and without adversarial training, ADAM and RADAM (Liu et al., “On the Variance of the Adaptive Learning Rate and Beyond”, arXiv preprint, arXiv: 1908.03265, 2019) were used as optimizers with a learning rate of (5 × 10⁻⁶). -6 8×10 -6 1×10 -5 2×10 -5 Unless otherwise specified, a linear learning rate decay scheme of 0.1 was applied to all tasks. Batch sizes were set to 16, 32, or 64. The signal loss rate for all task-specific layers was set to 0.1 (Williams et al., “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference,” Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol 1 (Long Papers), pp. 1112-1122, 2018) and 0.05 for CoLA, except for 0.3 for MNLI. Gradient norms were clipped to within 1 to avoid gradient explosion. All text was tokenized using WordPiece and split into up to 512 tokens. The models were fine-tuned for up to 10 epochs using the provided task-specific training set, and the most accurate task-specific model was selected based on its performance on the development set.

[0097] Model generalization

[0098] The following section reports the generalization results of the pre-trained model, fine-tuned for downstream tasks. Three models were compared in a pre-training setup from scratch:

[0099] ·BERTBASE is a base model of BERT, trained for 1M steps with a batch size of 256.

[0100] BERT+BASE is similar to BERT BASE, except that it uses 1.6M steps for training, which is consistent with ALUM. BERT-BASE The time spent was the same, as detailed below.

[0101] ·ALUM BERT-BASE This is a BERT model trained using ALUM, with the same training settings as BERTBASE, except that in the 1M-step training process, standard training is used for the first 500K steps, and adversarial training is used for the second 500K steps. Since each adversarial training step takes approximately 1.4 times longer than a standard training step, ALUM... BERT-BASE The training time is roughly the same as that of BERT+BASE.

[0102] Figure 5A The results of these models are shown on three benchmark datasets: SQuAD v1.1 and v2.0, and MNLI in GLUE. ALUM BERT-BASE It consistently outperforms BERT and BERT+ on all datasets; for example, on SQuADv1.1, ALUM... BERT-BASE The FI score is 2.3% higher than BERT+BASE and 1.2% higher than BERT+BASE. A ALUM score fine-tuned on MNLI... BERT-BASE The accuracy was analyzed as a function of the number of pre-training steps. When adversarial training was applied after 500K steps, ALUM began to outperform BERT, and the gap widened with the number of pre-training steps.

[0103] To further validate the generalization of ALUM, the pre-trained model was fine-tuned and evaluated on three NER tasks in the biomedical field, which differed significantly from the Wikipedia corpus used for pre-training. Figure 5BAs shown, ALUM outperformed BERTBASE in all three tasks, including BC2GM (Smith et al., “Overview of Biocreative II GeneMention Recognition”, Genome Biology, 9 Suppl. 2: S2, 2008), NCBI (Dogan et al., “NCB Idisease Corpus: A Resource for Disease Name Recognition and Concept Normalization”, Journal of Biomedical Informatics, 47, 2014), and JNLPBA (Collier and Kim, “Introduction to the Bio-Entity Recognition Task at JNLPBA”, Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA / BioNLP), pp. 73-78, 2004).

[0104] The following experiments demonstrate the generalization advantages of ALUM in continuous pre-training settings. Publicly available RoBERTa models were used as initial models and trained for 100K steps on an 82G text dataset. RoBERTa models use the same network architecture as BERT but are pre-trained on a large amount of text. They outperform BERT models on many NLP tasks.

[0105] Standard continuous pre-training was performed on RoBERTa. Figure 5C The results showed that this failed to yield any significant gains. Potentially, the existing RoBERTa is already well-trained, and achieving further improvements using standard pre-training methods is challenging.

[0106] Next, we will use ALUM to pre-train RoberTaLARGE and RoberTaBASE, and the resulting models will be represented as ALUM. RoBERTα-LARGE and ALUM RoBERTα-BASE Then, fine-tuning was performed on GLUE's eight downstream tasks. Figure 5D The results in ALUM show RoBERTα-BASEALUMROBERTα-LARGE outperformed RoBERTa's competitors in seven out of eight tasks. For example, consider the results on MNLI-m. RoBERTα-BASE Its accuracy is +0.5% higher than RoberTaBASE. ALUM RoBERTα-LARGE The accuracy is higher than RoberTa large High +0.7%. These results demonstrate that even with manageable computational costs (i.e., 100K training iterations on 82G of text), persistent adversarial pre-training can improve the generalization ability of models, surpassing models such as RoBERTa, and achieving significant improvements across a wide range of downstream NLP tasks.

[0107] Model robustness

[0108] The following sections describe experiments conducted on three benchmarks, including ANLI, HELLASWAG, and adversarial SQuAD, to evaluate the robustness of the pre-trained models. Four datasets were used as the final ANLI training data: ANLI, MNLI, SNLI, and FEVER (Thome et al., “Fever: A Large-scale Dataset for Fact Extraction and Verification,” arXiv preprint, arXiv: 1803.05355, 2018). The models were tested in three different rounds at their respective difficulty levels. The same fine-tuning approach was performed on the baseline models and their ALUM counterparts.

[0109] Figure 5E The results for ANLI are summarized. Please note ALUM. BERT-BASE and ALUM RoBERTα-LARGE Their performance was significantly better than that of BERTBASE and RoberTa. large Compared to similar products, including those from ALUM, the difference is significant. RoBERTα-LARGE It achieved 57.0% accuracy (compared to 49.7%), compared to RoBERTa. large Increased by 7.3%. ALUM BERT-BASE It not only outperforms the corresponding product BERTBASE, but also surpasses BERTLARGE. ALUM RoBERTα-LARGE It also outperformed XLNet by 5.0% on the ANLI task (57.0% vs. 52.0%).

[0110] Figure 5FReport results for adversarial training on SQuAD and HELLASWAG. Note that adversarial training with ALUM improves results on both adversarial datasets. For example, on the AddOneSent setting in adversarial SQuAD, ALUM outperforms BERTBASE in terms of FI score and EM. BERT-BASE They achieved absolute gains of 4.2% (63.2% vs. 59.0%) and 5% (69.8% vs. 64.8%), respectively. The gains exceeded those of ALUM. RoBERTα-LARGE Slightly smaller than comparable BASE products, but still significant. For example, in terms of the FI metric on the adversarial SQuAD dataset, it achieves 3.4% (69.4% vs. 66.0%) and 2.1% (75.0% vs. 72.9%). Similar observations were found on HELLASWAG. This clearly demonstrates that pre-trained models trained with adversarial training can significantly improve robustness compared to models without adversarial training. From all the experiments above, this leads to the observation that ALUM achieves excellent generalization and outstanding robustness to downstream tasks.

[0111] Continuous counter-adjustment

[0112] The following sections explore whether adversarial pre-training techniques complement adversarial fine-tuning. This study employed two representative benchmarks: MNLI and ANLI, using Robertal-Talarge and Alum. RoBERTα-LARGE As a baseline, ALUM was fine-tuned through adversarial training. RoBERTα-LARGE The model is called ALUM RoBERTα-LARGE-SMART The adversarial fine-tuning process is described in Jiang et al., “Robustand Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization,” arXiv preprint, arXiv: 1911.03437, 2019.

[0113] Figure 5G The results for these two downstream tasks are shown. These improvements are significant and consistent. For example, adversarial fine-tuning brings an additional gain of 0.5% (91.4% vs. 90.9%) for the MNLI task and 1.1% (58.8% vs. 57.7%) for the ANLI task, respectively. These empirical results demonstrate that even if the pre-trained model has already benefited from adversarial pre-training, model performance can still be improved through adversarial fine-tuning.

[0114] Example System

[0115] This implementation can run on various devices in various scenarios. As discussed further below, Figure 6 An example system 600 that can be implemented using this method is shown.

[0116] like Figure 6 As shown, system 600 includes client device 610, server 620, server 630, and client device 640 connected via one or more networks 650. Note that client devices can be mobile devices, such as smartphones or tablets, or fixed devices such as desktop computers, server devices, etc. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 6 Any of the devices shown, especially servers, can be implemented in data centers, server farms, etc.

[0117] Figure 6 Specific components of the devices shown herein may be indicated herein by reference numbers enclosed in parentheses. For the purposes described below, parentheses (1) indicate the presence of a given component on client device 610, (2) indicate the presence of a given component on server 620, (3) indicate the presence on server 630, and (4) indicate an event on client device 640. Unless identifying a specific instance of a given component, this document will generally refer to the component without parentheses.

[0118] Typically, devices 610, 620, 630, and / or 640 may have their own processing resources 601 and storage resources 602, which will be discussed in more detail below. These devices may also have various modules that use the processing and storage resources to perform the techniques discussed herein. Storage resources may include persistent storage resources, such as magnetic or solid-state drives, and volatile storage, such as one or more random access memory (RAM) devices. In some cases, modules are provided as executable instructions stored on persistent storage devices, loaded into RAM devices, and read from RAM by the processing resources for execution.

[0119] Client device 610 may include configuration module 611 that can interact with model training module 621 on server 620. Generally, the configuration module can provide specific configuration parameters to the model training module. Configuration parameters may include architecture parameters and training parameters (e.g., hyperparameters). Architecture parameters can specify the structure of the machine learning model, such as the number of nodes, the arrangement of nodes in layers, and the connectivity of nodes and / or layers. Architecture parameters can also specify the input and output data for each node and / or layer.

[0120] Configuration parameters may also include training parameters specifying information such as learning rate or other hyperparameters (perturbation limits, noise-free and noise-induced pre-training iterations, etc.), unsupervised or self-supervised learning parameters and / or data sources, and / or supervised learning parameters and / or data sources. The model training module 621 uses these training configuration parameters to perform model training functions on the model specified by the architecture parameters. Specifically, the model training module can execute training workflow 300 based on the training configuration parameters. Figure 3 As just one example, an unsupervised learning data source may include one or more sentence repositories, where the pre-training phase 302 may involve performing masked language modeling and / or next sentence prediction. As another example, a supervised learning data source may include task-specific labeled training examples and / or a single task's loss / objective function.

[0121] The model training module 621 can output the trained final model to the server 630. The model execution module 631 can execute the final model in response to received input. For example, the interface module 641 on the client device 640 can provide input data to the application module 632 executing on the server 630. The application module can call the model execution module to evaluate input data for any task provided in a task-specific layer. The model execution module can process the uploaded input data using the final model and provide the model's output to the application module. The application module can provide output directly to the client device 640 via (multiple) networks 650, or further process the output before sending a response to the client device. For example, the application module can select comments to include in the response based on the sentiment of the comments in the model output, or rank documents relative to the input query based on the similarity between the document and the query estimated by the model, etc.

[0122] Example Method

[0123] Figure 7 An example method 700 consistent with this concept is illustrated. As discussed further below, method 700 can be implemented on many different types of devices, such as via one or more cloud servers, via client devices such as laptops, tablets, or smartphones, or via a combination of one or more servers and client devices, etc.

[0124] Method 700 begins at box 702, which provides a machine learning model. The machine learning model may have one or more mapping layers, including a first mapping layer configured to map components of pre-trained examples to a first representation in the space. In a natural language scenario, pre-trained examples may include sequences of words or tokens, such as documents, paragraphs, or sentences, and components may include the words or tokens themselves. In an image processing scenario, pre-trained examples may include images, and components may include portions of images, other images in a sequence, etc.

[0125] Machine learning models can be neural networks, such as those discussed in the appendix or... Figure 1 As shown in the diagram. Note that block 702 may relate to the architecture (e.g., node structure, connections between nodes, etc.) for generating machine learning models manually or using automated techniques. Alternatively, block 702 may relate to receiving machine learning architectures from remote devices or via user input.

[0126] Method 700 continues in boxes 704 and 706, which together correspond to the pre-training phase.

[0127] In box 704, the pre-training phase includes adding noise to the first representation to obtain a noise-adjusted first representation of the training data items. As previously described, the noise can be added in the adversarial direction with random magnitude, subject to perturbation bounds.

[0128] In box 706, the pre-training phase includes performing a self-supervised learning task on one or more mapping layers using a first representation and a noise-adjusted first representation. In some cases, the self-supervised learning task involves learning from the structure of pre-trained examples. For example, the structure of a document, paragraph, or sentence can be used to learn the meaning of words or tokens. Similarly, a sequence of images (e.g., a video) can convey information about the individual objects present in the images, or one object in an image can convey information about another (e.g., a masked) object in the image.

[0129] Method 700 continues in box 708, where a supervised learning task is performed on the classification or regression layer and one or more mapping layers. As previously described, the supervised learning task can be performed using labeled training data. Traditional, adversarial, and / or virtual adversarial techniques can be used to perform the supervised learning task.

[0130] Method 700 continues in box 710, where the input data is processed using a classification layer and one or more mapping layers to obtain results. For example, the input data may include queries and documents, and the results may reflect the similarity between the query and the document. As another example, the input data may include sentences, and the results may characterize the sentiment of the sentence. As yet another example, the input data may include images, and the results may identify the classification of objects in the image.

[0131] Method 700 continues in box 712, where the results are output. For example, the results can be output to a local application, sent over a network, or displayed via a user interface.

[0132] Example User Experience

[0133] The following describes several user experiences that can be provided using machine learning models that have been trained and / or tuned using the disclosed techniques. For example, application module 632 on server 630 can generate a user interface such as the one shown below based on the output of the machine learning model.

[0134] Figure 8 The illustration shows a sample review page 800 for a laptop 802, serving as an example of a user interface used to communicate reviews to a user. Here, the user has selected to filter negative reviews via a dropdown menu 804, and in response, the review page displays three reviews: review 806, review 808, and review 810. Reviews 806 and 810 are clearly negative and therefore appropriate to display to the user in response to their request for negative reviews. However, review 808 is a very positive review of the laptop, but includes the words "amazing" and "critical." For example, if these are low-frequency terms used in one or more of the negative example reviews and the model overfits to these examples, a weak model might incorrectly describe the review as negative.

[0135] Figure 9 The illustration shows an example review page 900 for a laptop 802 generated using the disclosed techniques. Here, review 808 has been replaced by review 902, which is actually a negative review and is therefore correctly displayed to users who wish to view negative reviews of the laptop. Because the disclosed implementation provides a machine learning model that generalizes well to unseen or low-frequency terms, it can correctly learn that the terms "amazing" and "critical" are not strongly associated with negative reviews, and therefore does not misclassify review 808 as a negative review. More generally, the machine learning model trained using the disclosed implementation can be used to determine whether to output an individual review in response to requests for negative and / or positive reviews based on the review sentiment predicted by the machine learning model.

[0136] Figure 10 The illustration depicts an example scenario 1000 of user 1002 interacting with client device 640. As described above, the client device may have a local application and / or interact with a server application that can provide search functionality. In this scenario, the user requests the application to determine if they have any email complaints regarding customer service. The application responds by asking the user if they would like to search for the string "customer service" in their emails. This search may only display results containing this specific string, and other relevant results may be omitted.

[0137] Figure 11The illustration shows example scenario 1100, where an application has access to a model with a relevance ranking layer that ranks queries based on their semantic similarity to multiple answers in the semantic space. This allows the application to provide flexible search capabilities. Here, a user queries for emails related to customer service, and the application is able to identify an email from a customer named Sara that relates to delayed delivery. The email may not contain the terms "customer" or "service" or alternative forms of these terms, and therefore may not be identifiable through a regular text search. However, even though there may be little textual similarity between Sara's email and these terms, the relevance ranking layer may still indicate that the query "customer service" is semantically similar to an email complaint about delayed delivery. Therefore, the application is able to identify Sara's email as a response to the user's query. More generally, a machine learning model trained using the disclosed implementation can be used to rank documents relative to the query, at least based on the similarity between the documents and the query as the output of the machine learning model.

[0138] Equipment implementation

[0139] As mentioned above Figure 6 As shown, system 600 includes several devices, including client device 610, server 620, server 630, and client device 640. It is also noted that not all device implementations can be illustrated, and other device implementations should be obvious to those skilled in the art from the descriptions above and below.

[0140] As used herein, the terms “device,” “computer,” “computing device,” “client device,” and / or “server device” can refer to a quantity of hardware processing power and / or hardware storage / memory capacity. Processing power can be provided by one or more hardware processors (e.g., hardware processing units / cores) that can execute computer-readable instructions to provide functionality. Computer-readable instructions and / or data can be stored on storage devices (such as storage devices / memory and / or databases). As used herein, the term “system” can refer to a single device, multiple devices, etc.

[0141] Storage resources can be internal or external to their associated corresponding devices. Storage resources can include any one or more of volatile or non-volatile memory, hard disk drives, flash memory devices, and / or optical storage devices (e.g., CDs, DVDs, etc.). As used herein, the term "computer-readable medium" can include signals. Conversely, the term "computer-readable storage medium" does not include signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media such as RAM, and non-volatile storage media such as hard disk drives, optical disks, and flash memory.

[0142] In some cases, the device is configured with general-purpose hardware processors and storage resources. In others, the device may comprise a System-on-a-Chip (SOC) type design. In an SOC design implementation, the functionality provided by the device can be integrated onto a single SOC or multiple coupled SOCs. One or more associated processors may be configured to coordinate with shared resources (e.g., memory, storage devices, etc.) and / or one or more dedicated resources (e.g., hardware blocks configured to perform specific functions). Therefore, the terms “processor,” “hardware processor,” or “hardware processing unit” as used herein may also refer to a central processing unit (CPU), graphics processing unit (GPU), controller, microcontroller, processor core, or other type of processing device suitable for implementation in traditional computing architectures, as well as SOC designs.

[0143] Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. Exemplary types of hardware logic components that may be used, such as but not limited to, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0144] In some configurations, any modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, modules / code may be provided during device manufacturing or by an intermediary preparing to sell the device to the end user. In other cases, the end user may install these modules / code later, such as by downloading and installing the executable code on the appropriate device.

[0145] It's also important to note that devices can typically have input and / or output capabilities. For example, computing devices can have various input mechanisms, such as keyboards, mice, touchpads, voice recognition, gesture recognition (e.g., using depth cameras, such as stereo or time-of-flight camera systems, infrared camera systems, RGB camera systems, or using accelerometers / gyroscopes, facial recognition, etc.). Devices can also have various output mechanisms, such as printers, monitors, etc.

[0146] It should also be noted that the devices described herein can operate independently or collaboratively to implement the described technologies. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via network 650. However, this is not an limitation; network 650 may include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0147] Various examples have been described above. Other examples are described below. One example includes a method executed on a computing device, the method comprising providing a machine learning model having one or more mapping layers, including at least a first mapping layer configured to map components of pre-trained examples to a first representation in a space, and performing a pre-training phase on the one or more mapping layers using the pre-trained examples, wherein the pre-training phase includes... The method also includes adding noise to the first representation of the components of the pre-trained examples to obtain a noise-adjusted first representation, and performing a self-supervised learning process to pre-train the one or more mapping layers using at least the first representation of the components of the pre-trained examples and the noise-adjusted first representation.

[0148] Another example may include any of the above and / or the following examples, wherein one or more mapping layers include a second mapping layer configured to map the first representation and the noise-adjusted first representation to a second representation, the second representation reflecting the context in which a single component appears together with other components in a pre-trained example.

[0149] Another example may include any of the above and / or the following examples, wherein the first mapping layer includes a dictionary encoder and the second mapping layer includes a transformer encoder.

[0150] Another example may include any of the above and / or the following examples, wherein the pre-trained example includes a sentence, the components include words or tokens in the sentence, and the self-supervised learning process includes at least one of the following: masking word prediction, masking token prediction, or next sentence prediction.

[0151] Another example may include any of the above and / or the following examples, wherein the method further includes performing a supervised learning process on the classification layer and one or more mapping layers after the pre-training phase.

[0152] Another example may include any of the above and / or the following examples, in which adversarial training or virtual adversarial training is used to perform the supervised learning process.

[0153] Another example may include any of the above and / or the following examples, wherein the classification layer is selected from a group including a single-sentence classification layer, a pairwise text similarity layer, and a pairwise text classification layer.

[0154] Another example may include any of the above and / or the following examples, where the pre-trained examples include images or videos, and one or more mapping layers include convolutional layers.

[0155] Another example may include any of the above and / or the following examples, where a supervised learning process trains a classification layer to predict the classification of objects in an image or video.

[0156] Another example may include any of the above and / or the following examples, wherein the method further includes performing one or more initial training iterations of the self-supervised learning process without a noise-adjusted first representation, and one or more subsequent training iterations of the self-supervised learning process with a noise-adjusted first representation.

[0157] Another example may include any of the above and / or the following examples, where adding noise includes using virtual adversarial training to regularize the training objective.

[0158] Another example may include any of the above and / or the following examples, wherein the training objective facilitates a smooth output distribution of the first representation pair of the components of the pre-trained example and the corresponding noise-adjusted first representation pair by the machine learning model.

[0159] Another example includes a system comprising a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to: receive input data, process the input data using a machine learning model having a first layer and a second layer to obtain a result, the first layer having been pre-trained in a pre-training phase using virtual adversarial training for a self-supervised learning task, and output the result.

[0160] Another example may include any of the above and / or the following examples, wherein the virtual adversarial training used in the pre-training phase involves adding noise to the representation of the components of the pre-trained examples used to adjust the parameters of the first layer.

[0161] Another example may include any of the above and / or the following examples, wherein the input data includes comments, the result characterizes the sentiment associated with the comments predicted by the machine learning model, and the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to determine, at least based on the sentiment predicted by the machine learning model, whether to output an individual comment in response to a request for a negative comment.

[0162] Another example may include any of the above and / or the following examples, wherein the input data includes a query, the results reflect the similarity between the query and multiple documents output by a machine learning model, and the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to rank the multiple documents relative to the query, at least based on the similarity of the machine learning model outputs.

[0163] Another example includes a computer-readable storage medium storing instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform actions, including: providing a machine learning model having one or more mapping layers, the one or more mapping layers including at least a first mapping layer configured to map components of data items to a first representation in space; performing one or more initial pre-training iterations of a learning process to train the machine learning model with the first representation; performing noise adjustment on the first representation to obtain a noise-adjusted first representation; and performing one or more subsequent pre-training iterations of the learning process to train the machine learning model with the noise-adjusted first representation.

[0164] Another example may include any of the above and / or the following examples, wherein the first representation includes an embedding vector, and the noise-adjusted first representation includes a noise-adjusted embedding vector.

[0165] Another example may include any of the above and / or the following examples, wherein performing noise adjustment includes determining the adversarial direction to be performed with noise adjustment.

[0166] Another example may include any of the above and / or the following examples, in which one or more subsequent pre-training iterations facilitate a smooth output distribution for predictions made by the machine learning model using the embedding vector and the noise-adjusted embedding vector.

[0167] in conclusion

[0168] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims, and other features and actions that a person skilled in the art will recognize are intended to fall within the scope of the claims.

Claims

1. A method executed on a computing device, the method comprising: A machine learning model is provided having one or more mapping layers, the one or more mapping layers including at least a first mapping layer, the first mapping layer being configured to map components of unlabeled pre-trained examples to a first representation in a space, wherein the unlabeled pre-trained examples include sentences, or the unlabeled pre-trained examples include images or videos; as well as Perform a pre-training phase on the one or more mapping layers using the unlabeled pre-trained example, wherein the pre-training phase includes: Noise is added to the first representation of the components of the unlabeled pre-trained example to obtain a noise-adjusted first representation; as well as Perform a self-supervised learning process to pretrain the one or more mapping layers using at least the first representation of the components of the unlabeled pre-trained examples and the noise-adjusted first representation.

2. The method of claim 1, wherein the one or more mapping layers include a second mapping layer configured to map the first representation and the noise-adjusted first representation to a second representation, the second representation reflecting the context in which a single component appears together with other components in the unlabeled pre-trained example.

3. The method of claim 2, wherein the first mapping layer comprises a dictionary encoder, and the second mapping layer comprises a transformer encoder.

4. The method of claim 3, wherein the unlabeled pre-trained example includes the sentence, the component includes words or tokens in the sentence, and the self-supervised learning process includes at least one of the following: masking word prediction, masking token prediction, or next sentence prediction.

5. The method according to claim 1, further comprising: Following the pre-training phase, a supervised learning process is performed on the classification layer and the one or more mapping layers.

6. The method of claim 5, wherein the supervised learning process is performed using adversarial training or virtual adversarial training.

7. The method of claim 5, wherein the classification layer is selected from the group consisting of a single-sentence classification layer, a pairwise text similarity layer, and a pairwise text classification layer.

8. The method of claim 5, wherein the unlabeled pre-trained example comprises the image or the video, and the one or more mapping layers comprise convolutional layers.

9. The method of claim 8, wherein the supervised learning process trains the classification layer to predict the classification of objects in the image or video.

10. The method of claim 1, further comprising: One or more initial training iterations of the self-supervised learning process are performed without the noise-adjusted first representation; as well as One or more subsequent training iterations of the self-supervised learning process are performed with the first noise-adjusted representation.

11. The method of claim 1, wherein adding noise includes using virtual adversarial training to regularize the training objective.

12. The method of claim 11, wherein the training objective facilitates a smooth output distribution of the machine learning model for the first representation of the component of the unlabeled pre-trained example and the corresponding noise-adjusted first representation pair.

13. The method of claim 1, wherein the self-supervised learning process involves updating the machine learning model by learning from the structure of the unlabeled pre-trained examples using the added noise.

14. A system comprising: Hardware processing unit; as well as The storage resource stores computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to: Receive input data; The input data is processed using a machine learning model with a first layer and a second layer to obtain results. The first layer has been pre-trained in the pre-training phase using virtual adversarial training for a self-supervised learning task involving unlabeled pre-trained examples, wherein the unlabeled pre-trained examples include sentences, or the unlabeled pre-trained examples include images or videos; and Output the results.

15. The system of claim 14, wherein the virtual adversarial training used in the pre-training phase involves adding noise to the representation of components of unlabeled pre-training examples used to adjust the parameters of the first layer.

16. The system of claim 15, wherein the input data includes comments, the result characterizes the sentiment associated with the comments predicted by the machine learning model, and when the computer-readable instructions are executed by the hardware processing unit, the hardware processing unit causes the hardware processing unit to: Whether to output an individual comment in response to a request for a negative comment is determined at least based on the sentiment predicted by the machine learning model.

17. The system of claim 15, wherein the input data includes a query, the result reflecting the similarity of the query to a plurality of documents output by the machine learning model, and when the computer-readable instructions are executed by the hardware processing unit, the hardware processing unit causes the hardware processing unit to: The plurality of documents are ranked relative to the query, at least based on the similarity output by the machine learning model.

18. A computer-readable storage medium storing instructions, said instructions, when executed by one or more processing devices, causing said one or more processing devices to perform an action, said action comprising: A machine learning model is provided having one or more mapping layers, the one or more mapping layers including at least a first mapping layer, the first mapping layer being configured to map components of unlabeled data items to a first representation in a space, wherein the unlabeled data items include sentences, or the unlabeled data items include images or videos; Perform one or more initial pre-training iterations of the learning process to train the machine learning model with a first representation of the components of the unlabeled data items; Noise adjustment is performed on the first representation of the components of the unlabeled data item to obtain a noise-adjusted first representation; as well as One or more subsequent pre-training iterations of the learning process are performed to train the machine learning model with the noise-adjusted first representation of the components of the unlabeled data items.

19. The computer-readable storage medium of claim 18, wherein the first representation includes an embedding vector, and the noise-adjusted first representation includes a noise-adjusted embedding vector.

20. The computer-readable storage medium of claim 19, wherein the execution noise adjustment includes determining an execution noise-adjusted adversarial direction.

21. The computer-readable storage medium of claim 20, wherein one or more subsequent pre-training iterations facilitate the machine learning model to produce a smooth output distribution for predictions made using the embedding vector and the noise-adjusted embedding vector.

Images