Systems and methods for pre-training models for diverse downstream tasks

Abstract

An exemplary method for pre-training a machine learning model is provided. The exemplary method includes obtaining configuration parameters for a plurality of different combinations of a pre-training object framework. The exemplary method includes generating a plurality of corrupted training examples from one or more training examples using the pre-training object framework, the plurality of corrupted training examples being respectively generated according to a plurality of different combinations. The exemplary method includes inputting the plurality of corrupted training examples into a machine learning model, the machine learning model being configured to generate uncorrupted sub-portions corresponding to corrupted sub-portions of the corrupted training examples. The exemplary method includes obtaining, from the machine learning model, a plurality of outputs, each generated by the machine learning model based on the plurality of corrupted training examples. The exemplary method includes updating one or more parameters of the machine learning model based on an evaluation of the plurality of outputs.

Description

[Technical field]

[0001] Claiming priority This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 305,910, filed February 2, 2022. U.S. Provisional Patent Application No. 63 / 305,910 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to pre-training machine learning models, and more particularly, to improved objects for pre-training. [Background technology]

[0003] Training of machine learning models can be completed in stages: models can be pre-trained for a general release and then fine-tuned for a specific task. Pre-training may involve unsupervised object hunting across an unlabeled training dataset, and the fine-tuning stage is often followed by supervised learning on a smaller labeled dataset. Summary of the Invention

[0004] Aspects and advantages of embodiments of the present disclosure are set forth in part in the detailed description that follows, or may be learned from the detailed description or through practice of the embodiments.

[0005] An exemplary aspect of the present disclosure is directed to an exemplary computer-implemented method for pre-training a machine learning model having a variety of objects. The exemplary method can include obtaining a plurality of different combinations of configuration parameters of a pre-training object framework. The exemplary method can include generating a plurality of corrupted training examples from one or more training examples using the pre-training object framework. The plurality of corrupted training examples can be generated according to a plurality of different combinations of configuration parameters, respectively. The exemplary method can include inputting the plurality of corrupted training examples into a machine learning model. The machine learning model can be configured to generate uncorrupted sub-portions corresponding to corrupted sub-portions of the corrupted training examples. The exemplary method can include obtaining, from the machine learning model, a plurality of outputs, each generated by the machine learning model based on the plurality of corrupted training examples. The exemplary method can include updating one or more parameters of the machine learning model based on an evaluation of the plurality of outputs.

[0006] In another aspect, an exemplary embodiment of the present disclosure provides an exemplary non-transitory computer-readable medium storing executable instructions for causing one or more processors to perform exemplary operations. The exemplary operations can include obtaining a plurality of different combinations of configuration parameters of a pre-training object framework. The exemplary operations can include generating a plurality of corrupted training examples from one or more training examples using the pre-training object framework. The plurality of corrupted training examples can be generated according to a plurality of different combinations of configuration parameters, respectively. The exemplary operations can include inputting the plurality of corrupted training examples into a machine learning model. The machine learning model can be configured to generate uncorrupted sub-portions corresponding to corrupted sub-portions of the corrupted training examples. The exemplary operations can include obtaining, from the machine learning model, a plurality of outputs, each generated by the machine learning model based on the plurality of corrupted training examples. The exemplary operations can include updating one or more parameters of the machine learning model based on an evaluation of the plurality of outputs.

[0007] In another aspect, an exemplary embodiment of the present disclosure provides an exemplary system that includes one or more processors and an exemplary non-transitory computer-readable medium.

[0008] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following detailed description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the detailed description, serve to explain associated principles.

[0010] A detailed discussion of embodiments directed to those skilled in the art is set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 illustrates a block diagram of an exemplary computing system for performing pre-training, according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 illustrates a block diagram of an exemplary computing device for performing pre-training, according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 illustrates a block diagram of an exemplary computing device for performing pre-training, according to an exemplary embodiment of the present disclosure. [Diagram 2] 1 illustrates a block diagram of an exemplary pre-training framework, according to an exemplary embodiment of the present disclosure. [Figure 3A] 1 illustrates a block diagram of an exemplary training example, according to an exemplary embodiment of the present disclosure. [Figure 3B] FIG. 1 illustrates a block diagram of an exemplary corruption training example, in accordance with an exemplary embodiment of the present disclosure. [Figure 4A] FIG. 1 illustrates a block diagram of an exemplary corruption training example, in accordance with an exemplary embodiment of the present disclosure. [Figure 4B] FIG. 1 illustrates a block diagram of an exemplary corruption training example, in accordance with an exemplary embodiment of the present disclosure. [Diagram 5] 1 illustrates a flowchart diagram of an exemplary method for performing pre-training, according to an exemplary embodiment of the present disclosure.

[0012] Reference numbers repeated among the drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] overview Exemplary aspects of the present disclosure provide systems and methods for pre-training machine learning models for diverse downstream tasks. In some embodiments, the systems and methods of the present disclosure leverage multiple pre-training objects to simulate diverse implementations. In some embodiments, the pre-training objects can be based on a pre-training object framework that provides efficient construction of a diverse set of pre-training objects by tuning parameters of a common framework.

[0014] The pre-training objects can be configured based on a shared pre-training object framework. For example, the denoising object framework can accommodate corrupting (e.g., "noisifying") one or more selected sub-portion(s) of a training example, and then predicting / recovering the selected sub-portion(s) based on the remaining portion of the training example, so that the original training example can be reconstructed (e.g., "denoising"). The diverse pre-training objects can be obtained by adjusting one or more configuration parameters of the shared pre-training object framework. For example, the one or more configuration parameters can characterize the amount of selected sub-portion(s), the size of the selected sub-portion(s), the ratio at which the selected sub-portion(s) are corrupted, etc.

[0015] Advantageously, systems and methods according to exemplary aspects of the present disclosure can provide a unified approach to model selection, development, and implementation. For example, in some embodiments, machine learning models can be configured to process sequential information (e.g., language strings, gene sequencing, other sequencing data). For example, models can be configured to understand, generate, respond to, or otherwise interact with sequences of data. Pre-training a model according to exemplary embodiments of the present disclosure can provide a "universal" model that is effective for performing a variety of different downstream tasks on sequencing data (e.g., the same or different sequencing data), with or without optional subsequent fine-tuning.

[0016] In contrast, conventional techniques refer to model selection based on downstream tasks. Excessive individual model placements, architectures, training recipes, training datasets, etc., when large, may be poorly informed or otherwise suboptimal for model implementation. Furthermore, even if a model can be appropriately selected for a given task, the model may need to be reconfigured or even replaced if the task or other requirements change. For example, conventional approaches for processing sequence decision data have often relied on different categories of pre-training approaches. For example, in the context of natural language processing, one conventional approach involves pre-training with language modeling objects that unidirectionally generate sequences of text based on preceding text content. Another approach involves pre-training with masked language objects that identify masked text based on surrounding text (e.g., bidirectional). However, these pre-training objects have generally been shown to be insufficient for diverse implementations. For example, open text generation and prompt-based learning may be unfavorable settings for conventional masked language objects, while conventional language modeling approaches may be overly hindered by purely unidirectional causality.

[0017] Thus, the systems and methods according to the exemplary aspects of the present disclosure may provide several technical effects and advantages over conventional approaches. For example, the unified approach according to the exemplary aspects of the present disclosure may provide for the implementation of a small number of models (e.g., one model) instead of many models (e.g., multiple models). This may reduce the computational complexity of model deployment, model training, model updates, model deactivation, etc. In this manner, for example, model operations according to the unified techniques disclosed herein may be performed using fewer computational resources. Less storage may be used to store a small number of models (e.g., one model) instead of many models (e.g., multiple models). Less network transmission may be used to implement a small number of models (e.g., one model) instead of many models (e.g., multiple models) on one or more remote device(s) (e.g., client devices connected to a server device). Efficiency of update and patch cycles may be improved by dedicating resources (e.g., computational resources, human resources, etc.) to managing and versioning a small number of models (e.g., one model) instead of many models (e.g., multiple models). By using models trained with a diverse pre-training approach according to exemplary aspects of the present disclosure, target performance with less computational overhead can be achieved by utilizing a small number of models (e.g., one model) instead of many models (e.g., multiple models). Lower latency can be achieved by using a small number of models (e.g., one model) instead of switching between many models (e.g., multiple models).

[0018] Furthermore, systems and methods according to exemplary aspects of the present disclosure can provide improved performance across task domains. For example, a diverse pre-training approach according to exemplary aspects of the present disclosure can provide improved (e.g., more accurate, more precise, less expensive, less prone to error, etc.) processing of model inputs across task domains. For example, in real-world deployment scenarios where tasks may not always be well categorized into separate domains, models trained with a diverse pre-training approach according to exemplary aspects of the present disclosure can provide improved real-world performance and perform well on mixed or cross-domain tasks.

[0019] Furthermore, systems and methods according to exemplary aspects of the present disclosure can provide improved robustness from diverse pre-training. For example, a model pre-trained according to exemplary aspects of the present disclosure with diverse pre-training objects can provide improved response to new or unfamiliar contexts with diverse exposure to different objects in pre-training. For example, traditional adversarial attacks may be less effective if the model is less susceptible to being corrupted by different inputs. In this way, furthermore, for example, a model pre-trained with diverse objects according to exemplary aspects of the present disclosure can provide improved robustness in real-world implementations where tasks may not necessarily be well categorized or curated.

[0020] Furthermore, systems and methods according to exemplary aspects of the present disclosure are well suited for pre-training Transformer models. For example, exemplary techniques described herein leverage the internal parallel structure and processing streams of the Transformer model to provide a variety of pre-training objects that respond bidirectionally to inputs to the model and recover corrupted inputs. In some embodiments, the Transformer model can include efficiently parallelized computation of multi-head attention. In this way, for example, examples of inherently parallelizable Transformer models can be better pre-trained for immediate deployment and / or further fine-tuning, and can offer improved scalability and distributed computation by leveraging a small number of Transformer models (e.g., one Transformer model) instead of many fluctuating models (e.g., multiple models) that may not provide the same benefits at scale.

[0021] Referring now to the figures, exemplary embodiments of the present disclosure will be described in more detail.

[0022] Exemplary Devices and Systems 1A illustrates a block diagram of an example computing system 100 capable of performing pre-training, according to an example embodiment of the present disclosure. System 100 may include a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0023] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0024] The user computing device 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that may be executed by the processor 112 to cause the user computing device 102 to perform operations.

[0025] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the models 120 may be or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models may leverage attention mechanisms, such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., transformer models).

[0026] In some implementations, one or more models 120 may be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single model 120.

[0027] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a service for processing data using the models). Thus, one or more models 120 may be stored and implemented on the user computing device 102 and / or one or more models 140 may be stored and implemented on the server computing system 130.

[0028] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may function to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0029] The server computing system 130 may include one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0030] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0031] As described above, the server computing system 130 may store or otherwise include one or more models 140. For example, the models 140 may be or otherwise include various machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models may leverage attention mechanisms such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., transformer models).

[0032] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 can be separate from the server computing system 130 or can be part of the server computing system 130.

[0033] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operably connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0034] The training computing system 150 can include a model trainer 160 that trains the machine learning models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as backpropagation of error. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update the parameters over a number of training iterations.

[0035] In some implementations, performing backpropagation of the error may include performing truncated backpropagation of the error through time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0036] In particular, model trainer 160 can train models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, supervised and / or unsupervised training data. In some embodiments, the training data includes sequenced data (e.g., textual data, such as words or other symbolic representations arranged in a sequence, such as genetic information, natural language, etc.), such as a sequence of data elements.

[0037] In some implementations, if the user consents, the training examples may be provided by the user computing device 102. Thus, in those implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0038] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0039] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may occur over any type of wired or wireless links, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0040] The machine learning models described herein may be used for a variety of tasks, applications, and / or use cases.

[0041] In some implementations, the input to the machine learning model(s) of the present disclosure can be image data. The machine learning model(s) can process the image data to generate an output. As an example, the machine learning model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedded representation of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model(s) can process the image data to generate an image segmentation output. As another example, the machine learning model(s) can process the image data to generate an image classification output. As another example, the machine learning model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine learning model(s) can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model(s) can process the image data to generate an upscaled image data output. As another example, the machine learning model(s) can process the image data to generate a prediction output.

[0042] In some implementations, the input to the machine learning model(s) of the present disclosure can be text or natural language data. The machine learning model(s) can process the text or natural language data to generate an output. As an example, the machine learning model(s) can process the natural language data to generate a language encoding output. As another example, the machine learning model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model(s) can process the text or natural language data to generate a classification output. As another example, the machine learning model(s) can process the text or natural language data to generate a text segmentation output. As another example, the machine learning model(s) can process the text or natural language data to generate a semantic latent output. As another example, the machine learning model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model(s) can process the text or natural language data to generate a prediction output. As another example, the machine learning model(s) can process text or natural language data to generate speech output (e.g., audio output).

[0043] In some implementations, the machine learning model(s) can process text or natural language data to generate a translation output. In some embodiments, the translation output can be in a different language than the text or natural language data. In some embodiments, the translation output can be in a different language than the set of training examples (e.g., pre-training examples). For example, the machine learning model(s) can optionally provide prompt-based zero-shot translation output.

[0044] In some implementations, the input to the machine learning model(s) of the present disclosure can be audio data. The machine learning model(s) can process the audio data to generate an output. As an example, the machine learning model(s) can process the audio data to generate a speech recognition output. As another example, the machine learning model(s) can process the audio data to generate a speech translation output. As another example, the machine learning model(s) can process the audio data to generate a potential embedding output. As another example, the machine learning model(s) can process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate a predicted output.

[0045] In some implementations, the input to the machine learning model(s) of the present disclosure can be latent encoded data (e.g., a latent space representation of the input, etc.). The machine learning model(s) can process the latent encoded data to generate an output. As an example, the machine learning model(s) can process the latent encoded data to generate a recognition output. As another example, the machine learning model(s) can process the latent encoded data to generate a reconstruction output. As another example, the machine learning model(s) can process the latent encoded data to generate a search output. As another example, the machine learning model(s) can process the latent encoded data to generate a reclustered output. As another example, the machine learning model(s) can process the latent encoded data to generate a prediction output.

[0046] In some implementations, the input to the machine learning model(s) of the present disclosure can be statistical data. The statistical data can be, represent, or otherwise include data that has been computed and / or calculated from some other data source. The machine learning model(s) can process the statistical data to generate an output. As an example, the machine learning model(s) can process the statistical data to generate a recognition output. As another example, the machine learning model(s) can process the statistical data to generate a prediction output. As another example, the machine learning model(s) can process the statistical data to generate a classification output. As another example, the machine learning model(s) can process the statistical data to generate a segmentation output. As another example, the machine learning model(s) can process the statistical data to generate a visualization output. As another example, the machine learning model(s) can process the statistical data to generate a diagnostic output.

[0047] In some implementations, the input to the machine learning model(s) of the present disclosure can be sensor data. The machine learning model(s) can process the sensor data to generate an output. As an example, the machine learning model(s) can process the sensor data to generate a recognition output. As another example, the machine learning model(s) can process the sensor data to generate a prediction output. As another example, the machine learning model(s) can process the sensor data to generate a classification output. As another example, the machine learning model(s) can process the sensor data to generate a segmentation output. As another example, the machine learning model(s) can process the sensor data to generate a visualization output. As another example, the machine learning model(s) can process the sensor data to generate a diagnostic output. As another example, the machine learning model(s) can process the sensor data to generate a detection output.

[0048] In some cases, the machine learning model(s) can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos), and the output may include compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0049] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data of one or more images and the task is an image processing task. For example, the image processing task can be image classification and the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to that object class. The image processing task can be object detection and the image processing output identifies one or more regions in the one or more images and, for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task can be image segmentation and the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predefined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation and the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted in pixels between the images in the network input.

[0050] In some cases, the input includes audio data representing spoken speech and the task is a speech recognition task. The output may include text output that is mapped to the spoken speech. In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes microprocessor performance tasks, such as branch prediction or memory address translation.

[0051] 1A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can also be used. For example, in some implementations, the user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0052] 1B illustrates a block diagram of an exemplary computing device 10 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 9 can be a user computing device or a server computing device.

[0053] Computing device 9 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model(s). For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0054] 1B, each application may communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0055] 1C illustrates a block diagram of an exemplary computing device 11 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 11 can be a user computing device or a server computing device.

[0056] Computing device 11 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., an API common to all applications).

[0057] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, each machine learning model can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model for all applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 11.

[0058] The central intelligence layer can communicate with a central device data layer, which can be a central repository of data for the computing device 11. As shown in FIG. 1C, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0059] Example Pre-Training Pipeline Arrangement FIG. 2 illustrates a block diagram of an example pre-training pipeline 200. The pre-training pipeline 200 can be configured to process training data 202 using an object framework 204. The object framework 204 can provide a plurality of configurations (e.g., object configurations 206, 208, 210, 212, etc.). Based on the plurality of object configurations, corrupted training data 214 can be obtained for input to a machine learning model 216 as a training example. The machine learning model 216 can generate recovery data 218, and an evaluator 220 can evaluate the performance of the machine learning model 216 in recovering the corrupted training data 214. Based on the evaluated performance, one or more parameters of the machine learning model 216 can be updated. In this manner, for example, the machine learning model 216 can be trained, such as during a pre-training iteration before a subsequent fine-tuning training iteration.

[0060] In general, corrupted training data 214 can include both corrupted and uncorrupted aspects of training data 202. In this manner, for example, one or more pre-training object(s) can be included that attempt to recover and / or reconstruct corrupted aspects of training data 202 to provide unsupervised training objects.

[0061] The machine learning model 216 may be provided with corrupted training data 214 and obtained as output recovery data 218. The output recovery data 218 may be evaluated by an evaluator 220 to determine one or more updates to the machine learning model 216 (e.g., updates to one or more parameters of the machine learning model 216).

[0062] In some embodiments, a training example of training data 202 may include a sequence of data elements (which may optionally be tokenized for processing, e.g., by an encoder and / or a decoder of a Transformer model). In some embodiments, a training example may be subdivided into one or more sub-portions to generate corrupted training examples.

[0063] For example, in some embodiments, a plurality of corrupted training examples (e.g., from corrupted training data 214) may be generated from one or more training examples (e.g., from training data 202). In some embodiments, each training example of the one or more training examples includes a sequence of data tokens. In some embodiments, the plurality of corrupted training examples are each generated according to a plurality of configurations (e.g., object configurations 206, 208, 210, 212, etc.) of a pre-training object framework (e.g., object framework 204). In some embodiments, each of the plurality of corrupted training examples includes one or more corrupted subportions of a sequence of data tokens.

[0064] In some embodiments, the multiple configurations can effectively interpolate between long-distance generative language modeling objects and local prefix-based modeling objects. Advantageously, each of the multiple object configurations can test the performance of the model 216 in a different way. For example, by bounding the model by bidirectional context (or future) (e.g., span breaks), the task can be made easier and closer to factual completion. On the other hand, the language modeling objects can be more open-ended. This behavior can be observed, for example, by monitoring the cross-entropy loss of the different object configurations.

[0065] In some embodiments, modal tokens can be added to the input to the machine learning model 216 to signal the mode or paradigm of pre-training. For example, it may be beneficial for the model 216 to not only distinguish between different object configurations during pre-training, but also to adaptively switch modes when learning downstream tasks. Modal tokens can advantageously facilitate mode switching. Mode switching can include associating pre-training tasks with dedicated sentinel tokens, allowing dynamic mode switching via discrete prompts.

[0066] The object framework 204 can provide a selection from multiple object configurations based on one or more parameter values. One parameter value can include a span length parameter. The span length parameter can be an average span length parameter. For example, the span lengths of a given corrupted training example can be sampled from a desired distribution (e.g., a normal distribution) with the mean value set by the span length parameter. For sequence-based objects, the span length parameter can be extended by constraining the span to the end of the input sequence so that no uncorrupted tokens appear after the corrupted span.

[0067] One parameter value may include a corruption rate, which may indicate the probability that a sub-portion of a span is corrupted. For example, the corruption rate may be expressed as a percentage, a proportion, etc.

[0068] One parameter value may include an amount of span. The amount of span may be a function of the length of the original input. The amount of span may be a function of the span length or the average span length. For example, the amount of span may be determined based on calculating the result of dividing the input length by the span length.

[0069] By parameterizing the object framework based on span length, breakage rate, and amount of spans, multiple different object configurations can be produced and different types of learning objects can be interpolated between. As an example, to build an object similar to causal language modeling using this formulation, one may set the span length to the length of the input span, the breakage rate to 100%, and the amount of span to 1 (e.g., a single broken span whose span length is equal to the length of the input sequence). To represent something similar to a prefix-based language modeling object, one may set the span length to the difference between the input sequence length and the prefix length, and the amount of span to a single, post-prefix span, with the added constraint that the single broken span reaches the end of the sequence. The breakage rate can be set, for example, to 100% minus the ratio of the prefix length to the input span length.

[0070] Multiple different object configurations can be used. For example, a first object configuration can be used for a training example. A second object configuration can be used for a second training example. A third object configuration can be used for a third training example. Alternatively, multiple different object configurations can be used for each training example.

[0071] Examples of mixed object configurations are described herein for three different types or classes of configurations. The first two types or classes of configurations below can be considered distributed configurations in that they can be configured to generate multiple corrupted spans distributed (e.g., randomly distributed) across an input sequence. The third type or class can be considered sequential configurations in that they can be configured to generate corrupted spans in a specific sequence (e.g., a sequence of uncorrupted inputs followed by a single span of corrupted inputs).

[0072] The first object configuration can be a configuration that implements a relatively short corrupted span. The first object configuration can include a relatively short corrupted span with a relatively low corruption rate. The first object configuration can be similar to a "regular" span corrupted object as introduced in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, & Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019. An exemplary first object configuration can include parameters for using a span length of about 2-5 tokens, or less than about 10 tokens, and corrupting about 15% of the input tokens. The first object configuration can be a mildly corrupted configuration.

[0073] The second object configuration can be a configuration that implements more extreme corruption. The second object configuration can include a longer span of corruption. The second object configuration can include a higher corruption rate. For example, an exemplary second object configuration can include a span of corruption that is greater than about 12 tokens long. In some examples, about half of the input can be split due to corruption. An exemplary second object configuration can include a corruption rate greater than about 30%, such as about 50% or more.

[0074] The third object configuration can be a configuration that implements a relatively long-form language generation. The third object configuration can be a sequence-based object. The third object configuration can be set up to provide a predefined sequence order of uncorrupted and corrupted spans. For example, the third object configuration can provide a prefix-based language modeling task. The third object configuration can partition the input sequence into two subsequences of tokens as context and target, such that the target is future information independent.

[0075] The pre-training pipeline 200 can utilize any of one or more object configurations from three different classes. The pre-training pipeline 200 can implement all three classes of object configurations. The pre-training pipeline 200 can implement one or more object configurations from each of the three classes. For example, multiple sets of configuration parameters can be used within each class. For example, objects in a mild class can be implemented with a span length of 3 together with a span length of 8 (e.g., in parallel), both with a corruption rate of 15%. Objects in a more extreme class can be implemented with a span length of 3, a span length of 8, a span length of 64 (all with a corruption rate of 50%), and a span length of 64 (with a corruption rate of 15%). Objects in a sequence-based class can be configured with various span lengths, such as one-quarter of the input sequence length, with a corruption rate of 25%. In this manner, each class can be implemented with a different configuration, for example, in parallel with training the model 216. For example, all of the seven examples provided above can be used during training of the model 216.

[0076] In FIG. 3A, a block diagram of training examples 302a, 304a, and 306a shows a plurality of training examples subdivided into sub-portions. Each sub-portion includes one or more data elements (e.g., tokens). According to a plurality of configurations (e.g., object configurations 206, 208, 210, 212, etc.), one or more sub-portions of training examples 302a, 304a, 306a can be selected for corruption. For example, a training example can be subdivided based on configuration parameters of the object framework that characterize the count of the sub-portion and / or characterize the span length of the sub-portion (e.g., the amount of tokens / elements in the sub-portion). Once one or more sub-portions are selected for corruption, a corruption rate configuration parameter can characterize the likelihood that the sub-portion is corrupted.

[0077] FIG. 3B illustrates multiple corrupted training examples 302b, 304b, 306b. The corrupted training examples 302b, 304b, and 306b can be derived from the same or different uncorrupted training examples (e.g., optionally corresponding to the training examples 302a, 304a, 306a) from the training data 202. Each of the corrupted training examples 302b, 304b, and 306b can include one or more selected sub-portions of corruption. In some embodiments, at least one sub-portion of each of the corrupted training examples 302, 304, and 306 can be corrupted. For example, sub-portions 2 and 4 of the corrupted training example 302 can be corrupted (but other sub-portions can be corrupted in addition to or instead of sub-portions 2 and 4). For example, sub-portion 2 of the corrupted training example 304 can be corrupted (but other sub-portions can be corrupted in addition to or instead of sub-portion 2). For example, sub-portion 2 of corrupted training example 306 may be corrupted (but other sub-portions may also be corrupted in addition to or instead of sub-portion 2). As shown, in some embodiments, the corrupted sub-portions may be replaced with corrupted tokens (e.g., optionally a separate token for each corrupted sub-portion).

[0078] In this way, for example, the machine learning model 216 can learn to recover a corrupted sub-portion by processing the corrupted sub-portion (e.g., processing a replacement or altered token(s) of the sub-portion).

[0079] The corruption training examples 302, 304, and 306 may be corrupted according to the same object configuration. Each of the corruption training examples 302, 304, and 306 may be corrupted according to a different object configuration. Each of the corruption training examples 302, 304, and 306 may be corrupted according to a range of object configurations, such as each of a set of configurations.

[0080] FIG. 4A shows one diagram of how training examples can be split into corrupted training examples based on configurations of the object framework.

[0081] Under the first object configuration, for example, the original text "Thank you for inviting me to your party last week" is rewritten as "Thank you <x>me to your party <y>can be damaged as "week" <x>and <y>Since is an optional separate replacement token, the machine learning model <x>"for inviting" and <y>This can be an example of mild object composition.

[0082] In the second, more extreme object configuration, for example, the original text becomes "Thank you <x>party <y>" can be corrupted as <x>and <y>Since is an optional separate replacement token, the machine learning model <x>"you for inviting me to your" and <y>You can target to get the "last week" of.

[0083] In the third object configuration, the original text is "Thank you for inviting me <x>" can be corrupted as <x>is a replacement token, so the machine learning model is <x>This can be an example of a prefix-based language modeling object.

[0084] In some embodiments, configuration parameters of the object framework can be selected to, for example, interpolate between language modeling objects (e.g., unidirectionally predicting a subsequent word(s) based on a preceding word(s)) and in-place reconstruction (e.g., bidirectionally filling gaps based on surrounding context). For example, as the corrupted sub-portion length increases, the object can, in some embodiments, approximate the language modeling object locally within the corrupted sub-portion. Thus, by implementing multiple configurations of the pre-training object framework according to exemplary aspects of the present disclosure, a diverse mix of pre-training objects can be generated.

[0085] In some embodiments, modal tokens can be added to the input to the machine learning model 216 to signal the pre-training mode or paradigm. For example, in FIG. 4A , "[R]" can indicate a modal token that indicates a "regular" or "mild" class of object. "[X]" can indicate a modal token that indicates a more extreme class of object. "[S]" can indicate a modal token that indicates a sequence-based language modeling object. Modal tokens can be used during pre-training, fine-tuning, and downstream tasks. In this way, for example, a "mode switch" can be invoked at inference time to engage the associated operational mode of the trained model.

[0086] FIG. 4B illustrates an example of the application of a mixture of object configurations to the same input sequence. In the first object configuration, the relatively few sub-portions 2, 4, 6, 8, and 10 are selected for corruption. As shown in FIG. 4B, the target of prediction by the model 216 starts from the modal token "[R]", which indicates a regular or milder class of object configuration. For example, the average span length of sub-portions 2, 4, 6, 8, and 10 can be, for example, about 5. The sampling span lengths can be, in one example, 3, 5, 4, 5, and 2, respectively.

[0087] The symbols "<{letter}>" can all be the same or individually selected (e.g., individually different) and can be used to index subportions 2, 4, 6, 8, and 10. For example, a target can be input to the model 216 (e.g., to a decoder component of the model) to trigger a prediction of the original token that corresponds to the corrupted span indicated in the target. For example, the placeholder token " " can be associated (e.g., individually associated) with sub-portion 4. The input can be " ". Thus, the model 216 can include a placeholder token for " " may be configured to predict that subportion 4 follows based on processing ". Thus, the target may be used to guide model 216 to predict an output sequence that includes corrupted subportions separated by corresponding placeholder token(s). For example, for the first object configuration, an example output might be " ability emotion or copied. <c>Noughts & <d>Ellis, <e>" In this manner, for example, the exemplary implementations can effectively provide a fill-in-the-blank solution for masked subportions of the input sequence.

[0088] For the second object configuration, multiple sets of configuration parameters can be used. For example, with the first set of configuration parameters (left column), the average span length can be long (e.g., 20 tokens, 30 tokens, 40 tokens, etc.). The amount of spans can be relatively small. For example, spans 14, 16, 18, and 20 can be selected for corruption. The individual sampled span lengths can be 16, 32, 24, and 24, respectively, in one example. With the second set of configuration parameters (right column), the average span length can be short (e.g., 3 tokens, 5 tokens, 8 tokens, etc.). The amount of spans can be relatively large. For example, spans 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, and 48 can be selected for corruption. The individual sampled span lengths can be 3, 3, 5, 4, 4, 5, 5, 3, 3, 2, 4, 4, 2, 4, and 5, respectively, in one example. As shown in FIG. 4B, this example composition target begins with a modal token "[X]," which indicates a more extreme class of object composition.

[0089] In a third object composition, sequence-based objects can be used. A single longer span 50 can be selected for breaking. For example, the span length can be 95. The span can be anchored to the end of the input sequence. As shown in FIG. 4B, the target of this example composition starts with a modal token “[S]” indicating a sequence-based class of object composition.

[0090] Example Results For the pre-training object, the example is compared to the following pre-training baseline:

[0091] Causal Language Model (CLM) - This is a pre-training of a standard left-to-right autoregressive language model and is used in many standard pre-trained models such as GPT (Radford et al., 2019; Brown et al., 2020). This disclosure refers to this model as GPT-like in our experiments.

[0092] Prefix LM (PLM) - This is a slight variant of causal LM introduced in (Liu et al., 2018; Raffel et al., 2019) where M has a bidirectional receptive field. For this baseline, we uniformly sample PLM to the length of M and only compute the loss on the autoregressive target.

[0093] Span Corruption (SC) - This is the standard denoising object proposed in T5 (Raffel et al., 2019). The idea is to blank out certain text parts and replace them with sentinel tokens. The text replaced with sentinel tokens is then copied into the target, autoregressively generated by the model. This baseline uses the default T5 setup, followed by an average span of 3 and a denoising rate of 15%.

[0094] Span Corruption + LM (SCLM) - This baseline trains on an equal mixture of CLM and span corruption. It uses the same hyperparameters of SC for the SC component of this object.

[0095] UniLM (ULM) - This is an object proposed in Dong et al. (2019).

[0096] For all objects, these results explore both single-stack and encoder-decoder architectures. All architectures are input to target implemented in either an encoder-decoder or decoder-only model structure. This is because we believe that BERT-style masked language modeling pre-training is already effectively incorporated by this style of pre-training, as shown empirically in (Raffel et al., 2019).

[0097] The dataset used is SuperGLUE (Wang et al., 2019), which contains eight NLU subtasks. The experiments also cover three datasets from the GEM benchmark (Gehrmann et al., 2021), which focus on language generation problems: XSUM (summary), ToTTo (table-to-text generation) (Parikh et al., 2020), and SGD (Schema Guided Dialog) (Rastogi et al., 2019) from the GEM benchmark. For all of these tasks, the results are evaluated both with supervised fine-tuning and with prompt-based one-shot learning. Finally, the results also compare the general ability of the models in text generation using the perplexity scores on the C4 validation set.

[0098] For SuperGLUE, these results report well-established metrics such as accuracy, F1 or exact match, where appropriate. For the GEM benchmark, these results use the Rouge-L metric. For language modeling, these results report negative log perplexity. Model universality, i.e., collective performance across the full range of tasks, is the primary evaluation criterion here. To allow comparison between models in this respect, these results use aggregate performance scores. However, metrics for various tasks can vary widely in nature, e.g., F1 and perplexity. To address this, these results choose to report and use normalized relative gains against baselines as the overall metric. To this end, these results use standard language models (decoder-only) (GPT-like) and standard span denoising encoder-decoder (T5) as primary baselines, and report all methods against their relative performance against these well-established candidates. The overall gains are normalized to these results, making it difficult to exploit or be susceptible to benchmark lottery effects.

[0099] All experiments are conducted in JAX / Flax (Bradbury et al., 2018) using the open source T5X4 framework (Roberts et al., 2022) and Flaxformer. In our experiments, we pre-train all models for 500K steps using the C4 corpus, with a batch size of 128 and sequence lengths of input 512 and target 512. The approximate total number of tokens seen during pre-training is about 32 billion tokens. Each pre-training run is typically trained using 64–128 TPUv4 chips (Jouppi et al., 2020).

[0100] The experiments optimize the examples using the Adafactor (Shazeer & Stern, 2018) optimizer with an inverse square root learning rate. The examples run all baseline pre-training objects using both decoder-only and encoder-decoder architectures. The results report the main experimental results using a base architecture with approximately 167M parameters for the decoder model and 335M parameters for the encoder-decoder model. All models use the standard transformer with a SwiGLU layer as described in (Shazeer, 2020).

[0101] This example uses the default T5 English 32K sentence pieces for all models. Within the context of the decoder-only model, except for the decoder model trained with a causal LM, this experiment uses bidirectional receptive fields only for its input segments and autoregressive decoding for the target segments.

[0102] Table 1 reports the raw results for all benchmark tasks and datasets. Our examples are denoted as "UL2". To facilitate comparison across setups, we also report our results relative to well-established baselines such as the T5 and GPT models, which are reported in Tables 2 and 3, respectively. [Table 1] [Table 2] [Table 3]

[0103] When using T5 as the reference baseline, none of the pre-trained decoder models outperform T5, except for the UL2 decoder. Furthermore, they experience a 10%-30% decrease in overall relative performance. The prefix LM decoder model performs about 10% worse than the T5 baseline. The UL2 decoder outperforms the T5 encoder-decoder setup by +14.6%.

[0104] Overall, UL2 outperforms T5 by +43.4% and +76.2% when compared to the GPT-like CLM decoder model. This is the highest relative gain (overall gain) compared to all other alternatives. In all individual tasks, UL2 outperforms T5 in all nine of the nine tasks considered. Thus, UL2 is a universally better option compared to the span-corrupted T5 model. UL2 is highly consistent. Even when it underperforms another method on a task, the difference is relatively small (e.g., 6.5 vs. 7.3 on one-shot TOTTO). Conversely, when UL2 outperforms a baseline such as T5, the gain can be as large as +363%. UL2 remains the most consistent and robust method. The consistent improvement also suggests that it can be used as a more consistent replacement for T5 and GPT-like models.

[0105] Ablation results are provided to confirm that mode switching ability can be effective on performance. One-shot XSum and one-shot SuperGLUE experiments were conducted. Table 4 reports the results of varying the paradigm prompts for the model. The results show that using correct or incorrect prompts can result in a 48% gap in performance (XSum, Rouge-1). On the other hand, SuperGLUE was less sensitive to prompting. In SuperGLUE, using prompts was almost always better than not using prompts during the one-shot evaluation. [Table 4] [Table 5]

[0106] Experiments are provided to test the effectiveness of individual objects within the object framework. Table 5 reports the results of these ablations. Table 5 reports the results of varying the mean span and breakage rate along with the percentage of S noise removal used (shown in SD%). In this test, the total number of configurations in the mix was span x breakage rate + 1. Table 5 labels these configurations Var-A through Var-L for easy reference to them.

[0107] Additional experiments are conducted by scaling up both 1) the model size and 2) the pre-training dataset size. We scale up the UL2 encoder-decoder model to roughly 1B parameters and increase the number of pre-training tokens to 0.5 trillion tokens.

[0108] Table 6 reports the results for this scale setting. At larger scales, our UL2 encoder-decoder model remains competitive. The difference here is that UL2 underperforms on the SuperGLUE suite versus T5(1B). However, this is more than compensated for by a 2-4x performance improvement in one-shot evaluations, as well as outperforming it on 7 out of 8 tasks. The gains in supervised fine-tuning are smaller, but still significant across XSUM, SGD and TOT. [Table 6]

[0109] This example was also evaluated with a model size of about 20B parameters. This experiment follows the same training protocol as the previous experiment, pretraining on the C4 corpus, but also scales the number of tokens the model sees during pretraining. In this experiment, we use batch sizes of 1024 and 512 TPUv4 chips to pretrain this model. The model is trained on C4 for a total of 1 trillion tokens (2 million steps). The sequence length is set to 512 / 512 for input and target. Dropout during pretraining is set to 0. The model has 32 encoder layers and 32 decoder layers, with dmodel of 4096 and dff of 16384. The dimension of each head is 256, for a total of 16 heads. This model uses a model parallelism of 8. The results retain the same sentence piece tokenizer as T5 with a vocabulary size of 32k. UL20B can therefore be interpreted as a model quite similar to T5, but trained on different objects and with slightly different scaling knobs. Similar to the previous experiments, UL20B is trained on the Jax and T5X infrastructures.

[0110] In total, about 50+ NLP tasks are considered in this experiment to demonstrate the generality of the approach. The list of tasks and their categorization are shown below. Note that the categorization of tasks is generally flexible and some tasks may span different category boundaries.

[0111] Language generation - summarization and data-to-text generation tasks. We evaluate our models against CNN / Dailymail (Hermann et al., 2015), XSUM (Narayan et al., 2018), MultiNews (Fabbri et al., 2019), SAMSum (Gliwa et al., 2019), WebNLG (Castro Ferreira et al., 2020) (English), E2E (Dusek et al., 2019) and CommonGen (Lin et al., 2020). For WebNLG, E2E and CommonGen, we use the versions from the GEM benchmark (Gehrmann et al., 2021).

[0112] Human-Rated Language Generation - We evaluate a variety of text generation tasks using human ratings via the GENIE Leaderboard (Khashabi et al., 2021). These tasks include aNLG (Bhagavatula et al., 2019), ARC-DA (Clark et al., 2018), WMT19 (Foundation), and XSUM (Narayan et al., 2018).

[0113] Language Understanding, Classification, and Question Answering - We use reading comprehension, question answering, text classification, and natural language inference datasets, including RACE (reading comprehension) (Lai et al., 2017), QASC (Khot et al., 2020), OpenBookQA (Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., 2019), VitaminC (Schuster et al., 2021a), Civil Comments, and Wikipedia toxicity detection datasets (Borkan et al., 2019). We use the standard SuperGLUE (Wang et al., 2019) and GLUE (Wang et al., 2018) datasets.

[0114] Common sense reasoning - HellaSwag (Zellers et al., 2019), SocialIQA / SIQA (Sap et al., 2019), PhysicalIQA / PIQA (Bisk et al., 2020), CosmosQA (Huang et al., 2019), AbductiveNLI (Bhagavatula et al., 2019), CommonsenseQA (Talmor et al. al., 2018) and CommonsenseQA2 (Talmor et al., 2021).

[0115] Long-range inference - We use the Scroll benchmark (Shaham et al., 2022), which consists of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Kocisky et al., 2018), QuaLITY (Pang et al., 2021), and ContractNLI (Koreeda & Manning, 2021).

[0116] We use several component tasks from Structured Knowledge Grounding-UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al., 2017), MTOP (Li et al., 2020) and DART (Nan et al., 2020). We choose datasets that are relatively convenient to perform the evaluation on and use mainstream metrics such as accuracy or exact match instead of ambiguous metrics or metrics that require significant domain-specific post-processing.

[0117] Information Retrieval-IR is the task of retrieving relevant documents for a given query. In our experiments, we use the latest next-generation IR paradigm, namely the setup of differentiable search indexes (Tay et al., 2022). We use the same NQ (Kwiatkowski et al., 2019) partitioning as in the DSI paper.

[0118] For each dataset, the previous best state-of-the-art (SOTA) result is provided. [Table 7-1] [Table 7-2] [Table 7-3]

[0119] UL2 achieves at least SOTA performance on about 50+ NLP tasks and setups. In many cases, the margins are very wide, and even when UL2 does not achieve SOTA, its performance is generally very competitive. The difficulty of obtaining SOTA on each benchmark varies significantly. In some, the SOTA model is 32B density equivalent (Zoph et al., 2022). For others, it is the base model.

[0120] Exemplary Methods 5 shows a flow chart diagram of an exemplary method performed according to an exemplary embodiment of the present disclosure. Although FIG. 5 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particular order or arrangement shown. Various steps of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0121] At 502, the example method 500 may include obtaining a plurality of different combinations of configuration parameters for a pre-training object framework. The pre-training object framework (e.g., including the pre-training pipeline 200) may include a parameterized corruption function configured to generate training examples according to one or more configuration parameters. For example, the parameterized corruption function may be configured to receive original training examples (e.g., sequences of text, etc.) and output corrupted training examples. The plurality of different combinations of configuration parameters may respectively correspond to a plurality of object configurations, such as object configurations 206-212. The plurality of different combinations of configuration parameters may be obtained from a configuration file or other parameter storage.

[0122] At 504, the example method 500 may include generating a plurality of corrupted training examples from the one or more training examples using the pre-training object framework. The plurality of corrupted training examples may each be generated according to a plurality of different combinations of configuration parameters. For example, the different corrupted training examples may be generated according to each of a plurality of different combinations of configuration parameters (e.g., according to each of a plurality of object configurations).

[0123] At 506, the example method 500 can include inputting a plurality of corrupted training examples into a machine learning model. The machine learning model can be configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples. For example, the machine learning model can be configured to perform next word generation based on the surrounding context. The machine learning model can be configured to bidirectionally utilize the uncorrupted tokens as inputs to predict the corrupted subportions.

[0124] At 508, the example method 500 may include obtaining, from the machine learning model, a plurality of outputs, each generated by the machine learning model based on the plurality of corrupted training examples.

[0125] At 510, the example method 500 may include updating one or more parameters of the machine learning model based on an evaluation of the plurality of outputs.

[0126] In some implementations of the example method 500, the configuration parameters may include two or more different parameters of a sub-portion length parameter, a sub-portion quantity parameter, or a breakage rate parameter.

[0127] In some implementations of the example method 500, the different combinations of configuration parameters may include a distributed configuration configured to generate multiple corrupted subportions distributed across the training examples, and a sequential configuration configured to generate corrupted subportions corresponding to the ends of the training examples.

[0128] In some implementations of the exemplary method 500, the different combinations of configuration parameters can include a first distribution configuration configured to generate a first plurality of corrupted subportions distributed across the training examples, a second distribution configuration configured to generate a second plurality of corrupted subportions distributed across the training examples, and a sequential configuration configured to generate corrupted subportions corresponding to the ends of the training examples. In some implementations of the exemplary method 500, the second distribution configuration can be configured to cause more corruption of the training examples than the first distribution configuration.

[0129] In some implementations of the exemplary method 500, compared to the first distribution configuration, the second distribution configuration may include at least one of a sub-portion length parameter corresponding to a longer sub-portion length or a failure rate parameter corresponding to a higher failure rate.

[0130] In some implementations of the example method 500, the sequential configuration may correspond to a prefix-based language modeling object.

[0131] In some implementations of the exemplary method 500, the plurality of different combinations of configuration parameters may include a first plurality of distributed configurations, each of which may be associated with a sub-portion length parameter indicative of a sub-portion length of less than about 12 tokens, and a second plurality of distributed configurations, each of which may be associated with at least one of a sub-portion length parameter indicative of a sub-portion length of more than about 12 tokens, or a corruption rate parameter indicative of a corruption rate of more than about 30%. In some implementations of the exemplary method 500, the plurality of different combinations of configuration parameters may include sequential configurations. In some implementations of the exemplary method 500, the plurality of different combinations of configuration parameters may include an amount of one or more sequential configurations such that the amount is less than about 50% of the total amount of the plurality of configurations. In some implementations of the exemplary method 500, the plurality of different combinations of configuration parameters may include an amount of one or more sequential configurations such that the amount is about 20% of the total amount of the plurality of configurations.

[0132] In some implementations of the example method 500, the first plurality of distribution configurations can each be associated with a sub-portion length parameter that indicates a sub-portion length of less than about 10 tokens.

[0133] In some implementations of the exemplary method 500, the second plurality of distributed configurations can each be associated with a sub-portion length parameter indicative of a sub-portion length of greater than about 12 tokens. In some implementations of the exemplary method 500, the second plurality of distributed configurations can each be associated with a sub-portion length parameter indicative of a sub-portion length of greater than about 30 tokens.

[0134] In some implementations of the exemplary method 500, the second plurality of distributed configurations can each be associated with a failure rate parameter indicative of a failure rate of greater than about 30%. In some implementations of the exemplary method 500, the second plurality of distributed configurations can each be associated with a failure rate parameter indicative of a failure rate of at least about 50%.

[0135] In some implementations of the example method 500, generating the plurality of corrupted training examples from the one or more training examples may include, for each training example of the one or more training examples (each training example including a respective sequence of data tokens), determining one or more selected sub-portions of the respective sequence of data tokens, and replacing the one or more selected sub-portions with replacement tokens.

[0136] In some implementations of the exemplary method 500, the exemplary method 500 may include inputting, with each corrupted training example of the plurality of corrupted training examples, a mode-switching token (e.g., a modal token such as "[R]", "[X]", "[S]", etc.) corresponding to at least one configuration of a plurality of different combinations of configuration parameters, where the at least one configuration is used to corrupt the respective corrupted training example.

[0137] In some implementations of the example method 500, the mode switch token can trigger downstream operation of the machine learning model corresponding to the task prioritized by the at least one configuration. For example, the mode switch token can be prepended to the runtime input (e.g., at inference time) based on the type of task associated with the runtime input. For example, a short-form generation task can use a mode switch token associated with a short-form corrupted span (e.g., "[R]"). A long-form generation task can use a mode switch token associated with a long-form corrupted span (e.g., "[X]" or "[S]").

[0138] In some implementations of the exemplary method 500, at least one of the corruption parameters can be a probability parameter. In some implementations of the exemplary method 500, the probability parameter can be a corrupted sub-portion length parameter that characterizes the distribution from which the selected sub-portion lengths are sampled. In some implementations of the exemplary method 500, the probability parameter can be a corruption rate parameter that characterizes the proportion of one or more selected sub-portions of the training examples that are corrupted.

[0139] In some implementations of the example method 500, the sequence of data tokens may correspond to a natural language.

[0140] In some implementations of the exemplary method 500, the sequence of data tokens may correspond to genetic data.

[0141] In some implementations of the example method 500, the sequence of data tokens may correspond to textual data.

[0142] In some implementations of the example method 500, the machine learning model may include a transformer encoder. In some implementations of the example method 500, the machine learning model may include a transformer decoder.

[0143] In some implementations of the example method 500, the example method 500 may include generating a first fine-tuned version of the machine learning model for a first task and generating a second fine-tuned version of the machine learning model for a second, different task.

[0144] In some implementations of the exemplary method 500, the first task can be at least one of a classification task or a sequence-to-sequence task. In some implementations of the exemplary method 500, the second, different task can be at least one of an open text generation or a prompt-based reasoning task.

[0145] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions between components. For example, the processes discussed herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0146] Although the subject matter of the present disclosure has been described in detail with respect to various specific and exemplary embodiments thereof, each example is provided for the purpose of illustration and not for the purpose of limiting the present disclosure. Those skilled in the art, upon understanding the above content, can easily make modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, changes, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents.

[0147] Aspects of the present disclosure have been described with respect to exemplary embodiments thereof. Any and all features of the following claims can be combined or rearranged in any possible manner, including combinations of claims not expressly recited in combination with each other, as the dependency of the exemplary claims recited herein should not be read as limiting the scope of possible combinations of features disclosed herein. Thus, the scope of the present disclosure is exemplary, not limiting, and the disclosure of the present subject matter is not intended to exclude including such modifications, variations, or additions to the subject matter as would be readily apparent to one of ordinary skill in the art. Furthermore, terms are described herein with lists of exemplary elements joined by conjunctions such as "and," "or," "but," and the like. It should be understood that such conjunctions are provided for illustrative purposes only. For example, clauses and other series of items joined by certain conjunctions such as "or" can refer to "and / or," "at least one of," "any combination of," and the like of the exemplary elements recited therein. Additionally, terms such as "based on" should be understood as "based at least in part on."< / e> < / d> < / c> < / x> < / x> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x>

Claims

1. 1. A computer-implemented method for pre-training a machine learning model with diverse objects, comprising: obtaining, by a computing system including one or more processors, configuration parameters for a plurality of different combinations of the pre-training object framework; generating, by the computing system, a plurality of corrupted training examples from one or more training examples using the pre-training object framework, the plurality of corrupted training examples being each generated according to the plurality of different combinations of configuration parameters; inputting, by the computing system, the plurality of corrupted training examples into the machine learning model, the machine learning model being configured to generate uncorrupted sub-portions corresponding to the corrupted sub-portions of the corrupted training examples; obtaining, by the computing system, from the machine learning model, a plurality of outputs each generated by the machine learning model based on the plurality of corrupted training examples; updating, by the computing system, one or more parameters of the machine learning model based on an evaluation of the plurality of outputs; A method comprising:

2. The method of claim 1 , wherein the configuration parameters include two or more different parameters of a sub-portion length parameter, a sub-portion quantity parameter, or a damage rate parameter.

3. The plurality of different combinations of configuration parameters are a distribution scheme configured to generate a plurality of corrupted sub-portions distributed across the training examples; a sequential arrangement configured to generate a corrupted subportion corresponding to an end of the training example; The method of claim 1 or 2, comprising:

4. The plurality of different combinations of configuration parameters are a first distribution arrangement configured to generate a first plurality of corrupted sub-portions distributed across the training examples; a second distribution arrangement configured to generate a second plurality of corrupted subportions distributed across the training examples, the second distribution arrangement configured to cause greater corruption of the training examples than the first distribution arrangement; and a sequential arrangement configured to generate a corrupted subportion corresponding to an end of the training example; The method according to any one of claims 1 to 3, comprising:

5. Compared to the first distributed configuration, the second distributed configuration has the following features: a sub-part length parameter corresponding to a longer sub-part length, or a breakage rate parameter corresponding to a higher breakage rate; The method of claim 4 , comprising at least one of:

6. The method of claim 1 , wherein the sequential structure corresponds to a prefix-based language modeling object.

7. The plurality of different combinations of configuration parameters are a first plurality of distributed configurations each associated with a sub-portion length parameter indicative of a sub-portion length of less than about 12 tokens; a second plurality of distributed configurations, each of which comprises: a sub-portion length parameter indicating a sub-portion length of greater than about 12 tokens; or A failure rate parameter indicating a failure rate of greater than about 30%; the second plurality of distributed configurations associated with at least one of the The method of any one of claims 1 to 6, comprising:

8. The method of claim 7 , wherein the first plurality of distribution configurations are each associated with a sub-portion length parameter indicative of a sub-portion length of less than about 10 tokens.

9. 9. The method of claim 7 or 8, wherein the second plurality of distribution configurations are each associated with a sub-portion length parameter indicative of a sub-portion length of greater than about 12 tokens.

10. 10. The method of claim 7, 8 or 9, wherein the second plurality of distribution configurations are each associated with a sub-portion length parameter indicative of a sub-portion length of greater than about 30 tokens.

11. 11. The method of claim 7, 8, 9, or 10, wherein the second plurality of distributed configurations are each associated with a failure rate parameter indicative of a failure rate of greater than about 30%.

12. 12. The method of claim 7, 8, 9, 10, or 11, wherein the second plurality of distribution configurations are each associated with a failure rate parameter indicative of a failure rate of at least 50%.

13. Generating a plurality of corrupted training examples from the one or more training examples includes: for each training example of the one or more training examples, the each training example comprises a respective sequence of data tokens; determining, by said computing system, one or more selected sub-portions of each of said sequences of data tokens; replacing, by the computing system, the one or more selected subportions with a replacement token; 13. The method of any one of claims 1 to 12, comprising:

14. 14. The method of claim 1, further comprising: inputting, by the computing system, a mode-switching token corresponding to at least one configuration of the plurality of different combinations of configuration parameters with a corrupted training example of each of the plurality of corrupted training examples, the at least one configuration being used to corrupt the respective corrupted training example.

15. 15. The method of claim 14, wherein the mode-switching token triggers downstream operation of the machine learning model corresponding to tasks prioritized by the at least one configuration.

16. The method of claim 1 , wherein at least one of the corruption parameters is a probability parameter.

17. The method of claim 16 , wherein the probability parameter is a sub-portion length parameter that characterizes the variance with which a selected sub-portion length is sampled.

18. The method of claim 1 , wherein the probability parameter is the corruption rate parameter characterizing the proportion in which one or more selected sub-portions of training examples are corrupted.

19. 19. The method of claim 1, wherein the sequence of data tokens corresponds to a natural language.

20. 20. The method of claim 1, wherein the sequence of data tokens corresponds to genetic data.

21. 21. The method of claim 1, wherein the sequence of data tokens corresponds to text data.

22. The method of claim 1 , wherein the machine learning model includes a transformer encoder.

23. The method of claim 1 , wherein the machine learning model comprises a transformer decoder.

24. generating, by the computing system, a first fine-tuned version of the machine learning model for a first task; generating, by the computing system, a second fine-tuned version of the machine learning model for a second, different task; and 24. The method of any one of claims 1 to 23, comprising:

25. 25. The method of claim 24, wherein the first task is at least one of a classification task or a sequence-to-sequence task, and the second, different task is at least one of an open text generation or a prompt-based reasoning task.

26. 26. A non-transitory computer-readable storage medium storing instructions executable to cause one or more processors to perform operations, the operations comprising the method of any one of claims 1 to 25.

27. one or more processors; 27. The non-transitory computer readable medium of claim 26; Including, the system.

Images