Systems and methods for pre-training models of diverse downstream tasks

Abstract

An exemplary method for pre-training a machine learning model is provided. The exemplary method includes obtaining configuration parameters for a plurality of different combinations of a pre-training object framework. The exemplary method includes generating a plurality of corrupted training examples from one or more training examples using the pre-training object framework, the plurality of corrupted training examples being respectively generated according to a plurality of different combinations. The exemplary method includes inputting the plurality of corrupted training examples into a machine learning model, the machine learning model being configured to generate uncorrupted sub-portions corresponding to corrupted sub-portions of the corrupted training examples. The exemplary method includes obtaining, from the machine learning model, a plurality of outputs, each generated by the machine learning model based on the plurality of corrupted training examples. The exemplary method includes updating one or more parameters of the machine learning model based on an evaluation of the plurality of outputs.

Description

Technical Field

[0001] Claim of Priority This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 305,910, filed on February 2, 2022. U.S. Provisional Patent Application No. 63 / 305,910 is hereby incorporated by reference in its entirety.

[0002] This disclosure generally relates to pre-training machine learning models. More particularly, this disclosure relates to improved objects for pre-training.

Background Art

[0003] The training of machine learning models can be complemented step by step. The model can be pre-trained for general release and then fine-tuned for specific tasks. Pre-training may include the pursuit of teacherless objects across an entire unlabeled training dataset, and in the fine-tuning stage, supervised learning on a smaller labeled dataset often follows.

Summary of the Invention

[0004] Aspects and advantages of embodiments of this disclosure may be shown in part in the following detailed description, may be learned from the detailed description, or may be learned through the practice of the embodiments.

[0005] An exemplary aspect of this disclosure relates to an exemplary computer implementation method for pretraining a machine learning model having diverse objects. The exemplary method may include obtaining several different combinations of configuration parameters for a pretraining object framework. The exemplary method may include using the pretraining object framework to generate several corrupted training examples from one or more training examples. The several corrupted training examples may each be generated according to several different combinations of configuration parameters. The exemplary method may include inputting the several corrupted training examples into a machine learning model. The machine learning model may be configured to generate uncorrupted subparts corresponding to corrupted subparts of the corrupted training examples. The exemplary method may include obtaining several outputs from the machine learning model, each generated by the machine learning model based on the several corrupted training examples. The exemplary method may include updating one or more parameters of the machine learning model based on an evaluation of the several outputs.

[0006] In another aspect, exemplary embodiments of the present disclosure provide exemplary non-temporary computer-readable media for storing executable instructions for causing one or more processors to perform exemplary operations. Exemplary operations may include retrieving multiple different combinations of configuration parameters of a pre-training object framework. Exemplary operations may include generating multiple corrupted training examples from one or more training examples using the pre-training object framework. Each of the corrupted training examples may be generated according to multiple different combinations of configuration parameters. Exemplary operations may include inputting the multiple corrupted training examples into a machine learning model. The machine learning model may be configured to generate uncorrupted sub-parts corresponding to corrupted sub-parts of the corrupted training examples. Exemplary operations may include retrieving multiple outputs from the machine learning model, each generated by the machine learning model based on the multiple corrupted training examples. Exemplary operations may include updating one or more parameters of the machine learning model based on an evaluation of the multiple outputs.

[0007] In another aspect, exemplary embodiments of the present disclosure provide an exemplary system comprising one or more processors and exemplary non-temporary computer-readable media.

[0008] Other aspects of this disclosure cover a variety of systems, apparatus, non-transient computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects and advantages of the various embodiments of this disclosure will be better understood by referring to the modes for carrying out the invention and the appended claims below. The appended drawings incorporated herein and constituting part thereof illustrate exemplary embodiments of this disclosure and, together with the modes for carrying out the invention, illustrate the relevant principles.

[0010] A detailed discussion of embodiments for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1A] A block diagram of an exemplary computing system for performing pretraining according to an exemplary embodiment of the present disclosure is shown. [Figure 1B] A block diagram of an exemplary computing device for performing pretraining according to an exemplary embodiment of the present disclosure is shown. [Figure 1C] A block diagram of an exemplary computing device for performing pretraining according to an exemplary embodiment of the present disclosure is shown. [Figure 2] A block diagram of an exemplary pre-training framework according to an exemplary embodiment of the present disclosure is shown. [Figure 3A] A block diagram of an exemplary training example according to an exemplary embodiment of the present disclosure is shown. [Figure 3B] A block diagram of an exemplary failure training example according to an exemplary embodiment of the present disclosure is shown. [Figure 4A] A block diagram of an exemplary failure training example according to an exemplary embodiment of the present disclosure is shown. [Figure 4B] A block diagram of an exemplary failure training example according to an exemplary embodiment of the present disclosure is shown. [Figure 5] A flowchart illustrating an exemplary method for performing pretraining according to an exemplary embodiment of the present disclosure is shown.

[0012] The repeated reference numbers across multiple drawings are intended to identify the same features in various embodiments. [Modes for carrying out the invention]

[0013] overview Exemplary embodiments of this disclosure provide systems and methods for pretraining machine learning models for a variety of downstream tasks. In some embodiments, the systems and methods of this disclosure leverage a plurality of pretraining objects to simulate a variety of implementations. In some embodiments, the pretraining objects may be based on a pretraining object framework that provides efficient construction of a variety of pretraining objects by tuning the parameters of a common framework.

[0014] Multiple pre-training objects can be constructed based on a shared pre-training object framework. For example, a denoising object framework can be used to reconstruct the original training example (e.g., denoising) by corrupting (e.g., "noising") one or more selected sub-parts of a training example, and then predicting / recovering the selected sub-parts based on the rest of the training example. Diverse multiple pre-training objects can be obtained by tuning one or more configuration parameters of the shared pre-training object framework. For example, one or more configuration parameters could characterize the amount of selected sub-parts, the size of selected sub-parts, the ratio at which selected sub-parts are corrupted, etc.

[0015] Advantageously, the systems and methods according to exemplary embodiments of this disclosure can provide a unified approach to model selection, development, and implementation. For example, in some embodiments, machine learning models can be configured to process sequential information (e.g., language strings, gene sequencing, and other sequence determination data). For example, a model can be configured to understand, generate, respond to, or otherwise interact with sequences of data. By pre-training a model according to exemplary embodiments of this disclosure, it is possible to provide a “universal” model that is effective for performing a variety of different downstream tasks on sequence determination data (e.g., the same or different sequence determination data), with or without subsequent fine-tuning.

[0016] In contrast, traditional techniques refer to model selection based on downstream tasks. Excessive individual model placements, architectures, training recipes, and training datasets, when voluminous, can lead to selections that are not well-informed or otherwise suboptimal for the model implementation. Furthermore, even if a model can be adequately selected for a given task, it may need to be reconfigured or even replaced if the task or other requirements change. For example, traditional approaches to processing sequence decision data have often relied on different categories of pre-training approaches. For instance, in the context of natural language processing, one traditional approach involves pre-training with language modeling objects that unidirectionally generate sequences of text based on preceding text content. Another approach involves pre-training with masked language objects that identify masked text based on surrounding text (e.g., bidirectional). However, these pre-training objects have generally been shown to be inadequate for diverse implementations. For example, open text generation and prompt-based learning may be undesirable settings for traditional masked language objects, while traditional language modeling approaches may be excessively hindered by purely unidirectional causality.

[0017] Accordingly, the systems and methods according to exemplary embodiments of this disclosure can provide several technical advantages and benefits over conventional approaches. For example, the unified approach according to exemplary embodiments of this disclosure can provide implementation of a small number of models (e.g., one model) instead of many models (e.g., multiple models). This can reduce the computational complexity of tasks such as model deployment, model training, model updating, and model deactivation. In this way, for example, model operations using the unified techniques disclosed herein can be performed using fewer computational resources. A small number of models (e.g., one model) can be stored using fewer storage than many models (e.g., multiple models). A small number of models (e.g., one model) can be implemented on one or more remote devices (e.g., client devices connected to a server device) instead of many models (e.g., multiple models) using fewer network transmissions. The efficiency of update and patch cycles can be improved by allocating resources (e.g., computational resources, human resources, etc.) to the management and version control of a small number of models (e.g., one model) instead of many models (e.g., multiple models). By using models trained with various pre-training approaches as described in the exemplary embodiments of this disclosure, target performance with less computational overhead can be achieved by leveraging a small number of models (e.g., one model) instead of many models (e.g., multiple models). Reduced latency can be achieved by using a small number of models (e.g., one model) instead of switching between many models (e.g., multiple models).

[0018] Furthermore, the systems and methods in exemplary embodiments of this disclosure can provide performance improvements across task domains. For example, the diverse pre-training approaches in exemplary embodiments of this disclosure can provide improved (e.g., more accurate, more precise, less expensive, less prone to error, etc.) processing of model inputs across task domains. For example, in real-world deployment scenarios where tasks may not necessarily be neatly categorized into separate domains, models trained with the diverse pre-training approaches in exemplary embodiments of this disclosure can provide improved real-world performance and run well on mixed or cross-domain tasks.

[0019] Furthermore, the systems and methods according to exemplary embodiments of this disclosure can provide improved robustness through diverse pretraining. For example, a model pretrained according to exemplary embodiments of this disclosure with diverse pretraining objects can provide improved responses to new or unfamiliar contexts by exposing it to diverse objects during pretraining. For example, if a model is less susceptible to corruption by different inputs, traditional adversarial attacks may be less effective. In this way, furthermore, for example, a model pretrained with diverse objects according to exemplary embodiments of this disclosure can provide improved robustness in real-world implementations where tasks do not necessarily need to be well categorized or curated.

[0020] Furthermore, systems and methods according to exemplary embodiments of the present disclosure are well-suited for pre-training a transformer model. For example, the exemplary techniques described herein utilize the internal parallel structure and processing streams of a transformer model to provide various pre-training objects that respond bidirectionally to the input to the model and recover corrupted inputs. In some embodiments, the transformer model can include efficiently parallelized computations of multi-head attention. In this way, for example, examples of inherently parallelizable transformer models can be better pre-trained for immediate deployment and / or further fine-tuning, and by leveraging a small number of transformer models (e.g., one transformer model) instead of many varying models (e.g., multiple models) that may not provide the same advantages at scale, improvements in scalability and distributed computing can be presented.

[0021] Referring now to the figures, exemplary embodiments of the present disclosure will be described in more detail.

[0022] Exemplary Devices and Systems FIG. 1A shows a block diagram of an exemplary computing system 100 capable of performing pre-training according to an exemplary embodiment of the present disclosure. The system 100 can include a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0023] The user computing device 102 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0024] The user computing device 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 114 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that can be executed by the processors 112 to cause the user computing device 102 to perform operations.

[0025] In some embodiments, the user computing device 102 may store or include one or more machine learning models 120. For example, the models 120 may be various machine learning models such as neural networks (e.g., deep neural networks), or other types of machine learning models including nonlinear and / or linear models, or may otherwise include them. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models may leverage attention mechanisms such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., transformer models).

[0026] In some embodiments, one or more Model 120s may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used by one or more processors 112, or otherwise implemented. In some embodiments, a user computing device 102 may implement multiple parallel instances of a single Model 120.

[0027] Alternatively, one or more machine learning models 140 may be contained in, or otherwise stored in, and implemented by a server computing system 130 that communicates with a user computing device 102 according to a client-server relationship. For example, a model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a service for processing data using the model). Thus, one or more models 120 may be stored and implemented in the user computing device 102, and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0028] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, a user input component 122 could be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is highly sensitive to touch from a user input object (e.g., a finger or stylus). The touch-sensitive component could function as an implementation of a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which the user can provide user input.

[0029] The server computing system 130 may include one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 134 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processors 132 to cause the server computing system 130 to perform operations.

[0030] In some embodiments, the server computing system 130 includes or is implemented by one or more server computing devices. If the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0031] As described above, the server computing system 130 can store or otherwise include one or more models 140. For example, the models 140 can be various machine learning models or otherwise include various machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models can leverage attention mechanisms such as self-attention. For example, some exemplary machine learning models can include multi-head self-attention models (e.g., transformer models).

[0032] The user computing device 102 and / or the server computing system 130 can train models 120 and / or 140 by interacting with a training computing system 150 which is communicatively connected via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0033] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be one processor or multiple processors operably connected. The memory 154 can include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processors 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices, or is otherwise implemented by them.

[0034] The training computing system 150 may include a model trainer 160, which trains machine learning models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as backpropagation. For example, a loss function may be backpropagated by the model(s), and one or more parameters of the model(s) may be updated (for example, based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. The parameters may be iteratively updated over the number of training iterations using gradient descent.

[0035] In some embodiments, performing error backpropagation may include performing diachronic censored error backpropagation. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0036] In particular, the model trainer 160 can train models 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, supervised and / or unsupervised training data. In some embodiments, the training data includes sequenced data (text data such as genetic information, words or other symbolic representations arranged in a sequence such as natural language), such as a sequence of data elements.

[0037] In some embodiments, if the user consents, training examples may be provided by the user computing device 102. Thus, in such embodiments, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as model personalization.

[0038] The model trainer 160 includes computer logic used to provide a desired function. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored in a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in RAM, a hard disk, or a tangible computer-readable storage medium such as optical or magnetic media.

[0039] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof, and may include any number of wired or wireless links. Generally, communication over Network 180 can take place over any type of wired or wireless link using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0040] The machine learning models described herein can be used for a variety of tasks, applications, and / or use cases.

[0041] In some embodiments, the input to the machine learning model(s) of this disclosure may be image data. The machine learning model(s) may process the image data to produce an output. For example, the machine learning model(s) may process the image data to produce an image recognition output (e.g., recognition of image data, latent embedding representation of image data, encoded representation of image data, hash of image data, etc.). Another example is that the machine learning model(s) may process the image data to produce an image segmentation output. Another example is that the machine learning model(s) may process the image data to produce an image classification output. Another example is that the machine learning model(s) may process the image data to produce an image data modification output (e.g., modification of image data, etc.). Another example is that the machine learning model(s) may process the image data to produce an encoded image data output (e.g., encoded and / or compressed representation of image data, etc.). Another example is that the machine learning model(s) may process the image data to produce an upscaled image data output. Another example is that the machine learning model(s) may process the image data to produce a prediction output.

[0042] In some embodiments, the input to the machine learning model(s) of this disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to produce an output. For example, the machine learning model(s) may process natural language data to produce a language-encoded output. As another example, the machine learning model(s) may process text or natural language data to produce a latent text embedding output. As yet another example, the machine learning model(s) may process text or natural language data to produce a classification output. As yet another example, the machine learning model(s) may process text or natural language data to produce a text segmentation output. As yet another example, the machine learning model(s) may process text or natural language data to produce a semantic latent output. As yet another example, the machine learning model(s) may process text or natural language data to produce an upscaled text or natural language output (e.g., higher quality text or natural language data than the input text or natural language). As yet another example, the machine learning model(s) may process text or natural language data to produce a predictive output. As another example, a machine learning model(s) can process text or natural language data to generate speech output (e.g., audio output).

[0043] In some embodiments, a machine learning model(s) can process text or natural language data to generate a translated output. In some embodiments, the translated output may be in a different language than the text or natural language data. In some embodiments, the translated output may be in a different language than the set of training examples (e.g., pre-training examples). For example, a machine learning model(s) can optionally provide a prompt-based, zero-shot translated output.

[0044] In some embodiments, the input to the machine learning model(s) of this disclosure may be audio data. The machine learning model(s) may process the audio data to produce an output. For example, the machine learning model(s) may process the audio data to produce a speech recognition output. As another example, the machine learning model(s) may process the audio data to produce a speech translation output. As yet another example, the machine learning model(s) may process the audio data to produce a latent embedding output. As yet another example, the machine learning model(s) may process the audio data to produce an encoded audio output (e.g., an encoded and / or compressed representation of the audio data). As yet another example, the machine learning model(s) may process the audio data to produce an upscaled audio output (e.g., audio data of higher quality than the input audio data). As yet another example, the machine learning model(s) may process the audio data to produce a text representation output (e.g., a text representation of the input audio data). As yet another example, the machine learning model(s) may process the audio data to produce a predictive output.

[0045] In some embodiments, the input to the machine learning model(s) of this disclosure may be latent encoded data (e.g., a latent spatial representation of the input). The machine learning model(s) may process the latent encoded data to produce an output. For example, the machine learning model(s) may process the latent encoded data to produce a recognition output. As another example, the machine learning model(s) may process the latent encoded data to produce a reconstruction output. As yet another example, the machine learning model(s) may process the latent encoded data to produce a search output. As yet another example, the machine learning model(s) may process the latent encoded data to produce a reclustered output. As yet another example, the machine learning model(s) may process the latent encoded data to produce a prediction output.

[0046] In some embodiments, the input to the machine learning model(s) of this disclosure may be statistical data. Statistical data may be, represent, or otherwise include computer-processed and / or computed data from some other data source. The machine learning model(s) may process the statistical data to produce an output. For example, the machine learning model(s) may process the statistical data to produce a recognition output. As another example, the machine learning model(s) may process the statistical data to produce a prediction output. As yet another example, the machine learning model(s) may process the statistical data to produce a classification output. As yet another example, the machine learning model(s) may process the statistical data to produce a segmentation output. As yet another example, the machine learning model(s) may process the statistical data to produce a visualization output. As yet another example, the machine learning model(s) may process the statistical data to produce a diagnostic output.

[0047] In some embodiments, the input to the machine learning model(s) of this disclosure may be sensor data. The machine learning model(s) may process the sensor data to generate outputs. For example, the machine learning model(s) may process the sensor data to generate a recognition output. As another example, the machine learning model(s) may process the sensor data to generate a prediction output. As yet another example, the machine learning model(s) may process the sensor data to generate a classification output. As yet another example, the machine learning model(s) may process the sensor data to generate a segmentation output. As yet another example, the machine learning model(s) may process the sensor data to generate a visualization output. As yet another example, the machine learning model(s) may process the sensor data to generate a diagnostic output. As yet another example, the machine learning model(s) may process the sensor data to generate a detection output.

[0048] In some cases, a machine learning model(s) may be configured to perform a task that involves encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may contain audio data, and the output may contain compressed audio data. In another example, the input may contain visual data (e.g., one or more images or videos), and the output may contain compressed visual data, and the task is a visual data compression task. In yet another example, the task may involve generating embeddings for input data (e.g., input audio or visual data).

[0049] In some cases, the input includes visual data, and the task is a computer vision task. In other cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, an image processing task could be image classification, and the output would be a set of scores, each corresponding to a different object class, representing the likelihood that one or more images depict objects belonging to that object class. An image processing task could be object detection, and the image processing output would identify one or more regions within one or more images, and for each region, the likelihood that the region depicts an object of interest. As another example, an image processing task could be image segmentation, and the image processing output would define the likelihood for each pixel within one or more images, for each category within a given set of categories. For example, the set of categories could be foreground and background. As yet another example, the set of categories could be object classes. As yet another example, an image processing task could be depth estimation, and the image processing output would define the depth value for each pixel within one or more images. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted between the images in the network input, with each pixel in one of the input images representing the motion of the scene.

[0050] In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include text output mapped to the spoken utterances. In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes microprocessor performance tasks, such as branch prediction or memory address translation.

[0051] Figure 1A shows one exemplary computing system that can be used to implement the present disclosure. Other computing systems can also be used. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, the model 120 can be trained locally on the user computing device 102 or used. In some such embodiments, the user computing device 102 may implement the model trainer 160 for personalizing the model 120 based on user-specific data.

[0052] Figure 1B shows a block diagram of an exemplary computing device 10 that operates according to an exemplary embodiment of the present disclosure. The computing device 9 may be a user computing device or a server computing device.

[0053] Computing device 9 contains several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model(s). For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, and a browser application.

[0054] As shown in Figure 1B, each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0055] Figure 1C shows a block diagram of an exemplary computing device 11 that operates according to an exemplary embodiment of the present disclosure. The computing device 11 may be a user computing device or a server computing device.

[0056] The computing device 11 includes several applications (e.g., applications 1 to N). Each application communicates with the central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, and the like. In some embodiments, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., an API common to all applications).

[0057] The central intelligence layer includes several machine learning models. For example, as shown in Figure 1C, each machine learning model may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model to all applications. In some embodiments, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 11.

[0058] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data from the computing device 11. As shown in Figure 1C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, device state components, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0059] Exemplary pre-training pipeline configuration Figure 2 shows a block diagram of an exemplary pre-training pipeline 200. The pre-training pipeline 200 can be configured to process training data 202 using an object framework 204. The object framework 204 can provide multiple configurations (e.g., object configurations 206, 208, 210, 212, etc.). Based on multiple object configurations, corrupted training data 214 can be acquired as training examples for input to the machine learning model 216. The machine learning model 216 can generate recovery data 218, and the evaluator 220 can evaluate the performance of the machine learning model 216 in recovering the corrupted training data 214. Based on the evaluated performance, one or more parameters of the machine learning model 216 can be updated. In this way, the machine learning model 216 can be trained, for example, during pre-training iterations before subsequent fine-tuning training iterations.

[0060] In general, corrupted training data 214 can include both corrupted and uncorrupted forms of the training data 202. In this way, for example, one or more pre-training objects may attempt to recover and / or reconstruct the corrupted forms of the training data 202, providing unsupervised training objects.

[0061] The machine learning model 216 is provided with corrupted training data 214, which can be obtained as output recovery data 218. The output recovery data 218 can be evaluated by the evaluator 220 to determine one or more updates to the machine learning model 216 (for example, updates to one or more parameters of the machine learning model 216).

[0062] In some embodiments, the training examples of the training data 202 may include sequences of data elements (which may optionally be tokenized for processing, for example, by an encoder and / or decoder of a transformer model). In some embodiments, the training examples may be subdivided into one or more subparts to generate corrupted training examples.

[0063] For example, in some embodiments, multiple corrupted training examples (e.g., those of corrupted training data 214) can be generated from one or more training examples (e.g., those of training data 202). In some embodiments, each of the one or more training examples includes a sequence of data tokens. In some embodiments, each of the multiple corrupted training examples is generated according to one of several configurations (e.g., object configurations 206, 208, 210, 212, etc.) of a pre-trained object framework (e.g., object framework 204). In some embodiments, each of the multiple corrupted training examples includes one or more corrupted sub-parts of a sequence of data tokens.

[0064] In some embodiments, multiple configurations can effectively interpolate between long-range generative language modeling objects and local prefix-based modeling objects. Advantageously, each of the multiple object configurations can test the performance of Model 216 in different ways. For example, defining the model boundaries by bidirectional context (or future) (e.g., span breakage) can simplify the task and bring it closer to factual interpolation. Meanwhile, language modeling objects can be more open-ended. This behavior can be observed, for example, by monitoring the cross-entropy losses of different object configurations.

[0065] In some embodiments, modal tokens can be added to the input of the machine learning model 216 to signal the mode or paradigm of pretraining. For example, it may be beneficial not only for the model 216 to distinguish between different object configurations during pretraining, but also to adaptively switch modes when learning downstream tasks. Modal tokens can, advantageously, facilitate mode switching. Mode switching may involve associating the pretraining task with a dedicated sentinel token, and dynamic mode switching via discrete prompts may be possible.

[0066] The object framework 204 can provide a selection from multiple object configurations based on one or more parameter values. One parameter value may include a span length parameter. The span length parameter can be an average span length parameter. For example, the span length of a given corrupted training example can be sampled from a desired distribution (e.g., a normal distribution) using the mean value set by the span length parameter. For sequence-based objects, the span length parameter can be extended by constraining the span at the end of the input sequence so that no uncorrupted tokens appear after a corrupted span.

[0067] One parameter value can include the failure rate. The failure rate can indicate the probability that a sub-part of a span is damaged. For example, the failure rate can be expressed as a percentage, a proportion, or the like.

[0068] One parameter value can include the span quantity. The span quantity can be a function of the length of the original input. The span quantity can be a function of the span length or the average span length. For example, the span quantity can be determined based on calculating the result of dividing the input length by the span length.

[0069] By parameterizing the object framework based on span length, corruption rate, and span quantity, multiple different object configurations can be obtained, allowing for interpolation between different types of training objects. As an example, to construct an object similar to causal language modeling using this formulation, the span length may be set to the length of the input span, the corruption rate to 100%, and the span quantity to 1 (e.g., a single corrupted span whose span length is equal to the length of the input sequence). To represent something similar to a prefix-based language modeling object, the span length may be set to the difference between the input sequence length and the prefix length, and the span quantity to a single, post-prefixed span, with the added constraint that a single corrupted span reaches the end of the sequence. The corruption rate may be set to, for example, 100% minus the ratio of the prefix length to the input span length.

[0070] Multiple different object configurations can be used. For example, the first object configuration can be used for the training example. The second object configuration can be used for the second training example. The third object configuration can be used for the third training example. Alternatively, multiple different object configurations can be used for each training example.

[0071] Mixed examples of object configurations are described herein for three different types or classes of configurations. The first two types or classes of configurations below can be considered distributed configurations in that they can be configured to produce multiple corrupted spans that are distributed (e.g., randomly distributed) across an input sequence. The third type or class can be considered sequential configurations in that they can be configured to produce corrupted spans in a specific sequence (e.g., a single span of corrupted inputs following a sequence of uncorrupted inputs).

[0072] The first object configuration can be one that implements a relatively short corruption span. The first object configuration can include a relatively short corruption span with a relatively low corruption rate. The first object configuration can be similar to a “regular” span corruption object, such as the one introduced in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, & Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019. An exemplary first object configuration can use a span length of about 2 to 5 tokens, or less than about 10 tokens, and can include parameters to corrupt about 15% of the input tokens. The first object configuration can be a mild corruption configuration.

[0073] A second object configuration can be one that implements more extreme corruption. A second object configuration can include longer spans of corruption. A second object configuration can include higher corruption rates. For example, an exemplary second object configuration can include a corruption span with a length of over approximately 12 tokens. In some examples, about half of the input can be split for corruption. An exemplary second object configuration can include corruption rates exceeding approximately 30%, such as over 50%.

[0074] The third object configuration can be one that implements relatively long-form language generation. The third object configuration can be a sequence-based object. The third object configuration can be set up to provide a predetermined sequence order of unbroken spans and broken spans. For example, the third object configuration can provide a prefix-based language modeling task. The third object configuration can partition the input sequence into two subsequences of tokens, context and target, so that the target does not depend on future information.

[0075] The pre-training pipeline 200 can utilize one or more object configurations from three different classes. The pre-training pipeline 200 can implement all three classes of object configurations. The pre-training pipeline 200 can implement one or more object configurations from each of the three classes. For example, multiple sets of configuration parameters can be used within each class. For example, objects in the mild class can be implemented with span lengths of 3 and 8 (e.g., in parallel), both with a 15% failure rate. Objects in the more extreme class can be implemented with span lengths of 3, 8, 64 (all with a 50% failure rate), and 64 (with a 15% failure rate). Objects in the sequence-based class can be configured with a 25% failure rate and various span lengths, such as one-quarter of the input sequence length. In this way, each class can be implemented with different configurations in parallel with, for example, training model 216. For example, all seven examples provided above can be used during the training of model 216.

[0076] In Figure 3A, block diagrams of training examples 302a, 304a, and 306a show multiple training examples subdivided into sub-parts. Each sub-part contains one or more data elements (such as tokens). According to multiple configurations (e.g., object configurations 206, 208, 210, 212, etc.), one or more sub-parts of training examples 302a, 304a, and 306a can be selected for corruption. For example, a training example can be subdivided based on configuration parameters of an object framework that characterize the sub-part's counts and / or the sub-part's span length (e.g., the amount of tokens / elements in the sub-part). Once one or more sub-parts are selected for corruption, corruption rate configuration parameters can characterize the likelihood that the sub-parts are corrupted.

[0077] Figure 3B shows several corrupted training examples 302b, 304b, and 306b. Corrupted training examples 302b, 304b, and 306b can be derived from the training data 202 from the same or different uncorrupted training examples (for example, optionally corresponding to training examples 302a, 304a, and 306a). Each of the corrupted training examples 302b, 304b, and 306b may contain one or more selected subparts of corruption. In some embodiments, at least one subpart of each of the corrupted training examples 302, 304, and 306 may be corrupted. For example, subparts 2 and 4 of corrupted training example 302 may be corrupted (but other subparts may also be corrupted in addition to subparts 2 and 4). For example, subpart 2 of corrupted training example 304 may be corrupted (but other subparts may also be corrupted in addition to subpart 2). For example, sub-part 2 of the corrupted training example 306 may be corrupted (but other sub-parts may also be corrupted in addition to sub-part 2, or instead). As shown in the figure, in some embodiments, corrupted sub-parts can be replaced with corrupted tokens (e.g., a separate token for each corrupted sub-part, at an optional cost).

[0078] In this way, for example, the machine learning model 216 can learn to recover the corrupted subpart by processing it (for example, by processing the subpart's replacement or altered token(s)).

[0079] Breakdown training examples 302, 304, and 306 may break according to the same object configuration. Breakdown training examples 302, 304, and 306 may each break according to a different object configuration. Breakdown training examples 302, 304, and 306 may each break according to a set of object configurations, such as each of a set of configurations.

[0080] Figure 4A shows one diagram illustrating how training examples can be divided into multiple corrupted training examples based on multiple configurations of the object framework.

[0081] Under the first object structure, for example, the original text "Thank you for inviting me to your party last week" becomes "Thank you <x>me to your party <y>It can be damaged as a "week". <x>and <y>Since these are optional and individual substitution tokens, the machine learning model will <x>"for inviting" and <y>The target can be to obtain the "last" value. This can be an example of a minor object construction.

[0082] In the second, more extreme object configuration, for example, the original text is "Thank <x>party <y>It can be damaged as " <x>and <y>Since these are optional and individual substitution tokens, the machine learning model will <x>"you for inviting me to your" and <y>You can target to retrieve the "last week" value.

[0083] In the third object configuration, the original text reads, "Thank you for inviting me <x>It can be damaged as " <x>Since it is a substitution token, the machine learning model will <x>You can target to retrieve "to your party last week". This could be an example of a prefix-based language modeling object.

[0084] In some embodiments, the configuration parameters of the object framework can be selected to interpolate between, for example, language modeling objects (e.g., predicting subsequent words(or words) in one direction based on preceding words(or words)) and in-place reconstructions (e.g., filling gaps in both directions based on the surrounding context). For example, as the length of the corrupted sub-part increases, the object can, in some embodiments, approximate the language modeling object locally within the corrupted sub-part. Thus, by implementing multiple configurations of the pre-trained object framework according to exemplary embodiments of this disclosure, a diverse mixture of pre-trained objects can be generated.

[0085] In some embodiments, modal tokens can be added to the input of the machine learning model 216 to signal a pre-training mode or paradigm. For example, in Figure 4A, "[R]" can represent a modal token indicating an object of the "regular" or "mild" class. "[X]" can represent a modal token indicating an object of the more extreme class. "[S]" can represent a modal token indicating a sequence-based language modeling object. Modal tokens can be used during pre-training, fine-tuning, and downstream tasks. Thus, for example, "mode switching" can be invoked at inference to engage the trained model in the relevant operating mode.

[0086] Figure 4B shows an example of applying a mixture of object configurations to the same input sequence. In the first object configuration, relatively few sub-parts 2, 4, 6, 8, and 10 are selected for corruption. As shown in Figure 4B, the target of prediction by Model 216 starts with the modal token "[R]" which represents a regular or less severe class of object configuration. For example, the average span length of sub-parts 2, 4, 6, 8, and 10 could be, for example, about 5. The sampling span lengths could be 3, 5, 4, 5, and 2, respectively, in one example.

[0087] The symbol "<{letter}>" can all be the same or individually selected (e.g., individually different) and can be used to index subparts 2, 4, 6, 8, and 10. For example, the target can be input into model 216 (e.g., into the decoder component of the model) to trigger a prediction of the original token corresponding to the corrupted span indicated in the target. For example, placeholder token " " can be associated with subpart 4 (for example, individually). The input is " It can include a placeholder token corresponding to ". Therefore, Model 216 is " It can be configured to predict that subpart 4 follows based on processing ". Thus, the target can be used to guide model 216 to predict an output sequence containing corrupted subparts separated by the corresponding placeholder token(s). For example, for the first object configuration, the exemplary output is " yes emotion or copied. <c>Noughts & <d>Ellis, <e>This can be the case. In this way, for example, the exemplary embodiment can effectively provide a fill-in-the-blank solution for the masked sub-part of the input sequence.

[0088] In the case of the second object configuration, multiple sets of configuration parameters can be used. For example, in the first set of configuration parameters (left column), the average span length can be long (e.g., 20 tokens, 30 tokens, 40 tokens, etc.). The span amount can be relatively small. For example, spans 14, 16, 18, and 20 can be selected for corruption. Individual sampling span lengths can be 16, 32, 24, and 24, respectively, in one example. In the second set of configuration parameters (right column), the average span length can be short (e.g., 3 tokens, 5 tokens, 8 tokens, etc.). The span amount can be relatively large. For example, spans 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, and 48 can be selected for corruption. Individual sampling span lengths can be 3, 3, 5, 4, 4, 5, 5, 3, 3, 2, 4, 4, 2, 4, and 5, respectively, in one example. As shown in Figure 4B, the target of this exemplary configuration starts with the modal token "[X]", which represents an object configuration of a more extreme class.

[0089] In the third object configuration, sequence-based objects can be used. A single longer span of 50 can be selected for corruption. For example, the span length can be 95. The span can be fixed at the end of the input sequence. As shown in Figure 4B, the target of this exemplary configuration starts with the modal token "[S]" which indicates an object configuration of a sequence-based class.

[0090] Example Results For pre-trained objects, this example is compared to the following pre-training baseline.

[0091] Causal Language Models (CLMs) – these are pre-trained versions of standard left-to-right autoregressive language models used in many standard pre-trained models such as GPT (Radford et al., 2019; Brown et al., 2020). In this disclosure, we refer to this model as GPT-like in our experiments.

[0092] The prefix LM (PLM) is a slight variation of the causal LM in which M has a bidirectional receptive field, introduced by Liu et al., 2018; Raffel et al., 2019. For this baseline, we uniformly sample PLMs over the length of M and calculate only the loss on the autoregressive target.

[0093] Spanning corruption (SC) is a standard denoising object proposed in T5 (Raffel et al., 2019). The idea is to erase specific parts of the text, leaving them blank, and replace them with sentinel tokens. The text with the sentinel tokens replaced is then copied to the target and autoregressively generated by the model. This baseline uses a mean span of 3 and a denoising rate of 15%, following the default T5 setup.

[0094] Span Damage + LM (SCLM) - This baseline is trained with a mixture of CLM and span damage in equal proportions. This baseline uses the same hyperparameters for SC as the SC component of this object.

[0095] UniLM (ULM) - This is an object proposed by Dong et al. (2019).

[0096] For all objects, these results explore both single-stack and encoder-decoder architectures. All architectures are inputs to a target implemented in either an encoder-decoder or decoder-only model structure. This is because, as empirically demonstrated in Raffel et al., 2019, pretraining of BERT-style masked language modeling is already effectively incorporated by this style of pretraining.

[0097] The dataset used is SuperGLUE (Wang et al., 2019), which includes eight NLU subtasks. The experiment also covers three datasets from the GEM benchmark (Gehrmann et al., 2021), which focuses on language generation problems. We use XSUM (summarization), ToTTo (table-to-text generation) (Parikh et al., 2020), and SGD (Schema Guided Dialog) (Rastogi et al., 2019) from the GEM benchmark. For all of these tasks, these results are evaluated using both supervised fine-tuning and prompt-based one-shot learning. Finally, these results also compare the general ability of the models in text generation using perplexity scores from the C4 validation set.

[0098] For SuperGLUE, these results report well-established metrics such as accuracy, F1, or perfect match, where appropriate. For GEM benchmarks, these results use the Rouge-L metric. For language modeling, these results report negative log perplexity. Model universality, i.e., collective performance across all ranges of tasks, is the primary evaluation criterion here. To enable comparisons between models from this perspective, these results use aggregated performance scores. However, metrics for different tasks can vary significantly in nature, such as F1 and perplexity. To address this, these results choose to report and use normalized relative gain against a baseline as an overall metric. For this purpose, these results use a standard language model (decoder-only) (GPT-like) and a standard span-denoising encoder-decoder (T5) as primary baselines and report all methods against their relative performance to these well-established candidates. Because the overall gain is normalized to these results, it becomes more difficult to leverage or be influenced by benchmark lottery effects.

[0099] All experiments in this study were conducted using the open-source T5X4 framework (Roberts et al., 2022) and Flaxformer, with JAX / Flax (Bradbury et al., 2018). In these experiments, the C4 corpus was used, and all models were pre-trained over 500K steps with a batch size of 128 and sequence lengths of 512 input and 512 target. The approximate total number of tokens observed during pre-training was about 32 billion. Each pre-training run typically used 64-128 TPUv4 chips (Jouppi et al., 2020).

[0100] This experiment optimizes this example using the Adafactor (Shazeer & Stern, 2018) optimizer with an inverse square root learning rate. In this example, all baseline pre-trained objects are run using both a decoder-only architecture and an encoder-decoder architecture. The results report key experimental results using a basic architecture with approximately 167M parameters for the decoder model and 335M parameters for the encoder-decoder model. All models use standard transformers with SwiGLU layers as described in (Shazeer, 2020).

[0101] This example uses the default T5 English 32K sentence piece for all models. In the context of decoder-only models, except for decoder models trained with causal LM, this experiment uses bidirectional receptive fields only for the input segment and autoregressive decoding for the target segment.

[0102] Table 1 reports the raw results for all benchmark tasks and datasets. This example is represented as "UL2". To facilitate comparison of the overall setup, these results also report relative comparisons with well-established baselines such as the T5 and GPT models. These are reported in Tables 2 and 3, respectively. [Table 1] [Table 2] [Table 3]

[0103] When T5 is used as the reference baseline, none of the pre-trained decoder models outperform T5, except for the UL2 decoder. Furthermore, overall relative performance is 10% to 30% lower. The prefixed LM decoder models are about 10% worse than the T5 baseline. The UL2 decoder is only 14.6% better than the T5 encoder-decoder setup.

[0104] Overall, UL2 outperforms T5 by +43.4% and +76.2% compared to GPT-like CLM decoder models. This represents the best relative gain (overall gain) compared to all other alternatives. In every individual task, UL2 outperforms T5 in all nine tasks considered. Therefore, UL2 is a universally better option compared to span-broken T5 models. UL2 is very consistent. Even when it underperforms another method on a task, the difference is relatively small (e.g., 6.5 vs. 7.3 in one-shot TOTTO). Conversely, when UL2 outperforms a baseline like T5, the gain can be as large as +363%. UL2 remains the most consistent and robust method. The consistent improvement also suggests that it can be used as a more consistent replacement for T5 and GPT-like models.

[0105] Ablation results are provided to confirm that mode switching capability can be effective for performance. One-shot XSum and one-shot SuperGLUE experiments were conducted. Table 4 reports the results of varying the paradigm prompts for the models. The results show that using correct or incorrect prompts can result in a 48% performance gap (XSum, Rouge-1). On the other hand, SuperGLUE was not very sensitive to prompting. In SuperGLUE, using prompts during one-shot evaluation was almost always better than not using prompts. [Table 4] [Table 5]

[0106] Experiments are provided to test the effectiveness of individual objects within the object framework. Table 5 reports the results of these ablations. Table 5 reports the results with varying mean span and failure rate, along with the percentage of S-noise reduction used (expressed in SD%). In this test, the total number of configurations in the mixture was span × failure rate + 1. Table 5 labels these configurations Var-A to Var-L for easy reference.

[0107] Additional experiments were conducted by scaling up both 1) the model size and 2) the pre-training dataset size. The UL2 encoder-decoder model was scaled up to approximately 1B parameters, and the number of pre-training tokens was increased to 0.5 trillion tokens.

[0108] Table 6 reports the results for this scale setting. At larger scales, the UL2 encoder-decoder model in this example remains competitive. The difference here is that UL2 underperforms T5(1B) in the SuperGLUE suite. However, this is compensated for by not only outperforming T5(1B) in 7 out of 8 tasks, but also by a 2-4x performance improvement in one-shot evaluations. The gains in supervised fine-tuning are smaller, but still significant overall in XSUM, SGD, and TOT. [Table 6]

[0109] This example was also evaluated with a model size of approximately 20B parameters. This experiment follows the same training protocol as previous experiments, pre-training on the C4 corpus, but scaling the number of tokens the model references during pre-training. This experiment uses a batch size of 1024 and 512 TPUv4 chips to pre-train this model. The model is trained on a total of 1 trillion tokens (2 million steps) in C4. The sequence length is set to 512 / 512 for input and target. Dropout during pre-training is set to 0. The model has 32 encoder layers and 32 decoder layers, with dmodel being 4096 and dff being 16384. Each head has dimensions of 256, for a total of 16 heads. This model uses 8 model parallelisms. The result holds the same sentence-piece tokenizer as T5 with a vocabulary size of 32k. Therefore, UL20B can be interpreted as a model that is quite similar to T5, but trained on different objects and with slightly different scaling knobs. Similar to the previous experiment, UL20B will be trained on the Jax and T5X infrastructure.

[0110] To demonstrate the universality of the approach, this experiment examines a total of approximately 50+ NLP tasks. The list of tasks and their categorization are shown below. Note that the task categorization is generally flexible, and some tasks may straddle different category boundaries.

[0111] Language generation—summarization and data-to-text generation tasks. Our models were evaluated using CNN / Dailymail (Hermann et al., 2015), XSUM (Narayan et al., 2018), MultiNews (Fabbri et al., 2019), SAMSum (Gliwa et al., 2019), WebNLG (Castro Ferreira et al., 2020) (English), E2E (Dusek et al., 2019), and CommonGen (Lin et al., 2020). For WebNLG, E2E, and CommonGen, we used versions from the GEM benchmark (Gehrmann et al., 2021).

[0112] Human-based language generation—evaluating various text generation tasks using human evaluation via the GENIE leaderboard (Khashabi et al., 2021). These tasks include aNLG (Bhagavatula et al., 2019), ARC-DA (Clark et al., 2018), WMT19 (Foundation), and XSUM (Narayan et al., 2018).

[0113] Language comprehension, classification, and question answering – using reading comprehension, question answering, text classification, and natural language reasoning datasets. RACE (reading comprehension) (Lai et al., 2017), QASC (Khot et al., 2020), OpenBookQA (Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., 2019), VitaminC (Schuster et al., 2021a), Civil Comments, and Wikipedia toxicity detection datasets (Borkan et al., 2019) are used. We will use the standard SuperGLUE (Wang et al., 2019) and GLUE (Wang et al., 2018) datasets.

[0114] Common sense reasoning - HellaSwag (Zellers et al., 2019), SocialIQA / SIQA (Sap et al., 2019), PhysicalIQA / PIQA (Bisk et al., 2020), CosmosQA (Huang et al., 2019), AbductiveNLI (Bhagavatula et al., 2019), CommonsenseQA (Talmor et al. al., 2018) and CommonsenseQA2 (Talmor et al., 2021).

[0115] We use the scroll benchmark (Shaham et al., 2022), which consists of seven component tasks, including long-range inference - GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Kocisk y et al., 2018), QuaLITY (Pang et al., 2021), and ContractNLI (Koreeda & Manning, 2021).

[0116] Structured knowledge grounding—UnifiedSKG (Xie et al., 2022)—uses several component tasks from WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al., 2017), MTOP (Li et al., 2020), and DART (Nan et al., 2020). A dataset relatively convenient for performing evaluations is selected, and mainstream metrics such as exactness or perfect match are used instead of ambiguous metrics or metrics requiring significant domain-specific post-processing.

[0117] Information retrieval (IR) is the task of retrieving relevant documents for a given query. In this experiment, we will use the latest next-generation IR paradigm, namely the setup of a differentiable search index (Tay et al., 2022). We will use the same NQ partition as in the DSI paper (Kwiatkowski et al., 2019).

[0118] For each dataset, the previous best state-of-the-art (SOTA) result is provided. [Table 7-1] [Table 7-2] [Table 7-3]

[0119] UL2 achieves at least state-of-the-art performance in approximately 50+ NLP tasks and setups. Often, the margin is very wide, and even when UL2 doesn't achieve state-of-the-art, its performance is generally very competitive. The difficulty of achieving state-of-the-art on each benchmark varies considerably. In some cases, the state-of-the-art model is 32B density equivalent (Zoph et al., 2022). For others, it's the basic model.

[0120] Exemplary Method Figure 5 shows a flowchart of an exemplary method performed by an exemplary embodiment of the present disclosure. While Figure 5 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown therein. Various steps of Method 500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0121] In 502, exemplary method 500 may include obtaining configuration parameters for multiple different combinations of a pre-training object framework. The pre-training object framework (e.g., including a pre-training pipeline 200) may include a parameterized corruption function configured to generate training examples according to one or more configuration parameters. For example, a parameterized corruption function may be configured to receive an original training example (e.g., a sequence of text) and output a corrupted training example. Multiple different combinations of configuration parameters may each correspond to multiple object configurations, such as object configurations 206-212. Multiple different combinations of configuration parameters may be obtained from a configuration file or other parameter storage.

[0122] In 504, exemplary method 500 may include generating multiple corrupted training examples from one or more training examples using a pre-trained object framework. Each of the multiple corrupted training examples may be generated according to several different combinations of configuration parameters. For example, different corrupted training examples may be generated according to each of several different combinations of configuration parameters (e.g., each of several object configurations).

[0123] In 506, exemplary method 500 may include inputting multiple broken training examples into a machine learning model. The machine learning model may be configured to generate unbroken subparts corresponding to broken subparts of the broken training examples. For example, the machine learning model may be configured to perform next word generation based on the surrounding context. The machine learning model may be configured to use unbroken tokens bidirectionally as input for predicting broken subparts.

[0124] In 508, exemplary method 500 may include obtaining from a machine learning model multiple outputs, each generated by the machine learning model based on multiple broken training examples.

[0125] In 510, the exemplary method 500 may include updating one or more parameters of a machine learning model based on an evaluation of multiple outputs.

[0126] In some embodiments of the exemplary method 500, the configuration parameters may include two or more different parameters, such as sub-part length parameters, sub-part volume parameters, or failure rate parameters.

[0127] In some embodiments of the exemplary method 500, multiple different combinations of configuration parameters may include a distributed configuration configured to generate multiple corrupted sub-parts distributed across training examples, and a sequential configuration configured to generate corrupted sub-parts corresponding to the end of a training example.

[0128] In some embodiments of Exemplary Method 500, multiple different combinations of configuration parameters may include a first distributed configuration configured to generate a first set of multiple corrupted sub-sub

[0129] In some embodiments of the exemplary method 500, compared to the first distribution configuration, the second distribution configuration may include at least one of a sub-sub-length parameter corresponding to a longer sub-sub-length, or a failure rate parameter corresponding to a higher failure rate.

[0130] In some embodiments of the exemplary method 500, the sequential configuration can correspond to a prefix-based language modeling object.

[0131] In some embodiments of exemplary method 500, the configuration parameters of multiple different combinations may include a first set of multiple distribution configurations, each associated with a sub-length parameter indicating a sub-length of less than about 12 tokens, and a second set of multiple distribution configurations, each associated with at least one of a sub-length parameter indicating a sub-length of more than about 12 tokens, or a damage rate parameter indicating a damage rate greater than about 30%. In some embodiments of exemplary method 500, the configuration parameters of multiple different combinations may include sequential configurations. In some embodiments of exemplary method 500, the configuration parameters of multiple different combinations may include an amount of one or more sequential configurations, such that the amount is less than about 50% of the total amount of the multiple configurations. In some embodiments of exemplary method 500, the configuration parameters of multiple different combinations may include an amount of one or more sequential configurations, such that the amount is about 20% of the total amount of the multiple configurations.

[0132] In some embodiments of the exemplary method 500, the first multiple distributed configurations can each be associated with a sub-sub-length parameter indicating a sub-sub-length of less than approximately 10 tokens.

[0133] In some embodiments of exemplary method 500, the second multiple distribution configurations can each be associated with a sub-length parameter indicating a sub-length of more than approximately 12 tokens. In some embodiments of exemplary method 500, the second multiple distribution configurations can each be associated with a sub-length parameter indicating a sub-length of more than approximately 30 tokens.

[0134] In some embodiments of the exemplary method 500, the second plurality of dispersion configurations can each be associated with a failure rate parameter exhibiting a failure rate greater than approximately 30%. In some embodiments of the exemplary method 500, the second plurality of dispersion configurations can each be associated with a failure rate parameter exhibiting a failure rate of at least approximately 50%.

[0135] In some embodiments of the exemplary method 500, generating multiple corrupted training examples from one or more training examples may include determining one or more selected subparts of each sequence of data tokens for each of the one or more training examples (each training example containing each sequence of data tokens) and replacing one or more selected subparts with replacement tokens.

[0136] In some embodiments of the exemplary method 500, the exemplary method 500 may include inputting a mode switching token (e.g., modal tokens such as "[R]", "[X]", "[S]") corresponding to at least one configuration of several different combinations of configuration parameters, using each of the multiple failure training examples, where at least one configuration is used to cause each failure training example to fail.

[0137] In some embodiments of Exemplary Method 500, a mode-switching token can trigger downstream behavior of a machine learning model corresponding to a task preferred by at least one configuration. For example, a mode-switching token can be prepended to the runtime input (e.g., at inference) based on the type of task associated with the runtime input. For example, a short-form generation task can use a mode-switching token associated with a short-form broken span (e.g., "[R]"). A long-form generation task can use a mode-switching token associated with a long-form broken span (e.g., "[X]" or "[S]").

[0138] In some embodiments of Exemplary Method 500, at least one of the damage parameters may be a probability parameter. In some embodiments of Exemplary Method 500, the probability parameter may be a damage sub-length parameter that characterizes the variance at which selected sub-lengths are sampled. In some embodiments of Exemplary Method 500, the probability parameter may be a damage rate parameter that characterizes the ratio at which one or more selected sub-lengths of a training example are damaged.

[0139] In some embodiments of the exemplary method 500, the sequence of data tokens can correspond to natural language.

[0140] In some embodiments of the exemplary method 500, the sequence of data tokens can correspond to genetic data.

[0141] In some embodiments of the exemplary method 500, a sequence of data tokens can correspond to text data.

[0142] In some embodiments of Exemplary Method 500, the machine learning model may include a transformer encoder. In some embodiments of Exemplary Method 500, the machine learning model may include a transformer decoder.

[0143] In some embodiments of the exemplary method 500, the exemplary method 500 may include generating a first fine-tuned version of a machine learning model for a first task, and generating a second fine-tuned version of the machine learning model for a second different task.

[0144] In some embodiments of Exemplary Method 500, the first task may be at least one of a classification task or an intersequence task. In some embodiments of Exemplary Method 500, the second different task may be at least one of an open text generation task or a prompt-based inference task.

[0145] Additional disclosures The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions performed and the information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions between their components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0146] The subject matter of this disclosure has been described in detail in various specific and exemplary embodiments, each example provided for illustrative purposes only and not limiting the disclosure. Those skilled in the art, upon understanding the foregoing, will readily be able to modify, transform, and create equivalents of such embodiments. Therefore, this disclosure does not exclude the inclusion of such modifications, changes, and / or additions to the subject matter, as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used in another embodiment to create yet another embodiment. Thus, this disclosure is intended to cover such modifications, changes, and equivalents.

[0147] The aspects of this disclosure have been described in relation to their exemplary embodiments. All and all features of the following claims can be combined or rearranged in any possible way, including combinations of claims not expressly enumerated together, so that the dependency of the exemplary claims enumerated herein should not be read as limiting the scope of possible combinations of features disclosed herein. Accordingly, the scope of this disclosure is illustrative and not limiting, and the disclosure of the subject matter does not preclude such modifications, variations or additions to the subject matter as will be readily apparent to those skilled in the art. Furthermore, terms are described herein using lists of exemplary elements joined by conjunctions such as “and,” “or,” and “but.” It should be understood that such conjunctions are provided for illustrative purposes only. For example, clauses and other sets of items joined by certain conjunctions such as “or” may refer to “and / or,” “at least one of the exemplary elements enumerated therein,” “any combination of,” etc. Also, terms such as “based on” should be understood as “at least partially based on.”< / e> < / d> < / c> < / x> < / x> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x> < / y> < / x>

Claims

1. A method performed by one or more computers to pretrain a machine learning model having diverse objects, A computing system including one or more processors obtains configuration parameters for multiple different combinations of a pre-trained object framework, The computing system generates multiple corrupted training examples from one or more training examples using the pre-training object framework, wherein each of the multiple corrupted training examples is generated according to the multiple different combinations of configuration parameters. The computing system inputs the plurality of damaged training examples into the machine learning model, wherein the machine learning model is configured to generate undamaged sub-parts corresponding to damaged sub-parts of the damaged training examples. The computing system obtains from the machine learning model a plurality of outputs, each generated by the machine learning model based on the plurality of corruption training examples. The computing system provides a step of updating one or more parameters of the machine learning model based on an evaluation of the multiple outputs. Methods that include...

2. The method according to claim 1, wherein the configuration parameters include two or more different parameters selected from a sub-part length parameter, a sub-part quantity parameter, or one or more damage rate parameters.

3. The aforementioned multiple different combinations of configuration parameters are: A distributed configuration configured to generate multiple corrupted sub-parts distributed across training examples, A sequential configuration configured to generate a corrupted sub-part corresponding to the end of the aforementioned training example, The method according to claim 1, including the method described in claim 1.

4. The aforementioned multiple different combinations of configuration parameters are: A first distributed configuration configured to generate a first set of multiple damaged sub-parts distributed across training examples, A second distribution configuration configured to generate a second plurality of damaged sub-sub A sequential configuration configured to generate a corrupted sub-part corresponding to the end of the aforementioned training example, The method according to claim 1, including the method described in claim 1.

5. Compared to the first distributed configuration, the second distributed configuration is: Sub-part length parameter corresponding to a longer sub-part length, or One or more failure rate parameters corresponding to a higher failure rate, The method according to claim 4, comprising at least one of the following.

6. The method according to claim 3, wherein the sequential configuration corresponds to a prefix-based language modeling object.

7. The aforementioned multiple different combinations of configuration parameters are: A first set of distributed configurations associated with sub-length parameters, each representing a sub-length of less than 12 tokens, A second configuration is a set of multiple distributed configurations, each of which is: A sub-length parameter indicating a sub-length greater than 12 tokens, or One or more failure rate parameters indicating a failure rate exceeding 30%, The second plurality of distributed configurations associated with at least one of the following: The method according to claim 1, including the method described in claim 1.

8. The method according to claim 7, wherein the first plurality of distributed configurations are each associated with a sub-length parameter indicating a sub-length of less than 10 tokens.

9. The method according to claim 7, wherein the second plurality of distributed configurations are each associated with a sub-length parameter indicating a sub-length greater than 12 tokens.

10. The method according to claim 7, wherein the second plurality of distributed configurations are each associated with a sub-length parameter indicating a sub-length of more than 30 tokens.

11. The method according to claim 7, wherein the second plurality of dispersion configurations are each associated with one or more failure rate parameters exhibiting a failure rate of more than 30%.

12. The method according to claim 7, wherein the second plurality of dispersion configurations are each associated with one or more failure rate parameters that indicate a failure rate of at least 50%.

13. The step of generating multiple corrupted training examples from one or more training examples is: Each of the one or more training examples includes a sequence of data tokens. The computing system determines one or more selected sub-parts of each sequence of the data tokens, The computing system performs the steps of replacing one or more selected sub-parts with replacement tokens, The method according to claim 1, including the method described in claim 1.

14. The method according to claim 1, wherein the computing system includes the step of inputting a mode switching token corresponding to at least one configuration of the plurality of different combinations of configuration parameters using each of the plurality of corruption training examples, the at least one configuration being used to corrupt each of the respective corruption training examples.

15. The method according to claim 14, wherein the mode switching token triggers a downstream operation of the machine learning model corresponding to a task prioritized by the at least one configuration.

16. The method according to claim 2, wherein at least one of the one or more failure rate parameters is a probability parameter.

17. The method according to claim 16, wherein the probability parameter characterizes the variance from which the selected sub-length is sampled.

18. The method according to claim 16, wherein the probability parameter is one or more failure rate parameters that characterize the rate at which one or more selected subparts of a training example are damaged.

19. The method according to claim 13, wherein the sequence of data tokens corresponds to natural language.

20. The method according to claim 13, wherein the sequence of data tokens corresponds to genetic data.

21. The method according to claim 13, wherein the sequence of data tokens corresponds to text data.

22. The method according to claim 1, wherein the machine learning model includes a transformer encoder.

23. The method according to claim 1, wherein the machine learning model includes a transformer decoder.

24. The computing system comprises the steps of generating a first fine-tuned version of the machine learning model for a first task, The computing system comprises the steps of generating a second finely tuned version of the machine learning model for a second different task, The method according to claim 1, including the method described in claim 1.

25. The method according to claim 24, wherein the first task is at least one of a classification task or an intersequence task, and the second different task is at least one of an open-ended text generation task or a prompt-based inference task.

26. A non-temporary computer-readable storage medium that stores instructions executable to cause one or more processors to perform an operation, wherein the operation includes the method according to any one of claims 1 to 25.

27. One or more processors, A non-temporary computer-readable storage medium according to claim 26, A system that includes this.

Images