Improved training of large neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network may be configured as a generative neural network, e.g., an autoregressive generative neural network.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 465,487, filed May 10, 2023. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Background Art

[0002] This specification relates to neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current value input of the corresponding set of parameters.

[0004] Training large-scale neural networks can be very difficult due to their large model size and high computational cost. This is especially true because as model size increases, other issues arise, such as verbatim memorization of training data (which can potentially lead to unintended data leakage) and the generation of harmful content during inference. Summary of the Invention

[0005] This specification describes a system, implemented as a computer program on one or more computers in one or more locations, that trains a neural network to perform any of a variety of downstream machine learning tasks on network inputs.

[0006] Generally speaking, the training process has two phases: a pre-training phase, during which the neural network is pre-trained using a large amount of unlabeled training data; followed by an adaptation phase, during which the pre-trained neural network is adapted for one or more of the downstream tasks using training data specific to the downstream task—for example, fine-tuning or few-shot learning. Specifically, during the pre-training phase, the system uses a variety of techniques to improve the effectiveness, efficiency, or both of the training process.

[0007] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0008] Using some of the techniques described herein, for example, techniques related to inserting "canaries" into training data, a training system can test a neural network to determine the likelihood that the neural network has memorized the training data, and optionally modify the training of the neural network to reduce this likelihood. In this way, the techniques described herein can test whether a neural network is likely to be subject to data leakage, and optionally modify the training of the neural network to reduce the likelihood of data leakage.

[0009] Using some of the described techniques, e.g., techniques related to toxicity regulation / control techniques, a training system can improve the overall quality of output generated by a neural network after training—i.e., at inference time.

[0010] By combining a large, carefully curated pre-training corpus from a diverse set of sources—including web documents, books, code, mathematics, and conversational data—with efficient training techniques that improve the multilingual and reasoning capabilities of neural networks while alleviating scaling-related issues, some of the techniques described in this specification can train neural networks to converge significantly faster than existing training systems, e.g., in terms of wall-clock time, and while consuming reduced computational resources.

[0011] From another perspective, this improvement in training efficiency makes it possible to train neural networks that are smaller than some known neural networks using a considerable amount of computing resources. Once trained, the neural network will be able to achieve performance that is competitive with or even exceeds the state-of-the-art on a wide range of tasks, even with a relatively small model size, making it more suitable for deployment on mobile devices, embedded systems, or other hardware platforms with limited computing resources.

[0012] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A An example neural network is shown.

[0014] Figure 1B An example training system and an example inference system are shown.

[0015] Figure 2 is a flowchart of an example process for training a neural network.

[0016] Figure 3 yes Figure 2 A flowchart of the substeps of a step in a process.

[0017] Figure 4 yes Figure 2 A flowchart of the substeps of a step in a process.

[0018] Figure 5 is a flowchart of another example process for training a neural network.

[0019] Figure 6 is a flowchart of an example process for generating an output sequence using a trained neural network.

[0020] Figure 7 is a flowchart of another example process for training a neural network.

[0021] Figure 8 is a flowchart of another example process for training a neural network based on optimizing multiple different pre-trained objective functions.

[0022] Figure 9 yes Figure 8 A flowchart of the substeps of a step in a process.

[0023] Figure 10 yes Figure 8 A flowchart of the substeps of a step in a process.

[0024] Figure 11 yes Figure 8 A flowchart of the substeps of a step in a process.

[0025] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Figure 1A An example neural network 110 is shown. Neural network 110 can be configured through training to perform any kind of machine learning task in response to a received request, i.e., can be configured to receive any kind of numerical data input and generate any kind of score, classification, or regression output based on that input.

[0027] To improve the effectiveness, efficiency, or both of the training process, training of neural network 110 utilizes at least some of the techniques discussed in this specification, including canary-gram-based word-by-word memory assessment, training neural network 110 on multilingual training datasets, and training neural network 110 on a mixture of different pre-training tasks.

[0028] After training, neural network 110 can be caused to perform any of a variety of machine learning tasks, some examples of which are described below. For example, neural network 110 can be caused to perform a task by including a few-shot prompt for the task or a natural language instruction for the task in the input sequence of neural network 110.

[0029] In some cases, neural network 110 is a neural network configured to perform an image or video processing task, i.e., receive an input image or an input video having a plurality of frames (where each frame is an input image), and process intensity values of pixels of the input image to generate a network output for the input image or input video.

[0030] For example, the task may be image classification, and the output generated by neural network 110 for a given image may be a score for each of a set of object categories, where each score represents an estimated likelihood that the image contains an object belonging to that category. As another example, the task may be image embedding generation, and the output generated by neural network 110 may be a numerical embedding of the input image. As yet another example, the task may be object detection, and the output generated by neural network 110 may identify locations in the input image that depict objects of a particular type. As yet another example, the task may be image semantic segmentation, and the output generated by neural network 110 may assign each pixel of the input image to a category from a set of categories. As yet another example, the task may be image instance segmentation, and the output generated by neural network 110 may assign each pixel of the input image to a corresponding object instance from a set of object instances. As yet another example, the task may be image depth prediction, and the output generated by neural network 110 may assign a corresponding predicted depth value to each pixel of the input image.

[0031] As another example, if the input to neural network 110 is an internet resource (e.g., a web page), a document or a portion of a document, or features extracted from an internet resource, a document or a portion of a document, the task may be to classify the resource or document, i.e., the output generated by neural network 110 for a given internet resource, document or portion of a document may be a score for each of a set of topics, where each score represents an estimated likelihood that the internet resource, document or portion of a document is related to that topic.

[0032] As another example, if the input to neural network 110 is features of the impression context of a particular ad, the output generated by neural network 110 may be a score representing the estimated likelihood that the particular ad will be clicked.

[0033] As another example, if the input to neural network 110 is features of a personalized recommendation for a user, e.g., features characterizing the context of the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by neural network 110 may be a score for each content item in a set of content items, where each score represents an estimated likelihood that the user will react positively to being recommended the content item.

[0034] As another example, if the input to neural network 110 is a sequence of text in one language, the output generated by neural network 110 may be a text snippet in another language that is a predicted appropriate translation of the input text into the other language.

[0035] Some implementations can be used for automatic code generation. For example, the input can represent words, word fragments, or characters in a first natural language, and the output can represent instructions in a computer programming or markup language, or instructions for controlling an application to perform a task, such as constructing a data item such as an image or a web page.

[0036] As another example, the input to the neural network 110 may include a textual description of computer code or the functionality of the computer code, and the output generated by the neural network is a sequence of computer code in a programming language that completes the input code in the context input or performs the functionality described in the context input.

[0037] As a specific example of automatic code generation, Figure 1A It is shown that the task can be automatic code debugging, where the input of the neural network includes vulnerable computer code in a programming language that potentially has a defect or "bug", and the output generated by the neural network includes fixed computer code in the programming language, where the defect or "bug" is corrected or removed from the vulnerable computer code. Optionally, in the automatic code debugging task, the input of the neural network additionally includes a text description in any natural language, and the output generated by the neural network similarly includes a text description in any natural language, such as code comments.

[0038] As another example, the task may be an audio processing task. For example, if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 may be a text transcription of the utterance. As another example, the task may be a keyword detection task, where if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 may indicate whether a particular word or phrase ("hot word") was spoken in the utterance. As another example, if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 may identify the natural language in which the utterance was spoken.

[0039] As another example, the task may be a natural language processing or understanding task that operates on a text sequence of a certain natural language, such as an entailment task, a paraphrase task, a text similarity task, a sentiment task, a sentence completion task, a grammar task, etc.

[0040] As another example, the task may be a text-to-speech task where the input is natural language text or features of natural language text, and the network output is a spectrogram or other data defining the audio of the text spoken in the natural language.

[0041] As another example, the task may be a health prediction task, where the input is a sequence derived from a patient's electronic health record data, and the output is a prediction related to the patient's future health condition, such as a predicted treatment that should be taken for the patient, the likelihood that the patient will experience an adverse health condition event, or a predicted diagnosis for the patient. For example, such electronic health data may include one or more sequences of physiological data acquired from the patient, where the output is a corresponding prediction related to those data sequences. Examples of physiological data and corresponding predictions include: blood glucose measurements, where the prediction is a predicted future blood glucose measurement or a prediction of a hyperglycemic or hypoglycemic event; heart rate, where the prediction is the presence or absence of heart disease or a future cardiac event; blood pressure measurements, where the prediction is the risk of future heart disease; and so on.

[0042] As another example, the task can be a text generation task, where the input is a text sequence and the output is another text sequence, such as a completion of the input text sequence, a response to a question posed in the input sequence, or a text sequence about a topic specified by the first text sequence. As another example, the input of a text generation task can be input other than text, such as an image or video, and the output can be text describing the input.

[0043] The text input sequence and the text output sequence can be in the same language, or alternatively can be in different languages. For example, the task is a multilingual text generation task, where the input is a text sequence in a first language (e.g., English), and the output generated by the neural network is a text sequence in a second language (e.g., German). As another example, the input of the text generation task can be an input other than text, such as an image, and the output sequence can be text describing the input.

[0044] In some implementations, the input represents data to be compressed, e.g., image data, video data, text data, audio data, or any other type of data; and the output represents a compressed version of the data. The input and output can each include the data to be compressed / any representation of the compressed data, e.g., symbols or embeddings generated / decoded by a corresponding neural network.

[0045] As another example, a task can be an agent control task, where the input is a sequence of observations or other data representing the state of an environment, and the output defines an action to be performed by the agent in response to the latest data in the sequence. The agent can be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different types of agents. The observations can include sensor data captured by sensors associated with the agent (e.g., a part of the agent), such as visual data, lidar data, sonar data, agent configuration data (e.g., joint angles), agent orientation data, etc.

[0046] In some implementations, the environment is a real-world environment, the agent is a mechanical (or electromechanical) agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates in or navigates through the environment, and the action is an action taken by the mechanical agent in the real-world environment to perform a task. For example, the agent can be a robot that interacts with the environment to complete a specific task (e.g., locate or manipulate an object of interest in the environment, move an object of interest to a specified location in the environment, or navigate to a specified destination in the environment).

[0047] In these implementations, observations may include, for example, one or more of the following: images, object position data, and sensor data to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the following: position, linear or angular velocity, force, torque, or acceleration of one or more parts of the agent, and the global or relative pose. Observations may be defined in 1D, 2D, or 3D and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as captured by a camera or lidar sensor, such as data from the agent's sensors or data from sensors located separately from the agent in the environment.

[0048] In these implementations, actions may be control signals for controlling a robot or other mechanical agent, such as torques or higher-level control commands for the joints of the robot, or control signals for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torques or higher-level control commands for controlling a surface or other control element (e.g., a steering control element of a vehicle). Control signals may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. Control signals may also or instead include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which electronic devices has an effect on the state of the observed environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, control signals may define actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0049] In some implementations, the environment is a simulation of the aforementioned real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example, a system implementing neural network 110 can be used to select actions in the simulated environment during training or evaluation of the system, and after training or evaluation, or both, the action selection strategy can be deployed to control a real-world agent in the specific real-world environment that is the subject of the simulation. This can avoid unnecessary wear and tear on, and damage to, the real-world environment or the real-world agent, and can allow the control neural network 110 to be trained and evaluated in situations that are rare in the real-world environment or that are difficult or impossible to recreate safely. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in the specific real-world environment. Thus, in such situations, observations of the simulated environment are relevant to the real-world environment, and the actions selected in the simulated environment are relevant to the actions to be performed by the mechanical agent in the real-world environment.

[0050] In some implementations, as described above, the agent may not include a human (e.g., it may be a robot). Instead, in some implementations, the agent includes a human user of a digital assistant, such as a smart speaker, smart display, or other device. Information defining the task can then be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task.

[0051] For example, a system implementing neural network 110 may output, via a digital assistant, instructions for actions to be performed by the user at each of a plurality of time steps to a human user. For example, instructions may be generated in natural language (transmitted as voice and / or on-screen text) based on actions selected by the system. The system selects actions so that they facilitate the performance of the task. A monitoring system (e.g., a camera system) may be provided to monitor the actions (if any) actually performed by the user at each time step in case (e.g., due to human error) the actions differ from the actions the system instructed the user to perform. Using the monitoring system, the system may determine whether the task has been completed. The system may identify actions that the user incorrectly performed with a probability exceeding a certain limit. If so, the system may warn the user to be cautious when the system instructs the user to perform such identified actions. Alternatively or additionally, the system may learn not to instruct the user to perform the identified actions, i.e., actions that the user is likely to perform incorrectly.

[0052] More generally, a digital assistant instructing a user can include receiving a request for assistance from the user at the digital assistant and determining a series of tasks to be performed by the user in response to the request, such as steps or subtasks of an overall task. Then, for one or more tasks in the series of tasks, for example, for each task, for example, until the last task in the series, the digital assistant can be used to output instructions for the task to be performed, such as steps or subtasks, to the user. This can be done using natural language, for example, on a display and / or using a speech synthesis subsystem of the digital assistant. Visual (e.g., video) and / or audio observations of the user performing the task can be captured, for example, using the digital assistant. The system described above can then be used to determine whether the user has successfully completed the task, such as a step or subtask, i.e., based on the answer as described above. If there are further tasks to be completed, then in response, the digital assistant can advance to the next task in the series of tasks (if any), for example, by outputting an instruction for the next task to be performed. In this way, the user can be guided step by step through a series of tasks to perform the overall task. During training of neural network 110, training rewards may be generated, for example, from video data representing examples of the overall task (if a corpus of such data is available) or from simulations of the overall task.

[0053] In other aspects, a digital assistant device including a system as described above is provided. The digital assistant may also include a user interface to enable the user to request assistance and output information. In an implementation, this is a natural language user interface and may include a keyboard, a voice input and output subsystem, and / or a display. The digital assistant may further include an auxiliary subsystem configured to determine a series of tasks to be performed by the user in response to a request. In an implementation, this may include a generative (large) language model, particularly for dialogue, such as a dialogue agent such as Sparrow (Glaese et al., arXiv:2209.14375) or Chinchilla (Hoffmann et al., arXiv:2203.15556). The digital assistant may have an observation capture subsystem for capturing visual and / or audio observations of the user performing the task; and an interface for the above-mentioned language model neural network 110 (which may be implemented locally or remotely). The digital assistant may also have an auxiliary control subsystem configured to assist the user. The auxiliary control subsystem may be configured to perform the above steps for, for example, one or more tasks in a series of tasks—for example, until the last task in the series. More specifically, the assistant control subsystem outputs instructions for a task to be performed to the user, uses the observation capture subsystem to capture visual or audio observations of the user performing the task, and determines from the above answers whether the user has successfully achieved the task. In response, the digital assistant can advance to the next task in the series of tasks and / or control the digital assistant, for example, to stop capturing observations.

[0054] As another example, the task can be a genomics task, where the input is a sequence representing a fragment of a DNA sequence or other molecular sequence, and the output is an embedding of the fragment used in a downstream task (e.g., by utilizing unsupervised learning techniques on a dataset of DNA sequence fragments) or an output for a downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional effects of non-coding variants, and the like.

[0055] In some cases, a machine learning task is a combination of multiple separate machine learning tasks, i.e., the system is configured to perform multiple different separate machine learning tasks, such as two or more of the machine learning tasks mentioned above. For example, the system can be configured to perform multiple separate natural language understanding tasks, where the network input includes identifiers of the separate natural language understanding tasks to be performed on the network input.

[0056] In some cases, the machine learning task is a multimodal processing task that requires processing multimodal data. Generally speaking, multimodal data is a combination of two or more different types of data, for example, two or more of audio data, image data, text data, or graphic data. As an example, multimodal data may include audiovisual data, including a combination of pixels of an image or video and audio data representing the values of a digitized audio waveform. As another example, multimodal data may include a combination of: i) text data representing text in a natural language, and ii) pixels of an image or video or audio data representing the values of an audio waveform. Optionally, but not necessarily, different types of data can represent the same or overlapping objects using different modalities (types), and when processing multimodal data, the data can be mapped to a common embedding space.

[0057] As a specific example, the task is a multimodal processing task that requires processing both textual input and image input, such that the neural network 110 includes both a computer vision neural network and a text processing neural network. That is, the target output to be generated by the computer vision neural network for a given image depends on one or more outputs generated by the text processing neural network for one or more corresponding textual inputs (and vice versa). Examples of such tasks include open vocabulary image classification, open vocabulary object detection, image captioning, text-based image search, image-based retrieval, and the like.

[0058] More generally, a multimodal processing task can correspond to any task previously described for any type of data that constitutes a multimodal combination. For example, when a task is applied to multimodal data that combines data for which the task was previously described with data of another type, the accuracy of the previously described task can be improved. For example, when processing data of multiple different types (modalities), the detection or classification of objects or events can be improved.

[0059] The example machine learning tasks described above are not exhaustive, and neural network 110 may also be used to perform other tasks.

[0060] Figure 1B Shown are an example training system 100 and an example reasoning system 170. Training system 100 and reasoning system 150 are examples of systems implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0061] Training system 100 includes neural network 110. Neural network 110 is a neural network that can be configured through training to perform any one or more of the tasks mentioned above, and possibly other tasks, by processing network inputs to generate network outputs for the tasks according to a set of parameters of neural network 110. For example, the parameters of neural network 110 include weights and, optionally, biases, of the layers of the neural network.

[0062] The neural network 110 may have any of a variety of Transformer-based neural network architectures, such as an encoder-only Transformer architecture, an encoder-decoder Transformer architecture, a decoder-only Transformer architecture, other attention-based architectures, and the like.

[0063] Examples of such architectures include those described in: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020; and Aakanksha Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” arXiv preprint arXiv:2204.02311.

[0064] In some implementations, neural network 110 can be configured as or include a generative (large) language model or a multimodal model (e.g., a vision and language model) to perform these example machine learning tasks. Therefore, neural network 110 can also be referred to as a "generative neural network."

[0065] In some implementations, the neural network 110 may be configured to autoregressively generate an output sequence consisting of word-grams selected from a word-gram vocabulary as a network output based on a network input that is an input sequence consisting of word-grams selected from the word-gram vocabulary.

[0066] The word-gram vocabulary may include any of a variety of word-grams representing text symbols or other symbols. For example, the word-gram vocabulary may include one or more of characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. Therefore, neural network 110 may further be referred to as an "autoregressive generative neural network."

[0067] Additionally or alternatively, the word-meta vocabulary may include word-meta that may represent data other than text. For example, the word-meta vocabulary may include image word-meta that represents a set of discrete image embeddings of an image, which may be generated by an image encoder neural network based on processing the image. As another example, the word-meta vocabulary may include audio word-meta that represents code vectors in a codebook of a quantizer, such as a residual vector quantizer.

[0068] More specifically, to generate a particular word-gram at a particular position within the output sequence, the neural network 110 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a corresponding score, e.g., a corresponding probability, to each word-gram in the vocabulary of word-grams.

[0069] For example, when generating a word-gram at any given position in the output sequence, the current input sequence may include the word-gram at any previous position in the input sequence and the output sequence that precedes the given position. As a specific example, the current input sequence may include the input sequence followed by the word-gram at any previous position in the output sequence that precedes the given position.

[0070] The neural network 110 can then use the score distribution to select a word-gram from the vocabulary as a particular word-gram. For example, the neural network 110 can greedily select the word-gram with the highest score, or can sample word-grams from the distribution, for example, using kernel sampling or another sampling technique.

[0071] The neural network 110 includes a plurality of layers, each layer having corresponding parameters. Each layer in the plurality of layers is configured to receive a layer input and apply the corresponding parameters of the layer to the layer input to generate a layer output of the layer, and then provide the layer output to one or more other layers of the neural network, the one or more other layers being configured to receive input from the layer according to the neural network architecture.

[0072] The multiple layers of the neural network 110 include multiple self-attention layers 115, wherein each self-attention layer 115 is configured to receive a corresponding current representation of each text token in the text tokens in the current output sequence as a layer input, and process the corresponding current representation to generate a corresponding updated representation of each text token in the text tokens in the current output sequence as a layer output.

[0073] The self-attention layer 115 refers to a layer that includes a self-attention mechanism. To generate a layer output, the self-attention layer 115 applies a self-attention mechanism to the layer input, where the self-attention mechanism uses the key, query, and value calculated from the corresponding current representation based on the parameters of the attention layer.

[0074] Each self-attention layer 115 may optionally apply other operations to the representation as part of updating that representation (e.g., by utilizing a position-wise feed-forward neural network, by applying layer normalization, by utilizing residual connections, etc.).

[0075] Thus, the parameters of the neural network 110 may include at least: (i) weights of each self-attention layer in the sequence of self-attention layers 115, e.g., weights of one or more query transformation layers, weights of one or more key transformation layers, and weights of one or more value transformation layers within the attention layer; and (ii) weights and, optionally, biases of each feed-forward layer included in each position-wise feed-forward neural network.

[0076] The parameters of the neural network 110 may also include weights of an embedding layer of the neural network 110, which is configured to generate embeddings for text tokens in the current output sequence.

[0077] The neural network 110 may have additional layers and components without (trainable) parameters, such as normalization layers, pooling layers, residual connections, softmax layers, logistic layers, and so on.

[0078] The training system 100 trains the neural network 110 using a two-phase training process: a pre-training phase and an adaptation phase. During the pre-training phase, the training system 100 trains the neural network 110 on unlabeled training data 120 to repeatedly update the values of the parameters of the neural network 110, i.e., to generate pre-trained values of the parameters from the initial values.

[0079] During the adaptation phase, the training system 100 uses labeled training data specific to the downstream task to adapt the pre-trained neural network 110 for any one or more of the downstream tasks, such as through fine-tuning adaptation. The downstream tasks may include any combination of one or more of the machine learning tasks mentioned above and possibly other tasks. Examples of fine-tuning adaptation techniques include supervised fine-tuning (SFT), reinforcement learning based on human feedback (RLHF), instruction tuning, etc., which use different training objectives, different data, or both.

[0080] The unlabeled training data 120 includes a plurality of training input sequences. Each training input sequence has a plurality of positions. Each position has a text word selected from a text word list. As mentioned above, the text word list may include one or more of characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. Additionally or alternatively, the word list may include words that may represent data other than text. For example, the word list may include image words that represent a set of discrete image embeddings of an image, which may be generated by an image encoder neural network based on processing the image. As another example, the word list may include audio words that represent code vectors in a codebook of a quantizer, e.g., a residual vector quantizer.

[0081] Specifically, the plurality of training input sequences include text tokens across a diverse mix of languages. In some implementations, the plurality of training input sequences include text tokens in hundreds of natural and programming languages. In some implementations, the plurality of training input sequences include no more than a predetermined number of text tokens in any given language, for example, text tokens in any given language constitute at most a predetermined percentage of all text tokens included in unlabeled training data 120.

[0082] It is referred to as “unlabeled” to indicate that, for each training input sequence, the corresponding target output for that training input sequence—that is, the target output that the neural network 110 should generate by processing that training input sequence—was not available to or used by the training system 100 during pre-training.

[0083] Typically, the training engine 130 of the training system 100 performs pre-training in multiple update iterations. At each update iteration, the training engine 130 updates the parameters of the neural network 110 using multiple training input sequences ("batches" or "mini-batches") sampled from the unlabeled training data 120.

[0084] Specifically, during the pre-training phase, training engine 130 trains neural network 110 on a mixture of different pre-training tasks on unlabeled training data 120. Each pre-training task uses a corresponding pre-training objective function. Each pre-training task focuses on training neural network 110 to learn a different aspect of language modeling. By training neural network 110 on a mixture of different pre-training tasks, training system 100 can train neural network 110 in a manner that improves its generalization ability during the adaptation phase—that is, relative to training neural network 110 on a single pre-training task.

[0085] The mixture of different pre-training tasks can include two or more of the following:

[0086] (i) Causal Language Modeling Pretraining Task. The causal language modeling pretraining task requires predicting, for each text token in a causal language modeling input sequence generated from a training input sequence, the text token at the previous position in the causal language modeling input sequence. The causal language modeling pretraining task uses a causal language modeling objective function 131.

[0087] (ii) Prefix Language Modeling Pretraining Task. The prefix language modeling pretraining task requires predicting the text token that should occupy a specific position in the suffix for each text token in the suffix input sequence included in the prefix language modeling input sequence, conditioned on (a) the text token in the prefix input sequence preceding the suffix input sequence in the prefix language modeling input sequence generated from the training input sequence and (b) the text token at any previous position in the suffix input sequence. The prefix language modeling pretraining task uses the prefix language modeling objective function 132.

[0088] (iii) Span-impaired pre-training task. The span-impaired pre-training task requires predicting one or more text tokens that should occupy the corresponding positions of one or more masked tokens in the span-masked input sequence, generated from a training input sequence. The span-impaired pre-training task uses the span-impaired objective function 133.

[0089] In an implementation, during the pre-training phase, the training engine 130 trains the neural network 110 on at least two of the above-mentioned pre-training tasks, for example, training the neural network on both the causal language modeling pre-training task and the prefix language modeling pre-training task; training the neural network on the causal language modeling pre-training task, the prefix language modeling pre-training task, and the span corruption pre-training task.

[0090] In an implementation, different pre-training tasks may be mixed in a mixture of different pre-training tasks according to a predetermined ratio. In one example, the causal language modeling pre-training task, the prefix language modeling pre-training task, and the span damage pre-training task may be mixed in a ratio of 60% / 20% / 20%. That is, 60% of all training input sequences obtained from the unlabeled training data 120 are used by the training engine 130 to generate causal language modeling input sequences, which are then used to train the neural network 110 to perform the causal language modeling pre-training task; 20% of all training input sequences obtained from the unlabeled training data 120 are used by the training engine 130 to generate prefix language modeling input sequences, which are then used to train the neural network 110 to perform the prefix language modeling pre-training task; and 20% of all training input sequences obtained from the unlabeled training data 120 are used by the training engine 130 to generate span masking input sequences, which are then used to train the neural network 110 to perform the span damage pre-training task. In other examples, the causal language modeling pre-training task, the prefix language modeling pre-training task, and the span corruption pre-training task can be mixed in a ratio of 50% / 25% / 25%, 70% / 15% / 15%, 80% / 10% / 10%, 80% / 20% / 0%, and so on.

[0091] In some implementations, the predetermined proportions in which different pre-training tasks are mixed in the mixture of different pre-training tasks can be defined by respective weights assigned to the pre-training tasks. For example, a user of the training system 100 can provide user input that specifies respective weights assigned to each of a plurality of different pre-training tasks, e.g., 0.6 for the causal language modeling pre-training task, 0.2 for the prefix language modeling pre-training task, and 0.2 for the span corruption pre-training task, and then during the pre-training phase, the training engine 130 repeatedly selects pre-training tasks from the plurality of different pre-training tasks based on the specified weights to train the neural network 110.

[0092] A common problem faced by generative neural networks, such as neural network 110 in FIG1 , is data leakage in their output. Data leakage occurs when a trained neural network reveals information that is private to an individual or organization and that was included in the training data 120 of neural network 110 as part of its output after deployment. Depending on the downstream machine learning task, data leakage can lead to a range of negative consequences, particularly when the revealed information is sensitive, confidential, or otherwise protected.

[0093] To minimize the negative consequences of data leakage, optionally, in some implementations, at each of various points during the pre-training phase, training system 100 can estimate the likelihood that neural network 110 has memorized verbatim the content from unlabeled training data 120, and then, depending on the likelihood, determine whether to modify the training of neural network 110 to reduce the likelihood that neural network 110 has memorized the content from unlabeled training data 120. Memory evaluation is broadly applicable to any of the various pre-training tasks on which neural network 110 is trained, i.e., it can be performed regardless of what objective function training engine 130 uses to train neural network 110.

[0094] Verbatim memorization refers to a situation where a neural network reproduces a specific portion of text that it processed during training—for example, when the neural network reproduces a sequence of multiple consecutive text tokens from its training corpus (e.g., text tokens that make up a sentence or even a paragraph of text) verbatim. Verbatim memorization increases the risk of a neural network exposing private data from its training corpus.

[0095] To this end, training system 100 includes a special token insertion engine 140 that can be used to modify the training input sequences before training occurs—that is, before neural network 110 is trained using at least some of the training input sequences included in unlabeled training data 120. Special token insertion engine 140 is an optional component of training system 100.

[0096] Specifically, special token insertion engine 140 can modify a given training input sequence by inserting canary tokens as a special token into the given training input sequence. For any given training input sequence, canary tokens are tokens that are not initially included in unlabeled training data 120; instead, they are inserted into the given training input sequence before neural network 110 processes the given training input sequence during training.

[0097] To strike a balance between making the canary token appear as an outlier while also preserving specific characteristics of the training data, the canary token can take an appropriate form and can be inserted at an appropriate position in a given training input sequence.

[0098] For example, canary tokens may take the form of interleaved canary tokens. That is, the special token insertion engine 140 obtains two training input sequences, including a first training input sequence and a second training input sequence, from the unlabeled training data 120, and generates a modified first training input sequence by inserting text tokens obtained from the second training input sequence into the first training input sequence, and generates a modified second training input sequence by inserting text tokens obtained from the first training input sequence into the second training input sequence.

[0099] In this example, the modified first training input sequence and the modified second training input sequence are then used to train neural network 110. Because the canary tokens are obtained from the training input sequence from unlabeled training data 120, they retain some linguistic properties of unlabeled training data 120.

[0100] After training neural network 110 on the training input sequence comprising the modified first training input sequence and the modified second training input sequence, training engine 130 then estimates the extent to which the trained neural network has memorized data in unlabeled training data 120 .

[0101] Specifically, training engine 130 provides a test input sequence including a subset of tokens previously included in a first training input sequence as input to neural network 110, and processes the test input sequence using neural network 110 to generate one or more predicted continuations of the test input sequence. Each predicted continuation specifies a plurality of output tokens. That is, training engine 130 uses neural network 110 to predict subsequent tokens following the token subset in the first training input sequence, given the token subset previously included in the first training input sequence.

[0102] To estimate the degree to which the trained neural network has memorized data from the unlabeled training data, training engine 130 evaluates a plurality of output tokens specified by each predicted continuation for the remaining subset of text tokens included in the modified first training input sequence (including text tokens obtained from the second training input sequence). For example, the evaluation may involve counting the total number of output tokens in each predicted continuation that also appear in the text tokens obtained from the second training input sequence, and the estimated degree may be proportional to the total count.

[0103] Other forms of canary tokens may also be used in addition or alternatively. For example, canary tokens may take the form of shuffle canary tokens. That is, special token insertion engine 140 obtains a training input sequence and then generates a modified training input sequence by shuffling all text tokens included in the training input sequence—that is, by changing the positions of existing text tokens—in order to remove information associated with their sequence-level ordering. In this example, the modified training input sequence, including the shuffled text tokens, is then used to train neural network 110.

[0104] Another common problem faced by generative neural networks, such as neural network 110 in FIG1 , is toxicity in their output. Toxicity refers to output generated by a neural network that includes harmful, offensive, or otherwise inappropriate content. Toxicity in the output generated by a trained neural network can lead to a range of negative consequences after the trained neural network is deployed for downstream machine learning tasks, such as conversational tasks, generative question answering tasks, and other open-ended language modeling tasks.

[0105] To this end, the special word-gram insertion engine 140 may modify the given training input sequence by inserting a toxic word-gram as another special word-gram into the given training input sequence. The toxic word-gram identifies the toxicity level of the content represented by the given training input sequence.

[0106] The special word-gram insertion engine 140 may determine a determined toxicity level from a plurality of toxicity levels based on the content represented by a given training input sequence, and then prepend or append a specific toxic word-gram corresponding to the determined toxicity level to the given training input sequence.

[0107] Including toxicity tokens can alleviate the toxicity problem because, after training, neural network 110 can be conditioned on toxicity tokens that represent low toxicity levels, thereby causing neural network 110 to generate outputs with low toxicity levels. That is, by training neural network 110 to accurately estimate the toxicity of its outputs, the generation after training can be controlled to produce outputs with a specified toxicity level.

[0108] In some implementations, the plurality of toxicity levels may include two or more toxicity levels, for example, level 0 and level 1. In some implementations, the plurality of toxicity levels may include three or more toxicity levels, for example, level 0, level 1, and level 2, where a larger number indicates a higher toxicity level. For example, a level 2 toxicity word-gram will be prepended to or appended to a training input sequence that includes a greater amount of harmful, offensive, or otherwise inappropriate content, while a level 1 toxicity word-gram will be prepended to or appended to a training input sequence that includes a lesser amount of such content.

[0109] In some implementations, the special word-gram insertion engine 140 need only do so for a relatively small number of training input sequences included in the unlabeled training data 120. That is, for a relatively small portion of all the training input sequences included in the unlabeled training data 120—e.g., less than 0.5%, 1%, or 5%—the training system 100 uses the special word-gram insertion engine 140 to add poison words that identify the toxicity level of the content represented by those training input sequences.

[0110] In order to automatically determine the toxicity level of a given training input sequence, the special token insertion engine 140 includes or accesses a text-based classifier engine. The text-based classifier engine can be implemented in any suitable manner so that it can process a given training input sequence to classify the given training input sequence into one of a plurality of toxicity levels.

[0111] For example, the text-based classifier engine can implement a text classifier machine learning model that generates a toxicity classification output, such as a neural network (e.g., a language model neural network), a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model. As another example, the text-based classifier engine can implement a deterministic text-based classification algorithm that generates a toxicity classification output, such as a human-programmed algorithm.

[0112] Another common problem faced in training generative neural networks, such as neural network 110 in FIG1 , is the lack of multilingual training data in unlabeled training data 120. For example, unlabeled training data 120 may primarily contain training input sequences each consisting of text tokens in the same specific language, but training input sequences each consisting of text tokens in two or more different languages may still be rare.

[0113] The lack of multilingual training data in the unlabeled training data 120 may negatively impact the performance of the trained neural network 110, especially for multilingual downstream tasks involving processing text tokens in two or more languages, such as translation tasks, multilingual text generation tasks, including multilingual question answering tasks.

[0114] To address the lack of multilingual training data and the challenges it presents for training neural network 100, training system 100 includes a multilingual training data generation engine 145, which can be used to generate additional multilingual training data based on the existing content in unlabeled training data 120. Multilingual training data generation engine 145 is an optional component of training system 100.

[0115] When included, multilingual training data generation engine 145 can be used by training system 100 to generate multilingual training data from training input sequences included in unlabeled training data 120—and training engine 130 then trains neural network 100 on the multilingual training data.

[0116] The multilingual training data includes a plurality of first multilingual training pairs and a plurality of second multilingual training pairs. Each first multilingual training pair includes a first source training input sequence and a first target training input sequence. Each second multilingual training pair includes a second source training input sequence and a second target training input sequence. Specifically, the first source training input sequence and the second source training input sequence are in the same language, while the first target training input sequence and the second target training input sequence are in different languages. Figure 7 Generating multilingual training data is discussed further.

[0117] Including multilingual training data further improves the ability of neural network 110 to understand and generate multilingual output; it also enhances the performance of trained neural network 110 for multilingual downstream tasks involving processing text tokens in two or more languages, such as translation tasks, multilingual text generation tasks, including multilingual question answering tasks.

[0118] After the pre-training phase, in the adaptation phase, the training system 100 or a different training system can fine-tune some or all parameters of the neural network 110 to adapt the neural network 110 for any of a variety of downstream tasks. For example, the training system 100 can fine-tune one or more components of the neural network 110 for labeled training data specific to the downstream task, such as through supervised learning, reinforcement learning based on human feedback, or instructional tuning. As another example, the training system 100 can keep any portion of the neural network 110 that has been trained during the pre-training phase fixed while learning one or more additional customized components specific to the downstream task.

[0119] After the adaptation phase, the training system 100 or a different inference system 170 deploys the trained neural network 110 on one or more computing devices to perform inference, ie, generate new network outputs 112 for downstream tasks for new network inputs 102 .

[0120] Figure 2 is a flow chart of an example process 200 for training a neural network having parameters on a set of training data to repeatedly update the values of the parameters of the neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, e.g., Figure 1BTraining system 100 - executable process 200.

[0121] The set of training data includes multiple training input sequences. The set of training data may include unlabeled training input sequences. Each training input sequence has multiple positions. Each position has a word-gram selected from a word-gram vocabulary. As mentioned above, the word-gram vocabulary may include one or more of characters, subwords, words, punctuation marks, numbers or other symbols that appear in a corpus of natural language text and / or computer code. In addition or alternatively, the word-gram vocabulary may include words that can represent data other than text. For example, the word-gram vocabulary may include image words that represent a set of discrete image embeddings of an image, which image embeddings may be generated by an image encoder neural network based on processing the image. As another example, the word-gram vocabulary may include audio words that represent code vectors in a codebook of a quantizer, for example, a residual vector quantizer.

[0122] For example, the training input sequences included in the training data can be generated from: a large dataset of text in one or more natural languages, e.g., text publicly available from the internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc., e.g., computer code publicly available from the internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms representing audio recordings, a large dataset of images, each image comprising an array of pixels, a large dataset of videos, each video comprising a sequence of time frames, or a large multimodal dataset comprising a combination of two or more of these datasets.

[0123] Process 200 can be repeated as often as necessary throughout the training of the neural network to obtain an up-to-date estimate of the extent to which the neural network has memorized data from the training data on which the neural network was trained. For example, process 200 can be repeated every predetermined number of times the parameters of the neural network have been updated, every predetermined amount of wall-clock time has elapsed since training began, and so on. As another example, process 200 can be triggered upon receiving an instruction from a system user.

[0124] The system obtains a first training input sequence and a second training input sequence from training data, for example, by sampling (step 202). The first training input sequence includes a first word-gram selected from a word-gram vocabulary. The second training input sequence includes a second word-gram selected from the word-gram vocabulary.

[0125] In some implementations, the training data may be monolingual, and the first training input sequence and the second training input sequence may both be in the same language, e.g., the same natural language or the same programming language. In other implementations, the training data may be multilingual, and the first training input sequence and the second training input sequence may be in different languages. That is, the first training input sequence may be in a first language, and the second training input sequence may be in a second language different from the first language.

[0126] The system generates a modified first training input sequence based on the first training input sequence and the second training input sequence (step 204). Figure 3 Describes the generation of the modified first training input sequence, the figure is Figure 2 Flowchart of sub-steps 302 to 306 of step 204 of process 200.

[0127] The system selects a plurality of second word-grams from the second word-grams included in the second training input sequence (step 302). The number of second word-grams to be selected can be predetermined. The plurality of second word-grams can be sequences of text word-grams that are adjacent to each other within the second training input sequence, i.e., occupy a plurality of consecutive positions in the second training input sequence.

[0128] The system determines a first canary position within the first training input sequence (step 304). In some implementations, the first canary position can be determined independently of the positions of the selected plurality of second word-grams within the second training input sequence. For example, the first canary position can be determined uniformly randomly, i.e., it can be any position within the first training input sequence.

[0129] In some other implementations, the first canary position may be determined based on the positions of the selected second word-grams within the second training input sequence. For example, the first canary position within the first training input sequence may generally correspond to the positions of the selected second word-grams within the second training input sequence.

[0130] That is, if the plurality of second tokens are obtained from the first half of the second training input sequence, for example, from within a threshold number of tokens at the beginning of the second training input sequence, then the first canary position can be similarly located in the first half of the first training input sequence, for example, within a threshold number of tokens at the beginning of the first training input sequence.

[0131] Similarly, if the plurality of second tokens are obtained from the second half of the second training input sequence, for example, from within a threshold number of tokens at the end of the second training input sequence, the first canary position can similarly be located in the second half of the first training input sequence, for example, within a threshold number of tokens at the end of the first training input sequence.

[0132] The system generates a modified first training input sequence by inserting the selected plurality of second tokens into the first training input sequence after the first canary position—that is, by placing the selected plurality of second tokens at multiple (e.g., multiple consecutive) positions in the first training input sequence starting from the first canary position (step 306). Thus, the modified first training input sequence includes the first token originally included in the first training input sequence and the plurality of second tokens obtained from the second training input sequence. The plurality of second tokens obtained from the second training input sequence thus serve as canary tokens, that is, they are additional tokens that were not originally included in the first training input sequence. The first canary position can be the first position in the first training input sequence where one of the canary tokens is inserted. Once generated, the system can include the modified first training input sequence in the set of training data.

[0133] Optionally, in some implementations, the system further generates a modified second training input sequence based on the first training input sequence and the second training input sequence (step 206). Figure 4 Describes the generation of a modified second training input sequence, the diagram is Figure 2 Flowchart of sub-steps 402 to 406 of step 206 of process 200.

[0134] The system selects a plurality of first word-grams from the first word-grams included in the first training input sequence (step 402). The number of first word-grams to be selected can be predetermined, for example, it can be equal to the number of second word-grams in step 304. The plurality of first word-grams can be sequences of word-grams that are adjacent to each other in the first training input sequence, that is, occupy a plurality of consecutive positions in the first training input sequence.

[0135] The system determines a second canary position within the second training input sequence (step 404). In some implementations, the second canary position can be determined independently of the position of the selected plurality of first word-grams within the first training input sequence. For example, the second canary position can be determined uniformly randomly, i.e., it can be any position within the second training input sequence.

[0136] In some other implementations, the second canary position may be determined based on the positions of the selected first word-grams within the first training input sequence. For example, the second canary position within the second training input sequence may generally correspond to the positions of the selected first word-grams within the first training input sequence.

[0137] That is, if the plurality of first tokens are obtained from the first half of the first training input sequence, for example, from within a threshold number of tokens at the beginning of the first training input sequence, then the second canary position can be similarly located in the first half of the second training input sequence, for example, within a threshold number of tokens at the beginning of the second training input sequence.

[0138] Similarly, if the plurality of first tokens are obtained from the second half of the first training input sequence, for example, from within a threshold number of tokens at the end of the first training input sequence, then the second canary position can be similarly located in the second half of the second training input sequence, for example, within a threshold number of tokens at the end of the second training input sequence.

[0139] The system generates a modified second training input sequence by inserting the selected plurality of first tokens after the second canary position in the second training input sequence—that is, by placing the selected plurality of first tokens at multiple (e.g., multiple consecutive) positions in the second training input sequence starting from the second canary position (step 406). Thus, the modified second training input sequence includes the second tokens originally included in the second training input sequence and the plurality of first tokens obtained from the first training input sequence. The plurality of first tokens obtained from the first training input sequence thus serve as canary tokens, that is, they are additional tokens that were not originally included in the second training input sequence. The second canary position can be the first position in the second training input sequence where one of the canary tokens is inserted. Once generated, the system can include the modified second training input sequence in the set of training data.

[0140] The system trains the neural network on the training data including the modified first training input sequence and, optionally, the second training input sequence (step 208). The system performs training in multiple update iterations. In each update iteration, the system updates the parameters of the neural network based on the corresponding gradients of an objective function, e.g., a pre-trained objective function, calculated with respect to the parameters of the neural network using the multiple training input sequences sampled from the training data. The modified first training input sequence (or the second training input sequence) will be included in the multiple training input sequences sampled in at least one of the update iterations.

[0141] After training, the system uses the neural network trained on the training data including the modified first training input sequence and optionally the second training input sequence to generate one or more predicted continuations of the test input sequence by processing the test input sequence according to the updated values of the parameters of the neural network (step 210). The test input sequence includes a subset of the first tokens originally included in the first training input sequence. For example, the subset of the first tokens may include some tokens occupying positions in the modified first training input sequence that are located before the first canary position.

[0142] That is, the neural network is used to predict subsequent tokens following the subset of first tokens in the training data on which the neural network was trained, given a subset of first tokens. To this end, each predicted continuation specifies a plurality of output tokens. Each output token specified by a given predicted continuation is the predicted next token, as calculated by the neural network, that may follow the subset of first tokens in the training data.

[0143] Each predicted continuation generated by the neural network may specify an output word directly, e.g., where the predicted continuation comprises a plurality of output words, or may specify an output word indirectly, e.g., where the predicted continuation defines a score distribution over a word list for each of a plurality of output words, based on which the output word may be determined, i.e., the output word is selected from the word list.

[0144] The system determines an estimate of the extent to which the trained neural network has memorized data in the set of training data on which the neural network was trained by evaluating the plurality of output word-grams specified by each predicted continuation against the selected plurality of second word-grams included in the modified first training input sequence (step 212).

[0145] For example, the system can determine an estimate of the extent to which a trained neural network has memorized data verbatim. Verbatim memorization refers to how well a neural network reproduces specific portions of text that it processed during training, for example, when the neural network reproduces sequences of multiple consecutive tokens from its training corpus (e.g., text tokens that make up a sentence or even a paragraph of text).

[0146] As another example, the system can determine an estimate of the extent to which a trained neural network has gist-remembered the data. Gist memory refers to the ability of the neural network to reproduce summaries of the text it processed during training. For example, the summaries can be extractive summaries, where the output sequence generated by the neural network is a proper subset of one of the training input sequences processed by the neural network during training, i.e., consists of tokens from the training input sequence.

[0147] In an implementation, each predicted continuation includes a plurality of output tokens, and the system can compare the plurality of output tokens with the selected plurality of second tokens included in the modified first training input sequence to determine whether a match exists between any of the output tokens and any of the selected plurality of second tokens. In this way, the system can determine how many of the selected plurality of second tokens included in the modified first training input sequence are also included in the plurality of output tokens included in each predicted continuation.

[0148] In these implementations, the estimate of the degree of training data memorization can be determined based on the total match count between the output word-gram and the selected plurality of second word-grams, e.g., be (approximately) proportional to or otherwise dependent on the total match count.

[0149] In an implementation, each predicted continuation defines a score distribution over the word-gram vocabulary for each of the plurality of output word-grams, and the system can compute a likelihood assigned to each word-gram of at least a subset of the selected plurality of second word-grams by means of the score distribution of the positions included in each predicted continuation. For example, for a given second word-gram in the selected plurality of second word-grams, its assigned likelihood can be a combination, such as a product, of the scores included in the corresponding score distributions generated for the given second word-gram. As a specific example of this, assume that one of the selected plurality of second word-grams is the word-gram "cat" and that the predicted continuation has three positions. For each position, the score distribution generated by the neural network (which can be used to select the predicted word-gram for that position) includes a score for the word-gram "cat" as well as scores for other word-grams in the vocabulary. For example, when the score for the first position in the predicted continuation is P(cat)=0.05, the score for the second position in the predicted continuation is P(cat)=0.1, and the score for the third position in the predicted continuation is P(cat)=0.04, then the likelihood of the word "cat" can be calculated as the product of these three scores: 0.05×0.1×0.04=0.0002. In this way, the system can determine the likelihood that any of the selected plurality of second word-grams included in the modified first training input sequence can be selected for inclusion in the plurality of output word-grams included in each predicted continuation.

[0150] In these implementations, the estimate of the degree of training data memorization can be determined based on the calculated likelihood of each second word-gram in at least a subset of the selected plurality of second word-grams, e.g., be (approximately) proportional to or otherwise depend on the likelihood.

[0151] The system determines whether the estimate of how well the trained neural network has memorized data from the set of training data satisfies a predetermined threshold by comparing the estimate to a predetermined threshold (step 214). For example, step 214 may involve determining whether the plurality of output word-grams includes more than a threshold number of the selected plurality of second word-grams. As another example, step 214 may involve determining whether the calculated likelihood of each second word-gram in at least a subset of the selected plurality of second word-grams satisfies a likelihood threshold.

[0152] In response to determining that the estimate meets the predetermined threshold, the system applies one or more adjustments to the training of the neural network to adjust the estimate, step 216. For example, if the estimate is above the predetermined threshold, the estimate may meet the predetermined threshold when a lower estimate indicates a lower degree of memory of the training data.

[0153] Generally speaking, adjustments can modify any suitable aspect of the training of the neural network that may result in a change in the estimate, for example, to lower the estimate when a lower estimate indicates a lower degree of memorization of the training data. To name a few examples, the system can adjust the pre-training objective function, for example, using an alternative or new loss term that penalizes the neural network for verbatim memorization, assigning different weights to existing loss terms, adjusting hyperparameters—for example, adjusting the learning rate or the choice of optimizer, and so on.

[0154] This memory estimation technique enables neural networks to achieve a variety of technical improvements that were previously impossible. Importantly, incorporating memory estimation techniques into neural network training can lead to a significant reduction in verbatim memorization by trained neural networks, particularly for data that is not heavily repeated in the training data. Furthermore, memory estimation techniques make it more practical to systematically study potential privacy risks in downstream use cases of trained neural networks, including conversational or summarization tasks, and further facilitate the implementation of additional procedural and / or technical safeguards that may be necessary to protect against extraction attacks.

[0155] Figure 5 FIG1 is a flow chart of another example process 500 for training a neural network having parameters on a set of training data to repeatedly update the values of the parameters of the neural network. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, such as training system 100 of FIG1 , can perform process 500.

[0156] The set of training data includes multiple training input sequences. The set of training data may include unlabeled training input sequences. Each training input sequence has multiple positions. Each position has a word-gram selected from a word-gram vocabulary. As mentioned above, the word-gram vocabulary may include one or more of characters, subwords, words, punctuation marks, numbers or other symbols that appear in a corpus of natural language text and / or computer code. In addition or alternatively, the word-gram vocabulary may include words that can represent data other than text. For example, the word-gram vocabulary may include image words that represent a set of discrete image embeddings of an image, which image embeddings may be generated by an image encoder neural network based on processing the image. As another example, the word-gram vocabulary may include audio words that represent code vectors in a codebook of a quantizer, for example, a residual vector quantizer.

[0157] For example, the training input sequences included in the training data can be generated from: a large dataset of text in one or more natural languages, e.g., text publicly available from the internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc., e.g., computer code publicly available from the internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms representing audio recordings, a large dataset of images, each image comprising an array of pixels, a large dataset of videos, each video comprising a sequence of time frames, or a large multimodal dataset comprising a combination of two or more of these datasets.

[0158] Process 500 may be repeatedly performed for a relatively small portion of the training input sequences included in the training data. For example, iterations of process 500 may be performed for 0.5%, 1%, or 5% of all the training input sequences included in the training data.

[0159] The system obtains a third training input sequence from training data, such as by sampling (step 502).

[0160] The system determines a determined toxicity level from a plurality of toxicity levels based on the content represented by the third training input sequence (step 504).To automatically determine the toxicity level of the third training input sequence, the system can use, for example, a text classifier machine learning model or a deterministic text-based classification algorithm.

[0161] In some implementations, the plurality of toxicity levels may include two or more toxicity levels, e.g., level 0 and level 1. In some implementations, the plurality of toxicity levels may include three or more toxicity levels, e.g., level 0, level 1, and level 2, where a larger number indicates a higher toxicity level.

[0162] The system adds a specific toxic word corresponding to the determined toxicity level to the third training input sequence (step 506). For example, the specific toxic word can be prepended to the beginning of the third training input sequence, appended to the end of the third training input sequence, or inserted into any appropriate position within the third training input sequence.

[0163] In some implementations, the system determines whether to add a particular toxic word to the third training input sequence based on the total number of training input sequences to which toxic words have been added. For example, the system will determine to add a particular toxic word if toxic words (corresponding to the same or different toxicity levels) have been added to no more than a predetermined number of training input sequences included in the training data.

[0164] The system trains the neural network on the training data, including the third training input sequence, which now includes the particular toxic word (step 508). The system performs training in multiple update iterations. In each update iteration, the system updates the parameters of the neural network based on the corresponding gradients of the objective function—e.g., a pre-trained objective function—with respect to the parameters of the neural network using the multiple training input sequences sampled from the training data. The third training input sequence will be included in the multiple training input sequences sampled in at least one of the update iterations.

[0165] Advantageously, incorporating such toxic tokens into training data improves the performance of the trained neural network. For example, an output sequence generated by the trained neural network upon processing a context sequence that includes—e.g., is preceded or followed by—a toxic token corresponding to a low toxicity level will be less likely, sometimes significantly less likely, to contain toxic content, and the trained neural network will thus be more suitable for deployment in various environments, such as within educational or medical organizations, where any harmful behavior of the deployed neural network could result in serious consequences.

[0166] Figure 6 is a flow chart of an example process 600 for generating an output sequence using a neural network. The neural network may be a Figure 5 The process 600 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, such as the training system 100 of FIG. 1 , can perform the process 600.

[0167] The system obtains a context sequence comprising a plurality of input tokens (step 602). In some cases, the context sequence comprises user-specified prompt text, i.e., comprises input tokens submitted by a system user. In some cases, the context sequence comprises predetermined prompt text stored in the system. In some cases, the context sequence comprises both user-specified prompt text and predetermined prompts, e.g., it may be a concatenation of user-specified prompt text and predetermined prompts.

[0168] The system generates a quality-controlled context sequence from the context sequence (step 604). The quality-controlled context sequence can be generated by adding a specific toxicity token to the context sequence. The specific toxicity token can be any toxicity token selected from a plurality of toxicity tokens representing different toxicity levels of content to be represented by the output sequence. For example, the quality-controlled context sequence includes a specific quality token followed by a plurality of input tokens, or vice versa.

[0169] The system generates an output sequence based on processing the quality-controlled context sequence using a neural network (step 606). The output sequence includes a plurality of output tokens. For example, if a particular toxicity token included in the quality-controlled context sequence corresponds to a low toxicity level, then since the neural network has been described above with reference to Figure 5 The training is such that a particular toxicity will cause the neural network to generate an output sequence that is less likely (sometimes significantly less likely) to contain toxic content, e.g., relative to an output sequence generated by the same neural network (or another conventionally trained neural network) from a context sequence that does not include such toxic tokens.

[0170] Figure 7 FIG7 is a flow chart of another example process 700 for training a neural network having parameters on a set of training data to repeatedly update the values of the parameters of the neural network. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, such as training system 100 of FIG1 , can perform process 700.

[0171] The set of training data includes multiple training input sequences. The set of training data may include unlabeled training input sequences. Each training input sequence has multiple positions. Each position has a word selected from a text word list. As mentioned above, the text word list may include one or more of characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code.

[0172] For example, the training input sequences included in the training data can be generated from electronic documents (e.g., books, web pages (e.g., HTML pages), news articles, or other documents) that can be found in a corpus (e.g., a collection or repository of content) available on the internet or another text corpus.

[0173] The system obtains unlabeled text in multiple languages (step 702). That is, the system obtains one or more training input sequences each including text tokens in a first language, obtains one or more other training input sequences each including text tokens in a second language, and obtains one or more other training input sequences each including text tokens in a third language.

[0174] The system generates a first multilingual training dataset comprising a plurality of first multilingual training pairs from unlabeled text (step 704). Each first multilingual training pair comprises a first source text sequence and a first target text sequence. In some implementations, the first source text sequence and the first target text sequence have similar lengths, i.e., include approximately the same number of text tokens, while in other implementations, they may have different lengths.

[0175] The system generates a second multilingual training dataset comprising a plurality of second multilingual training pairs from the unlabeled text (step 706 ). Each second multilingual training pair comprises a second source text sequence and a second target text sequence.

[0176] In some implementations, the first multilingual training dataset and the second multilingual training dataset include a set of overlapping text sequences. For example, a first source text sequence included in the first multilingual training dataset is the same as a second source text sequence included in the second multilingual training dataset. In other implementations, the first multilingual training dataset and the second multilingual training dataset include different sets of text sequences. For example, a first source text sequence included in the first multilingual training dataset is different from a second source text sequence included in the second multilingual training dataset.

[0177] Specifically, however, the first source text sequence and the second source text sequence are both in the same language, while the first target text sequence and the second target text sequence are in different languages. For example, the first source text sequence and the second source text sequence both include text tokens in a first language, the first target text sequence includes text tokens in a second language, and the second target text sequence includes text tokens in a third language.

[0178] The system trains a neural network on the first multilingual training dataset and the second multilingual training dataset (step 708). The system performs training in multiple update iterations. In each update iteration, the system updates the parameters of the neural network based on the corresponding gradients of the objective function (e.g., a pre-trained objective function) with respect to the parameters of the neural network, which are calculated using the plurality of first multilingual training pairs sampled from the first multilingual training dataset or the plurality of second multilingual training pairs sampled from the second multilingual training dataset.

[0179] Merely as an illustrative example, for a given first multilingual training pair sampled from a first multilingual training dataset, the system may train a neural network on a pre-training language modeling task that requires predicting a predicted text token in a second language given (a portion of) a first source text sequence comprising text tokens in a first language, the predicted text tokens constituting a first target text sequence following the first source text sequence in the given first multilingual training pair.

[0180] After training, the system adapts the trained neural network to perform a multilingual downstream task, such as by fine-tuning, zero-shot, or few-shot adaptation (step 710). A multilingual downstream task can be any task that involves processing text tokens in two or more languages.

[0181] For example, a trained neural network may be adapted to perform a translation task by processing an input text sequence in a source language to generate an output text sequence in a target language, the output text sequence being a translation of the input text sequence into the target language.

[0182] As another example, a trained neural network may be adapted to perform a multilingual question answering task by processing an input text sequence in both a source language and a target language to generate an output text sequence in the target language, the output text sequence being a response to the input text sequence.

[0183] Training a neural network on multilingual training data improves its ability to understand and generate multilingual output, so it can be more easily adapted to any of a range of multilingual downstream tasks. Once adapted, the neural network can achieve or even exceed the performance of conventional pre-trained neural networks on any of the multilingual downstream tasks, even though the adaptation process consumes fewer computational resources, is faster in terms of wall-clock time, or both.

[0184] Figure 8800 is a flow chart of another example process 800 for training a neural network having parameters on a mixture of different pre-training tasks on a set of training data comprising a plurality of training input sequences. The plurality of training input sequences may include unlabeled training input sequences. The neural network may be an autoregressive generative neural network including one or more self-attention layers. For convenience, process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, such as training system 100 of FIG. 1 , may perform process 800.

[0185] For example, the training input sequences included in the training data can be generated from: a large dataset of text in one or more natural languages, e.g., text publicly available from the internet or another text corpus, a large dataset of computer code in one or more programming languages, e.g., Python, C++, C#, Java, Ruby, PHP, etc., e.g., computer code publicly available from the internet or another code repository, a large dataset of audio samples, e.g., audio recordings or waveforms representing audio recordings, a large dataset of images, each image comprising an array of pixels, a large dataset of videos, each video comprising a sequence of time frames, or a large multimodal dataset comprising a combination of two or more of these datasets.

[0186] The system may repeatedly perform iterations of process 800 to repeatedly update parameters of the neural network until a termination criterion has been met, e.g., until a threshold number of iterations of process 800 have been performed, until a threshold amount of wall-clock time has elapsed, or until the values of the parameters have converged.

[0187] The system obtains a plurality of training input sequences from training data, for example by sampling (step 802). Each training input sequence has a plurality of positions. Each position has a word-gram selected from a word-gram vocabulary. As mentioned above, the word-gram vocabulary may include one or more of characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. Additionally or alternatively, the word-gram vocabulary may include words that may represent data other than text. For example, the word-gram vocabulary may include image words that represent a set of discrete image embeddings of an image, which may be generated by an image encoder neural network based on processing the image. As another example, the word-gram vocabulary may include audio words that represent code vectors in a codebook of a quantizer, for example, a residual vector quantizer.

[0188] The system trains the neural network on a mixture of different pre-training tasks (step 804). Each pre-training task uses a corresponding pre-training objective function. Each pre-training task focuses on training the neural network to learn a different aspect of language modeling. By training the neural network on a mixture of different pre-training tasks, the system can train the neural network in a manner that improves its generalization ability during the adaptation phase—that is, relative to training the neural network on a single pre-training task.

[0189] A mixture of different pre-training tasks includes two or more of the following: (i) causal language modeling pre-training tasks (see below for Figure 9 further described), (ii) prefix language modeling pre-training task (as described below in Figure 10 further described), or (iii) span-corrupted pre-training tasks (as described below in Figure 11 further described).

[0190] In some implementations, the system maintains a corresponding weight for each of a plurality of different pre-training tasks. In some of these implementations, the system may repeatedly: sample a plurality of training input sequences from the training data, sample pre-training tasks from the plurality of different pre-training tasks according to the corresponding weights of the plurality of different pre-training tasks, and train the neural network to perform the sampled pre-training tasks by using the plurality of training input sequences. In this way, the system trains the neural network on the same pre-training task using the plurality of training input sequences. In some other implementations of these implementations, the system may sample a plurality of training input sequences from the training data, and then, for each training input sequence in a subset of the plurality of training input sequences: sample a pre-training task from the plurality of different pre-training tasks according to the corresponding weights of the plurality of different pre-training tasks, and train the neural network to perform the sampled pre-training task by using the subset of the plurality of training input sequences. In this way, the system trains the neural network across different pre-training tasks using the plurality of training input sequences.

[0191] Optionally, after training, the system adapts the trained neural network to perform one or more downstream tasks, such as by fine-tuning, zero-shot or few-shot adaptation. The downstream tasks may include any combination of one or more of the machine learning tasks mentioned above and possibly other tasks. Additionally or alternatively, the system outputs data specifying the trained neural network—e.g., data specifying the trained values of the parameters and, optionally, the architecture of the neural network—to another system or for storage in one or more memory devices for future use.

[0192] Figure 9 yes Figure 8Flowchart of sub-steps 902 to 906 of step 804 of the process of . By repeatedly performing iterations of sub-steps 902 to 906, the system can train a neural network on a causal language modeling pre-training task based on optimizing a causal language modeling objective function.

[0193] The system generates a plurality of causal language modeling input sequences from a plurality of training input sequences (step 902). In some implementations, the training input sequences can be used as causal language modeling input sequences without further processing.

[0194] The system processes each causal language modeling input sequence using a neural network to generate, for each word-gram in the causal language modeling input sequence, a causal prediction for the word-gram occupying the particular position of the word-gram in the causal language modeling input sequence, given the word-gram at a previous position preceding the particular position in the causal language modeling input sequence (step 904).

[0195] That is, for any given position in a causal language modeling input sequence, the neural network processes the tokens at the previous position preceding the given position in the causal language modeling input sequence to generate as output a prediction for the token occupying the given position in the causal language modeling input sequence.

[0196] For example, for any given position, the neural network can generate a score distribution over word-grams in the vocabulary from which predicted word-grams for occupying the given position can be sampled. The score distribution can assign a corresponding score to each word-gram in the vocabulary.

[0197] When processing an input sequence to generate as an output a prediction for a token occupying a given position, each of the one or more self-attention layers within the neural network applies a masked self-attention mechanism to previous positions in the causal language modeling input sequence, such that the given position does not pay attention to any data at positions that do not precede the given position - that is, the self-attention layer does not generate non-zero attention weights for any data at positions that do not precede the given position, or in other words, the self-attention layer only generates non-zero weights for positions that precede the given position in the causal language modeling input sequence.

[0198] The system determines updates to the parameter values of the neural network based on optimizing the causal language modeling objective function (step 906). The system can do this by calculating, for each given position in each causal language modeling input sequence, the corresponding gradient of the objective function with respect to the parameters of the neural network by backpropagating through the appropriate parameters of the neural network. The system can then determine the updates by applying an update rule, such as an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the corresponding gradients.

[0199] The causal language modeling objective function can be any function that measures the quality of the causal predictions generated by the neural network. For example, the causal language modeling objective function can include a cross-entropy loss term or a negative log-likelihood loss term that measures, for each given position in each causal language modeling input sequence, the difference between: (i) a predetermined fraction of true-valued tokens occupying the given position included in the causal language modeling input sequence and (ii) the corresponding scores assigned to the true-valued tokens by the score distribution generated by the neural network. The loss function can also include other terms, such as regularization terms, auxiliary loss terms, and the like.

[0200] Figure 10 yes Figure 8 Flowchart of sub-steps 1002 to 1006 of step 804 of the process of FIG. By repeatedly performing iterations of sub-steps 1002 to 1006, the system can train a neural network on the prefix language modeling pre-training task based on optimizing the prefix language modeling objective function.

[0201] The system generates a plurality of prefix language modeling input sequences from a plurality of training input sequences (step 1002). Each prefix language modeling input sequence includes a prefix input sequence followed by a suffix input sequence. The prefix input sequence and the suffix input sequence can each include a respective subset of the tokens included in the prefix language modeling text sequence.

[0202] In some implementations, a prefix language modeling input sequence can be generated from a training input sequence by dividing the training input sequence into two sequences (or subsequences), namely, a prefix input sequence and a suffix input sequence. For example, the prefix input sequence can include a first subset of consecutive tokens included in the prefix language modeling text sequence, and the suffix input sequence can include a second subset of consecutive tokens included in the prefix language modeling text sequence, the second subset following the first subset of consecutive tokens.

[0203] The system processes each prefix language modeling input sequence using a neural network to generate, for each word-gram in the suffix input sequence, a causal prediction corresponding to the word-gram occupying the particular position in the suffix input sequence, conditioned on (a) the word-gram in the prefix input sequence and (b) the word-gram at any previous position in the suffix input sequence that precedes the particular position in the prefix language modeling input sequence (step 1004).

[0204] That is, for any given position in the suffix input sequence in the prefix language modeling input sequence, the neural network processes an input sequence including (a) tokens in the prefix input sequence that precede the suffix input sequence in the prefix language modeling input sequence and (b) tokens at any previous positions in the suffix input sequence that precede the particular position in the prefix language modeling input sequence to generate as an output a prediction for the token occupying the given position in the suffix input sequence.

[0205] For example, for any given position, the neural network can generate a score distribution over word-grams in the vocabulary from which predicted word-grams for occupying the given position can be sampled. The score distribution can assign a corresponding score to each word-gram in the vocabulary.

[0206] When processing an input sequence to generate as an output a prediction for a token occupying a given position, each of one or more self-attention layers within the neural network (i) applies a bidirectional, unmasked attention mechanism to the position in the prefix input sequence, and (ii) applies a masked self-attention mechanism to the position in the suffix input sequence, such that the given position attends to all positions in the prefix input sequence and any previous positions in the suffix input sequence that precede the given position. Concurrently, any position in the prefix input sequence may attend to any other position in the prefix input sequence, including any subsequent positions after the position in the prefix input sequence. However, the given position does not attend to any data at positions after the given position in the suffix input sequence, i.e., the self-attention layer generates a zero attention weight for any data at positions after the given position, or in other words, the self-attention layer generates a non-zero attention weight only for the position in the prefix input sequence and the positions before the given position in the suffix input sequence.

[0207] The system determines updates to the parameter values of the neural network based on optimizing the prefix language modeling objective function (step 1006). The system can do this by, for each given position in the suffix input sequence within each prefix language modeling input sequence, calculating the corresponding gradient of the objective function with respect to the parameters of the neural network by backpropagating through the appropriate parameters of the neural network. The system can then determine the updates by applying an update rule, such as an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the corresponding gradients.

[0208] The prefix language modeling objective function can be any function that measures the quality of the prefix predictions generated by the neural network. For example, the prefix language modeling objective function can include a cross-entropy loss term or a negative log-likelihood loss term that measures, for each given position in the suffix input sequence in each prefix language modeling input sequence, the difference between: (i) a predetermined fraction of true-valued tokens occupying the given position included in the suffix input sequence and (ii) the corresponding scores assigned to the true-valued tokens by the score distribution generated by the neural network. The loss function can also include other terms, such as regularization terms, auxiliary loss terms, and the like.

[0209] Figure 11 yes Figure 8 Flowchart of sub-steps 1102 to 1106 of step 804 of the process of FIG. By repeatedly performing iterations of sub-steps 1102 to 1106, the system can train a neural network on the span damage pre-training task based on optimizing the span damage objective function.

[0210] The system generates a plurality of span-masked input sequences from a plurality of training input sequences (step 1102). Each span-masked input sequence includes a plurality of tokens separated by one or more mask tokens. A "masked token" is a token that includes a predetermined value and indicates that a token to occupy the position of the masked token has not yet been generated (e.g., has not yet been selected from a token vocabulary).

[0211] In some implementations, a span-masked input sequence can be generated from a corresponding training input sequence by replacing a subset of word-grams included in the training input sequence with masked word-grams. The word-gram subset can include word-grams that are adjacent to each other within the training input sequence, i.e., occupy multiple consecutive positions in the training input sequence.

[0212] For each span-masked input sequence, the system processes the span-masked input sequence using a neural network to generate predictions for one or more tokens that would occupy corresponding positions of the one or more masked tokens in the span-masked input sequence (step 1104). For example, for any position occupied by a masked token, the neural network can generate a score distribution over tokens in the vocabulary from which a predicted token to occupy that position, i.e., to replace the masked token, can be sampled. The score distribution can assign a corresponding score to each of the tokens in the vocabulary.

[0213] The system determines updates to the parameter values of the neural network based on optimizing the span-corruption objective function (step 1106). The system can do this by, for each given position in each prefix language modeling input sequence, calculating the corresponding gradient of the objective function with respect to the parameters of the neural network by backpropagating through the appropriate parameters of the neural network. The system can then determine the updates by applying an update rule, such as an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the corresponding gradients.

[0214] The span corruption objective function can be any function that measures the quality of the span predictions generated by the neural network. For example, the span corruption can include a cross-entropy loss term or a negative log-likelihood loss term that measures, for any position occupied by a masked token in each span-masked input sequence, the difference between: (i) a predetermined fraction of true-value tokens occupying that position included in the corresponding training input sequence (based on which the span-masked input sequence was generated) and (ii) the corresponding scores assigned to the true-value tokens by the score distribution generated by the neural network. The loss function can also include other terms, such as regularization terms, auxiliary loss terms, and the like.

[0215] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, this means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform that operation or action. For one or more computer programs to be configured to perform a particular operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0216] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0217] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0218] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages or declarative or procedural languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0219] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0220] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or computers.

[0221] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0222] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory, or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into a dedicated logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0223] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0224] To provide for user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, a computer can interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving responsive messages from the user in response.

[0225] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0226] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework or the JAX framework.

[0227] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with implementations of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0228] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, for example, the results of the user interaction, may be received from the device at the server.

[0229] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a subcombination or a variant of a subcombination.

[0230] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0231] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, wherein the method comprises: Obtaining a first training input sequence including a first word-gram and a second training input sequence including a second word-gram; Generate a modified first training input sequence, comprising: selecting a plurality of second word-grams from the second word-grams included in the second training input sequence; determining a first canary position within the first training input sequence; and inserting the selected plurality of second word-grams into the first training input sequence at positions after the first canary position; training a neural network using a training data set comprising the modified first training input sequence, including learning parameter values of the neural network; After the training, generating, using the trained neural network, from a test input sequence including a subset of the first word-grams, one or more predicted continuations of the test input sequence, wherein each predicted continuation specifies a plurality of output word-grams; and Determining an estimate of how well the trained neural network has memorized data in the training dataset includes evaluating the plurality of output word-grams specified by each predicted continuation against a selected plurality of second word-grams included in the modified first training input sequence.

2. The method of claim 1 , wherein the training data set comprises the modified first training input sequence and a modified second training input sequence, wherein the modified second training input sequence is generated by: selecting a plurality of first word-grams from the first word-grams included in the first training input sequence; determining a second canary position within the second training input sequence; as well as The selected plurality of first word-grams are inserted into the second training input sequence at positions subsequent to the second canary position.

3. The method of any one of claims 1 to 2, wherein each predicted continuation comprises the plurality of output word-grams, and wherein evaluating the plurality of output word-grams against the selected plurality of second word-grams comprises: It is determined how many second word-grams of the selected plurality of second word-grams included in the modified first training input sequence are included in the plurality of output word-grams.

4. The method of any one of claims 1 to 2, wherein each predicted continuation defines, for each output word-gram position, a corresponding probability distribution over a word-gram vocabulary, and wherein evaluating the plurality of output word-grams against the selected plurality of second word-grams comprises: The likelihood of being assigned to at least a subset of the selected plurality of second word-grams is calculated by the probability distribution of the output word-gram positions.

5. The method according to any one of claims 3 to 4, further comprising: determining the extent to which the estimate satisfies a threshold; as well as In response, one or more adjustments are applied to the training of the neural network to reduce the estimate.

6. The method of claim 5, wherein determining that the estimate satisfies the threshold degree comprises: It is determined that the plurality of output word-grams includes more than a threshold number of the selected plurality of second word-grams.

7. The method of claim 5, wherein determining that the estimate satisfies the threshold degree comprises: It is determined that the likelihood satisfies a likelihood threshold.

8. The method of any one of claims 1 to 7, wherein determining the first canary position within the first training input sequence comprises: The first canary position is determined based on positions of the selected plurality of second word-grams within the second training input sequence.

9. The method of any one of claims 1 to 8, wherein selecting the plurality of second word-grams comprises: A predetermined number of second word-grams adjacent to each other in the second training input sequence are selected.

10. The method of any one of claims 1 to 9, wherein the first training input sequence and the second training input sequence are both in the same natural language.

11. The method of any one of claims 1 to 9, wherein the first training input sequence is in a first natural language and the second training input sequence is in a second natural language different from the first language.

12. The method of any one of claims 1 to 10, wherein the training data set further comprises a modified third training input sequence, the modified third training input sequence being generated by: Obtaining a third training input sequence; determining a determined toxicity level from a plurality of toxicity levels based on content represented by the third training input sequence; A specific toxic word corresponding to the determined toxicity level is prepended to the third training input sequence.

13. The method of claim 12, wherein the plurality of toxicity levels comprises three or more toxicity levels.

14. The method according to any one of claims 12 to 13, wherein prepending the specific toxic word to the third training input sequence comprises: Whether to prepend the specific poison word to the third training input sequence is determined based on the total number of training input sequences to which poison words have been prepended.

15. The method of any one of claims 12 to 14, further comprising: Obtaining a context sequence including a plurality of input tokens; generating a quality-controlled context sequence from the context sequence by adding to the context sequence a specific toxic word-gram selected from a plurality of toxic word-grams representing a toxicity level of content represented by an output sequence; as well as An output sequence is generated based on processing the quality-controlled context sequence using the trained neural network.

16. The method of claim 15, wherein the quality-controlled context sequence includes the specific quality word-gram followed by the plurality of input word-grams.

17. The method of any one of claims 1 to 16, wherein the neural network is a Transformer neural network that generates the output tokens autoregressively.

18. A method performed by one or more computers, wherein the method comprises: Obtaining untagged text in multiple languages; generating a first multilingual training dataset from the unlabeled text, wherein the first multilingual training dataset includes a plurality of first multilingual training pairs, wherein each first multilingual training pair includes a first source text sequence and a first target text sequence; generating a second multilingual training dataset from the unlabeled text, the second multilingual training dataset comprising a plurality of second multilingual training pairs, wherein each second multilingual training pair comprises a second source text sequence and a second target text sequence, and wherein the first source text sequence and the second source text sequence are both in the same language, and the first target text sequence and the second target text sequence are in different languages; and A language model neural network is trained by using the first multilingual training dataset and the second multilingual training dataset based on optimizing a pre-training objective function, including learning parameter values of the neural network.

19. The method of claim 18, further comprising: After the training, the trained neural network is adapted to perform a translation task by processing an input text sequence in a source language to generate an output text sequence in a target language, the output text sequence being a translation of the input text sequence into the target language.

20. The method of claim 18, further comprising: After the training, the trained neural network is adapted to perform a multilingual question answering task by processing input text sequences in both a source language and a target language to generate an output text sequence in the target language, the output text sequence being a response to the input text sequence.

21. A method performed by one or more computers, wherein the method comprises: Obtain a plurality of unlabeled text sequences, wherein each unlabeled text sequence includes a plurality of text tokens; training an autoregressive generative neural network comprising one or more self-attention layers based on optimizing a plurality of different pre-trained objective functions, the plurality of different pre-trained objective functions comprising (i) a causal language modeling objective function and (ii) a prefix language modeling objective function, Wherein training the autoregressive generative neural network based on optimizing the causal language modeling objective function comprises: generating a plurality of causal language modeling text sequences from the plurality of unlabeled text sequences; processing each causal language modeling text sequence using the autoregressive generative neural network to generate, for each token in the causal language modeling text sequence, a causal prediction corresponding to the text token occupying a specific position of the text token in the causal language modeling text sequence, conditioned on the text token at a previous position in the causal language modeling text sequence, wherein the one or more self-attention layers within the autoregressive generative neural network apply a masked self-attention mechanism to the previous position in the causal language modeling text sequence; and determining an update to a parameter value of the autoregressive generative neural network based on a quality of the causal prediction, Wherein training the autoregressive generative neural network based on optimizing the prefix language modeling objective function comprises: generating a plurality of prefix language modeling text sequences from the plurality of unlabeled text sequences, each prefix language modeling text sequence comprising a prefix text sequence followed by a suffix text sequence; processing each prefix language modeling text sequence using the autoregressive generative neural network to generate, for each word-gram in the suffix text sequence, a causal prediction of a text word-gram corresponding to a specific position of the word-gram in the suffix text sequence, conditioned on the word-gram in the prefix text sequence and the word-gram at any previous position in the suffix text sequence, wherein the one or more self-attention layers within the autoregressive generative neural network apply a bidirectional, non-masked attention mechanism to the positions in the prefix text sequence and a masked self-attention mechanism to the positions in the suffix text sequence such that each position in the suffix text sequence pays attention to the position in the prefix text sequence and any previous position in the suffix text sequence; and An update to the parameter values of the autoregressive generative neural network is determined based on a quality of the prefix prediction.

22. The method of claim 21 , wherein the plurality of different pre-trained objective functions include (iii) a span-corruption objective function, and wherein training the autoregressive generative neural network based on optimizing the span-corruption objective function comprises: generating a plurality of span-masked text sequences from the plurality of unlabeled text sequences, wherein each span-masked text sequence comprises a plurality of text tokens separated by one or more mask tokens; processing each span-masked text sequence using the autoregressive generative neural network to generate span predictions corresponding to one or more text tokens occupying respective positions of the one or more masked tokens in the span-masked text sequence; as well as Updates to the parameter values of the autoregressive generative neural network are determined based on a quality of the span prediction.

23. The method of any one of claims 21 to 22, wherein training the autoregressive generative neural network based on optimizing the plurality of different pre-trained objective functions comprises: receiving a user input specifying a respective weight assigned to each of the plurality of different pre-trained objective functions; as well as A pre-trained objective function is repeatedly selected from the plurality of different pre-trained objective functions based on the specified weights.

24. The method of claim 23, further comprising: Sampling batches of unlabeled text sequences; as well as The batch of unlabeled text sequences is processed according to the selected pre-trained objective function.

25. The method of any one of claims 21 to 24, wherein the autoregressive generative neural network is a decoder-only attention neural network.

26. The method of any one of claims 21 to 24, wherein the autoregressive generative neural network is an encoder-decoder attention neural network.

27. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective methods of any one of claims 1 to 26.

28. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 26.

Images