Enhanced Training of Large Neural Networks

Description

1 / 67 Enhanced Training of Large Neural Networks Cross-reference to Related Requests

[0001] This application claims priority over U.S. Provisional Application No. 63 / 465,487, filed May 10, 2023. The disclosure of the earlier application is considered part of, and is incorporated by reference into, the disclosure of this application. FUNDAMENTALS

[0002] This descriptive report refers to neural networks.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input for the next layer in the network, that is, the next hidden layer or the output layer. Each layer of the network generates an output from a given input according to the current value inputs of a respective set of parameters.

[0004] Training neural networks on a large scale can be very difficult due to their immense model size and high computational cost. This is especially true because other issues, such as literal memorization of training data (which can potentially lead to unwanted data leakage) and the generation of toxic content during inference, emerge as model sizes increase. SUMMARY

[0005] This descriptive report describes a system implemented as computer programs on one or more computers in one or more locations that train a neural network to perform any one of a variety of downstream machine learning tasks on a network input.

[0006] In general, the training process has two stages: a pre-training stage, during which the neural network is pre-trained using a large amount of unlabeled training data, followed by a Petition 870250091754, dated 08 / 10 / 2025, page 28 / 116 2 / 67 adaptation stage during which the pre-trained neural network is adapted, for example, fine-tuned or with few trials, to one or more of the downstream tasks using training data that are specific to the downstream tasks. Particularly, during the pre-training stage, the system uses a series of techniques to improve the effectiveness, efficiency, or both of the training process.

[0007] Specific modalities of the subject matter described in this descriptive report may be implemented in order to achieve one or more of the following advantages.

[0008] Using some techniques described in this descriptive report, for example, techniques related to inserting canaries into the training data, a training system can test a neural network to determine the probability of the neural network having memorized training data and, optionally, modify the neural network training to reduce the probability. In this way, the techniques described in this descriptive report can test whether a neural network may be subject to data leakage and, optionally, modify the neural network training to reduce the probability of data leakage.

[0009] Using some of the techniques described, for example, techniques related to conditioning / toxicity control techniques, the training system can improve the overall quality of the outputs generated by the neural network after training, that is, at inference time.

[0010] Through a combination of a large, purposefully curated pretraining corpus from a diverse set of sources—including web documents, books, code, mathematics, and conversational data—and efficient training techniques that enhance the neural network's multilingual and reasoning capabilities while alleviating scaling-related issues, some techniques described in this descriptive report can train a neural network for significantly faster convergence, for example, in terms of time spent in Petition 870250091754, dated 08 / 10 / 2025, page 29 / 116 3 / 67 reality, as well as consuming fewer computing resources than existing training systems.

[0011] From another point of view, this increase in training efficiency makes it possible to train a neural network smaller than some known neural networks using comparable amounts of computing resources. Once trained, the neural network will be able to achieve competitive performance with or even exceeding the state of the art in a wide range of tasks, despite having relatively smaller model sizes, thus being more suitable for implementation in mobile devices, embedded systems, or other hardware platforms with limited computing resources.

[0012] The details of one or more embodiments of the subject matter of this descriptive report are set forth in the accompanying drawings and the description below. Other elements, aspects and advantages of the subject matter will become apparent from the description, figures and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1A shows an example of a neural network.

[0014] FIG. 1B shows an example of a training system and an example inference system.

[0015] FIG. 2 is a flow diagram of an example process for training a neural network.

[0016] FIG. 3 is a flow diagram of sub-steps of one of the steps in the process of FIG. 2.

[0017] FIG. 4 is a flow diagram of sub-steps of one of the steps in the process of FIG. 2.

[0018] FIG. 5 is a flow diagram of another example process for training a neural network.

[0019] FIG. 6 is a flow diagram of an example process for using a trained neural network to generate an output sequence.

[0020] FIG. 7 is a flow diagram of another example of a process for training a neural network. Petition 870250091754, dated 08 / 10 / 2025, page 30 / 116 4 / 67

[0021] FIG. 8 is a flow diagram of another example process for training a neural network based on the optimization of multiple different pretraining objective functions.

[0022] FIG. 9 is a flow diagram of sub-steps of one of the steps in the process of FIG. 8.

[0023] FIG. 10 is a flow diagram of sub-steps of one of the steps in the process of FIG. 8.

[0024] FIG. 11 is a flow diagram of sub-steps of one of the steps in the process of FIG. 8.

[0025] Similar reference numbers and designations in the various figures indicate similar elements. DETAILED DESCRIPTION

[0026] FIG. 1A shows an example of neural network 110. Neural network 110 can be configured through training to perform any type of machine learning task in response to received requests, that is, it can be configured to receive any type of digital data input and to generate any type of score, classification or regression output based on the input.

[0027] To improve the effectiveness, efficiency, or both of the training process, neural network 110 training makes use of at least some of the techniques discussed in this descriptive report, including literal memorization assessment based on canary tokens, neural network 110 training on a multilingual training dataset, and neural network 110 training on a mixture of different pretraining tasks.

[0028] After training, neural network 110 can be made to perform any of a variety of machine learning tasks, some examples of which are described below. For example, neural network 110 can be made to perform a task including a few attempts prompt for the task or a natural language instruction for the task in the input sequence to neural network 110. Petition 870250091754, dated 08 / 10 / 2025, page 31 / 116 5 / 67

[0029] In some cases, the neural network 110 is a neural network that is configured to perform an image or video processing task, that is, to receive an input image or an input video with multiple frames (where each frame is an input image) and process the intensity values ​​of the pixels of the input image to generate a network output for the input image or the input video.

[0030] For example, the task might be image classification, and the output generated by neural network 110 for a given image might be scores for each of a set of object categories, with each score representing an estimated probability that the image contains an image of an object belonging to the category. As another example, the task might be image embedding generation, and the output generated by neural network 110 might be a numerical embedding of the input image. As yet another example, the task might be object detection, and the output generated by neural network 110 might identify locations in the input image where specific types of objects are represented. As yet another example, the task might be semantic image segmentation, and the output generated by neural network 110 might assign each pixel of the input image to a category from a set of categories.As yet another example, the task could be image instance segmentation, and the output generated by neural network 110 could assign each pixel of the input image to a respective object instance from a set of object instances. As yet another example, the task could be image depth prediction, and the output generated by neural network 110 could assign a respective predicted depth value to each pixel of the input image.

[0031] As another example, if the inputs to neural network 110 are Internet resources (e.g., web pages), documents or portions of documents, or elements extracted from Internet resources, documents or portions of documents, the task might be to classify the resource or document, that is, the output generated by neural network 110 for a given resource of Petition 870250091754, dated 08 / 10 / 2025, page 32 / 116 6 / 67 Internet resource, document, or portion of a document can be scored for each of a set of topics, with each score representing an estimated probability that the Internet resource, document, or portion of a document is about the topic.

[0032] As another example, if the inputs to neural network 110 are elements of an impression context for a specific ad, the output generated by neural network 110 could be a score representing an estimated probability that the specific ad will be clicked.

[0033] As another example, if the inputs to neural network 110 are elements of a personalized recommendation for a user, for example, elements that characterize the context for the recommendation, for example, elements that characterize previous actions taken by the user, the output generated by neural network 110 could be a score for each of a set of content items, with each score representing an estimated probability that the user will respond favorably to being recommended the content item.

[0034] As another example, if the input to neural network 110 is a sequence of text in one language, the output generated by neural network 110 might be a piece of text in the other language that is a suitable predicted translation of the input text into the other language.

[0035] Some implementations can be used for automatic code generation. For example, the input can represent words, word fragments, or characters in a first natural language, and the output can represent instructions in a programming or computer markup language, or instructions to control an application program to perform a task, for example, constructing a data item, such as an image or web page.

[0036] As another example, the input to neural network 110 may include computer code or a text description of the computer code function, and the output generated by the neural network is a code sequence of Petition 870250091754, dated 08 / 10 / 2025, page 33 / 116 7 / 67 computer in a programming language that completes the input code in the context input or that performs the function described in the context input.

[0037] As a specific example of automatic code generation, FIG. 1A illustrates that the task could be automatic code bug removal, where the input to the neural network includes computer code with bugs in a programming language that potentially has defects or bugs, and the output generated by the neural network includes fixed computer code in the programming language with the defects or bugs being corrected or removed from the bugged computer code. Optionally, in the automatic code bug removal task, the input to the neural network additionally includes a text description in any natural language, and the output generated by the neural network similarly includes a text description, for example, code comments, in any natural language.

[0038] As another example, the task could be an audio processing task. For example, if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 could be a text transcription of the utterance. As another example, the task could be a keyword detection task where, if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 could indicate whether a specific word or phrase (trigger word) was spoken in the utterance. As another example, if the input to neural network 110 is a sequence representing a spoken utterance, the output generated by neural network 110 could identify the natural language in which the utterance was spoken.

[0039] As another example, the task may be a natural language processing or comprehension task, for example, an implication task, a paraphrasing task, a text similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on, which operates on a sequence of text in Petition 870250091754, dated 08 / 10 / 2025, page 34 / 116 8 / 67 some natural language.

[0040] As another example, the task could be a text-to-speech task, where the input is text in a natural language or elements of text in a natural language and the network output is a spectrogram or other data that define the audio of the text being spoken in the natural language.

[0041] As another example, the task could be a health prediction task, where the input is a sequence derived from electronic health record data for a patient and the output is a prediction that is relevant to the patient's future health, for example, a predicted treatment that should be prescribed for the patient, the probability that an adverse health event will occur for the patient, or a predicted diagnosis for the patient. Such electronic health data might comprise, for example, one or more sequences of physiological data taken from a patient, with the output being a corresponding prediction that refers to those data sequences.Examples of physiological data and a corresponding prediction include: blood glucose measurements, with the prediction being a predicted future blood glucose measurement or the prediction of a hyperglycemic or hypoglycemic event; a heart rate, with the prediction being the presence or absence of a cardiac condition, or a future cardiac event; blood pressure measurements, with the prediction being the risk of a future cardiac condition; or similar.

[0042] As another example, the task could be a text generation task, where the input is a text sequence and the output is another text sequence, for example, a conclusion to the text input sequence, an answer to a question asked in the input sequence, or a text sequence that is about a topic specified by the first text sequence. As another example, the input to the text generation task could be a different text input, for example, an image or a video, and the output could be text describing the input.

[0043] The text input sequence and the text output sequence may be in the same language or may alternatively be in Petition 870250091754, dated 08 / 10 / 2025, p. 35 / 116 9 / 67 different languages. For example, the task is a multilingual text generation task, where the input is a text sequence in a first language, for example, English, and the output generated by the neural network is a text sequence in a second language, for example, German. As another example, the input to the text generation task could be a non-text input, for example, an image, and the output sequence could be text describing the input.

[0044] In some implementations, the input represents data to be compressed, for example, image data, video data, text data, audio data, or any other type of data; and the output is a compressed version of the data. The input and output may each comprise any representation of the compressed / to-be-compressed data, for example, symbols or embeds generated / decoded by a respective neural network.

[0045] As another example, the task could be an agent control task, where the input is a sequence of observations or other data characterization states of an environment and the output defines an action to be performed by the agent in response to the most recent data in the sequence. The agent could be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls a different type of agent. The observations could comprise sensor data captured by sensors associated with (e.g., part of) the agent, for example, visual data, LIDAR data, sonar data, agent configuration data (e.g., joint angles), agent orientation data, or similar.

[0046] In some implementations, the environment is a real-world environment, the agent is a mechanical (or electromechanical) agent interacting with the real-world environment, for example, a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating or navigating the environment, and the actions are actions taken by the mechanical agent in the environment. Petition 870250091754, dated 08 / 10 / 2025, page 36 / 116 10 / 67 real world to perform the task. For example, the agent could be a robot interacting with the environment to perform a specific task, such as locating or manipulating an object of interest in the environment, moving an object of interest to a specified location in the environment, or navigating to a specified destination in the environment.

[0047] In these implementations, observations may include, for example, one or more of the following: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example, sensor data from an image, distance or position sensor, or from an actuator. For example, in the case of a robot, observations may include data that characterize the robot's current state, for example, one or more of the following: joint position, joint velocity, joint force, torque or acceleration, for example, gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent.Observations can be defined in 1, 2, or 3 dimensions and can be absolute and / or relative observations. Observations can also include, for example, detected electronic signals, such as motor current or a temperature signal; and / or image or video data, for example, captured by a camera or a LIDAR sensor, for example, data from the agent's sensors or data from sensors that are located separately from the agent in the environment.

[0048] In these implementations, actions can be control signals to control the robot or other mechanical agent, for example, torques to the robot's joints or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, for example, torques to the control surface or other control elements, for example, vehicle steering control elements or higher-level control commands. Petition 870250091754, dated 08 / 10 / 2025, page 37 / 116 11 / 67 top. Control signals may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. Control signals may also, or instead, include electronic control data, such as motor control data, or more generally data for controlling one or more electronic devices within the environment whose control has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, control signals may define actions to control navigation, e.g., steering, and movement, e.g., braking and / or acceleration of the vehicle.

[0049] In some implementations, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers interacting with the simulated environment. For example, a system implementing neural network 110 can be used to select actions in the simulated environment during system training or evaluation, and after training, or evaluation, or both, are complete, the action selection policy can be implemented to control a real-world agent in the specific real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear and damage to the real-world environment or the real-world agent and can allow the control neural network 110 to be trained and evaluated in situations that rarely occur or are difficult or unsafe to recreate in the real-world environment.For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment and then implemented to control the actual mechanical agent in the specific real-world environment. Thus, in such cases, observations from the simulated environment relate to the real-world environment, and actions selected in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.

[0050] In some implementations, as described above, the agent Petition 870250091754, dated 08 / 10 / 2025, page 38 / 116 12 / 67 may not include a human being (for example, it is a robot). On the other hand, in some implementations, the agent understands a human user of a digital assistant, such as a smart speaker, smart display, or other device. Then, the task-defining information can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task.

[0051] For example, a system implementing neural network 110 can issue instructions to the human user, via the digital assistant, for actions to be performed at each of a plurality of time steps. The instructions can, for example, be generated in the form of natural language (transmitted as sound and / or text on a screen) based on the actions chosen by the system. The system chooses the actions so that they contribute to the completion of a task. A monitoring system (e.g., a video camera system) can be provided to monitor the action (if any) that the user actually performs at each time step, in case (e.g., due to human error) it differs from the action the system instructed the user to perform. Using the monitoring system, the system can determine whether the task has been completed. The system can identify actions that the user performs incorrectly with more than a certain probability.If so, when the system instructs the user to perform such an identified action, the system can warn the user to be careful. Alternatively or additionally, the system can learn not to instruct the user to perform the identified actions, that is, those that the user is likely to perform incorrectly.

[0052] More generally, the digital assistant that instructs the user can understand receiving, within the digital assistant, a request from the user for assistance and determine, in response to the request, a series of tasks for the user to perform, for example, steps or subtasks of a general task. Then, for one or more tasks in the series of tasks, for example, for each task, for example, up to a final task in the series, the digital assistant can be Petition 870250091754, dated 08 / 10 / 2025, page 39 / 116 13 / 67 is used to provide the user with an indication of the task, for example, step or subtask, to be performed. This can be done using natural language, for example, on a screen and / or using a speech synthesis subsystem of the digital assistant. Visual observations, for example, from video, and / or audio of the user performing the task can be captured, for example, using the digital assistant. A system as described above can then be used to determine whether the user has successfully completed the task, for example, step or subtask, i.e., from the response as described above. If there are other tasks to be completed, the digital assistant can, in response, advance to the next task (if any) in the series of tasks, for example, by providing an indication of the next task to be performed. In this way, the user can be guided step-by-step through a series of tasks to accomplish an overall task.During neural network 110 training, training rewards can be generated, for example, from video data representing examples of the overall task (if such data bodies are available) or from a simulation of the overall task.

[0053] In another aspect, a digital assistant device is provided including a system as described above. The digital assistant may also include a user interface to allow a user to request assistance and output information. In implementations, this is a natural language user interface and may comprise a keyboard, voice input / output subsystem, and / or a screen. The digital assistant may further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations, this may comprise a generative (large) language model, particularly for dialogue, for example, a conversational agent such as Sparrow (Glaese et al. arXiv: 2209.14375) or Chinchilla (Hoffmann et al. arXiv: 2203.15556).The digital assistant may have an observation capture subsystem to capture visual and / or audio observations from the user performing a task; and an interface to the neural network model. Petition 870250091754, dated 08 / 10 / 2025, page 40 / 116 14 / 67 language described above (which can be implemented locally or remotely). The digital assistant may also have an assistance control subsystem configured to assist the user. The assistance control subsystem can be configured to perform the steps described above, for one or more tasks, for example, from a series of tasks, up to a final task in the series. More specifically, the assistance control subsystem gives the user an indication of the task to be performed, captures, using the observation capture subsystem, visual or audio observations from the user performing the task, and determines from the response described above whether the user has successfully completed the task. In response, the digital assistant can progress to the next task in the series of tasks and / or control the digital assistant, for example, to stop capturing observations.

[0054] As another example, the task could be a genomics task, where the input is a sequence representing a fragment of a DNA sequence or other molecular sequence, and the output is an incorporation of the fragment for use in a downstream task, for example, making use of an unsupervised learning technique on a dataset of DNA sequence fragments, or an output for the downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, prediction of functional effects of non-coding variants, and so on.

[0055] In some cases, the machine learning task is a combination of multiple individual machine learning tasks, that is, the system is configured to perform multiple different individual machine learning tasks, for example, two or more of the machine learning tasks mentioned above. For example, the system may be configured to perform multiple individual natural language understanding tasks, with the network input including an identifier for the individual natural language understanding task to be performed on the network input. Petition 870250091754, dated 08 / 10 / 2025, page 41 / 116 15 / 67

[0056] In some cases, the machine learning task is a multimodal processing task that requires the processing of multimodal data. In general, multimodal data is a combination of two or more different types of data, for example, two or more audio data, image data, text data, or graph data. As an example, multimodal data might comprise audiovisual data, comprising a combination of pixels from an image or video and audio data representing values ​​of a digitized audio waveform. As another example, multimodal data might comprise a combination of i) text data representing text in a natural language and ii) pixels from an image or video or audio data representing values ​​of an audio waveform.Optionally, but not necessarily, different data types can represent the same objects or overlapping objects using different modalities (types), and when processing multimodal data, the data can be mapped into a common embedding space.

[0057] As a specific example, the task is a multimodal processing task that requires processing text and image inputs, such that neural network 110 includes a computer vision neural network and a text processing neural network. That is, the target output to be generated by the computer vision neural network for a given image depends on one or more outputs generated by the text processing neural network for one or more corresponding text inputs (and vice versa). Examples of such tasks include open vocabulary image classification, open vocabulary object detection, image captioning, text-based image search, image-based retrieval, and so on.

[0058] More generally, the multimodal processing task can correspond to any of the tasks described above for any of the data types that make up the multimodal combination. For example, Petition 870250091754, dated 08 / 10 / 2025, page 42 / 116 16 / 67 The accuracy of the tasks described above can be increased when the task is applied to multimodal data by combining the data for which the task was previously described with another type of data. For example, the detection or classification of an object or event can be improved when data of several different types (modalities) are processed.

[0059] The example machine learning tasks described above are not exhaustive, and the 110 neural network can also be used to perform other tasks.

[0060] FIG. 1B shows an example of training system 100 and an example inference system 170. Training system 100 and inference system 150 are examples of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0061] Training system 100 includes a neural network 110. Neural network 110 is a neural network that can be configured through training to perform any one or more of the tasks mentioned above and possibly other tasks by processing a network input according to a set of neural network 110 parameters to generate a network output for the task. For example, the neural network 110 parameter includes weights and, optionally, biases of the neural network layers.

[0062] The neural network 110 can have any of a variety of transformer-based neural network architectures, for example, encoder-only transformer architectures, encoder-decoder transformer architectures, decoder-only transformer architectures, other attention-based architectures, and so on.

[0063] Exemplos de tais arquiteturas incluem aquelas descritas em Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li e Peter J Liu. Exploring the limits of transfer learning Petição 870250091754, de 08 / 10 / 2025, pág. 43 / 116 17 / 67 with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu e Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020; e Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311.

[0064] In some implementations, the 110 neural network can be configured as, or include, a (large) generative language model or a multimodal model, for example, a visual and language model, to perform these example machine learning tasks. Thus, the 110 neural network can also be referred to as a generative neural network.

[0065] In some implementations, the neural network 110 can be configured to autoregressively generate, as a network output, an output sequence made of tokens selected from a token vocabulary based on a network input that is an input sequence made of tokens selected from the token vocabulary.

[0066] The token vocabulary can include any of a variety of tokens that represent text symbols or other symbols. For example, the token vocabulary can include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. Thus, neural network 110 can still be referred to as an autoregressive generative neural network.

[0067] In addition, or alternatively, the token vocabulary may include tokens that can represent data other than text. For example, the token vocabulary may include image tokens that represent a discrete set of image embeddings of an image that may be Petition 870250091754, dated 08 / 10 / 2025, p. 44 / 116 18 / 67 generated by an image encoding neural network based on image processing. As another example, the token vocabulary might include audio tokens representing code vectors in a quantizer codebook, for example, a residual vector quantizer.

[0068] More specifically, to generate a specific token at a specific position within an output sequence, the neural network 110 can process a current input sequence to generate a scoring distribution, for example, a probability distribution, which assigns a respective score, for example, a respective probability, to each token in the token vocabulary.

[0069] For example, the current input sequence when generating a token at any position in the output sequence may include an input sequence and the tokens at any previous positions preceding the given position in the output sequence. As a specific example, the current input sequence may include the input sequence followed by the tokens at any previous positions preceding the given position in the output sequence.

[0070] Neural network 110 can then select, as the specific token, a token from the vocabulary using the score distribution. For example, neural network 110 can eagerly select the highest-scoring token or it can sample, for example, using core sampling or another sampling technique, a token from the distribution.

[0071] The 110 neural network includes multiple layers, each with its own parameters. Each of the multiple layers is configured to receive a layer input and apply the respective parameters to the layer input to generate the layer output, and then provide the layer output to one or more other layers of the neural network that are configured to receive layer input according to the neural network architecture.

[0072] The multiple layers of the neural network 110 include multiple Petition 870250091754, dated 08 / 10 / 2025, page 45 / 116 19 / 67 layers of self-attention 115, where each layer of self-attention 115 is configured to receive as layer input a respective current representation of each of the text tokens in a current output sequence and process the respective current representations to generate as layer output a respective updated representation of each of the text tokens in the current output sequence.

[0073] A self-attention layer 115 refers to a layer that includes a self-attention mechanism. To generate the layer output, a self-attention layer 115 applies the self-attention mechanism to the layer input, where the self-attention mechanism uses keys, queries, and values ​​that are calculated based on the attention layer parameters of the respective current representations.

[0074] Each self-attention layer 115 can optionally apply other operations to the representations as part of updating the representations, for example, making use of a feedforward neural network in position, applying layer normalization, making use of residual connections, and so on.

[0075] Thus, the neural network parameters 110 may include at least (i) the weights of each self-attention layer in the sequence of self-attention layers 115, for example, the weights of one or more query transformation layers, the weights of one or more key transformation layers and the weights of one or more value transformation layers within an attention layer and (ii) the weights and, optionally, the biases of each feedforward layer included in each feedforward neural network in position.

[0076] The parameters of neural network 110 may also include the weights of a neural network 110 embedding layer that is configured to generate the embeddings of the text tokens in the current output sequence.

[0077] The neural network 110 may have additional layers and components that have no (trainable) parameters, for example, normalization layers, Petition 870250091754, dated 08 / 10 / 2025, page 46 / 116 20 / 67 layers of aggregation, residual connections, softmax layers, logistic layers, and so on.

[0078] Training system 100 trains neural network 110 using a training process that has two stages: a pretraining stage and an adaptation stage. During the pretraining stage, training system 100 trains neural network 110 on unlabeled training data 120 to repeatedly update the values ​​of neural network 110 parameters, that is, to generate pretrained parameter values ​​from the initial values.

[0079] During the adaptation stage, the training system 100 adapts, for example, through fine-tuning adaptation, the pretrained neural network 110 to any one or more of the downstream tasks using labeled training data that are specific to the downstream tasks. The downstream tasks may include any combination of one or more of the machine learning tasks mentioned above and possibly other tasks. Examples of fine-tuning adaptation techniques include supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), instructional fine-tuning, and the like, which use different training objectives, different data, or both.

[0080] The untagged training data 120 includes multiple training input sequences. Each training input sequence has a plurality of positions. Each position has a text token selected from a vocabulary of text tokens. As mentioned above, the vocabulary of text tokens may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. In addition, or alternatively, the token vocabulary may include tokens that may represent data other than text. For example, the token vocabulary may include image tokens that represent a discrete set of image embeddings of an image that may be Petition 870250091754, dated 08 / 10 / 2025, page 47 / 116 21 / 67 generated by an image encoding neural network based on image processing. As another example, the token vocabulary might include audio tokens representing code vectors in a quantizer codebook, for example, a residual vector quantizer.

[0081] In particular, the multiple training input sequences include text tokens in a diverse mix of languages. In some implementations, the multiple training input sequences include text tokens in hundreds of natural languages ​​and programming languages. In some implementations, the multiple training input sequences include no more than a predetermined amount of text tokens in any language, for example, text tokens in any language make up at most a predetermined percentage of all text tokens included in the untagged training data 120.

[0082] Being referred to as unmarked means that, for each training input sequence, a corresponding target output for the training input sequences, i.e., a target output that should be generated by the neural network 110 processing the training input sequence, is unavailable or not used by the training system 100 during pretraining.

[0083] Generally, a training engine 130 of the training system 100 performs pretraining over a plurality of update iterations. In each update iteration, the training engine 130 updates the neural network parameters 110 using a plurality of training input sequences (a batch or a mini-batch of training input sequences) sampled from the unlabeled training data 120.

[0084] Specifically, during the pretraining stage, training engine 130 trains neural network 110 on a mixture of different pretraining tasks on unlabeled training data. Petition 870250091754, dated 08 / 10 / 2025, page 48 / 116 22 / 67 120. Each pretraining task uses a respective pretraining goal function. Each pretraining task focuses on training the neural network 110 to learn a different aspect of language modeling. By training the neural network 110 on a mixture of different pretraining tasks, the training system 100 can train the neural network 110 in a way that improves its generalization capabilities in the adaptation stage, that is, compared to training the neural network 110 on a single pretraining task.

[0085] The mix of different pre-training tasks may include two or more of the following: (i) A causal language modeling pretraining task. The causal language modeling pretraining task is a task that requires predicting, for each text token in a causal language modeling input sequence that is generated from a training input sequence, a text token that must occupy a specific text token position in the causal language modeling input sequence conditioned on text tokens in earlier positions in the causal language modeling input sequence. The causal language modeling pretraining task uses a causal language modeling objective function 131. (ii) A prefix language modeling pretraining task. The prefix language modeling pretraining task is a task that requires predicting, for each text token in a suffix input sequence included in a prefix language modeling input sequence that is generated from a training input sequence, a text token that must occupy a text token position in the suffix conditioned on (a) text tokens in a prefix input sequence preceding the suffix input sequence in the prefix language modeling input sequence and (b) text tokens in any earlier positions in the suffix input sequence. The pretraining task Petition 870250091754, dated 08 / 10 / 2025, page 49 / 116 23 / 67 prefix language modeling training uses a prefix language modeling objective function 132. (iii) A span corruption pretraining task. The span corruption pretraining task is a task that requires predicting, for a span-masked input sequence that is generated from a training input sequence and that replaces one or more contiguous text tokens previously included in the training input sequence with one or more mask tokens, one or more text tokens that should occupy the respective positions of one or more mask tokens in the span-masked input sequence. The span corruption pretraining task uses a span corruption objective function 133.

[0086] In implementations, during the pretraining stage, the training mechanism 130 trains the neural network 110 on at least two of the pretraining tasks mentioned above, for example, both the causal language modeling pretraining task and the prefix language modeling pretraining task; the causal language modeling pretraining task, the prefix language modeling pretraining task, and the span corruption pretraining task.

[0087] In implementations, the different pretraining tasks can be mixed according to a predetermined ratio in the mixing of different pretraining tasks. In one example, the causal language modeling pretraining task, the prefix language modeling pretraining task, and the span corruption pretraining task can be mixed in a 60% / 20% / 20% ratio. That is, 60% of all training input sequences obtained from the unlabeled training data 120 are used by the training engine 130 to generate causal language modeling input sequences and then used to train the neural network 110 to perform the causal language modeling pretraining task; 20% of all sequences Petition 870250091754, dated 08 / 10 / 2025, page 50 / 116 24 / 67 of the training input obtained from the untagged training data 120 are used by the training engine 130 to generate prefix language modeling input sequences and then used to train the neural network 110 to perform the prefix language modeling pretraining task; 20% of all training input sequences obtained from the untagged training data 120 are used by the training engine 130 to generate span-masked input sequences and then used to train the neural network 110 to perform the span corruption pretraining task.In other examples, the causal language modeling pretraining task, the prefix language modeling pretraining task, and the span corruption pretraining task can be mixed in a ratio of 50% / 25% / 25%, a ratio of 70% / 15% / 15%, a ratio of 80% / 10% / 10%, a ratio of 80% / 20% / 0%, and so on.

[0088] In some implementations, the predetermined ratio according to which the different pretraining tasks are mixed in the mix of different pretraining tasks can be defined by the respective weights assigned to the pretraining tasks. For example, a user of training system 100 can provide a user input specifying a respective weight assigned to each of the multiple different pretraining tasks, for example, 0.6 for the causal language modeling pretraining task, 0.2 for the prefix language modeling pretraining task, and 0.2 for the span corruption pretraining task—and then, during the pretraining stage, the training engine 130 repeatedly selects, based on the specified weights, a pretraining task from the multiple different pretraining tasks to train the neural network 110.

[0089] A common problem faced by generative neural networks, such as neural network 110 in FIG. 1, is data leakage in their outputs. Data leakage occurs when a trained neural network reveals Petition 870250091754, dated 08 / 10 / 2025, page 51 / 116 25 / 67 information that is private to an individual or organization and that was included in the training data 120 for the neural network 110 as part of its outputs generated after implementation. Depending on the downstream machine learning tasks, data leakage can lead to a number of negative consequences, especially when the information revealed is sensitive, confidential, or otherwise susceptible to protection.

[0090] To minimize the negative consequences of data leakage, optionally, in some implementations, at each of several points during the pre-training stage, the training system 100 can estimate the probability that the neural network 110 has memorized the content verbatim from the unlabeled training data 120 and then, depending on the probability, determine whether to modify the training of the neural network 110 to reduce the probability of the neural network 110 memorizing the content of the unlabeled training data 120. The memorization assessment is broadly applicable to any of a variety of pre-training tasks on which the neural network 110 is being trained, i.e., it can be performed independently of which objective functions the training mechanism 130 uses to train the neural network 110.

[0091] Literal memorization refers to the situation in which neural networks reproduce specific portions of text that they have processed during training, for example, when neural networks reproduce sequences of multiple consecutive text tokens, such as text tokens that make up sentences or even passages of text, from their training corpus, verbatim. Literal memorization increases the risk of neural networks exposing private data from their training corpus.

[0092] To this end, the training system 100 includes a special token insertion mechanism 140 that can be used to modify at least some of the training input sequences included in the untagged training data 120 before training occurs, that is, before those training input sequences are used to train the Petition 870250091754, dated 08 / 10 / 2025, page 52 / 116 26 / 67 neural network 110. The special token insertion mechanism 140 is an optional component of the training system 100.

[0093] In particular, the special token insertion mechanism 140 can modify a given training input sequence by inserting canary tokens as a type of special tokens into the given training input sequence. For any training input sequence, canary tokens are tokens that were not originally included in the untagged training data 120; instead, they are inserted into the training input sequence before the training input sequence is processed by the neural network 110 during training.

[0094] In order to strike a balance between making canary tokens appear as outliers while also retaining specific characteristics of the training data, canary tokens can take on an appropriate shape and can be inserted into appropriate positions within the given training input sequence.

[0095] For example, canary tokens can take the form of interleaved canary tokens. That is, the special token insertion mechanism 140 obtains two training input sequences that include a first training input sequence and a second training input sequence from the untagged training data 120, generates a modified first training input sequence by inserting text tokens taken from the second training input sequence into the first training input sequence, and generates a modified second training input sequence by inserting text tokens taken from the first training input sequence into the second training input sequence.

[0096] In this example, the first modified training input sequence and the second modified training input sequence will then be used to train neural network 110. As the canary tokens were removed from the training input sequences from the training data Petition 870250091754, dated 08 / 10 / 2025, page 53 / 116 27 / 67 unmarked 120, they preserve some linguistic properties of the unmarked 120 training data.

[0097] After training the neural network 110 on training input sequences that include the first modified training input sequence and the second modified training input sequence, the training engine 130 estimates the degree to which the trained neural network memorizes data in the unlabeled training data 120.

[0098] Specifically, training mechanism 130 provides a test input sequence that includes a subset of the tokens previously included in the first training input sequence as input to neural network 110 and uses neural network 110 to process the test input sequence to generate one or more predicted continuations of the test input sequence. Each predicted continuation specifies a plurality of output tokens. That is, training mechanism 130 uses neural network 110 to predict, given the subset of tokens previously included in the first training input sequence, the next tokens that follow the subset of tokens in the first training input sequence.

[0099] To estimate the degree to which the trained neural network memorizes data in the untagged training data, the 130 training engine evaluates the plurality of output tokens specified by each predicted continuation relative to the remaining subset of text tokens included in the first modified training input sequence (which includes text tokens taken from the second training input sequence). For example, the evaluation might involve counting the total number of output tokens in each predicted continuation that also appear in the text tokens taken from the second training input sequence, and the estimated degree might be proportional to the total number counted.

[0100] Other forms of canary tokens may be used additionally or alternatively. For example, canary tokens may take the form of scrambled canary tokens. That is, the mechanism of Petition 870250091754, dated 08 / 10 / 2025, page 54 / 116 28 / 67 Special token insertion 140 obtains a training input sequence and then generates a modified training input sequence by shuffling all the text tokens included in the training input sequence, that is, altering the positions of the existing text tokens so as to remove information associated with their sequence-level ordering. In this example, the modified training input sequence, which includes shuffled text tokens, will then be used to train the neural network 110.

[0101] Another common problem faced by generative neural networks, such as neural network 110 in FIG. 1, is toxicity in their outputs. Toxicity refers to the inclusion of harmful, offensive, or otherwise inappropriate content in the outputs generated by a neural network. Toxicity in the outputs generated by a trained neural network can lead to a number of negative consequences after it has been implemented for downstream machine learning tasks, for example, dialogue tasks, generative question answering tasks, and other open language modeling tasks.

[0102] To this end, the special token insertion mechanism 140 can modify a given training input sequence by inserting a toxicity token as another type of special token into the given training input sequence. The toxicity token identifies a toxicity level of the content represented by the given training input sequence.

[0103] The special token insertion mechanism 140 can determine, from a plurality of toxicity levels, a specific toxicity level based on the content represented by the given training input sequence and then prefix or append a specific toxicity token corresponding to the given toxicity level to the given training input sequence.

[0104] The inclusion of toxicity tokens can alleviate the toxicity problem because, after training, the neural network 110 can be conditioned Petition 870250091754, dated 08 / 10 / 2025, page 55 / 116 29 / 67 is assigned a toxicity token representing a low level of toxicity, thus causing the 110 neural network to generate outputs with low levels of toxicity. In other words, by training the 110 neural network to accurately estimate the toxicity of its outputs, it becomes possible to control the post-training generation to produce outputs with specified levels of toxicity.

[0105] In some implementations, the plurality of toxicity levels may include two or more toxicity levels, for example, level 0 and level 1. In some implementations, the plurality of toxicity levels may include three or more toxicity levels, for example, level 0, level 1, and level 2, where larger numbers indicate higher toxicity levels. For example, a level 2 toxicity token will be prefixed or appended to a training input sequence that includes a larger amount of harmful, offensive, or otherwise inappropriate content, while a level 1 toxicity token will be prefixed or appended to a training input sequence that includes a smaller amount of such content.

[0106] In some implementations, the special token insertion mechanism 140 only needs to do this for a relatively small number of training input sequences included in the untagged training data 120. That is, for a relatively small portion of, for example, less than 0.5%, 1%, or 5% of all training input sequences included in the untagged training data 120, the training system 100 uses the special token insertion mechanism 140 to add toxicity tokens that identify the toxicity levels in the content represented by those training input sequences.

[0107] To automatically determine the toxicity level for the given training input sequence, the special token insertion mechanism 140 includes or accesses a text-based classifier mechanism. The text-based classifier mechanism can be implemented in any appropriate manner to allow it to process the given training input sequence to classify the given sequence. Petition 870250091754, dated 08 / 10 / 2025, page 56 / 116 30 / 67 training input in one of a plurality of toxicity levels.

[0108] For example, the text-based classifier mechanism can implement a text-classifier machine learning model, for example, a neural network (e.g., a language model neural network), a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model, which generates toxicity classification outputs. As another example, the text-based classifier mechanism can implement a deterministic text-based classification algorithm, for example, a human-programmed algorithm, which generates toxicity classification outputs.

[0109] Another common problem encountered in training generative neural networks, such as neural network 110 in FIG. 1, is the lack of multilingual training data in the untagged training data 120. For example, the untagged training data 120 may predominantly contain training input sequences that are each composed of text tokens in the same specific language, but training input sequences that are each composed of text tokens in two or more different languages ​​may, however, be scarce.

[0110] The lack of multilingual training data in the untagged training data 120 may negatively affect the performance of the trained neural network 110, especially in multilingual downstream tasks involving the processing of text tokens in two or more languages, for example, translation tasks, multilingual text generation tasks, including multilingual question answering tasks.

[0111] To address the lack of multilingual training data and the challenges this presents in neural network training 100, the training system 100 includes a multilingual training data generation mechanism 145 that can be used to generate additional multilingual training data based on what already exists in the unlabeled training data 120. The multilingual training data generation mechanism Petition 870250091754, dated 08 / 10 / 2025, page 57 / 116 31 / 67 145 is an optional component of the 100 training system.

[0112] When included, the multilingual training data generation mechanism 145 can be used by the training system 100 to generate multilingual training data from the training input sequences included in the untagged training data 120 — and the training mechanism 130 trains the neural network 100 on the multilingual training data.

[0113] The multilingual training data includes a plurality of first multilingual training pairs and a plurality of second multilingual training pairs. Each first multilingual training pair, in turn, includes a first source training input sequence and a first target training input sequence. Each second multilingual training pair, in turn, includes a second source training input sequence and a second target training input sequence. In particular, the first source training input sequences and the second source training input sequences are both in the same language, while the first target training input sequences and the second target training input sequences are in different languages. The generation of multilingual training data will be discussed further below with reference to FIG. 7.

[0114] The inclusion of multilingual training data further improves the ability of neural network 110 to understand and generate multilingual outputs; it also enhances the performance of trained neural network 110 in downstream multilingual tasks involving the processing of text tokens in two or more languages, for example, translation tasks, multilingual text generation tasks, including multilingual question answering tasks.

[0115] After the pre-training stage, in the adaptation stage, the training system 100 or a different training system can adjust some or all of the parameters of the neural network 110 in order to adapt the neural network 110 to any of a variety of downstream tasks. For example, the Petition 870250091754, dated 08 / 10 / 2025, page 58 / 116 Training system 100 can adjust one or more of the neural network 110 components on training data labeled for a downstream task that is specific to the downstream task, for example, through supervised learning, reinforcement learning from human feedback, or instructional adjustment. As another example, training system 100 can hold any parts of neural network 110 that were trained during the pretraining stage fixed while learning one or more additional customized components that are specific to the downstream task.

[0116] After the adaptation stage, the training system 100 or a different inference system 170 implements the trained neural network 110 on one or more computing devices to perform inference, that is, to generate new network outputs 112 for downstream tasks for new network inputs 102.

[0117] FIG. 2 is a flow diagram of an example of process 200 for training a neural network that has parameters on a training dataset to repeatedly update the values ​​of the neural network parameters. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., training system 100 of FIG. 1B, appropriately programmed, can perform process 200.

[0118] The training dataset includes multiple training input sequences. The training dataset may include untagged training input sequences. Each training input sequence has a plurality of positions. Each position has a token selected from a token vocabulary. As mentioned above, the token vocabulary may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. In addition, or alternatively, the token vocabulary may include tokens that may represent data other than text. For example, the token vocabulary Petition 870250091754, dated 08 / 10 / 2025, page 59 / 116 33 / 67 can include image tokens that represent a discrete set of image embeddings of an image that can be generated by an image-encoding neural network based on image processing. As another example, the token vocabulary can include audio tokens that represent code vectors in a quantizer codebook, for example, a residual vector quantizer.

[0119] For example, the training input sequences included in the training data may be generated from a large dataset of text in one or more natural languages, for example, text that is publicly available on the Internet or in another text corpus, a large dataset of computer code in one or more programming languages, for example, Python, C++, C#, Java, Ruby, PHP and so on, for example, computer code that is publicly available on the Internet or in another code repository, a large dataset of audio samples, for example, audio recordings or waveforms representing the audio recordings, a large dataset of images in which each image includes a pixel array, a large dataset of videos in which each video includes a temporal sequence of frames, or a large multimodal dataset that includes a combination of two or more of these datasets.

[0120] Process 200 can be repeated as many times as necessary throughout the neural network training to obtain the most recent estimate of the degree to which the neural network memorizes data in the training data on which it is being trained. For example, process 200 can be repeated every predetermined number of times the neural network parameters have been updated, every predetermined amount of time that has passed in reality since the start of training, and so on. As another example, process 200 can be triggered upon receiving an instruction from a system user.

[0121] The system obtains, for example, through sampling, a Petition 870250091754, dated 08 / 10 / 2025, pp. 60 / 116 34 / 67 first training input sequence and a second training input sequence from the training data (step 202). The first training input sequence includes the first tokens selected from the token vocabulary. The second training input sequence includes second tokens selected from the token vocabulary.

[0122] In some implementations, the training data may be monolingual, and the first training input sequence and the second training input sequence are both in the same language, for example, the same natural language or the same programming language. In other implementations, the training data may be multilingual, and the first training input sequence and the second training input sequence are in different languages. That is, the first training input sequence may be in one language, and the second training input sequence may be in a second language that is different from the first language.

[0123] The system generates a first modified training input sequence based on the first training input sequence and the second training input sequence (step 204). The generation of the first modified training input sequence is described below with reference to FIG. 3, which is a flow diagram of substeps 302-306 of step 204 of process 200 in FIG. 2.

[0124] The system selects a plurality of second tokens from among the second tokens included in the second training input sequence (step 302). The number of second tokens to be selected can be predetermined. The plurality of second tokens can be a sequence of text tokens that are adjacent to each other within the second training input sequence, that is, that occupy multiple contiguous positions in the second training input sequence.

[0125] The system determines a first canary position within the first training input sequence (step 304). In some Petition 870250091754, dated 08 / 10 / 2025, pp. 61 / 116 In 35 / 67 implementations, the first canary position can be determined independently of the positions of the selected plurality of second tokens within the second training input sequence. For example, the first canary position can be determined with uniform randomness, that is, it can be any position within the first training input sequence.

[0126] In some other implementations, the first canary position may be determined based on the positions of the selected plurality of second tokens within the second training input sequence. For example, the first canary position within the first training input sequence may generally correspond to the positions of the selected plurality of second tokens within the second training input sequence.

[0127] That is, if the plurality of second tokens was taken from the first half of the second training input sequence, for example, taken from within a threshold number of token proportions at the beginning of the second training input sequence, then the first canary position may similarly be in the first half of the first training input sequence, for example, within the threshold number of token proportions at the beginning of the first training input sequence.

[0128] Similarly, if the plurality of second tokens was taken from the second half of the second training input sequence, for example, taken from within a threshold number of token proportions at the end of the second training input sequence, then the first canary position may similarly be in the second half of the first training input sequence, for example, within the threshold number of token proportions at the end of the first training input sequence.

[0129] The system generates the first modified training input sequence by inserting the selected plurality of second tokens into the first Petition 870250091754, dated 08 / 10 / 2025, page 62 / 116 36 / 67 training input sequence after the first canary position, that is, placing the selected plurality of second tokens in multiple positions (e.g., multiple contiguous positions) starting from the first canary position in the first training input sequence (step 306). Thus, the modified first training input sequence includes the first tokens that were initially included in the first training input sequence, as well as a plurality of second tokens that were removed from the second training input sequence. The plurality of second tokens that were removed from the second training input sequence thus acts as canary tokens, i.e., they are additional tokens that were not originally included in the first training input sequence.The first canary position can be the first position where one of these canary tokens is inserted into the first training input sequence. Once generated, the system can include the first modified training input sequence in the training dataset.

[0130] Optionally, in some implementations, the system also generates a second modified training input sequence based on the first training input sequence and the second training input sequence (step 206). The generation of the second modified training input sequence is described below with reference to FIG. 4, which is a flow diagram of substeps 402-406 of step 206 of process 200 in FIG. 2.

[0131] The system selects a plurality of first tokens from among the first tokens included in the first training input sequence (step 402). The number of first tokens to be selected can be predetermined, for example, it can be equal to the number of second tokens in step 304. The plurality of first tokens can be a sequence of tokens that are adjacent to each other within the first training input sequence, that is, that occupy multiple contiguous positions in the first training input sequence. Petition 870250091754, dated 08 / 10 / 2025, pp. 63 / 116 37 / 67

[0132] The system determines a second canary position within the second training input sequence (step 404). In some implementations, the second canary position may be determined independently of the positions of the selected plurality of first tokens within the first training input sequence. For example, the second canary position may be determined with uniform randomness, i.e., it may be any position within the second training input sequence.

[0133] In some other implementations, the second canary position may be determined based on the positions of the selected plurality of first tokens within the first training input sequence. For example, the second canary position within the second training input sequence may generally correspond to the positions of the selected plurality of first tokens within the first training input sequence.

[0134] That is, if the plurality of first tokens was taken from the first half of the first training input sequence, for example, taken from within a threshold number of token proportions from the beginning of the first training input sequence, then the second canary position may similarly be in the first half of the second training input sequence, for example, within the threshold number of token proportions from the beginning of the second training input sequence.

[0135] Similarly, if the plurality of first tokens was taken from a second half of the first training input sequence, for example, taken from within a threshold number of token proportions at the end of the first training input sequence, then the second canary position may similarly be in the second half of the second training input sequence, for example, within the threshold number of token proportions at the end of the second training input sequence. Petition 870250091754, dated 08 / 10 / 2025, pp. 64 / 116 38 / 67

[0136] The system generates the second modified training input sequence by inserting the selected plurality of first tokens into the second training input sequence after the second canary position, that is, placing the selected plurality of first tokens in multiple positions (e.g., multiple contiguous positions) starting from the second canary position in the second training input sequence (step 406). Thus, the second modified training input sequence includes the second tokens that were initially included in the second training input sequence, as well as a plurality of first tokens that were removed from the first training input sequence. The plurality of first tokens that were removed from the first training input sequence thus acts as canary tokens, that is, they are additional tokens that were not originally included in the second training input sequence.The second canary position can be the first position where one of these canary tokens is inserted into the second training input sequence. Once generated, the system can include the modified second training input sequence in the training dataset.

[0137] The system trains the neural network on training data that includes the first modified training input sequence and, optionally, the second training input sequence (step 208). The system performs training over a plurality of update iterations. In each update iteration, the system updates the neural network parameters based on the respective gradients of an objective function, for example, a pretraining objective function, calculated with respect to the neural network parameters using a plurality of training input sequences sampled from the training data. The first modified training input sequence (or the second training input sequence) will be included in the plurality of training input sequences that are sampled in at least one of the update iterations. Petition 870250091754, dated 08 / 10 / 2025, pp. 65 / 116 39 / 67

[0138] After training, the system uses the neural network, which has now been trained on training data that includes the first modified training input sequence and, optionally, the second training input sequence, to generate one or more predicted continuations of a test input sequence by processing the test input sequence according to the updated values ​​of the neural network parameters (step 210). The test input sequence includes a subset of the first tokens that were initially included in the first training input sequence. For example, the subset of first tokens might include some of the tokens that occupy positions in the first modified training input sequence that precede the first canary position.

[0139] In other words, the neural network is used to predict, given the subset of the first tokens, the next tokens that follow the subset of the first tokens in the training data on which they were trained. To this end, each predicted continuation specifies a plurality of output tokens. Each output token specified by a given predicted continuation is a next predicted token as calculated by the neural network that may follow the subset of the first tokens in the training data.

[0140] Each predicted continuation generated by the neural network can specify the output tokens directly, for example, where the predicted continuation includes the plurality of output tokens, or indirectly, for example, where the predicted continuation defines, for each of the plurality of output tokens, a score distribution over the token vocabulary based on which the output token can be determined, i.e., selected from the vocabulary.

[0141] The system determines an estimate of the degree to which the trained neural network memorizes data in the training dataset on which it was trained by evaluating the plurality of output tokens specified by each predicted continuation in relation to the selected plurality of second tokens included in the first modified training input sequence (step Petition 870250091754, dated 08 / 10 / 2025, pp. 66 / 116 40 / 67 212).

[0142] For example, the system can determine an estimate of the degree to which the trained neural network memorizes data literally. Literal memorization refers to the situation where neural networks reproduce specific portions of text that they processed during training, for example, when neural networks reproduce sequences of multiple consecutive tokens, such as text tokens that make up sentences or even passages of text, from their training corpus.

[0143] As another example, the system can determine an estimate of the degree to which the trained neural network memorizes essential data. Essential memorization refers to the situation where neural networks reproduce a summary of the text they processed during training. For example, the summary can be an extractive summary, where the output sequence generated by a neural network is a suitable subset of one of the training input sequences, that is, it is composed of tokens from the training input sequence, processed by the neural network during training.

[0144] In implementations, each predicted continuation includes a plurality of output tokens; the system can compare the plurality of output tokens with the selected plurality of second tokens included in the first modified training input sequence in order to determine if there is a match between any of the output tokens and any of the selected plurality of second tokens. In this way, the system can determine how many of the selected plurality of second tokens included in the first modified training input sequence are also included in the plurality of output tokens included in each predicted continuation.

[0145] In these implementations, the estimation of the degree of memorization of training data can be determined based, for example, on a (reference) proportion or otherwise dependent on the total count of Petition 870250091754, dated 08 / 10 / 2025, page 67 / 116 41 / 67 matches between the output tokens and the selected plurality of second tokens.

[0146] In implementations, each predicted continuation defines, for each of the plurality of output tokens, a scoring distribution over the token vocabulary. The system can calculate a probability assigned to each token in at least one subset of the selected plurality of second tokens by the scoring distributions for the positions included in each predicted continuation. For example, for a given second token in the selected plurality of second tokens, its assigned probability might be a combination, for example, a product, of the scores included in the respective scoring distributions that were generated for the given second token. As a specific example of this, suppose one of the selected plurality of second tokens is the token cat and the predicted continuation has three positions.For each position, the scoring distribution generated by the neural network (which can be used to select the predicted token for the position) includes a score for the cat token, among the scores for other tokens in the vocabulary. For example, when the scores are P(cat) = 0.05 for the first position in the predicted continuation, P(cat) = 0.1 for the second position in the predicted continuation, and P(cat) = 0.04 for the third position in the predicted continuation, then the probability for the cat token could be calculated as a product of the three scores: 0.05 x 0.1 x 0.04 = 0.0002. In this way, the system can determine the probability that any one of the selected plurality of second tokens included in the first modified training input sequence can be selected for inclusion in the plurality of output tokens included in each predicted continuation.

[0147] In these implementations, the estimation of the degree of memorization of training data can be determined based, for example, on a (reference) proportion or otherwise dependent on the probability calculated for each of at least the subset of the selected plurality of second tokens. Petition 870250091754, dated 08 / 10 / 2025, pp. 68 / 116 42 / 67

[0148] The system determines whether the estimate of the degree to which the trained neural network memorizes data in the training dataset satisfies a predetermined threshold by comparing the estimate with the predetermined threshold (step 214). For example, step 214 might involve determining whether the plurality of output tokens includes more than one threshold number of the selected plurality of second tokens. As another example, step 214 might involve determining whether the probability calculated for each of at least the subset of the selected plurality of second tokens satisfies a probability threshold.

[0149] In response to the determination that the estimate satisfies the predetermined threshold, the system applies one or more adjustments to the neural network training to adjust the estimate (step 216). For example, when lower estimates indicate a lower degree of memorization of training data, then the estimate may satisfy the predetermined threshold if the estimate is greater than the predetermined threshold.

[0150] In general, adjustments can modify any appropriate aspect of neural network training that would potentially result in changes in the estimate, for example, to reduce the estimate when lower estimates indicate a lower degree of memorization of training data. Just as a few examples for illustration, the system can adjust the pretraining objective function, for example, using alternative or new loss terms that penalize the neural network for literal memorization, assigning different weights to existing loss terms, adjusting hyperparameters, for example, adjusting learning rates or the choice of optimizer, and so on.

[0151] This memorization assessment technique allows for a variety of technological improvements in neural networks that were not previously possible. It is important to emphasize that incorporating the memorization assessment technique into the training of a neural network can lead to significant reductions in literal memorization by the trained neural network and, in particular, Petition 870250091754, dated 08 / 10 / 2025, pp. 69 / 116 43 / 67 for data that are not strongly repeated in the training data. Furthermore, the memorization assessment technique makes it more practical to conduct systematic studies on potential privacy risks in downstream use cases, including dialogue or summary tasks, of the trained neural network and further facilitates the adoption of procedural and / or additional technical safeguards that may be necessary against extraction attacks.

[0152] FIG. 5 is a flow diagram of another example of a 500 process for training a neural network that has parameters on a training dataset to repeatedly update the values ​​of the neural network parameters. For convenience, the 500 process will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, for example, the 100 training system of FIG. 1, appropriately programmed, can perform the 500 process.

[0153] The training dataset includes multiple training input sequences. The training dataset may include untagged training input sequences. Each training input sequence has a plurality of positions. Each position has a token selected from a token vocabulary. As mentioned above, the token vocabulary may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. In addition, or alternatively, the token vocabulary may include tokens that may represent data other than text. For example, the token vocabulary may include image tokens that represent a discrete set of image embeddings of an image that can be generated by an image-encoding neural network based on image processing.As another example, the token vocabulary might include audio tokens that represent code vectors in a quantizer's codebook, for example, a residual vector quantizer.

[0154] For example, the included training input sequences Petition 870250091754, dated 08 / 10 / 2025, pp. 70 / 116 44 / 67 of the training data can be generated from a large dataset of text in one or more natural languages, for example, text that is publicly available on the Internet or in another text corpus; a large dataset of computer code in one or more programming languages, for example, Python, C++, C#, Java, Ruby, PHP, and so on, for example, computer code that is publicly available on the Internet or in another code repository; a large dataset of audio samples, for example, audio recordings or waveforms representing the audio recordings; a large dataset of images in which each image includes a pixel matrix; a large dataset of videos in which each video includes a temporal sequence of frames; or a large multimodal dataset that includes a combination of two or more of these datasets.

[0155] Process 500 can be repeatedly performed for a relatively small portion of the training input sequences included in the training data. For example, iterations of process 500 can be performed for 0.5%, 1%, or 5% of all training input sequences included in the training data.

[0156] The system obtains, for example, through sampling, a third training input sequence from the training data (step 502).

[0157] The system determines, from a plurality of toxicity levels, a specific toxicity level based on the content represented by the third training input sequence (step 504). To automatically determine the toxicity level for the third training input sequence, the system can use a text classifier machine learning model or a deterministic text-based classification algorithm, for example.

[0158] In some implementations, the plurality of toxicity levels may include two or more toxicity levels, for example, level 0 and level 1. In Petition 870250091754, dated 08 / 10 / 2025, pp. 71 / 116 45 / 67 In some implementations, the plurality of toxicity levels may include three or more toxicity levels, for example, level 0, level 1, and level 2, where larger numbers indicate higher toxicity levels.

[0159] The system adds a specific toxicity token corresponding to the given toxicity level to the third training input sequence (step 506). For example, the specific toxicity token can be prefixed to the beginning of the third training input sequence, appended to the end of the third training input sequence, or inserted at any appropriate position within the third training input sequence.

[0160] In some implementations, the system determines whether to add the specific toxicity token to the third training input sequence based on a total number of training input sequences to which toxicity tokens have been added. For example, the system will determine whether to add the specific toxicity token if toxicity tokens (corresponding to the same or different toxicity levels) have been added to no more than a predetermined number of training input sequences included in the training data.

[0161] The system trains the neural network on training data that includes the third training input sequence, which now includes the specific toxicity token (step 508). The system performs training over a plurality of update iterations. In each update iteration, the system updates the neural network parameters based on the respective gradients of an objective function, for example, a pretraining objective function, calculated with respect to the neural network parameters using a plurality of training input sequences sampled from the training data. The third training input sequence will be included in the plurality of training input sequences that are sampled in at least one of the iterations of Petition 870250091754, dated 08 / 10 / 2025, pp. 72 / 116 46 / 67 update.

[0162] Advantageously, incorporating such toxicity tokens into the training data improves the performance of the trained neural network. For example, an output sequence generated by the trained neural network from processing a context sequence that includes, for example, is prefixed or appended with a toxicity token that corresponds to a low level of toxicity will have a lower, sometimes significantly lower, probability of containing toxic content, and the trained neural network is therefore more suitable for deployment in a variety of environments, such as within an educational or medical organization, where any harmful behavior of the deployed neural network could result in serious consequences.

[0163] FIG. 6 is a flow diagram of an example of process 600 for using a neural network to generate an output sequence. The neural network can be a neural network that has been trained using at least the technique discussed above with reference to FIG. 5. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, for example, training system 100 of FIG. 1, appropriately programmed, can perform process 600.

[0164] The system obtains a context string that includes a plurality of input tokens (step 602). In some cases, the context string includes user-specified prompt text, that is, it includes input tokens sent by a system user. In some cases, the context string includes predefined prompt text stored in the system. In some cases, the context string includes both user-specified prompt text and predefined prompt text; for example, it might be a concatenation of user-specified prompt text and predefined prompt text.

[0165] The system generates a quality-controlled context sequence from the context sequence (step 604). The quality-controlled context sequence can be generated by adding to the context sequence of Petition 870250091754, dated 08 / 10 / 2025, pp. 73 / 116 47 / 67 a specific toxicity token. The specific toxicity token can be any of the toxicity tokens selected from the plurality of toxicity tokens that represent different levels of toxicity of the content to be represented by an output sequence. For example, the quality-controlled context sequence includes the specific quality token followed by the plurality of input tokens, or vice versa.

[0166] The system generates, using the neural network, an output sequence based on the processing of the quality-controlled context sequence (step 606). The output sequence includes a plurality of output tokens. For example, if the specific toxicity token that is included in the quality-controlled context sequence corresponds to a low level of toxicity, then, due to the way the neural network was trained as described above with reference to FIG. 5, the specific toxicity will cause the neural network to generate an output sequence that has a lower probability, sometimes significantly lower, of containing toxic content, for example, compared to an output sequence generated by the same neural network (or another conventionally trained neural network) from a context sequence that does not include such a toxicity token.

[0167] FIG. 7 is a flow diagram of another example of process 700 for training a neural network that has parameters on a training dataset to repeatedly update the values ​​of the neural network parameters. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., training system 100 of FIG. 1, appropriately programmed, can perform process 700.

[0168] The training dataset includes multiple training input sequences. The training dataset may include untagged training input sequences. Each training input sequence has a plurality of positions. Each position has a token selected from a vocabulary of text tokens. As Petition 870250091754, dated 08 / 10 / 2025, pp. 74 / 116 48 / 67 mentioned above, the vocabulary of text tokens may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code.

[0169] For example, the training input sequences included in the training data can be generated from electronic documents (e.g., books, web pages (e.g., HTML pages), news articles, or other documents) that can be found in a corpus (e.g., a collection or repository of content) that is available on the Internet or in another text corpus.

[0170] The system obtains unmarked text in multiple languages ​​(step 702). That is, the system obtains one or more training input sequences that each include text tokens in a first language, obtains one or more other training input sequences that each include text tokens in a second language, and obtains one or more other training input sequences that each include text tokens in a third language.

[0171] The system generates, from the unmarked text, a first set of multilingual training data that includes a plurality of first multilingual training pairs (step 704). Each first multilingual training pair includes a first source text sequence and a first target text sequence. In some implementations, the first source text sequence and the first target text sequence have similar lengths, that is, they include approximately the same number of text tokens, while in other implementations, they may have different lengths.

[0172] The system generates, from the unmarked text, a second multilingual training dataset that includes a plurality of second multilingual training pairs (step 706). Each second multilingual training pair includes a second source text sequence and a Petition 870250091754, dated 08 / 10 / 2025, pp. 75 / 116 49 / 67 second target text sequence.

[0173] In some implementations, the first multilingual training dataset and the second multilingual training dataset include an overlapping set of text sequences. For example, the first source text sequences included in the first multilingual training dataset are the same as the second source text sequences included in the second multilingual training dataset. In other implementations, the first multilingual training dataset and the second multilingual training dataset include distinct sets of text sequences. For example, the first source text sequences included in the first multilingual training dataset are different from the second source text sequences included in the second multilingual training dataset.

[0174] Specifically, however, the first source text string and the second source text string are both in the same language, while the first target text string and the second target text string are in different languages. For example, the first source text string and the second source text string include text tokens in the first language, the first target text string includes text tokens in the second language, and the second target text string includes text tokens in the third language.

[0175] The system trains a neural network on the first and second multilingual training datasets (step 708). The system performs training over a plurality of update iterations. In each update iteration, the system updates the neural network parameters based on the respective gradients of an objective function, for example, a pretraining objective function, calculated with respect to the neural network parameters using a plurality of first multilingual training pairs sampled from the first multilingual training dataset, or using a plurality of second training pairs. Petition 870250091754, dated 08 / 10 / 2025, pp. 76 / 116 50 / 67 multilingual samples from the second multilingual training dataset.

[0176] Merely as an example for illustration, for a given first multilingual training pair sampled from the first multilingual training dataset, the system can train the neural network on a pretraining language modeling task that requires predicting, given (a portion of) a first source text sequence that includes text tokens in the first language, predicted text tokens in the second language that make up the first target text sequence that follows the first source text sequence in the given first multilingual training pair.

[0177] After training, the system adapts, for example, through fine-tuning, zero-shot or few-shot adaptation, the trained neural network to perform a multilingual downstream task (step 710). The multilingual downstream task can be any task that involves processing text tokens in two or more languages.

[0178] For example, the trained neural network can be adapted to perform a translation task by processing an input text sequence in a source language to generate an output text sequence in a target language that is a translation of the input text sequence into the target language.

[0179] As another example, the trained neural network can be adapted to perform a multilingual question answering task by processing an input text sequence in the source and target languages ​​to generate an output text sequence in the target language that is a response to the input text sequence.

[0180] Training the neural network on multilingual training data improves its ability to understand and generate multilingual outputs, therefore it can be more easily adapted to any of a variety of multilingual downstream tasks. Once adapted, the neural network can Petition 870250091754, dated 08 / 10 / 2025, pp. 77 / 116 51 / 67 to achieve or even exceed the performance of a conventionally pre-trained neural network on any of the downstream multilingual tasks, despite an adaptation process that consumes fewer computing resources, is faster in terms of time spent in reality, or both.

[0181] FIG. 8 is a flow diagram of another example of an 800 process for training a neural network that has parameters on a mixture of different pretraining tasks on a training dataset that includes multiple training input sequences. The multiple training input sequences may include untagged training input sequences. The neural network may be a generative autoregressive neural network that includes one or more layers of self-attention. For convenience, the 800 process will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the 100 training system of FIG. 1, appropriately programmed, may perform the 800 process.

[0182] For example, the training input sequences included in the training data may be generated from a large dataset of text in one or more natural languages, for example, text that is publicly available on the Internet or in another text corpus, a large dataset of computer code in one or more programming languages, for example, Python, C++, C#, Java, Ruby, PHP and so on, for example, computer code that is publicly available on the Internet or in another code repository, a large dataset of audio samples, for example, audio recordings or waveforms representing the audio recordings, a large dataset of images in which each image includes a pixel array, a large dataset of videos in which each video includes a temporal sequence of frames, or a large multimodal dataset that includes a combination of two or more of these datasets.

[0183] The system can repeatedly execute iterations of process 800 Petition 870250091754, dated 08 / 10 / 2025, pp. 78 / 116 52 / 67 to repeatedly update the neural network parameters until a termination criterion has been met, for example, until a threshold number of process iterations of 800 has been performed, until a threshold amount of time actually spent has elapsed, or until the parameter values ​​have converged.

[0184] The system obtains, for example, through sampling, a plurality of training input sequences from the training data (step 802). Each training input sequence has a plurality of positions. Each position has a token selected from a token vocabulary. As mentioned above, the token vocabulary may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text and / or computer code. In addition, or alternatively, the token vocabulary may include tokens that can represent data other than text. For example, the token vocabulary may include image tokens that represent a discrete set of image embeddings of an image that can be generated by an image-encoding neural network based on image processing.As another example, the token vocabulary might include audio tokens that represent code vectors in a quantizer's codebook, for example, a residual vector quantizer.

[0185] The system trains the neural network on a mixture of different pretraining tasks (step 804). Each pretraining task uses a respective pretraining goal function. Each pretraining task focuses on training the neural network to learn a different aspect of language modeling. By training the neural network on the mixture of different pretraining tasks, the system can train the neural network in a way that improves its generalization capabilities in the adaptation stage, that is, compared to training the neural network on a single pretraining task.

[0186] The mixture of different pre-training tasks includes two or Petition 870250091754, dated 08 / 10 / 2025, pp. 79 / 116 53 / 67 more than: (i) a causal language modeling pretraining task (as described further below with reference to FIG. 9), (ii) a prefix language modeling pretraining task (as described further below with reference to FIG. 10), or (iii) a span corruption pretraining task (as described further below with reference to FIG. 11).

[0187] In some implementations, the system maintains a respective weight for each of the multiple different pretraining tasks. In some of these implementations, the system can repeatedly: sample a plurality of training input sequences from the training data, sample a pretraining task from the multiple different pretraining tasks according to their respective weights, and train the neural network to perform the sampled pretraining task using the plurality of training input sequences. In this way, the system trains the neural network using the plurality of training input sequences on the same pretraining task.In some other such implementations, the system may sample a plurality of training input sequences from the training data and then, for each of a subset of the plurality of training input sequences: sample a pretraining task from multiple different pretraining tasks according to their respective weights and train the neural network to perform the sampled pretraining task using the subset of the plurality of training input sequences. In this way, the system trains the neural network using the plurality of training input sequences on different pretraining tasks.

[0188] Optionally, after training, the system adapts, for example, through fine-tuning, zero-shot or few-shot adaptation, the trained neural network to perform one or more downstream tasks. The downstream tasks may include any combination of one or more of the machine learning tasks mentioned above and possibly other tasks. Additionally or alternatively, the system outputs data specifying the trained neural network, Petition 870250091754, dated 08 / 10 / 2025, pp. 80 / 116 54 / 67 for example, data specifying the trained values ​​of the parameters and, optionally, the neural network architecture, for another system or for storage on one or more memory devices for future purposes.

[0189] FIG. 9 is a flow diagram of substeps 902-906 of step 804 of the process in FIG. 8. By repeatedly performing iterations of substeps 902-906, the system can train the neural network on the causal language modeling pretraining task based on the optimization of a causal language modeling objective function.

[0190] The system generates, from the plurality of training input sequences, a plurality of causal language modeling input sequences (step 902). In some implementations, a training input sequence can be used as a causal language modeling input sequence without additional processing.

[0191] The system processes, using the neural network, each causal language modeling input sequence to generate, for each token in the causal language modeling input sequence, a causal prediction of a token that should occupy a specific token position in the causal language modeling input sequence when given the tokens in previous positions that precede the specific position in the causal language modeling input sequence (step 904).

[0192] That is, for any position in the causal language modeling input sequence, the neural network processes the tokens in previous positions that precede the given position in the causal language modeling input sequence to generate, as output, a prediction of a token that should occupy the given position in the causal language modeling input sequence.

[0193] For example, for any given position, the neural network can generate a scoring distribution over the tokens in the vocabulary from which a token predicted to occupy the given position can be sampled. The scoring distribution can assign a respective score to each of the Petition 870250091754, dated 08 / 10 / 2025, pp. 81 / 116 55 / 67 tokens in the vocabulary.

[0194] When processing the input sequence to generate, as output, a prediction of a token that should occupy a given position, each of the one or more layers of self-attention within the neural network applies a masked self-attention mechanism over the previous positions in the causal language modeling input sequence, so that the given position does not match, that is, the self-attention layer does not generate a non-zero attention weight for any data that is not in a position prior to the given position or, put another way, the self-attention layer generates non-zero weights only for the positions prior to the given position in the causal language modeling input sequence.

[0195] The system determines, based on the optimization of the causal language modeling objective function, an update for the neural network parameter values ​​(step 906). The system can do this by computing, for each given position in each causal language modeling input sequence, the respective gradients of the objective function with respect to the neural network parameters by backpropagation through the appropriate neural network parameters. The system can then determine the updates by applying an update rule, for example, an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule to the respective gradients.

[0196] The objective function of causal language modeling can be any function that measures a quality of the causal predictions generated by the neural network. For example, the objective function of causal language modeling might include a cross-entropy loss term or a negative log probability loss term that measures, for each given position in each causal language modeling input sequence, a difference between (i) a predetermined score for a fundamental truth token included in the causal language modeling input sequence that occupies the given position (ii) the respective score assigned to the truth token Petition 870250091754, dated 08 / 10 / 2025, pp. 82 / 116 56 / 67 fundamental by the scoring distribution generated by the neural network. The loss function can also include other terms, for example, regularization terms, auxiliary loss terms, and so on.

[0197] FIG. 10 is a flow diagram of substeps 1002-1006 of step 804 of the process in FIG. 8. By repeatedly performing iterations of substeps 1002-1006, the system can train the neural network on the prefix language modeling pretraining task based on the optimization of a prefix language modeling objective function.

[0198] The system generates, from the plurality of training input sequences, a plurality of prefix language modeling input sequences (step 1002). Each prefix language modeling input sequence includes a prefix input sequence followed by a suffix input sequence. The prefix input sequence and the suffix input sequence may each include a respective subset of the tokens included in the prefix language modeling text sequence.

[0199] In some implementations, a prefix language modeling input sequence can be generated from a training input sequence by splitting the training input sequence into two sequences (or subsequences), namely, a prefix input sequence and a suffix input sequence. For example, the prefix input sequence might include a first subset of contiguous tokens included in the prefix language modeling text sequence, and the suffix input sequence might include a second subset of contiguous tokens included in the prefix language modeling text sequence that follows the first subset of contiguous tokens.

[0200] The system processes, using the neural network, each prefix language modeling input sequence to generate, for each token in the suffix input sequence, a causal prediction of a token that should occupy a specific token position in the suffix input sequence. Petition 870250091754, dated 08 / 10 / 2025, pp. 83 / 116 57 / 67 conditional on (a) tokens in the prefix input sequence and (b) tokens in any earlier positions in the suffix input sequence that precede the specific position in the prefix language modeling input sequence (step 1004).

[0201] That is, for any position in the suffix input sequence in the prefix language modeling input sequence, the neural network processes an input sequence that includes (a) tokens in the prefix input sequence that precedes the suffix input sequence in the prefix language modeling input sequence and (b) tokens in any earlier positions in the suffix input sequence that precedes the specific position in the prefix language modeling input sequence to generate, as output, a prediction of a token that should occupy the given position in the suffix input sequence.

[0202] For example, for any given position, the neural network can generate a scoring distribution over the tokens in the vocabulary from which a token predicted to occupy the given position can be sampled. The scoring distribution can assign a respective score to each of the tokens in the vocabulary.

[0203] When processing the input sequence to generate, as output, a prediction of a token that should occupy a given position, each of the one or more layers of self-attention within the neural network applies (i) a bidirectional and unmasked attention mechanism to the positions in the prefix input sequence and (ii) a masked self-attention mechanism to positions in the suffix input sequence, such that the given position matches all positions in the prefix input sequence and any preceding positions in the suffix input sequence that precede the given position. Meanwhile, any position in the prefix input sequence may match any other position in the prefix input sequence, including any subsequent position that is after the position in the prefix input sequence. However, the given position does not match, i.e., the layer of Petition 870250091754, dated 08 / 10 / 2025, pp. 84 / 116 58 / 67 self-attention generates a zero attention weight for any data that is in a position after the given position in the suffix input sequence or, put another way, the self-attention layer generates non-zero attention weights only for positions in the prefix input sequence and positions prior to the given position in the suffix input sequence.

[0204] The system determines, based on the optimization of the prefix language modeling objective function, an update for the neural network parameter values ​​(step 1006). The system can do this by computing, for each given position in the suffix input sequence in each prefix language modeling input sequence, the respective gradients of the objective function with respect to the neural network parameters by backpropagation through the appropriate neural network parameters. The system can then determine the updates by applying an update rule, for example, an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule to the respective gradients.

[0205] The prefix language modeling objective function can be any function that measures a quality of the prefix predictions generated by the neural network. For example, the prefix language modeling objective function might include a cross-entropy loss term or a negative log likelihood loss term that measures, for each given position in the suffix input sequence in each prefix language modeling input sequence, a difference between (i) a predetermined score for a fundamental truth token included in the suffix input sequence that occupies the given position (ii) the respective score assigned to the fundamental truth token by the scoring distribution generated by the neural network. The loss function might also include other terms, for example, regularization terms, auxiliary loss terms, and so on.

[0206] FIG. 11 is a flow diagram of substeps 1102-1106 of step 804 of the process in FIG. 8. By repeatedly performing iterations of Petition 870250091754, dated 08 / 10 / 2025, pages 85 / 116 In substeps 59 / 67, 1102-1106, the system can train the neural network on the span corruption pretraining task based on the optimization of a span corruption objective function.

[0207] The system generates, from the plurality of training input sequences, a plurality of span-masked input sequences (step 1102). Each span-masked input sequence includes a plurality of tokens separated by one or more mask tokens. A mask token is a token that includes predetermined numeric values ​​and means that the token to occupy the mask token position has not yet been generated, for example, selected from the token vocabulary.

[0208] In some implementations, a span-masked input sequence can be generated from a matching training input sequence by replacing a subset of the tokens included in the training input sequence with mask tokens. The subset of tokens may include tokens that are adjacent to each other within the training input sequence, that is, that occupy multiple contiguous positions in the training input sequence.

[0209] For each span-masked input sequence, the system processes, using the neural network, the span-masked input sequence to generate a prediction of one or more tokens that should occupy the respective positions of one or more mask tokens in the span-masked input sequence (step 1104). For example, for any position occupied by a mask token, the neural network can generate a score distribution over the tokens in the vocabulary from which a token predicted to occupy the position, i.e., to replace the mask token, can be sampled. The score distribution can assign a respective score to each of the tokens in the vocabulary.

[0210] Based on the optimization of the span corruption objective function, the system determines an update for the neural network parameter values ​​(step 1106). The system can do this by computing, for each position Petition 870250091754, dated 08 / 10 / 2025, pages 86 / 116 60 / 67 given in each prefix language modeling input sequence, the respective gradients of the objective function with respect to the neural network parameters are backpropagated through the appropriate neural network parameters. The system can then determine the updates by applying an update rule, for example, an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule to the respective gradients.

[0211] The objective span corruption function can be any function that measures a quality of the span predictions generated by the neural network. For example, span corruption might include a cross-entropy loss term or a negative log probability loss term that measures, for any position occupied by a mask token in each span-masked input sequence, a difference between (i) a predetermined score for a fundamental truth token included in the corresponding training input sequence (based on which the span-masked input sequence is generated) that occupies the position (ii) the respective score assigned to the fundamental truth token by the scoring distribution generated by the neural network. The loss function might also include other terms, for example, regularization terms, auxiliary loss terms, and so on.

[0212] This descriptive report uses the term configured in connection with computer systems and program components. A system of one or more computers being configured to perform specific operations or actions means that the system has installed software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operations or actions. One or more computer programs being configured to perform specific operations or actions means that the one or more programs include instructions that, when executed by the data processing apparatus, cause the apparatus to perform the operations or actions. Petition 870250091754, dated 08 / 10 / 2025, page 87 / 116 61 / 67

[0213] The material modalities and functional operations described in this descriptive report may be implemented in digital electronic circuits, in tangibly embedded computer software or firmware, in computer hardware, including the structures disclosed in this descriptive report and their structural equivalents, or in combinations of one or more thereof. The material modalities described in this descriptive report may be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded in a tangible non-transient storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial memory device, or a combination of one or more thereof.Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0214] The term data processing device refers to data processing hardware and encompasses all types of devices, apparatus, and machines for data processing, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be, or include, special-purpose logic circuits, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The device may optionally include, in addition to hardware, code that creates an execution environment for computer programs, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. Petition 870250091754, dated 08 / 10 / 2025, pp. 88 / 116 62 / 67

[0215] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and may be implemented in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a portion of a file that contains other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, subprograms, or portions of code.A computer program can be implemented to run on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0216] In this descriptive report, the term database is used broadly to refer to any collection of data: the data need not be structured in any specific way, nor structured in any way, and may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0217] Similarly, in this descriptive report, the term tool is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, a tool will be implemented as one or more software modules or components, installed on one or more computers. Petition 870250091754, dated 08 / 10 / 2025, pp. 89 / 116 63 / 67 in one or more locations. In some cases, one or more computers will be dedicated to a specific tool; in other cases, multiple tools may be installed and run on the same computer or computers.

[0218] The processes and logical flows described in this descriptive report can be performed by one or more programmable computers running one or more computer programs to perform functions operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuits, for example, an FPGA or an ASIC, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0219] Computers suitable for running a computer program may be based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit to perform or execute instructions and one or more memory devices to store instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, special-purpose logic circuits. In general, a computer will also include, or be operationally coupled to receive data from, or transfer data to, or both, one or more mass storage devices to store data, for example, magnetic disks, magnetic optical disks, or optical disks.However, a computer does not need to have such devices. Furthermore, a computer can be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few. Petition 870250091754, dated 08 / 10 / 2025, pp. 90 / 116 64 / 67

[0220] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, memory media and devices, including, by way of example, semiconductor memory devices, for example, EPROM, EEPROM and flash memory devices; magnetic disks, for example, internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0221] To provide interaction with a user, the modalities of the subject matter described in this descriptive report may be implemented on a computer with a display device, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, to display information to the user, and a keyboard and a pointing device, for example, a mouse or a trackball, by which the user may provide input to the computer. Other types of devices may be used to provide interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback; and user input may be received in any form, including acoustic, speech, or tactile input.Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of messaging to a personal device, for example, a smartphone running a messaging application, and receiving responsive messages from the user in return.

[0222] The data processing apparatus for implementing machine learning models may also include, for example, special-purpose hardware accelerator units for processing common and computationally intensive parts of training or output. Petition 870250091754, dated 08 / 10 / 2025, pp. 91 / 116 65 / 67 machine learning, i.e., inference, workloads.

[0223] Machine learning models can be implemented and deployed using a machine learning framework, for example, a TensorFlow framework or a JAX framework.

[0224] The material described in this descriptive report may be implemented in a computing system that includes a back-end component, for example, a data server, or that includes a middleware component, for example, an application server, or that includes a front-end component, for example, a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the material described in this descriptive report, or any combination of one or more of these back-end, middleware, or front-end components. The system components may be interconnected by any form or means of digital data communication, for example, a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), for example, the Internet.

[0225] A computer system may include clients and servers. A client and a server are generally remote from each other and typically interact through a communication network. The client-server relationship emerges by virtue of computer programs running on their respective computers and a client / server relationship with each other. In some embodiments, a server transmits data, for example, an HTML page, to a user device, for example, for purposes of displaying data and receiving user input from a user interacting with the device, which acts as a client. The data generated on the user's device, for example, a result of the user's interaction, may be received on the server from the device.

[0226] Although this descriptive report contains many specific implementation details, these should not be interpreted as limitations. Petition 870250091754, dated 08 / 10 / 2025, pp. 92 / 116 66 / 67 within the scope of any invention or within the scope of what may be claimed, but rather as descriptions of features that may be specific to specific embodiments of specific inventions. Certain elements that are described in this descriptive report in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, several elements that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although elements may be described above as acting in certain combinations and even initially claimed as such, one or more elements of a claimed combination may, in some cases, be extirpated from the combination and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0227] Similarly, although operations are depicted in the figures and cited in the claims in a specific order, this should not be understood as requiring that such operations be performed in the specific order shown or in sequential order, or that all illustrated operations be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0228] The specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions mentioned in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the order Petition 870250091754, dated 08 / 10 / 2025, pp. 93 / 116 A specific 67 / 67 sequence, or sequential order, is used to achieve desired results. In some cases, multitasking and parallel processing can be advantageous. Petition 870250091754, dated 08 / 10 / 2025, pp. 94 / 116

Claims

1 / 9 CLAIMS 1. A method performed by one or more computers characterized in that the method comprises: obtaining a first training input sequence that includes first tokens and a second training input sequence that includes second tokens; generating a modified first training input sequence comprising: selecting a plurality of second tokens from the second tokens included in the second training input sequence; determining a first canary position within the first training input sequence; and inserting the selected plurality of second tokens into the first training input sequence at positions after the first canary position; training a neural network including neural network learning parameter values ​​using a training dataset comprising the modified first training input sequence;After training, use the trained neural network to generate, from a test input sequence comprising a subset of the first tokens, one or more predicted continuations of the test input sequence, where each predicted continuation specifies a plurality of output tokens; and determine an estimate of the degree to which the trained neural network memorizes data in the training dataset, comprising evaluating the plurality of output tokens specified by each predicted continuation against the selected plurality of second tokens included in the modified first training input sequence.

2. Method, according to claim 1, characterized in that the training dataset comprises the first modified training input sequence and a second modified training input sequence, the second modified training input sequence being generated by: selecting a plurality of first tokens from the first tokens included in the first training input sequence; determining a second canary position within the second training input sequence; and inserting the selected plurality of first tokens into the second training input sequence at positions after the second canary position.

3. A method, according to any one of claims 1-2, characterized in that each predicted continuation includes the plurality of output tokens and in which the evaluation of the plurality of output tokens with respect to the selected plurality of second tokens comprises: determining how many of the selected plurality of second tokens included in the first modified training input sequence are included in the plurality of output tokens.

4. A method, according to any one of claims 1-2, characterized in that each predicted continuation defines, for each exit token position, a corresponding probability distribution over a vocabulary of tokens and wherein the evaluation of the plurality of exit tokens against the selected plurality of second tokens comprises: calculating a probability assigned to at least one subset of the selected plurality of second tokens by the probability distributions for the exit token positions.

5. A method, according to any one of claims 3-4, characterized in that it further comprises: determining whether the estimate satisfies a threshold degree; and in response, applying one or more adjustments to the neural network training to reduce the estimate. Petition 870250091754, dated 10 / 08 / 2025, p. 96 / 116 3 / 9 6. Method according to claim 5, characterized in that determining whether the estimate satisfies the threshold degree comprises: determining that the plurality of output tokens includes more than one threshold number of the selected plurality of second tokens.

7. A method according to claim 5, characterized in that determining whether the estimate satisfies the threshold degree comprises: determining whether the probability satisfies a probability threshold.

8. A method, according to any one of claims 1-7, characterized in that determining the first canary position within the first training input sequence comprises: determining the first canary position based on the positions of the selected plurality of second tokens within the second training input sequence.

9. A method, according to any one of claims 1-8, characterized in that selecting the plurality of second tokens comprises: selecting a predetermined number of second tokens that are adjacent to each other within the second training input sequence.

10. A method, according to any one of claims 1-9, characterized in that the first training input sequence and the second training input sequence are both in the same natural language.

11. A method, according to any one of claims 1-9, characterized in that the first training input sequence is in a first natural language and the second training input sequence is in a second natural language different from the first language.

12. Method, according to any one of claims 1-10, characterized in that the training dataset comprises Petition 870250091754, dated 08 / 10 / 2025, page 97 / 116 4 / 9, and also a third modified training input sequence, the third modified training input sequence being generated by: obtaining a third training input sequence; determining, from a plurality of toxicity levels, a specific toxicity level based on the content represented by the third training input sequence; prefixing a specific toxicity token corresponding to the specific toxicity level to the third training input sequence.

13. Method according to claim 12, characterized in that the plurality of toxicity levels comprises three or more toxicity levels.

14. A method, according to any one of claims 12-13, characterized in that prefixing the specific toxicity token to the third training input sequence comprises: determining whether to prefix the specific toxicity token to the third training input sequence based on a total number of training input sequences to which toxicity tokens have been prefixed.

15. A method, according to any one of claims 12-14, characterized in that it further comprises: obtaining a context sequence that includes a plurality of input tokens; generating, from the context sequence, a quality-controlled context sequence by adding to the context sequence a specific toxicity token selected from the plurality of toxicity tokens that represent toxicity levels of the content represented by the output sequence; and generating, using the trained neural network, an output sequence based on the processing of the quality-controlled context sequence.

16. Method, according to claim 15, characterized by the fact that the quality-controlled context sequence comprises the specific quality token followed by the plurality of input tokens.

17. A method, according to any one of claims 1-16, characterized in that the neural network is a Transformer neural network that autoregressively generates the output tokens.

18. Method performed by one or more computers, the method characterized by the fact that it comprises: obtaining unmarked text in multiple languages; generating, from the unmarked text, a first multilingual training dataset comprising a plurality of first multilingual training pairs, wherein each first multilingual training pair includes a first source text sequence and a first target text sequence;Generate, from the unmarked text, a second multilingual training dataset comprising a plurality of second multilingual training pairs, wherein each second multilingual training pair includes a second source text sequence and a second target text sequence, and wherein the first source text sequence and the second source text sequence are both in the same language while the first target text sequence and the second target text sequence are in different languages; and train a language model neural network including neural network learning parameter values ​​using the first and second multilingual training datasets based on the optimization of a pretraining objective function.

19. Method, according to claim 18, characterized in that it further comprises: after training, adapting the trained neural network to perform a translation task by processing an input text sequence in a source language to generate an output text sequence in a target language that is a translation of the input text sequence into the target language.

20. A method according to claim 18, characterized in that it further comprises: after training, adapting the trained neural network to perform a multilingual question-answering task by processing an input text sequence in the source and target languages ​​to generate an output text sequence in the target language that is a response to the input text sequence.

21. A method performed by one or more computers, characterized in that the method comprises: obtaining a plurality of untagged text sequences, wherein each untagged text sequence comprises a plurality of text tokens; training an autoregressive generative neural network comprising one or more layers of self-attention based on the optimization of multiple different pretraining objective functions comprising (i) a causal language modeling objective function and (ii) a prefix language modeling objective function, wherein training the autoregressive generative neural network based on the optimization of the causal language modeling objective function comprises: generating, from the plurality of untagged text sequences, a plurality of causal language modeling text sequences;To process, using the autoregressive generative neural network, each causal language modeling text sequence to generate, for each token in the causal language modeling text sequence, a causal prediction of a text token that should occupy a specific text token position in the causal language modeling text sequence conditioned on text tokens in preceding positions in the causal language modeling text sequence, wherein one or more layers of self-attention Petition 870250091754, dated 10 / 08 / 2025, page 100 / 116 7 / 9 within the autoregressive generative neural network apply a masked self-attention mechanism on the preceding positions in the causal language modeling text sequence;and determine, based on a quality of causal predictions, an update of the parameter values ​​of the autoregressive generative neural network, wherein training the autoregressive generative neural network based on the optimization of the prefix language modeling objective function comprises: generating, from the plurality of unmarked text sequences, a plurality of prefix language modeling text sequences, each prefix language modeling text sequence comprising a prefix text sequence followed by a suffix text sequence;To process, using the autoregressive generative neural network, each prefix-modeling language text sequence to generate, for each token in the suffix text sequence, a causal prediction of a text token that must occupy a specific token position in the suffix text sequence conditioned on tokens in the prefix text sequence and tokens in any preceding positions in the suffix text sequence, wherein one or more layers of self-attention within the autoregressive generative neural network apply a bidirectional and unmasked attention mechanism on the positions in the prefix text sequence and apply a masked self-attention mechanism on positions in the suffix text sequence such that each position in the suffix text sequence meets the positions in the prefix text sequence and any preceding positions in the suffix text sequence;and determine, based on a quality of the prefix predictions, an update of the parameter values ​​of the generative autoregressive neural network.

22. Method according to claim 21, characterized in that the multiple different pre-training objective functions comprise (iii) a span corruption objective function and in which training Petition 870250091754, dated 08 / 10 / 2025, p.101 / 116 8 / 9 The autoregressive generative neural network based on the optimization of the span corruption objective function comprises: generating, from a plurality of unmarked text sequences, a plurality of span-masked text sequences, wherein each span-masked text sequence comprises a plurality of text tokens separated by one or more mask tokens; processing, using the autoregressive generative neural network, each span-masked text sequence to generate a span prediction of one or more text tokens that should occupy the respective positions of one or more mask tokens in the span-masked text sequence; and determining, based on a quality of the span prediction, an update to the parameter values ​​of the autoregressive generative neural network.

23. A method, according to any one of claims 21-22, characterized in that training the autoregressive generative neural network based on the optimization of multiple different pretraining objective functions comprises: receiving user input specifying a respective weight assigned to each of the multiple different pretraining objective functions; and repeatedly selecting, based on the specified weights, a pretraining objective function from the multiple different pretraining objective functions.

24. Method according to claim 23, characterized in that it further comprises: sampling a batch of untagged text sequences; and processing the batch of untagged text sequences according to the selected pretraining objective function.

25. Method, according to any one of claims 21-24, characterized in that the autoregressive generative neural network is a decoding-only attentional neural network.

26. A method, according to any one of claims 21-24, characterized in that the autoregressive generative neural network is an encoder-decoder attentional neural network.

27. A system characterized in that it comprises one or more computers and one or more storage devices storing instructions which, when executed by one or more computers, will cause one or more computers to perform the operations of the respective method as defined in any one of claims 1-26.

28. One or more computer storage media characterized in that they store instructions which, when executed by one or more computers, cause one or more computers to perform operations of the respective method as defined in any of claims 1-26. Petition 870250091754, dated 10 / 08 / 2025, pp. 103-116

Images