Executing tasks using generative neural networks

Abstract

A method, system, and apparatus for performing a task, including a computer program encoded on a computer storage medium. One method comprises obtaining a sequence of input tokens, each token selected from a vocabulary of tokens including text tokens and audio tokens, the sequence of input tokens including tokens describing a task to be performed and data for performing the task, the method further comprises generating a sequence of embeddings by embedding each token in the sequence of input tokens into an embedding space, and processing the sequence of embeddings using a language model neural network to generate a sequence of output tokens for a task, each token selected from a vocabulary.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 522,650, filed Jun. 22, 2023. The disclosure of the prior application is considered a part of the disclosure of this application and is incorporated into the disclosure of this application by reference.

Background Art

[0002] This specification relates to performing tasks using neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of non - linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters.

Summary of the Invention

[0004] This specification describes a system that is executed as a computer program on one or more computers in one or more locations, using one or more generative neural networks to perform a task. For example, the task can include generating text, generating an audio signal, or generating both text and an audio signal.

[0005] Generally, an output audio signal contains samples of the audio wave at each of a sequence of time steps spanning a specified time window. For example, the time steps may be arranged at regular intervals within the specified time window. An audio sample at a given time step may be the amplitude value of the audio wave, or the amplitude value after compression, decompression, or both. For example, an audio sample may be the raw amplitude value or the μ-law decompressed representation of the amplitude value.

[0006] In general, one innovative aspect of the subject matter described herein can be embodied in a method comprising the action of obtaining a sequence of input tokens, each token selected from a vocabulary of tokens including text tokens and audio tokens, the sequence of input tokens including tokens describing a task to be performed and data for performing the task, and the method further comprising the action of generating a sequence of embeddings by embedding each token in the sequence of input tokens into an embedding space, and the action of processing the sequence of embeddings using a language model neural network to generate a sequence of output tokens for a task, each token selected from a vocabulary.

[0007] In some embodiments, generating a sequence of embeddings by embedding each token in the sequence of input tokens into the embedding space includes maintaining the respective embedding for each token within the token vocabulary and, for each token in the sequence of input tokens, mapping the token to the respective embedding for that token.

[0008] In some embodiments, processing a sequence of embeddings using a language model neural network to generate a sequence of output tokens for a task includes processing a sequence of embeddings using a language model neural network to autoregressively generate a sequence of output tokens for a task.

[0009] In some embodiments, embeddings of arbitrary text tokens in the vocabulary are learned during pre-training of the language model neural network specifically for text, embeddings of arbitrary audio tokens in the vocabulary are learned during audio-text training of the language model neural network, and text token embeddings are fixed and retained during audio-text training of the language model neural network.

[0010] In some embodiments, obtaining a sequence of input tokens includes receiving an input text tag describing a task to be performed, receiving one or more sequences of data for performing the task, and generating a sequence of input tokens from the input text tag and the one or more sequences of data.

[0011] In some embodiments, one or more sequences of data include text, and generating a sequence of input tokens includes applying a text tokenizer to the text to generate a sequence of text tokens, and including a sequence of text tokens in the sequence of input tokens.

[0012] In some embodiments, one or more sequences of data include an audio signal, and generating a sequence of input tokens includes applying an audio tokenizer to the audio signal to generate a sequence of audio tokens and including a sequence of audio tokens in the sequence of input tokens.

[0013] In some embodiments, applying an audio tokenizer to an audio signal involves generating a semantic representation of the audio signal that specifies each audio signal at each of a plurality of first time steps spanning the audio signal, where each audio token is selected from an audio token lexicon and represents the semantic content of the audio signal at the corresponding first time step.

[0014] In some embodiments, generating a semantic representation of an audio signal involves processing the audio signal using an audio representation neural network trained to generate a representation of the input audio in order to generate a semantic representation of the audio signal.

[0015] In some embodiments, processing an audio signal using an audio representation neural network trained to generate a representation of an input audio in order to generate a semantic representation of the audio signal includes processing the audio signal using the audio representation neural network to generate an encoded vector for each of the first time steps, and for each of the first time steps, selecting the audio token that is closest to the encoded vector generated by the audio representation neural network for the first time step as the audio token for the first time step.

[0016] In some embodiments, generating a sequence of input tokens includes applying a text tokenizer to input text tags to generate a sequence of text tokens, and including a sequence of text tokens in the sequence of input tokens.

[0017] In some embodiments, the method further includes detoxifying a sequence of output tokens in order to generate an output that satisfies the task.

[0018] In some embodiments, the task's output includes text, audio signals, or both.

[0019] In some embodiments, the sequence of output tokens includes multiple text tokens, and detoxifying the sequence of output tokens involves processing the text tokens to generate a text prediction.

[0020] In some embodiments, the sequence of output tokens includes a plurality of audio tokens, and detokenizing the sequence of output tokens includes using one or more neural networks to generate an acoustic representation of the audio signal, conditionally on at least a plurality of audio tokens, wherein the acoustic representation specifies one or more sets of respective acoustic tokens at each of a plurality of second time steps spanning the audio signal, and each of the one or more respective acoustic tokens at each second time step represents the acoustic properties of the audio signal at the corresponding second time step, and detokenizing the sequence of output tokens further includes processing at least the acoustic representation using a decoder neural network to generate a prediction of the audio signal.

[0021] In some embodiments, generating an acoustic representation includes generating an acoustic representation of an audio signal, conditionally on a plurality of audio tokens and a speech context, and processing at least the acoustic representation using a decoder neural network includes processing the acoustic representation of the audio signal and the acoustic representation of the speech context using a decoder neural network to generate a prediction of the audio signal, conditionally on the speech context.

[0022] In some embodiments, the audio context includes each audio token at each of a plurality of first time steps spanning at least a portion of the audio signal, and each audio token is selected from an audio token lexicon to represent the semantic content of the audio signal at the corresponding first time step.

[0023] In some embodiments, the task includes one or more of the following: automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, or text-to-text machine translation.

[0024] In some embodiments, the task includes a plurality of subtasks, the input text tags specify each of the subtasks, and the sequence of output tokens includes each output for each of the subtasks.

[0025] In some embodiments, the subtask includes any one of automatic speech recognition, automatic speech translation, cross-lingual speech translation, text-to-speech, or machine translation between texts.

[0026] In some embodiments, the language model neural network is trained on one or more tasks.

[0027] Other embodiments of this aspect include a corresponding computer system, an apparatus, and a computer program recorded on one or more computer storage devices each configured to perform the actions of the method.

[0028] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0029] The system described herein is a multimodal architecture capable of processing and generating text and speech. The system may include a large-scale language model neural network and an audio framework for generating audio. For example, the system may use a shared vocabulary to represent utterances and text. A single decoder-only model can be trained to perform multiple tasks, or combinations of tasks with optionally interleaved utterances and text. For example, the system may obtain a sequence of input tokens containing text tokens, audio tokens, or both text and audio tokens, selected from a vocabulary of text and audio tokens. The system may generate a sequence of embeddings by embedding each token in an embedding space. The system may then process the sequence of embeddings using a language model neural network to generate a sequence of output tokens, each token selected from the vocabulary.

[0030] For example, speech recognition, text-to-speech synthesis, and speech translation are special cases of tasks involving speech and text. As an example, a system can generate speech, text, or both from input data that preserves paralinguistic information such as speaker identification and intonation. For example, to perform a task, the system obtains a sequence of input tokens that includes tokens describing the task to be performed and the data for performing the task. In an example where the data for performing the task is an audio signal input, the system can use an audio tokenizer that generates audio tokens that preserve paralinguistic information from the audio signal input to generate tokens that describe the data for performing the task. The system can generate a sequence of embeddings by embedding each token within an embedding space. The system can use a language model neural network to process the sequence of embeddings to generate a sequence of output tokens, where each token is selected from a vocabulary. Thus, the output tokens can include audio tokens that preserve paralinguistic information from the audio signal input.

[0031] The system can also generate speech, text, or both from input data that preserves the linguistic knowledge of the input data. For example, to perform a task, the system obtains a sequence of input tokens that includes tokens describing the task to be performed and the data for performing the task. In an example where the data for performing the task includes an audio signal input representing speech, the system can use an audio tokenizer that generates audio tokens that preserve the linguistic information from the audio signal input to generate tokens that describe the data for performing the task. In an example where the data for performing the task includes text, the system can use a text tokenizer that generates text tokens that preserve the linguistic information from the text to generate tokens that describe the data for performing the task.

[0032] Typically, conventional systems use heterogeneous models to perform different tasks. In contrast, the systems described herein can be trained to perform various tasks using a single architecture and training run, saving computational resources required for training and inference. For example, the system can represent different tasks using text tags. In some examples, to perform a task, the system takes a sequence of input tokens containing tokens that describe the task to be performed and the data needed to perform that task. The system can generate tokens describing a task by applying a text tokenizer to the task's text tag. The system can be trained with training data containing input text tags that specify tasks for training examples.

[0033] The system described herein can incorporate existing text-only language model neural networks without requiring the training of a language model neural network from scratch, thereby reducing the computing time and resources required for training. For example, the language model neural network can be initialized with the weights of a text-only language model neural network. The system can acquire data specifying a pre-trained text-only language model neural network. The system can extend the embedding matrix of the pre-trained language model neural network to include audio token mappings. The system can further train the language model neural network, including the embedding matrix, from the pre-trained values, for example, by fine-tuning it. Thus, the system can leverage existing language model neural networks to train faster and more efficiently.

[0034] The system can also improve model performance by leveraging larger amounts of available text training data to assist with speech or text-related tasks. Because a large amount of text training data is available for speech data, the system can use the language and reasoning knowledge learned by the language model neural network when performing speech-related tasks. The system can improve model performance on audio / speech-related tasks by leveraging larger amounts of text training data, specifically by using a pre-trained text-only language model neural network as the model's starting foundation.

[0035] Language modeling neural networks perform better than existing systems in tasks such as speech translation and zero-shot speech-to-text translation for languages ​​where the input / target language combination was not known during training. By using pre-trained language modeling neural networks trained on large amounts of text training data, the system can leverage the translation capabilities of the pre-trained language modeling neural network.

[0036] The system offers superior performance, such as generating high-quality audio, and in some cases, faithfully preserves the speaker's voice in tasks like speech-to-speech translation involving the transmission of an unseen speaker's voice, compared to conventional systems. Conventional speech-to-speech translation systems typically consist of a cascade of automatic speech recognition, text-to-text machine translation, and text-to-speech synthesis. However, cascading methods focus primarily on text and may overlook important aspects such as paralinguistic features, computational efficiency, compound errors, and accurate handling of proper nouns, nouns, and non-linguistic communication that does not require translation. The system described herein surpasses conventional methods in terms of speech quality and speech preservation by directly generating audio in the target language from audio in the source language. For example, the system can obtain a sequence of input tokens containing text tokens, audio tokens, or both text and audio tokens, selected from a vocabulary of text tokens and audio tokens. The system can generate a sequence of embeddings by embedding each token in an embedding space. The system can process the sequence of embeddings using a language model neural network to generate a sequence of output tokens, each token selected from a vocabulary. In the case of speech-to-speech translation, each output token selected from the vocabulary may be an audio token. Therefore, the system can perform speech-to-speech translation using only a language model neural network.

[0037] In some examples, a system may be required to perform multiple tasks or subtasks. For example, a system may be required to perform inter-speech translation by performing subtasks such as automatic speech recognition, automatic speech translation, and inter-speech translation. The system performs subtasks as a single autoregressive decoder (i.e., producing a single sequential output, one token at a time, and the output sequence contains all tasks / subtasks), which allows the model to handle the input and all pre-decoded content at each stage or subtask, as opposed to separate pipeline methods. For example, a system can be trained to perform subtasks by being trained with training examples that contain input text tags identifying subtasks for a task. The training output of each training example may contain the output of each subtask so that the language model neural network is trained to produce a sequence of output tokens containing the output tokens of each subtask.

[0038] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0039] [Figure 1] This is an illustrative block diagram of a system for performing tasks that require generating text, generating audio, or both. [Figure 2] This is a block diagram of another exemplary system for performing tasks that require generating text, generating audio, or both. [Figure 3] This is an illustrative process flowchart for performing tasks that require generating text, generating audio, or both. [Figure 4]This is an exemplary process flowchart for training a system to perform tasks that require generating text, generating audio, or both. [Figure 5] This diagram illustrates the performance of an exemplary system for performing tasks that require generating text, generating audio, or both. [Modes for carrying out the invention]

[0040] Similar reference numbers and names in various drawings refer to the same elements.

[0041] Figure 1 is a block diagram of an exemplary system 100 for performing tasks that require generating text, generating audio, or both. System 100 is an example of a system that runs as a computer program on one or more computers in one or more locations, on which the systems, components, and technologies described below are performed.

[0042] System 100 can perform tasks that require generating text, generating audio, or both, given a sequence of input tokens 104. System 100 can be configured to perform any of the following tasks that require (i) receiving an input containing both text and audio signals, (ii) generating an output containing audio signals, or (iii) receiving an input containing audio signals and generating an output containing audio signals. Some examples of such tasks are described below. In some examples, the input or output may include any interleaved utterances and text.

[0043] To generate text or audio, the system obtains a sequence of input tokens 104. This sequence may include tokens describing tasks to be performed and data for performing those tasks (e.g., task input data). For example, some tokens might correspond to input text tags indicating or describing tasks to be performed.

[0044] Some tokens in the input token sequence 104 may correspond to one or more sequences of data 102 for performing a task. Sequence data can include, for example, text or audio signals. For example, some tokens in the input token sequence 104 may correspond to text in a sequence of data. Some tokens in the input token sequence 104 may correspond to audio signals in a sequence of data. The system can retrieve text tokens or audio tokens selected from a vocabulary of text tokens and audio tokens.

[0045] Figure 1 shows a sequence of multiple data 102 that can be represented by tokens in the input to system 100. For example, system 100 receives audio signals such as audio signal 102a representing the utterance "bonjour le monde" and audio signals such as audio signal 102b representing the utterance "ciao mondo". System 100 can also receive text such as text 102c containing "hello world". In the example in Figure 1, "bonjour le monde" is the French translation of "hello world" and "ciao mondo" is the Italian translation of "hello world".

[0046] In some embodiments, system 100 can generate a sequence of input tokens 104. System 100 can receive an input text tag describing a task to be performed, and a sequence of one or more data 102 for performing the task. System 100 can generate a sequence of input tokens from the input text tag and the sequence of one or more data 102. The sequence of one or more data 102 may include text or audio signals. Thus, the sequence of input tokens 104 can be generated from data from multiple modalities. Generating the sequence of input tokens 104 is described in more detail below with reference to Figure 2.

[0047] In some examples, system 100 receives input text tags and a sequence of one or more data 102 for performing a task from the user. For example, system 100 can receive input text tags and a sequence of one or more data 102 from the user through the user interface of the user device.

[0048] System 100 uses a language model neural network to process a sequence of input tokens 104 to generate an output 110 that satisfies a task. The task output 110 may include text, an audio signal, or both. Different tasks can generate output 110 by having the language model neural network generate different types of output tokens that the system processes. The task output 110 is defined by the sequence of input tokens 104. For example, the language model neural network can generate different types of output tokens, such as audio tokens or text tokens, according to the tokens in the sequence of input tokens 104 that describe the task to be performed and the data needed to perform the task.

[0049] Figure 1 shows an exemplary output, an audio signal 110a representing the utterance "helloworld". Figure 1 also shows an exemplary output, text 110b containing "ciao mondo". Generating output 110 is described in further detail below with reference to Figure 2.

[0050] Generally, the output 110 generated by system 100 fulfills tasks such as automatic speech recognition (ASR), automatic speech translation (AST), speech-to-speech translation (S2ST), text-to-speech (TTS), or text-to-machine translation (MT). In other words, system 100 can perform any appropriate task specified by the input text tag. An example input text tag is described in more detail below with reference to Figure 2.

[0051] For example, system 100 can perform automatic speech recognition. System 100 can receive an audio signal as input and generate text as output. The audio signal may be a speech signal, and the text may include a transcript of the content of the speech signal. In the example in Figure 1, system 100 can receive an audio signal 102b representing the utterance "ciao mondo" and generate text 110b containing the text "ciao mondo" as output. In some examples, the system can receive an input text tag that specifies the task as automatic speech recognition.

[0052] As another example, system 100 can also perform automatic speech translation. System 100 can receive an audio signal as input and generate text as output. The audio signal may be a speech signal, and the text may include a translated transcript of the content of the speech signal in a different language than the one contained in the speech signal. In the example in Figure 1, system 100 can perform automatic speech translation from French to Italian. For example, the system can receive an audio signal 102a representing the utterance "bonjour le monde" and generate text 110b as output, which includes the text "ciao mond". In some examples, the system can receive an input text tag specifying the task and target language, Italian, as the automatic speech translation. In some examples, the input text tag may also specify the source language, French. System 100 can perform the translation directly, that is, it converts the audio signal to text and does not need to translate the text separately.

[0053] As another example, system 100 can perform speech-to-speech translation. System 100 can receive an audio signal as input and produce an audio signal as output. The input audio signal may be a spoken signal, and the output audio signal may contain the same semantic content as the spoken signal but be spoken differently. For example, the input audio signal may contain a spoken utterance in one natural language, and the output audio signal may represent a spoken utterance in a different natural language of the target, which is a translation of the input utterance into the target language. In the example in Figure 1, system 100 can perform speech-to-speech translation from French to English. For example, the system can receive an audio signal 102a representing the utterance "bonjour le monde" and produce an audio signal 110a representing "hello world" as output. In some examples, the system can receive an input text tag specifying the task and target language, English, as the speech-to-speech translation. In some examples, the input text tag may also specify the source language, French. System 100 can perform translation directly using audio data; that is, it does not require separate steps of converting the audio signal to text, translating the text, and generating speech from the translated text.

[0054] As another example, system 100 can perform speech-to-speech translation from Italian to English. For example, the system can receive an audio signal 102b representing the utterance "ciao mondoe" and produce an audio signal 110a representing the utterance "hello world" as output. In some examples, the system can receive an input text tag specifying the task and target language, English, for speech-to-speech translation. In some examples, the input text tag can also specify the source language, Italian.

[0055] As another example, system 100 can perform text-to-speech generation. System 100 can receive text as input and generate an audio signal as output. The output audio signal may be a spoken text signal; that is, the output audio signal may contain spoken words corresponding to the input text. In the example in Figure 1, system 100 can receive text 102c containing the text "hello world" and generate an audio signal 110a as output representing the spoken word "hello world". In some examples, the system can receive an input text tag specifying a task as text-to-speech.

[0056] In addition to the tasks described above, system 100 can also perform text-to-text tasks that do not require audio input or output. For example, the system can perform text-to-text machine translation. The system can receive text as input and produce text as output. The output text may include text that is a translation of the input text into a target language. In the example in Figure 1, system 100 can perform machine translation from English to Italian. For example, system 100 can receive text 102c containing the text "hello world" and produce text 110b containing the text "ciao mond" as output. In some examples, the system can receive an input text tag that specifies the task as machine translation and the target language, Italian. In some examples, the input text tag may also specify the source language, English. Thus, using different input text tags that specify different tasks, the system can perform both unimodal and multimodal tasks, such as text-to-text machine translation.

[0057] In some embodiments, system 100 can execute combinations of tasks. For example, a task may include multiple subtasks, and system 100 can execute combinations of subtasks. Also, for convenience, individual tasks within a combination of tasks may be called subtasks. Input text tags can identify the subtasks to be executed. A sequence of output tokens can include the respective outputs for each of the subtasks. Thus, in some examples, system 100 can output intermediate steps for complex tasks.

[0058] For example, when the input tag text specifies a single task of inter-speech translation, system 100 can perform inter-speech translation to directly output an audio signal in French from an input audio signal in English. Alternatively, system 100 can output English text, followed by French text, followed by a French audio signal. For example, the input text tag could specify a combination of subtasks: automatic speech recognition, automatic speech translation, and inter-speech translation. System 100 can perform the combination of subtasks with a single call to a language model neural network rather than multiple separate calls to a language neural network. The language model neural network can handle the input and all pre-decoded content at each stage, resulting in improved performance compared to a pipelined approach that performs automatic speech recognition, machine translation, and text-to-speech. In this example, system 100 performs all subtasks using the provided input audio, whereas in a pipelined approach, the input audio is processed only by the automatic speech recognition system. The output of the automatic speech recognition system, i.e., the text transcript of the audio, is sent to a machine translation system that provides a translation of the text transcript. The translated transcript is sent to a text-to-speech system to generate translated audio. However, prosodic and paralinguistic information present in the input audio is not transmitted to the text-to-speech system through a pipelined process. In contrast, in system 100, when generating translated audio, the language model neural network can refer to information within the input audio signal. This allows the translated audio to more accurately reflect the prosody of the input audio, as well as paralinguistic information such as speaker identification.

[0059] In some cases, it may be helpful to specify additional subtasks to generate additional output that can be used to assist in the performance of the desired task. Some subtasks include automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, or text-to-text machine translation. In some embodiments, any task may be a subtask.

[0060] For example, to assist in a speech-to-speech translation task, additional tasks may include the automatic speech recognition and automatic speech translation described above. System 100 can perform automatic speech recognition on the input audio to generate a transcript of the input audio. System 100 can perform automatic speech translation to generate a translated text transcript from the input audio, and to do so, it can work with the generated transcript of the input audio. System 100 can also perform a desired speech-to-speech translation task, and to do so, it can work with both the generated transcript of the input audio and the generated translated transcript. As described above, the system performs each subtask in a single call. For example, if autoregressive decoding is used, system 100 can successively generate each output token one at a time for a first specified subtask, then a second specified subtask, and so on. In this way, earlier generated outputs of earlier subtasks can be taken into account in order to generate the output of subsequent subtasks.

[0061] As another example, to accomplish the task of speech-to-speech translation, the subtasks may include automatic speech recognition and speech-to-speech translation. System 100 can perform automatic speech recognition on the input audio to generate a transcript of the input audio. System 100 can also perform a desired speech-to-speech translation task and to do so, can work with the generated transcript of the input audio.

[0062] As another example, to accomplish a speech-to-speech translation task, the subtasks may include automatic speech translation and speech-to-speech translation. System 100 can perform automatic speech translation to generate a translated text transcript from the input. System 100 can also perform a desired speech-to-speech translation task and, to do so, can handle the generated translated transcript.

[0063] As another example, to accomplish the task of automatic speech translation, the subtasks may include automatic speech recognition and automatic speech translation. System 100 can perform automatic speech recognition on the input audio to generate a transcript of the input audio. System 100 can perform the desired automatic speech translation task to generate a translated text transcript from the input audio, and to do so, it can correspond to the generated transcript of the input audio.

[0064] As another example, to accomplish a speech-to-speech translation task, the subtasks may include automatic speech recognition, text-to-text machine translation, and text-to-speech. To accomplish a speech-to-speech translation task, the subtasks may include automatic speech recognition and text-to-text machine translation. To accomplish a speech-to-speech translation task, the subtasks may include automatic speech recognition and text-to-text machine translation. To accomplish a speech-to-speech translation task from an input language to a target language, the subtasks may include automatic speech translation from an input language to another language and text-to-text machine translation from another language to a target language. To accomplish a text-to-text machine translation task, the subtasks may include text-to-speech, speech-to-speech translation, and automatic speech translation. To accomplish a speech-to-text machine translation task, the subtasks may include text-to-speech, speech-to-speech translation, and automatic speech recognition.

[0065] In some examples, system 100 provides output 110 for presentation to the user. For example, system 100 can provide data representing an audio signal 110a to the user device, triggering playback of the audio signal 110a. In another example, system 100 can provide data representing text 110b for display on the user device.

[0066] In some examples, output 110 may include text, audio, or both representing one or more commands for controlling an agent interacting with the environment. For example, the commands may include natural language commands. For instance, system 100 may receive an audio signal as input containing utterances representing commands for controlling an agent in a source language. System 100 may perform tasks such as speech-to-speech translation to produce an output containing audio representing commands for controlling an agent in a target language. System 100 may also perform tasks such as automatic speech translation to produce an output containing a transcript of the commands in the target language. System 100 may also perform tasks such as automatic speech recognition to produce an output containing a transcript of the commands in the source language.

[0067] As another example, system 100 can receive text containing instructions for controlling an agent in a source language. System 100 can perform tasks such as machine translation to generate output containing text containing instructions in a target language. System 100 can also perform tasks such as text-to-speech to generate output containing utterances representing the instructions.

[0068] In particular, natural language instructions can define, or otherwise specify, a high-level plan that includes a sequence of two or more actions to be performed by the agent. Generally, a high-level plan defines the actions to be performed by the agent and the sequential order in which those actions should be performed. Each of these actions may itself contain multiple lower-level actions that the agent can perform.

[0069] In some embodiments, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent may be a robot that interacts with the environment to achieve a goal, for example, to position a target object within the environment, to move the target object to a specified location within the environment, to physically manipulate the target object within the environment in a specified manner, or to navigate to a specified destination within the environment. Alternatively, the agent may be an autonomous or semi-autonomous land, air, or water vehicle that navigates through the environment to a specified destination within the environment.

[0070] The action may be a control input for controlling a robot, such as torque to the robot's joints or a higher level of control command, or a control input for controlling an autonomous or semi-autonomous land, air, or water vehicle, such as torque to the vehicle's control surface or other control elements or a higher level of control command.

[0071] In other words, an action may include, for example, position, velocity, and / or force / torque / acceleration data for one or more joints or other mechanical agent parts of a robot. Additionally or alternatively, an action may include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment whose control affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or water vehicle, an action may include actions that control navigation, such as steering, and actions that control movement, such as braking and / or accelerating the vehicle.

[0072] In some embodiments, the environment is a simulated environment, and the agent runs as one or more computer programs that interact with the simulated environment. For example, the environment could be a computer simulation of a real-world environment, and the agent could be a simulated machine agent that navigates the computer simulation.

[0073] For example, the simulated environment could be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent could be a simulated vehicle navigating the motion simulation. In these embodiments, the action could be a control input for controlling a simulated user or a simulated vehicle. In another example, the simulated environment could be a computer simulation of a real-world environment, and the agent could be a simulated robot interacting with the computer simulation.

[0074] In some embodiments, the environment is a suitable execution environment, e.g., a runtime environment or operating system environment, running on one or more computing devices such as smartphones, tablet computers, wearable devices, automotive systems, or standalone personal assistant devices, and the agent is a virtual agent (also known as an "automation assistant" or "mobile assistant") that can be interacted with by a user through the computing device. The virtual agent can receive input from the user (e.g., typed or spoken natural language input) and respond with response content (e.g., visual and / or audible natural language output). The virtual agent can provide a wide range of functionality through interaction with various local and / or third-party applications, websites, or other agents. In these embodiments, the action may include any activity or behavior that can be performed or initiated by a user on the computing device, e.g., within application software installed on the computing device.

[0075] The components of system 100 can be trained by a training system implemented as one or more computer programs on one or more computers located in one or more locations. In some embodiments, the components of system 100 can be trained independently by the training system. In some embodiments, some components of system 100 can be pre-trained. The training of system 100 is described in more detail below with reference to Figure 4.

[0076] Figure 2 is a block diagram of an exemplary system 100, which is described with reference to Figure 1. In particular, system 100 uses a text tokenizer 210, an audio tokenizer 220, a language model neural network 250, a text detokenizer 260, and an audio detokenizer 270 to generate text, audio, or both, given text, audio, or both inputs. Figure 2 shows an exemplary task of interspeech translation from French to English, and an automatic speech recognition task with Italian utterances.

[0077] The system generates a sequence of input tokens 104 by tokenizing the input text tag 206 and the data 208 for performing the task. In the example in Figure 2, system 100 generates a sequence of input tokens 104a for a French-to-English speech translation task. System 100 generates a sequence of input tokens 104b for an automatic speech recognition task in Italian.

[0078] For example, system 100 receives an input text tag 206 that describes a task to be performed. The system can process the input text tag 206 to generate a text token that describes the task to be performed.

[0079] The system can generate one or more text tokens corresponding to the input text tag 206 using the text tokenizer 210, which will be described in more detail below. The system includes the text tokens corresponding to the input text tag 206 within the sequence of input tokens 104.

[0080] For example, an input text tag can include the target language of the output text or audio. Specifically, the tag can specify the English name of the input language and, optionally, the output language if the input language is different. In the example in Figure 2, the input text tag 206a contains "[S2ST French English]" which describes the task type, the task's source language, and the task's target language. For example, "S2ST" specifies the task type, i.e., speech-to-speech translation. The source language is French, and the target language is English. Therefore, system 100 can include a text token in the sequence of input tokens 104a that corresponds to the input text tag 206a.

[0081] The input text tag 206b contains "[ASR Italian]", which describes the task type and source language. For example, "ASR" specifies the task type, automatic speech recognition. The source language is Italian. Therefore, the system 100 can include a text token in the sequence of input tokens 104b that corresponds to the input text tag 206b.

[0082] As another example, to query the system and perform ASR on a word in French, the input text tag can include "[ASR French]". To perform TTS in English, the input text tag can include "[TTS English]". To perform S2ST from English to French, the input text tag can include "[S2ST English French]".

[0083] In some examples, the input text tag can include a natural language description of the task to be performed. For example, to perform automatic speech recognition on French utterances, the input text tag could include "Transcribe the following French audio."

[0084] In some examples, tags do not specify the name of the input or output language. For example, to perform automatic speech recognition on French utterances, the input text tag could include "ASR" or "Transcribe Audio".

[0085] In examples with multiple subtasks, a tag can specify multiple subtasks. For example, to perform English-to-French speech translation by performing automatic speech recognition on English audio to generate English text, performing automatic speech translation on English audio to generate French text, and performing speech-to-speech translation on English audio to generate French audio, the input text tag could contain "[ASR AST S2STEnglishFrench]".

[0086] System 100 also receives one or more sequences of data 208 to perform a task. System 100 may include tokens in the sequence of input tokens 104 that correspond to the sequences of data 208. The sequences of data 208 may include, for example, text input or input audio signals.

[0087] In the example in Figure 2, the input audio signal 208a represents a French utterance. Therefore, system 100 can tokenize the input audio signal 208a into audio tokens and include the audio tokens corresponding to the input audio signal 208a within the sequence of input tokens 104a.

[0088] The input audio signal 208b represents an utterance in Italian. Therefore, system 100 can tokenize the input audio signal 208b into audio tokens and include the audio tokens corresponding to the input audio signal 208b within the sequence of input tokens 104b.

[0089] Tokenization refers to the process of mapping a sequence of inputs to tokens selected from a fixed vocabulary of tokens. For example, the token vocabulary may be a system-maintained vocabulary. The token vocabulary can include text tokens representing text and audio tokens representing audio.

[0090] For example, to generate a sequence of input tokens for text input, system 100 can apply the text tokenizer 210 to the text to generate a sequence of text tokens. System 100 can include a sequence of text tokens in the sequence of input tokens.

[0091] The text tokenizer 210 is configured to tokenize text. For example, the system can tokenize text input and input text tags 206 that describe a task. The token vocabulary can include any of the various tokens that represent text symbols. For example, the token vocabulary can represent one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a corpus of natural language text. The SentencePiece tokenizer, described in detail by Kudo, T. and Richardson, J., in "SentencePiece: A simple and language-independent subword tokenizer and detokenizer for neural text processing," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018a, Association for Computational Linguistics, doi:10.18653 / v1 / D18-2012, is an example of a suitable tokenizer in this case.

[0092] To generate a sequence of input tokens for an input audio signal, system 100 can apply the audio tokenizer 220 to the audio signal to generate a sequence of audio tokens. System 100 can include a sequence of audio tokens in the sequence of input tokens.

[0093] In some examples, the audio tokenizer 220 is configured to generate a semantic representation of an input audio signal 208a that includes audio tokens. The audio tokenizer 220 can generate a semantic representation of an audio signal that specifies each audio signal at each of several first time steps spanning the audio signal, and each audio token is selected from the audio token vocabulary and represents the semantic content of the audio signal at the corresponding first time step. Examples of semantic content that can be represented by tokens include the linguistic content of an utterance, as well as the melody and rhythm of music.

[0094] The audio tokenizer 220 can generate semantic representations of audio signals as described in AudioLM: a Language Modeling Approach to Audio Generation.arXiv pre-review manuscript arXiV:2209.03143,2022. For example, the audio tokenizer 220 can generate semantic representations by processing the audio signal using an audio representation neural network trained to generate representations of input audio and thus generate semantic representations of audio signals. That is, the audio tokenizer 220 can process the audio signal using the audio representation neural network to generate an encoded vector for each of the first time steps. For each of the first time steps, the audio tokenizer 220 can select the audio token that is closest to the encoded vector generated by the audio representation neural network for the first time step as the audio token for the first time step.

[0095] The target semantic representation for training an audio representation neural network may be generated by clustering the outputs of the intermediate layers of a self-attention-based model, such as a transformer-based or conformer-based model, and using the centroids of the clusters as semantic tokens. For example, a system or a training system for a system can generate a sequence of high-density embeddings from an embedding model. For example, the embedding model may be a w2v-BERT model. The system can generate high-density embeddings for all of the training data. The system can normalize the embeddings by subtracting the mean and scaling them to have unit variance. The system can perform k-means clustering on the normalized embeddings. The system can then compute tokens by assigning the normalized high-density embeddings to the identifiers of the nearest neighbor cluster centers.

[0096] In some embodiments, the system does not normalize the embeddings before performing k-means clustering, for example, to improve performance with multilingual data.

[0097] In some embodiments, the w2v-BERT model can be trained on multilingual data.

[0098] In some embodiments, the system can use a multilingual speech encoder, such as a Universal Speech Model (USM) encoder, as the embedding model. Given raw audio input, the USM encoder generates a sequence of integers of a length proportional to the length of the audio. For example, the system can extract the embeddings from the intermediate layers (or any other suitable layers or combination of layers) of the USM encoder and quantize the embeddings to compute tokens. For example, the quantizer may be a random projection quantizer, as described in Zhang et al, Google USM: Scaling automatic speech recognition beyond 100 languages.arXiv pre-review manuscript arXiv:2303.01037,2023.

[0099] Therefore, the input token sequence 104 includes tokens that describe the task to be performed and the data for performing that task. Each token in the input token sequence 104 is selected from a vocabulary of tokens, which includes text tokens and audio tokens.

[0100] The system processes the input token sequence 104 to generate a sequence of embeddings. The system can generate a sequence of embeddings by embedding each token in the input token sequence 104 into the embedding space. For example, system 100 can maintain the respective embedding for each token in the token vocabulary. For each token in the input token sequence 104, the system can map the token to the respective embedding for that token.

[0101] For example, the language model neural network 250 may be a decoder-only transformer containing a token embedding matrix E with learned values. Generally, the token embedding matrix E maps each token, represented, for example, as an integer or one-hot coded vector, to a corresponding high-density embedding. For example, given a vocabulary of t tokens and embeddings of size m, E is a t × m matrix, where its i-th row gives the embedding for the i-th token. The language model neural network 250 also contains another embedding matrix E' within the final softmax layer, which is used to compute logits across all tokens at each position. For example, the embedding matrix E' is an m × t matrix, which is multiplied by the m-dimensional output of the final attention layer of the language model neural network 250 to obtain a t-dimensional vector of logits, one for each token. In some examples, the embedding matrix E' has a shared variable with E and is the transpose of E.

[0102] To modify the language model neural network 250 to model audio, the system can use an augmented embedding matrix. For example, the augmented embedding matrix can include mappings of text tokens and audio tokens. In the example in Figure 2, the augmented embedding matrix is ​​a (t+a)×m matrix, where a is the number of audio tokens. Thus, the augmented embedding matrix maps a vocabulary of t text tokens and a audio tokens to their respective embeddings in the embedding space. The system can then expand the token vocabulary of the existing decoder and use the new tokens to represent tokenized audio.

[0103] At least some of the values ​​may be learned in collaboration with the language model neural network 250. Embeddings of any text tokens in the vocabulary can be learned during the text-only pre-training of the language model neural network. For example, the system can obtain a vocabulary embedding matrix of t text tokens from the pre-trained language model neural network 250.

[0104] Embeddings for any text token within a vocabulary can be learned during audio-text pre-training of a language model neural network. The system can learn audio token embeddings by expanding the embedding matrix by adding a new row, and then fine-tuning the expanded embedding matrix. For example, the first t embeddings may correspond to a vocabulary of t text tokens, e.g., SentencePiece text tokens. The next a embeddings from t to t+a can represent audio tokens.

[0105] Text embeddings can be reused from a pre-trained language model neural network 250, while the training system of system 100 or other systems can randomly initialize and then train the audio embeddings. The system can train the audio embeddings using mixed utterance and text tasks, as described below with reference to Figure 4. In some examples, the system can train the audio embeddings and then further train, for example, fine-tune, the text embeddings. In some examples, the text token embeddings can be fixed and retained during audio-text training of the language model neural network.

[0106] The system uses a language model neural network 250 to process a sequence of embeddings and generate a sequence of output tokens 252 for the task. The sequence of output tokens 252 can include text tokens, audio tokens, or both. By extending the embedding matrix to include text and audio embeddings, the language model neural network 250 can model both text and audio.

[0107] In the example in Figure 2, the language model neural network 250 processes the input token sequence 104a to generate the output token sequence 252a, which includes audio tokens. The language model neural network 250 then processes the input token sequence 104b to generate the output token sequence 252b, which includes text tokens.

[0108] For example, the language model neural network 250 may be a modified pre-trained text decoder with an extended embedding matrix as described above. As an example, the text decoder, and therefore the language model neural network 250, may have a transformer-based architecture dedicated to the decoder, where a sequence of input tokens 104a is provided to the language model neural network 250 as a "prompt".

[0109] Generally, transformer-based architectures are sometimes characterized by having a series of self-aware neural network layers. Each self-aware neural network layer has an attention layer input for each element of the input, and is configured to apply an attention mechanism to the attention layer input to generate an attention layer output for each element of the input. Many different attention mechanisms can be used.

[0110] As a concrete example, the language model neural network 250 can autoregressively generate a sequence of output tokens 252. For example, the language model neural network 250 can generate a sequence of output tokens 252 by generating each specific output token in sequence 252, given the current input sequence which includes all output tokens preceding a particular output token in sequence 252, that is, output tokens already generated for all previous positions in sequence 252 that precede a particular position of a particular output token.

[0111] More specifically, to generate a specific output token, the language model neural network 250 can process the current input sequence to generate a score distribution, e.g., a probability distribution, which assigns a score, e.g., a probability, to each token in the token vocabulary. The language model neural network 250 can then use the score distribution to select a token from the vocabulary as a specific token. For example, the language model neural network 250 can greedily select the token with the highest score, or it can sample tokens from the distribution using, for example, top-k sampling, nuclear sampling, or other sampling techniques.

[0112] As a specific example, the language model neural network 250 may be an autoregressive transformer-based neural network containing multiple layers, each applying self-attentional behavior. The language model neural network 250 can have any of the various transformer-based neural network architectures. Examples of such architectures include J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdLCasas, LAHendricks, J. Welbl, A. Clark, et.al., Training compute-optimal large language models, arXiv pre-peer review manuscript arXiv:2203.15556,2022, JWRae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HFSong, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche,LAHendricks,M.Rauh,P.Huang,A.Glaese,J.Welbl,S.Dathathri,S.Huang,J.Uesato,J.Mellor,I.Hig gins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskay, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, XLLi, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mes ch, J. Lespiau, M. Tsimpoukelli, N. Grigoev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A.Guy, C. Jones, J. Bradbury, M. Johnson, BAHechtman, L. Weidinger, I. Gabriel, WSIsaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving, Scaling Google Scholar Crossref , CAS 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu Unified Text-to-Text Transformer,arXiv Retrieved from "arXiv:1910.10683,2019" Lu, and Quoc V.Le, Towards a Human-Like Open-Domain Chatbot, CoRR, abs / 2001.09977, Shyam, Girish Sastry, Amanda Askell, etc. al.,Language models are few-shot learners.,arXiv Proceedings of ArXiv:2005.14165,2020.

[0113] In some examples, the language modeling neural network 250 may include an attention neural network containing one or more parallel attention layers. Each parallel attention layer contains an attention sublayer positioned in parallel with a feedforward sublayer. Examples of the architecture of the language modeling neural network 250 are described in Chery, A., et al. Palm: Scaling language modeling with pathways.arXiv pre-review manuscript arXiv:2204.02311,2022, and Anil, R., et al. Palm 2 Technical Report.arXiv pre-review manuscript arXiv:2305.10403,2023.

[0114] The system converts the sequence of output tokens 252, output by the language model neural network 250, into an output 282 that satisfies the task. For example, the system can detoxify the sequence of output tokens 252 to generate an output 282 that satisfies the task. The sequence of output tokens 252 can contain text tokens, audio tokens, or both. The output may be an audio signal, text, or both.

[0115] In the example in Figure 2, system 100 detoxifies the sequence of output tokens 252a to produce output 282a. Output 282a is an audio signal containing English utterances that satisfy the task of French-to-English speech translation, as specified by the input text tag 206a.

[0116] In the example in Figure 2, system 100 detoxifies the sequence of output tokens 252b to produce output 282b. Output 282b is text that satisfies the task of automatic speech recognition from Italian utterances, as specified by the input text tag 206b.

[0117] System 100 can process text tokens and generate text predictions. The system can generate text predictions from text tokens using a text detokenizer 260. The system can convert text tokens to text using, for example, SentencePiece.

[0118] System 100 can generate audio from audio tokens using the audio detokenizer 270. In some examples, the system can detokenize audio tokens by using one or more neural networks to generate an acoustic representation of the audio signal, provided that at least the audio tokens are present. The system can use a decoder neural network to process at least the acoustic representation to generate a prediction of the audio signal.

[0119] An acoustic representation can specify one or more sets of acoustic tokens for each of several second time steps spanning an audio signal. Each of the one or more acoustic tokens for each second time step can represent the acoustic characteristics of the audio signal in the corresponding second time step. Each acoustic token may be a token of a neural audio codec, such as the SoundStream neural audio codec. Acoustic characteristics capture the details of the audio waveform, enabling high-quality synthesis. Acoustic characteristics can include recording conditions such as reverberation level, distortion, and background noise.

[0120] In some examples, the system can synthesize audio signals from audio tokens using autoregressive or non-autoregressive decoding, as will be further described below with reference to Figure 3. For example, the system can generate acoustic tokens from audio tokens autoregressively or non-autoregressively.

[0121] In some embodiments, the prediction of an audio signal may be conditional on a speech context that represents or describes the desired speech of the audio signal. The speech context may include a reference input audio signal of the desired speech. In some examples, the input audio signal of the desired speech is converted into, or received as, a semantic representation, such as audio tokens at each of a plurality of first time steps spanning at least a portion of the audio signal of the desired speech. Each audio token is selected from an audio token vocabulary and represents the semantic content of the audio signal at the corresponding first time step. In some examples, the speech context may include an acoustic representation of the input audio signal of the desired speech. For example, the acoustic representation may include one or more acoustic tokens at each of a plurality of second time steps spanning the audio signal. Each of the one or more acoustic tokens at each second time step may represent the acoustic characteristics of the audio signal at the corresponding second time step.

[0122] In some embodiments, where the prediction of an audio signal is conditional on an audio context describing the desired sound of the audio signal, the audio tokenizer 220 can generate audio tokens for a sequence of output tokens 252, conditional on the audio tokens of the audio context. In some embodiments, the system can generate an acoustic representation, conditional on the audio tokens for a sequence of output tokens 252 and the acoustic tokens of the audio context.

[0123] In some examples, the input audio signal for the desired speech may be a sample from a sequence of data 208, which are audio signals representing utterances, such as input audio signals 208a and 208b. The system can generate a speech context from the samples. For example, the system can use the audio tokenizer 220 to generate audio tokens for the speech context.

[0124] After generating an acoustic representation, for example, autoregressively or non-autoregressively, the system uses a decoder neural network to process at least the acoustic representation to generate predictions of the audio signal. For example, each audio sample in each of multiple output time steps spanning a time window may be based on one or more acoustic tokens of the acoustic representation.

[0125] In some embodiments, the decoder neural network may be the decoder neural network of a neural audio codec. For example, the decoder neural network may be a convolutional decoder. The neural audio codec may be, for example, the SoundStream neural audio codec.

[0126] A neural audio codec may include a decoder neural network and an encoder neural network. For example, an encoder neural network can convert audio into a coded signal quantized into an acoustic representation. A decoder neural network can convert the acoustic representation into a predicted audio signal.

[0127] In some embodiments where the prediction of an audio signal is conditional on an audio context that describes the desired speech of the audio signal, the system can use a decoder neural network to process the acoustic representation of the audio signal and the acoustic representation of the audio context to generate a prediction of the audio signal that is conditional on the audio context.

[0128] Figure 3 is a block diagram of an exemplary process 300 for performing tasks that require generating text, generating audio, or both. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, system 100, e.g., in Figures 1 and 2, as appropriately programmed according to this specification, can perform process 300.

[0129] The system obtains a sequence of input tokens (step 302). The sequence of input tokens includes tokens that describe the task to be performed and the data required to perform that task.

[0130] In some examples, the system generates a sequence of input tokens by receiving input text tags that describe the tasks to be performed and the data required to perform those tasks. The data required to perform the tasks can include input text, input audio, or both. The system generates input tokens by applying tokenization. For example, the system can use a text tokenizer to tokenize input text tags and input text into text tokens, and an audio tokenizer to tokenize input audio into audio tokens.

[0131] The system generates a sequence of embeddings (step 304). For example, the system can embed each token in the sequence of input tokens within the embedding space. The system can generate a sequence of embeddings using an embedding matrix, as described above with reference to Figure 2.

[0132] The system processes the sequence of embeddings using a language model neural network to generate a sequence of output tokens for the task (step 306). The sequence of output tokens may include text tokens, audio tokens, or both.

[0133] The system can detoxify a sequence of output tokens to produce an output that satisfies the task. For example, the system can use a text detoxifier to process text tokens and generate a text prediction. The system can use an audio detoxifier to process audio tokens and generate an audio signal prediction. The audio detoxifier can generate acoustic tokens based on audio tokens, process the acoustic tokens, and generate an audio signal prediction. An example of acoustic token generation is described in more detail below.

[0134] For example, a system can autoregressively generate acoustic tokens from an input sequence of audio tokens. For instance, a system can generate acoustic tokens for an acoustic representation using coarse and fine generative neural networks. These networks can be trained to predict the generated acoustic representation based on the output of an encoder neural network by processing an audio signal.

[0135] For example, an encoder neural network might be a convolutional encoder that maps an audio signal to a sequence of embeddings. Each embedding at each of several second time steps can correspond to a feature of the audio signal at each second time step. A ground-truth acoustic representation of the audio signal can be generated by applying quantization to each of the embeddings. For example, an encoder neural network might be part of a neural audio codec, such as the SoundStream neural audio codec. For example, the quantization might be residual vector quantization that encodes each embedding using a hierarchy of multiple vector quantizers, each generating its respective acoustic token from a corresponding vocabulary of acoustic tokens of vector quantizers.

[0136] Each set of one or more acoustic tokens at each of the multiple second time steps contains multiple acoustic tokens that collectively represent the prediction of the output of the residual vector quantization applied to the embedding representing the acoustic properties of the audio signal at the second time step. The residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each generating its respective acoustic token from the corresponding vocabulary of the vector quantizer's acoustic tokens. The hierarchy contains one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy. Thus, each set of acoustic tokens at each second time step contains, for each vector quantizer, the respective acoustic tokens selected from the vector quantizer's vocabulary.

[0137] For example, the hierarchy could include coarse vector quantizers and fine vector quantizers.

[0138] To generate acoustic representations, a coarse generative neural network can generate acoustic tokens for a coarse vector quantizer, provided at least a semantic representation exists. For example, for each of one or more coarse vector quantizers in a hierarchy, the coarse generative neural network can generate acoustic tokens for each of the second time steps of the vector quantizer, provided at least a semantic representation exists. The acoustic tokens of the coarse vector quantizer can be used to materialize utterances, albeit at a low bitrate. In some embodiments where the prediction of an audio signal is conditional on a speech context describing the desired speech of the audio signal, the coarse generative neural network can generate acoustic tokens, provided that the semantic representation and the acoustic tokens of the speech context exist.

[0139] A coarse generative neural network may be an autoregressive neural network configured to autoregressively generate acoustic tokens of a coarse vector quantizer according to a first generation order. In some embodiments, the coarse generative neural network has a transformer architecture dedicated to the decoder. In some embodiments, the coarse generative neural network has an encoder-decoder transformer architecture.

[0140] To generate acoustic representations, a fine-grained generative neural network can generate acoustic tokens for a fine-grained vector quantizer, conditionally on the acoustic tokens of at least a coarse vector quantizer. For example, for each fine-grained vector quantizer in the hierarchy, a fine-grained generative neural network can generate acoustic tokens for each second time step of a vector quantizer, conditionally on the acoustic tokens for each second time step of one or more coarse vector quantizers in the hierarchy. The acoustic tokens of a fine-grained vector quantizer can be used to materialize utterances at a higher bitrate than the acoustic tokens of a coarse vector quantizer, thereby improving the quality of the materialized utterances.

[0141] Fine-grained generative neural networks may be autoregressive neural networks configured to autoregressively generate acoustic tokens according to a second generation order. In some embodiments, fine-grained generative neural networks have a transformer architecture dedicated to the decoder. In some embodiments, fine-grained generative neural networks have an encoder-decoder transformer architecture. Further details are described in Borsos et al., AudioLM: a Language Modeling Approach to Audio Generation. arXiv pre-review manuscript arXiV:2209.03143, 2022.

[0142] In some other examples, a system can generate audio tokens from a sequence of audio tokens using non-autoregressive decoding. A system can generate a sequence of acoustic tokens from a sequence of tokens over multiple iterations using a generative neural network. For example, a system could include a bidirectional attention-based conformer model trained to predict masked acoustic tokens given a conditioned signal, such as audio tokens in a sequence of output tokens. In some embodiments, where the prediction of the audio signal is conditioned on an audio context describing the desired speech of the audio signal, the conditioned signal could also include audio tokens of the audio context. In some embodiments, the conditioned signal could include acoustic tokens representing the audio context.

[0143] Before the first iteration, the system can generate a sequence of tokens from a sequence of output tokens. The sequence of tokens contains each token at each of several positions within the sequence. Positions typically correspond to time steps spanning a specified time window of the output audio signal. Positions can be divided into multiple frames (or segments), each of which can contain a fixed number of positions.

[0144] Each token can contain an audio token from a sequence of output tokens, or a masked token. In some examples, each token can contain an audio token or an acoustic token from a speech context. A "masked token" is a token that contains a given number and means that the corresponding token in the sequence of tokens has not yet been generated, for example, has not yet been selected from a given set of tokens.

[0145] The system generates a sequence of output tokens by gradually unmasking all the masked tokens that were initially included in the sequence of tokens. During each iteration, the system performs a forward pass through the generative neural network, that is, it uses the generative neural network to process the network input according to its parameters to generate an updated sequence of tokens. In the first iteration, the network input contains the sequence of input tokens. For each subsequent iteration, the network input contains the updated sequence of tokens generated in the previous iteration.

[0146] Next, in each iteration, the system uses a generative neural network to process the network input and generate one or more new tokens that replace each masked token in the sequence of tokens. That is, in each iteration, the generative neural network is used to generate an updated sequence of tokens with fewer masked tokens.

[0147] A generative neural network is configured to generate a sequence of embeddings for each token in the network input. The generative neural network can sum up embeddings corresponding to the same frame containing conditioned tokens (or multiple) i.e., audio tokens (or multiple) and provide a sequence of embeddings as input to a conformer model configured to apply bidirectional self-attention using one or more attention blocks.

[0148] Each attention block in the conformer model includes one or more convolutional layers and one or more attention layers. Therefore, each attention block may also be called a “convolutional-extended attention block.” Each attention block updates the contiguous embeddings by processing the contiguous embeddings, or data derived from contiguous embeddings, and applying both convolutional and attentional operations to generate updated contiguous embeddings.

[0149] The system can identify a subset of sequences of tokens that are eligible to be unmasked in the current iteration. For example, each token in a sequence of tokens can be associated with its respective vector quantizer at a specific level / position in a sequence of vector quantizers. The system can be configured to unmask tokens in the input sequence level by level, starting from the first level in the sequence of vector quantizers. Thus, the system can identify a subset of sequences of input tokens that are eligible to be unmasked in the current iteration as any masked tokens in the input sequence associated with the level being unmasked in the current iteration.

[0150] Specifically, when different positions in the sequence of output tokens are associated with different vector quantizers in the sequence of vector quantizers 1...Q',(Q'+1)...Q contained in a neural audio codec arranged in a hierarchical order (e.g., the Soundstream neural audio codec), for each position occupied by a masked token in the sequence of tokens, the system can decide whether to select a position to unmask based on the residual vector quantizer associated with that position.

[0151] For example, a hierarchical order can range from coarse to fine. That is, a hierarchy can include one or more coarse vector quantizers at one or more initial levels within the hierarchy, and one or more fine vector quantizers at one or more final levels within the hierarchy.

[0152] The generative neural network processes the updated sequential embeddings for each frame to generate predictions characterizing each of the tokens selected to be unmasked. It can then generate an updated sequence of tokens for a given iteration by replacing one or more masked tokens in the identified subset with unmasked tokens—that is, by including unmasked tokens in place of masked tokens in the sequence of output tokens. Thus, at the end of a given iteration, the generative neural network can generate an updated sequence of tokens.

[0153] After the last iteration, the system uses the updated sequence of tokens generated in the last iteration as the sequence of acoustic tokens. Further details can be found in Borsos et al., SoundStorm: Efficient Parallel Audio Generation.arXiv pre-review manuscript arXiv:2305.09636,2023.

[0154] Throughout this specification, “embedding” refers to an ordered set of numbers (e.g., integers or floating-point numbers), such as a vector of numbers, a matrix, or other tensor.

[0155] Throughout this specification, “residual vector quantizer” (RVQ) may refer to a multi-stage vector quantization technique based on a sequence of (residual) vector quantizers. A vector quantizer can quantize an input vector, for example, by identifying a code vector from a codebook of code vectors associated with a vector quantizer having the smallest distance from the input vector, for example, according to a distance criterion (for example, based on an L1 criterion). A residual vector quantizer can quantize an input vector (or “signal”) by iteratively quantizing the residual error from previous quantization stages. Thus, each stage of a residual vector quantizer encodes the difference (or residual) between the original signal and the signal reconstructed from the previous stage, thereby progressively refining the approximation of the original signal at each step.

[0156] Figure 4 is a flowchart of an exemplary process 400 for training a system to perform tasks that require generating text, generating audio, or both. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a training system for a system to perform tasks, such as system 100 in Figures 1-2, or other training systems, preferably programmed according to this specification, can perform process 400.

[0157] The system obtains data specifying one or more pre-trained components of the system (step 402). For example, the system may obtain data specifying a pre-trained text tokenizer, a pre-trained audio tokenizer, a pre-trained language model neural network, a pre-trained text detoxifier, and a pre-trained audio detoxifier.

[0158] For example, a language modeling neural network may be pre-trained on language modeling tasks, such as a task that requires predicting the text token following the current sequence in the training data, given the current sequence of text tokens. Specifically, a language modeling neural network can be pre-trained with a maximum likelihood objective function on a large dataset of text, such as text publicly available from the internet or other text corpora.

[0159] Several exemplary pre-trained language model neural networks are described in Chowdhery, A., et.al. Palm: Scaling language modeling with pathways.arXiv pre-review manuscript arXiv:2204.02311,2022, and Anil, R., et.al. PaLM 2 Technical Report.arXiv pre-review manuscript arXiv:2305.10403,2023. In these examples, pre-training the language model neural network may also involve learning pre-trained values ​​of the embedding matrix.

[0160] As another example, a text tokenizer may be pre-trained on a text tokenization task, and a text detokenizer may be pre-trained on a text detokenization task. Further details of exemplary text tokenizers and detokenizers are described in Kudo et al., Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.arXiv pre-review manuscript arXiv:1808.06226,2018.

[0161] Audio tokenizers are described, for example, in Zhang et al., Google USM: Scaling automatic speech recognition beyond 100 languages.arXiv pre-review manuscript arXiv:2303.01037,2023, or in Borsos et al., AudioLM: a Language Modeling Approach to Audio Generation.arXiv pre-review manuscript arXiV:2209.03143,2022. Audio detokenizers can include, for example, in Borsos et al., AudioLM: a Language Modeling Approach to Audio Generation.arXiv pre-review manuscript arXiV:2209.03143,2022, or in Borsos et al., Soundstorm: Efficient parallel audio generation.arXiv pre-review manuscript arXiv:2305.09636,2023, and may include a neural network pre-trained to generate acoustic tokens conditionally on audio tokens. An audio detokenizer may also include a decoder neural network pre-trained to generate predictions of audio signals from acoustic tokens, as described, for example, in Zeghidour et al., SoundStream: An end-to-end neural audio codec. IEEE / ACM Transactions on Audio, Speech, and Language Processing 30:495-507, 2021.

[0162] The system obtains a set of training data for one or more tasks (step 404). The set of training data may include multiple training examples, each containing a training input and a ground truth output. Each training input and ground truth output may include one of the following: an utterance in the source language, a transcript of the utterance, a speech translation of the utterance, and a written translation of the utterance. In some examples, the speech translation of the utterance has the phonetic characteristics of the speaker of the utterance in the source language.

[0163] As described above, tasks can include automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, or text-to-text machine translation. For example, in the case of speech-to-speech translation, the training input would include utterances in the source language, and the training output would include a speech translation of those utterances in the target language. In some examples, the training input could include input text tags that specify the task of the training example.

[0164] In some cases, the training data set can include training examples for a specific task. For example, the training data set can include training examples for only the automatic speech translation task.

[0165] In some other examples, the training data set can include training examples for multiple tasks. Training a system on multiple tasks can result in improved system performance compared to training it on a single task. For example, training on a training data set that includes training examples for both automatic speech recognition and automatic speech translation can yield better performance on the automatic speech translation task compared to training on the automatic speech translation task alone, for example, by helping the language model neural network connect audio input to its previous understanding of text. A system can be trained on multiple tasks by representing different tasks with input text tags.

[0166] For example, the training data set could include training examples for automatic speech recognition and automatic speech translation only. In this example, the system is trained to output text.

[0167] As another example, the training data set could include training examples for tasks such as automatic speech recognition, automatic speech translation, text-to-speech, and speech-to-speech translation. In this example, the system is trained to output text and speech.

[0168] In some embodiments, the training data set may include training examples of combined tasks. For example, the training input may include input text tags that identify subtasks of the task. The training output may include the output of each subtask so that the language model neural network is trained to generate a sequence of output tokens containing the output tokens of each subtask.

[0169] In some embodiments, the training input includes a training conditioning input. For example, the training conditioning input may include a speech context. Specifically, the speech context can be represented as audio tokens, acoustic tokens, or both. In some embodiments, the speech context can be derived from the training audio input of the training input. For example, the speech context may include a portion of the training audio input. In some embodiments where the training audio input is shorter than a threshold time length, the speech context may include the training audio input multiple times to reach the threshold time length.

[0170] In some cases, the system can generate training examples. For example, the system can use a pre-trained neural network of a language model to translate audio transcripts into the source language. The system can then use a finely tuned language model neural network to generate utterances from the translated transcripts. For a speech-to-speech translation task, the system can include audio and generated utterances in its training examples.

[0171] The system trains one or more pre-trained components on the training data set (step 406). For example, instead of updating all the parameters of a component, the system updates only some of the parameters of the component, keeping the other parameters fixed during training. As an example, the system can further train the language model neural network (including the embedding matrix) from the pre-trained values, for example, by fine-tuning them. Thus, the system can train the language model neural network on one or more tasks of the training data. In some examples, the parameters of the pre-trained tokenizer and detokenizer can be kept fixed during training of the language model neural network.

[0172] For example, the system can extend the embedding matrix to include audio token embeddings. The system can initialize the audio token embeddings randomly or to zero. The audio token embeddings can be learned by further training the language model neural network on the aforementioned audio-text training dataset. In some examples, the text token embeddings can be fine-tuned during the audio-text training of the language model neural network. In some examples, the text token embeddings can be fixed and held in place during the audio-text training of the language model neural network.

[0173] The system updates the values ​​of the language model neural network using a suitable optimizer, e.g., stochastic gradient descent, RMSprop, Adam optimizer, or Adafactor optimizer, to optimize the objective function, such as a cross-entropy objective function specific to the next token prediction task, using machine learning training techniques, for example. In some examples, the system uses loss masking on the training input.

[0174] The system can then determine one or more updates to the values ​​of the parameters of the language model neural network, based on calculating the gradient of the objective function with respect to the parameters of the language model neural network.

[0175] By combining audio tokenization techniques with language modeling neural networks, the system can combine audio tokens and text into a multimodal set of tokens. This multimodal set of tokens can be used interchangeably as input and output. Thus, language modeling neural networks can model any sequence of audio and text tokens. For example, a language modeling neural network dedicated to text can be used to initialize a language modeling neural network dedicated to decoders, which can be fine-tuned for a mixture of tasks that freely map between utterances and text.

[0176] Figure 5 shows the performance of exemplary systems for performing tasks that require text generation, audio generation, or both. For example, Figure 5 shows a comparison of the quality of audio signals and text produced by several other systems with variations of the AudioPaLM system (corresponding to the systems for performing tasks that require text generation, audio generation, or both described herein). The variations use an AudioPaLM system (AudioPaLM 8B AST) using a pre-trained PaLM language model trained with AST, an AudioPaLM system (AudioPaLM 8B S2ST) using a pre-trained PaLM language model trained with S2ST and TTS, an AudioPaLM system (AudioPaLM-2 8B AST) using a pre-trained PaLM-2 language model trained with AST, and a cascaded system (AudioPaLM-2 8B Cascaded ASR+ Translation) with an AudioPaLM-2 ASR model followed by translation using other AudioPaLM-2 models fine-tuned specifically for text-to-text translation.

[0177] Other systems include the Whisper Large system, mSLAM system, MAESTRO system, USM-M system, and Translatotron system.

[0178] The AST BLEU score represents a measure of the quality of automatic speech translation, with higher scores indicating better performance. Figure 5 shows that the system has the best BLEU score for the automatic speech translation task.

[0179] The S2ST ASR-BLEU score represents a measure of the quality of speech-to-speech translation by comparing the text output generated by providing the system's audio output to an automatic speech recognition model with the ground truth target text; a higher score indicates better performance. Figure 5 shows that the system has the best BLEU score for the automatic speech recognition task.

[0180] The ASR WER score represents a measure of the quality of automatic speech recognition based on word error rate, with lower scores indicating better performance. Figure 5 shows that the system is competitive for the automatic speech recognition task.

[0181] This specification uses the term “configured” in relation to systems and computer program components. A system of one or more computers being configured to perform a particular operation or action means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. A system of one or more computer programs being configured to perform a particular operation or action means that, when executed by a data processing device, the program contains instructions that cause the device to perform that operation or action.

[0182] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, and include structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer program instructions, that is, as one or more modules of computer program instructions encoded in a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable memory board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in artificially generated propagating signals, such as mechanically generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0183] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further than, a special-purpose logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). In addition to hardware, a device may optionally include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code that constitutes one or more of these.

[0184] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one location or distributed across multiple locations and interconnected by a data communication network.

[0185] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0186] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0187] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0188] A computer suitable for running computer programs may be based on a general-purpose microprocessor, a dedicated microprocessor, both, or other types of central processing units. Generally, a central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a central processing unit for executing and running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented or incorporated by dedicated logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, to give some examples, a computer can be embedded in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0189] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0190] Embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a key vector board and pointing device that allows the user to input into the computer, such as a mouse or trackball, in order to provide user interaction. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, such as acoustic input, voice input, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0191] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for handling the general and computationally intensive parts of machine learning training or production, i.e., inference workloads.

[0192] Machine learning models can be executed and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0193] Embodiments of the subject matter described herein may be implemented in a computing system that includes a backend component, for example, a data server, or a middleware component, for example, an application server, or a frontend component, for example, a client computer having a graphical user interface, a web browser, or an app that allows a user to interact with embodiments of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, for example, a communication network. Examples of communication networks include local area networks (LANs), wide area networks (WANs), for example, the Internet.

[0194] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, may be received by the server from the device.

[0195] While this specification includes details of many specific embodiments, these should not be construed as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Specific features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features of the invention described in the context of a single embodiment may also be implemented separately or in any preferred subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover subcombinations or variations of subcombinations.

[0196] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0197] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the desired results can still be obtained by performing the actions described in the claims in a different order. As an example, the process shown in the accompanying diagram does not necessarily require that the actions be performed in a specific or sequential order shown in order to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. Obtaining a sequence of input tokens, each token being selected from a vocabulary of tokens including text tokens and audio tokens, wherein the sequence of input tokens includes tokens describing a task to be performed and data for performing the task. The embedding sequence is generated by embedding each token in the sequence of input tokens into the embedding space, To generate a sequence of output tokens for the task, a language model neural network is used to process the sequence of embeddings, wherein each token is selected from the vocabulary. Methods that are performed on a computer, including [specific examples].

2. Generating an embedding sequence by embedding each token in the sequence of input tokens into the embedding space is, Maintaining the respective embedding of each token within the vocabulary of the aforementioned tokens, For each token in the sequence of input tokens, the token is mapped to the respective embedding of the token. The method according to claim 1, including the method described in claim 1.

3. The method according to any one of claims 1 to 2, wherein processing the sequence of embeddings using a language model neural network to generate a sequence of output tokens for the task comprises processing the sequence of embeddings using a language model neural network to autoregressively generate a sequence of output tokens for the task.

4. The method according to claim 3, wherein the embeddings of any text tokens in the vocabulary are learned during pre-training of the language model neural network specifically for text, the embeddings of any audio tokens in the vocabulary are learned during audio-text training of the language model neural network, and the embeddings of text tokens are fixed and held during the audio-text training of the language model neural network.

5. Obtaining a sequence of input tokens is possible. Receiving an input text tag that describes the task to be executed, Receiving one or more sequences of data for performing the aforementioned task, The process involves generating a sequence of input tokens from the aforementioned input text tags and the sequence of one or more data, The method according to any one of claims 1 to 4, including the method described in any one of claims 1 to 4.

6. The sequence of one or more data includes text, and generating a sequence of input tokens is: To generate a sequence of text tokens, a text tokenizer is applied to the text, The sequence of input tokens includes the sequence of text tokens, The method according to claim 5, including the method described in claim 5.

7. The sequence of one or more data includes an audio signal, and generating a sequence of input tokens is: To generate a sequence of audio tokens, an audio tokenizer is applied to the audio signal, The sequence of input tokens includes the sequence of audio tokens, The method according to any one of claims 5 to 6, including the method described in any one of claims 5 to 6.

8. Applying an audio tokenizer to the audio signal means This includes generating a semantic representation of the audio signal that specifies each audio signal in each of a plurality of first time steps spanning the audio signal, wherein each audio token is selected from an audio token vocabulary and represents the semantic content of the audio signal in the corresponding first time step. The method according to claim 7.

9. To generate a semantic representation of the aforementioned audio signal, To generate the semantic representation of the audio signal, the audio signal is processed using an audio representation neural network trained to generate a representation of the input audio. The method according to claim 8, including the method described in claim 8.

10. To generate the semantic representation of the audio signal, processing the audio signal using an audio representation neural network trained to generate a representation of the input audio is: To process the audio signal using the audio representation neural network in order to generate the respective encoded vectors for each of the first time steps, For each first time step, the audio token for the first time step is selected to be the one that is closest to the encoded vector generated by the audio representation neural network for the first time step, The method according to claim 9, including the method described in claim 9.

11. Generating a sequence of input tokens is To generate a sequence of text tokens, a text tokenizer is applied to the input text tag, The sequence of input tokens includes the sequence of text tokens, The method according to any one of claims 5 to 10, including the method described in any one of claims 5 to 10.

12. The method according to any one of claims 1 to 11, further comprising detoxifying the sequence of output tokens in order to generate an output that satisfies the task described above.

13. The method according to claim 12, wherein the output of the task includes text, an audio signal, or both.

14. The sequence of output tokens includes multiple text tokens, and detoxifying the sequence of output tokens is To generate text predictions, the text tokens are processed. The method according to any one of claims 12 to 13, including the method described in any one of claims 12 to 13.

15. The sequence of output tokens includes multiple audio tokens, and detoxifying the sequence of output tokens is Using one or more neural networks, generate an acoustic representation of an audio signal, conditionally on at least one set of audio tokens, wherein the acoustic representation specifies one or more sets of audio tokens in each of a set of second time steps spanning the audio signal, and each of the one or more audio tokens in each second time step represents the corresponding acoustic characteristics of the audio signal in the second time step. To generate predictions of audio signals, at least the acoustic representation is processed using a decoder neural network. The method according to any one of claims 12 to 14, including the method described in any one of claims 12 to 14.

16. The method according to claim 15, wherein generating the acoustic representation includes generating the acoustic representation conditional on the plurality of audio tokens and the speech context in order to generate the acoustic representation of the audio signal, and processing at least the acoustic representation using a decoder neural network includes processing the acoustic representation of the audio signal and the acoustic representation of the speech context using a decoder neural network in order to generate a prediction of the audio signal conditional on the speech context.

17. The method according to claim 16, as dependent on claim 7, wherein the audio context includes each audio token in each of a plurality of first time steps spanning at least a portion of the audio signal, and each audio token is selected from an audio token vocabulary to represent the semantic content of the audio signal in the corresponding first time step.

18. The method according to any one of claims 1 to 17, wherein the task includes one or more of automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, or text-to-text machine translation.

19. The method according to any one of claims 17 to 18, wherein the task comprises a plurality of subtasks, the input text tag specifies each of the subtasks, and the sequence of output tokens comprises the respective output for each of the subtasks.

20. The method according to claim 19, wherein the subtask includes one of the following: automatic speech recognition, automatic speech translation, speech-to-speech translation, text-to-speech, or text-to-text machine translation.

21. The method according to any one of claims 1 to 20, wherein the language model neural network is trained on one or more tasks.

22. A system comprising one or more computers and one or more storage devices for storing instructions, wherein, when the instructions are executed by the one or more computers, they are operable to cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 21.

23. A computer storage medium encoded with instructions, wherein, when executed by one or more computers, the instructions cause the one or more computers to perform the operations of each of the methods described in any one of claims 1 to 21.

Images