Discrete token processing using diffusion models

Abstract

In one aspect, the method includes: generating a final latent representation of the sequence of discrete tokens using the diffusion model, the final latent representation including determined values ​​for each of a plurality of latent variables; applying a de-embedding matrix to the final latent representation of the output sequence of discrete tokens to generate, for each of the plurality of latent variables, a de-embedding final latent representation including a respective numerical score for each discrete token in a vocabulary of the plurality of discrete tokens; selecting, for each of the plurality of latent variables, a discrete token having a highest numerical score from among the plurality of discrete tokens in the vocabulary; and generating an output sequence of discrete tokens including the selected discrete token.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 411,045, filed on 28 September 2022. The disclosures of the prior application are deemed to be part of the disclosures of this application and are incorporated into the disclosures of this application by reference. [Background technology]

[0002] This specification relates to data processing using neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from a given input according to the current values ​​of its respective set of parameters. [Overview of the Initiative]

[0004] According to a first embodiment, a computer-implementable method is provided for generating an output sequence of discrete tokens using a diffusion model. This method includes using the diffusion model to generate a final latent representation of the sequence of discrete tokens containing determined values ​​for each of a plurality of latent variables. Generating the final latent representation includes, at each of a plurality of despread time steps, processing a diffusion model input containing an intermediate latent representation of the sequence of discrete tokens at the time of the despread time step to generate an estimate of the sequence of discrete tokens at the time of the despread time step; using the estimate to define a distribution over a continuous space of possible values ​​for each of the plurality of latent variables; and generating an updated intermediate latent representation of the sequence of discrete tokens at the despread time step through sampling from the distribution. This method further includes, for each of the plurality of latent variables, applying a deembedded matrix having learned values ​​to the final latent representation of the output sequence of discrete tokens to generate a deembedded final latent representation containing the respective numerical score of each discrete token in the plurality of discrete token vocabulary. This method further includes, for each of the multiple latent variables, selecting the discrete token with the highest numerical score from among multiple discrete tokens in the vocabulary, and generating an output sequence of discrete tokens that includes the selected discrete token.

[0005] In the first despreading time step, the intermediate latent representation may be the initial latent representation. In subsequent despreading time steps, the intermediate latent representation may be the updated intermediate latent representation generated in the previous despreading time step.

[0006] In this embodiment, the estimate of the sequence of discrete tokens at the despreading time step is the estimate of the final latent representation of the output sequence of discrete tokens at the despreading time step.

[0007] In this embodiment, the discrete token includes text, symbols, or signals.

[0008] In this embodiment, the diffusion model input further includes an estimate of the sequence of discrete tokens generated at the previous despreading time step. In this embodiment, the estimate of the sequence of discrete tokens at the previous despreading time step is an estimate of the final latent representation of the output sequence of discrete tokens at the previous despreading time step.

[0009] In this embodiment, generating an output sequence of discrete tokens using a diffusion model includes generating unconditional discrete tokens.

[0010] In this embodiment, generating an output sequence of discrete tokens using a diffusion model includes generating discrete tokens conditioned on an input sequence of discrete tokens, and the method is: Receiving a discrete token input sequence, Convert each discrete token in the input sequence into a one-hot vector, This includes applying an embedding matrix with pre-trained values ​​to each one-hot vector in order to embed one-hot vectors into continuous vectors.

[0011] In this embodiment, generating an output sequence of discrete tokens using a diffusion model includes generating discrete tokens conditioned on an input sequence of discrete tokens, and the method is: Receiving a discrete token input sequence, Convert each discrete token in the input sequence into a one-hot vector, This includes applying an embedding matrix to each one-hot vector in order to embed the one-hot vectors into the embedding.

[0012] In this embodiment, the output sequence of discrete tokens also includes the input sequence of discrete tokens received by the spreading model.

[0013] In this embodiment, the discrete token input sequence includes, for example, discrete tokens representing speech data inputs that include spoken words characterizing the waveform of speech in the time domain or the time-frequency domain.

[0014] In this embodiment, the output token represents, for example, speech data containing a spoken word that characterizes the waveform of the speech in the time domain or the time-frequency domain.

[0015] In this embodiment, the input sequence is generated from an audio signal including spoken speech. In this embodiment, the input may be received in the form of an audio (speech) signal, which is converted by a speech-to-text converter to form the input sequence.

[0016] In this embodiment, the output sequence of discrete tokens represents text. In this embodiment, the output sequence of discrete tokens is converted by a text-to-speech converter to form an audio signal.

[0017] In this embodiment, the discrete token input sequence represents a sequence of actions performed by an agent, such as a machine agent in a real-world environment that implements actions to perform a mechanical task.

[0018] In this embodiment, the discrete token output sequence represents a sequence of actions performed by an agent, such as a machine agent in a real-world environment that implements the actions of performing a mechanical task.

[0019] In this embodiment, the method further, To generate a projected continuous vector, we apply linear projection to the continuous vector, This includes processing projected continuous vectors using a diffusion model.

[0020] In an embodiment, generating discrete tokens conditioned on an input sequence of discrete tokens includes using a classifier-free guidance technique.

[0021] In an embodiment, an embedding matrix is fixed during training of the diffusion model, while an unembedding matrix is learned during training of the diffusion model.

[0022] In an embodiment, the method further includes training the diffusion model with unlabeled discrete token data including the discrete token input to minimize the mean squared error between each discrete token input and an estimated value of the discrete token input generated by the diffusion model at the time of the sampled reverse diffusion time step.

[0023] In an embodiment, the training also minimizes a cross-entropy loss evaluated with respect to the final latent representation of the sequence of discrete tokens generated by the diffusion model from the discrete token input.

[0024] In an embodiment, the training includes applying a binary mask that includes one or more first masks that define conditional tokens within the sequence and one or more second masks that define infilling tokens within the sequence to a sequence of discrete token inputs, and training the diffusion model with the masked sequence of discrete token inputs to generate an estimated value of the sequence of discrete token inputs that includes infilling tokens within the sequence.

[0025] In an embodiment, the training includes learning the values of the unembedding matrix while keeping the pre-trained values of the embedding matrix fixed.

[0026] In another embodiment, a system is provided that includes one or more computers and one or more storage devices that, when executed by one or more computers, store instructions that are operable to cause one or more computers to perform any of the operations of the above-described methods.

[0027] In another embodiment, a computer storage medium is provided which, when executed by one or more computers, encodes instructions causing one or more computers to perform any of the operations of the above-described methods.

[0028] This specification describes a system implemented as a computer program on one or more computers in one or more locations that generates an output sequence of discrete tokens using a diffusion model that performs a dediffusion process on continuous embeddings in an embedding space.

[0029] In some cases, the generation process may be unconditional if the output sequence can be generated by a diffusion model from random noise. In other cases, for example, if the output text, computer program code, symbols, or signal sequence generated by the diffusion model is a complement or extension of the input text, computer program code, symbols, or signal sequence, the generation process may be conditional.

[0030] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages:

[0031] The diffusion models described herein are models that perform a diffusion process in a continuous latent space, yet can process data that is inherently discrete, such as text data; that is, they can receive discrete input data and / or generate discrete output data. By performing a diffusion process on continuous embeddings in the latent space and returning the final latent representation to discrete space in a continuous-to-discrete transformation step using a learnable deembling matrix, the diffusion models described possess both the flexibility to be easily configured for both conditional and unconditional discrete data generation, for example, and the scalability to be extended to any large-scale model with, for example, 100 million, 500 million, or more parameters, allowing them to better suit the needs of a variety of generative tasks. The diffusion models described achieve, or surpass, state-of-the-art performance in a wide range of conditional and unconditional text generation tasks achieved by many existing autoregressive models.

[0032] Advantageously, unlike most existing autoregressive models that predict output tokens sequentially in a way that initial tokens are not notified by later tokens, the diffusion model described can predict all tokens in a sequence at once. This allows for bidirectional attention (e.g., not causal attention), so that token selection later in the sequence influences the selection of earlier tokens, resulting in higher quality data generation. This significantly improves the parallelization of the inference process; since all tokens are generated simultaneously using the diffusion model rather than in the order required by autoregressive models, it becomes possible to generate any amount of data within a given time budget by running multiple diffusion processes in parallel with each other.

[0033] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0034] [Figure 1]This is a block diagram of an exemplary data generation system. [Figure 2] This is a flowchart illustrating an exemplary process for generating a discrete output sequence. [Figure 3] Figure 2 is an illustrative process flow diagram of one substep of the process steps. [Modes for carrying out the invention]

[0035] Similar reference symbols and designations in various drawings refer to the same elements.

[0036] Figure 1 shows an exemplary data generation system 100. The data generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, and can implement the systems, components, and technologies described below.

[0037] The data generation system 100 is a system that generates a discrete output sequence 162 in response to a received request. The data generation system 100 can store the discrete output sequence 162 in an output data repository or provide the discrete output sequence 162 for use for any other immediate purpose. For example, the data generation system 100 can then output the discrete output sequence 162 for display, for example, on a client device that sent the data generation request.

[0038] The discrete output sequence 162 contains multiple discrete tokens. Each discrete token may be a separate discrete data item. In this specification, continuous data refers to data that is constrained only by the precision of the numerical format used in the computer system and can take any value within a specified range. In contrast, discrete data has additional constraints that further restrict the possible values ​​that the data can take within a specified range, in addition to the possible values ​​of the numerical format used in the computer system.

[0039] In one example, a discrete output sequence contains multiple tokens from a finite number of possible tokens.

[0040] For example, a discrete output sequence 162 may include each of several tokens from a discrete token vocabulary at each of its multiple locations. The token vocabulary may include any of the various discrete tokens that represent text, symbols, or signals. For example, the discrete token vocabulary may include one or more characters, subwords, words, punctuation marks, numbers, or other symbols that appear in a text corpus. The text may be, for example, natural language text or computer program code. In some examples, the output tokens represent speech data containing, for example, spoken words that characterize the waveform of speech in the time domain or the time-frequency domain. In some examples, the discrete token output sequence represents text. In one example, the discrete token output sequence is transformed by a speech synthesis converter to form a speech signal. In some examples, the discrete token output sequence represents a sequence of actions performed by an agent, for example, a machine agent in a real-world environment that implements actions to perform a mechanical task.

[0041] In some cases, the data generation system 100 can be configured as an unconditional output sequence generation system that generates a discrete output sequence 162 without any conditions on any conditional inputs. In these embodiments, the discrete output sequence 162 generated by the system 100 approximates a sample of the distribution of training input sequences used during the training of the system 100.

[0042] In other cases, the data generation system 100 can be configured as a conditional output sequence generation system that generates a discrete output sequence 162 conditioned on an input sequence 102. In some of these cases, the input sequence 102 includes a discrete input sequence. Similar to the discrete output sequence 162, the discrete input sequence may include each of several positions containing a token from a token vocabulary. The token vocabulary may include any of the various discrete tokens representing text, symbols, or signals. In other cases, the input sequence 102 includes a continuous input sequence representing, for example, a sequence of pixels in an image, or a waveform or spectrogram of an audio.

[0043] For example, when configured as a conditional output sequence generation system, the data generation system 100 receives an input sequence 102 which includes text data and additional non-text data such as images, videos, speech, etc., or both, and can generate a discrete output sequence 162 which is a sequence of text such as completion of the input sequence 102, expansion of the input sequence 102, a response to a question presented in the input sequence 102, a sequence of text on a topic specified by the input sequence 102, or a textual description of data of different data types.

[0044] As another example, the data generation system 100 can receive an input sequence 102 containing one or more code segments and generate a discrete output sequence 162 that is conditioned on the input sequence 102, such as one or more code segments that, when combined with the code segments contained in the input sequence 102, constitute a sequence of executable applications, programs, objects, or instructions.

[0045] In both cases, text (both natural language text and computer program code), symbols, and signals are understood merely as examples of discrete data for illustrative purposes, and discrete tokens will be understood to be discrete data items of many other forms or styles. For example, a discrete output sequence 162 could be, or include, a sequence of biological data (e.g., a sequence of gene expression), a sequence of electronic health record data (e.g., a sequence of health events), a sequence of clinical procedure data (e.g., a sequence of physician's orders, clinical documents, notes, diagnostic codes, medications, etc.).

[0046] In one example, the input may be received in the form of a speech signal captured by a microphone, which is then converted by a speech recognition converter to form an input sequence 102. In some examples, the discrete token input sequence includes tokens representing speech data input, such as utterances that characterize the waveform of the speech in the time domain or the time-frequency domain. In some examples, the discrete token input sequence represents a sequence of actions performed by an agent, such as a machine agent in a real-world environment that implements actions to perform a mechanical task.

[0047] To generate the discrete output sequence 162, the data generation system 100 initializes the latent representation using the initialization engine 120, i.e., generates the initial latent representation 122 of the discrete output sequence 162, and then generates the final latent representation 136 of the discrete output sequence 162 by performing a despreading process that updates the initial latent representation 122 over multiple despreading time steps using the spread model neural network 130 (or "spread model 130" for short) and the update engine 140. Thus, the final latent representation 136 is generated in the last despreading time step of the despreading process.

[0048] The initial latent representation 122 has the same dimensions as the final latent representation 136, but its values ​​are different. That is, the initial latent representation 122 contains multiple latent variables, and the final latent representation 136 contains the same number of latent variables, but the values ​​of these latent variables are usually different between these two representations 122 and 136.

[0049] When configured as an unconditional output sequence generation system, the initialization engine 120 generates the initial latent representation 122 by sampling initial values ​​for each of the multiple latent variables contained in the initial latent representation 122 from a corresponding noise distribution, such as a Gaussian distribution or another predetermined distribution. Thus, the initial latent representation 122 contains multiple latent variables, and the initial value of each latent variable is sampled from the corresponding noise distribution.

[0050] When configured as a conditional output sequence generation system, the preprocessing engine 110, an optional component of the data generation system 100, processes the received input sequence 102 to generate a conditional embedding 116 containing a sequence of numbers. The initialization engine 120 then generates the initial latent representation 122 based on using the numbers contained in the conditional embedding 116 as initial values ​​for some of the latent variables contained in the initial latent representation 122, and sampling the initial values ​​for each of the remaining latent variables from a corresponding noise distribution, such as a Gaussian distribution or another predetermined distribution. Thus, the initial latent representation 122 contains multiple latent variables, the initial values ​​for some of the latent variables are determined from the input sequence 102, and the initial values ​​for each of the other latent variables are sampled from a corresponding noise distribution.

[0051] More specifically, the preprocessing engine 110 includes an embedding matrix 112 that defines a mapping from the discrete space to the embedding space. To define this mapping, the embedding matrix 112 can be a matrix of dimension D × V, i.e., a matrix that can have D × V predefined values ​​included as entries in the matrix, where D is the embedding size (the size of each continuous vector in the embedding space) and V is the vocabulary size (the total number of discrete tokens contained in the token's vocabulary). Thus, in one example, each token ω in the vocabulary has a fixed norm that matches the norm of a random Gaussian sample of dimension D used for denoising data.

number

number

[0052] The preprocessing engine 110 converts each token in the input sequence 102 into a discrete one-hot vector, applies the embedding matrix 112 to each discrete one-hot vector, and maps the discrete one-hot vector to a continuous vector in the embedding space having the numerical values ​​of the tokens, for example, based on element-wise multiplication. Thus, in some embodiments, the conditional embedding 116 consists of the numerical values ​​contained in the continuous vector generated as a result of applying the embedding matrix 112 to the input sequence 102. In other embodiments, the preprocessing engine 110 additionally applies linear projection to each continuous vector to generate a projected continuous vector, and then generates the conditional embedding 116 from the numerical values ​​contained in the projected continuous vector. The conversion to tokens can be performed using the SentencePiece tokenizer described in Kudo et. al., "Sentencepiece: A simple and language-independent subword tokenizer and detokenizer for neural text processing," EMNLP (Demonstration), pp. 66-71, Association for Computational Linguistics, 2018, the full content of which is incorporated herein by reference and consists of 32,000 words.

[0053] The diffusion model 130 can have any suitable architecture, so that the diffusion model can process the diffusion model input at a desuplash time step, which includes an intermediate latent representation (at the time of the desuplash time step) having the same dimensions as the discrete output sequence 162, at any given desuplash time step, and generate the diffusion model output 132 at the given desuplash time step. The diffusion model output 132 includes the discrete output sequence 162 and an estimate of the discrete output sequence 162 (at the time of the desuplash time step), which also has the same dimensions as the discrete output sequence 162.

[0054] For example, the diffusion model 130 may be a convolutional neural network, such as U-Net, or other architecture, that includes one or more convolutional residual blocks. Another example is that the diffusion model 130 may be an attention neural network that includes one or more attention blocks, such as self-attention blocks, gated attention blocks, or cross-attention blocks. Yet another example is that the diffusion model 130 may include both convolutional residual blocks and attention blocks.

[0055] At a specified despreading time step, the data generation system 100 then uses the update engine 140 to update the intermediate latent representation based on the estimate of the discrete output sequence 162 for the specified despreading time step. The intermediate latent representation after being updated by the update engine 140 at a specified despreading time step is referred herein to as the updated intermediate latent representation 134, and the intermediate latent representation after being updated at the last despreading time step in the despreading process is referred herein to as the final intermediate latent representation 136.

[0056] Therefore, at any given despreading time step, the intermediate latent representation provided as (part of) the diffusion model input to the diffusion model 130 will be the updated intermediate latent representation 134 generated in the immediately preceding despreading time step. In the first despreading time step, the intermediate latent representation is the initial latent representation 122 generated by the initialization engine 120.

[0057] More specifically, at any given despreading time step, the update engine 140 defines a corresponding distribution across a continuous space of possible values ​​for each of the multiple latent variables contained in the initial latent representation 122, using at least an estimate of the discrete output sequence 162 for that given despreading time step. The update engine 140 then generates an updated intermediate latent representation 134 of the discrete output sequence 162 for the despreading time step by selecting a value for each latent variable from the corresponding distribution, for example, by selecting the value with the highest probability, or by sampling from the distribution. Thus, the updated latent representation 134 contains multiple latent variables, the value of each latent variable being sampled from the corresponding distribution defined by the diffusion model output 132 of the diffusion model 130.

[0058] To define such distributions for each latent variable, the update engine 140 can calculate the mean and, optionally, the variance of the distribution based on the estimates of the discrete output sequence 162 and the intermediate latent representation before it is updated at a specified despreading time step. The update steps are further described below with reference to Figure 3.

[0059] In some embodiments of the data generation system 100, a self-conditioning input 138 is used when performing a dediffusion process. When the data generation system 100 is configured as a conditional output sequence generation system, the use of self-conditioning can improve the quality of the discrete output sequence 162.

[0060] At any given despreading time step, the self-conditioned input 138 can be an estimate of the discrete output sequence 162 included in the diffusion model output generated at the immediately preceding despreading time step. Thus, for any given despreading time step, the diffusion model 130 can process the diffusion model input, which includes (i) an intermediate latent representation (at the despreading time step) and (ii) an estimate of the discrete output sequence 162 generated at the immediately preceding despreading time step, to generate the diffusion model output 132 for the given despreading time step.

[0061] In some embodiments of the data generation system 100, guidance is used when performing the despreading process. That is, the despreading process is, in some cases, a guided despreading process. To reduce the need for individually trained guidance neural networks, the guidance can be classifier-less guidance.

[0062] Using classifier-less guidance may involve the diffusion model 130 processing a sequence of conditioned tokens as part of the diffusion model input at any specified despreading time step to generate the diffusion model output. Conditioned tokens can be defined by a binary conditioning mask set to 1 for each position in the discrete output sequence that has a given discrete token ("conditioning position") and to 0 for each position in the discrete output sequence that has an unknown discrete token ("filling position"), i.e., a position where a discrete token needs to be generated by the system.

[0063] After the final despreading time step, the data generation system 100 provides the post-processing engine 150 with a final latent representation 136 in the same embedding space as the initial latent representation 122. The final latent representation 136 contains multiple latent variables. Each latent variable has a determined value determined as a result of the despreading process, and this value may differ from the initial value. The post-processing engine 150 processes the final latent representation 136 to generate a discrete output sequence 162 in the same discrete space as the input sequence 102.

[0064] To this end, the post-processing engine 150 includes a deembedding matrix 152 that defines a mapping from the embedding space to the discrete space. To define this mapping, the deembedding matrix 152 can be a matrix of dimension V × D, i.e., a matrix that can have V × D pre-trained values ​​included as entries in the matrix, where V is the vocabulary size (the total number of discrete tokens contained in the token vocabulary) and D is the embedding size (the size of each continuous vector in the embedding space).

[0065] The post-processing engine 150 applies the deembedding matrix 152 to each latent variable in the final latent representation 136, for example, based on element-wise multiplication operations, to generate a deembedding final latent representation. The deembedding final latent representation contains, for each latent variable in the final latent representation 136, a lexical distribution of tokens corresponding to the latent variable. For each latent variable, the corresponding distribution contains the respective numerical scores of each token in the lexical.

[0066] Next, the post-processing engine 150 generates a discrete output sequence 162 for each latent variable by selecting tokens from the vocabulary according to the corresponding distribution, for example, by selecting the token with the highest numerical score, or by sampling from the corresponding distribution.

[0067] Since the calculations involved in generating each discrete output sequence can be parallelized over the entire length of the discrete output sequence, the data generation system 100 makes more effective use of the computational resources of the hardware accelerator capable of implementing the generation system 100.

[0068] A hardware accelerator is a computing device equipped with specialized hardware configured to perform specialized computations, such as parallel computing. Examples of hardware accelerators include application-specific integrated circuits (ASICs) such as graphics processing units (GPUs), field-programmable gate arrays (FGPAs), and tensor processing units (TPUs).

[0069] The process of training the diffusion model 130 and other trainable components of the data generation system 100 to determine the trained parameter values ​​of these components is described further below.

[0070] Figure 2 is a flowchart of an exemplary process 200 for generating a discrete output sequence. For convenience, process 200 is described as being executed by a system of one or more computers located in one or more locations. For example, a data generation system appropriately programmed according to this specification, such as the data generation system 100 shown in Figure 1, can execute process 200.

[0071] The system performs a despreading process on the initial latent representation of the discrete output sequence to generate the final latent representation of the discrete output sequence (step 202). The discrete output sequence contains, at each of several locations, a token from the token vocabulary. The token vocabulary can contain any of the various discrete tokens representing text, symbols, or signals.

[0072] The initial latent representation has the same dimensions as the final latent representation, but its values ​​are different. That is, the initial latent representation contains multiple latent variables, and the final latent representation contains the same number of latent variables, but the initial values ​​of these latent variables in the initial latent representation and the determined values ​​of these latent variables in the final latent representation are usually different from each other.

[0073] When configured as an unconditional output sequence generation system, the system can generate an initial latent representation based on sampling initial values ​​for each of several latent variables from a corresponding noise distribution (e.g., a Gaussian distribution or another predetermined distribution).

[0074] When configured as a conditional output sequence generation system, the system receives an input sequence and can then map the received input sequence to a conditional embedding containing a sequence of numbers. The mapping is an embedding matrix where D is the embedding size and V is the vocabulary size.

number

[0075] Next, the system generates the final latent representation by updating the initial latent representation over several despreading time steps using a diffusion model. In other words, the final latent representation is the updated intermediate latent representation generated in the last despreading time step.

[0076] Updating the initial latent representation will be described in more detail below with reference to Figure 3, which shows a flowchart of an exemplary process 300 in substeps 302-306 of step 202. In general, the system can repeatedly run process 300 in each of multiple despreading time steps T, T-1, ..., 2, 1.

[0077] The system processes the diffusion model input at the despreading time step to generate the diffusion model output (step 302). The diffusion model output may include estimates of the discrete output sequence at the despreading time step. On the other hand, the diffusion model input may include (i) the intermediate latent representation of the discrete output sequence (at the despreading time step) x t , and (ii) a time index t indicating the current despreading time step. In one example, the spreading model is a trained neural network and the intermediate latent representation (despreading time step time) x of the discrete output sequence. t It takes a time index t representing the current despreading time step as input, and the final latent representation of the discrete output sequence at the despreading time step.

number

[0078] In the first despreading time step, the intermediate latent representation x t x is the initial latent representation. At any subsequent despreading time step, the intermediate latent representation x t This is the updated intermediate latent representation generated in the immediately preceding despreading time step.

[0079] For example, estimates of discrete output sequences

number

number

[0080] Optionally, the diffusion model input may also include a self-conditioning input. At any given desplash time step, the self-conditioning input can be an estimate of the discrete output sequence contained in the diffusion model output generated at the previous desplash time step. Therefore, for any given desplash time step, the diffusion model input may be an estimate of the discrete output sequence contained in the diffusion model output generated at the previous desplash time step t+1.

number

[0081] Another example is the estimation of discrete output sequences.

number

number

[0082] Furthermore, the diffusion model input may optionally include guidance, such as no-classifier guidance. When using no-classifier guidance, for any specified despreading time step, the diffusion model input may additionally include a fixed sequence of conditioning tokens.

[0083] Another example is the estimation of discrete output sequences.

number

number

[0084] The system estimates the discrete output sequence.

number

number

[0085] For example, the system can calculate the mean μ of the distribution as follows:

number

Number

Number

[0086] The system generates an updated intermediate latent representation x t-1 of the discrete output sequence of inverse diffusion time steps from the distribution (step 306). The system can do this by selecting the value of each latent variable from the corresponding distribution, for example, by sampling a value from the continuous space of possible values according to the distribution:

Number

[0087] Therefore, the updated latent representation x t-1 contains a plurality of latent variables, and the value of each latent variable is sampled from the corresponding distribution defined by the diffusion model output of the diffusion model at the current inverse diffusion time step.

[0088] Returning to FIG. 2, the system applies an embedding removal matrix to the final latent representation x0 of the discrete output sequence (step 204) to generate an embedded and removed final latent representation. The embedded and removed final latent representation contains, for each latent variable, a distribution over the vocabulary of tokens corresponding to the latent variable. For each latent variable, the corresponding distribution contains the respective numerical scores of each token within the vocabulary of tokens.

[0089] For example, the system can generate the deembedded final latent representation by the following calculation:

number

number

number

[0090] The system selects tokens from the vocabulary according to the corresponding distribution for each of the multiple latent variables (step 206). For each latent variable, the system can, for example, select the token with the highest numerical score from all the tokens in the vocabulary.

[0091] The system generates a discrete output sequence (step 208). The discrete output sequence contains the tokens selected from the vocabulary in step 206. If configured as a conditional output sequence generation system, the discrete output sequence may optionally also include the input sequence received by the system as input.

[0092] By repeatedly executing process 200, the system can generate different discrete output sequences. In other words, process 200 can be executed as part of predicting a discrete output sequence from an input sequence where the desired output, i.e., the discrete output sequence that the system should generate from the input sequence, is unknown.

[0093] Some steps of process 200, for example, substep 302 of step 202, can also be performed as part of processing input sequences derived from a set of training data, i.e., inputs derived from a set of inputs whose discrete output sequences to be produced by the system are known, in order to train the trainable components of the system and determine the training values ​​of the parameters of these components. As an example, the diffusion model can be trained on the C4 dataset described in Raffel et al., "Exploring the limits of transfer learning with a unified text-to-text transformer", J. Mach. Learn. Res., 21:140:1-140:67, 2020, the full contents of which are incorporated herein by reference. For example, the training data may be converted into tokens using the SentencePiece tokenizer described in Kudo et. al., "SentencePiece: A simple and language-independent subword tokenizer and detokenizer for neural text processing," EMNLP (Demonstration), pp. 66-71, Association for Computational Linguistics, 2018, the full content of which is incorporated herein by reference, and consists of 32,000 words. For example, the diffusion model may be trained with a sequence length of 256. For example, 10% of the padding tokens may be uniformly inserted into the training set (i.e., not necessarily at the end of the sequence) to allow the model to generate samples of varying sizes and provide greater flexibility.

[0094] Specifically, as part of the diffusion model training process, the system can train a trainable component of the system by repeatedly performing substep 302 of step 202 on an input sequence selected from a set of unlabeled discrete training data, and optimize an objective function suitable for the discrete data generation task that the diffusion model is configured to perform.

[0095] For example, the objective function might include an input sequence x selected from unlabeled discrete training data. 0で The diffusion model θ is trained, and each input sequence x0 and the estimated input sequence generated by the diffusion model at a sampled desplash time step in the desplash process are taken.

number

number

[0096] The objective function may additionally include a reconstruction loss term that trains the unembedding matrix R to minimize the cross-entropy loss, which is evaluated with respect to the final latent representation of the discrete output sequence generated by the diffusion model from the input sequence.

number

number

[0097] During training, the system can incorporate any number of techniques to improve the speed, effectiveness, or both of the training process.

[0098] For example, the system can use span masking techniques to train a diffusion model on several filling tasks, such as fill-in-the-middle and spans-in-filling tasks. In this example, for each input sequence selected from unlabeled discrete training data, the system can apply a binary mask to the discrete tokens contained in the input sequence. The binary mask includes one or more first masks that define the conditioned tokens in the sequence and one or more second masks that define the filling tokens in the sequence. The system can then train a diffusion model on the masked input sequences to generate estimates of the filling tokens contained in the masked input sequences, i.e., estimates of the original discrete tokens contained in the input sequences that are masked by the second masks.

[0099] As another example, the system can initialize the deembedding matrix R to the transpose of the embedding matrix E before training. The values ​​of the embedding matrix E can be pre-trained. In the pre-training phase, a BERT model with a fixed size (e.g., 150m parameters) and feature dimension of 896 can be trained to generate word embeddings.

[0100] As yet another example, to stabilize training and avoid a decrease in unigram entropy, the system can train the deembedding matrix R in particular, while keeping the embedding matrix E fixed during training. Thus, the deembedding matrix R is learned during the training of the diffusion model, while the embedding matrix E is not learned during training.

[0101] In this example, "learned" means that one or more values ​​included as entries in the deembedding matrix R are adjusted during the training of the diffusion model. In contrast, "not learned" means that predefined values ​​included as entries in the embedding matrix E, which were determined before the diffusion model was trained (for example, values ​​determined as a result of training another neural network with which the embedding matrix E was learned), remain fixed throughout the entire training of the diffusion model.

[0102] As mentioned earlier, in one example where self-conditioning is used, the x0 estimate is gradually improved by passing the estimate obtained in the previous sampling step as input to the diffusion model. To approximate the inference behavior during training while maintaining computational efficiency, the initial estimate is calculated with self-conditioning set to zero:

number

[0103] next,

number

number

[0104] Next, the diffusion model is optimized using the outputs from two forward propagations to accurately estimate x0, with or without self-conditioning.

[0105] The term “configured” is used herein in relation to systems and computer program components. A system of one or more computers being configured to perform a particular operation or action means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. A system of one or more computer programs being configured to perform a particular operation or action means that the program contains instructions that cause the data processing device to perform that operation or action when executed by the device.

[0106] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, or in one or more combinations thereof, including structures and structural equivalents thereof disclosed herein. Embodiments of the subject matter described herein may be implemented as one or more modules of computer program instructions, i.e., as one or more computer programs encoded in a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into artificially generated propagating signals, such as mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0107] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may optionally include code that generates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code that constitutes one or more of these.

[0108] Computer programs may also be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, and can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed as standalone programs or as modules, in any form including components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a part of a file that holds one or more scripts stored in other programs or data, for example, a document in a markup language, in a single file dedicated to the program of interest, or in multiple collaborative files, for example, files that store one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or it may be deployed to run on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0109] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on a storage device in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0110] Similarly, the term “engine” in this specification is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines can be installed and run on the same one or more computers.

[0111] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0112] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing units. Typically, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or a computer may be functionally connected to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive, transmit, or both. However, a computer is not required to have such devices. Furthermore, computers can be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives (these are just a few examples).

[0113] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0114] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from the user's device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving a response message from the user.

[0115] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for processing the general and numerical computation portions of machine learning training or production, i.e., inference, workloads.

[0116] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the JAX framework.

[0117] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component as an application server, or a frontend component as a client computer having a graphical user interface, web browser, or app on which a user can interact with embodiments of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0118] A computing system can include a client and a server. The client and server are generally geographically distant from each other and typically communicate through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received by the server from the device.

[0119] This specification includes details of many specific embodiments, but these should not be construed as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, individually or in any preferred secondary combination. Furthermore, even if multiple features are described above as functioning in a particular combination and are initially described as such in the claims, one or more features may be removed from the combination described in the claims, and furthermore, the combination described in the claims may cover secondary combinations or variations of secondary combinations.

[0120] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0121] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require that the actions be performed in the specific order or sequence shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implementable method for generating a discrete token output sequence using a diffusion model, Using the diffusion model, generate a final latent representation of the sequence of discrete tokens, which includes values ​​determined for each of the multiple latent variables, wherein generating the final latent representation involves, in each of the multiple despreading time steps, To generate an estimate of the sequence of discrete tokens at the time step in the despreading time step, the diffusion model input is processed, which includes an intermediate latent representation of the sequence of discrete tokens at the despreading time step. Using the aforementioned estimates, define the distribution of possible values ​​over a continuous space for each of the multiple latent variables, The process of generating the final latent representation includes generating an updated intermediate latent representation of the sequence of discrete tokens for the despread time step through sampling from the distribution, For each of the multiple latent variables, in order to generate a deembedded final latent representation that includes the numerical score of each discrete token in the vocabulary of multiple discrete tokens, a deembedded matrix having learned values ​​is applied to the final latent representation of the output sequence of the discrete tokens, For each of the aforementioned multiple latent variables, the discrete token with the highest numerical score is selected from among the aforementioned multiple discrete tokens in the vocabulary. The process includes generating an output sequence of discrete tokens, which includes the selected discrete tokens. method.

2. The method according to claim 1, wherein the discrete token includes text, symbols, or signals.

3. The method according to claim 1, wherein the diffusion model input further includes an estimate of the sequence of discrete tokens generated at the previous despreading time step.

4. The method according to claim 1, wherein generating the output sequence of the discrete tokens using the diffusion model includes generating an unconditional discrete token.

5. Generating the output sequence of discrete tokens using the diffusion model includes generating discrete tokens conditioned on the input sequence of discrete tokens, and the method is Receiving the input sequence of the discrete tokens, Convert each discrete token in the input sequence into a one-hot vector, The method according to claim 1, comprising applying an embedding matrix having pre-trained values ​​to each one-hot vector in order to embed the one-hot vectors into continuous vectors.

6. The method according to claim 5, wherein the output sequence of the discrete tokens also includes the input sequence of the discrete tokens received by the spreading model.

7. To generate a projected continuous vector, linear projection is applied to the continuous vector, The method according to claim 5, further comprising processing the projected continuous vector using the diffusion model.

8. The method according to claim 5, wherein generating the discrete tokens conditioned on the input sequence of discrete tokens includes using a classifier-less guidance technique.

9. The method according to claim 5, wherein the embedding matrix is ​​fixed during the training of the diffusion model, while the deembedding matrix is ​​learned during the training of the diffusion model.

10. The method according to claim 1, further comprising training the diffusion model with unlabeled discrete token data including the discrete token inputs to minimize the mean squared error between each discrete token input and an estimate of the discrete token input generated by the diffusion model at a sampled inverse diffusion time step.

11. The method according to claim 10, wherein the training also minimizes the cross-entropy loss evaluated with respect to the final latent representation of the sequence of discrete tokens generated by the diffusion model from the discrete token input.

12. The aforementioned training is Applying a binary mask to a sequence of discrete token inputs, wherein the binary mask includes one or more first masks defining conditional tokens in the sequence and one or more second masks defining filler tokens in the sequence. The method according to claim 10, comprising training the diffusion model on a masked sequence of discrete token inputs to generate an estimate of the sequence of discrete token inputs, which includes the filling tokens in the sequence.

13. Further comprising training the diffusion model with unlabeled discrete token data including the discrete token inputs to minimize the mean squared error between each discrete token input and an estimate of the discrete token input generated by the diffusion model at a sampled despread time step, The method according to claim 5, wherein the training includes learning the values ​​of the unembedding matrix while keeping the pre-trained values ​​of the embedding matrix fixed.

14. A system comprising one or more computers and one or more storage devices for storing instructions, wherein, when executed by the one or more computers, the instructions are operable to cause the one or more computers to perform an operation according to any one of claims 1 to 13.

15. A computer-readable storage medium that, when executed by one or more computers, stores instructions causing the one or more computers to perform an operation according to any one of claims 1 to 13.

Images