Sequence Modeling with Imputation

Abstract

A method, system, and apparatus for sequence modeling, including a computer program encoded on a computer storage medium. One of the methods includes: receiving an input sequence having a plurality of input positions; determining a plurality of blocks of consecutive input positions; processing the input sequence using a neural network to generate a latent alignment, at each input time step in a plurality of input time steps, including: receiving a partial latent alignment from a previous input time step; selecting an input position in each block, wherein a language symbol at the selected input position of the partial latent alignment in each block is a mask language symbol; and processing the partial latent alignment and the input sequence using the neural network to generate a new latent alignment, wherein the new latent alignment includes an output language symbol or a blank language symbol at the selected input position in each block; and generating an output sequence using the latent alignment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 62 / 971,769, filed on February 7, 2020, and U.S. patent application serial number 63 / 009,970, filed on April 14, 2020, the entire contents of which are incorporated herein by reference. Background Art

[0003] This specification relates to using neural networks to generate sequence data.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network (i.e., one or more other hidden layers, the output layer, or both). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding parameter set. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that receives an input sequence consisting of corresponding input language symbols at each of a plurality of input positions and generates an output sequence consisting of corresponding output language symbols at each of a plurality of output positions. The system can generate the output sequence by processing the input sequence using a neural network. This specification also describes techniques for training the neural network to generate the output sequence.

[0006] The neural network can first generate a latent alignment between the input sequence and the output sequence, and then generate the output sequence from the latent alignment. The neural network can generate the output sequence in a fixed number of time steps by dividing the input sequence into multiple blocks of the same size B and generating a corresponding new element in the latent alignment of each block at each time step. Therefore, regardless of the length of the input sequence, the neural network can generate the latent alignment in exactly B time steps and then generate the output sequence from the latent alignment.

[0007] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0008] The system described in this specification can generate output sequences faster than some existing systems that rely on autoregressive neural networks. In particular, the system described in this specification can generate output sequences in a fixed, constant number of time steps. This guaranteed efficiency may be critical for use cases where fast output is required regardless of the size of the input. In this way, the system described in this specification allows real-time operation in real-world systems. For example, an on-device speech recognition system, such as a system operating on a user's mobile device, may need to respond to the user in real time, such as by receiving an input audio sequence including a question spoken by the user, generating an output text sequence of the question asked by the user, and determining an answer for the user. As another example, a machine translation system may need to receive an input text sequence in a first language and generate an output text sequence in a second language in real time to facilitate communication between two users speaking different languages. More generally, the features of the various systems described in this specification are derived from technical considerations related to achieving real-time operation in real-world systems.

[0009] Systems as described herein can generate output sequences that are more accurate than some existing systems that rely on non-autoregressive neural networks because they do not rely on conditional independence assumptions between output language symbols. In particular, some systems described herein marginalize over all possible alignments and generation orders, making them well-suited for problems with naturally underlying monotonic alignments.

[0010] The system as described in this specification is not limited to generating output sequences from left to right, but is capable of modeling languages ​​with bidirectional contextualization and both local and global conditional dependencies.

[0011] The details of the subject matter of this invention are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of an example sequence modeling system.

[0013] Figures 2A-2C is an illustration of an example partial potential alignment between the input and output sequences.

[0014] Figure 3 is a flowchart of an example process for processing an input sequence to generate an output sequence.

[0015] Figure 4 is a diagram of an example neural network.

[0016] Figure 5 is a diagram of an example training system.

[0017] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0018] This specification describes a system for processing an input sequence to generate an output sequence.

[0019] Figure 1 is a diagram of an example sequence modeling system 100. The sequence modeling system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented. The sequence modeling system 100 includes a neural network 110 and an output generation engine 120.

[0020] The sequence modeling system 100 is configured to obtain an input sequence 102 and generate an output sequence 122. The input sequence 102 includes a corresponding input language token at each of a plurality of input positions, and the output sequence 122 includes a corresponding output language token at each of the plurality of output positions.

[0021] The input language symbols of the input sequence 102 and the output language symbols of the output sequence 122 can be of any suitable type. In some implementations, the input language symbols and the output language symbols are of different types. In some other implementations, the input language symbols and the output language symbols are of the same type.

[0022] For example, the input language symbols and / or the output language symbols may represent text; for example, each language symbol may represent a character, a phoneme, or a word. As another example, the input language symbols and / or the output language symbols may represent audio data; for example, each language symbol may represent the original, compressed, or compressed amplitude value of an audio wave. As another example, the input language symbols and / or the output language symbols may represent sensor data captured by a sensor in the environment at a corresponding point in time. As another example, the input language symbols and / or the output language symbols may represent an embedding of the state of the environment at a corresponding point in time. As another example, the input language symbols and / or the output language symbols may represent an action to be taken by a robot interacting with the environment.

[0023] As a specific example, the input language symbols may represent audio samples, while the output language symbols may represent text samples; that is, the neural network 110 may be configured through training to perform speech recognition. As another specific example, the input language symbols may represent text samples, while the output language symbols may represent audio samples; that is, the neural network 110 may be configured through training to perform speech synthesis. As another specific example, the input language symbols may represent text samples in a first language, while the output language symbols may represent text samples in a second language; that is, the neural network 110 may be configured through training to perform machine translation. As another specific example, the input language symbols may represent text samples from a document and text samples representing questions about the document, while the output language symbols may represent text samples representing answers to the questions; that is, the neural network 110 may be configured through training to perform question answering. As another specific example, the input language symbols may represent text samples from a document, while the output language symbols may represent text samples representing a summary of the document; that is, the neural network 110 may be configured through training to perform text summarization.

[0024] As another example, the input sequence 102 and / or the output sequence 122 may represent an image. For example, each linguistic symbol in the sequence may represent a corresponding pixel of the image; that is, the sequence may be a "flat" version of the image. As a specific example, the input sequence 102 may represent an image, and the output sequence 122 may represent text describing the image. As another specific example, the input sequence 102 may represent a first image, and the output sequence 122 may represent a second image that is a transformed version of the first image. For example, the sequence modeling system 100 may be configured to process an input sequence 102 representing a first image that includes artifacts (e.g., noise or low resolution), and generate an output sequence 122 representing a second image that represents the first image without the artifacts. As another example, the sequence modeling system 100 may be configured to process an input sequence 102 representing a black and white image, and generate an output sequence 122 representing a colorized version of the black and white image. As another example, the sequence modeling system 100 may be configured to process an input sequence 102 representing a first image from a first domain and generate an output sequence 122 representing a second image in a second domain; for example, the first image may depict a simulated environment and the second image may depict a real-world environment, or vice versa.

[0025] The input sequence 102 may also be referred to as an input sequence x={x0, x1, ..., x N-1}, and the output sequence 122 can also be called the output sequence y={y0, y1, ..., y M-1}.

[0026] Neural network 110 is configured to process input sequence 102 at a first time step and generate partial latent alignment 112. Then, at each of a plurality of subsequent time steps, neural network 110 processes input sequence 102 and partial latent alignment 112 generated at the previous time step to generate a new partial latent alignment 112. Then, at a final time step, neural network 110 processes input sequence 102 and the previous partial latent alignment 112 to generate a final latent alignment.

[0027] A potential alignment, or simply "alignment," between an input sequence and an output sequence is a mapping between input language symbols of the input sequence and output language symbols of the output sequence. Each input language symbol can be mapped to an output language symbol such that each output language symbol has at least one input language symbol mapped to the output language symbol. A partial potential alignment between an input sequence and an output sequence is a partial mapping between input language symbols of the input sequence and output language symbols of the output sequence. That is, some input language symbols of the input sequence can be mapped to corresponding output language symbols, while the remaining input language symbols of the input sequence have not yet been mapped to corresponding output language symbols.

[0028] In some implementations, there is a monotonic potential alignment between the input language symbols of the input sequence 102 and the output language symbols of the output sequence 122. If each input language symbol can be mapped to a corresponding output language symbol, such that there is no input language symbol mapped to the output language symbol y i Input language symbol x i and are mapped to the output language symbol y j Input language symbol x j , so that x i In the input sequence than x j Early and y i In the output sequence than y j Late, then there is a monotonic potential alignment between the input language symbol and the output language symbol. That is, if the input language symbol x i Mapping output language symbol y i , then the following input language symbol x i +1 must map to the same output language symbol y i or mapped to the following output language symbol y i+1 Note that multiple input language symbols can be mapped to the same output language symbol such that |y| ≤ |x|, i.e., M ≤ N. In other words, for any language symbol at any given position in the input sequence, the corresponding output language symbol in the potential alignment is no earlier in the output sequence than the output language symbol corresponding to any language symbol at a position after the given position in the input sequence.

[0029] The partial potential alignment 112 and the final potential alignment 114 may be any data representing a mapping (partial or complete, respectively) between the input language symbols of the input sequence 102 and the output language symbols of the output sequence 112 .

[0030] For example, if y i ∈V, where V is the vocabulary of the output sequence 122, then there may be a monotonic final potential alignment a={a0, a1, ..., a N-1}, so that |a|=|x|, where a i ∈V + , V + =V∩{″_″}, where “_” is a special language symbol called a “blank language symbol.” If multiple input language symbols map to the same output language symbol in alignment a, the blank language symbol helps align input sequence 102 with output sequence 122. In this specification, the elements of V may also be referred to as “lexicon language symbols.” That is, every language symbol in a monotone potential alignment a is either a blank language symbol or a lexicon language symbol.

[0031] For each input language symbol in the input sequence 102 that corresponds to a blank language symbol in the final potential alignment 114, the blank language symbol recognition input language symbol is mapped to the same output language symbol as the previous input language symbol in the input sequence 102 (i.e., the most recent input language symbol that does not correspond to a blank language symbol in the final potential alignment 114). For example, if the system is configured to perform speech recognition, three input audio samples of the input sequence 102 may be mapped to the output language symbol "MY". In this case, the language symbol in the final potential alignment 114 corresponding to the first of the three input audio samples may be "MY", while the two language symbols in the final potential alignment 114 corresponding to the other two of the three input audio samples may be blank language symbols.

[0032] Before the first time step of the neural network 110, the sequence modeling system 100 can generate an initial partial potential alignment 112 that is the same size as the input sequence 102 and consists entirely of "masked" language symbols. A masked language symbol is a language symbol that represents a partial potential alignment for which the corresponding language symbol in the final potential alignment 114 has not yet been selected. Thus, each language symbol in the partial potential alignment is either a masked language symbol or a blank language symbol or a vocabulary language symbol (if it has already been selected). Then, at each time step, the neural network can process i) the input sequence 102 and ii) the current partial potential alignment 112 to generate one or more new language symbols (e.g., vocabulary language symbols or blank language symbols) to replace the corresponding masked language symbols in the partial potential alignment 112. That is, at each time step, the neural network 110 generates a new partial potential alignment 112 with fewer masked language symbols.

[0033] For example, neural network 110 may process input sequence 102 to generate an embedding of input sequence 102, and process partial latent alignment 112 to generate an embedding of partial latent alignment 112. Then, neural network 110 may process both embeddings to generate an updated partial latent alignment 112. Figure 4 An example network architecture for neural network 110 is discussed in more detail.

[0034] In some implementations, neural network 110 generates a fixed number of new language symbols in partial latent alignment 112 at each time step of neural network 110. For example, neural network 110 may partition partial latent alignment 114 into blocks of a fixed size B, where each block consists of B consecutive language symbols in partial latent alignment 114, and each language symbol is in exactly one block. Then, at each time step, neural network 110 may generate a corresponding new language symbol in each block. That is, at each time step, neural network 110 may select a language symbol in each block as a mask language symbol, and select a vocabulary language symbol or a blank language symbol to replace the selected language symbol, thereby generating a new partial latent alignment 112.

[0035] In this example, final potential alignment 114 is guaranteed to be generated in a fixed amount of time equal to B time steps. That is, the amount of time to generate final potential alignment 114 is constant relative to the length of input sequence 102. This efficiency can be advantageous compared to some existing systems that use autoregressive neural networks, as the running time of autoregressive neural networks generally increases, e.g., linearly, with the size of input sequence 102.

[0036] However, by generating the final latent alignment 114 over multiple time steps and adjusting each time step based on the most recent partial latent alignment 112, the neural network 110 avoids making a conditional independence assumption on the output language symbols of the output sequence. In many existing non-autoregressive systems, the system assumes that the output language symbols in the output sequence are independent of each other and generates the output sequence within a single time step. This conditional independence assumption can lead to poor performance in existing systems because the output language symbols are not actually independent. Figure 1 The depicted neural network 110 uses the language symbols in the partial latent alignment 112 generated at the previous time step to update the partial latent alignment 112; that is, the new language symbols in the partial latent alignment 112 are dependent on the old language symbols in the partial latent alignment 112. Thus, the neural network 110 can generate the final latent alignment 114 in a fixed amount of time without relying on erroneous conditional independence assumptions.

[0037] After generating the final potential alignment 114, the neural network 110 can provide the final potential alignment 114 to an output generation engine 120. The output generation engine 120 is configured to process the final potential alignment 114 to generate an output sequence 122. In some implementations, the output generation engine 120 removes only whitespace language symbols from the potential alignment 114 to generate the output sequence 122. In some implementations, the subsystem also removes duplicate lexical language symbols; that is, if the potential alignment 114 includes two or more identical lexical language symbols in a row (in some implementations, even if they are separated by one or more whitespace language symbols), the output generation engine 120 can generate an output sequence 122 that includes only a single copy of the lexical language symbol at the corresponding position in the output sequence 122.

[0038] In some implementations, the sequence modeling system 100 can generate the output sequence 122 from the input sequence 102 even if there is no monotonic underlying alignment between the input sequence 102 and the output sequence 122 .

[0039] For example, the sequence modeling system 100 can generate the output sequence 122 by augmenting or modifying the input sequence 102 to generate a modified input sequence that satisfies the monotonic potential alignment property. As another example, the sequence modeling system 100 can generate the output sequence 122 by first generating a modified output sequence that satisfies the monotonic potential alignment property and then processing the modified output sequence to generate the output sequence 122.

[0040] As a specific example, for example, in some machine translation use cases, the input sequence 102 may be shorter than the output sequence 122, that is, the input sequence 102 may not satisfy the requirement of |y|≤|x|. In some implementations, the sequence modeling system 100 may upsample the input sequence 102 to generate a modified input sequence x', such that the modified input sequence is longer than the output sequence 122. In some such implementations, the system may upsample the input sequence by repeating each language symbol in the input sequence s times, such that |x'|=s·|x|. In some other implementations, each language symbol in the input sequence x is represented as a vector of length d, such that the input sequence x can be represented as an input matrix In these implementations, a modified input matrix X may be generated by performing a linear transformation on the input matrix X To upsample the input sequence. For example, X′s i：s(i+1)-1 =WX i +b, where and is a learnable parameter.

[0041] In some implementations, the sequence modeling system 100 can be configured to generate the output sequence 122 even if there is no monotonic underlying alignment between the input sequence 102 and the output sequence 122. As a specific example, for example, in some machine translation use cases, the sequence modeling system 100 can receive an input sequence 102 for which there is no potential alignment that satisfies the monotonicity requirement with the corresponding output sequence 122. For example, when translating from a first language to a second language, there is often a reordering of words such that the linguistic symbols of the input sequence 102 are not monotonically aligned with the linguistic symbols of the output sequence 122. In these implementations, the sequence modeling system 100 can still generate the output sequence 122 because the depth of the neural network 110 (which may include, for example, a stack of transformer self-attention layers) can allow the system 100 to automatically learn through training to reorder the input sequence 102 such that the output embeddings of the neural network 110 have an approximately monotonic alignment with the output sequence 112. Figure 4 Discuss this process in more detail.

[0042] Figures 2A-2C is a diagram of example partial latent alignments between an input sequence and an output sequence. For example, the input sequence may represent audio data, while the output sequence may represent corresponding text data. In response to processing the input sequence and the previous partial latent alignments, each partial latent alignment may be generated by a neural network such as Figure 1 The neural network 110 depicted in is generated at the corresponding time step.

[0043] refer to Figure 2A, each language symbol in the first partial latent alignment 202 is a masked language symbol. The first partial latent alignment may be an initial partial latent alignment before the first time step (ie, before the neural network processes the input sequence).

[0044] The first portion of potential alignments 202 is divided into blocks, each block comprising four consecutive speech symbols. Figure 2A These blocks are depicted with thick lines in the . Figure 2A , three blocks of four language symbols are illustrated, but in general, some potential alignments may include more blocks, where each block may include any number of language symbols. For example, each block may include 2, 4, 8, 16, 32, 64, 128, 256, or 512 language symbols.

[0045] At each time step, the neural network may select a mask language symbol in each block and generate a new language symbol (e.g., a blank language symbol or a vocabulary language symbol) to replace the selected mask language symbol.

[0046] The second partial potential alignment 204 is the output of the neural network after the first time step, i.e., the output of the neural network in response to processing the input sequence and the first partial potential alignment 202. For each block, the neural network has generated a new non-masked language symbol. In particular, the neural network has generated the "A" language symbol in the first block, the blank language symbol in the second block, and the "F" language symbol in the third block. Note that the neural network does not necessarily have to select the first language symbol in each block; for example, the neural network selected the final language symbol in the third block. Figure 4 and Figure 5 Discusses an example procedure for selecting mask language symbols in a block.

[0047] The third partial potential alignment 206 is the output of the neural network after the second time step, i.e., the output of the neural network in response to processing the input sequence and the second partial potential alignment 204. In particular, the neural network has generated a blank language symbol in the first block, an “E” language symbol in the second block, and a blank language symbol in the third block.

[0048] Fourth partial potential alignment 208 is the output of the neural network after the third time step, i.e., the output of the neural network in response to processing the input sequence and third partial potential alignment 206. In particular, the neural network has generated "B" language symbols in the first block, "D" language symbols in the second block, and blank language symbols in the third block.

[0049] The fifth partial potential alignment 210 is the output of the neural network after the fourth time step, i.e., the output of the neural network in response to processing the input sequence and the fourth partial potential alignment 208. In particular, the neural network has generated the “C” language symbol in the first block, the blank language symbol in the second block, and the blank language symbol in the third block.

[0050] The fifth portion of potential alignments 210 is the final potential alignment because each of the mask language symbols of the first portion of potential alignments 202 has been removed. Figure 1 The output generation engine 120 depicted in FIG. 1 may process the fifth potential alignment 210 to generate an output sequence. For example, the system may remove each of the whitespace language symbols from the fifth potential alignment 210 to generate the output sequence "ABCDEF".

[0051] Note that at the second time step (i.e., the time step at which the neural network generated the third partial potential alignment 206), the neural network generated two adjacent language symbols in the third partial potential alignment 206. Specifically, the neural network generated the final language symbol (the "E" language symbol) in the second block and the first language symbol (the blank language symbol) in the third block. Generating consecutive language symbols of a partial potential alignment in parallel may be uncommon, as only a single language symbol per block is generated at each time step; however, this may occur, as in this example, when the neural network generates the final language symbol in the first block and the first language symbol in the subsequent block.

[0052] By generating partially aligned consecutive tokens in parallel, the neural network implicitly makes a conditional independence assumption about adjacent tokens. As mentioned previously, this conditional independence assumption can be problematic, especially for adjacent tokens that are a priori highly dependent on each other. Therefore, at each time step, the neural network can choose tokens in each block to avoid generating consecutive tokens. Figure 2B and Figure 2C Two strategies for doing this are described in .

[0053] refer to Figure 2B , the neural network can retain the final language symbol in each block to be generated in the final time step. Therefore, the neural network can avoid generating the final language symbol in the first block and the first language symbol in the subsequent blocks in parallel.

[0054] As a specific example, if the neural network is configured to select the masked language symbol in each block with the highest likelihood among all the masked language symbols in the block as the corresponding vocabulary language symbol or blank language symbol (as described below with reference to Figure 4described in more detail), the neural network may determine not to select the final language symbol in the block; if the final language symbol in a particular block has the highest likelihood, the neural network may select a masked language symbol with the next highest likelihood.

[0055] For example, at a first time step, the neural network may process a first partial potential alignment 212 consisting entirely of masked language symbols to generate a second partial potential alignment 214. In particular, the neural network (constrained to generate language symbols in one of the first three positions in each block) may generate an "A" language symbol in the first block, a blank language symbol in the second block, and a blank language symbol in the third block.

[0056] At a second time step, the neural network (again limited to the first three positions in each block) can generate a third partial potential alignment 216 by generating a blank language symbol in the first block, a "D" language symbol in the second block, and a blank language symbol in the third block.

[0057] At a third time step, the neural network (again limited to the first three positions in each block) may generate a fourth partial potential alignment 218 by generating a “B” language symbol in the first block, a blank language symbol in the second block, and a blank language symbol in the third block.

[0058] Finally, at a fourth time step, the neural network may generate a final language symbol in each chunk to generate a final potential alignment 220. In particular, the neural network may generate a "C" language symbol at the final position of the first chunk, an "E" language symbol at the final position of the second chunk, and an "F" language symbol at the final position of the third chunk.

[0059] refer to Figure 2C , in order to avoid generating the final language symbol in the first block and the first language symbol in the subsequent blocks in parallel, the neural network can split each block into two sub-blocks. For each block, the first half of the language symbols of the block can be in the first sub-block, and the second half of the language symbols of the block can be in the second sub-block. For each block, Figure 2C The first sub-block is depicted with a lighter grayscale, while the second sub-block is depicted with a darker grayscale.

[0060] The neural network may alternate between generating language symbols in the first sub-block of each block and in the second sub-block of each block. For example, the system may generate language symbols in the first sub-block of each block at odd time steps and generate language symbols in the second sub-block of each block at even time steps.

[0061] As a specific example, at a first time step, the neural network can process the first partial potential alignment 222 to generate the second partial potential alignment 224. The neural network is constrained to generate language symbols in the first sub-block of each block. In particular, the neural network generates an "A" language symbol in the first sub-block of the first block, a blank language symbol in the first sub-block of the second block, and a blank language symbol in the first sub-block of the third block.

[0062] At a second time step, the neural network can process the second partial potential alignment 224 to generate a third partial potential alignment 226. The neural network is constrained to generate language symbols in the second sub-block of each block. Specifically, the neural network generates a blank language symbol in the second sub-block of the first block, an "E" language symbol in the second sub-block of the second block, and an "F" language symbol in the second sub-block of the third block.

[0063] At a third time step, the neural network can process the third partial potential alignment 226 to generate a fourth partial potential alignment 228. The neural network is again restricted to generating language symbols in the first sub-block of each block. Specifically, the neural network generates a "B" language symbol in the first sub-block of the first block, a blank language symbol in the first sub-block of the second block, and a blank language symbol in the first sub-block of the third block.

[0064] At a fourth time step, the neural network can process the fourth partial potential alignment 228 to generate a final potential alignment 230. The neural network is again restricted to generating language symbols in the second sub-block of each block. Specifically, the neural network generates a "C" language symbol in the second sub-block of the first block, a "D" language symbol in the second sub-block of the second block, and a blank language symbol in the second sub-block of the third block.

[0065] Figure 3 is a flow chart of an example process 300 for processing an input sequence to generate an output sequence. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a sequence modeling system appropriately programmed according to the present specification, such as Figure 1 The sequence modeling system 100 depicted in FIG. 1 may perform process 300 .

[0066] The system obtains an input sequence having a corresponding input language symbol at each input position in a plurality of input positions (step 302).The output sequence includes a corresponding output language symbol from a vocabulary of output language symbols at each output position in the plurality of output positions.

[0067] The system determines a plurality of blocks from the input sequence (step 304). Each block inputs a plurality of consecutive input positions of the input sequence.

[0068] The system processes the input sequence using a neural network to generate potential alignments of the input sequence (step 306).The potential alignments include an output language symbol from the vocabulary of output language symbols or a blank language symbol at each input position.

[0069] The processing may include receiving, at each time step in a plurality of time steps, a partial latent alignment from a previous input time step, wherein the partial latent alignment comprises, at each input position, one of: an output language symbol, a blank language symbol, or a mask language symbol. The neural network may select input positions in each block, wherein the language symbols at the selected input positions of the partial latent alignment in each block are mask language symbols. The neural network may then process i) the partial latent alignment and ii) the input sequence to generate a new latent alignment, wherein the new latent alignment comprises, at the selected input position in each block, either the output language symbol or the blank language symbol.

[0070] The system generates an output sequence from the potential alignment (step 308).For example, the system can remove each whitespace language symbol from the potential alignment to generate the output sequence.

[0071] Figure 4 is a diagram of an example neural network 400. The neural network is configured to process i) an input sequence 402 and ii) a partial latent alignment 404 and generate an updated partial latent alignment 442. For example, the neural network 400 may be Figure 1 The neural network 110 depicted in FIG.

[0072] The neural network 400 may generate a corresponding updated partial latent alignment 404 at each of a plurality of time steps. Prior to the first time step, the partial latent alignment 404 may consist entirely of masked language symbols. Then, at each time step, the neural network 400 may select one or more masked language symbols in the partial latent alignment 404 and replace each selected masked language symbol with a vocabulary language symbol or a blank language symbol. At each time step, the neural network 400 may process the partial latent alignment generated at the previous time step. After the last time step, when each masked language symbol in the partial latent alignment 404 has been replaced, the updated partial latent alignment 442 may be considered the final latent alignment. The updated partial latent alignment 442 may then be considered the final latent alignment, for example, using Figure 1 The final potential alignment is processed by the output generation engine 120 depicted in

[0046] to generate an output sequence.

[0073] The neural network 400 includes an input embedding subnetwork 410 , an alignment embedding subnetwork 420 , a combination engine 430 , and a prediction subnetwork 440 .

[0074] The input embedding subnetwork 410 is configured to process the input sequence 402 and generate an embedding 412 of the input sequence 402. In this specification, an embedding is an ordered set of numerical values ​​representing the input in a specific embedding space. For example, the embedding can be a vector of floating points or other numerical values ​​with a fixed dimension.

[0075] In some implementations, for example, when the input sequence 402 is a sequence of audio samples, the input embedding subnetwork 410 may include one or more one-dimensional convolutional neural network layers. In some implementations in which the input sequence 402 is upsampled, as described above, the input embedding subnetwork 410 may first process the input sequence 402 to generate an initial input embedding, and then upsample the initial input embedding to generate the input embedding 412. In some other implementations in which the input sequence 402 is upsampled, the input embedding subnetwork 410 may first upsample the input sequence 402 to generate an upsampled sequence (e.g., by repeating each language symbol in the input sequence 402 p times), and then process the upsampled sequence to generate the input embedding 412.

[0076] Because the input sequence 402 does not change across the time steps of the neural network 400, in some implementations, the input embedding subnetwork 410 processes the input sequence 402 only once to generate the input embedding 412, and then at each subsequent time step, the neural network 400 uses the same input embedding 412.

[0077] The alignment embedding sub-network 420 is configured to process the partial latent alignment 404 and generate an embedding 432 of the partial latent alignment 404. In some implementations, the alignment embedding sub-network 420 can include one or more one-dimensional convolutional neural network layers.

[0078] The combination engine 430 is configured to obtain the input embedding 412 and the alignment embedding 422, and combine the two embeddings to generate a combined embedding 432. In some implementations, the input embedding 412 and the alignment embedding 422 are of the same size. For example, the combination engine 430 may determine the combined embedding 432 as the sum of the input embedding 412 and the alignment embedding 422. As another example, the combination engine 430 may concatenate the input embedding 412 and the alignment embedding 422 to generate the combined embedding 432. As a specific example, the combination engine 430 may concatenate the two embeddings such that for each linguistic token in the input sequence 402, the representation of the linguistic token in the input embedding 412 and the representation of the linguistic token in the alignment embedding 422 are concatenated. As another example, the combination engine 430 may use a neural network, such as a multilayer perceptron, to process both the input embedding 412 and the alignment embedding 422 to generate the combined embedding 432.

[0079] The prediction subnetwork 440 is configured to process the combined embeddings 432 to generate an updated partial latent alignment 442 .

[0080] For example, for each masked language symbol in partial potential alignment 404, prediction sub-network 440 can generate, for each vocabulary language symbol in the vocabulary, a corresponding likelihood value that the masked language symbol in partial potential alignment 404 should be assigned the vocabulary language symbol. Similarly, prediction sub-network 440 can generate a likelihood value that the blank language symbol should be assigned to the masked language symbol in partial potential alignment 404. Prediction sub-network 440 can then select one or more masked language symbols in partial potential alignment 404 to replace with corresponding vocabulary or blank language symbols to generate an updated partial potential alignment 442. For example, for each selected masked language symbol, prediction sub-network 440 can select the vocabulary language symbol or blank language symbol with the highest corresponding likelihood value.

[0081] As a specific example, the neural network 400 can partition the partial potential alignments 404 into a plurality of blocks as described above. For each block, the prediction sub-network 440 can determine the specific mask language symbol with the single highest likelihood score for the corresponding vocabulary language symbol or blank language symbol from all remaining mask language symbols in the block (i.e., the "arg max" mask language symbol in the block). The prediction sub-network 440 can then select the specific mask language symbol and replace it with the vocabulary language symbol or blank language symbol with the highest likelihood score. In some implementations, the prediction sub-network 440 performs these calculations in parallel across all blocks.

[0082] As another specific example, the prediction subnetwork 440 can determine one or more specific mask language symbols (i.e., one or more "arg max" mask language symbols) that have a single highest likelihood score for a corresponding vocabulary language symbol or blank language symbol from all remaining mask language symbols in the entire partial potential alignment 404. For example, the prediction subnetwork 440 can select a fixed number of specific mask language symbols at each time step, where the fixed number of specific mask language symbols can depend on the length of the input sequence 402 to ensure that the output sequence is generated in a fixed number of time steps, as described above. The prediction subnetwork 440 can then select the specific mask language symbols and then replace the specific mask language symbols with the corresponding vocabulary language symbol or blank language symbol having the highest likelihood score. In other words, the neural network 400 can determine that the partial potential alignment 404 is a single block and greedily select the one or more mask language symbols with the highest likelihood score at each time step.

[0083] As another specific example, prediction sub-network 440 can randomly select one or more remaining mask language symbols for each block of partial potential alignments 404. For example, prediction sub-network 440 can randomly and consistently select the remaining mask language symbols in each block. For each selected mask language symbol, prediction sub-network 440 can then determine the vocabulary language symbol or blank language symbol with the highest likelihood score.

[0084] In some implementations, the prediction subnetwork 440 can use a self-attention subnetwork to process the combined embedding 432. As a specific example, the self-attention subnetwork can include a stack of one or more transformer self-attention layers (Vaswani et al., arXiv:1706.03762). In implementations where the latent alignment between the input sequence 402 and the output sequence is not monotonic, as described above, the self-attention subnetwork can be trained to automatically reorder the representations of the language symbols in the combined embedding 432 to generate an updated combined embedding with a monotonic latent alignment with the output sequence. That is, because each self-attention layer in the self-attention subnetwork processes the entire sequence generated by the previous layer (i.e., has full information about every element in the sequence), the self-attention subnetwork can learn to implicitly reorder the elements of the combined embedding 432 as needed to generate the updated partial alignment 442.

[0085] The prediction subnetwork 440 may include a subnetwork (e.g., a subnetwork including a multilayer perceptron and / or a softmax layer) that generates, for each mask language symbol in the partial potential alignment 404, a probability distribution across possible vocabulary language symbols (and blank language symbols) that can be assigned to the mask language symbol.

[0086] In some implementations, the input embedding subnetwork 410, the alignment embedding subnetwork 420, and the prediction subnetwork 440 can be trained end-to-end. That is, the training system can backpropagate errors through each subnetwork and determine updates to the parameters of each subnetwork in parallel. Figure 5 Discussing this process in more detail, in some other implementations, one or more sub-networks can be "frozen" while the other sub-networks are being trained. That is, the system does not determine updates to the parameters of the "frozen" sub-networks during backpropagation.

[0087] In some implementations, the neural network 400 does not include one or more of the input embedding subnetwork 410 or the alignment embedding subnetwork 420. That is, the combination engine 430 can directly obtain the input sequence 402 and / or the partial latent alignment 404 to generate the combined embedding 432, as described above.

[0088] Figure 5is a diagram of an example training system 500. Training system 500 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0089] The training system 500 is configured to train a neural network, e.g. Figure 1 The neural network 100 depicted in FIG. 5 is configured to receive an input sequence and process the input sequence to generate an output sequence. The training system 500 includes a policy sample engine 510 , a training engine 520 , and a parameter storage 530 .

[0090] The strategy sample engine 510 is configured to generate training examples for training the neural network. Each training example can include i) a training input sequence 502, ii) a training portion potential alignment 504, and iii) a ground truth output sequence 506. The ground truth output sequence 506 represents the output that the neural network should generate in response to processing the training input sequence 502 and the training portion potential alignment 504.

[0091] In some implementations, the policy sample engine 510 samples training examples from an “expert” policy φ. Specifically, for a given input sequence 502 and a ground truth output sequence 506, the policy sample engine 510 may sample from a distribution q over all possible potential alignments under the expert policy φ φ The expert policy can be a pre-trained model configured to perform the same task as the neural network (e.g., speech recognition or machine translation). As a specific example, the expert policy φ can be an autoregressive neural network that performs well but takes too much time to generate the output sequence at inference time.

[0092] In some implementations, the policy sample engine 510 does not directly φ Instead, we sample from different strategies φ′ generated according to the expert strategy φ Sampling. The memory cost of computing the probability of each alignment under φ for each training example and storing them offline can be prohibitively expensive, so the more tractable q is used in its place φ′ may be a computationally feasible replacement.

[0093] In some implementations, the strategy sample engine 510 performs the task by first computing, for each (x, y) training example (where x represents the training input sequence 502 and y represents the example ground truth output sequence 506), the empirical best alignment of x under φ. Right now,

[0094]

[0095] Using this expression, Dynamic programming can be used to efficiently and accurately calculate the policy sample engine 510 can then determine by adding the noise distribution N Right now, As a specific example, N may randomly shift the alignment to the left or right. As another specific example, N may randomly switch one or more pairs of language symbols.

[0096] In some other implementations, the policy sample engine 510 will φ′ Set to q θ′ , where q θ′ is a fixed distribution created from a stale copy θ′ of the parameters of the neural network, e.g., the current parameter values ​​532 stored in the parameter store 530. That is, the policy sample engine 510 uses its own stale copy θ′ of the neural network θ as the expert policy.

[0097] When training examples (e.g., or q φ ) to train a neural network, the strategic sample engine 510 may "mask" certain language symbols of the sampled potential alignment a to generate the training partial potential alignment 504. That is, the strategic sample engine 510 may generate the training partial potential alignment 504 by selecting N language symbols from each block of the sampled potential alignment a and replacing them with masked language symbols. This can be written as using the masking function To process the sampled potential alignment a. In some implementations, r(a) is a Bernoulli distribution or a uniform distribution.

[0098] In some implementations, the policy sample engine 510 stores a set of And it can be computationally fast and efficient to store samples N on the fly during training of the neural network to generate new training portion alignments 504. In some other implementations, the policy sample engine 510 generates each training portion alignment 504 before training.

[0099] Parameter storage 530 is configured to store current values ​​532 of parameters of the neural network.

[0100] The training engine 520 is configured to perform training of the neural network, i.e., determine updates to the parameters of the neural network. In particular, at each training time step in a plurality of training time steps, the training engine 520 obtains from the policy sample engine 510 i) a training input sequence 502, ii) a training partial alignment 504, and iii) a ground truth output sequence 506 corresponding to the training input sequence 502. The training engine 520 may also obtain current values ​​532 of the parameters of the neural network from the parameter storage 530.

[0101] The training engine 520 may then process the training input sequence 502 and the training partial alignment 504 to generate an updated partial alignment, as described above with reference to Figure 1 and Figure 4 The training engine 520 can then determine updates to the parameters of the neural network based on the difference between the updated partial alignment and the ground-truth output sequence 506. In some implementations, the training engine 520 executes the neural network for only one or a few time steps to generate a still incomplete updated partial alignment (i.e., still including one or more masked language symbols), and then determines the difference between the incomplete updated partial alignment and the ground-truth output sequence 506 (e.g., by ignoring the remaining masked language symbols and comparing the vocabulary language symbols in the updated partial alignment with the corresponding language symbols in the ground-truth output sequence 506). In some other implementations, the training engine 520 generates a final alignment (i.e., fully completing the training partial alignment 504), and then determines the difference between the final alignment and the output sequence 506 (e.g., by discarding blank language symbols in the final alignment and comparing the vocabulary language symbols in the final alignment with the language symbols in the ground-truth output sequence 506).

[0102] In some implementations, the training engine 520 may use imitation learning to imitate the expert policy φ′. In some such implementations, the training engine 520 may use the following objective function:

[0103]

[0104] In some other implementations, the training engine 520 may use dynamic programming to calculate the objective function. In some such implementations, the training engine 520 may use the following objective function:

[0105]

[0106] in Returns the value from the distribution q φ′ Extracted from ×r The set of all possible new alignments that are compatible. The inner logarithm term can be computed exactly and efficiently through dynamic programming.

[0107] Once the objective function is calculated, the training engine 520 can use backpropagation and stochastic gradient descent to determine updates to the parameters θ of the neural network.

[0108] After updating the parameters of the neural network, training engine 520 may provide updated parameter values ​​522 to parameter storage 530 .

[0109] After completing training, the training system 500 can output final training values ​​534 for the parameters of the neural network. In some implementations, the training system 500 can determine that training is complete after processing a predetermined number of training examples. In some other implementations, the training system 500 can determine that training is complete after a performance metric of the neural network (e.g., prediction accuracy of a validation or test data set) exceeds a predetermined threshold. In some other implementations, the training system 500 can determine that training is complete after the incremental improvement in the performance metric of the neural network across multiple training time steps falls below a predetermined threshold, i.e., after the performance of the neural network no longer significantly improves.

[0110] For example, the training system 500 can provide the trained parameter values ​​534 to an inference system that is configured to receive an input sequence and process the input sequence using the trained neural network to generate a network output. In some implementations, the inference system can be deployed on a user's local device. In some other implementations, the inference system can be deployed on a cloud system, i.e., a distributed computing system having multiple computing nodes (e.g., hundreds or thousands of computing nodes) in one or more locations.

[0111] In some implementations, the neural network segments the input sequence into blocks of the same block size B during training and inference. As specific examples, the block size B can be 2, 4, 8, 16, or 32 during both training and inference. In some other implementations, the neural network can segment the input sequence into blocks of different sizes during training and inference. As specific examples, the system can use a block size of B=8 during training and a block size of B=16 or 32 during inference, or a block size of B=4 during training and a block size of B=8 or 16 during inference.

[0112] In some implementations, the neural network may segment the input sequence into chunks during training, but then not segment the input sequence into chunks during inference. For example, during inference, the neural network may use a greedy top-k strategy, in which the neural network selects k arg max input language symbols (as described above with reference to Figure 4 (described above), regardless of the position of the language symbols in the potential alignment. To ensure that the system generates potential alignments in exactly B time steps during training, the neural network can choose k = |x| / B. In some such implementations, the neural network may not allow adjacent language symbols to be selected in the same time step.

[0113] This specification uses the term "configured to" in connection with systems and computer program components. With respect to a system of one or more computers to be "configured to" perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, when executed, causes the system to perform the operation or action. With respect to one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operation or action.

[0114] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, to be executed by a data processing apparatus or to control the operation of the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information to be sent to a suitable receiver device for execution by the data processing apparatus.

[0115] The term "data processing apparatus" refers to data processing hardware and includes all types of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may be or also include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an operating environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0116] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines, or code). A computer program may be deployed to run on one computer or on multiple computers, the multiple computers being located at one site or distributed across multiple sites and interconnected by a data communications network.

[0117] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and can be stored in one or more locations on a storage device. Thus, for example, an index database may include multiple data collections, each of which can be organized and accessed differently.

[0118] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0119] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., an FPGA or ASIC) or a combination of special purpose logic circuitry and one or more programmed computers.

[0120] The computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and numbers. The central processing unit and memory can be supplemented or incorporated into them by dedicated logic circuits. Typically, a computer will also include or be operably connected to receive and / or transmit data from and / or to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.

[0121] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0122] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, a keyboard, and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user in exchange.

[0123] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0124] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0125] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or a middleware component, such as an application server, or a front-end component, such as a client computer with a graphical user interface, a web browser, or an application program through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0126] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers, and each has a client-server relationship with the other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, for example, the results of a user interaction, may be received from the device at the server.

[0127] In addition to the above embodiments, the following embodiments are also innovative:

[0128] Embodiment 1 is a method of generating, from an input sequence having a corresponding input language symbol at each of a plurality of input positions, an output sequence having a corresponding output language symbol from a vocabulary of output language symbols at each of a plurality of output positions, the method comprising:

[0129] receiving the input sequence;

[0130] determining a plurality of blocks, wherein each block comprises a plurality of consecutive input positions from the input positions;

[0131] processing the input sequence using a neural network to generate a potential alignment of the input sequence, wherein the potential alignment comprises, at each input position of the input position, an output language symbol from a vocabulary of the output language symbol or a blank language symbol, the processing comprising, at each input time step in a plurality of input time steps:

[0132] receiving a partial latent alignment from a previous input time step, wherein the partial latent alignment comprises, at each of the input positions, one of: an output language symbol, a blank language symbol, or a mask language symbol;

[0133] selecting input positions in each block, wherein the language symbols at the selected input positions that are partially potentially aligned in each block are mask language symbols; and

[0134] processing i) the partial potential alignment and ii) the input sequence using the neural network to generate a new potential alignment, wherein the new potential alignment includes an output language symbol or a blank language symbol at the selected input position in each block; and

[0135] The output sequence is generated using the potential alignment.

[0136] Embodiment 2 is the method of embodiment 1, wherein each block includes a same number of input language symbols, and wherein the same number of input language symbols is equal to the number of input time steps.

[0137] Embodiment 3 is the method of any one of embodiments 1 or 2, wherein using the neural network to process i) the partial potential alignment and ii) the input sequence to generate a new potential alignment comprises:

[0138] processing the partial latent alignment using an embedding sub-network to generate a partial latent alignment embedding;

[0139] Processing the input sequence using a convolutional subnetwork to generate an input sequence embedding;

[0140] combining the partial latent alignment embedding and the input sequence embedding to generate a combined embedding; and

[0141] The combined embedding is processed using a self-attention sub-network to generate the new latent alignment.

[0142] Embodiment 4 is the method of any one of embodiments 1-3, wherein the input language symbol is an audio sample and the output language symbol is a text sample.

[0143] Embodiment 5 is the method of any of embodiments 1-4, wherein the neural network has been trained by updating parameters θ of the neural network using an objective function that is marginal over all possible new partial potential alignments that are compatible with a particular partial potential alignment.

[0144] Embodiment 6 is the method of embodiment 5, wherein the objective function is:

[0145]

[0146] where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible potential alignments of x under the pseudo-expert policy φ′, r(a) is the distribution over all possible mask permutations of the potential alignments of x, and Returns the value from the distribution q φ′ ×r extracts the specific part of the potential alignment The set of all possible new partial potential alignments that are compatible.

[0147] Embodiment 7 is the method of any of embodiments 1-4, wherein the neural network has been trained by updating parameters θ of the neural network using an objective function that calculates loss according to a pseudo-expert strategy.

[0148] Embodiment 8 is the method of embodiment 7, wherein the objective function is:

[0149]

[0150] where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible alignments of x under the pseudo-expert strategy φ′, and r(a) is the distribution over all possible mask permutations of the alignments of x.

[0151] Embodiment 9 is the method of any one of embodiments 6 or 8, wherein where N is the noise distribution and is the best empirical alignment under the expert strategy φ,

[0152]

[0153] Among them, q φ is the distribution over all possible alignments of x under the expert policy φ.

[0154] Embodiment 10 is the method of embodiment 9, wherein Compute using dynamic programming.

[0155] Embodiment 11 is the method of any one of embodiments 6 or 8, wherein q φ′ =q θ′ , where q θ′ is a fixed distribution created from a stale copy θ′ of the parameters θ of the neural network.

[0156] Embodiment 12 is the method of any one of embodiments 5-11, wherein training the neural network comprises:

[0157] For a particular potential alignment a~q for a particular input sequence x φ′ Take samples;

[0158] Sample a specific partial potential alignment by sampling a specific mask permutation from r and applying it to a

[0159] Using the neural network to process the specific portion of the potential alignment and the specific input sequence x to generate a prediction;

[0160] calculating the objective function;

[0161] calculating an error in the prediction using the calculated objective function;

[0162] The error is back-propagated through the neural network to determine updates to parameters θ of the neural network.

[0163] Embodiment 13 is the method of embodiment 12, wherein the objective function is calculated using dynamic programming.

[0164] Embodiment 14 is the method of any of embodiments 5-13, wherein r(a) is a Bernoulli or uniform distribution.

[0165] Embodiment 15 is the method of any of embodiments 1-14, wherein selecting the input position in each block comprises computing the arg max input position for each block in parallel across all blocks.

[0166] Embodiment 16 is the method of any one of embodiments 1-15, wherein selecting the input position in each block comprises:

[0167] At each of a plurality of non-final input time steps, selecting an input position such that i) a language symbol at the selected input position in the partial potential alignment is a mask language symbol, and ii) the selected input position is not a final input position in the block; and

[0168] At the final input time step, the final input position in the block is selected.

[0169] Embodiment 17 is the method of any of embodiments 1-15, wherein selecting the input position in each block at each time step in the plurality of time steps comprises alternating between i) selecting the input position in each block that is in a first sub-block of the block and ii) selecting the input position in each block that is in a second sub-block of the block.

[0170] Embodiment 18 is a method of generating, from an input sequence having a corresponding input language symbol at each of a plurality of input positions, an output sequence having a corresponding output language symbol from a vocabulary of output language symbols at each of a plurality of output positions, the method comprising:

[0171] receiving the input sequence;

[0172] determining a plurality of blocks, wherein each block comprises a plurality of consecutive input positions from the input positions;

[0173] processing the input sequence using a neural network to generate a potential alignment of the input sequence, wherein the potential alignment comprises, at each input position of the input position, an output language symbol from a vocabulary of the output language symbol or a blank language symbol, the processing comprising, at each input time step in a plurality of input time steps:

[0174] receiving a partial latent alignment from a previous input time step, wherein the partial latent alignment comprises, at each of the input positions, one of: an output language symbol, a blank language symbol, or a mask language symbol;

[0175] selecting input positions in each block, wherein the language symbols at the selected input positions that are partially potentially aligned in each block are mask language symbols; and

[0176] processing i) the partial potential alignment and ii) the input sequence using the neural network to generate a new potential alignment, wherein the new potential alignment includes an output language symbol or a blank language symbol at the selected input position in each block; and

[0177] The output sequence is generated using the potential alignment.

[0178] Embodiment 19 is the method of embodiment 18, wherein each block includes a same number of input language symbols, and wherein the same number of input language symbols is equal to the number of input time steps.

[0179] Embodiment 20 is the method of any one of embodiments 18 or 19, wherein processing i) the partial potential alignment and ii) the input sequence using the neural network to generate a new potential alignment comprises:

[0180] processing the input sequence using a first embedding subnetwork to generate an input sequence embedding;

[0181] processing the partial latent alignment using a second embedding sub-network to generate a partial latent alignment embedding;

[0182] combining the partial latent alignment embedding and the input sequence embedding to generate a combined embedding; and

[0183] The combined embedding is processed using a self-attention sub-network to generate the new latent alignment.

[0184] Embodiment 21 is the method of any one of embodiments 18-20, wherein the input language symbol is an audio sample and the output language symbol is a text sample.

[0185] Embodiment 22 is the method of any one of embodiments 18-20, wherein the input language symbol is a text sample in a first language and the output language symbol is a text sample in a second language.

[0186] Embodiment 23 is the method of any one of embodiments 18-22, wherein using the neural network to process i) the partial potential alignment and ii) the input sequence to generate a new potential alignment comprises:

[0187] upsampling the input sequence to generate a modified input sequence; and

[0188] The neural network is used to process i) the partial potential alignment and ii) the modified input sequence to generate the new potential alignment.

[0189] Embodiment 24 is the method of any of embodiments 18-23, wherein the neural network has been trained by updating parameters θ of the neural network using an objective function that marginalizes over all possible new partial potential alignments that are compatible with a particular partial potential alignment.

[0190] Embodiment 25 is the method of embodiment 24, wherein the objective function is:

[0191]

[0192] where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible potential alignments of x under the pseudo-expert policy φ′, r(a) is the distribution over all possible mask permutations of the potential alignments of x, and Returns the value from the distribution q φ′ ×r extracts the specific part of the potential alignment The set of all possible potential alignments of the new part that are compatible.

[0193] Embodiment 26 is the method of any of embodiments 18-23, wherein the neural network has been trained by updating parameters θ of the neural network using an objective function that calculates loss according to a pseudo-expert strategy.

[0194] Embodiment 27 is the method of embodiment 26, wherein the objective function is:

[0195]

[0196] where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible alignments of x under the pseudo-expert strategy φ′, and r(a) is the distribution over all possible mask permutations of the alignments of x.

[0197] Embodiment 28 is the method of any one of embodiments 25 or 27, wherein where N is the noise distribution and is the best empirical alignment under the expert strategy φ,

[0198]

[0199] Among them, q φ is the distribution over all possible alignments of x under the expert policy φ.

[0200] Embodiment 29 is the method of embodiment 28, wherein Compute using dynamic programming.

[0201] Embodiment 30 is the method of any one of embodiments 25 or 27, wherein q φ′ =q θ′ , where q θ′ is a fixed distribution created from a stale copy θ′ of the parameters θ of the neural network.

[0202] Embodiment 31 is the method of any one of embodiments 24-30, wherein training the neural network comprises:

[0203] For a particular potential alignment a~q for a particular input sequence x φ′ Take samples;

[0204] Sample a specific partial potential alignment by sampling a specific mask permutation from r and applying it to a

[0205] Using the neural network to process the specific portion of the potential alignment and the specific input sequence x to generate a prediction;

[0206] calculating the objective function;

[0207] calculating an error in the prediction using the calculated objective function;

[0208] The error is back-propagated through the neural network to determine updates to parameters θ of the neural network.

[0209] Embodiment 32 is the method of embodiment 31, wherein the objective function is calculated using dynamic programming.

[0210] Embodiment 33 is the method of any of embodiments 24-32, wherein r(a) is a Bernoulli or uniform distribution.

[0211] Embodiment 34 is the method of any of embodiments 18-33, wherein selecting the input position in each block comprises computing the arg max input position for each block in parallel across all blocks.

[0212] Embodiment 35 is the method of any one of embodiments 18-34, wherein selecting the input position in each block comprises:

[0213] At each of a plurality of non-final input time steps, selecting an input position such that i) a language symbol at the selected input position in the partial potential alignment is a mask language symbol, and ii) the selected input position is not a final input position in the block; and

[0214] At the final input time step, the final input position in the block is selected.

[0215] Embodiment 36 is the method of any of embodiments 18-34, wherein selecting the input position in each block at each time step in the plurality of time steps comprises alternating between i) selecting the input position in each block in a first sub-block of the block and ii) selecting the input position in each block in a second sub-block of the block.

[0216] Embodiment 37 is a system comprising: one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform the method according to any one of embodiments 1-36.

[0217] Embodiment 38 is a computer storage medium encoded with a computer program comprising instructions that, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform a method according to any one of embodiments 1-36.

[0218] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that can be claimed, but rather as descriptions of features for specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be combined or implemented in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve subcombinations or variations of subcombinations.

[0219] Similarly, although operations are depicted in the drawings and recited in the claims as being described in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0220] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A method of generating, from an input sequence having a corresponding input language symbol at each of a plurality of input positions, an output sequence having a corresponding output language symbol from a vocabulary of output language symbols at each of a plurality of output positions, the method comprising: receiving the input sequence; determining a plurality of blocks, wherein each block includes a plurality of input language symbols having a plurality of consecutive input positions from the plurality of input positions, wherein the input language symbols include a first language symbol modality associated with an audio language symbol, a text language symbol, or an image language symbol; processing the input sequence using a neural network to generate a potential alignment of the input sequence, wherein the potential alignment comprises, at each of the plurality of input positions, an output language symbol from a vocabulary of the output language symbol or a blank language symbol, the processing comprising, at each of a plurality of input time steps: receiving a partial latent alignment from a previous input time step, wherein the partial latent alignment comprises, at each input position of the plurality of input positions, one of: an output language symbol, a blank language symbol, or a mask language symbol; selecting input positions in each block, wherein the language symbols at the selected input positions that are partially potentially aligned in each block are mask language symbols; and processing i) the partial potential alignment and ii) the input sequence using the neural network to generate the potential alignment, wherein the potential alignment includes either an output language symbol or a blank language symbol at the selected input position in each block; and The output sequence is generated using the potential alignment, wherein the output language symbols include a second language symbol modality associated with an audio language symbol, a text language symbol, or an image language symbol.

2. The method according to claim 1, wherein Each block includes a same number of input language symbols, and wherein the same number of input language symbols is equal to the number of input time steps.

3. The method according to claim 1, wherein Processing i) the partial potential alignment and ii) the input sequence using the neural network to generate the potential alignment comprises: processing the input sequence using a first embedding subnetwork to generate an input sequence embedding; processing the partial latent alignment using a second embedding sub-network to generate a partial latent alignment embedding; combining the partial latent alignment embedding and the input sequence embedding to generate a combined embedding; and The combined embedding is processed using a self-attention sub-network to generate the latent alignment.

4. The method according to claim 1, wherein The first language symbol modality is an audio language symbol modality including audio sample language symbols, and the second language symbol modality is a text language symbol modality including text sample language symbols.

5. The method according to claim 1, wherein The first language symbol modality is a text language symbol modality including text samples in a first language, and the second language symbol modality is a text language symbol modality including text samples in a second language.

6. The method according to claim 1, wherein Processing i) the partial potential alignment and ii) the input sequence using the neural network to generate the potential alignment comprises: upsampling the input sequence to generate a modified input sequence; and The neural network is used to process i) the partial potential alignment and ii) the modified input sequence to generate the potential alignment.

7. The method according to claim 1, wherein The neural network has been trained by updating parameters θ of the neural network using an objective function that is marginal over all possible new partial latent alignments that are compatible with a particular partial latent alignment.

8. The method according to claim 7, wherein: The objective function is: where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible potential alignments of x under the pseudo-expert policy φ′, r(a) is the distribution over all possible mask permutations of the potential alignments of x, and Returns the value from the distribution q φ′ ×r extracts the specific part of the potential alignment The set of all possible potential alignments of the new part that are compatible.

9. The method according to claim 1, wherein: The neural network has been trained by updating the parameters θ of the neural network using an objective function that calculates the loss according to a pseudo-expert strategy.

10. The method according to claim 9, wherein: The objective function is: where x is the input sequence, a is a specific potential alignment, is a specific partial potential alignment of the potential alignment a, φ′ is a pseudo-expert strategy, q φ′ is the distribution over all possible alignments of x under the pseudo-expert strategy φ′, and r(a) is the distribution over all possible mask permutations of the alignments of x.

11. The method according to claim 8, wherein where N is the noise distribution and is the best empirical alignment under the expert strategy φ, Among them, q φ is the distribution over all possible alignments of x under the expert policy φ.

12. The method according to claim 11, wherein Compute using dynamic programming.

13. The method according to claim 8, wherein q φ′ =q θ′ , where q θ′ is a fixed distribution created from a stale copy θ′ of the parameters θ of the neural network.

14. The method according to claim 7, wherein: Training the neural network includes: For a particular potential alignment a~q for a particular input sequence x φ′ Take samples; Sample a specific partial potential alignment by sampling a specific mask permutation from r and applying it to a Using the neural network to process the specific portion of the potential alignment and the specific input sequence x to generate a prediction; calculating the objective function; calculating an error in the prediction using the calculated objective function; The error is back-propagated through the neural network to determine updates to parameters θ of the neural network.

15. The method according to claim 14, wherein The objective function is calculated using dynamic programming.

16. The method according to claim 7, wherein r(a) is either Bernoulli or uniformly distributed.

17. The method according to claim 1, wherein Selecting the input position in each block includes computing the arg max input position for each block in parallel across all blocks.

18. The method according to any one of claims 1 to 17, wherein The input locations selected in each block include: At each of a plurality of non-final input time steps, selecting an input position such that i) a language symbol at the selected input position in the partial potential alignment is a mask language symbol, and ii) the selected input position is not a final input position in the block; and At the final input time step, the final input position in the block is selected.

19. The method according to any one of claims 1 to 17, wherein: Selecting the input position in each block at each time step in the plurality of time steps includes alternating between: i) selecting the input position in a first sub-block of each block and ii) selecting the input position in a second sub-block of each block.

20. A system for generating an output sequence, comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method according to any one of claims 1-19.

21. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1-19.

Images