Training a neural network to generate structured embeddings

Abstract

A method (200) for training a machine learning model (108) to generate an embedding (106) of an input to the machine learning model, the machine learning model having an encoder (110) that generates the embedding from the input and a decoder (120) that generates an output from the generated embedding, wherein the embedding is partitioned into a sequence of embedding partitions, the embedding partitions each comprising one or more dimensions of the embedding. The method comprises: for a first embedding partition (106A) in the sequence of embedding partitions: performing initial training to train the encoder and a decoder replica (122) corresponding to the first embedding partition; for each particular embedding partition after the first embedding partition in the sequence of embedding partitions: performing incremental training to train the encoder and a decoder replica corresponding to the particular partition.

Description

TECHNICAL FIELD

[0001] This specification relates to training neural networks. BACKGROUND

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0003] This specification describes techniques related to training an encoder-decoder engine to generate structured embeddings that can later be modified in an intuitive manner.

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods performed by one or more computers for training a machine learning model to generate an embedding for an input to the machine learning model, the machine learning model having an encoder that generates the embedding from the input and a decoder that generates an output from the generated embedding, where the embedding is partitioned into a sequence of embedding partitions, each of the embedding partitions including one or more dimensions of the embedding, the method including: for a first embedding partition in the sequence of embedding partitions: performing initial training to train the encoder and a decoder replica corresponding to the first embedding partition, where the decoder replica receives, as input during the initial training, a first masked embedding that includes (i) values generated by the encoder for the first embedding partition and (ii) masking values for all subsequent embedding partitions in the sequence; and for each particular embedding partition after the first embedding partition in the sequence of embedding partitions: performing incremental training to train the encoder and a decoder replica corresponding to the particular partition, where the decoder replica corresponding to the particular partition receives, as input during the incremental training, an incremental masked embedding for the particular partition that includes (i) values generated by the encoder for the particular embedding partition and each embedding partition preceding the particular embedding partition in the sequence and (ii) masking values for any subsequent embedding partitions following the particular embedding partition in the sequence. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

[0005] In some implementations, performing the incremental training further includes, for each preceding embedding partition that precedes the particular embedding partition in the sequence, training the encoder and a decoder replica corresponding to the preceding embedding partition, wherein the decoder replica receives, during the incremental training, the incremental masked embedding for the preceding partition as input.

[0006] In some implementations, during the incremental training, parameters of the decoder replicas corresponding to the particular partition and the preceding partition are constrained to have the same values.

[0007] In some implementations, performing the initial training to train the encoder and the decoder replica corresponding to the first embedding partition includes: determining a gradient of an objective function with respect to an output generated by the decoder replica; backpropagating the gradient from the decoder replica only to a corresponding portion of the encoder that generates the first embedding partition; and updating respective parameter values of the decoder replica and the corresponding portion of the encoder using the backpropagated gradient.

[0008] In some implementations, performing the incremental training to train the encoder and the decoder replica corresponding to the particular partition includes: determining a gradient of an objective function with respect to an output generated by the decoder replica; backpropagating the gradient from the decoder replica only to a corresponding portion of the encoder that generates the particular embedding partition; and updating respective parameter values of the decoder replica and the corresponding portion of the encoder using the backpropagated gradient.

[0009] In some implementations, the method further includes: determining that the gradient of the objective function has converged to a predetermined value; and in response to the determination, terminating the initial training and beginning the incremental training for a second partition in the sequence.

[0010] In some implementations, the masking values for all subsequent embedding partitions in the sequence are zero.

[0011] In some implementations, the encoder applies an activation function having a fixed output range to the intermediate encoder output to generate the embedding.

[0012] In some implementations, the input is a text unit and the output is a discourse representing the text unit.

[0013] In some implementations, the method further includes, after performing all of the incremental training: receiving a new input; processing the new input using the trained encoder to generate an initial embedding for the new input; receiving user input that modifies values of a given embedding partition in the initial embedding to generate a new embedding; and processing the new embedding using the trained decoder to generate an output for the new embedding.

[0014] Particular embodiments that enable the subject matter described in this specification can be implemented to realize one or more of the following advantages. Generally, an encoder-decoder engine includes an encoder and a decoder, both of which are implemented as machine learning models, e.g., neural networks. The engine can be configured to perform a machine learning task by encoding a received input into an embedding comprising an ordered set of values and then decoding the embedding to generate an output. Following conventional training techniques, the embedding representation is typically entangled, e.g., a single value included in the embedding can represent multiple output features. In practice, having an entangled embedding representation has several problems. One problem is that it increases the difficulty of optimization, which further leads to a prolonged training process. Another problem is that the entangled representation reduces the overall explainability of the embedding to a human user and prevents the embedding from being modified in a desired manner after generation. However, this specification discloses a technique for training an encoder-decoder engine in a structured process. As a result of the structured training process, the generated embedding is a disentangled representation of the output. In particular, the embedding includes multiple partitions, each of which represents a different feature of the output. Moreover, a primary feature of the output is typically represented by a first embedding partition. Having the ability to generate an embedding that represents the output features in such an explainable manner enables a human user to easily modify a particular feature of the output by adjusting the corresponding value within the embedding on which the output is based. This ability, in turn, broadens the range of applications for the encoder-decoder engine to include use cases that often require fine-tuning of the generated output. For example, in a text-to-speech application, different human users can want to pronounce certain input text in a particular way that is different from the initial synthesized pronunciation.

[0015] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a block diagram of an example encoder-decoder system.

[0017] Figure 2 is a flow diagram of an example process for training an encoder-decoder engine.

[0018] Figure 3A is a flow diagram of an example process for performing initial training to train an encoder-decoder engine.

[0019] Figure 3B is a flow diagram of an example process for performing incremental training to train an encoder-decoder engine.

[0020] Figure 4A is an illustration of an example of performing initial training for a first embedding partition to train an encoder-decoder engine.

[0021] Figure 4B is a diagram of an example of performing incremental training on a second embedding partition to train an encoder-decoder engine.

[0022] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0023] This specification describes a system implemented as a computer program on one or more computers in one or more locations that trains an encoder-decoder engine to generate structured embeddings that can later be modified in an intuitive manner by a human user.

[0024] Figure 1 is a block diagram of an example encoder-decoder system 100. The encoder-decoder system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0025] The encoder-decoder system 100 is configured to receive an input 102 and encode the input 102 into an embedding 106 in a lower dimensional space relative to the input 102. In other words, the embedding 106 is a numerical representation in a space with a fixed dimensionality that is lower than the dimensionality of the input 102, i.e., the embedding 106 is an ordered collection of numerical values with a fixed number of values. For example, the embedding 106 can be a fixed size vector or matrix.

[0026] The system 100 then decodes the embedding 106 to an output 112.

[0027] The system 100 uses an encoder-decoder engine 108 to perform the encoding and decoding processes, which in turn includes an encoder 110 and a decoder 120. The encoder-decoder engine 108 can be interchangeably referred to as a machine learning model 108.

[0028] In general, the system 100 can be configured to perform any of a range of machine learning tasks involving an encoding process, a decoding process, or both an encoding process and a decoding process. Depending on the particular task, the input 102 and the output 112 can be any of a variety of types of data, and the input 102 and the output 112 can or can not be the same type. In cases where the same type of data is included in both the input 102 and the output 112, the task performed by the encoder-decoder engine 108 can be referred to as an autoencoding task.

[0029] For example, if the task is an image autoencoding task, the input 102 is structured data (e.g., an array) representing one image, and the output 112 generated by the engine 108 is data representing a reconstructed image.

[0030] As another example, if the task is an audio autoencoding task, the input 102 is data representing audio (e.g., a waveform audio file), and the output 112 generated by the engine 108 is data representing reconstructed audio. Example audio can include sound, speech, music, etc.

[0031] In particular, the encoder 110 and the decoder 120 are each neural networks that can each include one or more neural network layers, e.g., fully connected layers, convolutional layers, recurrent layers, activation layers, etc. Each layer of a neural network generates an output from received input according to current values of a respective set of parameters. The output of preceding layers is used to construct the input of subsequent layers.

[0032] The encoder 110 is configured to receive the input 102 and process the input 102 according to current encoder parameter values to generate the embedding 106 based on the input 102.

[0033] In some implementations, the encoder 110 applies a bounded activation function having a fixed output range to the intermediate encoder output (i.e., the output of a layer preceding an activation layer) to generate the final embedding 106. In other words, the last layer of the encoder 110 is an activation layer that applies an activation function having a fixed output range to the intermediate encoder output generated in the encoder 110 by a layer preceding the activation layer. In general, an activation function transforms a distribution of inputs to the function into a different distribution. A bounded activation function, such as a tanh or sigmoid function, forces the distribution of the generated embedding 106 to be predictable, i.e., within a fixed interval.

[0034] The decoder 120 is configured to receive the generated embedding 106 and process the embedding 106 according to current decoder parameter values to generate the output 112.

[0035] The encoder-decoder system 100 also includes a training engine 130 configured to train the encoder-decoder engine 108 on training inputs 132. In particular, the training engine 130 trains the encoder 110 and the decoder 120 using an iterative training process to determine training values for the parameters of the encoder-decoder engine 108, including the encoder and decoder parameter sets.

[0036] Generally, training quality can be measured by an objective function that describes how well the trained encoder-decoder engine 108 can perform a given machine learning task. In particular, the objective function evaluates how close the generated output for a given input is to a desired target output. Moreover, as described above, the output 112 is generated by the decoder 120 based on processing the embedding 106, which in turn is generated by the encoder 110 based on processing the received input 102. Thus, training quality further determines how well the generated embedding 106 can represent the unique features of the generated output 112.

[0037] In some implementations, the system 100 divides the embedding 106 into a sequence of embedding partitions. Each embedding partition includes one or more dimensions of numerical values. For example, if the embedding 106 is a 5-dimensional vector, i.e., a vector [x1, x2, x3, x4, x5] having 5 numerical values, then the first dimension, i.e., the numerical value x1, can be the first embedding partition in the sequence. Correspondingly, each subsequent dimension can be a subsequent embedding partition that results in a total of five embedding partitions.

[0038] The encoder-decoder engine 108 is trained by appropriately using a structured training process that will be described below with reference to Figure 2 By training the encoder such that, for all generated outputs, each given embedding partition represents the same corresponding feature of the output 112 and each partition corresponds to a different feature than all other partitions, the system trains the encoder-decoder engine 108 to generate outputs 112 that are more accurate and more representative of the input 102. In other words, as a result of the training, a particular feature of the output 112 will be sensitive to changes in the numerical values within one corresponding embedding partition, but relatively invariant to changes in the numerical values within other embedding partitions. Moreover, in some cases, the embedding partitions in the sequence represent different features of the output in an ordered manner. In such cases, the particular feature represented by the first embedding partition corresponds to a primary feature of the output. The primary feature corresponds to the feature that, during training, allows the engine 108 to minimize the error calculated using the objective function that measures training quality. Optionally, the primary feature refers to the most representative characteristic of the output, e.g., the identity of an object depicted in an image output, the tone of a voice recorded in a speech output, etc. In some implementations, in a similar manner, the corresponding features represented by subsequent embedding partitions in the sequence correspond to secondary features of the output, i.e., features that allow the engine 108 to generate outputs having a second-minimal error during training.

[0039] For example, when used in a text-to-speech task in which the input is tokenized data representing one or more text units and the output is a set of utterances (or waveforms of utterances) representing the text units, the trained encoder-decoder engine 108 can be configured to generate a set of embedding partitions that can each represent different prosodic features of the output utterances corresponding to the input text. Example prosodic features can include pitch, length, loudness, timbre, and the like. Thus, one partition can represent pitch, another partition can represent length, and so on. Optionally, where each embedding partition in turn comprises more than one dimension of numerical values, each dimension within one embedding partition can further represent a particular property of the corresponding prosodic feature, e.g., height, slope, or curvature of the pitch.

[0040] In some implementations, after being trained, the system 100 can be configured to receive a new input that is the same type of data as the training input. The system 100 processes the new input using the trained encoder to generate an initial embedding for the new input. The system then processes the initial embedding using the trained decoder to generate an initial output. As a result of the structured training process, the initial embedding comprises multiple partitions that each represent different features of the output.

[0041] For example, if the new input and the training input are both image data for an image reconstruction task, each embedding partition can represent a particular graphical feature of the reconstructed image, e.g., scale, lighting, or color.

[0042] As another example, if the new input and the training input are both text data for a text-to-speech task, each embedding partition can represent a particular prosodic feature of the utterance, e.g., pitch, duration, loudness, or timbre.

[0043] To allow the features of the initial output to be modified after they have been generated, the system 100 can be further configured to receive user input that modifies the values of one or more given embedding partitions in the initial embedding to generate a new embedding, e.g., through an application programming interface (API) provided by the system or through a user interface presentation generated by the system and displayed on a user device. The user input can specify modifications to a set of numerical values within the one or more given embedding partitions. Upon receiving the user input, the system 100 processes the new embedding using the trained decoder to generate a modified output based on the new embedding. The modified output then incorporates the one or more modified features in its representation, e.g., modified colors in a reconstructed image, rescaled pitches in a generated utterance, and the like.

[0044] As a particular example, in an audio processing task, the system processes input data representing audio (e.g., a recording of a user’s voice or a computer-generated voice sample) to generate an initial embedding. The system can be configured to receive user input specifying a modification to a value within one or more embedding partitions. The system then processes the modified embedding to generate a modified audio. The modified audio incorporates one or more user-specified modifications to corresponding features of the input audio.

[0045] On the other hand, if the training engine 130 has trained the encoder-decoder engine 108 in a conventional manner (i.e., not following the structured training process), the trained encoder 110 is likely to generate entangled embeddings 106, i.e., embeddings composed of values each representing more than one feature of the output 112, which in turn is generated by the trained decoder 120 based on processing the embedding 106. In other words, a modification made to any one of the values included in such an embedding 106, after it is generated, will cause more than one feature in the output 112 to change. Additionally, while there are other training techniques with the goal of training the encoder 110 to generate disentangled embeddings 106, the correspondence between different embedding dimensions and features of the output 112 is difficult to identify. Specifically, if another disentangled embedding training technique is used, it will be difficult to determine which embedding dimension corresponds to which feature once the encoder-decoder engine has been trained, as these techniques do not train the encoder-decoder engine in the structured and iterative manner described in this specification. In these cases, it will be difficult, and often impractical, to fine-tune particular features of the generated output in the desired manner.

[0046] Figure 2 is a flowchart of an example method 200 for training an encoder-decoder engine. For convenience, the method 200 will be described as performed by a system of one or more computers located in one or more locations. For example, an encoder-decoder system, e.g., the encoder-decoder system 100 of Figure 1 , appropriately programmed according to this specification, is able to perform the method 200.

[0047] The method 200 can be referred to as a structured training process. Briefly, the structured training process includes an initial training process and one or more incremental training processes.

[0048] In some implementations, prior to or concurrent with performing the method 200, the system trains the encoder-decoder engine 108 based on a decoder (i.e., included in the encoder-decoder system 100 as described in this specification) to generate an output based on processing an embedding. The system then trains the encoder to generate an embedding that, when processed by the decoder, generates the output. The system can train the encoder-decoder engine 108 in this manner prior to performing the method 200, or the system can perform the method 200 concurrently with training the encoder-decoder engine 108 in this manner. Figure 1The decoder 120 in the depicted encoder-decoder engine 108 generates one or more decoder replicas. The one or more decoder replicas are generated to have the same architecture and parameter values as the decoder. Thus, for convenience, the following description will only refer to the decoder replicas, even in cases where the decoder is to be referenced.

[0049] In such implementations, the number of decoder replicas is typically the same as the number of partitions included in the embedding. That is, the system generates a corresponding decoder replica for each embedding partition.

[0050] Furthermore, when performing the method 200, the system synchronously applies any changes made to the parameter values of a particular decoder replica to all other decoder replicas. In this way, the system constrains the parameter values of all decoder replicas to be the same.

[0051] The system receives training inputs (202). Each training input is associated with a target output. The target output is an output that should be generated by the engine by performing a machine learning task that the engine is configured to perform on the training input.

[0052] The system performs initial training for a first embedding partition in a sequence of embedding partitions (204). The first embedding partition includes values from a first, one or more dimensions of the embedding, where the embedding is generated by the engine based on processing a received training input. In particular, the system does not process the complete embedding during the initial training, but only the first embedding partition. That is, the system generates a training output based only on the values included in the first embedding partition. Thus, only such values are updated at the end of each initial training iteration.

[0053] In general, performing the initial training encourages the engine to learn the first embedding partition that generates a representation of a primary feature of the training output. A primary feature refers to a most representative characteristic of the training output, e.g., the identity of an object depicted in an image training output, the tone of voice recorded in a speech training output, etc. Processing the first embedding partition that represents the primary feature of the training output, even without processing the complete embedding, allows the engine to generate a training output that is as close as possible to the target output, thereby reducing the error computed using an objective function that measures the quality of the training.

[0054] The method 200 will be described in more detail below with reference to Figure 3A and Figure 4A The performing of the initial training is described in more detail.

[0055] After the initial training has terminated (e.g., after a predetermined number of initial training iterations have been completed or after the computed gradient of the objective function has converged to a predetermined value), the system proceeds to perform incremental training.

[0056] The system performs incremental training for each particular embedding partition in the sequence that follows the first embedding partition (206). In some implementations, the incremental training begins with a second embedding partition in the sequence. The second embedding partition includes values for one or more dimensions that follow the first embedding partition in the sequence. Similarly, each incremental training process terminates when a certain termination criterion is met, such as after a predetermined number of incremental training iterations have been completed or after the computed gradient of the objective function has converged to a predetermined value.

[0057] Performing incremental training encourages the engine to learn to generate embedding partitions that each represent corresponding features of the training output. This will be described below with reference to Figure 3B and Figure 4B Performing incremental training is described in more detail.

[0058] Figure 3A is a flowchart of an example process 300 for performing initial training to train an encoder-decoder engine. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an encoder-decoder system, e.g., the encoder-decoder system 100 of Figure 1 , appropriately programmed according to the present specification, is able to perform the process 300A.

[0059] The system can repeatedly perform the process 300 for different training inputs in order to perform the initial training, i.e., the training for the first embedding partition in the sequence.

[0060] The system receives a training input associated with a target output (302). The target output is an output that should be generated by the system by performing a machine learning task, where the system is configured to perform the machine learning task based on the training input.

[0061] The system generates a first masked embedding based on processing the training input according to current parameter values of the encoder (304). In particular, the first masked embedding includes (i) values generated by the encoder for the first embedding partition, and (ii) masking values for all subsequent embedding partitions in the sequence of embedding partitions. In some implementations, the system masks the values for all subsequent embedding partitions in the sequence by replacing the values with predetermined values, e.g., zeros.

[0062] Referring now to Figure 4A , an example of performing initial training for a first embedding partition to train an encoder-decoder engine is illustrated. In particular, as Figure 4A illustrated, the system generates a first masked embedding based on processing the training input according to current parameter values of the encoder 110. The first masked embedding includes values generated by the encoder for the first embedding partition 106A and masking values for the rest of the embedding partitions.

[0063] The system generates a training output based on processing the generated first masked embedding using the decoder replica corresponding to the first embedding partition according to the current parameter values (306).

[0064] The system computes a target function with respect to the generated training output (308). The target function evaluates the quality of the generated training output, i.e., by measuring the error between the generated training output and a target output. Typically, the system uses a target function that is well-suited for the machine learning task that the neural network is being trained to perform. For example, an L2 loss function that computes the least squares error between two outputs is a common choice for a target function in regression machine learning tasks involving, e.g., image and speech data.

[0065] The system determines the gradient of the target function (310). For example, as Figure 4A illustrated, the system determines the gradient 142 of the target function with respect to the training output generated by the decoder replica 120.

[0066] The system backpropagates the gradient from the decoder replica to only the corresponding portion of the encoder that generated the first embedding partition (312). For example, as Figure 4A illustrated, the system backpropagates the gradient from the decoder replica 120 to only the corresponding portion of the encoder 110 that generated the first embedding partition 106A.

[0067] Typically, the gradient of the target function is propagated backwards from the decoder layers to the embeddings, and to the encoder layers. Thus, because the embeddings only include encoder-generated values in the first embedding partition, the gradient that is backpropagated to the encoder layers from the embeddings is only passed to the corresponding portion of the encoder that generated values in the first embedding partition (i.e., a subset of the encoder parameters or a portion of the encoder parameter values).

[0068] The system updates the respective parameter values of the decoder replica and the corresponding portion of the encoder, e.g., based on the backpropagated gradient (314). In particular, the system can update the parameter values using any appropriate machine learning training technique. Examples of training techniques include stochastic gradient descent, Adam, and rms-prop.

[0069] By repeatedly performing the process 300, the system trains the encoder to generate a first embedding partition in a sequence that represents the same corresponding feature of all training outputs generated based on the same type of training input. Typically, the feature corresponds to a primary feature of the training outputs.

[0070] Figure 3Bis a flowchart of an example process 350 for performing incremental training to train an encoder-decoder engine. For convenience, the process 350 will be described as being performed by a system of one or more computers located in one or more locations. For example, an encoder-decoder system, e.g., the encoder-decoder system 100, appropriately programmed according to the present specification, is able to perform the process 350. Figure 1 The system 100 is able to perform the process 350 repeatedly for different training inputs in order to perform incremental training, i.e., training for each particular embedding partition after the first embedding partition in the sequence of embedding partitions.

[0071] The system is able to perform the process 350 repeatedly for different training inputs in order to perform incremental training, i.e., training for each particular embedding partition after the first embedding partition in the sequence of embedding partitions.

[0072] The system receives a training input associated with a target output (352). The target output is an output that should be generated by the system by performing a machine learning task for which the system is configured to perform based on the training input.

[0073] The system generates an incremental masked embedding for the particular embedding partition based on processing the training example according to the current parameter values of the encoder (354). Specifically, the incremental masked embedding for the particular partition includes (i) values generated by the encoder for the particular embedding partition and each embedding partition preceding the particular embedding partition in the sequence, and (ii) masking values for any subsequent embedding partitions following the particular embedding partition in the sequence.

[0074] Referring now to Figure 4B , an example of performing incremental training to train an encoder-decoder engine for a second embedding partition is illustrated. Specifically, as Figure 4B illustrated, the system generates an incremental masked embedding for the second embedding partition 106B based on processing the training input according to the current parameter values of the encoder 110. The incremental masked embedding includes values generated by the encoder for the first embedding partition 106A and the second embedding partition 106B and masking values for all other partitions.

[0075] The system generates a training output based on processing the incremental masked embedding for the particular embedding partition according to the current parameter values using the decoder replica corresponding to the particular embedding partition (356).

[0076] The system computes a target function with respect to the generated training output (358).

[0077] The system determines a gradient of the target function computed with respect to the generated training output (360). For example, as Figure 4B illustrated, the system determines a gradient 146 of the target function computed with respect to the training output generated by the decoder replica 122.

[0078] The system backpropagates the gradient from the decoder replica only to the corresponding portion of the encoder that generated the particular embedding partition from the decoder replica (362). That is, the system backpropagates the gradient from the decoder replica only back to the corresponding particular embedding partition. The system then backpropagates the gradient from the particular embedding partition back to the corresponding portion of the encoder that generated the particular embedding partition.

[0079] For example, as Figure 4B illustrated, the system backpropagates the gradient 146 from the decoder replica 122 through the second embedding partition 106B only to the corresponding portion of the encoder 110 that generated the second embedding partition 106B.

[0080] The system updates respective parameter values of the decoder replica and the corresponding portion of the encoder based on the backpropagated gradient, for example (364).

[0081] In some implementations, during each process 350, the system trains all preceding embedding partitions in the sequence in addition to the particular embedding partition. That is, when performing the process 350, the system jointly performs the following steps with the corresponding steps described above.

[0082] In particular, the system also generates a corresponding delta masked embedding for each embedding partition that precedes the particular embedding partition in the sequence (354). The delta masked embedding for each preceding embedding partition includes (i) values generated by the encoder for the preceding embedding partition and each embedding partition that precedes the preceding embedding partition in the sequence, and (ii) masking values for any subsequent embedding partitions that follow the preceding embedding partition in the sequence.

[0083] For example, as Figure 4B illustrated, the system also generates a delta masked embedding for the first embedding partition 106A that precedes the second embedding partition 106B in the sequence of partitions. The first embedding partition 106A includes values generated by the encoder only in the first embedding partition 106A.

[0084] The system generates one or more additional training outputs using the corresponding decoder replica based on processing the one or more delta masked embeddings for each embedding partition that precedes the particular embedding partition (356).

[0085] The system computes a respective objective function with respect to each of the one or more additional training outputs (358). Each objective function measures a corresponding error between the generated training output and a target output.

[0086] The system determines respective gradients of the one or more objective functions computed with respect to the additional training outputs (360). For example, as Figure 4BThe illustrated system computes the objective function with respect to the training output generated by the decoder replica 120.

[0087] The system backpropagates the gradient from the respective decoder replica to only the corresponding portion of the encoder that generated each preceding embedding partition (362). For example, as illustrated, the system backpropagates the gradient 144 from the decoder replica 120 to only the corresponding portion of the encoder 110 that generated the first embedding partition 106A. Figure 4B The illustrated system backpropagates the gradient 144 from the decoder replica 120 to only the corresponding portion of the encoder 110 that generated the first embedding partition 106A.

[0088] The system updates respective parameter values of the corresponding portions of the decoder replica and the encoder, e.g., based on the backpropagated gradient (364).

[0089] By repeatedly performing the process 350, the system trains the encoder to generate subsequent embedding partitions in the sequence that each represent the same corresponding feature of all training outputs generated based on the same type of training input. Typically, each embedding partition corresponds to a different feature than all other partitions.

[0090] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded as propagated signals, e.g., machine-generated electrical, optical, or electromagnetic signals, that are generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

[0091] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special-purpose logic, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0092] A computer program, which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0093] As used in this specification, an "engine" or "software engine" refers to a software implemented input / output system that provides outputs that are different from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any appropriate type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other stationary or portable device that includes one or more processors and computer readable media. Additionally, two or more engines can be implemented on the same computing device, or on different computing devices.

[0094] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) and

[0095] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0096] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0097] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0098] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.

[0099] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0100] Although this specification contains many specifics, these should not be construed as limiting the scope of any invention or restricting the meaning of terms given to particular embodiments described. Some features set forth in the context of separate embodiments can also be implemented in combination, in a single embodiment. Conversely, various features set forth in the context of a single embodiment can also be implemented in multiple embodiments or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0101] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0102] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As another example, the processes depicted in the accompanying figures can not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method of training a machine learning model to generate an input embedding to the machine learning model, the machine learning model having an encoder and a decoder, wherein the encoder generates the embedding from the input, the decoder generates an output from the generated embedding, wherein, The input comprises text units and the output comprises utterances representing the text units, or the input comprises structured data representing images and the output comprises data representing reconstructed images, or the input comprises data representing audio and the output comprises data representing reconstructed audio, and wherein the embedding is partitioned into a sequence of embedding partitions, the embedding partitions each comprising one or more dimensions of the embedding, the method comprising: generating a plurality of decoder replicas, each of the plurality of decoder replicas corresponding to a respective embedding partition in the sequence of embedding partitions, and each of the plurality of decoder replicas having one or more parameter values set equal to one or more parameter values of the decoder; for a first embedding partition in the sequence of embedding partitions, performing initial training to train the encoder and the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition, wherein the decoder replica receives as input, during the initial training, a first masked embedding comprising: values generated by the encoder for the first embedding partition; and masked values for all subsequent embedding partitions in the sequence of embedding partitions; for each particular embedding partition after the first embedding partition in the sequence of embedding partitions, performing incremental training to train the encoder and a particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition, wherein the particular decoder replica corresponding to the particular embedding partition receives as input, during the incremental training, an incremental masked embedding for the particular embedding partition, the incremental masked embedding comprising: values generated by the encoder for the particular embedding partition and each embedding partition preceding the particular embedding partition in the sequence of embedding partitions; and masked values for any subsequent embedding partitions following the particular embedding partition in the sequence of embedding partitions; and during the incremental training, for each preceding embedding partition preceding the particular embedding partition in the sequence of embedding partitions, synchronously applying any changes made to any of the parameter values of the particular decoder replica to decoder replicas of the plurality of decoder replicas corresponding to the preceding embedding partition, such that the one or more parameter values of the particular decoder replica and the one or more parameter values of the decoder replicas of the plurality of decoder replicas corresponding to the preceding embedding partition are constrained to have the same values.

2. The method of claim 1, wherein, Performing the incremental training further comprises, for each preceding embedding partition preceding the particular embedding partition in the sequence of embedding partitions: training the encoder and the particular decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition, wherein the particular decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition receives as input, during the incremental training, an incremental masked embedding for the preceding embedding partition.

3. The method of claim 1, wherein, performing initial training to train the encoder and the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition comprises: determining gradients of an objective function with respect to outputs generated by the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition; backpropagating the gradients from the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition only to a corresponding portion of the encoder that generated the first embedding partition of the sequence of embedding partitions; and updating respective parameter values of the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition and the corresponding portion of the encoder using the backpropagated gradients.

4. The method of claim 1, wherein, performing incremental training to train the encoder and the particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition comprises: determining gradients of an objective function with respect to outputs generated by the particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition; backpropagating the gradients from the particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition only to a corresponding portion of the encoder that generated the particular embedding partition of the sequence of embedding partitions; and updating respective parameter values of the particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition and the corresponding portion of the encoder using the backpropagated gradients.

5. The method of claim 3, further comprising: determining that the gradients of the objective function have converged to a predetermined value; and in response to the determining, terminating the initial training and starting the incremental training for a second embedding partition of the sequence of embedding partitions.

6. The method of claim 1, wherein, the masking values for all subsequent embedding partitions of the sequence of embedding partitions are zero.

7. The method of claim 1, wherein, the encoder applies an activation function having a fixed output range to an intermediate encoder output to generate the embedding.

8. The method of any of claims 1-7, further comprising: after performing all incremental training: receiving a new input; processing the new input using the trained encoder to generate an initial embedding for the new input; receiving user input that modifies values of a given embedding partition in the initial embedding to generate a new embedding; and processing the new embedding using the trained decoder to generate an output for the new embedding.

9. A system for training a machine learning model, the system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for training a machine learning model to generate input embeddings to the machine learning model, the machine learning model having an encoder and a decoder, wherein the encoder generates the embeddings from the input, the decoder generates an output from the generated embeddings, wherein, the input comprises text units and the output comprises discourse representing the text units, or the input comprises structured data representing an image and the output comprises data representing a reconstructed image, or the input comprises data representing audio and the output comprises data representing reconstructed audio, and wherein the embedding is partitioned into a sequence of embedding partitions, the embedding partitions each comprising one or more dimensions of the embedding, the operations comprising: generating a plurality of decoder replicas, each of the plurality of decoder replicas corresponding to a respective embedding partition of the sequence of embedding partitions and each of the plurality of decoder replicas having one or more parameter values set equal to one or more parameter values of the decoder; performing initial training to train the encoder and the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition of the sequence of embedding partitions, wherein the decoder replica receives as input, during the initial training, a first masked embedding comprising: values generated by the encoder for the first embedding partition; and masked values for all subsequent embedding partitions of the sequence of embedding partitions; and performing incremental training to train the encoder and a particular decoder replica of the plurality of decoder replicas corresponding to a particular embedding partition of the sequence of embedding partitions that is subsequent to the first embedding partition, wherein the embedding decoder replica corresponding to the particular embedding partition receives as input, during the incremental training, an incremental masked embedding for the particular embedding partition, the incremental masked embedding comprising: values generated by the encoder for the particular embedding partition and each embedding partition of the sequence of embedding partitions that precedes the particular embedding partition; and masked values for any subsequent embedding partitions of the sequence of embedding partitions that follow the particular embedding partition; and during the incremental training, for each preceding embedding partition of the sequence of embedding partitions that precedes the particular embedding partition, synchronously applying any changes made to any of the parameter values of the particular decoder replica to the decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition, such that the one or more parameter values of the particular decoder replica and the one or more parameter values of the decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition are constrained to have the same values.

10. The system of claim 9, wherein, performing incremental training further comprises, for each preceding embedding partition of the sequence of embedding partitions that precedes the particular embedding partition: training the encoder and the particular decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition, wherein the particular decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition receives as input, during the incremental training, an incremental masked embedding for the preceding embedding partition.

11. The system of claim 9, wherein, performing initial training to train the encoder and the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition comprises: determining a gradient of an objective function with respect to an output generated by the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition; propagating the gradient from the decoder replica corresponding to the particular embedding partition only to a corresponding portion of the encoder that generates the particular embedding partition in the sequence of embedding partitions; and updating respective parameter values of the decoder replica corresponding to the particular embedding partition and the corresponding portion of the encoder using the backpropagated gradient.

12. The system of claim 9, wherein, performing incremental training to train the encoder and the particular decoder replica corresponding to the particular embedding partition includes: determining a gradient of an objective function with respect to an output generated by the particular decoder replica corresponding to the particular embedding partition; propagating the gradient from the decoder replica corresponding to the particular embedding partition only to a corresponding portion of the encoder that generates the particular embedding partition in the sequence of embedding partitions; and updating respective parameter values of the decoder replica corresponding to the particular embedding partition and the corresponding portion of the encoder using the backpropagated gradient.

13. The system of claim 12, wherein, The operations further include: determining that the gradient of the objective function has converged to a predetermined value; and in response to the determination, terminating the initial training and starting the incremental training for a second embedding partition in the sequence of embedding partitions.

14. The system of claim 9, wherein, The masking values for all subsequent embedding partitions in the sequence of embedding partitions are zero.

15. The system of claim 9, wherein, The encoder applies an activation function having a fixed output range to an intermediate encoder output to generate the embedding.

16. The system of any of claims 9-15, the operations further comprising, after performing all incremental training: receiving a new input; processing the new input using the trained encoder to generate an initial embedding for the new input; receiving user input modifying values of a given embedding partition in the initial embedding to generate a new embedding; and processing the new embedding using the trained decoder to generate an output for the new embedding.

17. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform a method for training a machine learning model to generate input embeddings to the machine learning model, the machine learning model having an encoder and a decoder, wherein the encoder generates the embeddings from the input, the decoder generates an output from the generated embeddings, wherein, The input comprises text units and the output comprises discourse representing the text units, or the input comprises structured data representing an image and the output comprises data representing a reconstructed image, or the input comprises data representing audio and the output comprises data representing reconstructed audio, and wherein the embedding is partitioned into a sequence of embedding partitions, the embedding partitions each comprising one or more dimensions of the embedding, the method comprising: generating a plurality of decoder replicas, each of the plurality of decoder replicas corresponding to a respective embedding partition in the sequence of embedding partitions, and each of the plurality of decoder replicas having one or more parameter values set equal to one or more parameter values of the decoder; For a first embedding partition in a sequence of embedding partitions, perform initial training to train the encoder and the decoder replica of the plurality of decoder replicas corresponding to the first embedding partition, wherein the decoder replica receives as input during the initial training a first masking embedding comprising: values generated by the encoder for the first embedding partition; and masking values for all subsequent embedding partitions in the sequence of embedding partitions; and For each particular embedding partition following the first embedding partition in the sequence of embedding partitions, perform incremental training to train the encoder and a particular decoder replica of the plurality of decoder replicas corresponding to the particular embedding partition, wherein the particular decoder replica corresponding to the particular embedding partition receives as input during the incremental training an incremental masking embedding for the particular embedding partition comprising: values generated by the encoder for the particular embedding partition and each embedding partition preceding the particular embedding partition in the sequence of embedding partitions; and masking values for any subsequent embedding partitions following the particular embedding partition in the sequence of embedding partitions; and during the incremental training, for each preceding embedding partition preceding the particular embedding partition in the sequence of embedding partitions, apply any changes made to any of the parameter values of the particular decoder replica synchronously to the decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition, such that the one or more parameter values of the particular decoder replica and the one or more parameter values of the decoder replica of the plurality of decoder replicas corresponding to the preceding embedding partition are constrained to have the same value.

Images