Unsupervised Learning of Disentangled Speech Content and Style Representations

Abstract

The linguistic content and speaking style disentanglement model (100) includes a content encoder (110), a style encoder (130), and a decoder (150). The content encoder is configured to receive an input utterance (102) as input and generate as output a latent representation of the linguistic content (120) of the input utterance. The content encoder is trained to disentangle speaking style information from the latent representation of the linguistic content. The style encoder is configured to receive the input utterance as input and generate as output a latent representation of the speaking style (140) of the input utterance. The style encoder is trained to disentangle the linguistic content information from the latent representation of the speaking style. The decoder is configured to generate an output utterance (152) based on the latent representation of the linguistic content of the input utterance and the latent representation of the speaking style of the same or different input utterance.

Description

[Technical field]

[0001] This disclosure relates to unsupervised learning of disentangled speech content and style representations. [Background technology]

[0002] Speech waveforms are complex, high-dimensional forms of data that are influenced by several underlying factors that can be broadly categorized as linguistic contents and speaking styles. Learning disentangled latent representations from speech has broad applications in generative tasks, including speech synthesis, data augmentation, speech transfer, and speech compression. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the disclosure provides a linguistic content and speech style disentangling model, including a content encoder, a style encoder, and a decoder. The content encoder is configured to receive an input utterance as an input and generate a latent representation of the linguistic content of the input utterance as an output. The content encoder is trained to disentangle speech style information from the latent representation of the linguistic content. The style encoder is configured to receive the same or a different input utterance as an input and generate a latent representation of the speech style of the same or a different input utterance as an output. The style encoder is trained to disentangle the linguistic content information from the latent representation of the speech style. The decoder is configured to generate an output utterance based on the latent representation of the linguistic content of the input utterance and the latent representation of the speech style of the same or a different input utterance.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the content encoder generates the latent representations of the linguistic content as a discrete time-step latent representation of the linguistic content that discards speech style variations in the input utterance. The content encoder may include one or more convolutional layers configured to receive the input utterance as an input and generate an initial discrete time-step latent representation of the linguistic content, and a vector-quantization (VQ) layer configured to apply an information bottleneck with a linear gradient to each initial discrete time-step latent representation of the linguistic content to generate a latent representation of the linguistic content as a set of latent variables that represent the linguistic content of the input utterance. Here, the content encoder may be trained using a content VQ loss based on the latent representation of the linguistic content generated for each time step, whereby the VQ loss prompts the content encoder to minimize the distance between the output and a nearest neighbor codebook.

[0005] In some examples, the style encoder includes one or more convolutional layers configured to receive an input utterance as an input, and a variational layer with Gaussian posterior configured to summarize the output from the one or more convolutional layers by a global average pooling operation over time to extract a global latent style variable corresponding to a latent representation of the utterance style. In these examples, during training, the global latent style variable may be sampled from the mean and variance of the style latent variables predicted by the style encoder, and during inference, the global latent style variable may be sampled from the mean of the global latent style variables predicted by the style encoder. Additionally or alternatively, the style encoder may be trained using a style regularization loss based on the mean and variance of the style latent variables predicted by the style encoder, where the style encoder uses the style regularization loss to minimize Kullback-Leibler (KL) divergence between unit Gaussian prior and Gaussian posterior.

[0006] In some implementations, the decoder is configured to receive as input a latent representation of the linguistic content of an input utterance and a latent representation of a speech style of the same input utterance, and to generate as output an output utterance that includes a reconstruction of the input utterance. The model may be trained using a reconstruction loss between the input utterance and the reconstruction of the input utterance output from the decoder.

[0007] In some additional implementations, the decoder is configured to receive as input a latent representation of the linguistic content of the input utterance and a latent representation of the speech style of the different input utterance, and to generate as output an output utterance that includes the linguistic content information specified by the input utterance and the speech style information specified by the different input utterance. The content encoder and the style encoder may be trained using mutual information loss to minimize the amount of mutual information captured in the latent representation of the linguistic content and the speech style.

[0008] Another aspect of the disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving an input utterance and processing the input utterance using a content encoder to generate a latent representation of a linguistic content of the input utterance. The content encoder is trained to disentangle speech style information from the latent representation of the linguistic content. The operations also include processing the same or a different input utterance using a style encoder to generate a latent representation of a speech style of the same or a different input utterance, and processing the latent representation of the linguistic content of the input utterance and the latent representation of the speech style of the same or a different input utterance using a decoder to generate an output utterance. The style encoder is trained to disentangle the linguistic content information from the latent representation of the speech style.

[0009] This aspect may include one or more of the following optional features. In some implementations, processing the input utterance to generate a latent representation of the linguistic content includes processing the input utterance to generate the latent representation of the linguistic content as a latent representation for each individual time step of the linguistic content that discards speech style variations in the input utterance. The content encoder may include one or more convolutional layers configured to receive the input utterance as an input and generate an initial, individual time step latent representation of the linguistic content, and a vector quantization (VQ) layer configured to apply an information bottleneck with a linear gradient to each initial, individual time step latent representation of the linguistic content to generate a latent representation of the linguistic content as a set of latent variables representing the linguistic content from the input utterance. Here, the content encoder may be trained using a content VQ loss based on the latent representation of the linguistic content generated for each time step, whereby the VQ loss prompts the content encoder to minimize the distance between the output and a nearest neighbor codebook.

[0010] In some examples, the style encoder includes one or more convolutional layers configured to receive an input utterance as an input, and a variational layer with Gaussian posterior configured to summarize the output from the one or more convolutional layers by a global average pooling operation over a time axis to extract a global latent style variable corresponding to a latent representation of the utterance style. In these examples, the operations may further include, during training, sampling the global style latent variable from the mean and variance of the style latent variables predicted by the style encoder, and during inference, sampling the global style latent variable from the mean of the global latent style variables predicted by the style encoder. Additionally or alternatively, the style encoder may be trained using a style regularization loss based on the mean and variance of the style latent variables predicted by the style encoder, where the style encoder uses the style regularization loss to minimize Kullback-Leibler (KL) divergence between the unit Gaussian prior and the Gaussian posterior.

[0011] In some implementations, the decoder is configured to receive as input a latent representation of the linguistic content of an input utterance and a latent representation of a speech style of the same input utterance, and to generate as output an output utterance that includes a reconstruction of the input utterance. The model may be trained using a reconstruction loss between the input utterance and the reconstruction of the input utterance output from the decoder.

[0012] In some additional implementations, the decoder is configured to receive as input a latent representation of the linguistic content of the input utterance and a latent representation of the speech style of the different input utterance, and to generate as output an output utterance that includes the linguistic content information specified by the input utterance and the speech style information specified by the different input utterance. The content encoder and the style encoder may be trained using mutual information loss to minimize the amount of mutual information captured in the latent representations of the linguistic content and the speech style.

[0013] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an exemplary system for disentangling latent representations of linguistic content and speaking style from an input utterance. [Diagram 2] FIG. 1 is a schematic diagram of a linguistic content and speech style disentanglement model, including a style encoder, a content encoder, and a decoder. [Diagram 3] FIG. 13 shows a plot of top-k speaker accuracy compared to content speaker ID and style speaker ID for various codebook sizes. [Figure 4A]FIG. 3 is a schematic diagram of an exemplary speaker recognition model implementing a style encoder from the linguistic content and speaking style disentanglement models of FIGS. 1 and 2. [Figure 4B] FIG. 3 is a schematic diagram of an exemplary speech recognition model implementing a content encoder from the linguistic content and speech style disentangling models of FIGS. 1 and 2. [Diagram 5] 1 is a flow diagram of an example configuration of operations for generating an output utterance. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Speech waveforms are complex, high-dimensional forms of data that are influenced by several underlying factors that can be broadly categorized into linguistic content (e.g., phonemes) and speech style. Learning disentangled latent representations of linguistic content and speech style from speech has broad applications in generative tasks, including speech synthesis, data augmentation, voice transfer, and speech compression. Disentangling latent representations from speech can also be useful for downstream tasks such as automatic speech recognition and speaker classification.

[0017] Techniques for learning unsupervised latent representations of speech have attracted interest due to the cost, complexity, and privacy concerns associated with collecting labeled speech data. In particular, learning latent representations of speech styles from unsupervised data is of particular interest due to the inherent inability to describe the prosody of speech with human labels. Learning global style tokens from a dictionary of embeddings from utterances that do not have prosodic labels provides the ability to learn global representations from entire speech sequences. Furthermore, fine-grained localized speech representations can be learned by applying self-supervised learning to unlabeled speech data and extracting localized latent representations that can be fine-tuned for speech recognition.

[0018] Implementations described herein are directed to unsupervised representation learning of speech by learning both global and localized representations. Specifically, implementations are directed to disentangling latent representations of linguistic content and speech style from an utterance using an autoencoder model, including a content encoder, a style encoder, and a decoder. The autoencoder model may be synonymously referred to as a linguistic content and speech style disentanglement model. The content encoder is associated with a local encoder having a vector quantization (VQ) layer configured to learn a discrete per-time-step (e.g., per-frame) latent representation that captures linguistic content from the input utterance, while the style encoder is associated with a variational autoencoder (VAE) configured to extract per-utterance latent representations that capture speech style from the input utterance. During training, the decoder reconstructs input speech samples to the content and style encoder using the latent representations of linguistic content and speech style output from the content and style encoder. During inference, the decoder can generate new speech features by combining latent representations of linguistic content extracted from a first speech sample with latent representations of speech style extracted from a different second speech sample.

[0019] In an additional implementation, mutual information loss is applied to the content and style encoders to minimize mutual information from the latent representations of linguistic content and speaking style output from the content and style encoders. As will be apparent, training the content and style encoders to minimize the extracted mutual information at their respective outputs further disentangles the latent representations of linguistic content (e.g., local representations) and latent representations of speaking style (e.g., global representations) from the speech.

[0020] The autoencoder model includes a content encoder, a style encoder 130, and a decoder. The decoder is configured to receive latent representations of both content and style as input and generate speech features as output. That is, the decoder is configured to reconstruct the input utterance as output speech features. In some applications, a trained autoencoder model can dispense with the use of a decoder.

[0021] FIG. 1 illustrates an exemplary system 10 for training an autoencoder model (e.g., a linguistic content and speech style disentanglement model) 100 to disentangle latent representations of linguistic content 120 and speech style 140 from an input utterance 102. The input utterance 102 may include a set of speech features. In some examples, the set of speech features representing the input utterance 102 includes log-mel filter bank features. The system 10 includes a computing system 20 having data processing hardware 22 and memory hardware 24 in communication with the data processing hardware 22 and storing instructions that cause the data processing hardware 22 to perform operations. In some implementations, the computing system 20 (e.g., the data processing hardware 22) performs a training process to train the autoencoder model 100 with multiple samples of the input utterance 102 corresponding to utterances spoken by one or more different speakers. The training process aims to reconstruct each sample of the input utterance 102 by decoding the latent representations of the linguistic content 120 and speech style 140 extracted from the input utterance 102 and combined to generate the corresponding speech features 152. The training process may store the latent representations of the linguistic content 120 and speech style 140 extracted from each sample of the input utterance 102 and the corresponding output speech features 152 in data storage 185 in the memory hardware 24.

[0022] During inference, the computing system 20 (e.g., data processing hardware 22) or a user computing device (not shown) executes the trained autoencoder model 100, including the content encoder 110, the style encoder 130, and the decoder 150, to generate new speech features 152 as synthesized utterances that propagate linguistic content with speech styles extracted from a first speech sample 50, 50a and extracted from a second speech sample 50, 50b. Here, the content encoder 110 is configured to predict a time-step-wise latent representation of the linguistic content 120 from a first speech sample 50a spoken by a first speaker, while the style encoder 130 is associated with a variational autoencoder (VAE) configured to extract an utterance-by-utterance latent representation of the speech style 140 from a different second speech sample 50b spoken by a different speaker. In particular, the first and second speech samples 50a, 50b spoken by different speakers may contain different linguistic content, and the first and second speakers may speak in different speaking styles (e.g., prosody / accent). In this scenario, the trained autoencoder model 100 is adapted for use in speech transfer applications (e.g., "put it like this"), where the new speech features 152 generated as a synthesized utterance preserve the linguistic content of the first speech sample 50a spoken by the first speaker, but in the voice and speaking style of the second speaker. In some examples, the first speech sample 50a is spoken in a different language than the second speech sample 50b. The trained autoencoder model 100 can similarly be applied in data augmentation applications, where latent representations of speech styles 140 are extracted from speech samples spoken by different users and conveying different speech styles, to yield new speech features 152 from a decoder 150 that convey different augmentations of synthesized speech of the same linguistic content.

[0023] The decoder 150 is configured to generate speech features 152 as output by combining the latent representations of the linguistic content 120 output by the content encoder 110 and the latent representations of the speech style 140 output by the style encoder 130. The output 190 may receive the output speech features 152 from the decoder 150. The output 190 may include a speech recognition model or a speaker identification model trained on the output speech features 152. Optionally, the output 190 may not receive the speech features 152, but instead receive the latent representations of the linguistic content 120 and the speech style 140 from the content and style encoders 110, 130, respectively.

[0024] FIG. 2 shows an input utterance 102,

number

[0025] The content encoder 110 may include a neural network with one or more convolutional layers 112 and a vector quantization (VQ) layer 114. In some examples, the content encoder 110 includes ten one-dimensional convolutional layers with residual connections. A time stride of 2 may be applied in the third convolutional layer to reduce the final output length by half (e.g., from T to T / 2). The one or more convolutional layers 112 of the content encoder 110 receive the input utterance 102 and generate an initial latent representation 113 from the utterance 102, and the VQ layer 114 applies an information bottleneck with a linear gradient to the initial latent representation 113 to capture the required localized information, e.g., phoneme or subword-like latent representations, and discard the variation of the speaking style. Thus, the VQ layer 114 generates a set of latent variables representing the linguistic content from the input utterance 102,

number

[0026] In some implementations, during training, the VQ loss module 122 determines a content VQ loss 124 for the content encoder 110 based on the latent representation of the linguistic content 120 predicted for each time step. The content VQ loss 124 is calculated by multiplying the content VQ loss 124 by itself and the nearest neighbor codebook E i The content encoder output z is prompted to minimize the distance between z and the content VQ loss module 122 may determine the content VQ loss 124 using the following formula:

[0027]

number

[0028] where sg(·) denotes the stopping gradient operation. The VQ loss 124 corresponds to a self-supervised loss, similar to a clustering loss, in which the model is encouraged to cluster consecutive data points and move other data points closer to the clusters.

[0029] The style encoder 130 may include a neural network having one or more convolutional layers 132 and a variational layer 134 with Gaussian posterior probabilities configured to summarize the output from the convolutional layers 132 by a global average pooling operation over the time axis. In some examples, the style encoder 130 processes the input utterance 102 using six residual one-dimensional convolutional layers with a time stride of 2 for three different layers, thereby resulting in an 8x reduction in time length. The global average pooling operation extracts from the input utterance 102 a global latent style variable representing the speaking style 140,

number

[0030]

number

[0031] Here, S μ denotes the mean of the style variables predicted by the style encoder 130,

number

[0032] During training, the style loss module 142 calculates the average S of the style variables predicted by the style encoder 130. μ and Dispersion

number

number

[0033]

number

[0034] In some implementations, the decoder 150 receives as input both the latent representations of linguistic content and speech style 120, 140 output from the content and style encoders 110, 130, respectively, during training, and generates features 152,

number

number

number

[0035] While the content encoder 110 and the style encoder 130 are optimized to predict the latent representations of the disentangled linguistic content 120 and speaking style 140, respectively, the quantized variable c 1 , ..., c T can inherently capture non-content information, such as speech style information. Without using explicit labels during training, the model 100 can extract linguistic content (c 1 , ..., c T ) 120 and speech style 140(s), respectively. Minimizing mutual information (MI) can be estimated from the latent representations of the encoder outputs 120 and speech style 140(s). Thus, minimizing the mutual information to reduce the amount of correlation between the encoder outputs optimizes the model 100 to further disentangle local and global representations of linguistic content and speech style.

[0036] In some implementations, a noise contrast estimation (I NCE ) loss is used to estimate a lower bound MI between content and style as follows:

[0037]

number

[0038] where K is the number of samples across different utterances, Sc(·,·) is the neural network scorer, and C i is the initial latent representation from the content encoder 110 before quantization, and S i is the latent representation of the speaking style 140 output from the style encoder 130.

[0039] Then, I NCE is maximized with respect to Sc, and I NCE The final loss of model 100 is calculated using the following formula:

number

[0040]

number

[0041] The Final Loss of the Model 100

number

number

[0042]

number

[0043] To further disentangle the local and global representations, mutual information loss can be applied. In particular, the global style representation captured by the global VAE provides sufficient speaker information from only one example per speaker to allow training a linear projection layer on top of the global style representation to obtain a speaker classification model.

[0044] Experiments show that increasing the VQ-VAE codebook size of the content encoder 110 optimizes the model 100 to preserve linguistic content from the input utterance 102. Also, applying mutual information loss to minimize the amount of mutual information captured by the content and style encoders 110, 130 further improves linguistic content preservation. In a first non-shuffled scenario, optimized to measure how well the model 100 compresses an utterance, the content and style encoders 110, 130 compress the same input utterance 102, X i , respectively, and the decoder 150 generates speech features 152,

number

number

number

number

[0045] [Table 1]

[0046] The experiments further measure how well the style encoder extracts and preserves the original speaking style from the input utterance 102. In these experiments, the content and style encoders 110, 130 extract and preserve the original speaking style from different input utterances X i , X j The decoder 150 receives the speech features corresponding to the generated utterance based on the linguistic content and the latent representations of the speech style 130, 140 output from the encoders 110, 130.

number

number

[0047] [Table 2]

[0048] After training the model 100 to learn disentangled latent representations of linguistic content and speaking style from an utterance, the trained model 100 can be used in a wide range of applications in generative tasks, including speech synthesis, data augmentation, voice transfer, and voice compression. The ability of the trained model 100 to disentangle these latent representations from an utterance can also be useful for downstream tasks such as training automatic speech recognition and / or speaker recognition / classification models. In general, the trained model 100 can be used in two groups of applications: encoder-only applications and encoder-decoder applications. Encoder-decoder applications, such as voice transfer applications and data augmentation applications, have been described above with reference to FIG. 1.

[0049] FIG. 4A illustrates a notable encoder-only application that includes using a trained style encoder 130 as a speaker recognition model 400 for a speaker recognition task. Here, the trained style encoder 130 is suited for speaker recognition applications by pre-training the style encoder 130 with a large amount of unlabeled data that includes input utterances corresponding to utterances spoken by one or more speakers. A small amount of labeled data then fine-tunes the pre-trained style encoder to perform a few-shot speaker recognition task. The labeled data may vary the number of utterance examples 402 per speaker, referred to as "one shot" and "three shot". For the speaker recognition task, a linear projection layer (e.g., a softmax layer) is placed on top of the style encoder to compute the logits of speaker classification based on the latent representation of the speaking style 140 output from the style encoder 130 for a given input utterance. During fine-tuning, all parameters are frozen except for the projection layer. The style encoder 130 and the projection layer 450 can form the speaker recognition model 400. The speaker recognition model 400a may be used to perform various speaker recognition tasks, such as, but not limited to, detecting different speakers (e.g., diarization), calculating speaker ID, and / or voice matching for enrollment / verification.

[0050] FIG. 4B illustrates another encoder-only application example, including using the trained content encoder 110 as a speech recognition model 400b to generate speech recognition results for an input utterance 402. The content encoder 110 can be leveraged to extract latent representations of the linguistic content 120 to provide local information for use in bootstrapping automatic speech recognition (ASR) training by using latent representations of the linguistic content 120 and leveraging unsupervised data. In the illustrated example, a neural network 470 is placed on top of the content encoder 110 to provide the speech recognition model 400b. In some implementations, the content encoder 110 and the neural network 470 are retrained using labeled data of the input utterance example 402 and corresponding transcripts. In this configuration, the content encoder 110 functions as a feature extractor to encode linguistic content from the utterance to improve speech recognition accuracy.

[0051] Notable encoder-decoder applications include speech transfer applications (e.g., "Say it like this") and data augmentation applications. In a typical speech transfer application, the content encoder 110 will receive an utterance from a first speaker, and the style encoder 130 will receive a different utterance from a second speaker. The utterances spoken by the first and second speakers may contain different content. In this scenario, the decoder 150 can generate a synthesized speech representation that conveys the linguistic content of the utterance from the first speaker, but in the voice of the second speaker. That is, the decoder 150 receives as input the latent representation of the linguistic content 120 encoded by the content encoder 110 of the utterance from the first speaker and the speech style 140 encoded by the style encoder 130 of the utterance from the second speaker, and generates as output a synthesized utterance that preserves the linguistic content of the utterance from the first speaker and the speech style of the second speaker. In data augmentation applications, the latent representation of speech style can be varied to produce different augmentations of synthetic utterances from the decoder of the same linguistic content.

[0052] 5 is a flow diagram of an example configuration of operations of a method 500 for generating an output utterance. The data processing hardware 22 may execute instructions stored in the memory hardware 24 to cause the data processing hardware to perform the operations of the method 500. At operation 502, the method 500 includes receiving an input utterance 102, and at operation 504, the method 500 includes processing the input utterance 102 using a content encoder 110 to generate a latent representation of the linguistic content 120 of the input utterance 102. The content encoder 110 is trained to disentangle speaking style information from the latent representation of the linguistic content 120.

[0053] At operation 506, the method 500 includes processing the same or different input utterance using a style encoder 130 to generate a latent representation of a speaking style 140 of the same or different utterance. The style encoder 130 is trained to disentangle linguistic content information from the latent representation of the speaking style 140. At operation 508, the method 500 includes processing the latent representation of the linguistic content 120 of the input utterance 102 and the latent representation of the speaking style 140 of the same or different utterance using a decoder 150 to generate an output utterance.

[0054] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0055] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., a sequence of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0056] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0057] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or in storage device 630 to display graphical information for a graphical user interface (GUI) on an external input / output device, e.g., a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, as desired, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (such as, for example, a bank of servers, a group of blade servers, or a multi-processor system).

[0058] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., a sequence of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, e.g., boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0059] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or the memory of the processor 610.

[0060] The high-speed controller 640 manages the bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages the less bandwidth-intensive operations. Such an allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, to the display 680 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 650, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and to a low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, e.g., a keyboard, a pointing device, a scanner, or, e.g., via a network adapter, to a networking device such as a switch or a router.

[0061] The computing device 600 may be implemented in a number of different ways, as shown in the figure. For example, the computing device 600 may be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0062] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuit components, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and to transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0063] These computer programs (also known as programs, software, software applications or codes) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logic flows described herein may be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be executed by dedicated logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and dedicated microprocessors and any one or more processors of any kind of digital computer. In general, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, dedicated logic circuitry.

[0065] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide interaction with the user as well, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser at the user's client device in response to a request received from the web browser.

[0066] Several implementations have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0067] 10. System 20. Computing Systems 22 Data Processing Hardware 24 Memory Hardware 50,50a First speech sample 50,50b Second speech sample 100 Autoencoder Models 102 Utterances 110 Content Encoder 112 Convolutional Layer 113 Initial latent representation 114 Vector Quantization (VQ) Layer 120 Linguistic content 122 VQ Loss Module 124 Contents VQ Loss 130 Style Encoder 132 Convolutional Layer 134 Variational Sheaf 140 Speech Style 142 Style Loss Module 144 Style Regularization Loss 150 Decoder 152 Speech Features 160 Reconstruction Loss Module 162 Reconstruction loss 185 Data Storage 190 Output 300 Plots 400 speaker recognition models 402 Input utterance, utterance example 450 projection layer 470 Neural Networks 500 ways 600 Computing Devices 610 Processor 620 Memory 630 Storage Devices 640 High Speed ​​Interface / Controller 650 High Speed ​​Expansion Port 660 Low Speed ​​Interface / Controller 670 Slow Bus 680 Display 690 Low Speed ​​Expansion Port

Claims

1. A disentanglement model (100) of linguistic content (120) and speech style (140), receiving an input utterance (102) as input; generating as output a latent representation of the linguistic content (120) of the input utterance (102); a content encoder (110) configured to: receiving as input the same or a different input utterance (102); generating as output a latent representation of the speech style (140) of the same or different input utterance (102); a style encoder (130) trained to disentangle linguistic content information from the latent representation of a speaking style (140); one or more convolutional layers (132) configured to receive as input the input utterance (102); a variational layer (134) with Gaussian posterior probabilities configured to summarize outputs from the one or more convolutional layers (132) by a global average pooling operation over time to extract global style latent variables corresponding to the latent representation of a speaking style (140); A function of a style encoder (130), including: a decoder (150) configured to generate an output utterance based on the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the same or a different input utterance (102); A model (100) that realizes the above on a computer.

2. 2. The model (100) of claim 1, wherein the content encoder (110) generates the latent representation of linguistic content (120) as a latent representation for each distinct time step of the linguistic content (120) that discards speaking style variations in the input utterance (102).

3. The content encoder (110): one or more convolutional layers (112) configured to receive as input the input utterance (102) and generate an initial discrete time-step latent representation of the linguistic content (120); a vector quantization (VQ) layer configured to apply an information bottleneck with a linear gradient to a latent representation of the linguistic content (120) for each initial discrete time step to generate the latent representation of the linguistic content (120) as a set of latent variables representing the linguistic content (120) of the input utterance (102); 3. The model (100) of claim 1 or 2, comprising:

4. 4. The model (100) of claim 3, wherein the content encoder (110) is trained using a content VQ loss (124) based on the latent representation of the linguistic content (120) generated for each time step, the content VQ loss (124) prompting the content encoder (110) to minimize the distance between an output and a nearest neighbor codebook.

5. During training, the global style latent variables are sampled from the mean and variance of the style latent variables predicted by the style encoder (130); During inference, the global style latent variables are sampled from the average of the global style latent variables predicted by the style encoder (130).

5. A model (100) according to any one of claims 1 to 4.

6. 6. The model of claim 1, wherein the style encoder is trained using a style regularization loss based on the mean and variance of style latent variables predicted by the style encoder, and the style encoder uses the style regularization loss to minimize Kullback-Leibler (KL) divergence between unit Gaussian priors and Gaussian posterior probabilities.

7. The decoder (150): receiving as input the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the same input utterance (102); generating as output the output utterance comprising a reconstruction of the input utterance (102); The model (100) according to any one of claims 1 to 6, configured so as to

8. 8. The model (100) of claim 7, wherein the model (100) is trained using a reconstruction loss (162) between the input utterance (102) and the reconstruction of the input utterance (102) output from the decoder (150).

9. The decoder (150): receiving as input the latent representation of the linguistic content (120) of the input utterances (102) and the latent representation of the speech style (140) of the different input utterances (102); generating as output the output utterance comprising linguistic content information specified by the input utterance (102) and speech style information specified by the different input utterance (102); The model (100) according to any one of claims 1 to 8, configured so as to

10. 10. The model (100) of any one of claims 1 to 9, wherein the content encoder (110) and the style encoder (130) are trained to minimize the amount of mutual information captured in the latent representation of linguistic content (120) and the latent representation of speech style (140) using mutual information loss.

11. A method (500) implemented by data processing hardware (22), comprising: receiving an input utterance (102); processing the input utterance (102) using a content encoder (110) to generate a latent representation of linguistic content (120) of the input utterance (102), the content encoder (110) being trained to disentangle speaking style information from the latent representation of linguistic content (120); processing the same or different input utterances (102) using a style encoder (130) to generate latent representations of speech styles (140) of the same or different input utterances (102), the style encoder (130) being trained to disentangle linguistic content information from the latent representations of speech styles (140); one or more convolutional layers (132) configured to receive as input the input utterance (102); a variational layer (134) with Gaussian posterior probabilities configured to summarize outputs from the one or more convolutional layers (132) by a global average pooling operation over time to extract global style latent variables corresponding to the latent representation of a speaking style (140); and processing, using a decoder (150), the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the same or different input utterance (102) to generate an output utterance; A method (500).

12. Processing the input utterance (102) to generate the latent representation of the linguistic content (120) of the input utterance (102), processing the input utterance (102) to generate a latent representation of linguistic content (120) for each distinct time step of the linguistic content (120) that discards speech style variations in the input utterance (102); 12. The method (500) of claim 11, comprising:

13. The content encoder (110): one or more convolutional layers (112) configured to receive as input the input utterance (102) and generate an initial discrete time-step latent representation of the linguistic content (120); a vector quantization (VQ) layer configured to apply an information bottleneck with a linear gradient to a latent representation of the linguistic content (120) for each initial discrete time step to generate the latent representation of the linguistic content (120) as a set of latent variables representing the linguistic content (120) of the input utterance (102); 13. The method (500) of claim 11 or 12, comprising:

14. 14. The method (500) of claim 13, wherein the content encoder (110) is trained using a content VQ loss (124) based on the latent representation of the linguistic content (120) generated for each time step, the content VQ loss (124) prompting the content encoder (110) to minimize the distance between an output and a nearest neighbor codebook.

15. During training, sampling the global style latent variables from the mean and variance of style latent variables predicted by the style encoder (130); During inference, sampling the global style latent variables from the average of the global style latent variables predicted by the style encoder (130); 15. The method (500) of any one of claims 11 to 14, further comprising:

16. 16. The method (500) of any one of claims 11 to 15, wherein the style encoder (130) is trained using a style regularization loss (144) based on the mean and variance of style latent variables predicted by the style encoder (130), and the style encoder (130) uses the style regularization loss (144) to minimize Kullback-Leibler (KL) divergence between unit Gaussian prior and Gaussian posterior probabilities.

17. The decoder (150): receiving as input the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the same input utterance (102); processing the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the same input utterance (102) to generate as output the output utterance comprising a reconstruction of the input utterance (102); The method (500) according to any one of claims 11 to 16, configured to:

18. 20. The method (500) of claim 17, further comprising training a disentanglement model (100) of linguistic content (120) and speaking style (140) using a reconstruction loss (162) between the input utterance (102) and the reconstruction of the input utterance (102) output from the decoder (150).

19. The decoder (150): receiving as input the latent representation of the linguistic content (120) of the input utterances (102) and the latent representation of the speech style (140) of the different input utterances (102); processing the latent representation of the linguistic content (120) of the input utterance (102) and the latent representation of the speech style (140) of the different input utterance (102) to generate as output the output utterance including linguistic content information specified by the input utterance (102) and speech style information specified by the different input utterance (102); The method (500) according to any one of claims 11 to 18, configured to:

20. 20. The method (500) of any one of claims 11 to 19, wherein the content encoder (110) and the style encoder (130) are trained to minimize the amount of mutual information captured in the latent representation of linguistic content (120) and the latent representation of speech style (140) using mutual information loss.

Images