Converting non-native speech into native speech while preserving vocal identity

Abstract

A computer-implemented method includes providing input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features. The input features include a lower-dimensionality representation of a speech signal. The first compact representations include lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

Description

CONVERTING NON-NATIVE SPEECH INTO NATIVE SPEECH WHILE PRESERVING VOCAL IDENTITYCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from US Provisional Application No. 63 / 558,989, filed on 28 February 2024, and European Patent Application No. EP 24174712.0, filed on 8 May 2024, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to audio processing and, more particularly, to processing techniques for transforming non-native speech into native speech while preserving vocal identity.SUMMARY

[0003] Applying machine learning techniques to audio signals can unlock a variety of benefits across various domains. For example, machine learning models may transcribe spoken language into text, translate audio in near-real-time, analyze the emotion and sentiment of speakers, and the like. However, raw audio signals typically need to be processed and transformed into a form more suited for machine learning applications. For example, raw audio signals are typically sampled at a relatively high rate, resulting in a high dimensionality. This high dimensionality makes processing raw audio signals computationally expensive.Furthermore, the performance of machine learning models often degrades as the dimensionality of the input data increases. As the dimensionality of input data increases, meaningful data points that the machine learning model can leam from may be spaced farther apart, increasing the difficulty for models to generalize from training data. To maintain a sufficient density of relevant data points in a high-dimensional space, the required sample size may also increase exponentially. This increase in sample size can increase the likelihood of overfitting the models to training data (because, for example, the models may leam noise in the training data as if it were a meaningful signal). Thus, it is often technically beneficial to identify and extract the features from the audio signal that are relevant for a specific task for machine learning tasks.

[0004] However, identifying the right signals to extract from an audio signal can be technically challenging. For example, audio signals include complex information encompassing a wide range of frequencies, amplitudes, and / or temporal patterns. As a result of this complexity, the relevant features for any given task may be deeply embedded within the signaland not immediately apparent, often requiring advanced signal processing techniques to uncover. Additionally, the high-dimensional nature of audio signals further complicates the task of identifying which dimensions (e.g., features) carry meaningful information for the machine learning task. Since audio information may also be inherently temporal, meaningful information may be spread across different time scales of the audio signal. For example, while the spectral content of individual sounds might be captured in short-term features, the rhythm or prosody of speech might be captured in long-term features. Thus, identifying and effectively capturing the varying temporal dependencies may be challenging. Accordingly, what is needed are processing techniques for simultaneously capturing relevant features and reducing the dimensionality7of audio signals, transforming the raw audio signals into a form suitable for machine learning applications.

[0005] For example, capturing both representations of the phonetic content of speech and its articulatory characteristics in a compact form may offer a variety of technical benefits for machine learning models performing accent conversion. Phonetic content may refer to the sounds (e.g., phonemes) used in a language and their organization into words and sentences. Phonetic content may be an abstract representation of speech sounds and patterns present in the audio signal. Articulatory characteristics may describe the coordinated movements of the articulators (e.g., the tongue, lips, jaw. velum, etc.) of the speaker to produce speech. Representing the speech content of audio signals through both phonetic content and articulatory characteristics enables downstream machine learning models to better abstract accents from the audio signals.

[0006] For example, variations in accent may be represented by differences in the phonemes that are present in speech and / or how the phonemes are pronounced. For example, the presence or absences of certain sounds and / or specific intonation patterns can contribute to the listener’s perception of an accent. Articulatory7characteristics (e.g.. the physical production of sounds) can also vary significantly with accent. For example, the same phoneme can be physically articulated differently in different accents, leading to an audible distinction between accents. Thus, the phonetic content and articulatory' characteristics of speech together provide a comprehensive framework for representing speech, especially for accent conversion machine learning applications.

[0007] Systems, apparatuses, methods, and techniques described in this specification provide technical solutions to these problems by automatically extracting relevant features from audio signals (such as speech signals) using an audio model. In various implementations, the speechsignals are input to an acoustic feature extraction model, which outputs speech embeddings that capture the temporal and contextual relationships between sounds in the speech signal as a lower-dimensional vector representation. In some examples, the speech embeddings are input to the audio model, which maps the speech embeddings to speech representations. In various implementations, the audio model extracts speech representations including phonetic content such as phonetic posteriorgrams (PPGs) and articulatory- characteristics such as vocal tract variables from the speech embeddings. In some examples, the audio model is jointly trained using a combined loss function to estimate both PPGs and vocal tract variables. In various implementations, the audio model outputs a compact representation of the speech representations. For example, the compact representations are provided to other machine learning models for various applications, such as, for example, accent conversion applications.

[0008] According to some examples, a computer-implemented method includes providing input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features. The input features include a lower- dimensionality representation of a speech signal. The first compact representations include lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

[0009] Other examples provide a non-transitory computer-readable medium that includes executable instructions, which when executed by an electronic processor causes the electronic processor to provide input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features. The input features include a lower-dimensionality representation of a speech signal. The first compact representations include lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

[0010] Other implementations provide a system that includes memory hardware storing instructions and processor hardware configured to execute the instructions. Executing the instructions causes the system to provide input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features. The input features include a lower-dimensionality representation of a speech signal. The first compact representations include lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

[0011] Other examples, embodiments, features, and aspects will become apparent by consideration of the detailed description and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 is a block diagram illustrating an example computing system.

[0013] FIG. 2 is a block diagram illustrating an example audio processing and transformation platform.

[0014] FIG. 3 is a schematic illustration of an example architecture of a machine learning model that implements some examples of an acoustic model.

[0015] FIG. 4 is a flowchart of an example process for training a machine learning model.

[0016] FIG. 5 is a schematic illustration of example reference audio data.

[0017] FIG. 6 is a block diagram of an example implementation of a process for generating compact representations from speech signals.

[0018] FIG. 7 is a message sequence chart illustrating example interactions between components of an audio processing and transformation platform as the platform automatically generates compact representations from speech signals.

[0019] FIG. 8 is a block diagram of an example implementation of a process for generating a synthesized speech signal from a reference native speech signal and a non-native speech signal.

[0020] FIG. 9 is a message sequence chart illustrating example interactions between components of an audio processing and transformation platform as the platform automatically generates a synthesized speech signal from the reference native speech signal and the non-native speech signal.

[0021] FIG. 10 is a block diagram of an example implementation of a process for generating a synthesized speech signal from a non-native speech signal.

[0022] FIG. 11 is a message sequence chart illustrating example interactions between components of an audio processing and transformation platform as the platform automatically generates a synthesized speech signal from the non-native speech signal.

[0023] In the drawings, reference numbers may be reused to identify similar and / or identical elements.DETAILED DESCRIPTION

[0024] FIG. 1 is a block diagram illustrating an example computing system 100. As illustrated in FIG. 1, some examples of the system 100 include an audio processing and transformation platform 102, one or more user devices 104 (such as user device 104-1 and user device 104-2), a media delivery' platform 106, a speech recognition platform 108, and / or a communications system 110. Although a single audio processing and transformation platform 102, two user devices 104, a single media delivery platform 106, a single speech recognition platform, and a single communications system 110 are illustrated in FIG. 1, various implementations of the system 100 include one or more (e.g., any number) of each device, platform, and / or system. In some examples, one or more of the user devices 104, the media delivery platform 106. the speech recognition platform 108. and / or the communications system 110 are omitted from the system 100. In various implementations, the audio processing and transformation platform 102, user devices 104, media delivery' platform 106, and / or speech recognition platform 108 communicate with one another via the communications system 110.

[0025] In some examples, the audio processing and transformation platform 102 may be a computing platform that performs audio processing on speech signals using machine learning models. For example, the audio processing and transformation platform 102 may use machine learning models to transform speech signals from a non-native speaker of a language to speech signals mimicking the pronunciation and intonation of a native speaker while preserving characteristics of the speaker’s voice. In various implementations, the user devices 104 may include one or more computing platforms, such as smartphones, tablet computers, laptop computers, desktop computers, computer servers, etc. The media delivery platform 106 may include one or more computing platforms that host digital content such as videos, music tracks, podcasts, etc. The media delivery platform 106 may deliver the digital content to other platforms in the system 100 via the communications system. The speech recognition platform 108 may include one or more computing platforms that convert the spoken language of digital content into text.

[0026] In some examples, the communications system 110 includes one or more networks, such as a General Packet Radio Service (GPRS) network, a Time-Division Multiple Access (TDMA) network, a Code-Division Multiple Access (CDMA) network, a Global System ofMobile Communications (GSM) network, an Enhanced Data Rates for GSM Evolution (EDGE) network, a High-Speed Packet Access (HSPA) network, an Evolved High-Speed Packet Access (HSPA+) network, a Long Term Evolution (LTE) network, a Worldwide Interoperability for Microwave Access (WiMAX) network, a 5th-generation mobile network (5G), an Internet Protocol (IP) network, a Wireless Application Protocol (WAP) network, or an IEEE 802.11 standards network, as well as any suitable combination of the above networks. In various implementations, the communications system 110 includes an optical network, a local area network, and / or a global communication network, such as the Internet.

[0027] FIG. 2 is a block diagram illustrating an example audio processing and transformation platform 102. In various implementations, the audio processing and transformation platform 102 includes system resources 202, a communications interface 204, and non-transitory computer- readable storage media, such as, for example, storage 206. The non-transitory computer- readable storage media may contain instructions that, when executed, cause one or more electronic processors to perform various functions described herein. In some examples, the system resources 202 include one or more electronic processors, one or more graphics processing units, volatile computer memory, non-volatile computer memory. and / or one or more system buses interconnecting the components of the audio processing and transformation platform 102. In various implementations, the communications interface 204 includes hardware and software components that communicate with other devices, platforms, and / or systems over the communications system 110. For example, the communications interface 204 may include one or more transceivers for sending and / or receiving data over the communications system 110.

[0028] In various implementations, the storage 206 includes an audio processing application 208, one or more machine learning models (such as an audio feature extraction model 210, a first training features generation model 212, a second training features generation model 214, an acoustic model 216, a speaker encoder model 218, a translator model 220, a prosody encoder model 222. a synthesizer model 224. and or a vocoder model 226), and / or reference audio data 228. In some examples, the various devices and / or platforms in the system 100 (such as the user devices 104, media delivery platform 106, and / or speech recognition platform 108) connect to the audio processing and transformation platform 102 and interact with the one or more machine learning models and / or the reference audio data 228 via the audio processing application 208.

[0029] In various implementations, the audio feature extraction model 210 is a machine learning model that generates speech embeddings from an input speech signal. The speech signal may be an audio file, and the speech embeddings may include a set of vectors. The set ofvectors may sample the speech signal at 50 Hz (e.g., for every second of the speech signal, 50 distinct embedding vectors may be generated). Each vector may have a dimensionality of 1024 (e.g., each vector may consist of 1024 elements). Speech embeddings output from the audio feature extraction model 210 may capture the temporal and contextual relationships between sounds in the speech signal. By transforming the speech signal into a dense vector representation as speech embeddings, the audio feature extraction model 210 effectively compresses the high-dimensionality speech signal into a lower-dimensional vectorized space as speech embeddings, which dramatically reduces the computational burden for the various machine learning models of the audio processing and transformation platform 102 to work with the content of the speech signal.

[0030] In various implementations, the speech embeddings capture salient features of the speech signal, such as linguistic characteristics (e.g., phonetic information and / or world-level information) and / or prosodic features. In some examples, the speech embeddings encapsulate details about the phonemes (e.eg., the basic units of sound in speech) present in the speech file, such as information about how sounds are articulated and perceived. In various implementations, the speech embeddings capture word-level information, such as word boundaries. In some examples, the speech embeddings capture prosodic features, such as features related to intonation and rhythm. For example, the speech embeddings can capture information about intonation (e.g., pitch variation) and / or rhythm (e.g., timing and / or stress patterns).

[0031] In various implementations, the audio feature extraction model 210 may include a transformer model. Some examples of transformer models may be especially suited to capturing contextual information present in speech because they include a self-attention (or intra-attention) mechanism that allows the model to adjust the focus on different parts of the input sequence, enabling them to capture long-range dependencies and contextual relationships between elements present in the speech signal, regardless of their distance in the sequence. In some examples, the audio feature extraction model 210 may employ a Bidirectional Encoder Representations from Transformers (BERT) architecture or a variant of the architecture, such as a Hidden Unit BERT (HuBERT) architecture.

[0032] In various implementations, the training features generation model 212 is a machine learning model that maps lower dimensional representations of speech, such as mel-frequency cepstral coefficients (MFCCs) or speech embeddings generated by a machine learning model, such as the audio feature extraction model 210, to a set of phonetic units or symbols, such asphonetic posteriorgrams (PPGs) or tri-phone PPGs. MFCCs may be compact representations of the short-term power spectrum of sound, and PPGs may be a form of representation that captures the likelihood of phonetic units occurring across a timeline of the speech signal. Tri-phone PPGs may capture the likelihood of phonetic units (not just individual phonemes) occurring across the timeline of the speech signal. For example, the phonetic units may be "tn-phones." which include a central phoneme and its immediate-left and immediate-right phonetic context. By capturing this context, tri-phone PPGs may be able to better capture the nuance of how the pronunciation of a phoneme can change based on the phonemes that precede and / or follow it.

[0033] In some examples, the training features generation model 212 may include a deep neural network (DNN), such as a convolution neural network (CNN) or a recurrent neural network (RNN). In various implementations, the training features generation model 214 may have a hybrid architecture, such as a Long Short-Term Memory' (LSTM) architecture, a gated recurrent units (GRUs) architecture, or a transformer architecture.

[0034] In various implementations, the training features generation model 214 is a machine learning model that maps lower dimensional representations of speech, such as mel-frequency cepstral coefficients (MFCCs) or speech embeddings generated by a machine learning model such as the audio feature extraction model 210, to vocal tract variables describing the articulatory movements of the speaker that produced the speech. For example, vocal tract variables (or articulatory' features) refer to the anatomical and / or physiological characteristics of the vocal tract that can be modified to produce different sounds in speech. The vocal tract variables may track positions of various parts of the vocal tract, such as the upper lips, the lower lips, the jaw, the tongue tip, the tongue body, the velum (or soft palate), and / or the glottis (e.g., the vocal folds within the larynx and / or the space between them).

[0035] In various implementations, the vocal tract variables include lip protrusion (LP) variables, lip aperture (LA) variables, tongue tip constriction degree (TTCD) variable, tongue tip constriction location (TTCL) variables, tongue body constriction degree (TBCD) variables, and / or tongue body constriction location (TBCL) variables. The LP variables may track the positions and / or relative positions of the speaker’s upper lips, lower lips, and / or jaws. For example, the LP variables may track how far the speaker’s lips extend forward or rearwards from the face. The LA variables may track the positions and / or relative positions of the speaker’s upper lips, lower lips, and / or jaws. For example, the LA variables may describe the opening between the speaker's upper lips and lower lips. The TTCD variable may track the amount / degree of constriction made by the tongue tip with the hard palate or the alveolar ridge inthe mouth. The TTCL variables may track the positions and / or relative positions of the speaker’s tongue tip. For example, the TTCL variables may specify the locations along the speaker’s vocal tract that the tongue tip constricts and / or narrows the space.

[0036] The TBCD variables may track the positions and / or relative positions of the speaker’s tongue body. For example, the TBCD variables may describe how tightly the speaker’s tongue body approaches or contacts a constriction location. The TBCL, like the TTCL may track the positions and / or relative positions of the speaker’s tongue body. In various implementations, the vocal tract variables include velic aperture (VEL) variables and / or glottal aperture (GLO) variables. The VEL variables may track the positions and / or relative positions of the speaker’s velum. For example, the VEL variables may describe the opening and / or closing of the speaker’s velum in producing nasal sounds in speech. The GLO variables may track the positions and / or relative positions of the speaker’s glottis. For example, the GLO variables may describe the space between the vocal folds within the speaker’s larynx. In various implementations, the training features generation model 214 includes a DNN, such as a CNN or an RNN. In some examples, the training features generation model 214 employs a hybrid architecture such as an LSTM architecture, a GRU architecture, or a transformer architecture. In various implementations, the training features generation model 214 may include a hidden Markov model (HMM), an autoencoder, or a generative adversarial model (GAN).

[0037] In some examples, the acoustic model 216 is a machine learning model that maps input features representing speech, such as speech embeddings output by the audio feature extraction model 210, to speech representations. The speech representations may include a vector or set of vectors representing PPGs and vocal tract variables present in the speech signal. In various implementations, the input speech embeddings capture information related to the phonetic and articulator characteristics present in a speech signal, and the acoustic model 216 maps the input speech embeddings to output structured speech representations that capture both the linguistic content and the characteristics of the speech signal. For example, the speech representations output by the acoustic model 216 may include PPGs (such as tri-phone PPGs) and vocal tract variables.

[0038] Speech representations that capture both PPGs and vocal tract variables offer a variety of technical benefits for speech-related machine learning applications over speech representations capturing only PPGs or only vocal tract variables. Capturing both PPGs and vocal tract variables in the speech representations may offer improved synergies that make the speech representations more suitable for use in speech-related machine learning applications.For example, the combination of PPGs and vocal tract variables may improve machine-learning - based speech synthesis using the speech representations. The PPGs capture the phonetic content (such as the specific sounds or phonemes that are being produced and their linguistic properties) present in a speech signal, while the vocal tract variables capture the natural variations and nuances of the speaker’s speech. By capturing both the PPGs and vocal tract variables, the speech representations better capture the subtleties of the emotional content, intonation patterns, and / or stress patterns from speech.

[0039] In various implementations, the acoustic model 216 outputs a compact representation of the speech representations. The compact representation may be a lower-dimensional representation of relevant features of the speech representations (e.g., a lower-dimensional representation of relevant features of the PPGs and / or the vocal tract variables). Using the compact representation output from the acoustic model 216 in speech-related machine learning applications (such as applications related to capturing and / or converting accents) may improve the performance of the machine learning models used in those applications. This may be because vocal tract variables can be more strongly correlated with accents than PPGs (for example, because accents may be characterized by variations in pronunciation arising from differences in the way speakers with different accents physically articulate speech sounds). For example, the goal in accent conversion applications is to change the accent of a speaker’s speech from accent to another while keeping the content and identity of the speaker unchanged. Using input features including both PPGs and vocal tract variables together provide a more comprehensive representation of the speech signal.

[0040] PPGs, which may capture the phonetic content of speech, may help in understanding what is being said, irrespective of the speaker’s accent. Vocal tract variables may encode the physiological and articulatory characteristics of how sounds are produced, which may be closely related to the speaker’s accent characteristics. Thus, while PPGs abstract away from the speaker’s identity by focusing on the linguistic content of the speech signal, the vocal tract variables carry information about the speaker’s unique vocal tract configuration (e.g., features related to the speaker’s unique accent). Using PPGs and vocal tract variables together allows the machine learning model to maintain the speaker’s identity by preserving the speaker's unique vocal traits while converting the speaker’s accent, ensuring that the output speech still sounds like the original speaker. Furthermore, because the compact representation is a lowerdimensional representation of relevant features of the PPGs and / or the vocal tract variables, using the compact representation in downstream machine learning applications removes noiseand / or unwanted features from the input dataset and simultaneously reduces the computational burden for the model to process the input dataset.

[0041] FIG. 3 is a schematic illustration of an example architecture of a machine learning model 300 that implements some examples of the acoustic model 216. In various implementations, the machine learning model 300 includes one or more layers. Each layer may refer to a collection of nodes (also referred to as neurons or units) that operate together to perform a specific function on the input data they receive. Thus, each layer may function as a processing step where certain features or patterns in the input data are extracted or transformed. In the following description and in the claims, the various layers of machine learning models (such as the enumerated layers of the machine learning model 300) may be referred to as layers because they may, in some examples, include multiple layers. For example, each of the referenced layers may include an input layer responsible for receiving the input data, one or more hidden layers lying between the input and output layers performing computation, and an output layer that is the final layer and produces the output of the enumerated layers. However, in various implementations, the referenced layers may refer to only an input layer connected to an output layer. In some examples, although referred to in plural form, the referenced layers may include only a single layer.

[0042] In various implementations, the machine learning model 300 includes two bidirectional long short-term memory' (BiLSTM) layers (such as BiLSTM layers 302 and BiLSTM layers 304) followed by upsampling layers 306, dropout layers 308, and two parallel fully-connected layers (such as fully-connected layers 310 and fully-connected layers 312). The BiLSTM layers 302 may accept input features 314, the fully-connected layers 310 may generate output features 316, and the fully-connected layers 312 may generate output features 318. In some examples, BiLSTM layers 302 accept speech embeddings (such as speech embeddings generated by the audio feature extraction model 210) as inputs features 314, fully-connected layers 310 generate output features 316 representing PPGs mapped from the speech embeddings, fully-connected layers 312 generate output features 318 representing vocal tract variables mapped from the speech embeddings, and dropout layers 308 generate output features 320 (which may be a compact representation of relevant features from the PPGs and vocal tract variables).

[0043] The BiLSTM layers 302 and 304 may include one or more RNN layers designed to process sequence data in both forward and backward directions. This bidirectionality allows the BiLSTM layers 302 and 304 to capture temporal context from the input features 314 (e.g., fromboth past and future speech frames), enhancing the ability of the machine learning model 300 to extract temporal dependencies and nuances from the speech embeddings. Additionally, including two BiLSTM layers 302 and 304 in succession may enable the machine learning model 300 to extract and refine high-level features from the input speech embeddings at multiple levels of granularity, improving the representation for dow nstream layers. The upsampling layers 306 may increase the temporal resolution of the feature maps produced by the BiLSTM layers 302 and 304. In implementations where the output features 316 represent PPGs and the output features 318 represent vocal tract variables, it may be beneficial to increase the temporal resolution of the feature maps produced by the BiLSTM layers 302 and 304 because the output features 316 and 318 may have a higher temporal dimensionality than the input features 314.

[0044] The dropout layers 308 may function as a form of regularization used to prevent overfitting. For example, the dropout layers 308 may randomly drop out (or set to zero) a fraction of their outputs to force the downstream fully-connected layers 310 and 312 to leam more robust features. The dropout layers 308 may generate output features 320, which are a compact representation of relevant features mapped from the input features 314 that the fully- connected layers 310 and 312 may use to generate PPGs and vocal tract vectors (e.g., as represented by output features 316 and 318, respectively). Since the output features 320 are a compact representation of the input features 314 and the downstream fully-connected layers 310 and 312 generate output features 316 and 318 using the output features 320 as an input, the output features 320 may be referred to as “bottleneck features.”

[0045] As the fully-connected layers 310 and 312 may generate output features 316 and 318 representing the phonetic content of speech (such as PPGs) and articulatory characteristics of speech (such as vocal tract variables) from the output features 320, the output features 320 may be considered a compact representation or a compressed representation of the relevant content of the output features 316 and 318. For example, for the fully-connected layers 310 and 312 to be able to recreate the phonetic content of speech (such as PPGs) and the articulatory characteristics of speech (such as vocal tract variables) from the output features 320, the outputs features 320 may need to include distilled essential features of the phonetic content and the articulatory characteristics of speech. How ever, as a compact or compressed representation, the output features 320 may contain less content than the output features 316 and 318.

[0046] Fully-connected layers 310 and 312 may be parallel branches downstream of the dropout layers 308 (e.g., the output features 320 from the dropout layers 308 may be provided in parallel to both fully-connected layers 310 and fully-connected layers 312). In variousimplementations, the fully -connected layers 310 and 312 may be traditional neural network layers where every input is connected to every output by a learned weight and / or bias. The fully- connected layers 310 and 312 may serve as shared output layers allowing the machine learning model 300 to output both output features 316 and 318.

[0047] In various implementations, the fully-connected layers 310 map the features extracted by the previous layers 302-308 to PPGs. In various some examples, the nodes of the fully- connected layers 310 may have a softmax activation function. In various implementations, the fully-connected layers 312 map the features extracted by the previous layers 302-308 to vocal tract variables. In some examples, the nodes of the fully-connected layers 312 may have a tank activation function. Thus, together, the layers 302-312 of the machine learning model 300 may form a comprehensive model that takes speech embeddings as input features 314 and outputs detailed phonetic representations (e.g., PPGs) as output features 316, vocal tract variables as output features 318, and a compact representation of relevant features of the PPGs and / or vocal tract variables as output features 320, facilitating further advanced speech processing applications (such as accent conversion or extraction applications) using the output features 320.

[0048] In various implementations, the input features 314 may be a set of speech embedding vectors that sample a speech signal at 50 Hz, with each vector having a dimensionality of 1024. In some examples, the output features 316 may be a set of vectors that sample the speech signal at 100 Hz, with each vector having a dimensionality' of 5816. In various implementations, the output features 318 may be a set of vectors that sample the speech signal at 100 Hz, with each vector having a dimensionality of 6. In some examples, the output features 320 may be a set of vectors that sample the speech signal at 100 Hz, with each vector having a dimensionality of 256.

[0049] FIG. 4 is a flowchart of an example process 400 for training the machine learning model 300. In the example process 400, the audio processing application 208 generates reference speech features (at block 402). For example, the audio processing application 208 may generate MFCCs and / or speech embeddings (e.g., using the audio feature extraction model 210) representing reference speech signals from reference audio data 228. FIG. 5 is a schematic illustration of example reference audio data 228. In various implementations, the reference audio data 228 includes reference native speech signals 502 and / or reference non-native speech signals 504. The reference native speech signals 502 may include speech signals representing one or more native speakers of a language reciting one or more passages of speech. The nonnative speech signals 504 may include speech signals representing one or more non-nativespeakers of the language reciting one or more passages of speech. In various implementations, the audio processing application 208 generates reference native speech features 506 (e.g., MFCCs and / or speech embeddings) representing the reference native speech signals 502. In some examples, the audio processing application 208 generates reference non-native speech features 508 (e.g., MFCCs and / or speech embeddings) representing the reference non-native speech signals 504.

[0050] Returning to FIG. 4, the reference speech features generated at block 402 may include the reference native speech features 506 and / or the reference non-native speech features 508. In the example process 400, the audio processing application 208 generates a first target dataset for training the machine learning model 300 (at block 404). For example, the audio processing application 208 provides speech features from the reference native speech features 506 and / or the reference non-native speech features 508 to the training features generation model 212 to generate the first target dataset. In various implementations, the first target dataset may include target PPGs generated from the reference native speech features 506 and / or the reference non- native speech features 508. The audio processing application 208 may save the target PPGs to the reference audio data 228 as reference PPGs 510 (illustrated in FIG. 5).

[0051] In the example process 400, the audio processing application 208 generates a second target dataset for training the machine learning model 300 (at block 406). For example, the audio processing application 208 provides speech features from the reference native speech features 506 and / or the reference non-native speech features 508 to the training features generation model 214 to generate the second target dataset. In various implementations, the second target dataset may include target vocal tract variables generated from the reference native speech features 506 and / or the reference non-native speech features 508. The audio processing application 208 may save the target vocal tract variables to the reference audio data 228 as reference vocal tract features 512 (illustrated in FIG. 5).

[0052] In the example process 400, the audio processing application 208 trains the acoustic model 216 using the reference speech features to minimize a combined loss of the first and second target dataset (at block 408). For example, during training, the audio processing application 208 provides the reference speech features to the machine learning model 300 (e.g., as input features 314 to the BiLSTM layers 302). In various implementations, the reference speech features include only the reference native speech features 506. In some examples, the reference speech features include only the non-native speech features 508. In various implementations, the reference speech features include both the reference native speech features506 and the non-native speech features 508. The machine learning model 300 may generate output features 316 representing PPGs and output features 318 representing vocal tract variables (e.g., as outputs from the fully-connected layers 310 and the fully-connected layers 310, respectively). The audio processing application 208 may adjust the machine learning model 300 to minimize a loss of the model in estimating PPGs (PPGioss) and a loss of the model in estimating vocal tract variables (TVioss . In various implementations, the audio processing application 208 determines PPGioss by comparing the estimated PPGs from the output features 316 against the reference PPGs 510. In some examples, the audio processing application 208 determines TVioss by comparing the estimated vocal tract variables from the output features 318 against the reference vocal tract features 512.

[0053] In various implementations, the audio processing application 208 adjusts the machine learning model 300 to minimize a combined loss function (AMLOSScombtned) defined by Equation (1) below:AMLOSScombined= a x TVloss+ (1 - a) X PPGloss(1)

[0054] In some examples, a may be set to about 1. In examples where a is set to about 1, the combined loss function minimizes only PPGioss. which trains the machine learning model 300 to minimize PPGioss while ignoring TVioss. In some examples, a may be set to about 0. In examples where a is set to about 0, the combined loss function minimizes only TVioss, which trains the machine learning model 300 to minimize TVioss while ignoring PPGioss. Accordingly, to minimize both PPGioss and TVioss, a may be set to a value between 0 and 1, such as between approximately 0.35 and 0.65 (including, for example, 0.40, 0.45, 0.5, 0.55, or values therebetween) to minimize both PPGioss and TVioss in a balanced manner. Implementations where a is set to between 0 and 1 allow the audio processing application 208 to train the machine learning model 300 to accurately estimate both PPGs and vocal tract vectors and may be particularly suitable for speech-related applications.

[0055] Returning to FIG. 2, in various implementations, the speaker encoder model 218 is a machine learning model that generates speaker embeddings from speech signals or input speech features (such as MFCCs or speech embeddings generated by the audio feature extraction model 210). The speaker encoder model 218 may extract a compact representation from the input speech signals or speech features that capture unique characteristics of a speaker’s voice. For example, the speaker embeddings may capture information about the speaker’s pitch, tone, timbre, and / or accent, and / or other vocal attributes of the speaker. In various implementations,the speaker encoder model 218 may include time-delay neural networks (TDNNs), LSTM networks, CNNs. Gaussian mixture models (GMMs), RNNs, and / or GRUs.

[0056] In some examples, the translator model 220 is a machine learning model that translates compact representations of the speech of a non-native speaker (such as the output features 320 generated by the dropout layers 308 of the machine learning model 300) to compact representations that would have been generated by the acoustic model 216 from inputs representing a native speaker speaking the same passage of speech. The output compact representations of the translator model 220 retain representations of the original message and intent of the speech but modify representations of the pronunciation patterns so that they match those of a native speaker. In various implementations, the translator model 220 may be a sequence-to-sequence model employing an encoder-decoder architecture. In some examples, the encoder portion of the sequence-to-sequence model encodes the compact representation output from the acoustic model 216 into a high-dimensional representation, and the decoder portion generates compact representations matching those that the acoustic model 216 would have generated had a native speaker spoken the same passage of speech.

[0057] In various implementations, the translator model 220 is trained on parallel data including reference compact representations from non-native speakers and corresponding reference compact representations from native speakers speaking the same content. For example, the reference non-native compact representations may be input to the translator model 220, and the translator model 220 outputs predicted native compact representations. The translator model 220 may be trained to minimize losses between the output predicted native compact representations and corresponding reference native compact representations.

[0058] In some examples, the prosody encoder model 222 is a machine learning model that generates prosodic features from input speech signals or speech features (e.g., MFCCs or speech embeddings generated by the audio feature extraction model 210). Prosodic features may be aspects of speech that go beyond the individual phonemes (e.g., the units of sound that distinguish one word from another) and include the rhythm, stress, intonation, and / or the overall melody of the speech. For example, the prosodic features may include pitch, loudness, duration, stress, rhythm, intonation, and / or pauses. Pitch may describe the perceived frequency of sound. Loudness may describe the intensity of amplitude of a sound wave. Duration may describe the length of time that phonemes, syllables, words, and / or phrases are held during speech. Stress may describe the emphasis placed on certain syllables or words within a phrase or sentence. Rhythm may describe the pattern of stresses, syllables, and / or pauses in speech. Intonation maydescribe the variation of pitch over the course of a phrase or sentence. Pauses may describe breaks in speech. In various implementations, the prosody encoder model 222 may include a HMM, a DNN, a CNN, an RNN, an LSTM, or a transformer model.

[0059] In some examples, the synthesizer model 224 is a machine learning model that generates a predicted mel-spectrogram from input compact representations (e.g., generated by the acoustic model 216 or the translator model 220), prosodic features (e.g., generated by the prosody encoder model 222), and / or speaker embeddings (e.g., generated by the speaker encoder model 218). A mel-spectrogram may be a representation (such as a visual representation) of frequencies of a sound signal as it varies with time. Mel-spectrograms may use the mel-scale to map frequencies (as opposed to a linear scale). The mel-scale may be a perceptual scale of pitches judged by listeners to be equal in distance from one other, which may more closely approximate human hearing. Mel-spectrograms may provide a compact and perceptually relevant representation of sound making them especially suited for use as inputs into machine learning models for speech-related applications.

[0060] In various implementations, the synthesizer model 224 is a sequence-to-sequence synthesizer that employs an encoder-decoder model. The encoder portion may process the compact representations and the prosodic features to create a representation that captures the complex relationships between the linguistic content, vocal tract characteristics, and prosody of the speech. The decoder portion may condition on the speaker embeddings and the context provided by the encoder to generate the output mel-spectrogram (frame by frame). In some examples, the prosody encoder model 222 and the synthesizer model 224 may be trained together in an unsupervised fashion to reconstruct a mel-spectrogram based on input compact representations, prosodic features, and speaker embeddings. For example, reference input compact representations, reference prosodic features, reference speaker embeddings, and reference mel-spectrograms may be generated for reference speech signals. The prosody encoder model 222 and the synthesizer model 224 may be trained together to minimize the loss between the predicted mel-spectrograms output from the synthesizer model 224 and the reference mel-spectrograms corresponding to a given set of reference inputs.

[0061] In some examples, the vocoder model 226 is a speech synthesis model that generates an audio signal of synthesized speech from input mel-spectrograms (such as the mel- spectrograms generated by the synthesizer model 224). In various implementations, the vocoder model 226 uses a machine learning model, such as a neural network to convert the input mel- spectrogram back into a waveform that can be played as sound.

[0062] FIG. 6 is a block diagram 600 of an example implementation of a process for generating compact representations from speech signals. FIG. 7 is a message sequence chart 700 illustrating example interactions between components of the audio processing and transformation platform 102 as the platform 102 automatically generates compact representations from speech signals. Referring to FIGS. 6 and 7 together, in the message sequence chart 700, the audio processing application 208 sends a speech signal 602 to the audio feature extraction model 210 (at operation 702). In the message sequence chart 700, the acoustic feature extraction model 210 generates speech embeddings 604 from the speech signal 602 (at operation 704). As previously described, the speech embeddings 604 may be a lower-dimensional representation of relevant features of the speech signal 602 and capture temporal and contextual relationships between sounds in the speech signal 602. As previously described, reducing the dimensionality and number of features present in the speech signal 602 reduces the computational burden of dow nstream models using the speech embeddings 604 as input and reduces the likelihood of the downstream models overfitting to noise in the speech signal 602 during training.

[0063] In the message sequence chart 700, the audio feature extraction model 210 sends the speech embeddings 604 to the acoustic model 216 (at operation 706). The acoustic model 218 generates compact representations 606 from the speech embeddings 604 (at operation 708). In various implementations, the speech embeddings 604 are provided to the BiLSTM layers 302 of the machine learning model 300 as input features 314 and the compact representations 606 are the output features 320 generated by the dropout layers 308 of the machine learning model 300. As previously described, the compact representations 606 may include representations of relevant features of the speech embeddings 604 that can be mapped to PPGs and / or vocal tract variables.

[0064] FIG. 8 is a block diagram 800 of an example implementation of a process for generating a synthesized speech signal from a reference native speech signal and a non-native speech signal. FIG. 9 is a message sequence chart 900 illustrating example interactions between components of the audio processing and transformation platform 102 as the platform 102 automatically generates a synthesized speech signal from the reference native speech signal and the non-native speech signal. Referring to FIGS. 8 and 9 together, the non-native speech signal may represent audio from a non-native speaker of a language reciting one or more passages of speech and the reference native speech signal may represent audio from a native speaker of the language reciting the same one or more passages of speech. The synthesized speech signal retains the original content and unique characteristics of the speaker’s voice from the non-native speech signal but modifies the accent to match that of a native speaker. Thus, while thesynthesized speech signal transforms the accent of the non-native speaker, it leaves the content of the speech and the distinctive qualities of the speaker's voice unchanged. Accordingly, the block diagram 800 and the message sequence chart 900 describe the acoustic model 216 and the audio processing and transformation platform 102 performing accent conversion applications.

[0065] In the message sequence chart 900, the audio processing application 208 sends the reference native speech signal 802 to the audio feature extraction model 210 (at operation 902). In the message sequence chart 900, the audio feature extraction model 210 generates speech embeddings 804 from the reference native speech signal 802 (at operation 904). In the message sequence chart 900, the audio feature extraction model 210 sends the speech embeddings 804 to acoustic model 216 (at operation 906). In the message sequence chart 900, the acoustic model 216 generates compact representations 806 from the speech embeddings 804 (at operation 908). In various implementations, the speech embeddings 804 are provided to the BiLSTM layers 302 of the machine learning model 300 as input features 314 and the compact representations 806 are the output features 320 generated by the dropout layers 308 of the machine learning model 300.

[0066] In the message sequence chart 900, the audio processing application 208 sends the non-native speech signal 808 to the prosody encoder model 222 (at operation 910). In the message sequence chart 900. the prosody encoder model 222 generates prosodic features 810 from the non-native speech signal 808 (at operation 910). Thus, the prosodic features 810 may capture the prosodic elements of the non-native speaker reciting the one or more passages of speech. In the message sequence chart 900, the audio processing application 208 sends the non- native speech signal 812 to the speaker encoder model 218 (at operation 914). In various implementations, the non-native speech signal 812 may be identical to the non-native speech signal 808. In some examples, the non-native speech signals 808 and 812 may take different forms but represent the same audio content (e.g., as different encodings of the same audio content or be MFCCs and / or speech embeddings generated from the audio content). In the message sequence chart 900. the speaker encoder model 218 generates speaker embeddings 814 from the non-native speech signal 812 (at operation 916). In various implementations, the speaker embeddings 814 capture the unique characteristics of the non-native speaker’s voice from the non-native speech signal 812.

[0067] In the message sequence chart 900, the acoustic model 216 sends the compact representations 806 generated at operation 908 to the synthesizer model 224 (at operation 918). In the message sequence chart 900, the prosody encoder model 222 sends the prosodic features 810 generated at operation 912 to the synthesizer model 224 (at operation 920). In the messagesequence chart 900, the speaker encoder model 218 sends the speaker embeddings 814 generated at operation 916 to the synthesizer model 224 (at operation 922). In the message sequence chart 900, the synthesizer model 224 generates mel -spectrograms 81 from the compact representations 806, the prosodic features 810, and the speaker embeddings 814 (at operation 924). In various implementations, the mel-spectrograms 816 represent the speech content of the non-native speech signals 808 and 812 but with the accent of the non-native speaker replaced with the accent of a native speaker. In the example message sequence chart 900, the synthesizer model 224 sends the mel-spectrograms 816 generated at operation 924 to the vocoder model 226 (at operation 926). In the example message sequence chart 900, the vocoder model 226 generates the synthesized speech signal 818 from the mel-spectrograms 816 (at operation 928). In various implementations, the synthesized speech signal 818 may have the same speech content as the non-native speech signals 808 and 812 but replaces the accent of the non-native speaker with the accent of a native speaker.

[0068] FIG. 10 is a block diagram 1000 of an example implementation of a process for generating a synthesized speech signal from anon-native speech signal. FIG. 11 is a message sequence chart 1100 illustrating example interactions between components of the audio processing and transformation platform 102 as the platform 102 automatically generates a synthesized speech signal from the non-native speech signal. Referring to FIGS. 10 and 11 together, the non-native speech signal may represent speech from audio of a non-native speaker of a language reciting one or more passages of speech. The synthesized speech signal may retain the original content and unique characteristics of the non-native speaker’s voice from the non- native speaker but modifies the accent to match that of a native speaker. Thus, while the synthesized speech signal transforms the accent of the non-native speaker, it leaves the content of the speech and the distinctive qualities of the non-native speaker’s voice unchanged.Accordingly, the block diagram 1000 and the message sequence chart 1100 describe the acoustic model 216 and the audio processing and transformation platform 102 performing accent conversion applications.

[0069] In the message sequence chart 1100, the audio processing application 208 sends the non-native speech signal 1002 to the audio feature extraction model 210 (at operation 1102). In the message sequence chart 1100, the audio feature extraction model 210 generates speech embeddings 1004 from the non-native speech signal 1002 (at operation 1 104). In the message sequence chart 1100, the audio feature extraction model 210 sends the speech embeddings 1004 to the acoustic model 216 (at operation 1106). In the message sequence chart 1100, the acoustic model 216 generates compact representations 1006 from the speech embeddings 1004 (atoperation 1108). In various implementations, the speech embeddings 1004 are provided to the BiLSTM layers 302 of the machine learning model 300 as input features 314 and the compact representations 1006 are the output features 320 generated by the dropout layers 308 of the machine learning model 300.

[0070] In the message sequence chart 1100, the acoustic model 216 sends the compact representations 1006 to the translator model 220 (at operation 1110). In the message sequence chart 1100, the translator model 220 generates compact representations 1008 from the compact representations 1006 (at operation 1112). In various implementations, translator model 220 generates the compact representations 1008 by modifying the representations of the pronunciation patterns of the non-native speaker in the compact representations 1006 to match those of a native speaker while retaining the representations of the original message and intent of the speech. Thus, the compact representations 1008 may simulate compact representations that would have been generated from a speech signal of a native speaker reciting the same speech content as recited by the non-native speaker in the non-native speech signal 1002.

[0071] In the message sequence chart 1100, the audio processing application 208 sends the non-native speech signal 1010 to the prosody encoder model 222 (at operation 1114). In various implementations, the non-native speech signal 1010 may be identical to the non-native speech signal 1002. In some examples, the non-native speech signals 1002 and 1010 may take different forms but represent the same audio content (e.g., as different encodings of the same audio content or be MFCCs and / or speech embeddings generated from the audio content). In the message sequence chart 1100, the prosody encoder model 222 generates prosodic features 1012 from the non-native speech signal 1010 (at operation 11 16). The prosodic features 1012 may capture the prosodic elements of the non-native speaker reciting the one or more passages of speech.

[0072] In the message sequence chart 1100, the audio processing application 208 sends the non-native speech signal 1014 to the speaker encoder model 218 (at operation 1118). In various implementations, the non-native speech signal 1014 may be identical to the non-native speech signals 1002 and 1010. In some examples, the non-native speech signals 1002, 1010, and / or 1014 may take different forms but represent the same audio content (e.g.. as different encodings of the same audio content or be MFCCs and / or speech embeddings generated from the audio content). In the message sequence chart 1100, the speaker encoder model 218 generates speaker embeddings 1016 from the non-native speech signal 1014 (at operation 1120). The speaker embeddings 1016 may capture unique characteristics of the non-native speaker's voice from thenon-native speech signal 1014. In the message sequence chart 1100, the translator model 220 sends the compact representations 1008 to the synthesizer model 224 (at operation 1122). In the message sequence chart 1100, the prosody encoder model 222 sends the prosodic features 1012 to the synthesizer model 224 (at operation 1124). In the message sequence chart 1100, the speaker encoder model 218 sends the speaker embeddings 1016 to the synthesizer model 224 (at operation 1126).

[0073] In the message sequence chart 1100, the synthesizer model generates mel- spectrograms 1018 from the compact representations 1008, the prosodic features 1012, and the speaker embeddings 1016 (at operation 1128). In various implementations, the mel- spectrograms 1018 represent the speech content of the non-native speech signals 1002, 1010, and 1014 but with the accent of the non-native speaker replaced with the accent of a native speaker. In the message sequence chart 1100, the synthesizer model 224 sends the mel-spectrograms 1018 to the vocoder model 226 (at operation 1130). In the message sequence chart 1100, the vocoder model 226 generates the synthesized speech signal 1020 from the mel-spectrograms 1018 (at operation 1 132). In various implementations, the synthesized speech signal 1020 may have the same speech content as the non-native speech signals 1002, 1010, and 1014 but replaces the accent of the non-native speaker with the accent of a native speaker.

[0074] In various implementations, the accent conversion techniques described with reference to FIGS. 8-11 have a variety of applications. For example, users may live stream content from a user device 104-1 to other user devices 104 (such as user device 104-2) via the communications system 110. In examples where the user is streaming content in anon-native language, the user device 104-1 may transmit the content to the audio processing and transformation platform 102, which converts the accent of the content to that of a native speaker in real time or near-real time before transmitting the content with the converted accent to the user device 104-2. In other examples, the media delivery’ platform 106 may transmit content to the audio processing and transformation platform 102 to convert the accent of the content (for example, in real time or near-real time) before transmitting the content with the converted accent to the user devices 104. In various implementations, the speech recognition platform 108 may convert the accent of content spoken by a non-native speaker to the accent of a native speaker before performing speech recognition operations on the content (for example, to improve the effectively of speech recognition models trained using content spoken by native speakers).

[0075] Additional use cases may be achieved by modifying the training data used to train the acoustic model 216. In various implementations, media content such as movies or televisionshows are dubbed from a non-native language into a native language. During the dubbing process, native speakers having accents and / or speech characteristics distinct from those of the original actors may be used to record the native language dub. Techniques described in this specification may be used to facilitate the transformation of the original non-native accented speech into the native language, all while retaining the unique vocal identity of the original performers. In some examples, non-native speakers may be able to acquire native pronunciation more rapidly when they are able to hear themselves pronounce words with a native accent. Techniques described in this specification may be used to generate audio of non-native speakers pronouncing words or speech with a native accent to facilitate their acquiring native pronunciation.

[0076] In various implementations, the compact representations output from the acoustic model 216 can be used along with a custom trained translator model 220 to synthesize enhanced speech from unintelligible speech (e.g., where the speaker is mumbling or where the audio recording is corrupted) to improve the quality of the speech. Such applications may be used to increase the intelligibility of speech in cinema and other video content. In some examples, the compact representations output from the acoustic model 216 can be used with a translator model 220 trained to map input speech from a given speaker to another target speaker for zero-shot voice conversion (e.g., to modify the input speech to sound like speech of the target speaker).

[0077] In various implementations, the compact representations output from the acoustic model 216 may encode articulatory-level information that can be used to detect speech disorders and help speech therapists design corrective mechanisms. In some examples, the compact representations output from the acoustic model 216 can be used along with a custom trained translator model 220 to synthesize enhanced speech for speakers with speech disorders (e.g., to improve the clarity of speech from speakers with speech disorders).

[0078] The following paragraphs provide examples of systems, methods, and devices implemented in accordance with this specification.

[0079] Example 1. A computer-implemented method comprising: providing input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features; wherein the input features comprise a lower- dimensionality representation of a speech signal; and wherein the first compact representations comprise lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

[0080] Example 2. The computer-implemented method of example 1, further comprising generating first speech representations and second speech representations mapped from the input features.

[0081] Example 3. The computer-implemented method of example 2, wherein the first speech representations comprise representations of phonetic posteriorgrams (PPGs).

[0082] Example 4. The computer-implemented method of any one of examples 2 or 3, wherein the second speech representations comprise representations of vocal tract variables.

[0083] Example 5. The computer-implemented method of any one of examples 2-4, wherein: the first compact representations are a compressed representation of the phonetic content of speech and the articulatory characteristics of speech; and the first compact representations are different from the first speech representations and the second speech representations.

[0084] Example 6. The computer-implemented method of any one of examples 2-5, wherein: the acoustic model is trained to minimize a combined loss function; and the combined loss function comprises (i) a loss of the first speech representations and (ii) a loss of the second speech representations.

[0085] Example 7. The computer-implemented method of any one of examples 1-6. wherein the acoustic model comprises: first bidirectional long short-term memory (BiLSTM) layers; second BiLSTM layers; upsampling layers; dropout layers; first fully -connected layers; and second fully-connected layers.

[0086] Example 8. The computer-implemented method of example 7, wherein: the first BiLSTM layers accept the input features; the second BiLSTM layers are downstream of the first BiLSTM layers; the upsampling layers are dow nstream of the second BiLSTM layers; the dropout layers are downstream of the upsampling layers; and the first fully-connected layers and the second fully -connected layers are parallel layers and downstream of the dropout layers.

[0087] Example 9. The computer-implemented method of any one of examples 7 or 8, wherein nodes of the first fully-connected layers have softmax activation functions.

[0088] Example 10. The computer-implemented method of any one of examples 7-9, wherein nodes of the second fully-connected layers have tanh activation functions.

[0089] Example 11. The computer-implemented method of any one of examples 1-10, further comprising: providing a speech signal to an acoustic feature extraction model to generate speech embeddings; wherein the speech embeddings are a lower-dimensional representation of features of the speech signal; and wherein the input features comprise the speech embeddings.

[0090] Example 12. The computer-implemented method of any one of examples 1-11, further comprising: providing a non-native speech signal to a prosody encoder model to generate prosodic features; providing the non-native speech signal to a speaker encoder model to generate speaker embeddings; and providing the first compact representations, the prosodic features, and the speaker embeddings to a synthesizer model to generate a synthesized speech signal; wherein the input features are a representation of a native speech signal; wherein the synthesized speech signal includes a same speech content as the non-native speech signal and an accent of a native speaker of a language.

[0091] Example 13. The computer-implemented method of example 12, wherein the synthesizer model comprises a sequence-to-sequence synthesizer.

[0092] Example 14. The computer-implemented method of any one of examples 1-11, further comprising: providing the first compact representations to a translator model to generate second compact representations; providing a non-native speech signal to a prosody encoder model to generate prosodic features; providing the non-native speech signal to a speaker encoder model to generate speaker embeddings; and providing the second compact representations, the prosodic features, and the speaker embeddings to a synthesizer model to generate a synthesized speech signal; wherein the input features are a representation of the non-native speech signal; and wherein the synthesized speech signal includes a same speech content as the non-native speech signal and an accent of a native speaker of a language.

[0093] Example 15. The computer-implemented method of example 14, wherein the synthesizer model comprises a sequence-to-sequence synthesizer.

[0094] Example 16. The computer-implemented method of example 15, wherein the translator model comprises a sequence-to-sequence translator model.

[0095] Example 17. The computer-implemented method of any one of examples 15 or 16, wherein the translator model generates second compact representations by modifying representations of pronunciation patterns in the first compact representation to match those of a native speaker.

[0096] Example 18. The computer-implemented method of example 17, wherein providing the second compact representations, the prosodic features, and the speaker embeddings to the synthesizer model to generate the synthesized speech signal comprises: providing the second compact representations, the prosodic features, and the speaker embeddings to the synthesizer model to generate mel-spectrograms; and providing the mel-spectrograms to a vocoder model to generate the synthesized speech signal.

[0097] Example 19. A non-transitory computer-readable medium comprising executable instructions, which when executed by an electronic processor causes the electronic processor to perform the method of any one of examples 1-18.

[0098] Example 20. A system comprising: memory hardware storing instructions; and processor hardware configured to execute the instructions, wherein executing the instructions causes the system to perform the method of any one of examples 1-18.

[0099] The foregoing description is merely illustrative in nature and does not limit the scope of the disclosure or its applications. The broad teachings of the disclosure may be implemented in many different ways. While the disclosure includes some particular examples, other modifications will become apparent upon a study of the drawings, the text of this specification, and the following claims. In the written description and the claims, one or more processes within any given method may be executed in a different order — or processes may be executed concurrently or in combination with each other — without altering the principles of this disclosure. Similarly, instructions stored in a non-transitory computer-readable medium may be executed in a different order — or concurrently — without altering the principles of this disclosure. Unless otherwise indicated, the numbering or other labeling of instructions or method steps is done for convenient reference and does not necessarily indicate a fixed sequencing or ordering.

[0100] Unless the context of their usage unambiguously indicates otherwise, the articles '‘a,” “an,” and “the” should not be interpreted to mean “only one.” Rather, these articles should be interpreted to mean “at least one” or “one or more.” Likewise, when the terms “the” or “said” are used to refer to a noun previously introduced by the indefinite article “a” or “an,” the terms “the” or “said” should similarly be interpreted to mean “at least one” or “one or more” unless the context of their usage unambiguously indicates otherwise.

[0101] Spatial and functional relationships between elements — such as modules — are described using terms such as (but not limited to) “connected,” “engaged,” “interfaced,” and / or“coupled.” Unless explicitly described as being “direct,” relationships between elements may be direct or include intervening elements. The phrase “at least one of A. B, and C” should be construed to indicate a logical relationship (A OR B OR C), where OR is a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The term “set” does not necessarily exclude the empty set. For example, the term “set” may have zero elements. The term “subset” does not necessarily require a proper subset. For example, a “subset” of set A may be coextensive with set A, or include elements of set A. Furthermore, the term “subset” does not necessarily exclude the empty set.

[0102] In the figures, the directions of arrows generally demonstrate the flow of information — such as data or instructions. The direction of an arrow does not imply that information is not being transmitted in the reverse direction. For example, when information is sent from a first element to a second element, the arrow may point from the first element to the second element. However, the second element may send requests for data to the first element, and / or acknowledgements of receipt of information to the first element. Furthermore, while the figures illustrate a number of components and / or steps, any one or more of the components and / or steps may be omitted or duplicated, as suitable for the application and setting.

[0103] The term computer-readable medium does not encompass transitory electrical or electromagnetic signals or electromagnetic signals propagating through a medium — such as on an electromagnetic carrier wave. The term “computer-readable medium” is considered tangible and non-transitory. The functional blocks, flowchart elements, and message sequence charts described above serve as software specifications that can be translated into computer programs by the routine work of a skilled technician or programmer.

[0104] It should also be understood that although certain drawings illustrate hardware and software as being located within particular devices, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware, and / or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device, or they may be distributed among different computing devices — such as computing devices interconnected by one or more networks or other communications systems.

[0105] In the claims, if an apparatus or system is claimed as including an electronic processor or other element configured in a certain manner, the claim or claimed element should be interpreted as meaning one or more electronic processors (or other element as appropriate). If the electronic processor (or other element) is described as being configured to make one or more determinations or one or execute one or more steps, the claim should be interpreted to mean that any combination of the one or more electronic processors (or any combination of the one or more other elements) may be configured to execute any combination of the one or more determinations (or one or more steps).

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: providing input features representing a speech signal to an input layer of an acoustic model to generate first compact representations mapped from the input features; wherein the input features comprise a lower-dimensionality representation of a speech signal; and wherein the first compact representations comprise lower-dimensional representations of (i) phonetic content of speech mapped from the speech signal and (ii) articulatory characteristics of speech mapped from the speech signal.

2. The computer-implemented method of claim 1. further comprising generating first speech representations and second speech representations mapped from the input features.

3. The computer-implemented method of claim 2, wherein the first speech representations comprise representations of phonetic posteriorgrams (PPGs).

4. The computer-implemented method of any one of claims 2 or 3, wherein the second speech representations comprise representations of vocal tract variables.

5. The computer-implemented method of any one of claims 2-4, wherein: the first compact representations are a compressed representation of the phonetic content of speech and the articulatory characteristics of speech; and the first compact representations are different from the first speech representations and the second speech representations.

6. The computer-implemented method of any one of claims 2-5, wherein: the acoustic model is trained to minimize a combined loss function; and the combined loss function comprises (i) a loss of the first speech representations and (ii) a loss of the second speech representations.

7. The computer-implemented method of any one of claims 1-6, wherein the acoustic model comprises: first bidirectional long short-term memory (BiLSTM) layers; second BiLSTM layers; upsampling layers; dropout layers; first fully-connected layers; and second fully-connected layers.

8. The computer-implemented method of claim 7, wherein: the first BiLSTM layers accept the input features; the second BiLSTM layers are dow nstream of the first BiLSTM layers; the upsampling layers are downstream of the second BiLSTM layers; the dropout layers are downstream of the upsampling layers; and the first fully-connected layers and the second fully-connected layers are parallel layers and downstream of the dropout layers.

9. The computer-implemented method of any one of claims 7 or 8, wherein nodes of the first fully-connected layers have softmax activation functions.

10. The computer-implemented method of any one of claims 7-9, wherein nodes of the second fully-connected layers have tanh activation functions.

11. The computer-implemented method of any one of claims 1-10, further comprising: providing a speech signal to an acoustic feature extraction model to generate speech embeddings; w herein the speech embeddings are a low er-dimensional representation of features of the speech signal; and wherein the input features comprise the speech embeddings.

12. The computer-implemented method of any one of claims 1-11, further comprising: providing a non-native speech signal to a prosody encoder model to generate prosodic features; providing the non-native speech signal to a speaker encoder model to generate speaker embeddings; and providing the first compact representations, the prosodic features, and the speaker embeddings to a synthesizer model to generate a synthesized speech signal; wherein the input features are a representation of a native speech signal; wherein the synthesized speech signal includes a same speech content as the non-native speech signal and an accent of a native speaker of a language.

13. The computer-implemented method of claim 12, wherein the synthesizer model comprises a sequence-to-sequence synthesizer.

14. The computer-implemented method of any one of claims 1-11, further comprising: providing the first compact representations to a translator model to generate second compact representations; providing a non-native speech signal to a prosody encoder model to generate prosodic features; providing the non-native speech signal to a speaker encoder model to generate speaker embeddings; and providing the second compact representations, the prosodic features, and the speaker embeddings to a synthesizer model to generate a synthesized speech signal; wherein the input features are a representation of the non-native speech signal; and wherein the synthesized speech signal includes a same speech content as the non-native speech signal and an accent of a native speaker of a language.

15. The computer-implemented method of claim 14, wherein the synthesizer model comprises a sequence-to-sequence synthesizer.

16. The computer-implemented method of claim 15, wherein the translator model comprises a sequence-to-sequence translator model.

17. The computer-implemented method of any one of claims 15 or 16, wherein the translator model generates second compact representations by modifying representations of pronunciation patterns in the first compact representation to match those of a native speaker.

18. The computer-implemented method of claim 17, wherein providing the second compact representations, the prosodic features, and the speaker embeddings to the synthesizer model to generate the synthesized speech signal comprises: providing the second compact representations, the prosodic features, and the speaker embeddings to the synthesizer model to generate mel-spectrograms; and providing the mel-spectrograms to a vocoder model to generate the synthesized speech signal.

19. A non-transitory computer-readable medium comprising executable instructions, which when executed by an electronic processor causes the electronic processor to perform the method of any one of claims 1-18.

20. A system comprising: memory hardware storing instructions; and processor hardware configured to execute the instructions, wherein executing the instructions causes the system to perform the method of any one of claims 1-18.

Images