Adapting and training neural speech synthesis

Abstract

Systems, methods, and other implementations for speech generation are disclosed, including a method that includes obtaining a speech sample of a target speaker, processing the speech sample using a trained encoder to generate a parametric representation of the speech sample of the target speaker, receiving configuration data for a speech synthesis system that accepts the parametric representation as input, and adapting the configuration data of the speech synthesis system with the input including the parametric representation and a time-domain representation of the speech sample to generate adapted configuration data for the speech synthesis system. The method further includes resulting in a configuration of the speech synthesis system with the adapted configuration data, the speech synthesis system being implemented to generate synthetic speech output data having estimated voice and time-domain voice characteristics that approximate the actual voice and time-domain voice characteristics of the target speaker.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is an international application claiming priority to U.S. Provisional Patent Application No. 63 / 288,907, filed December 13, 2021, the contents of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates to voice cloning, and more particularly to voice cloning of a target speaker who provides a short (e.g., 10 minutes, although other sample lengths shorter (e.g., 2-3 minutes) or longer than 10 minutes may alternatively be used) voice sample based on which a natural-sounding voice is synthesized incorporating the speaker's specific voice characteristics (accent, style, prosody, etc.).

[0003] Speech synthesis systems process text input to generate output speech that aims to emulate human speech (such systems are also called Text-to-Speech (TTS) systems). For example, various techniques for generating speech output based on phonetic and prosodic information include sample-based and parameter-based techniques. Sample-based techniques may use a database of pre-recorded speech samples. The phonetic and prosodic information may be used as the basis for both selecting a set of pre-recorded speech samples and concatenating this selected set together to form the output speech. The overall performance of sample-based techniques may depend on the size of the database of pre-recorded speech samples and / or the manner in which the pre-recorded speech samples are organized and selected in the database. Because the pre-recorded speech samples rely on segmentation techniques to determine the pre-recorded samples, audible artifacts may exist in the synthesized speech.

[0004] Parameter-based techniques do not use pre-recorded speech samples at run time. Instead, output speech may be generated based on an acoustic model that parameterizes human speech. Parameter-based techniques can generate synthetic speech that is easy to understand, but the generated speech may have a less natural sounding quality (e.g., compared to sample-based techniques). Summary of the Invention [Means for solving the problem]

[0005] The present disclosure is directed to a solution for generating natural-sounding synthetic speech of a target speaker based on a relatively short sample (e.g., 10 minutes or less) acquired for the target speaker. The system utilizes a trainable multi-speaker parameterized model (e.g., implemented using a learning machine system such as a neural network-based speech synthesis system). The multi-speaker model is trained using a large corpus of recorded speech from multiple speakers, each of which is associated with an encoded vector (also called an embedding). The encoded vector defines an embedding space that represents at least some voice characteristics of the speaker in the large corpus (e.g., typically characterizing spectral domain characteristics). The target speaker, typically different from any of the speakers used to train the multi-speaker model utilized for TTS, provides a short sample of speech based on which a trained encoder generates an embedding vector that represents the voice characteristics of the target speaker (e.g., as a centroid or average value of a set of speech-level embeddings for the target speaker). The encoded embedding for the target speaker lies within the embedding space defined for the trained encoder, and this embedding can be used to condition (effectively control or adjust, as described in more detail below) the characteristics of the voice produced by the speech synthesis system to synthesize a voice that closely emulates / approximates the characteristics of the target speaker's voice.

[0006] To further enhance the naturalness of the synthesized speech and more closely approximate the speech of the target speaker as it would be spoken by any given utterance, the multi-speaker model implementation also uses information representative of the time-domain speech characteristics of speech uttered by the target speaker (e.g., the pronunciation of the speech by the target speaker, the accent of the target speaker, the speech style of the target speaker, and / or the prosodic characteristics of the target speaker). To achieve this naturalness of the cloned speech to match the additional stylistic characteristics of the speaker, a non-parametric adaptation process (i.e., a process that goes beyond conditioning on the embedding of the target speaker) is applied to the speech synthesis system to adjust the model to represent the stylistic characteristics of the target speaker's speech (e.g., based on short speech samples provided and recorded for the target speaker). For example, the speech synthesis system can be subjected to an optimization tuning process to adjust (e.g., adapt, modify) the trained configuration (e.g., weights of one or more neural network-based implementations of the synthesis system's components / subsystems) based on short speech samples provided to the target speaker and annotation information matched to the speech samples to achieve a more optimal performance for the particular target speaker whose speech is to be cloned.

[0007] In some embodiments, further improvement of the performance of the speech synthesis system to generate a more natural sounding cloned voice of the target speaker is achieved by an adaptation monitoring process that uses the calculated variance metric to determine the stability of the voice cloning process (e.g., by determining the stability of an optimization process used in the course of a non-parametric adaptation process). As described in more detail below, a stability metric (e.g., as an entropy metric) is calculated (e.g., in a non-parametric adaptation process) to evaluate the stability of an emerging optimization solution. In some embodiments, if the stability metric (as indicated by the variance behavior of the calculated entropy metric) is determined to be converging to a non-stable solution, the current adaptation process is stopped and new speech samples are obtained (possibly using different utterances / words prompted by the target speaker) or the adaptation process is restarted.

[0008] The techniques and solutions described herein can be used with a variety of speech synthesis architectures or frameworks. For example, the presently described neural speech synthesis models for multi-speaker training and adaptation techniques for creating individualized voices and synthesizing natural-sounding speech may be applied to the Tacotron model architecture (including the Tacotron 2 model architecture), the Noise Diffusion model class, the DiffWave model class, Generative Adversarial Network (GAN)-based waveform synthesis, and the like. The techniques and solutions can also be used with audio watermarking techniques configured to determine the origin of an audio fragment (e.g., whether the audio portion is authentic or generated by a voice clone). The techniques and solutions described herein perform non-parametric adaptation along with style embedding to jointly model the speaker that is the subject of the voice cloning. The techniques and solutions described herein achieve improved voice cloning that better mimics the prosody of the target speaker.

[0009] The techniques and methods described herein include implementations for finding and refining acquired speech samples for creating individual synthetic speech using unsupervised signal enhancement, annotation, transcription, and / or selective elimination of training processes. The techniques and methods of this disclosure can be performed using speech samples (from target speakers whose voices and speech characteristics are sought to be cloned) that are 10 minutes long, or shorter (e.g., 3 minutes), or longer (e.g., 2 hours).

[0010] Thus, in one variation, a method of generating speech is provided that includes obtaining speech samples of a target speaker, processing the speech samples of the target speaker using a trained encoder to generate a parametric representation of the speech samples of the target speaker, receiving configuration data for a speech synthesis system that accepts the parametric representation as input, and adapting the configuration data of the speech synthesis system with the input including the parametric representation of the target speaker and a time-domain representation of the speech samples of the target speaker to generate adapted configuration data for speech synthesis representing the target speaker. The method also includes providing configuration of the speech synthesis system with the adapted configuration data (which may include the parametric representation of the target speaker), where the speech synthesis system including the adapted configuration data is implemented to generate synthetic speech output data having estimated voice and time-domain voice characteristics that approximate the actual voice and time-domain voice characteristics of the target speaker.

[0011] Embodiments of this method may include at least some of the features described in this disclosure, including one or more of the following features.

[0012] The configuration data may include weights for a neural network-based implementation of the speech synthesis system.

[0013] Adapting the configuration data according to the time domain representation may include matching a speech sample with a corresponding language annotation for the speech sample to generate an annotated speech sample that identifies speech and non-speech portions and their respective temporal information, where the annotated speech sample represents time-domain speech attribute data of a target speaker, and adapting configuration data of a speech synthesis system at least in part according to the annotated speech sample representing the time-domain speech attribute data of the target speaker.

[0014] The time-domain speech attribute data of the target speaker may include, for example, one or more of the following: speech pronunciation by the target speaker, accent of the target speaker, speech style of the target speaker, and / or prosodic features of the target speaker.

[0015] The linguistic annotation may include word and / or sub-word transcriptions, and matching the speech sample to the corresponding linguistic annotation may include aligning word and / or sub-word elements of the transcription to a time-domain representation of the target speech sample of the target speaker.

[0016] The method may further include generating synthetic speech output data, including processing the target language input by applying a speech synthesis system configured with the adapted configuration data to synthesize speech having voice characteristics and time-domain voice characteristics that approximate the actual voice and time-domain voice characteristics of a target speaker speaking the target language input.

[0017] Obtaining a speech sample of a target speaker may include obtaining speech corresponding to a linguistic representation of the verbal content of the speech sample.

[0018] Obtaining a voice sample of a target speaker may include conducting a scripted data collection session with the target speaker, including prompting the target speaker to speak verbal content.

[0019] The method may further include performing an audio validation analysis on the voice sample to determine whether the voice sample satisfies one or more audio quality criteria, and in response to determining that the voice sample does not satisfy the one or more audio quality criteria, obtaining a new voice sample.

[0020] The method may further comprise applying filtering and speech enhancement operations to the audio samples to improve the quality of the audio samples.

[0021] The received configuration data for the speech synthesis system may be derived from training speech samples from a number of training speakers distinct from the target speaker.

[0022] Adapting the configuration data may include calculating an adaptation stability index representative of an adaptation performance for adapting the configuration data, and in response to the calculated adaptation stability index being determined to indicate unstable adaptation of the configuration data, discontinuing the adaptation of the configuration data.

[0023] The method may further include restarting the adaptation of the configuration data using speech samples of the target speaker.

[0024] The method may further include, following the stopping, obtaining a new speech sample of the target speaker and performing adaptation of the configuration data using the new speech sample.

[0025] Computing an adaptation stability metric may include computing attention data for a portion of the speech samples. Ceasing adaptation of the learning machine based synthesizer may include ceasing adaptation of the learning machine based synthesizer in response to determining that an attention variance level derived from the attention data indicates a non-converged adaptation solution of the speech synthesis system.

[0026] Processing the target speaker's voice samples using the trained encoder to generate the parametric representation may include converting the target speaker's voice samples into a vector representation in the spectral domain.

[0027] Converting the audio samples into a vector representation in the spectral domain may include converting the audio samples into a plurality of mel spectrogram frames and mapping the plurality of mel spectrogram frames into a fixed dimensional vector.

[0028] The method may further include generating a parametric style representation of a prosodic style associated with the speech sample using a variational autoencoder. Adapting the configuration data may include adapting configuration data of the speech synthesis system further based on the parametric style representation.

[0029] Adapting configuration data of a speech synthesis system with a parametric representation of a target speaker and a time-domain representation of the speech samples may include adapting the configuration data using a non-parametric adaptation procedure to minimize an error between predicted spectral representation data generated by the speech synthesis system in response to the parametric representation and text data matching the speech samples of the target speaker and actual spectral data derived directly from the speech samples.

[0030] In one variation, a speech generation system is provided that includes a speech acquisition unit for acquiring speech samples of a target speaker, an encoder for applying to the speech samples of the target speaker to generate a parametric representation of the speech samples of the target speaker, and a speech synthesis and cloning system. The speech synthesis and cloning system includes a receiver for receiving configuration data of the speech synthesis system, the receiver being configured such that the speech synthesis system accepts the parametric representation as input, and an adaptation module for adapting the configuration data of the speech synthesis system with inputs including the parametric representation of the target speaker and the time-domain representation of the speech samples of the target speaker to generate adapted configuration data of the speech synthesis system representing the target speaker. The adaptation module provides for configuration of the speech synthesis system with the adapted configuration data, and the speech synthesis system including the adapted configuration data is implemented to generate synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker.

[0031] In one variation, a non-transitory computer-readable medium is provided that stores a set of instructions executable on at least one programmable device for obtaining voice samples of a target speaker, processing the voice samples of the target speaker using a trained encoder to generate a parametric representation of the voice samples of the target speaker, receiving configuration data of a speech synthesis and cloning system that accepts the parametric representation as input, and adapting the configuration data of the speech synthesis and cloning system with inputs including the parametric representation of the target speaker and the time-domain representation of the voice samples of the target speaker to generate adapted configuration data of the speech synthesis and cloning system representing the target speaker, which set of instructions, when executed on the at least one programmable device, results in configuration of the speech synthesis and cloning system with the adapted configuration data, and the speech synthesis and cloning system including the adapted configuration data is implemented to generate synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker.

[0032] In a variation, a computing device is provided that includes a speech capture unit for obtaining speech samples of a target speaker, and one or more programmable processor-based devices for generating synthetic speech according to any of the aforementioned method steps.

[0033] In some variations, a programmed non-transitory computer readable medium is provided having a set of computer instructions executable on a processor that, when executed, causes operations including any of the various method steps described above.

[0034] The aforementioned system, apparatus, and / or computer-readable medium embodiments may include at least some of the features described in this disclosure and may be combined with any other embodiment, variation, or feature of the methods described herein.

[0035] Some embodiments may include cross-language training to generate speech synthesis in a language different from the language the speech sample was collected in. Some embodiments may use audio samples obtained from a data repository on an accessible network (such as the Internet) or other source, which are then annotated.

[0036] Other features and advantages of the invention will be apparent from the following description and from the claims.

[0037] These and other aspects will now be described in detail with reference to the following drawings. [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 is a schematic diagram of an exemplary voice cloning system. [Diagram 2] 1 is a schematic diagram of an exemplary voice synthesis unit of a voice cloning system. [Diagram 3] 1 is a flow diagram of an exemplary procedure for generating audio.

[0039]

[0040] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Like reference symbols in the various drawings indicate like elements.

[0042] FIG. 1 is a diagram of a voice cloning system 100. The system 100 is shown with two separate main parts, namely a voice acquisition part 110 and a voice synthesis cloning part 120. In general, the voice synthesis cloning part 120 is located in a central location (and may be implemented as a single or several distributed computing servers or devices), and the voice acquisition part 110 (or at least a part thereof) is located remotely from the voice synthesis cloning part 120, both processing parts communicating via wired and / or wireless channels. Alternatively, the two processing parts may be present in the same device. Furthermore, while only one voice acquisition part 110 is shown, the voice synthesis part 120 may communicate with and perform voice synthesis operations based on data provided by multiple independent devices. It should be noted that the speech synthesis cloning unit 120 is configured to perform a "cloning-time" (typically offline) cloning operation (e.g., to configure and adapt a previously constructed synthesis model, previously constructed at "multiple speaker training time," to form a cloned model for a particular target speaker), and the resulting cloned model is then transferred (not shown in FIG. 1 ) to a downstream speech synthesis system (e.g., on a personal device, in a car, in a cloud server, etc.) for later "run-time" speech synthesis in the target speaker's voice. However, in some embodiments, as described in more detail below, this processing unit 120 may also be configured to perform run-time speech synthesis as well.

[0043] An example of a device in which the voice capture unit 110 may be implemented may be a wireless device, such as a smartphone, or a personal computing device (such as a laptop computing device) that can establish a communication link with a platform implementing voice synthesis by a number of different communication technologies or protocols (e.g., WLAN-based communication technologies such as WiFi protocols, WWAN technologies such as LTE-based communication protocols, 5G protocols, etc.). As shown in FIG. 1, the voice capture unit 110 generally includes a voice collection unit 112 including one or more voice transducers (microphones) and a recording medium (typically a memory storage device for storing digitized samples of recorded audio samples). The voice collection unit 112 may also include a user input / output interface, via which a user may be prompted to utter a given voice sample. For example, in embodiments in which the audio collection unit 112 is implemented using a device that includes a display device, a predetermined passage of text may be presented to a user via the display device, and then the target speaker may repeat the passage aloud, causing the smartphone or computing device to convert the audio sample into digital form (using an analog-to-digital converter and any other filtering required) and store it on the storage medium of the recording device.

[0044] In some embodiments, the recorded voice samples may be analyzed by an audio validation unit 114 (shown in FIG. 1 ), which may be configured to process the recorded voice to perform filtering operations to improve the quality of the recorded voice samples from the target user and / or to determine whether the quality of the recorded signal meets a quality criterion. For example, the audio validation unit 114 may calculate a signal-to-noise ratio (SNR) or some other audio quality metric for the recorded voice samples. The audio validation unit 114 may then determine whether the calculated audio quality metric meets (e.g., exceeds) a criterion value (threshold) used to evaluate the audio quality criterion. If, as a result of the analysis / evaluation performed by the audio validation unit 114, it is determined that the obtained voice sample does not satisfy one or more audio quality criteria (e.g., the SNR calculated for the voice sample is below some predefined reference SNR value), the audio validation unit 114 may cause the audio collection unit 112 (e.g., by sending a request message via the link 115) to request the target speaker (via a user interface of the audio collection unit 112) to record a new voice sample (based on the same language content used to record the current voice sample or based on new content). In some embodiments, the audio validation unit 114 may also be configured to perform various filtering processes / operations to improve the audio quality of the current acquired voice sample. For example, the audio validation unit may include an equalizer realized using a fixed or adaptive filtering implementation (e.g., based on a finite impulse response (FIR) or infinite filter response (IIR) filter, or according to other filtering expressions). Such an equalizer may be used to remove noise or otherwise condition in any desired manner the input signal provided to it.Other types of enhancement / filtering that may be performed on the speech samples to improve the quality of the collected samples (make them more suitable for training) include various types of de-noising techniques, various types of dereverberation, etc.

[0045] Once a speech sample that satisfies one or more of the audio quality criteria has been acquired, the audio validation unit 114 forwards the resulting speech sample to an automatic transcription unit 116 configured to generate a time-domain representation including matching of the speech sample with transcription units (e.g., word or sub-word elements, phonemes, etc.) that is used to adapt a speech synthesis system (such as the speech synthesis system 130 shown in FIG. 1). It should be noted that in some examples, only a portion of the circuitry of the speech acquisition unit 112, and possibly the audio validation unit, will reside on the device (of the personal computer with which the target speaker interacts) or browser, and the automatic transcription unit 116 runs on a central server (e.g., a cloud-based server). Adapting a speech synthesis system means that the synthesized speech will have speech characteristics (e.g., spectral characteristics such as timbre) and time-domain speech characteristics (speech style, accent, prosody, etc.) that approximate the actual time-domain speech characteristics and actual voice characteristics of the target speaker providing the input speech sample via the audio collection unit 112.

[0046] In some implementations, the time domain representation may include linguistic annotations generated for the speech samples (processed by the audio validation unit 114) that match the speech sample signal with the textual content that the target speaker was prompted to speak and record. Such matching of the time domain audio content of the speech samples with the textual content aligns the words and / or subword elements of the transcription with the target speech sample. The alignment of the words and subwords provided in the textual content with the actual audio content spoken by the speaker may be performed through a machine learning process, where a learning machine (implemented for the automatic transcription unit 116) is trained to associate or match written representations such as words, subwords, phonemes, silences, etc. with the corresponding audio content. The transcription process may be performed through speech recognition and / or alignment to known transcriptions processes. Alternatively, or in addition, the automatic transcription unit 116 may perform natural language processing operations on the audio speech sample to independently recognize the linguistic content of the audio speech sample, for example, when the input does not follow a known script for which any required natural language-based annotations are already available. For example, available target speech samples may be retrieved via a shared audio repository (e.g., on YouTube) and used for the cloning-based speech synthesis operation. In such a situation, NLP operations may be applied to the retrieved speech sample (for which corresponding annotated text-based linguistic content may not be available) to generate annotations that match the audio content. Natural language processing is applied to the data source (in this case, the speech sample output by the speech validation unit 114) to process human language for meaning (semantics) and structure (syntax). NLP can distinguish the meaning of words / phrases and larger text units based on the surrounding semantic context. In some embodiments, a syntax processor assigns or "parses" units of text into grammatical categories or "parts of speech" (nouns, verbs, prepositions, etc.).The semantic processor assigns units of text to lexical classes to standardize the representation of meaning. Thus, the automatic transcription unit 116 may utilize an NLP engine that recognizes words uttered by a speaker and determines the phonological order of those utterances. Such an NLP engine then enables the automatic transcription unit 116 to identify boundaries (start and end points) of word and subword parts or phonemes in the audio speech sample and match them to the textual content (i.e., linguistic content) of the passage spoken by the target speaker when providing the speech sample. When a mismatch occurs between the audio content and the textual content, for example when the confidence level associated between the matched parts falls below some threshold level or when a certain percentage of independently recognized words and subwords or phonemes do not match the words, subwords, and / or phonemes in the textual content (which may occur as a result of the speaker providing a speech sample that is inconsistent with what the audio collection unit prompted the speaker to provide), a decision may be made to discard the speech sample and acquire a new speech sample. In such a situation, the automatic transcription unit 116 may request the audio collection unit (using a message request sent via link 117) to reacquire a new source sample from the target speaker. Note that in some embodiments, the actual speech sample may not match the expected speech transcription due to the speaker pronouncing one of the expected words in an unclear or unexpected way. In such a situation, a process of adapting the speech transcription to the audio may need to be performed, or alternatively the audio speech sample may need to be discarded and / or the speaker may need to be asked to provide a new sample.

[0047] In situations where the speech sample can be substantially matched to the linguistic content used to record the raw speech sample provided by the speaker, the automatic transcription unit 116 generates an annotated speech sample that includes the audio content of the speech sample and information representative of a phonetic transcription of the speech sample. In some embodiments, the automatic transcription unit may alternatively generate an annotated speech sample that includes the audio content and the linguistic (semantic and syntactic) content of the speech sample. As described in more detail below, the annotated speech sample is used to perform two tasks. The first task is to generate parametric inputs (e.g., in the form of fixed dimensional vectors) that are provided to a trained speech synthesis system by a separate encoder (e.g., a trained learning machine-based encoder) to cause the speech synthesis system to generate outputs from which synthetic speech having vocal characteristics that approximate those of the target speaker is generated. The second task of using the annotated speech samples is to adapt the speech synthesis system to the speech characteristics of the speaker represented by the annotated speech samples (e.g., via adjusting configuration data such as weights in neural network-based systems that define the implementation and functionality of the speech synthesis system).

[0048] Thus, with continued reference to FIG. 1, the speech synthesis unit 120 may include an encoder 126 coupled to a speech synthesizer 130, which may be implemented as a learning machine, and configured to generate an output that can be converted into a synthetic speech having vocal and phonetic characteristics determined by adapted configuration data of the speech synthesis system 130. In some embodiments, the encoder 126 (which may be realized as a learning machine implementation) may be configured to calculate a fixed-dimensional embedding vector from the reference speech of the target speaker (i.e., the speech samples 102). In such an embodiment, the speech synthesizer system 130 may be configured to generate a synthetic speech based on the embedding vector provided by the encoder 126 and the input text. The speech synthesis / cloning system 130 may optionally include a speech encoder (vocoder), which may also be implemented as a learning machine system, configured to estimate / generate a time-domain waveform from a mel-spectrogram generated by a token-to-spectrogram converter (also referred to as a "converter" or "encoder module"). In some embodiments, the converter may convert the input tokens into other types of output representations to be processed by the vocoder. The speech encoder may be implemented in the same or different device as other components of the speech synthesis system 130. As shown in FIG. 1, in some examples, the speech synthesis system (or at least a portion thereof) may be deployed (e.g., following individual adaptation of multi-speaker models for specific target speakers) in individual devices (e.g., speech interfaces included in various objects such as smartphones, laptops, automobiles, etc.) in an embedded deployment configuration (shown diagrammatically as implementation 150), or in a central location accessible by multiple users in a cloud deployment configuration (shown diagrammatically as implementation 152). In some embodiments, the deployment of the speech synthesis system may follow adaptation of multi-speaker models to specific target speakers (adaptation of multi-speaker models is also referred to as adaptation during cloning).

[0049] More specifically, at clone time (i.e., when the speech synthesis system is configured for a particular target speaker), the encoder 126 receives as input a short reference utterance (e.g., speech sample 102) and generates a parametric representation (e.g., embedding vector) of the reference utterance that represents the vocal characteristics of the target speaker through its learned speaker feature space (embedding vectors are typically generated for each input utterance, or alternatively, based on multiple or all of the speaker's utterances). The reference utterance is also provided to a non-parametric adaptation module 132 of the speech synthesis system 130, which is configured to adapt the configuration data generated for the multi-speaker synthesis model based on the time-domain phonetic characteristics (e.g., style, accent, prosody, etc.) represented by the annotated speech sample 102. As mentioned above, the annotated speech sample 102 includes timing information related to the manner in which words and subwords (or phonemes) are spoken, thus providing phonetic characteristic information about the manner in which the target speaker speaks or enunciates linguistic content. The non-parametric adaptation module can then use this speech characteristic information to adjust the processing behavior defined by the configuration data provided to the speech synthesis system 130. Once the configuration data of the speech synthesis system 130 has been non-parametrically adapted, the speech synthesis system 130 processes the input sequence to generate intermediate outputs, e.g., mel-spectrogram outputs, that are conditioned by the speaker encoder embedding vectors. In some examples, the output cloned speech synthesis model is provided and / or the embedding vectors used during the adaptation of the configuration data may be calculated as centroid vectors of embedding vectors generated from the speech sample utterances. A vocoder of the speech synthesis system 130 generates speech waveforms from the intermediate outputs (i.e., in situations where a vocoder is included, such that, for example, vocoder adaptation can achieve improved performance of the downstream speech synthesis system).

[0050] The configuration data adapted based on the annotated voice samples 102 was previously generated by an optimization process implemented by a learning machine training controller for the multi-speaker model (generally identified as the multi-speaker model 124) using training audio data from multiple speakers (accessed from the data repository 122). For example, the system 100 may no longer have access to the original training audio, but may have the configuration data of the multi-speaker model, including the encoders used to form the embedding vectors of the speakers of the original training audio. The multi-speaker model may be implemented as a separate and remote system from the speech synthesis system 130. The controller of the multi-speaker model 124 (which previously determined the configuration data based on a corpus of training data of the multiple speakers) is configured to determine and / or adapt parameters (e.g., neural network weights) of the learning engine-based speech data synthesizer that will generate, for a particular input text content, an output representing voice characteristics and speech characteristics that match or approximate the actual voice characteristics and speech characteristics of the input data derived from the target speaker's voice samples (which in the embodiment of FIG. 1 are parametric representations). It should be noted that during training, the speech synthesis system used a training data set that may include speech samples (each associated with its own vocal characteristics and phonetic properties) from multiple speakers (tens, hundreds, or thousands of speakers, with each sample containing minutes to hundreds of hours of speech content). Such speech (audio) data may be associated with male and female speakers of various ages and may include samples with various voice attributes (style, prosody, etc.) and phonetic properties. In some embodiments, male and female speakers may be used to form different separate models (separate models may be formed for different age groups for each gender-based grouping). In general, the more training data there is, the higher the accuracy and confidence level associated with the output the speech synthesis system produces.

[0051] After the learning machine-based implementation of the speech synthesis system is operational and capable of processing actual run-time data (following the training stage and the transfer of configuration data from the controller of the multi-speaker model 124 to the system 130), subsequent training may be performed intermittently (regularly or irregularly) to dynamically adapt the speech synthesis system to new, more recent training data samples to maintain or even improve the performance of the speech synthesis system. Such intermittent training would complement the non-parametric adaptation typically performed by the speech synthesis system. Thus, the intermittent training may be parametric adaptation performed using the controller of the multi-speaker model 124 (as opposed to non-parametric adaptation, which is performed on the configuration data of the speech synthesis system 130 to adapt the system to the specific inherent speech characteristics of the target speaker).

[0052] When the speech synthesis system 130 is implemented as a neural network, training (performed by a controller of the multi-speaker model 124) is used to define parameter values ​​(weights), represented as a vector θ, that are assigned to the links of the neural network, for example, based on a procedure that minimizes a loss metric between the predictions made by the neural network and the labeled instances of data. One example of an optimization procedure for computing weights for a neural network-based learning machine is a stochastic gradient descent procedure, which may be used to minimize a loss metric. The calculated parameter values ​​may be stored in a memory storage device (not shown) and transferred intermittently to the speech synthesis system 130 (e.g., upon receiving input for a new target speaker).

[0053] As described above, the speaker encoder 126 generates a parametric representation (e.g., an embedding vector) to represent the characteristics of the target speaker's voice. In some implementations, the encoder 126 is trained to identify the characteristics of the target speaker's voice regardless of linguistics (e.g., phonetic or semantic content or language) and / or background noise. The encoder 126 may be implemented, for example, using a neural network model trained on a text-independent speaker verification task that seeks to optimize a generalized end-to-end (GE2E) loss such that embeddings of utterances from the same speaker have high cosine similarity and embeddings of utterances from different speakers are spaced apart in the embedding space. To generate the resulting vector from a time-domain audio sample (e.g., sample 102), the encoder may be configured to convert the audio sample into multiple mel spectrogram frames and map the multiple mel spectrogram frames to a fixed-dimensional vector. Various implementations and configurations may be used to perform the mapping from the mel spectrogram to the embedding vector. For example, in some embodiments, the resulting mel-spectrogram is passed to a network including a stack of multiple long short-term memory (LSTM) layers, each followed by a projection into multiple dimensions. The resulting embedding vector may be generated by L2-normalizing the output of the top layer in the final frame. At runtime, the encoder may be fed portions of the speech sample divided into manageable windows (with or without overlap) that are processed independently by the encoder, and the output for each portion of the speech sample is averaged and normalized to generate the completed embedding vector. In other exemplary embodiments, the mapping of the mel-spectrogram to a parametric representation may be implemented using a convolutional neural network (CNN) followed by a stack of multiple (e.g., three) gated recurrent units (GRUs). Additional layers (such as a projection layer) may be included.

[0054] As further shown in FIG. 1, the speech synthesis and cloning system 130 includes a multi-speaker synthesizer 134, which typically includes a circuit (neural learning engine circuit) that performs processing on the parametric representation generated by the encoder 126 and the linguistic content input (to be converted into cloned / synthesized speech), and a non-parametric adaptation unit 132 that adjusts at least some of the adaptable parts of the multi-speaker synthesizer 134 (e.g., neural network weights) with the annotated speech samples 102. The non-parametric adaptation unit 132 adjusts the multi-speaker model to a speaker-dependent model of the target speaker, which takes into account the time-domain speech characteristics of the target speaker. The non-parametric adjustment is typically performed during cloning time (i.e., prior to run-time), during which a clone model is generated (by the adaptation process described herein) and loaded into a downstream synthesis device (not shown) that generates audio from the text content with the adapted clone model. The multi-speaker synthesis device 134 implements a framework (which may be similar to the Tacotron™ architecture in some embodiments) that includes a token-to-spectrogram converter 140 for converting text-based linguistic content into mel-spectrogram frames using a recursive sequence-to-sequence feature prediction model. The token-to-spectrogram converter 140 may be trained on pairs of text-derived token sequences and audio-derived mel-spectrogram sequences. In some examples, the token-to-spectrogram converter may first map the input text-based linguistic content to a set of phonemes (using a learning machine stage in the token-to-spectrogram converter 140), which may further speed up the convergence of the synthetic speech and improve its pronunciation. As previously mentioned, the speaker encoder 126 generates embedding vectors that condition the performance of the multi-speaker synthesis system 134, thus enabling the speech synthesis system to implement transfer learning functionality.For example, the parametric representations (embedding vectors) generated by encoder 126 can be combined (concatenated) with internal representations of intermediate values ​​generated by circuitry in token-to-spectrogram converter 140 (e.g., using an attention layer that receives the embedding vectors as input via a concatenation module that applies the embedding vectors to the output of an intermediate stage in token-to-spectrogram converter 140). The output generated by token-to-spectrogram converter 140 (e.g., mel spectrogram frames conditioned by the embedding vectors) is provided to speech encoder (vocoder) 142, which generates time-domain waveform samples from the mel spectrogram frames generated by token-to-spectrogram converter 140 (one example of a vocoder that may be used with system 100 of FIG. 1 is WaveNet™). It should be noted that in embodiments where the multi-speaker synthesizer is configured to perform further adaptations to the vocoder individually, or jointly for a combined implementation including a converter and a vocoder, a vocoder 142 is optionally included in the multi-speaker synthesizer 134. In some embodiments, the vocoder may be provided only in the deployed runtime system (e.g., in a remote device or central server that performs synthesis of speech based on runtime language content).

[0055] The non-parametric adaptation unit 132 adapts the configuration data of the speech synthesis system 130 by optimizing parameters of a learning machine implementation of the multi-speaker synthesizer 134 (specifically, the token-to-spectrogram converter 140) with the annotated speech samples 102. Specifically, the non-parametric adaptation unit 132 uses the annotated speech samples 102 as a compact training set that includes the actual target speech content (i.e., the audio content of the speech samples 102) and corresponding text for that speech content that is aligned with the audio content. Thus, the audio content of the speech samples and the text-based content together define an aligned ground truth that can be used to further adjust the configuration data to minimize the error between a representation of the actual speech sample (e.g., a mel spectrogram representation derived from the speech sample) and a predicted mel spectrogram frame generated by the token-to-spectrogram converter 140 conditioned by the parametric representation generated from the speech sample. The adaptation of the configuration data may be performed directly on the configuration data provided to the speech synthesizer 130, or may be performed in a separate learning machine training controller (similar to the training controller engine used to compute the base configuration data from the multi-speaker training set 122), and the non-parametrically adapted configuration data is then forwarded to the multi-speaker synthesiser 134.Non-parametric adaptation, achieved by optimization using actual speech samples of the target users, can thus fine-tune the behavior of the multi-speaker synthesizer 134 to better match the speech attributes (speech style, accent, prosody, etc.) of the target users, so that when any text-based content is provided to the speech synthesis system at run-time, the predicted output data representation (e.g., mel spectrogram, or time domain waveform) of the speech synthesis system 130 will more closely match the actual data representation that would be generated if the target speaker were to speak the text-based content. In some embodiments, the optimization procedure that adjusts the weights of the speech synthesis system based on the ground truth defined by the annotated speech samples (when the speech synthesis system is conditioned by embedding vectors of the speech samples) may be performed using a stochastic gradient descent procedure that minimizes a predefined loss metric. Other optimization procedures may be used to implement non-parametric adaptation.

[0056] Continuing with reference to FIG. 1, as shown, speech synthesis system 130 may optionally include a stability detector 136 in communication with non-parametric adaptation unit 132 (or alternatively in communication with multi-speaker synthesizer 134 in embodiments where further training and adaptation is performed directly on the configuration data maintained by multi-speaker synthesizer 134). As described in more detail below (with reference to FIG. 2), this stability detector is configured to determine whether non-parametric adaptation performed using short (e.g., 2-3 minutes, 10 minutes) annotated speech samples of a target speaker (e.g., during cloning operations) has converged to a stable solution. To that end, the stability detector calculates an adaptation stability metric representative of the adaptation performance for adapting the configuration data, and in response to determining that the calculated adaptation stability metric indicates an unstable state in the adaptation of the configuration data, ceases adaptation of the configuration data. In some examples, when instability is detected and the nonparametric adaptation is aborted, the nonparametric adaptation unit 132 is configured to cause a new speech sample of the target speaker to be acquired (e.g., by the audio collection unit 112) and subsequently perform the adaptation of the configuration data using this new speech sample (this acquired new speech sample would generally also need to be processed by the audio validation unit 114 and the automatic transcription unit 116). Alternatively, instead of acquiring a new speech sample (which may not be feasible if the target speaker is not available to generate another sample), the nonparametric adaptation unit 132 may simply restart the adaptation process by reinitializing the configuration data (i.e., discarding the already partially modified configuration data) and reacquiring the base configuration data (from the multi-speaker synthesizer 134 or the multi-speaker model 124).Unstable adaptation is often caused by a bad initialization of the adaptation process, and therefore under such circumstances, this unstable condition can be improved by restarting the adaptation process with the same annotated speech samples from the target speaker. The initialization generally relies on a random number seed. Restarting will change this seed with the goal of avoiding instabilities that may occur with other random number seeds, and may result in different results.

[0057] In some embodiments, the calculated adaptation stability metric may be attention data for each portion of the speech sample (which may represent entropy), and discontinuing adaptation of the learning machine based synthesizer may include discontinuing adaptation of the learning machine based synthesizer in response to determining that an attention variance level derived from the attention data is indicative of a non-converged adaptation solution for the speech synthesis system.

[0058] 2 is a schematic diagram of an exemplary embodiment of a speech synthesizer 200, which may be configured and implemented at least in part similarly to speech synthesizer 120. The schematic diagram of FIG. 2 provides some further details to what has been shown and discussed in connection with FIG.

[0059] The speech synthesis unit 200 includes a document-to-token stage 210 for processing and adapting input language content for compatibility with a token-to-spectrogram (TC) stage 220 (part of the functionality and / or implementation of the token-to-spectrogram stage 220 may be similar to the functionality and / or implementation of the token-to-spectrogram converter 140 of the multi-speaker synthesis unit 134 of FIG. 1). The document-to-token stage 210, which may be implemented as a learning machine having an encoder / attention / decoder configuration (identified as units 212, 214, and 216, respectively), is trained to perform natural language processing on text-based language content, such as content provided in a document 202, and generate an output (usually segmented into multiple parts that are gradually provided as input to the token-to-spectrogram unit 220). It should be noted that other configurations of the document-to-token stage 210 (or any other stage) as well as the encoder / attention / decoder configuration may alternatively be used. The NLP process divides a document into linguistically meaningful, processable parts (e.g., into syllables or into groupings of one or more phonemes) according to the semantic and syntactic content recognized by the machine learning implementation of the document-to-token stage 210. The actual output provided to the token-to-spectrogram stage 220 may depend on the implementation of the token-to-spectrogram stage 220, and in particular the type of input that the stage 220 accepts. The tokens may be actual sequences of text-based characters (e.g., corresponding to the syllables and phonemes detected and analyzed by the document-to-token stage 210), or may be some other type of encoded output data, such as, for example, encoded parametric data representing the linguistic content analyzed by the document-to-token stage 210.

[0060] The token-to-spectrogram stage 220 is configured to generate a linguistic representation of the tokenized text-based content (e.g., in the form of an output spectrogram or mel-spectrogram) that represents speech characteristics that approximate the speech characteristics (e.g., timbre) of the target speaker. The token-to-spectrogram stage 220 includes a speech synthesizer 230 that may be similar in functionality and implementation to the converter 140 (of the multi-speaker synthesizer 134) of FIG. 1 and may be implemented as a learning machine having an encoder / attention / decoder configuration (identified as units 232, 234, and 236, respectively). As described in connection with FIG. 1, a speaker encoder 206 (similar to the encoder 126 of FIG. 1) receives speech samples 204 (which may be annotated) of the target speaker and converts the speech samples into parametric representations (e.g., embedding vectors) that are provided to the token-to-spectrogram stage 220 to condition the output generated by the speech synthesizer 230. In some embodiments, the computation of the embedding vector may be performed at speech clone training time. At inference time, the pre-computed speaker embedding vector is used for that particular speaker.

[0061] The parametric representation represents the characteristics of the speaker's voice (so different voice characteristics will result in different parametric representations). In the embodiment of Figure 2, the parametric representation is combined with the data processed by the speech synthesizer 230 via an attention unit 234 (which is configured to identify and track (and amplify) important parts of the output of the encoder unit 232).

[0062] To more closely approximate the predicted spectrogram generated by the token-to-spectrogram stage 220 to a spectrogram that would be generated directly from the target speaker's audio samples, a non-parametric adaptor 224 (which may be similar in function and implementation to the non-parametric adaptation unit 132 of FIG. 1 ) receives the speech sample 204 and corresponding text-based content that is consistent with the audio content of the speech sample (this adaptation is typically performed during cloning adaptation prior to runtime deployment). The non-parametric adaptor 224 uses the speech sample and accompanying text-based annotations to further train the speech synthesizer 230 for the particular target speaker (thus adapting the base configuration data of the speech synthesizer 230). When adapting, utterance-level speaker embeddings are also used as input. As mentioned above, this process typically occurs during cloning adaptation prior to runtime deployment. As also mentioned above, adaptation may be performed by using the text-based annotations to predict the resulting spectrogram (using base configuration data conditioned on the parametric representation) and adapting / adjusting weights (or other parameters) of a learning machine implementation of the speech synthesizer 230 to reduce the error (using some pre-specified loss or error function) between the predicted spectrogram and a spectrogram generated directly from the target speech sample 204.

[0063] In some embodiments, speech characteristics such as prosodic style may be approximated by the output of the system 200 by modeling the speech characteristics with a variational autoencoder. In such an embodiment, during the training stage, style embeddings are computed (at the utterance level) from the input mel spectrograms of the training speakers. During inference (runtime), a fuzzy matching scheme may be used to find the most relevant style embeddings (for the sequences to be generated) from the data available in the target speaker data. Thus, as shown in FIG. 2, the token-to-spectrogram stage 220 optionally includes a variational autoencoder (VAE) style encoder 226 configured to encode the reference audio (e.g., the speaker's mel spectrogram frames) into a parametric style representation (e.g., a fixed-dimension vector) of the speaker's prosodic style. The resulting parametric stylistic representation, in the example of Fig. 2, is provided to the encoder unit 232 of the token-to-spectrogram stage 220, which combines the parametric stylistic representation with the encoder states (either as input to the encoder or as additional data added to the intermediate processed encoder data). The parametric stylistic representation may be added to the output of the encoder 232. In some embodiments, the parametric stylistic representation may be combined with other modules of the token-to-spectrogram stage 220, or may be combined earlier or later than the processing performed by the speech synthesis unit 200. The parametric stylistic representation (which is part of the multi-speaker model) may be included during non-parametric adaptation (performed by the adapter 224) or at other times.

[0064] The use of the attention unit 234 in an exemplary configuration of the speech synthesizer 230 implementation also presents a convenient way to track the performance of non-parametric adaptation training and determine the stability of the optimized solution, for example during non-parametric adaptation using the speech samples 204. In general, for a converged optimized solution in a sequence-to-sequence model, the attention matrix (in the attention unit 234) should exhibit monotonic behavior with few or no skips (which may result from inaccurate phonetic transcriptions and / or the presence of empty syntactic or boundary symbols). Typically, in the initial training stage, the attention matrix has a wide range of values, and in the further decoding steps, it gradually moves towards a unimodal probability distribution over the attention weights. In such a system, the attention weight entropy averaged over the validation set utterances may indicate attention convergence when the system is evaluated in inference mode. Therefore, when the optimization solution is converging, the time-dependent (or step-dependent) behavior of various stability indices (eg, entropy, robustness, etc.) should have predictable properties.

[0065] In non-parametric adaptation, one example of a stability metric that may be calculated (e.g., by stability detector 222) is the stability metric of attention weight entropy. Entropy is used to represent the variance of attention. When attention is relatively scattered, entropy increases. One way to calculate an entropy metric of the attention behavior of a system is as follows: Given a training set with K-point input speech sequences and respective N-point output acoustic sequences, the average speech entropy can be calculated as follows:

number

[0066] In general, since attention behavior depends on initial conditions, an unstable solution can be improved by restarting the non-parametric adaptation. Alternatively, the stability detector may be configured such that a speech capture system (such as the audio collection unit 112 of the speech capture section 110 in FIG. 1) can prompt the target speaker to record a new speech sample, which is then processed (validated and transcribed) in the same manner as discussed in connection with FIG.

[0067] Continuing with reference to 2, system 200 also includes a spectrogram-to-waveform stage 240 configured to convert the resulting output of speech synthesizer 230 into an audio waveform. The spectrogram-to-waveform stage 240 may be implemented similarly to the speech encoder (vocoder) 142 of FIG. 1. For example, the spectrogram-to-waveform stage may be implemented using a WaveNet™ vocoder. The spectrogram-to-waveform stage 240 may be implemented on the same device as the rest of system 200, or may be implemented on a separate, remote device from the devices on which the other stages reside.

[0068] Referring now to FIG. 3, a flow diagram of an exemplary procedure 300 for speech generation is shown. The procedure 300 includes acquiring (310) a speech sample of a target speaker. In some examples, acquiring the speech sample of the target speaker may include acquiring speech corresponding to a linguistic representation of the verbal content of the speech sample. In such examples, acquiring the speech sample of the target speaker may include conducting a scripted data collection session with the target speaker, including prompting the target speaker to speak the verbal content. The procedure may further include performing an audio validation check of the speech sample to determine whether the speech sample meets one or more audio quality criteria, and acquiring a new speech sample in response to determining that the speech sample does not meet one or more audio quality criteria. In some circumstances, the procedure may further include applying filtering and speech enhancement operations (e.g., equalization, noise reduction, dereverberation, etc.) to the speech sample to improve the quality of the speech sample.

[0069] The procedure 300 further includes processing (320) the target speaker's voice samples using a trained encoder (such as encoders 126 and 206 of FIGS. 1 and 2, respectively) to generate a parametric representation (e.g., an embedding vector) of the target speaker's voice samples. For example, processing the target speaker's voice samples to generate a parametric representation may include converting the target speaker's voice samples into a spectral domain vector representation (e.g., representative of the target speaker's voice characteristics). In some embodiments, converting the voice samples into a spectral domain vector representation may include converting the voice samples into a plurality of mel spectrogram frames and mapping the plurality of mel spectrogram frames into a fixed dimensional vector.

[0070] Continuing to refer to FIG. 3, the procedure 300 also includes receiving (330) configuration data for a speech synthesis system that accepts the parametric representation as an input. The configuration data may include weights for a neural network-based implementation of the speech synthesis system. In some examples, the received configuration data for the speech synthesis system is derived from training speech samples from multiple training speakers that are different from the target speaker.

[0071] The procedure 300 further includes adapting (340) configuration data of the speech synthesis system with inputs including a parametric representation of the target speaker and a time-domain representation of speech samples of the target speaker to generate adapted configuration data of the speech synthesis system representing the target speaker. As mentioned above, in some examples, during adaptation, the parametric representation of the speaker may not be a single vector, but rather may be provided at an utterance level. In some embodiments, adapting the configuration data with the time-domain representation may include matching the speech samples with corresponding language annotations of the speech samples to generate annotated speech samples that identify speech and silence portions and respective time information, using the annotated speech samples to represent time-domain speech attribute data of the target speaker, and adapting the configuration data of the speech synthesis system at least in part according to the annotated speech samples representing the time-domain speech attribute data of the target speaker. The time-domain speech attribute data of the target speaker may include, for example, one or more of a speech pronunciation by the target speaker, an accent of the target speaker, a speech style of the target speaker, and / or a prosodic characteristic of the target speaker. Note that when adapting configuration data, various speech attributes may not be explicitly represented in the configuration data (e.g., a single parameter representing a particular accent) but will typically be implicitly represented (i.e., these speech attributes will be encoded in the configuration data as a result of the adaptation process and may not be able to be released). The linguistic annotation may include word and / or sub-word transcriptions, and matching a speech sample with the corresponding linguistic annotation may include aligning word and / or sub-word elements of the transcription with a time-domain representation of the target speech sample for the target speaker.

[0072] An example of adapting configuration data of a speech synthesis system with a parametric representation of a target speaker and a time-domain representation of a speech sample may include adapting the configuration data using a non-parametric adaptation procedure to minimize an error between predicted spectral representation data generated by the speech synthesis system in response to matching the parametric representation and text data to the target speaker's speech sample and actual spectral data derived directly from the speech sample.

[0073] In some embodiments, adapting the configuration data may include calculating an adaptation stability metric representative of an adaptation performance for adapting the configuration data, and halting the adaptation of the configuration data in response to determining that the calculated adaptation stability metric indicates an unstable adaptation of the configuration data. In such embodiments, the procedure may further include restarting the adaptation of the configuration data using a speech sample of the target speaker. In some examples, the procedure may further include, after the halt, obtaining a new speech sample of the target speaker, and performing the adaptation of the configuration data using the new speech sample. Calculating the adaptation stability metric may include calculating attention data for each portion of the speech sample. In such embodiments, halting the adaptation of the learning machine-based synthesizer may include halting the adaptation of the learning machine-based synthesizer in response to determining that an attention variance level derived from the attention data indicates a non-converged adaptation solution of the speech synthesis system.

[0074] In some embodiments, this procedure may also include generating a parametric style representation of a prosodic style associated with the speech sample using a variational autoencoder. In such embodiments, adapting the configuration data may include adapting configuration data of the speech synthesis system further based on the parametric style representation.

[0075] Returning to FIG. 3, this step further includes providing a configuration of a speech synthesis system with the adapted configuration data, the speech synthesis system including the adapted configuration data being implemented to generate synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker.

[0076] The procedure 300 may further include generating synthetic speech output data, which includes processing the target language input by applying a speech synthesis system configured with the adapted configuration data to synthesize speech having voice and time-domain voice characteristics that approximate the actual voice and time-domain voice characteristics of the target speaker speaking the target language input. It is noted that generating synthetic speech output data also includes cross-language situations where synthetic speech is generated in a language different from the language used in the collected speech samples.

[0077] The above techniques can be implemented, for example, using a programmable computing system executing suitable software instructions, or in suitable hardware such as a field programmable gate array (FPGA), or in some hybrid form. For example, in a programmed technique, the software may include the steps of one or more computer programs executed on one or more programmed or programmable computing systems (which may be of various architectures, such as distributed, client / server, or grid), each including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), at least one user interface (for receiving input using at least one input device or input port, and for output using at least one output device or output port). The software may include one or more modules of a relatively large program. Each module of the program may be implemented as a data structure or other organized data that conforms to a data model stored in a data repository.

[0078] The software may be stored in a persistent form, such as embodied in a volatile or non-volatile storage medium, or any other persistent medium, using the physical characteristics of the medium (e.g., pits and lands on a surface, magnetic domains, or charges) for a period of time (e.g., the time between refresh periods of a dynamic memory device such as dynamic RAM). In preparation for loading the instructions, the software may be created on a tangible, persistent medium, such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or computing device), or may be delivered (e.g., encoded into a propagating signal) via a communication medium of a network to the tangible, persistent medium of a computing system on which the software is executed. Some or all of the processing may be performed on a special-purpose computer, or may be performed using special-purpose hardware, such as a coprocessor or a field programmable gate array (FPGA), or a special-purpose application-specific integrated circuit (ASIC). The processing may be performed in a distributed manner, with different parts of the computations specified by the software being performed by different computing elements. Each such computer program is preferably stored or downloaded into a computer-readable storage medium (e.g., solid-state storage or media, or magnetic or optical media) of a storage device accessible by a general-purpose or special-purpose programmable computer, so as to configure and operate the computer when the storage medium is read by the computer to perform the processes described herein. The system may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, where the medium so configured causes the computer to operate in a particular, predefined manner to perform one or more of the process steps described herein.

[0079] Implementations using neural networks can be implemented on any computing platform, including computing platforms that include one or more microprocessors, microcontrollers, and / or digital signal processors that provide processing functions and other computational and control functions. The computing platform can include one or more CPUs, one or more graphics processing units (GPUs, such as NVIDIA GPUs), and may also include special purpose logic circuitry, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), DSP processors, accelerated processing units (APUs), application processors, customized special purpose circuits, etc., for at least partially implementing the processes and functions of the neural networks (or other types of learning machines), procedures, and methods described herein. Computing platforms used to implement neural networks typically also include memory for storing data and software instructions for executing the functions programmed within the device. Various learning processes implemented using neural networks may be configured or programmed using TensorFlow, an open source software library used for machine learning applications such as neural networks. Other programming platforms available include the Keras (an open source neural network library) building blocks and the NumPy (an open source programming library useful for implementing modules for processing arrays) building blocks.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly or customarily understood. As used herein, the articles "a" and "an" refer to one or more than one (i.e., at least one) of the grammatical object of the article. As an example, "an element" means one element or more than one element. As used herein when referring to a measurable value, such as an amount, duration, etc., "about" and / or "approximately" encompasses a variation of ±20% or ±10%, ±5%, or ±0.1% from the specified value, and thus the variation is appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein. As used herein, "substantially" when referring to a measurable value, such as an amount, duration, physical attribute (such as frequency), etc., encompasses variations of ±20%, or ±10%, ±5%, or even ±0.1% from the specified value, and thus variations are appropriate in the context of the systems, devices, circuits, methods, and other implementations described herein.

[0081] In this specification, including the claims, "or" used in a list of items prefaced with "at least one" or "one or more" indicates a disjunctive list, such as, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C), or a combination having two or more features (e.g., AA, AAB, ABBC, etc.). Also, in this specification, unless otherwise noted, a statement that a function or operation is "based on" an item or condition means that the function or operation is based on the specified item or condition and may be based on one or more items and / or conditions in addition to the specified item or condition.

[0082] Several embodiments of the present invention have been described. Nevertheless, it should be understood that the foregoing description is illustrative, not limiting, of the scope of the present invention, which is defined by the scope of the appended claims. Accordingly, other embodiments are within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. Furthermore, some of the steps described above may be order-independent, and thus may be performed in a different order than described.

Claims

1. 1. A method of generating speech, comprising: obtaining a speech sample of a target speaker; processing the speech samples of the target speaker using a trained encoder to generate a parametric representation of the speech samples of the target speaker; receiving configuration data for a speech synthesis system that accepts the parametric representation as input; adapting the configuration data of the speech synthesis system with inputs including the parametric representation of the target speaker and a time-domain representation of the speech samples of the target speaker to generate adapted configuration data of the speech synthesis system representing the target speaker; providing a configuration of the speech synthesis system with the adapted configuration data, the speech synthesis system including the adapted configuration data being implemented to generate synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker; A method of generating speech comprising:

2. The method of claim 1 , wherein the configuration data includes weights for a neural network-based implementation of the speech synthesis system.

3. Adapting the configuration data according to the time domain representation, Matching the speech samples with corresponding language annotations of the speech samples to generate annotated speech samples that identify speech and non-speech portions and their respective time information, wherein the annotated speech samples represent time-domain speech attribute data of the target speaker; adapting the configuration data of the speech synthesis system at least in part according to the annotated speech samples representing the time-domain speech attribute data of the target speaker; The method of claim 1 or 2, comprising:

4. 4. The method of claim 3, wherein the time-domain speech attribute data of the target speaker includes one or more of a speech pronunciation by the target speaker, an accent of the target speaker, a speech style of the target speaker, or a prosodic feature of the target speaker.

5. The linguistic annotations include transcriptions of words and / or subwords, and matching the speech samples to the corresponding linguistic annotations includes: Aligning word and / or sub-word elements of the transcription to the time domain representation of a target speech sample of the target speaker. The method of claim 3 , comprising:

6. 6. The method of claim 1, further comprising: generating the synthetic speech output data, the method comprising: processing a target language input by applying the speech synthesis system configured with the adapted configuration data to synthesize speech having voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker speaking the target language input.

7. The method of any one of claims 1 to 6, wherein obtaining the speech sample of the target speaker comprises obtaining speech corresponding to a linguistic representation of the verbal content of the speech sample.

8. 8. The method of claim 7, wherein obtaining the voice sample of the target speaker comprises conducting a scripted data collection session with the target speaker, the data collection session including prompting the target speaker to speak the verbal content.

9. performing an audio validation analysis on the voice sample to determine whether the voice sample satisfies one or more audio quality criteria; obtaining a new audio sample in response to the audio sample being determined to not satisfy the one or more audio quality criteria; The method of claim 8 further comprising:

10. The method of claim 7 , further comprising applying filtering and speech enhancement operations to the speech samples to improve a quality of the speech samples.

11. The method of any one of claims 1 to 10, wherein the received configuration data of the speech synthesis system is derived from training speech samples from a number of training speakers different from the target speaker.

12. Adapting the configuration data calculating an adaptation stability index representative of an adaptation performance for adapting the configuration data; halting the adaptation of the configuration data in response to determining that the calculated adaptation stability index indicates unstable adaptation of the configuration data. The method according to any one of claims 1 to 11, comprising:

13. 13. The method of claim 12, further comprising restarting the adaptation of the configuration data using the speech samples of the target speaker.

14. obtaining a new speech sample of the target speaker following said ceasing; and performing said adaptation of said configuration data using said new voice samples; and The method of claim 12 further comprising:

15. calculating the adaptive stability metric includes calculating attention data for each portion of the audio sample; 13. The method of claim 12, wherein discontinuing adaptation of a learning machine based synthesizer comprises discontinuing the adaptation of the learning machine based synthesizer in response to determining that an attention variance level derived from the attention data indicates a non-converged adaptation solution of the speech synthesis system.

16. processing the speech samples of the target speaker using the trained encoder to generate the parametric representation; A method according to any one of the preceding claims, comprising converting the speech samples of the target speaker into a vector representation in the spectral domain.

17. converting the audio samples into a vector representation in the spectral domain; converting the audio samples into a plurality of mel-spectrogram frames; mapping the plurality of mel-spectrogram frames into a fixed dimensional vector; 17. The method of claim 16, comprising:

18. generating a parametric style representation for a prosodic style associated with the speech sample using a variational autoencoder; 18. The method of claim 1, wherein adapting the configuration data comprises adapting the configuration data of the speech synthesis system further based on the parametric style representation.

19. adapting the configuration data of the speech synthesis system according to the parametric representation of the target speaker and the time domain representation of the speech samples; 19. The method of claim 1, comprising adapting the configuration data using a non-parametric adaptation procedure to minimize an error between predicted spectral representation data generated by the speech synthesis system in response to the parametric representation and text data matching the speech samples of the target speaker and actual spectral data derived directly from the speech samples.

20. 1. A speech generation system comprising: a speech capture unit for acquiring speech samples of a target speaker; an encoder applied to the speech samples of the target speaker to generate a parametric representation of the speech samples of the target speaker; 1. A voice synthesis and cloning system comprising: a receiver for receiving configuration data for a speech synthesis system, said speech synthesis system configured to accept said parametric representation as an input; and an adaptation module for adapting the configuration data of the speech synthesis system according to inputs including the parametric representation of the target speaker and a time domain representation of the speech samples of the target speaker to generate adapted configuration data of the speech synthesis system representative of the target speaker; a speech synthesis and cloning system, the speech synthesis system being implemented such that the adaptation module provides configuration of the speech synthesis system with the adapted configuration data, and the speech synthesis system including the adapted configuration data generates synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker; 13. A speech generation system comprising:

21. 21. The system of claim 20, wherein the audio capture unit comprises one or more of: i) an audio collection unit for collecting and recording the audio samples; ii) an audio validation unit configured to perform an audio validation analysis of the audio samples to determine whether the audio samples satisfy one or more audio quality criteria and / or apply filtering operations to the audio samples to improve a quality of the audio samples; or iii) an automatic audio transcription unit configured to generate annotated audio samples from the collected audio samples.

22. A non-transitory computer-readable medium, comprising: Obtain a speech sample of the target speaker; processing the speech samples of the target speaker using a trained encoder to generate a parametric representation of the speech samples of the target speaker; receiving configuration data for a speech synthesis system that accepts the parametric representation as input; adapting the configuration data of the speech synthesis system with inputs including the parametric representation of the target speaker and a time-domain representation of the speech samples of the target speaker to generate adapted configuration data of the speech synthesis system representing the target speaker; the adapted configuration data results in a configuration of the speech synthesis system, and the speech synthesis system including the adapted configuration data is implemented to generate synthetic speech output data having estimated voice characteristics and time-domain voice characteristics that approximate the actual voice characteristics and time-domain voice characteristics of the target speaker. A non-transitory computer-readable medium storing a set of instructions executable by at least one programmable device.

23. A computing device comprising a speech capture unit for obtaining speech samples of a target speaker, and one or more programmable processor-based devices for generating synthetic speech by the steps according to any one of claims 1 to 19.

24. A non-transitory computer readable medium programmed with a set of computer instructions executable on a processor which, when executed, causes operations including the method steps of any one of claims 1 to 19.

Images