Systems and methods for adapting human speaker embeddings to speech synthesis

Abstract

A new method and system for adapting a speech cloning synthesizer to new speakers using real speech data is disclosed. Utterances from one or more target speakers are parameterized and used to initialize embedding vectors for use in the speech synthesizer by clustering the speech data and determining the centroids of the data, using a speaker identification neural network, and / or finding the closest stored embedding vector to the utterance data.

Description

[Technical field]

[0001] [Related Applications] This application claims priority to U.S. Provisional Patent Application No. 62 / 889,675, filed August 21, 2019, and U.S. Provisional Patent Application No. 63 / 023,673, filed May 12, 2020, each of which is incorporated by reference in its entirety.

[0002] [Technical field] The present disclosure relates to improvements for processing audio signals, in particular for speech style transfer implementations. [Background technology]

[0003] Voice style transfer, or voice cloning, can be achieved by deep learning neural network models trained to synthesize voice like a particular identified speaker using inputs other than that speaker, such as speech waveforms from another speaker or speech waveforms from text. An example of such a system is a recurrent neural network, such as the SampleRNN generative model for voice conversion (see, for example, Cong Zhou, Michael Horgan, Vivek Kumar, Cristina Vasco, and Dan Darcy, Voice Conversion with Conditional SampleRNN", Proc. Interspeech 2018, 2018, pp.1973-1977). Initializing the embedding vector for the new voice style is important for efficient convergence, since the model needs to be rebuilt (fitted) to synthesize the voice style of each speaker.

[0004] Training datasets used in speech synthesis development are primarily clean data with consistent speaking styles and similar recording conditions for each speaker, such as people reading audiobooks. Using real speech data (e.g., samples from movies and other media sources) is much more difficult due to the limited amount of clean speech, the effects of different recording channels, and the fact that audio sources may have a variety of speaking styles for a single speaker, including different emotions and different acting roles. Therefore, it is difficult to build a speech synthesizer using real data. Summary of the Invention

[0005] Various audio processing systems and methods are disclosed herein. Some such systems and methods include training for speech synthesis. The methods may be computer-implemented in some embodiments. For example, the methods may be implemented, at least in part, via a control system including one or more processors and one or more non-transitory storage media.

[0006] In some examples, systems and methods are described for adapting a voice cloning synthesizer for a new speaker using real speech data, including creating embedding data for different speaking styles of a given speaker without the difficult task of manually labeling all the data bit by bit. Improved methods for initializing embedding vectors for a voice synthesizer are also disclosed, providing faster convergence of the voice synthesis model.

[0007] In some such examples, the method comprises: receiving as input a plurality of waveforms, each waveform corresponding to speech in said target style; extracting features of the plurality of waveforms to generate a plurality of embedding vectors; clustering the plurality of embedding vectors to generate at least one cluster, each cluster having a centroid; determining the centroid of a cluster of the at least one cluster; and designating the centroid of the cluster as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on at least the initial embedding vector, thereby generating synthetic speech in the target style; may include:

[0008] According to some implementations, at least some operations of the method may include modifying a physical state of at least one non-transitory storage medium location, for example, updating a speech synthesizer table with the initial embedding vector.

[0009] In some examples, the method further includes pre-processing the plurality of waveforms to remove non-speech sounds and silences. In some examples, each cluster has a threshold distance from its centroid, and the adapting step further includes fine-tuning based on a plurality of embedding vectors of the target style at the threshold distance. In some examples, the speech synthesizer is a neural network. In some examples, extracting the features further includes combining sample embedding vectors extracted from windowed samples of a waveform to generate an embedding vector for the waveform. In some examples, the combining includes averaging the sample embedding vectors. In some examples, the input is from a film or video source. In some examples, the target style includes a speaking style of a target person. In some examples, the target style further includes at least one of age, accent, emotion, and behavioral role.

[0010] In some examples, a method of synthesizing a target style of speech includes: receiving as input a plurality of waveforms, each waveform corresponding to speech in said target style; extracting features from the waveforms to generate embedding vectors; calculating a vector distance on an embedding vector of the plurality of embedding vectors to determine an embedding vector distance to each of a plurality of known embedding vectors; determining a known embedding vector among the known embedding vectors that has a shortest distance from the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; synthesizing speech in the target style by the adapted speech synthesizer; The method includes:

[0011] In some examples, the method comprises: receiving as input a plurality of waveforms, each waveform corresponding to speech in the target style; extracting features of the at least one waveform to generate a plurality of embedding vectors; using a speech recognition system on an embedding vector of the plurality of embedding vectors to generate a known embedding vector corresponding to a speech identified by the speech recognition system as closest to the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; synthesizing speech in the target style by the adapted speech synthesizer; The method includes:

[0012] In some examples, the voice recognition system is a neural network.

[0013] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. Accordingly, various innovative aspects of the subject matter described in this disclosure may be embodied in a non-transitory medium having software stored thereon. The software may be executable by one or more components of a control system, for example, as disclosed herein. The software may include, for example, instructions for performing one or more of the methods disclosed herein.

[0014] At least some aspects of the disclosure may be implemented by or through an apparatus. For example, one or more apparatus may be configured to at least partially execute the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between the control system and a memory system, one or more interfaces between the control system and another device, and / or one or more external device interfaces. The control system may include at least one of a general purpose single-chip or multi-chip processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic device, discrete gate or transistor logic, or discrete hardware components. Thus, in some implementations, the control system may include one or more processors and one or more non-transitory storage media operably coupled to the one or more processors.

[0015] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. The relative dimensions of the following drawings may not be drawn to scale. As with reference numbers and designations in the various drawings, the reference numbers and designations in the various drawings generally refer to like elements, but the various reference numbers do not necessarily refer to different elements between the various drawings. [Brief description of the drawings]

[0016] [Figure 1] An example of a method for voice cloning is shown below.

[0017] [Diagram 2] We present an example of how to initialize embedding vectors for phonetic cloning by using clustering.

[0018] [Diagram 3] 1 shows an example of histogram data of audio pitch data for determining the number of clusters to use for clustering.

[0019] [Figure 4A] 1 illustrates an exemplary 2-D projection of clustered speech data. [Figure 4B] 1 illustrates an exemplary 2-D projection of clustered speech data. [Figure 4C] 1 illustrates an exemplary 2-D projection of clustered speech data.

[0020] [Diagram 5] 13 shows an example of a method for initializing embedding vectors for phonetic cloning using vector distance calculations.

[0021] [Figure 6] FIG. 1 illustrates an example of a method for initializing embedding vectors for phonetic cloning using phonetic ID machine learning.

[0022] [Figure 7] ,An example of calculating a representative embedding vector by sampling is shown.

[0023] [Figure 8] FIG. 2 illustrates an exemplary speech synthesizer method according to one embodiment of the present disclosure.

[0024] [Figure 9] 1 illustrates an example hardware implementation of the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] A voice "style" as used herein refers to any group of waveform parameters that distinguish it from another source and / or another context. Examples of "style" include distinctions between different speakers. It can also refer to differences in waveform parameters of a single speaker speaking in different contexts. Different contexts include, for example, speakers speaking at different ages (e.g., a teenage speaker sounds different from a middle-aged speaker when he speaks, so they are two different styles), speakers speaking in different emotional states (e.g., angry, sad, calm, etc.), speakers speaking with different accents or languages, speakers speaking in different business or social contexts (e.g., speaking with friends, family, strangers), actors speaking when playing different roles, or other situational differences that affect a person's speaking style (and thus generally generate different voice waveform parameters). For example, Person A speaking with an English accent, Person B speaking with an English accent, and Person A speaking with a Canadian accent are considered three different "styles."

[0026] As used herein, "waveform parameters" refer to quantifiable information that can be derived from an audio waveform (digital or analog). The derivation can be in the time and / or frequency domain. Examples include pitch, amplitude, pitch variance, amplitude variance, phase alignment, intonation, phoneme duration, phoneme sequence alignment, mel-scale pitch, spectrum, mel-scale spectrum, etc. Also, some or all of the parameters may be values ​​derived from the input speech waveform that are not specifically understood (e.g., combination / transformation of other values). In practice, waveform parameters can refer to both directly measured and estimated parameters.

[0027] As used herein, an "utterance" is a relatively short sample of speech, typically equivalent to a line of dialogue from a scenario (eg, a phrase, sentence, or sequence of sentences lasting a few seconds).

[0028] As used herein, a "speech synthesizer" is a machine learning model that can convert text or audio input into an output of that text or audio spoken with a particular quality that the model has been trained on. Speech synthesizers use embedding vectors for a particular "identity" of the output speaking style. See Chen, Y. et al, "Sample efficiency adaptive text-to-speech", International Conference on Learning Representations, 2019.

[0029] FIG. 1 shows an example of speech cloning using an initialized embedding vector approach. Speech waveforms of a target speech style are obtained from one or more sources (105). Examples of sources include movie / television / video clips, audio recordings, and live sampling / broadcasts. The waveforms may be filtered prior to feature extraction to remove some or all non-verbal components, such as sighs, silences, laughs, coughs, etc. For example, a voice activity detector may be used to trim the non-verbal components. Additionally or alternatively, a noise suppression algorithm may be used to remove background noise. The noise suppression algorithm may be subtractive or based on computational auditory scene analysis (CASA) or similar techniques known in the art. Additionally or alternatively, an audio leveller may be used to adjust the waveform to the same level on a frame-by-frame basis. For example, the audio leveller may set the waveform to -23 dB.

[0030] The waveform from the target source is then parameterized (110) into a number of waveform parameters by feature extraction such that a vector is formed for each utterance. The number of parameters depends on the input of the speech synthesizer (135) and can be any number (such as 32, 64, 100, or 500).

[0031] These vectors can be used to determine the initialization vectors (115) for entering into the embedding vector table (125), i.e., a list of all styles that can be used by the speech synthesizer (135) to train new models for cloning. Additionally, some or all of the vectors can be used as tuning data (120) for fine-tuning the speech synthesizer (135). The speech synthesizer (135) applies machine learning models, such as neural networks, to take linguistic input (130) in the form of speech audio or text and generate a synthetic speech output waveform (140) in the style of the target source (105). Model adaptation can be done by updating the model and embedding vectors through stochastic gradient descent.

[0032] One example of parameterization is phoneme sequence alignment estimation. This can be done using a forced aligner (such as Gentile™) based on a speech recognition system (such as Kaldi™). It converts the audio into Mel-frequency cepstral coefficient (MFCC) features and converts the text to known phonemes via a dictionary. It then aligns the MFCC features with the phonemes. The output includes 1) a sequence of phonemes, and 2) a timestamp / duration for each phoneme. Based on the phonemes and phoneme durations, statistics of phoneme durations and how often the phonemes are spoken can be calculated as parameters.

[0033] Another example of parameterization is pitch estimation, i.e. pitch contour extraction. This can be done with programs such as the WORLD vocoder (DIO and Harvest Pitch Tracker) or the CREPE neural net pitch estimator. For example, the pitch can be extracted every 5 ms, resulting in 200 floating points in a sequence that represent the absolute value of the pitch for every 1 s of speech data as input. If we take the logarithm of these floating points and then normalize it for each target speaker, we can generate a contour of about 0.0 (e.g. a value like "0.5") rather than an absolute pitch value (e.g. 200.0 Hz). Systems like the WORLD pitch estimator use high-level speech time characteristics. First, we use low-pass filters with different cutoff frequencies, and if the filtered signal consists of only the fundamental frequency, it forms a sine wave, and based on the period of this sine wave, we can obtain the fundamental frequency. The zero crossing and peak-dip intervals can be used to select the best fundamental frequency candidate. Since the contour is indicative of pitch variation, we can calculate the variance of the normalized contour to know the magnitude of the variation in the waveform.

[0034] Another example of parameterization is amplitude derivation. This can be done, for example, by first computing the short-time Fourier transform (STFT) of the waveform to obtain the spectrum of the waveform. A Mel filter can be applied to the spectrum to obtain a mel-scale spectrum, which can be converted to a logarithmic mel-scale spectrum. Parameters such as absolute magnitude and amplitude variance can be calculated based on the logarithmic mel-scale spectrum.

[0035] In some embodiments, the parameterization step (110) includes labeling data from speakers. Because this is source-based, the labeling step can be performed on the data in bulk, rather than piece by piece. Note that data labeled for a single speaker may contain multiple speaker styles.

[0036] In some embodiments, parameterization (110) includes phenotype extraction and alignment with the input waveform. One example of this process is to represent the waveform in text (manual or automatic speech recognition system), then convert the sequence of text to a sequence of phonemes by dictionary lookup (e.g., using t2p Perl script), and then align the phoneme sequence with the waveform. A timestamp (start and end time) can be associated with each phoneme (e.g., use Montreal Forced Aligner to convert the audio to MFCC features and create an alignment of the MFCC features with the phonemes). The output, in turn, includes 1) the sequence of phonemes, 2) the timestamp / duration of each phoneme.

[0037] Figures 2-7 show further embodiments of the present disclosure. The following description of such further embodiments will focus on the differences between such embodiments and the embodiment described above with reference to Figure 1. Thus, features common to the embodiment of Figures 2-7 and one of the embodiments of Figure 1 may be omitted from the following description. If so, it should be assumed that features of the embodiment of Figure 1 are possible, or at least possible, in the further embodiments of Figures 2-7, unless the following description requires otherwise.

[0038] In one embodiment, the initialization can be performed by clustering. Figure 2 shows an exemplary method of the clustering method. As also described for Figure 1, the input sample waveforms (205) are directly encoded into parameterized vectors (215) by feature extraction or they are first sent through a speech filtering algorithm (210) and then parameterized (215). The input can cover multiple different styles (one speaker or multiple styles from different speakers) and the data is appropriately labeled. An analysis can be performed on the input to determine the number of clusters (220) expected to be found in the vector space.

[0039] In some embodiments, the number of clusters is determined using statistical analysis of the input, attempting to represent the number of different styles in the input data. In some embodiments, phoneme and triphone duration statistics (indicating how fast the speaker speaks), pitch variance statistics (indicating how much the speaker varies in timbre), and absolute loudness statistics (indicating how fast the speaker speaks) are analyzed as features to estimate the number of speaking styles (clusters), e.g., computing one mean and one variance for each feature sequence, then looking at all the means and variances to roughly estimate how many mean / variance clusters there are.

[0040] In some embodiments, the number of clusters is determined automatically for a particular data by a clustering algorithm. A clustering algorithm (225) is run on the data to find clusters of inputs. This can be, for example, a k-means or Gaussian mixture model (GMM) clustering algorithm. The clusters are identified and the centroids of each cluster are determined (230). The centroids are used as initialized embedding vectors for each cluster / style to train / fit the synthesizer (235) for that style. The labeled input data for that style in the corresponding cluster variance from the corresponding centroid (in the cluster space) can be used as fine-tuning data (240) for the synthesizer adaptation (235).

[0041] Some embodiments of the synthesizer adaptation (235) adapt only the speaker embedding vector, for example: p(x|x 1…t-1 ,emb,c,w), where x is the sample (at time t), x 1…t-1where is the sample history, emb is the embedding vector, c is the conditioning information including extracted conditioning features (e.g., pitch contour, phoneme sequence with timestamp, etc.), and w is the weight of the conditioned SampleRNN. We fix c and w and only perform stochastic gradient descent on emb. Once training converges, we stop training. The updated emb is assigned to the speaker target (new speaker).

[0042] In some embodiments of synthesizer adaptation (235), the speaker embedding vector is adapted first and then the model (all or part) is updated directly. For example, p(x|x 1…t-1 ,emb,c,w), where x is the sample (at time t), x 1…t-1 where is the sample history, emb is the embedding vector, c is the conditioning information including extracted conditioning features (e.g., pitch contour, phoneme sequence with timestamp, etc.), and w is the weight of the conditioned SampleRNN. Fix c and w and only perform stochastic gradient descent on emb. Once the training of emb reaches convergence, start stochastic gradient descent on w. Alternatively, once the training of emb reaches convergence, start stochastic gradient descent on the last output layer of the conditioned SampleRNN. Train for several steps of gradient updates (e.g., 1000 steps) if necessary. The updated w and emb are jointly assigned to the speaker target (new speaker).

[0043] As used herein, training reaching "convergence" refers to a subjective judgment when training shows no substantial improvement. Speech cloning involves listening to the synthesized speech and subjectively assessing its quality. When training the synthesizer, both the training set loss curve and the validation set loss curve can be monitored, and if the validation set loss does not decrease for a threshold number of epochs (e.g., 2 epochs), the learning rate can be decreased (e.g., 50% rate).

[0044] In some embodiments, during the adaptation phase, only the speaker embeddings are adapted. The loss curve can be monitored and a subjective assessment can be made as to whether the training has reached convergence. If there is no subjective improvement, training can be stopped and the remainder of the model will continue to perform well over a number of gradient update steps with low (e.g., 1x10 -6 ) learning rate can be fine-tuned. Again, subjective evaluation can be used to determine when to stop training. Subjective evaluation can also be used to measure the effectiveness of the training procedure.

[0045] Different approaches can be used to select the most appropriate number of clusters. In some embodiments, a pitch analysis can be performed to determine the number of clusters. Before pitch extraction, pre-processing such as silence trimming and non-speech region trimming (similar to filtering (210) shown in FIG. 2) can be applied. FIG. 3 shows an example of a histogram of pitch (Hz) for one person speaking at two different ages. The bars below the dashed line (305) show the pitch values ​​(e.g., extracted in 5 ms increments) of a person aged 50-60. The bars below the dotted line (310) and the dotted line (315) show the pitch values ​​of the same person aged 20-30. This indicates that the appropriate number of clusters is three (one for 50-60 years and two for 20-30 years). This means that there were at least two speech styles in the 20s, which may reflect differences in accent, emotion, and other situations. Note that in this example, the 50-60 age range (305) shows very low variance and a central pitch below 100 Hz, while the 20-30 age range (310 and 315) shows large variance and central pitch around both 130 Hz and 140 Hz. This indicates that there are at least two speaking styles in the 20-30 age range. A pitch variance threshold can be set to determine the number of clusters to use. If the pitch variance is too large to estimate the number of clusters, this indicates that other parameters (beyond or beyond pitch) should be used to determine the number of clusters (the network needs to learn styles beyond just pitch-based styles). In some embodiments, sentiment analysis can be performed on the transcriptions, and the sentiment classification results can be used as an initial estimate of the number of speaking styles. In some embodiments, the number of behavioral roles played by the speaker (who is the actor in this case) in these sources can be used as an initial estimate of the number of speaking styles.

[0046] 4A-4C show examples of clustering projected into 2-D space (the actual space is N-dimensional, where N is the number of parameters, e.g., 64-D). FIG. 4A shows speech data points (vectors of parameters) for three sources, represented as squares (405), circles (410), and triangles (415), respectively. FIG. 4B shows the data clustered into three clusters (420, 435, and 440) with a threshold distance of each cluster's centroid (not shown in FIG. 4B) shown with a dotted line. The threshold distance can be set by the user or can be set equal to the variance of the cluster as determined by the algorithm. FIG. 4C shows the centroids of the three clusters (445, 450, and 455). The centroids do not necessarily correlate directly with any input data, but are calculated from the clustering algorithm. These centroids (445, 450, and 455) can then be used as the initial embedding vectors for the speech synthesis model and can be stored in a table with other styles for future use (each style is treated as a separate ID in the table, even if it is from the same person). Input data whose labels match the centroids of the clusters can be used to fine-tune the speech synthesis model, and outlier data (example shown as 460) can be pruned from being used as tuning data for falling outside a threshold distance (420, 435, 440) from its corresponding centroid (445, 450, 455). In some embodiments, only a single (global) cluster is used to embed speaker, alias speaker identity information, without clustering. In some embodiments, there are multiple clusters used for speaker, alias style embedding.

[0047] Figure 5 shows an example of initializing embedding vectors by vector distance to previously established embedding vectors. Machine learning based speech synthesizers can have an embedding vector table (125) that provides embedding vectors associated with different speech styles (different speakers or different styles depending on how the table was constructed) available for simulation or speech cloning. This resource can be used to generate initial embedding vectors (510) for adapting the synthesizer (235) to new styles.

[0048] The parameterized vector (110) is compared (distance) (505) with values ​​from an embedding vector table (125) to determine the closest vector from the table, which is used as an initialized embedding vector (510) to adapt the synthesizer (235). A random (e.g., initially generated) parameterized vector can be used for the distance calculation (505), or an average parameterized vector can be constructed from multiple parameterized vectors and used for the distance calculation (505). The more embedding vectors from the table (125) used for the distance calculation (505), the higher the accuracy of the resulting initialized embedding vector (510), since it increases the likelihood that a speech style very close to the input is available. The adaptation (235) can also be fine-tuned (520) from the parameterized vector (110). The adaptation (235) can update the embedding vectors based on the fine-tuning (520) for input to the embedding vector table (125), or the initialized embedding vectors (510) can be populated into the table (125) with new identifiers associated with the new style.

[0049] The vector distance calculations may include Euclidean distance, vector dot product, and / or cosine similarity.

[0050] Figure 6 shows an example of embedding vector initialization by speech recognition deep learning. The utterance (105, 210) is feature extracted for use in the speech recognition machine learning system (610). The feature extraction may be the same as or different from the feature extraction of the speech synthesizer (235). The speech recognition machine learning system may be a neural network.

[0051] If they are the same, the parameterized vector (605) is run through the Voice ID System (610) to "identify" which entry in the Voice ID Database (625) matches the utterance. Obviously, the speaker will not typically be present in the Voice ID Database at this point, but if there are a large number of entries in the table (e.g., 30k), the speaker (625) identified from the table should closely match the style of speech. This means that the embedding vector from the Voice ID Database (625) selected by the Voice ID Model (610) can be used as an initialized embedding vector for adapting the speech synthesizer (235). As with the other initialization methods, this can be fine-tuned with the parameterized vector (605) for the utterance.

[0052] If the parameters of the Voice ID system are different from the parameters of the synthesizer, the method is generally the same, but the initialized embedding vector must be retrieved from the database (625) in a format suitable for the synthesizer (235), and the fine-tuning data (120) must undergo a separate feature extraction from the Voice ID parameterization (605).

[0053] In some embodiments, feature extraction of an utterance can be done by combining vectors extracted from shorter segments of a longer utterance. Figure 7 shows an example of averaged extracted vectors for an utterance. Utterance X (705) is input as a waveform for a duration, e.g., 3 seconds. The waveform (705) is sampled over several shorter durations, e.g., 5 milliseconds, moving sampling windows (710). The windowed samples can overlap (715). Windowing can be performed sequentially on the waveform or in parallel on some or all of the waveform at the same time. Each sample undergoes feature extraction (720) to produce n embedding vectors (725) e1-e n These embedding vectors are the representative embedding vectors (735) e for the utterance X (705). x The vectors (705) are combined (730) to generate vectors (725). An example of combining vectors (730) is taking the average of the vectors (725) from the window samples (710). Another example of combining vectors (730) is using a weighted sum. For example, a speech detector can be used to identify speech frames (e.g., "i" and "aw") and non-speech frames (e.g., "t", "s", "k"). Speech frames can be weighted relative to non-speech frames because speech frames contribute more to the perception of how a voice sounds. The speech (705) can be raw audio or pre-processed audio with mute portions and / or non-speech portions of the waveform trimmed.

[0054] According to some embodiments, a speech synthesizer system may be as shown in FIG. 8. Given an input of a waveform from a speech utterance (805), the waveform data may first be "cleaned" (810). This may include the use of a noise suppression algorithm (811) and / or an audio leveller (812). The data may then be labeled (815) to identify the waveform to the speaker. Next, phonemes are extracted (820) and the phoneme sequence is aligned with the waveform (825). Also, a pitch contour (830) may be extracted from the waveform. The aligned phonemes (825) and pitch contour (830) provide parameters for adaptation (835). The adaptation sets a training goal based on conditional SampleRNN weights (840), and then stochastic gradient descent is performed on the embedding vector (845). Once the training on the embedding vectors has converged, a) training is stopped and updated embedding vectors are assigned to the speakers (850a), or b) stochastic gradient descent is performed on the weights (or the final output layer of the conditioned SampleRNN) resulting in an updated embedding vector being assigned to the speaker (850b), as in this example embodiment.

[0055] FIG. 9 is an exemplary embodiment of target hardware (10) (e.g., a computer system) for implementing the embodiments of FIGS. 1-8. The target hardware includes a processor (15), a memory bank (20), a local interface bus (35), and one or more input / output devices (40). The processor can execute one or more instructions associated with the implementation of FIGS. 1-8 as provided by an operating system (25) based on any executable programs (30) stored in the memory (20). These instructions are sent to the processor (15) via a local interface (35) as dictated by some data interface protocol specific to the local interface and the processor (15). It should be noted that the local interface (35) is a symbolic representation of some elements generally intended to provide address, control, and / or data connections between multiple elements of a processor-based system, such as controllers, buffers (caches), drivers, repeaters, and receivers. In some embodiments, the processor (15) can include some local memories (caches) where some of the instructions to be executed can be stored for some added execution speed. Execution of instructions by the processor may require the use of some input / output device (40), such as input of data from a file stored on a hard disk, input of commands from a keyboard, input of data and / or commands from a touch screen, output of data to a display, or output of data to a USB flash drive. In some embodiments, an operating system (25) facilitates these tasks by being the central element for collecting the various data and instructions required for the execution of the program and providing them to the microprocessor. In some embodiments, there is no operating system and all tasks are under the direct control of the processor (15), although the basic architecture of the target hardware device (10) is the same as that shown in FIG. 9.In some embodiments, multiple processors may be used in a parallel configuration for additional execution speed. In such cases, the executable programs may be specifically adapted for parallel execution. Also, in some embodiments, the processor (15) may perform some of the implementations of FIGS. 1-8, while other parts may be implemented using dedicated hardware / firmware located at input / output locations accessible to the target hardware (10) via a local interface (35). The target hardware (10) may include multiple executable programs (30), each of which may be executed independently or in combination with one another.

[0056] A number of embodiments of the present disclosure have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other aspects are within the scope of the following claims.

[0057] The present disclosure is directed to certain implementations for purposes of describing some innovative aspects described herein and examples of contexts in which these innovative aspects may be implemented. However, the teachings herein may be applied in a variety of different ways. Moreover, the described embodiments may be implemented in various hardware, software, firmware, and the like. For example, aspects of the present application may be realized, at least in part, in an apparatus, a system including one or more devices, a method, a computer program product, and the like. Accordingly, aspects of the present application may take the form of hardware embodiments, software embodiments (including firmware, resident software, microcode, and the like), and / or embodiments combining both software and hardware aspects. Such embodiments may be referred to herein as "circuits," "modules," "engines," "processes," or "blocks." Some aspects of the present application may take the form of a computer program product embodied in one or more non-transitory medium(s) having computer-readable program code embodied therein. Such non-transitory media may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. Thus, the teachings of the present disclosure are not limited to the implementations illustrated and / or described herein, but rather have broad applicability.

Claims

1. 1. A method for synthesizing a target style of speech, comprising: receiving at least one waveform as input, each waveform corresponding to speech in said target style; extracting features of the at least one waveform to generate at least one embedding vector; clustering the at least one embedding vector to generate at least one cluster, each cluster having a centroid; determining the centroid of a cluster of the at least one cluster; designating the centroids of the clusters as initial embedding vectors for a speech synthesizer; adapting the speech synthesizer based on at least the initial embedding vector, thereby generating synthetic speech in the target style; The method includes:

2. The method of claim 1 , further comprising the step of pre-processing the at least one waveform to remove non-speech sounds and silences.

3. The method of claim 1 or 2, wherein each cluster has a threshold distance from its centroid, and the adapting step further comprises fine-tuning based on at least one embedding vector of the target style at the threshold distance.

4. The method according to any one of claims 1 to 3, wherein the speech synthesizer is a neural network.

5. 5. The method of claim 1, wherein extracting features further comprises combining sample embedding vectors extracted from windowed samples of the at least one waveform to generate an embedding vector for the waveform.

6. The method of claim 5 , wherein the combining comprises averaging the sample embedding vectors.

7. The method of any one of claims 1 to 6, wherein the input is from a film or video source.

8. The method according to any one of claims 1 to 7, wherein the target style comprises a speaking style of a target person.

9. The method of claim 8 , wherein the target style further comprises at least one of an age, an accent, an emotion, and a behavioral role.

10. The method of claim 8 , wherein the target person is an actor and the target style is the target person at a younger age than their current age.

11. receiving further waveforms as the input, each further waveform corresponding to a second style of speech different from the target style; extracting features of the further waveform to generate at least a second embedding vector; The method of claim 1 , further comprising:

12. The method of claim 11 , further comprising, prior to the clustering, determining an expected number of clusters, and wherein the clustering is based on the expected number of clusters.

13. The method of claim 12 , wherein determining the expected number of clusters uses a statistical analysis of the input.

14. 1. A method for synthesizing a target style of speech, comprising: receiving as input at least one waveform, each waveform corresponding to speech in said target style; extracting features from the at least one waveform to generate at least one embedding vector; calculating a vector distance on the embedding vector of the at least one embedding vector to determine an embedding vector distance to each of a plurality of known embedding vectors; determining a known embedding vector among the known embedding vectors that has a shortest distance from the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; synthesizing speech in the target style by the adapted speech synthesizer; The method includes:

15. 1. A method for synthesizing a target style of speech, comprising: receiving as input at least one waveform, each waveform corresponding to speech in the target style; extracting features of the at least one waveform to generate at least one embedding vector; using a speech recognition system on an embedding vector of the at least one embedding vector to generate a known embedding vector corresponding to a speech identified by the speech recognition system as closest to the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; synthesizing speech in the target style by the adapted speech synthesizer; The method includes:

16. The method of claim 15 , wherein the voice recognition system is a neural network.

17. The method according to any one of claims 1 to 16, further comprising the step of updating a speech synthesizer table with said initial embedding vector.

18. A computer program arranged to carry out, on a computer, the method according to any one of claims 1 to 16.

19. Apparatus configured to carry out the method according to any of the preceding claims.

Images