System and method for adapting human speaker embeddings in speech synthesis

Abstract

A novel method and system for adapting a voice cloning synthesizer for a new speaker using real speech data is disclosed. Utterances from one or more target speakers are parameterized and used to initialize embedding vectors for use by the voice synthesizer by clustering the utterance data and determining the centroid of the data using a speaker identification neural network and / or by finding the stored embedding vector closest to the utterance data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 889,675, filed on August 21, 2019, and U.S. Provisional Patent Application No. 63 / 023,673, filed on May 12, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to improvements in the processing of audio signals. In particular, the present disclosure relates to processing audio signals for speech style transfer implementations. Background Art

[0004] Speech style conversion or voice cloning can be accomplished by a deep learning neural network model that is trained to synthesize speech that sounds like a specific identified speaker using input different from that speaker (e.g., a speech waveform from another speaker or from text). An example of such a system is a recurrent neural network, such as the SampleRNN generative model for speech conversion (see, e.g., Cong Zhou, Michael Horgan, Vivek Kumar, Cristina Vasco, and Dan Darcy, “Voice Conversion with Conditional SampleRNN,” Interspeech 2018, 2018, pp. 1973–1977). Since the model needs to be rebuilt (adapted) for each speaker's voice style to be synthesized, initializing the embedding vector for the new voice style is important for efficient convergence.

[0005] Training datasets used in speech synthesis development are mostly clean data with a consistent speaking style and similar recording conditions for each speaker, such as people reading audiobooks. Using real speech data (for example, samples extracted from movies or other media sources) is more challenging because the amount of clean speech is limited, there are multiple recording channel effects, and for a single speaker, the source can have a variety of speaking styles, including different emotions and different roles, making it difficult to build a speech synthesizer using real data. Summary of the Invention

[0006] Various audio processing systems and methods are disclosed herein. Some of these systems and methods may involve training speech synthesis. In some embodiments, the methods may be computer-implemented. For example, the methods may be implemented, at least in part, via a control system comprising one or more processors and one or more non-transitory storage media.

[0007] In some examples, systems and methods are described for adapting a voice cloning synthesizer for a new speaker using real speech data, including creating embedding data for different speaking styles for a given speaker (as opposed to embedding data that differentiates solely by speaker identity) without the arduous task of manually labeling all the data bit by bit. Also disclosed are improved methods for initializing embedding vectors for a speech synthesizer, providing faster convergence of the speech synthesis model.

[0008] In some such examples, the method can involve receiving as input a plurality of waveforms, the plurality of waveforms comprising a plurality of waveforms each corresponding to an utterance in a target style; extracting features of at least one waveform to create a plurality of embedding vectors; clustering the embedding vectors to produce at least one cluster, each cluster having a centroid; determining a centroid of a cluster in the at least one cluster; designating the centroid of the cluster as an initial embedding vector for a speech synthesizer; and adapting the speech synthesizer based at least on the initial embedding vector to produce synthesized speech in the target style.

[0009] According to some embodiments, at least some operations of the method may involve changing the physical state of at least one non-transitory storage medium location. For example, updating a speech synthesizer table with the initial embedding vector.

[0010] In some examples, the method further includes pre-processing the plurality of waveforms to remove non-speech sounds and silence. In some examples, each cluster has a threshold distance from its centroid, and the adapting further includes fine-tuning a plurality of embedding vectors within the threshold distance based on the target style. In some examples, the speech synthesizer is a neural network. In some examples, extracting features further includes combining sample embedding vectors extracted from window samples of the waveform to produce an embedding vector for the waveform. In some examples, combining includes averaging the sample embedding vectors. In some examples, the input is from a movie or video source. In some examples, the target style includes a speaking style of a target person. In some examples, the target style also includes at least one of age, accent, emotion, and role playing.

[0011] In some examples, the method may involve receiving a plurality of waveforms as input, the plurality of waveforms including a plurality of waveforms each corresponding to an utterance in a target style; extracting features of at least one waveform to create a plurality of embedding vectors; calculating vector distances for embedding vectors in the plurality of embedding vectors, and comparing the embedding vector distances with a plurality of known embedding vectors; determining a known embedding vector in the plurality of known embedding vectors that has a shortest distance to the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; and synthesizing speech in the target style using the adapted speech synthesizer.

[0012] In some examples, the method can involve receiving as input a plurality of waveforms, the plurality of waveforms including a plurality of waveforms each corresponding to an utterance in a target style; extracting features of at least one waveform to create a plurality of embedding vectors; using a speech recognition system on an embedding vector from the plurality of embedding vectors to produce a known embedding vector corresponding to an utterance identified by the speech recognition system as corresponding most closely to the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; and synthesizing speech in the target style using the adapted speech synthesizer.

[0013] In some examples, the speech recognition system is a neural network.

[0014] Some or all of the methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, the various innovative aspects of the subject matter described in this disclosure can be implemented in a non-transitory medium with software stored thereon. Software, for example, can be performed by one or more components of control systems such as those disclosed herein. Software, for example, may include instructions for executing one or more methods in the methods disclosed herein.

[0015] At least some aspects of the present disclosure may be implemented via one or more devices. For example, one or more devices may be configured to at least partially perform the method disclosed herein. In some embodiments, the device may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between a control system and a memory system, one or more interfaces between a control system and another device, and / or one or more external device interfaces. The control system may include at least one of a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic or a discrete hardware component. Therefore, in some embodiments, the control system may include one or more processors and one or more non-transitory storage media operably coupled to the one or more processors.

[0016] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Additional features, aspects, and advantages will become apparent from the description, drawings, and claims. Please note that the relative dimensions of the following figures may not be drawn to scale. Similar reference numbers and names in different figures generally indicate similar elements, but different reference numbers do not necessarily indicate different elements between different figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An example of a method of voice cloning is shown.

[0018] Figure 2 An example of a method of initializing an embedding vector for voice cloning by using clustering is shown.

[0019] Figure 3 An example of histogram data of voice pitch data used to determine the number of clusters to be used for clustering is shown.

[0020] Figures 4A-4C An exemplary 2-D projection of clustered speech data is shown.

[0021] Figure 5 An example of a method for initializing an embedding vector for voice cloning using vector distance calculation is shown.

[0022] Figure 6 An example of a method for initializing embedding vectors for voice cloning using voice ID machine learning is shown.

[0023] Figure 7 An example of computing a representative embedding vector by sampling is shown.

[0024] Figure 8 An exemplary speech synthesizer method according to an embodiment of the present disclosure is shown.

[0025] Figure 9 An exemplary hardware implementation of the methods described herein is shown. DETAILED DESCRIPTION

[0026] As used herein, speech " style " refers to any grouping of waveform parameters that distinguish it from another source and / or another environment.The example of " style " includes distinguishing between different speakers.It can also refer to the difference of waveform parameters that a single speaker speaks in different environments.Different environments can include, for example, speakers speak at different ages (for example, people speak when they are teenagers and they speak when they are middle-aged and sound different, so this will be two different styles), speakers speak under different emotional states (for example anger, sadness, calmness, etc.), speakers speak with different accents or languages, speakers speak in different business or social environments (for example, talking to friends, talking to family, talking to strangers, etc.), actors speak when playing different roles or any other environmental differences that will affect people's way of speaking (and therefore, usually different speech waveform parameters are produced).Therefore, for example, people A speaks with a British accent, people B speaks with a British accent and people A speaks with a Canadian accent and will be considered as 3 different " styles ".

[0027] As used herein, "waveform parameters" refers to quantifiable information that can be derived from an audio waveform (digital or analog). The derivation can be performed in the time domain and / or the frequency domain. Examples include pitch, amplitude, pitch variation, amplitude variation, phasing, intonation, utterance duration, phoneme sequence alignment, Mel-scaled pitch, frequency spectrum, Mel-scaled spectrum, etc. Some or all of the parameters may also be values ​​derived from the input audio waveform without any specific understood meaning (e.g., a combination / transformation of other values). In practice, waveform parameters may refer to both directly measured parameters and estimated parameters.

[0028] As used herein, an "utterance" is a relatively short speech sample, typically equivalent to a line of dialogue in a script (e.g., a phrase, sentence, or series of sentences lasting a few seconds).

[0029] As used herein, a "speech synthesizer" is a machine learning model that can convert text or speech input into output that text or speech spoken with a specific quality that the model has learned. Speech synthesizers use embedding vectors for a specific "identity" of the output speaking style. See, for example, Chen, Y. et al., "Sample Efficient Adaptive Text-to-Speech," presented at the 2019 International Conference on Representation Learning.

[0030] Figure 1An example of voice cloning using an initialized embedding vector method is shown. Waveforms of speech in the target voice style are taken from one or more sources (105). Examples of sources include movie / television / video clips, audio recordings, and live samples / broadcasts. The waveforms can be filtered before feature extraction to eliminate some or all non-speech components, such as sighs, silences, laughter, coughs, etc. For example, a voice activity detector (VAD) can be used to trim non-speech components. Additionally or alternatively, a noise suppression algorithm can be used to remove background noise. The noise suppression algorithm can be subtractive or can be based on computational auditory scene analysis (CASA) or can be based on similar techniques known in the art. Additionally or alternatively, an audio leveler can be used to adjust the waveforms frame by frame to be at the same level. For example, the audio leveler can set the waveform to -23dB.

[0031] The waveforms from one or more target sources are then parameterized (110) into a plurality of waveform parameters through feature extraction, thereby forming a vector for each utterance. The number of parameters depends on the input to the speech synthesizer (135) and can be any number (e.g., 32, 64, 100, or 500).

[0032] These vectors can be used to determine the initialization vector (115) to enter into the embedding vector table (125), which is a list of all styles that can be used by the speech synthesizer (135) to train a new model for cloning. Additionally, some or all of the vectors can be used as adjustment data (120) for fine-tuning the speech synthesizer (135). The speech synthesizer (135) adapts a machine learning model such as a neural network to obtain language input (130) in the form of speech audio or text, and produces an output waveform (140) of synthesized speech in the style of the target source (105). The adaptation of the model can be performed by updating the model and the embedding vectors by stochastic gradient descent.

[0033] An example of parameterization is the estimation of phoneme sequence alignment. This can be achieved by using a speech recognition system such as Kaldi TM )'s mandatory aligner (e.g. Gentile TM ) is performed. This converts the audio into Mel-Frequency Cepstral Coefficient (MFCC) features and converts the text into known phonemes using a dictionary. It then performs alignment between the MFCC features and the phonemes. The output contains 1) a sequence of phonemes and 2) a timestamp / duration for each phoneme. Based on the phonemes and phoneme duration, statistics on the phoneme duration and the frequency of the spoken phoneme can be calculated as parameters.

[0034] Another example of parameterization is pitch estimation or pitch contour extraction. This can be accomplished using programs such as the WORLD vocoder (DIO and Harvest pitch tracker) or the CREPE neural network pitch estimator. For example, pitch can be extracted every 5 ms, so that for every 1 second of speech data as input, 200 floating-point numbers representing the absolute value of the pitch are sequentially obtained. Taking the logarithm of these floating-point numbers and then normalizing them for each target speaker can produce contours around 0.0 (e.g., a value like "0.5") rather than absolute pitch values ​​(e.g., 200.0 Hz). Systems like the WORLD pitch estimator use high-level temporal features of speech. They first apply low-pass filters with different cutoff frequencies. If the filtered signal contains only the fundamental frequency, a sine wave is formed, and the fundamental frequency can be obtained based on the period of this sine wave. The zero crossings and peak-to-valley intervals can be used to select the best fundamental frequency candidate. Since the contours show pitch variation, the variance of the normalized contours can be calculated to understand how much variation there is in the waveform.

[0035] Another example of parameterization is amplitude derivation. For example, this can be accomplished by first computing the short-time Fourier transform (STFT) of the waveform to obtain the waveform's spectrum. A Mel filter can be applied to the spectrum to obtain a Mel-scale spectrum, and the Mel-scale spectrum can be logarithmically scaled to convert it to a log-Mel-scale spectrum. Parameters such as absolute loudness and amplitude variance can be calculated based on the log-Mel-scale spectrum.

[0036] In some embodiments, the parameterization step (110) includes labeling the data from the speaker. Since this is source-based, the labeling step can be performed on the data as a whole, rather than on a per-speaker basis. Note that the data labeled for a single speaker can contain multiple speaking styles.

[0037] In some embodiments, parameterization (110) includes phoneme extraction and alignment with the input waveform. An example of this process is to transcribe the waveform into text (manually or via an automatic speech recognition system), then convert the text sequence into a phoneme sequence via a dictionary search (e.g., using the t2p Perl script), and then align the phoneme sequence with the waveform. A timestamp (start time and end time) can be associated with each phoneme (e.g., using a Montreal force aligner to convert the audio into MFCC features and create an alignment between the MFCC features and the phoneme). For this, the output contains: 1) the phoneme sequence 2) the timestamp / duration of each phoneme.

[0038] Figure 2-Figure 7 Further embodiments of the present disclosure are described. The following description of such further embodiments will focus on the differences between such embodiments and the previously described embodiments. Figure 1Therefore, the differences between the embodiments described above may be omitted from the following description. Figure 2-Figure 7 One of the embodiments and Figure 1 If so, then unless the following Figure 2-Figure 7 If the description requires otherwise, it should be assumed Figure 1 The features of the embodiments have been or at least can be Figure 2-Figure 7 is implemented in a further embodiment of the present invention.

[0039] In one embodiment, initialization may be performed by clustering. Figure 2 An exemplary method of clustering method is shown. Figure 1 As described, input sample waveforms (205) are either encoded directly into parameterized vectors (215) through feature extraction, or they are first sent through a speech filtering algorithm (210) and then parameterized (215). The input can be for a variety of different styles (from a single speaker or multiple styles from different speakers), where the data is appropriately labeled. Analysis can be performed on the input to determine the number of clusters expected to be found in the vector space (220).

[0040] In some embodiments, the number of clusters is determined using statistical analysis of the input and attempts to represent the number of different styles in the input data. In some embodiments, statistics of phoneme and triphone durations (indicating how fast the speaker speaks), statistics of pitch variation (indicating how drastically the speaker changes pitch), and statistics of absolute loudness (indicating how loud the speaker speaks) are analyzed as features to estimate the number of speaking styles (clusters), for example, by calculating a mean and a variance for each feature sequence in the feature sequence, then looking at all the means and variances, and then roughly estimating how many mean / variance clusters there are.

[0041] In some embodiments, for certain data, the number of clusters is automatically determined by a clustering algorithm. A clustering algorithm (225) is performed on the data to find input clusters. For example, this can be a k-means or Gaussian mixture model (GMM) clustering algorithm. Where clusters are identified, the centroid of each cluster is determined (230). The centroid is used as an initialization embedding vector for each cluster / style for training / adapting a synthesizer for that style (235). Input data tagged for that style that is within the corresponding cluster variance (in cluster space) from the corresponding centroid can be used as fine-tuning data (240) for synthesizer adaptation (235).

[0042] Some embodiments of synthesizer adaptation (235) adapt only the speaker embedding vector. For example, let the training objective be: p(x|x 1…t-1 ,emb,c,w), where x is a sample (at time t), x 1…t-1is the sample history, emb is the embedding vector, c is the conditional information containing the extracted conditional features (e.g., pitch contour, timestamped phoneme sequence, etc.), and w represents the weights of the conditional SampleRNN. c and w are fixed and stochastic gradient descent is performed only on emb. Training stops once convergence is reached. The updated emb is assigned to the speaker target (new speaker).

[0043] In some embodiments of synthesizer adaptation (235), the speaker embedding vector is adapted first, and then the model (in whole or in part) is updated directly. For example, let the training objective be: p(x|x 1…t-1 ,emb,c,w), where x is a sample (at time t), x 1…t-1 is the sample history, emb is the embedding vector, c is the conditional information containing the extracted conditional features (such as pitch contours, time-stamped phoneme sequences, etc.), and w represents the weights of the conditional SampleRNN. Fix c and w and perform stochastic gradient descent only on emb. Once the training of emb reaches convergence, start stochastic gradient descent on w. Alternatively, once the training of emb reaches convergence, start stochastic gradient descent on the last output layer of the conditional SampleRNN. Optionally, train the gradient updates for a few steps (e.g., 1000 steps). The updated w and emb are assigned to the speaker target (new speaker) together.

[0044] As used herein, training reaching "convergence" refers to a subjective determination of when training no longer shows significant improvement. For voice cloning, this can include listening to the synthesized speech and subjectively assessing the quality. While training the synthesizer, both the loss curve for the training set and the loss curve for the validation set can be monitored, and if the loss on the validation set does not decrease within a certain threshold number of epochs (e.g., 2 epochs), the learning rate can be reduced (e.g., by a rate of 50%).

[0045] In some embodiments, only speaker embeddings are adapted during the adaptation phase. The loss curve can be monitored and a subjective evaluation can be performed to determine whether the training has reached convergence. If there is no subjective improvement, the training can be stopped and the training can be performed at a low (e.g., 1x10 -6 ) learning rate. Similarly, subjective evaluation can be used to determine when to stop training. Subjective evaluation can also be used to measure the effectiveness of the training process.

[0046] Different methods can be used to select the most appropriate number of clusters. In some embodiments, a pitch analysis can be performed to determine the number of clusters. Applications such as silence trimming and non-speech region trimming (similar to Figure 2The pre-processing of the filtration (210) shown in FIG. Figure 3 An exemplary histogram of the pitch (in Hertz) of a person speaking at two different ages is shown. The bar below the dashed line (305) shows the pitch values ​​of the person when they were 50-60 years old (e.g., extracted in 5 ms increments). The bar below the dotted line (310) and the dotted dashed line (315) shows the pitch values ​​of the same person when they were 20-30 years old. This indicates that the number of suitable clusters is 3 - one for 50-60 years old and two for 20-30 years old, which means that this person has at least two speaking styles in their 20s, perhaps reflecting differences in accent, emotion, or other circumstances. Note that in this example, the 50-60 age range (305) shows very low variance and a center pitch below 100 Hz, while the 20-30 age range (310 and 315) shows a larger variance and a center pitch around 130 Hz and 140 Hz. This indicates that there are at least two speaking styles in the 20-30 age range. A pitch variance threshold can be set to determine how many clusters to use. If the pitch variance is too large to estimate the number of clusters, this indicates that other parameters (other than or in addition to pitch) should be used to determine the number of clusters (the network needs to learn styles other than just pitch-based styles). In some embodiments, sentiment analysis can be performed on the transcriptions, and the sentiment classification results can be used as an initial estimate of the number of voice styles. In some embodiments, the number of roles played by the speaker (in this case, the actor) in the source is used as an initial estimate of the number of voice styles.

[0047] Figures 4A-4C An example of clustering projected into a 2-D space (the actual space will be N-dimensional, where N is the number of parameters, e.g., 64-D) is shown. Figure 4A Three sources of utterance data points (vectors of parameters) are shown, represented here as squares (405), circles (410) and triangles (415). Figure 4B shows data clustered into three clusters (420, 435, and 440), where the centroid of each cluster ( Figure 4B The threshold distance (not shown) is indicated by a dashed line. The threshold distance can be set by the user; or it can be set equal to the variance of the cluster determined by the algorithm. Figure 4CThe centroids (445, 450, and 455) of three clusters are shown. The centroids are not necessarily directly associated with any input data - they are calculated based on the clustering algorithm. These centroids (445, 450, and 455) can then be used as initial embedding vectors for the speech synthesis model and can be stored in a table with other styles for future use (each style is treated as a separate ID in the table even if it is from the same person). Input data whose labels match the centroids of the clusters can be used to fine-tune the speech synthesis model; outlier data (shown as an example of 460) can be pruned away from being used as adjustment data because it is outside a threshold distance (420, 435, 440) from its corresponding centroid (445, 450, 455). In some embodiments, there is only a single (global) cluster for the speaker, that is, there is no clustered speaker identity embedding. In some embodiments, there are multiple clusters for the speaker, that is, style embeddings.

[0048] Figure 5 An example of initializing embedding vectors by vector distance from previously established embedding vectors is shown. A machine learning-based speech synthesizer can have an embedding vector table (125) that provides embedding vectors associated with different speech styles (different speakers or different styles, depending on how the table is constructed) that can be used for simulation or speech cloning. This resource can be used to generate initial embedding vectors (510) to adapt the synthesizer (235) to a new style.

[0049] The parameterized vector (110) can be compared (distance) (505) with the values ​​in the embedding vector table (125) to determine the closest vector in the table, which is used as an initialization embedding vector (510) for adapting the synthesizer (235). A random (e.g., first generated) parameterized vector can be used for the distance calculation (505), or an average parameterized vector can be constructed from multiple parameterized vectors and used for the distance calculation (505). The more embedding vectors in the table (125) used for the distance calculation (505), the more accurate the initialization embedding vector (510) obtained will be, because this provides a greater probability that a voice style that is very close to the input is available. The adaptation (235) can also be fine-tuned (520) based on the parameterized vector (110). The adaptation (235) can update the embedding vector for entry into the embedding vector table (125) based on the fine-tuning (520), or the initialization embedding vector (510) can be populated into the table (125) with a new identifier that associates it with a new style.

[0050] Vector distance calculations may include Euclidean distance, vector dot product, and / or cosine similarity.

[0051] Figure 6An example of initializing an embedding vector by speech recognition deep learning is shown. Feature extraction is performed on the utterance (105, 210) for use by the speech recognition machine learning system (610). The feature extraction can be the same as the feature extraction used for the speech synthesizer (235), or it can be different. The speech recognition machine learning system can be a neural network.

[0052] If the feature extraction is the same as that used for the speech synthesizer (235), the parameterized vector (605) is run through the voice ID system (610) to "identify" which entry in the voice ID database (625) matches the utterance. Obviously, the speaker will usually not be in the voice ID database at this point, but if there are a large number of entries in the table (e.g., 30k), the speaker identified from the table (625) should closely match the style of the utterance. This means that the embedding vector from the voice ID database (625) selected by the voice ID model (610) can be used as the initialization embedding vector to adapt the speech synthesizer (235). As with other initialization methods, this can be fine-tuned using the parameterized vector (605) for the utterance.

[0053] If the parameters of the Voice ID system are different from those of the synthesizer, the approach is largely the same, but the initialization embedding vectors will have to be looked up from the database (625) in a form suitable for the synthesizer (235), and the fine-tuning data (120) will have to be feature extracted separately from the Voice ID parameterization (605).

[0054] In some embodiments, feature extraction of an utterance may be accomplished by combining vectors extracted from shorter segments of a longer utterance. Figure 7 An example of an average extracted vector for an utterance is shown. An utterance X (705) is input as a waveform of some duration, e.g., 3 seconds. The waveform (705) is sampled over some moving sampling window (710) of smaller duration (e.g., 5 ms). The windowed samples can overlap (715). Windowing can be run sequentially on the waveform or in parallel on part or all of the waveform simultaneously. Each sample is subjected to feature extraction (720) to produce a set of n embedding vectors (725) e1-e n These embedding vectors are combined (730) to produce a representative embedding vector (735) for utterance X (705) x. One example of a combined vector (730) is to average the vectors (725) from the window samples (710). Another example of a combined vector (730) is to use a weighted sum. For example, a speech detector can be used to identify speech frames (e.g., "i" and "aw") and unspeech frames (e.g., "t", "s", "k"). Because speech frames contribute more to the perception of speech sounds, speech frames can be weighted over unspeech frames. The speech (705) can be raw audio or pre-processed audio with silence and / or non-speech portions of the waveform trimmed.

[0055] According to some embodiments, the speech synthesizer system may be configured as follows: Figure 8 As shown. Given a waveform input (805) from a speech utterance, the waveform data can first be "cleaned" (810). This can include using a noise suppression algorithm (811) and / or an audio leveler (812). Next, the data can be labeled (815) to identify the speaker's waveform. Phonemes are then extracted (820) and the phoneme sequence is aligned with the waveform (825). Pitch contours (830) can also be extracted from the waveform. The aligned phonemes (825) and pitch contours (830) provide parameters for adaptation (835). Adaptation is based on the established training target (840) of the conditional SampleRNN weights and then performing stochastic gradient descent on the embedding vector (845). Once the training of the embedding vector converges, a) training is stopped and the updated embedding vector is assigned to the speaker (850a), or b) stochastic gradient descent is performed on the weights (or the last output layer of the conditional SampleRNN) and the resulting updated embedding vector is assigned to the speaker (850b). Embodiments of this example

[0056] Figure 9 Is used to implement Figures 1-8 An exemplary embodiment of target hardware (10) (e.g., a computer system) of an embodiment of the present invention. The target hardware includes a processor (15), a memory bank (20), a local interface bus (35), and one or more input / output devices (40). The processor can execute the same Figures 1-8One or more instructions related to the embodiment of the present invention and provided by the operating system (25) based on some executable program (30) stored in the memory (20). These instructions are transmitted to the processor (15) via the local interface (35) and are specified by some data interface protocol specific to the local interface and the processor (15). It should be noted that the local interface (35) is a symbolic representation of a plurality of elements such as controllers, buffers (caches), drivers, repeaters and receivers, which are generally used to provide address, control and / or data connections between the plurality of elements of the processor-based system. In some embodiments, the processor (15) may be equipped with some local memory (cache) in which it can store some of the instructions to be executed to increase some execution speed. The execution of instructions by the processor may require the use of some input / output devices (40), such as inputting data from a file stored on the hard disk, inputting commands from the keyboard, inputting data and / or commands from a touch screen, outputting data to a display or outputting data to a USB flash drive. In some embodiments, the operating system (25) facilitates these tasks by acting as a central element to collect the various data and instructions required to execute the program and provide these to the microprocessor. In some embodiments, there may be no operating system present, and although the basic architecture of the target hardware device (10) will be similar to Figure 9 The same as described in , but all tasks are under the direct control of the processor (15). In some embodiments, multiple processors may be used in a parallel configuration to increase execution speed. In this case, the executable program may be customized specifically for parallel execution. In addition, in some embodiments, the processor (15) may execute Figures 1-8 Some of the embodiments of the present invention may be implemented in part, and some other parts may be implemented using dedicated hardware / firmware placed at input / output locations accessible by the target hardware (10) via the local interface (35). The target hardware (10) may include multiple executable programs (30), each of which can be run independently or in combination with each other.

[0057] A number of embodiments of the present disclosure have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the following claims.

[0058] This disclosure is directed to describing specific embodiments of some of the innovative aspects described herein, as well as examples of environments in which these innovative aspects can be implemented. However, the teachings herein can be applied in a variety of different ways. In addition, the described embodiments can be implemented in various hardware, software, firmware, etc. For example, various aspects of the present application can be implemented at least in part in an apparatus, a system including more than one device, a method, a computer program product, etc. Therefore, various aspects of the present application can take the form of hardware embodiments, software embodiments (including firmware, resident software, microcode, etc.) and / or embodiments combining software and hardware aspects. Such embodiments may be referred to herein as "circuits," "modules," "devices," "devices," or "engines." Some aspects of the present application can take the form of a computer program product embodied in one or more non-transitory media having computer-readable program code embodied thereon. Such non-transitory media may, for example, include a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. Thus, the teachings of the present disclosure are not intended to be limited to the embodiments shown in the figures and / or described herein, but rather have broad applicability.

Claims

1. A method for synthesizing a speech of a target style, comprising: receiving as input at least one waveform, each waveform corresponding to an utterance in the target style; extracting features of the at least one waveform to create at least one embedding vector; clustering the at least one embedding vector to generate at least one cluster, each cluster having a centroid; determining the centroid of a cluster in the at least one cluster; designating the centroid of the cluster as an initial embedding vector for a speech synthesizer; as well as The speech synthesizer is adapted based on at least the initial embedding vector to produce synthesized speech in the target style.

2. The method according to claim 1, further comprising: The at least one waveform is pre-processed to remove non-speech sounds and silence.

3. The method according to claim 1 or 2, wherein: Each cluster has a threshold distance from its centroid, and the adapting further comprises fine-tuning the at least one embedding vector within the threshold distance based on the target style.

4. The method according to claim 1 or 2, wherein: The speech synthesizer is a neural network.

5. The method according to claim 1 or 2, wherein: Extracting features of the at least one waveform further includes combining sample embedding vectors extracted from window samples of a waveform in the at least one waveform to generate an embedding vector for the waveform.

6. The method according to claim 5, wherein: The combining includes averaging the sample embedding vectors.

7. The method according to claim 1 or 2, wherein: The input is from a movie or video source.

8. The method according to claim 1 or 2, wherein: The target style includes the speaking style of the target person.

9. The method according to claim 8, wherein The target style further includes at least one of age, accent, emotion, and role playing.

10. The method according to claim 8, wherein The target person is an actor, and the target style is that the target person is at an age younger than the current age.

11. The method of claim 1 or 2, further comprising receiving as the input additional waveforms, each waveform corresponding to an utterance in a second style different from the target style; and extracting features of the additional waveform to create at least a second embedding vector; in, The clustering further includes clustering the second embedding vectors. 12 . The method of claim 11 , further comprising determining an expected number of clusters prior to the clustering, wherein the clustering is based on the expected number of clusters.

13. The method according to claim 12, wherein: The determining the expected number of clusters uses statistical analysis of the input.

14. A method for synthesizing speech in a target style, comprising: receiving as input at least one waveform, each waveform corresponding to an utterance in the target style; extracting features from the at least one waveform to create at least one embedding vector; calculating a vector distance for an embedding vector in the at least one embedding vector to determine an embedding vector distance to each of a plurality of known embedding vectors; determining a known embedding vector having a shortest distance to the embedding vector among the plurality of known embedding vectors; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; fine-tuning the adaptation based on the at least one embedding vector; as well as The target-style speech is synthesized using the adapted speech synthesizer.

15. A method for synthesizing speech in a target style, comprising: receiving as input at least one waveform, each waveform corresponding to an utterance in the target style; extracting features of the at least one waveform to create at least one embedding vector; applying a speech recognition system to an embedding vector of the at least one embedding vector to produce a known embedding vector corresponding to a corresponding speech identified by the speech recognition system as a best match to the embedding vector; designating the known embedding vector as an initial embedding vector for a speech synthesizer; adapting the speech synthesizer based on the initial embedding vector; as well as The target-style speech is synthesized using the adapted speech synthesizer.

16. The method according to claim 15, wherein The speech recognition system is a neural network.

17. The method of claim 15 or 16, further comprising updating a speech synthesizer table using the initial embedding vector.

18. A non-transitory computer-readable medium configured to execute the method according to any one of claims 1 to 17 on a computer.

19. An electronic device configured to perform the method according to any one of claims 1-17.

20. A computer program product comprising a computer program comprising executable instructions which, when executed by a computer having one or more processors, cause the computer to perform the method according to any one of claims 1 to 17.

Images