Unsupervised alignment for text-to-speech synthesis using neural networks

Abstract

This disclosure relates to unsupervised alignment of text-to-speech synthesis using neural networks. Generating synthesized speech from an input text sequence can be difficult when the duration of individual phonemes forming the input text sequence is unknown. A primarily parallel process can model speech rhythms as separate generation distributions so that phoneme durations can be sampled at inference time. Additional information such as pitch or energy can also be sampled to provide improved diversity for synthesized speech generation.

Description

Background Technology

[0001] Speech synthesis is typically modeled in an autoregressive manner, where statistical models are used to generate output speech based on an input text sequence. These models predict the lengths of different phonemes in the input text sequence, but a single poorly predicted audio frame can lead to additional errors in the entire synthesized speech sequence. Autoregressive models also scalable poorly, especially as sequence length increases. Furthermore, attempts to integrate autoregressive models into parallel architectures have introduced their own problems, such as audio-text alignment issues. Additionally, autoregressive models may lack diversity in synthesized speech results, where the input text sequence leads to similar outputs each time the model is executed, which may be undesirable in many applications. Attached Figure Description

[0002] Various embodiments of this disclosure will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 An example of a pipeline for speech synthesis according to at least one embodiment is shown;

[0004] Figure 2A An example of a training pipeline for speech synthesis according to at least one embodiment is shown;

[0005] Figure 2B An example of an inference pipeline for speech synthesis according to at least one embodiment is shown;

[0006] Figure 2C An example of an alignment architecture for speech synthesis according to at least one embodiment is shown;

[0007] Figure 3 An example of an alignment attention matrix according to at least one embodiment is shown;

[0008] Figure 4A An example phoneme distribution according to at least one embodiment is shown;

[0009] Figure 4B An example pitch distribution according to at least one embodiment is shown;

[0010] Figure 5 An example environment for speech synthesis according to at least one embodiment is shown;

[0011] Figure 6A An example flowchart of a process for generating synthetic speech according to at least one embodiment is shown;

[0012] Figure 6BAn example flowchart of a process for generating synthetic speech according to at least one embodiment is shown;

[0013] Figure 6C An example flowchart of a process for training a text-to-speech system according to at least one embodiment is shown;

[0014] Figure 7 An example data center system according to at least one embodiment is shown;

[0015] Figure 8 A computer system according to at least one embodiment is shown;

[0016] Figure 9 A computer system according to at least one embodiment is shown;

[0017] Figure 10 At least a portion of a graphics processor according to one or more embodiments is shown; and

[0018] Figure 11 At least a portion of a graphics processor according to one or more embodiments is shown. Detailed Implementation

[0019] Methods according to various embodiments provide systems and methods for end-to-end text-to-speech (TTS) models. In at least one embodiment, the TTS model is parallel or at least partially parallel. Various embodiments may include further modeling the speech rhythm as a sampled distribution to facilitate variable token durations during inference. In at least one embodiment, the systems and methods provide online extraction of speech-text alignment.

[0020] Various embodiments relate to systems and methods for TTS generation using primarily parallel end-to-end models. Embodiments include using probabilistically deterministic unsupervised alignment to align individual phonemes, which may have variable lengths, with their corresponding text. Generative models are used to sample distributions of reasonable phoneme durations, where boundaries can be set to eliminate consideration of unlikely or unrealistic durations. Furthermore, alternative generative models can be developed for the pitch and energy of different phonemes, which can be extracted during the training phase. Diversity can be achieved by sampling different locations from the same probability distribution. Systems and methods can also be optimized for training using distribution-aware data augmentation, where synthetic data can be labeled / classified during training and then ignored or not used during inference.

[0021] Various embodiments of this disclosure are designed to overcome problems associated with traditional alignment tools with TTS, where these tools attempt to extract alignment prior to inference or rely on attention mechanisms. Using a forced aligner can have limited functionality, for example, where the aligner is not applicable to a particular language or alphabet. Furthermore, the attention mechanisms of aligners have proven unstable and may fail to converge to a meaningful solution. The systems and methods of this disclosure address these and other problems by extending the attention-based mechanism to add a cigar-shaped prior distribution, which may be present in one or more embodiments. Additionally, the systems and methods can deploy various techniques associated with Hidden Markov Models (HMMs), such as forward sub-algorithms and Viterbi, to identify the most probable text string for a given signal.

[0022] The embodiments can also address issues related to insufficient or minimally relevant datasets used for training. Regarding TTS synthesis training pipelines, augmented data samples are rarely incorporated because these samples may be considered outside the distributions required for modeling, as it is undesirable in many cases to synthesize text that sounds similar to the augmented data samples. In at least one embodiment, systems and methods can incorporate data augmentation to improve the model's generalization without infiltrating the inference results. For example, augmented data can be labeled or otherwise identified in one or more generated distributions, where sampling during inference avoids these regions.

[0023] The systems and methods of this disclosure also address the diversity of synthesized speech. In at least one embodiment, the same text can be synthesized multiple times, with each result being plausible and distinct in quality. A generative model (normalized flow) can be used to sample plausible phoneme durations during inference, which can provide improved diversity, as phoneme duration is a significant variation in speech. The systems and methods may also involve pitch and energy modeling, thereby providing more options for the diversity of synthesized speech. In at least one embodiment, pitch and energy can be modeled, and probability components can be sampled from these models during the generation of output speech.

[0024] Various embodiments can also improve speaker modulation variables. For example, a traditional multi-speaker TTS model might assign a dimension vector to a specific speaker and then apply that vector when generating the output. However, using a single vector for each speaker can limit the model's ability to generate similar or different sound outputs without modeling individual differences among speakers. Embodiments can model the distribution of the generated vector as a function of the input and across speakers during inference.

[0025] Figure 1The diagram illustrates a text-to-speech (TTS) pipeline 100, also known as speech synthesis, which includes an input 102 that may correspond to text input. It should be understood that the input can be initial text input, such as user-provided input, converted text input, such as utterances that have been evaluated and then converted into text, text sequences extracted from input images or videos, and so on. In at least one embodiment, the input 102 can respond to user-provided questions or comments, such as conversational artificial intelligence (AI) systems providing answers in response to user queries, and other applications. The illustrated input 102 can be formatted for inclusion in a processing framework 104, which may include one or more trained machine learning systems to evaluate one or more features of the input 102, which can convert the input 102 into an audio output that simulates human speech.

[0026] In this example, the processing framework 104 includes a Natural Language Understanding (NLU) system 106, a prosody model 108, and a TTS module 110. As will be understood, the NLU system 106 can be used with one or more conversational AI systems to enable humans to interact naturally with the device. The NLU system 106 can be used to interpret the context and intent of the input 102 to generate a response. For example, the input 102 can be preprocessed, which may include tokenization, lemmatization, stemming, and other processes. Furthermore, the NLU system 106 may include one or more deep learning models, such as the BERT model, to enable features such as entity recognition, intent recognition, sentiment analysis, etc. Additionally, the NLU system 106 can convert the linguistic units of the input 102 into phonemes, which can then be combined using the prosody model 108.

[0027] In at least one embodiment, the TTS model 110 can acquire the text response generated by the NLU system 106 and transform it into speech that sounds natural. It should be understood that, in various embodiments, the prosodic model 108 can be part of the TTS model 110. The output from the NLU system 106 can undergo various processes associated with the TTS model 110, such as language analysis, synthesis, etc. Furthermore, parts of speech can be tagged. In various embodiments, the output can be further analyzed to improve pronunciation, calculate word duration, decipher the prosodic structure of the utterance, and understand grammatical information. Additionally, the text can be converted to a Mel spectrogram for output to a vocoder 112 to generate speech that sounds natural. As mentioned above, it should be understood that, in various embodiments, the vocoder 112 can be incorporated into the TTS model 110. Thus, an audio output 114 that sounds like human speech is generated.

[0028] Speech synthesis can be modeled sequentially in a fully autoregressive manner, where the training and perturbation rates cannot or ineffectively scale with increasing sequence length. Furthermore, errors within one or more audio frames can propagate to other parts of the synthesized speech. Parallelizing speech synthesis is desirable, but may encounter problems related to phoneme length. For example, it is difficult to know the individual lengths of phonemes without synthesizing them. Additionally, methods that might first determine the phonemes in the input text and then sample them from the Mel spectrogram miss the full end-to-end parallelism of the systems and methods of this disclosure. The systems and methods of this disclosure are applicable to end-to-end systems that include online alignment and implementations of generative models used for reasoning different results.

[0029] Existing alignment techniques are often insufficient or unusable in parallel architectures. For example, existing techniques may extract attention from autoregressive models to parallel architectures involving two-stage processes, which can be very costly to train. Furthermore, these techniques are often limited, requiring independent training for the individual languages ​​or letters used by the system. Additionally, these techniques often exhibit a loss of inference diversity, such as the variability of speech rhythm. Systems and methods can overcome these shortcomings by combining one or more generative models for token duration; these generative models can be standalone models.

[0030] Training pipeline 200, such as Figure 2A As shown, this can be used to build generative models for sampling mel-spectrograms given text input and speaker information. In this example, the mel-spectrogram 202 is provided with a speaker vector 204(ξ), which can encode speaker-specific features for augmentation 206. For example, mel-spectrograms can be used for audio clips of human speech, which can be represented as mel-spectrogram tensors. Where T is the number of Mel frames on the time axis, C mel This is the number of bands per frame dimension. The speaker vector 204 can include information about corresponding changes in the data distribution. In this example, enhancement 206 can be applied with a certain probability, which can change the Mel spectrogram 202 by corresponding changes to the speaker vector 204, and then the output is directed to the training pipeline 208. It should be understood that the Mel spectrogram 202 can also be directed to the training pipeline 208 along with the input text 210 without enhancement; the input text 210 can be represented as a tensor of an embedded text sequence, such as... As shown, where N is the length. The resulting output is a maximum likelihood estimate (MLE)212 on the latent space. This output may correspond to the optimized latent random variables of the Mel and duration (speech rate).

[0031] Inference pipeline 250, such asFigure 2B As shown, samples can be taken from MLE 212 to generate input samples 252. Random sampling of MLE 212 can achieve different inference results. In at least one embodiment, sample 252 is processed by inference pipeline 254 to output a second Mel spectrogram 256, which can then be processed to produce an output audio clip.

[0032] Various embodiments of this disclosure are capable of sampling Mel spectrogram frames and their durations at inference time while maintaining a parallel architecture for modeling. In operation, a temporal alignment is established between audio clips (e.g., Mel spectrogram 202) and text (e.g., input text 210). The alignment can be represented as... Therefore, the conditional distribution can be represented by equation (1).

[0033] P(X, A, F0, E|Φ, ξ)=P mel (X|Φ, ξ, A, F0, E)P dur (A|Φ,ξ)P pitch (F0|Φ,ξ)P energy (E|Φ,ξ) (1)

[0034] As previously mentioned, in the conditional representation, X represents the Mel spectrogram tensor, A represents alignment, F0 represents pitch, and E represents energy. Therefore, as will be described below, various embodiments can be implemented to sample a variety of different distributions, which may include duration, pitch, energy, and other speech properties not captured by duration, pitch, and energy, such as, but not limited to, intonation, stress, velocity, rhythm, etc.

[0035] It should be understood that, in various embodiments, one or more portions of the instructions (e.g., software instructions) used to execute at least a portion of equation (1) may be parallel. For example, P mel It can be parallelized. However, one or more other parts or components of equation (1) may include one or more autoregressive components, such as duration, pitch, energy, and text. Therefore, the different components of various embodiments can balance the expensive training steps.

[0036] In operation, a standardized process is applied to Mel encoding in TTS. The distribution can be modeled so that each time step in the distribution can be sampled from a simple distribution. In various embodiments, independent and identically distributed random variables are assumed. Therefore, the MLE 212 for data sample x can be expressed by equation (2).

[0037] log p x (x)=log p z (g -1(x))+log|det J(g -1 (x))|, (2)

[0038] Where p x (x) represents each mermaid frame P mel The unknown likelihood function of (), pz(z) denotes the Gaussian likelihood function, and J is the Jacobian matrix of the invertible transformation g such that z = g -1 (x).

[0039] In one or more embodiments, reasoning can be performed as represented by equation (3).

[0040] z~N(O,I)and x=g(z) (3)

[0041] During the reasoning process, the duration of each phoneme can be obtained using the phoneme stream, which may correspond to alignment A, used to prepare for pairing P. mel The input to the parallel Mel-decoder stream is modeled in (). As described herein, the decoder can sample latent vectors, such as those from MLE 212, and map them to Mel frames that sound plausible. Alignment between text and speech, independent of an external aligner, is a crucial process for providing an end-to-end TTS system. Various embodiments of this disclosure can utilize one or more affine coupling layers to segment the input data for inference scaling and translation parameters. In various embodiments, the input can be contextual conditions, which can be temporally correlated. The affine coupling layers can be used to generate one or more context matrices. This can be used in conjunction with speaker embedding vectors to provide a matrix with temporal alignment between textual information and Mel spectrogram frames, as well as speaker-related information. Recent work may utilize stream-based models to map Mel spectrogram frames to latent vectors. While this approach may yield stable results, the irreversible architecture often leads to attentional collapse onto trivial solutions, which may limit transferability to other models. Furthermore, as mentioned above, stream-based models do not account for the loss of inference diversity. Therefore, the systems and methods of this disclosure continue to illustrate improvements to existing models.

[0042] Alignment Architecture 270 in Figure 2CAs shown, it can form one or more parts of a machine learning system that can be used with embodiments of this disclosure. In this example, input text 210 and mel frame 272, which may at least partially correspond to mel spectrogram 202, are both encoded at encoder 274, which may include separate encoders for the mel frame and input text 210. For example, the text encoder may evaluate text tokens (e.g., phoneme embeddings). In at least one embodiment, each of input 210 and frame 272 is encoded with a 1D convolutional network with a limited receptive field to incorporate local context. Encoder 274 generates an output that serves as input to softmax function 276, which can generate a probability matrix 278 comprising one or more probability vectors for the text and mel frame, which can be used to generate one or more visualizations as described above herein.

[0043] Various embodiments enable unsupervised or partially unsupervised alignment learning. Alignments can be developed without relying on external aligners. In at least one embodiment, a combination of Viterbi and forward-backward algorithms used in Hidden Markov Models (HMMs) can be utilized to learn hard and soft alignments, respectively. As described above, alignment can be represented as... The different alignments can be specified as "hard" alignment or "soft" alignment. A soft The alignment between text Φ and Mel frames X of lengths N and T can be represented respectively, such that A soft Each column of A is normalized to a probability distribution. soft Processing is performed to extract the monotonic, binarized alignment matrix A. hard Thus, for each frame, the probabilistic quality is concentrated on a single symbol, and Generate a vector of the duration of each symbol.

[0044] Soft alignment can be based on learning pairwise affines between all text tokens φ∈Φ and mermaid frames x∈X, which can be normalized on the text dimension represented in equations (4) and (5) using softmax.

[0045]

[0046] A soft =softmax(-D, dim=0) (5)

[0047] Regarding equations (4) and (5), x enc and φ encThese are encoding variants of x and φ, each using 2 or 3 1D convolutional layers. In various embodiments, a loss module can be used to develop a monotonic sequence that begins and ends at the first and last text tokens, respectively, using each text token once, and advancing 0 or 1 text tokens for each advance of a mermaid frame.

[0048] Various embodiments can accelerate alignment learning by using priors that promote proximity to elements on the diagonal path. A β-binomial distribution can be used to encourage forward movement of the attention mechanism. This β-binomial distribution can be used in A soft A 2D cigar-shaped prior is constructed along the diagonal of the matrix, which widens at the center and narrows towards the corners. The prior can be weighted or scaled, where lower weights increase the width of the prior.

[0049] Because the alignments generated through duration prediction are inherently binary, the model can be conditional on a binary alignment matrix to avoid training-test domain gaps. This can be achieved using the Viterbi algorithm, while applying the same constraints to monotonic alignments, which can provide the most probable monotonic alignments from the distributions on monotonic paths. Furthermore, the results can be managed such that A is optimized by minimizing the loss. soft Match A as much as possible hard .

[0050] Figure 3 The diagram illustrates a soft alignment visualization 300, along with a β-binomial prior visualization 302, and soft alignment 304 and hard alignment 306 with the prior visualization. In these examples, the melodic frame is represented on the x-axis 308 and the text token on the y-axis 310. Soft alignment 300 shows various potential alignments over a period of time, with the most likely alignments shown as lighter shading near the corners (e.g., bottom left and top right), corresponding to the start and end times of the audio clip. The application of prior 312 is shown in visualization 302, where prior 312 has a cigar shape (e.g., a middle that is wider than the edges) and extends substantially along a diagonal line from the bottom left to the top right. As mentioned above, this configuration can impose alignment constraints on potentially aligned portions, such as the start (bottom left) and end (top right), thereby potentially improving accuracy. Prior 312 can apply boundaries to constrain sampling of the most probable portion of the distribution, as shown in visualization 304, where soft alignment is improved, illustrated by the lighter shading shown along the diagonal represented by prior 312. Visualization 306 illustrates an even more improved alignment that substantially conforms to the angles and sizes of prior 312. Therefore, synthesized speech can be generated using alignment developed with the prior to determine how to apply the duration of sampled phonemes.

[0051] Example phoneme-level duration distribution 400 in Figure 4A The following is illustrated. In this example, the phonemes constituting the word "climate" are shown. As shown, x-axis 402 represents the phonemes 404 that form the word "climate," corresponding in this case to "k," "l," "ayi," "m," "ahd," and "t." Each of these phonemes 404A-404F includes a corresponding set of distributions 406A-406F corresponding to duration, represented on y-axis 408. These distributions can be determined based on an assessment of a group of speakers uttering the word "climate" to determine how long each phoneme takes to generate. As shown, each of the phonemes 404A-404F may have slightly different durations, so attempting to force alignment between text and speech using a fixed duration will result in unnatural audio output. That is, the duration of "ayi" might be approximately twice that of "k," and a fixed duration for each phoneme would result in a pause or delay after "k," or a clipped or abrupt response to "ayi." As noted, various embodiments of this disclosure can achieve improved versatility by sampling different distributions of various phonemes. As an example, in this case, the phoneme 404A corresponding to "k" has three different distributions 406A (e.g., a set of distributions with slightly different durations). Therefore, during inference, one of the distributions can be selected, and within that distribution, the duration can be chosen. In another example, using a similar sound, different distributions and durations can be selected. In this way, by providing different phoneme durations during inference, or at least providing the possibility of selecting different durations, the diversity of synthesized speech can be increased.

[0052] Figure 4B The figure illustrates a pitch distribution 420, which can be further used to provide improved diversity in synthesized speech, as described herein. As shown, the x-axis 422 corresponds to time, and the y-axis 424 corresponds to relative pitch. In this example, distribution 420 can be computed by evaluating a set of sentences or words, for example, during one or more training phases. During inference, pitch can also be sampled from distribution 420 to provide further diversity in synthesized speech. That is, different attempts to synthesize the same word may result in different sets of results, each presenting a different pitch. When further combined... Figure 4A When choosing the phoneme duration, the same word can be synthesized to sound different, for example, from completely different speakers.

[0053] Synthetic Speech System 500 Figure 5As shown, it may include one or more components previously described herein. It should be understood that components may be grouped for illustrative purposes, but one or more systems may be integrated into or used with different components of system 500. Furthermore, one or more systems may utilize or otherwise share the architecture associated with one or more machine learning systems. Thus, different components may be described as separate modules or systems based on one or more functions, but may be part of a single integrated system. In this example, TTS system 502 may be used to generate synthesized speech based at least in part on text input 504 and speaker vector 506. In at least one embodiment, speaker vector 506 includes one or more speaker attributes, such as desired pitch, energy, intonation, stress, etc. In various embodiments, speaker vector 506 may include weighted attributes corresponding to various speech portions that can be applied to the synthesized speech. Furthermore, speaker vector 506 may be an adjustable component to achieve speech conversion, as well as other features. Additionally, in at least one embodiment, speaker vector 506 may be selected from a distribution of speaker vectors or a database. Furthermore, in at least one embodiment, the different components of the speaker vector 506 may be sampled or otherwise obtained from the distribution.

[0054] In this example, input text 504 and speaker vector 506 are provided as input to TTS system 502, where input text 504 can be evaluated to at least partially determine duration. For example, duration module 508 (e.g., duration system) can be used to sample one or more distributions to determine the corresponding duration of each phoneme forming input text 504. As previously described, training data can be used to generate phoneme distributions from which each phoneme forming input text 504 can be sampled to determine its respective length. In at least one embodiment, the length can be different each time a phoneme is evaluated, which increases the diversity of the output synthesized speech. The determination of duration can then be used during alignment, as discussed herein.

[0055] Further illustrations show pitch and energy modules 510, 512, which can be used to sample their respective distributions to obtain the pitch (e.g., fundamental frequency) and energy (e.g., amplitude) of the different phonemes forming the input text 504. In at least one embodiment, at least a portion of the pitch and energy can be determined at least partially by the speaker vector 506. For example, the speaker vector 506 can provide weights to influence the pitch and energy. In some embodiments, pitch and energy can be determined by the speaker vector 506, for example, in a directed attempt at speech conversion. In various embodiments, the distributions of duration, energy, and pitch can be stored in a distribution data repository 514, which can be accessed at inference time. As will be understood, the distributions can vary based on language or alphabet, and can be updated using one or more training processes. While duration, pitch, and energy can be described as components used for sampling, it should be understood that various other data points and distributions can be used for sampling. By way of example only, various prosodic features can also be used for sampling, such as emphasis, contrast, focus, or one or more elements not otherwise represented in the speaker's grammatical choices. In addition, additional factors in the evaluation may include tone, stress, speed, rhythm, pauses, etc.

[0056] In at least one embodiment, alignment between the input text and the output audio can be performed by one or more alignment modules 516, which may include various machine learning systems capable, in part, of recursively computing a set of probabilities to determine the most likely or highest probability alignment between the different phonemes forming the input text 504. The alignment probability between phonemes (from the text) and Mel samples (from the audio training data) is based on L2 distance, which can be interpreted as proportional to Gaussian likelihood, as described herein. In this example, the audio length module 520 may determine the audio length at least in part based on the duration of the input text 504 and / or the phonemes forming the input text. Furthermore, the text length module 522 may determine the text length, which can then be correlated with the audio length and presented as a matrix, which may be formed by the matrix generation module 524. In at least one embodiment, the matrix generation module 524 may be used to generate a matrix of potential phoneme durations regarding their positions, which the prior module 518 may then apply a prior to the matrix to limit or otherwise restrict the evaluation to the most likely positions. For example, the prior of cigar shape can be constructed from a β-binomial distribution, as shown in equation (6).

[0057] P(mel, text, alignment)=P(mel t text n |alignment)P(alignment), (6)

[0058] Where P(alignment) is the prior of the β-binomial cigar shape, and P(mel, text|alignment) is the L2 distance between the Mel sample at time step t and the nth text phoneme in the sequence. As described herein, the prior may include a boundary and is located on a diagonal line extending from the lower left to the upper right. It should be understood that the prior can be adjustable, allowing the use of different boundaries. Therefore, an alignment between the input text 504 and the audio duration can be generated.

[0059] In at least one embodiment, the audio generation module 526 can generate output audio based at least in part on alignment and input text 504, as well as other features such as speaker vector 506 and / or pitch, energy, etc. In various embodiments, audio generation may have improved diversity due to sampling from various distributions during inference, which may alter or modify different phoneme durations, potentially leading to different alignments and thus different output speech. Furthermore, sampling variations in pitch or energy can further improve output diversity.

[0060] Various embodiments may also include one or more training systems 528 that can generate synthetic training data for improving the TTS system 502. In various embodiments, the training system 528 may implement distribution-aware enhancement, where synthetic training information can be generated, used during training, and then deleted at inference time. For example, the synthetic speech generation 530 may obtain samples from a sample data repository 532 and modify one or more attributes of the speech, such as changing pitch or energy, or modifying various phoneme lengths, and other possibilities. The synthetic speech may be tagged or otherwise identified using a tagging module 534, which may append information such as metadata to the synthetic speech for later identification. This identification may be applied to all synthetic speech, where the synthetic speech may be considered “dirty” or “unclean” compared to actual ground-based training data. In other embodiments, one or more attributes may be evaluated to determine whether the data is considered clean or dirty, for example, identifying one or more features to compare against a threshold. The distribution generator 536 may then use the information from the synthetic speech to modify or generate one or more distributions. These generated distributions can then be used for training purposes, but during inference, synthetic speech can be identified, for example, using labels, and then removed. In this way, additional training data can be generated to improve the model, but during inference, only the data that forms ground truth information is used to generate synthetic speech in response to user input.

[0061] Figure 6AAn example process 600 for generating synthesized speech is illustrated. It should be understood that, for this process and other processes presented herein, unless specifically stated otherwise, additional, fewer, or alternative steps may be performed in a similar or alternative order or at least partially in parallel within the scope of various embodiments. In this example, multiple audio segments are received 602. The audio segments may form at least a portion of a training dataset that is evaluated to determine different aspects of different parts of speech. In various embodiments, the audio segments may undergo one or more preprocessing or processing steps to extract different parts of the audio segments, such as phonemes forming words, pitches, energy, etc. In at least one embodiment, phoneme duration, phoneme pitch, and phoneme energy are determined from the multiple audio segments 604. The information extracted from the multiple phonemes can then be used to generate one or more distributions indicating certain features found in the audio segments. For example, a first distribution 606 may be generated using phoneme duration, a second distribution 608 may be generated using phoneme pitch, and a third distribution 610 may be generated using phoneme energy. It should be understood that additional distributions may also be generated to capture other speech portions or features of the audio segments.

[0062] In various embodiments, distributions can be used to generate synthesized speech, for example, in relation to conversational AI. The system can receive text input represented as a text sequence and determine an alignment 612 between the text sequence and the audio length. The alignment can be at least partially based on a first distribution, which associates the different phonemes forming the text input with their respective durations. In various embodiments, the distributions of the individual phonemes are sampled to select phoneme durations. It should be understood that this type of probabilistic sampling can improve the diversity of the generated synthesized speech because, by sampling during inference, the same text input can present different output audio, for example, due to differences in phoneme durations and other factors. Therefore, synthesized speech can be generated at least partially based on alignment, a second distribution, and a third distribution 614. Furthermore, as described above, in various embodiments, one or more additional distributions can also be utilized to generate synthesized speech.

[0063] Figure 6BAn example process 620 for generating synthesized speech is illustrated. In this example, corresponding alignments 622 are determined between the text of multiple audio segments and the durations of the multiple audio segments. For example, the multiple audio segments may correspond to training information determined to identify phoneme length distributions of different words or phrases. An alignment distribution may be generated, which may be at least partially based on the corresponding alignments 624. The alignment distribution may be presented in the form of a matrix illustrating the probability that a certain phoneme will align with a portion of a text sample. In at least one embodiment, one or more vectors from the alignment distribution are determined 626. One or more vectors may correspond to one or more speaker features.

[0064] As noted, various embodiments can be used to generate synthetic speech, wherein a text sequence 628 is received. The text sequence may be input by a user or extracted from an image, among other options. Synthetic audio clips may be generated at least in part based on the text sequence and one or more vectors, wherein the synthetic audio clips correspond to text sequence 630.

[0065] Figure 6C An example process 650 for training a TTS system is illustrated. In this example, one or more synthetic training clips 652 are generated. The synthetic training clips can be generated at least in part based on one or more sample audio segments, which may correspond to ground-based training data provided for training the TTS system, among other options. In at least one embodiment, the synthetic training clips modify one or more attributes of the sample audio segments, such as changing the pitch or speed of the speech. The synthetic training clips can be tagged 654, for example, by identifying their location within a distribution or by associating metadata with the clips. One or more machine learning systems can then be trained using at least some of the sample audio segments and at least some of one or more synthetic training clips 656. The synthetic training clips can enable a larger training set, which can improve subsequent inference.

[0066] In at least one embodiment, a trained machine learning system is used to generate synthetic audio clips. A request 658 to generate synthetic speech can be received. This speech can be generated by processing text input using a TTS system that can align portions of the speech with different durations to generate output audio. In at least one embodiment, one or more locations 660 within a distribution associated with the synthetic training clip are identified, and these locations are avoided or not sampled from during the generation of the synthetic speech 662. In this way, the synthetic training clip can be used to improve modeling by increasing the dataset, but may not be used during inference.

[0067] Data Center

[0068] Figure 7 An example data center 700 that can be used with at least one embodiment is shown. In at least one embodiment, the data center 700 includes a data center infrastructure layer 710, a framework layer 720, a software layer 730, and an application layer 740.

[0069] In at least one embodiment, such as Figure 7 As shown, the data center infrastructure layer 710 may include a resource coordinator 712, grouped computing resources 714, and node computing resources (“nodes CR”) 716(1)-716(N), where “N” represents any positive integer. In at least one embodiment, nodes CR 716(1)-716(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (“NWI / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR 716(1)-716(N) may be servers having one or more of the aforementioned computing resources.

[0070] In at least one embodiment, the grouped computing resources 714 may include individual groups (not shown) of node CRs housed in one or more racks, or a plurality of racks (also not shown) housed in data centers in various geographic locations. The individual groups of node CRs within the grouped computing resources 714 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches, in any combination.

[0071] In at least one embodiment, resource coordinator 712 may configure or otherwise control one or more nodes CR716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, resource coordinator 712 may include a Software Design Infrastructure (“SDI”) management entity for data center 700. In at least one embodiment, resource coordinator may include hardware, software, or some combination thereof.

[0072] In at least one embodiment, such asFigure 7 As shown, framework layer 720 includes a job scheduler 722, a configuration manager 724, a resource manager 726, and a distributed file system 728. In at least one embodiment, framework layer 720 may include a framework of software 732 supporting software layer 730 and / or one or more applications 742 supporting application layer 740. In at least one embodiment, software 732 or application 742 may respectively include web-based service software or applications, such as services or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 720 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 728 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 722 may include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 700. In at least one embodiment, the configuration manager 724 may be able to configure different layers, such as the software layer 730 and the framework layer 720, which includes Spark and a distributed file system 728 for supporting large-scale data processing. In at least one embodiment, the resource manager 726 is able to manage cluster or group computing resources mapped to or allocated to support the distributed file system 728 and the job scheduler 722. In at least one embodiment, the cluster or group computing resources may include group computing resources 714 on the data center infrastructure layer 710. In at least one embodiment, the resource manager 726 may coordinate with the resource coordinator 712 to manage these mapped or allocated computing resources.

[0073] In at least one embodiment, the software 732 included in the software layer 730 may include software used by at least a portion of the nodes CR716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 728 of the framework layer 720. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0074] In at least one embodiment, the application layer 740 may include one or more applications 742 that can be used by at least a portion of nodes CR716(1)-716(N), grouped computing resources 714, and / or the distributed file system 728 of the framework layer 720. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0075] In at least one embodiment, any of the configuration manager 724, resource manager 726, and resource coordinator 712 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modification actions can mitigate potentially poor configuration decisions by data center operators of data center 700 and can prevent underutilization and / or poor performance of the data center.

[0076] In at least one embodiment, data center 700 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 700. In at least one embodiment, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to data center 700 by using weight parameters calculated through one or more training techniques described herein.

[0077] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to utilize the aforementioned resources to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0078] Such components can be used to generate synthesized speech.

[0079] Computer System

[0080] Figure 8This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system of interconnected devices and components, a system-on-a-chip (SOC), or some combination thereof 800 formed with a processor, which may include an execution unit to execute instructions. In at least one embodiment, according to this disclosure, such as the embodiments described herein, the computer system 800 may include, but is not limited to, components such as processor 802, whose execution unit includes logic to execute algorithms for process data. In at least one embodiment, the computer system 800 may include a processor, such as one available from Intel Corporation of Santa Clara, California. Processor family, Xeon TM , XScale TM and / or StrongARM TM , Core TM or Nervana TM A microprocessor may be used, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors may also be used. In at least one embodiment, the computer system 800 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0081] The embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (IP) devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip (SoC), a network computer (“NetPC”), an edge computing device, a set-top box, a network hub, a wide area network (“WAN”) switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0082] In at least one embodiment, the computer system 800 may include, but is not limited to, a processor 802, which may include, but is not limited to, one or more execution units 908, to perform machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 800 is a single-processor desktop or server system, but in another embodiment, the computer system 800 may be a multiprocessor system. In at least one embodiment, the processor 802 may include, but is not limited to, a Complex Instruction Set Computer (“CISC”) microprocessor, a Reduced Instruction Set Computing (“RISC”) microprocessor, a Very Long Instruction Word (“VLIW”) microprocessor, a processor implementing instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 802 may be coupled to a processor bus 810, which can transmit data signals between the processor 802 and other components in the computer system 800.

[0083] In at least one embodiment, processor 802 may include, but is not limited to, a Level 1 (“L1”) internal cache memory (“cache”) 804. In at least one embodiment, processor 802 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may reside external to processor 802. Depending on specific implementation and requirements, other embodiments may also include a combination of internal and external caches. In at least one embodiment, register file 806 may store different types of data in various registers, including but not limited to integer registers, floating-point registers, status registers, and instruction pointer registers.

[0084] In at least one embodiment, a logic execution unit 808, including but not limited to performing integer and floating-point operations, is also located within the processor 802. In at least one embodiment, the processor 802 may further include a microcode (“ucode”) read-only memory (“ROM”) for storing microcode of certain macro instructions. In at least one embodiment, the execution unit 808 may include logic for processing a packaged instruction set 809. In at least one embodiment, by including the packaged instruction set 809 in the instruction set of the general-purpose processor 802, along with the associated circuitry for executing the instructions, the packaged data in the processor 802 can be used to perform operations used by many multimedia applications. In one or more embodiments, many multimedia applications can be executed more quickly and efficiently by using the full width of the processor's data bus to perform operations on the packaged data, which may eliminate the need to transfer smaller data units on the processor's data bus to perform one or more operations on one data element at a time.

[0085] In at least one embodiment, execution unit 808 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuitry. In at least one embodiment, computer system 800 may include, but is not limited to, memory 820. In at least one embodiment, memory 820 may be implemented as a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or other storage device. In at least one embodiment, memory 820 may store instructions 819 and / or data 821 represented by data signals that can be executed by processor 802.

[0086] In at least one embodiment, the system logic chip may be coupled to a processor bus 810 and a memory 820. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 816, and the processor 802 may communicate with the MCH 816 via the processor bus 810. In at least one embodiment, the MCH 816 may provide a high-bandwidth memory path 818 to the memory 820 for instruction and data storage, as well as for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 816 may initiate data signals between the processor 802, the memory 820, and other components in the computer system 800, and bridge data signals between the processor bus 810, the memory 820, and system I / O 822. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 816 can be coupled to memory 820 via high-bandwidth memory path 818, and graphics / video card 812 can be coupled to MCH 816 via Accelerated Graphics Port (“AGP”) interconnect 814.

[0087] In at least one embodiment, the computer system 800 may use system I / O 822, which is a proprietary hub interface bus, to couple MCH 816 to I / O controller hub (“ICH”) 830. In at least one embodiment, ICH 830 may provide direct connectivity to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to memory 820, chipset, and processor 802. Examples may include, but are not limited to, an audio controller 829, a firmware hub (“Flash BIOS”) 828, a wireless transceiver 826, a data storage 824, a conventional I / O controller 823 including a user input and keyboard interface, a serial expansion port 827 (e.g., a Universal Serial Bus (USB) port), and a network controller 834. Data storage 824 may include a hard disk drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

[0088] In at least one embodiment, Figure 8 A system including interconnected hardware devices or "chips" is shown, while in other embodiments, Figure 8 An exemplary system-on-a-chip (SoC) may be illustrated. In at least one embodiment, the device may be interconnected with a proprietary interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of the computer system 800 are interconnected using a compute fast link (CXL) interconnect.

[0089] Such components can be used to generate synthesized speech.

[0090] Figure 9 This is a block diagram illustrating an electronic device 900 for utilizing a processor 910 according to at least one embodiment. In at least one embodiment, the electronic device 900 may be, for example, but not limited to, a laptop computer, tower server, rack server, blade server, laptop computer, desktop computer, tablet computer, mobile device, telephone, embedded computer, or any other suitable electronic device.

[0091] In at least one embodiment, system 900 may include, but is not limited to, processor 910 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, processor 910 uses a bus or interface coupling, such as an I2C bus, system management bus (“SMBus”), low pin count (LPC) bus, serial peripheral interface (“SPI”), high-definition audio (“HDA”) bus, serial advanced technology accessory (“SATA”) bus, universal serial bus (“USB”) (versions 1, 2, and 3), or universal asynchronous receiver / transmitter (“UART”) bus. In at least one embodiment, Figure 9 The system shown includes interconnected hardware devices or "chips," while in other embodiments, Figure 9 An exemplary system-on-a-chip (SoC) can be illustrated. In at least one embodiment, Figure 9 The device shown can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, Figure 9 One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0092] In at least one embodiment, Figure 9 This may include a display 924, a touchscreen 925, a touchpad 930, a near-field communication unit (“NFC”) 945, a sensor hub 940, a thermal sensor 946, a fast chipset (“EC”) 935, a trusted platform module (“TPM”) 938, a BIOS / firmware / flash (“BIOS, FWFlash”) 922, a DSP 960, a drive 920 (e.g., a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 950, a Bluetooth unit 952, a wireless wide area network unit (“WWAN”) 956, a global positioning system (GPS) 955, a camera (“USB 3.0 camera”) 954 (e.g., a USB 3.0 camera), and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 915 implemented in, for example, the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0093] In at least one embodiment, other components may be communicatively coupled to processor 910 via the components described above. In at least one embodiment, accelerometer 941, ambient light sensor (“ALS”) 942, compass 943, and gyroscope 944 may be communicatively coupled to sensor hub 1040. In at least one embodiment, thermal sensor 939, fan 937, keyboard 936, and touchpad 930 may be communicatively coupled to EC 935. In at least one embodiment, speaker 963, earphone 964, and microphone (“mic”) 965 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 962, which in turn may be communicatively coupled to DSP 960. In at least one embodiment, audio unit 962 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 957 may be communicatively coupled to WWAN unit 956. In at least one embodiment, components such as WLAN unit 950, Bluetooth unit 952, and WWAN unit 956 can be implemented as next-generation form factor (NGFF).

[0094] Such components can be used to generate synthesized speech.

[0095] Figure 10 This is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1000 includes one or more processors 1002 and one or more graphics processors 1008, and may be a single-processor desktop system, a multi-processor workstation system, or a server system or data center having a large number of collectively or individually managed processors 1002 or processor cores 1007. In at least one embodiment, system 1000 is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0096] In at least one embodiment, system 1000 may include or be integrated into a server-based gaming platform, cloud computing hosting platform, virtualization computing platform, or game console, including a game and media console, mobile game console, handheld game console, or online game console. In at least one embodiment, system 1000 is a mobile phone, smartphone, tablet computing device, or mobile internet device. In at least one embodiment, processing system 1000 may also include components coupled to or integrated into a wearable device, such as a smartwatch wearable device, smart glasses device, augmented reality device, edge device, Internet of Things (“IoT”) device, or virtual reality device. In at least one embodiment, processing system 1000 is a television or set-top box device having one or more processors 1002 and a graphical interface generated by one or more graphics processors 1008.

[0097] In at least one embodiment, each of the one or more processors 1002 includes one or more processor cores 1007 for processing instructions that, when executed, perform operations against the system and user software. In at least one embodiment, each of the one or more processor cores 1007 is configured to process a specific instruction set 1009. In at least one embodiment, the instruction set 1009 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). In at least one embodiment, each processor core 1007 may process a different instruction set 1009, which may include instructions that facilitate the emulation of other instruction sets. In at least one embodiment, the processor core 1007 may also include other processing devices, such as a digital signal processor (DSP).

[0098] In at least one embodiment, processor 1002 includes cache memory 1004. In at least one embodiment, processor 1002 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among various components of processor 1002. In at least one embodiment, processor 1002 also uses an external cache (e.g., a Level 3 (L3) cache or a last-level cache (LLC)) (not shown), which can be shared among processor cores 1007 using known cache coherence techniques. In at least one embodiment, processor 1002 further includes a register file 1006, which may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, register file 1006 may include general-purpose registers or other registers.

[0099] In at least one embodiment, one or more processors 1002 are coupled to one or more interface buses 1010 to transmit communication signals, such as address, data, or control signals, between the processor 1002 and other components in the system 1000. In at least one embodiment, the interface bus 1010 may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus 1010 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor 1002 includes an integrated memory controller 1016 and a platform controller hub 1030. In at least one embodiment, the memory controller 1016 facilitates communication between memory devices and other components of the processing system 1000, while the platform controller hub (PCH) 1030 provides connectivity to I / O devices via a local I / O bus.

[0100] In at least one embodiment, memory device 1020 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or a device with suitable performance for use as processor memory. In at least one embodiment, memory device 1020 may be used as system memory of processing system 1000 to store data 1022 and instructions 1021 for use when one or more processors 1002 execute an application or process. In at least one embodiment, memory controller 1016 is also coupled to an optional external graphics processor 1012, which may communicate with one or more graphics processors 1008 of processor 1002 to perform graphics and media operations. In at least one embodiment, display device 1011 may be connected to processor 1002. In at least one embodiment, display device 1011 may include one or more internal display devices, such as in mobile electronic devices or laptop devices, or external display devices connected via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 1011 may include a head-mounted display (HMD), such as a stereoscopic display device for virtual reality (VR) or augmented reality (AR) applications.

[0101] In at least one embodiment, the platform controller hub 1030 enables peripheral devices to connect to the storage device 1020 and the processor 1002 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 1046, a network controller 1034, a firmware interface 1028, a wireless transceiver 1026, a touch sensor 1025, and a data storage device 1024 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1024 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCIe). In at least one embodiment, the touch sensor 1025 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1026 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or LTE transceiver. In at least one embodiment, the firmware interface 1028 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, network controller 1034 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to interface bus 1010. In at least one embodiment, audio controller 1046 is a multi-channel high-definition audio controller. In at least one embodiment, processing system 1000 includes an optional legacy I / O controller 1040 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to system 1000. In at least one embodiment, platform controller hub 1030 may also be connected to one or more Universal Serial Bus (USB) controllers 1042 that connect input devices, such as a keyboard and mouse combination 1043, a camera 1044, or other USB input devices.

[0102] In at least one embodiment, instances of the memory controller 1016 and platform controller hub 1030 may be integrated into a discrete external graphics processor, such as external graphics processor 1012. In at least one embodiment, the platform controller hub 1030 and / or the memory controller 1016 may be external to one or more processors 1002. For example, in at least one embodiment, system 1000 may include external memory controller 1016 and platform controller hub 1030, which may be configured as a memory controller hub and peripheral controller hub in a system chipset communicating with processor 1002.

[0103] Such components can be used to generate synthesized speech.

[0104] Figure 11 This is a block diagram of a processor 1100 having one or more processor cores 1102A-1102N, an integrated memory controller 1114, and an integrated graphics processor 1108 according to at least one embodiment. In at least one embodiment, the processor 1100 may include additional cores, up to and including additional cores 1102N indicated by dashed boxes. In at least one embodiment, each processor core 1102A-1102N includes one or more internal cache units 1104A-1104N. In at least one embodiment, each processor core may also access one or more shared cache units 1106.

[0105] In at least one embodiment, internal cache units 1104A-1104N and shared cache unit 1106 represent a cache memory hierarchy within processor 1100. In at least one embodiment, cache memory units 1104A-1104N may include at least one level of instruction and data cache within each processor core and one or more levels of cache in a shared intermediate cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, wherein the highest level of cache preceding external memory is classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 1106 and 1104A-1104N.

[0106] In at least one embodiment, the processor 1100 may further include a set of one or more bus controller units 1116 and a system agent core 1110. In at least one embodiment, the one or more bus controller units 1116 manage a set of peripheral buses, such as one or more PCI or PCIe buses. In at least one embodiment, the system agent core 1110 provides management functions for various processor components. In at least one embodiment, the system agent core 1110 includes one or more integrated memory controllers 1114 to manage access to various external memory devices (not shown).

[0107] In at least one embodiment, one or more processor cores 1102A-1102N include support for multi-threaded concurrent processing. In at least one embodiment, system agent core 1110 includes components for coordinating and operating cores 1102A-1102N during multi-threaded processing. In at least one embodiment, system agent core 1110 may additionally include a power control unit (PCU) including logic and components for regulating one or more power states of processor cores 1102A-1102N and graphics processor 1108.

[0108] In at least one embodiment, processor 1100 further includes a graphics processor 1108 for performing graph processing operations. In at least one embodiment, graphics processor 1108 is coupled to a shared cache unit 1106 and a system proxy core 1110 including one or more integrated memory controllers 1114. In at least one embodiment, system proxy core 1110 further includes a display controller 1111 for driving graphics processor outputs to one or more coupled displays. In at least one embodiment, display controller 1111 may also be a separate module coupled to graphics processor 1108 via at least one interconnect, or it may be integrated within graphics processor 1108.

[0109] In at least one embodiment, the ring-based interconnect unit 1112 is used to couple internal components of the processor 1100. In at least one embodiment, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies, may be used. In at least one embodiment, the graphics processor 1108 is coupled to the ring interconnect 1112 via I / O link 1113.

[0110] In at least one embodiment, I / O link 1113 represents at least one of a variety of I / O interconnects, including packaged I / O interconnects that facilitate communication between various processor components and high-performance embedded memory modules 1118 (e.g., eDRAM modules). In at least one embodiment, each of the processor cores 1102A-1102N and the graphics processor 1108 uses the embedded memory module 1118 as a shared last-level cache.

[0111] In at least one embodiment, processor cores 1102A-1102N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 1102A-1102N are heterogeneous in terms of instruction set architecture (ISA), with one or more processor cores 1102A-1102N executing a common instruction set, while one or more other processor cores 1102A-1102N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 1102A-1102N are heterogeneous in terms of microarchitecture, with one or more cores having relatively high power consumption coupled to one or more power cores having lower power consumption. In at least one embodiment, processor 1100 may be implemented on one or more chips or implemented as a SoC integrated circuit.

[0112] Such components can be used to generate synthesized speech.

[0113] The embodiments of this disclosure may be implemented in light of the following provisions.

[0114] 1. A computer-implemented method, comprising:

[0115] Alignment between phonemes and phoneme durations is determined from multiple audio samples, including human speech.

[0116] Generate an alignment matrix corresponding to the distribution from the alignment;

[0117] Receive input text sequence;

[0118] Sample the phonemes and phoneme durations of corresponding portions of the input text sequence from the alignment matrix; and

[0119] The generated audio clips are inferred at least in part based on the samples.

[0120] 2. The computer-implemented method as described in Clause 1 further includes:

[0121] Generate synthesized training audio samples;

[0122] Provide identifiers for the synthesized training audio samples;

[0123] At least in part, the generated synthetic training audio samples are used to train one or more machine learning systems.

[0124] 3. The computer-implemented method as described in Clause 2, wherein the inference is derived from an unenhanced distribution of training audio samples without the synthesized audio.

[0125] 4. The computer-implemented method as described in Clause 1 further includes:

[0126] The L2 distance between the text and the Mel code pair is determined, and the alignment matrix is ​​based at least in part on the corresponding L2 distance.

[0127] 5. The computer-implemented method as described in Clause 1, wherein the alignment is performed as part of a normalized flow framework without an external aligner.

[0128] 6. The computer-implemented method as described in Clause 1 further includes:

[0129] Use an evolved set of alignments to train the text-to-speech system.

[0130] 7. The computer-implemented method as described in Clause 6, wherein the evolved alignment set includes one or more training alignments introduced at different stages of the training sequence.

[0131] 8. The computer-implemented method as described in Clause 1, wherein the phoneme duration is determined during inference.

[0132] 9. The computer-implemented method as described in Clause 1 further includes:

[0133] Enhance at least a portion of the plurality of audio samples.

[0134] 10. A method comprising:

[0135] The first audio samples, including human speech, are processed using a first trained neural network.

[0136] The text sequence is processed in parallel using a trained second neural network;

[0137] The alignment between the text component of the first audio sample and the duration of the text component is determined at least in part based on the first audio sample;

[0138] A second audio sample corresponding to the text sequence is generated, at least in part based on the alignment.

[0139] 11. The method as described in Clause 10 further includes:

[0140] Determine one or more distributions corresponding to pitch and energy from the first audio sample.

[0141] 12. The method as described in Clause 10 further includes:

[0142] Sampling is performed from at least one of the one or more distributions.

[0143] 13. The method as described in Clause 10 further includes:

[0144] Enhance the first audio sample to create an enhanced audio sample;

[0145] The first neural network is trained at least in part based on the enhanced audio samples.

[0146] 14. The method as described in Clause 13, wherein the portion of the distribution corresponding to the enhanced audio sample is not selected during inference.

[0147] 15. The method described in Clause 14 further includes:

[0148] The enhanced audio samples are identified within the distribution corresponding to the alignment.

[0149] 16. A processor, comprising:

[0150] One or more processing units are used for:

[0151] Generate one or more synthetic training clips from multiple sample audio segments;

[0152] Use at least a portion of one or more synthetic training clips to train one or more text-to-speech machine learning systems;

[0153] Identify one or more locations within the distribution corresponding to the one or more synthetic training clips; and

[0154] During inference using a text-to-speech machine learning system, sampling is performed from one or more regions of the distribution that do not have one or more synthetic training clips.

[0155] 17. The processor as described in Clause 16, wherein the text-to-speech machine learning system forms at least a portion of an end-to-end parallel speech synthesis system.

[0156] 18. The processor as described in Clause 16, wherein the one or more processing units are further configured to enhance one or more features of the plurality of sample audio segments at least in part based on the enhancement probability.

[0157] 19. The processor as described in Clause 18, wherein one or more features correspond to at least one of tone or energy.

[0158] 20. The processor as described in Clause 16, wherein the one or more processing units are further configured to use the trained text-to-speech system to generate synthesized speech at least in part based on the input text sequence.

[0159] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0160] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar designations, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. Unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term "subset" of a corresponding set does not necessarily refer to an appropriate subset of the corresponding set, but rather the subset and the corresponding set can be equal.

[0161] Unless explicitly stated otherwise or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). The number of items in multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0162] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that are executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues). In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions, while the graphics processing unit (“GPU”) and / or data processing unit (“DPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0163] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0164] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0165] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0166] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0167] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “operation,” “calculation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0168] In a similar manner, the term "processor" can refer to any device or part of memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, "processor" can be any processor capable of general-purpose processing, such as a CPU, GPU, or DPU. As a non-limiting example, "processor" can be any microcontroller or special-purpose processing unit, such as a DSP, image signal processor ("ISP"), arithmetic logic unit ("ALU"), vision processing unit ("VPU"), tree traversal unit ("TTU"), ray tracing core, tensor tracing core, tensor processing unit ("TPU"), embedded control unit ("ECU"), etc. As a non-limiting example, "processor" can be a hardware accelerator, such as a PVA (programmable vision accelerator), DLA (deep learning accelerator), etc. As a non-limiting example, "processor" can also include one or more virtual instances of a CPU, GPU, etc., hosted on underlying hardware components executing one or more virtual machines. A "computing platform" can include one or more processors. As used herein, a “software” process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Similarly, each process can refer to multiple processes that execute instructions sequentially or intermittently, or in parallel. The terms “system” and “method” are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0169] This document refers to the process of acquiring, obtaining, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. Analog and digital data can be acquired, obtained, received, or input in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface (API). In some implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In another implementation, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference can also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an API, or an inter-process communication mechanism.

[0170] While the discussion above illustrates example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for discussion purposes, various functions and responsibilities can be assigned and divided in different ways depending on the circumstances.

[0171] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A computer-implemented method comprising: determining, from a plurality of audio segments, respective phoneme durations, phoneme pitches, and phoneme energies; determining a first alignment between a sequence of text and a total speech duration, the first alignment corresponding to possible positions of the respective phoneme durations; determining, based at least in part on a first distribution corresponding to the phoneme durations and the first alignment, a second alignment of an audio segment of synthesized speech; and generating, based at least in part on the second alignment and at least one of a second distribution corresponding to the phoneme pitches or a third distribution corresponding to the phoneme energies, an audio segment comprising a synthesized reading of the sequence of text for the sequence of text.

2. The computer-implemented method of claim 1, further comprising: generating a fourth distribution corresponding to one or more attributes, the one or more attributes being associated with the synthesized reading based at least in part on the fourth distribution.

3. The computer-implemented method of claim 1, further comprising: applying a prior distribution to the second alignment to exclude phonemes and duration pairs outside of a specified range from the plurality of audio segments.

4. The computer-implemented method of claim 3, wherein the prior distribution is cigar-shaped.

5. The computer-implemented method of claim 3, wherein the prior distribution is constructed from a beta-binomial distribution.

6. The computer-implemented method of claim 1, further comprising: determining a plurality of text tokens from the sequence of text; and aligning each of the plurality of text tokens with a respective mel frame based at least in part on the second alignment.

7. The computer-implemented method of claim 6, wherein the second alignment is based at least in part on an L2 distance between a mel frame at a first time and a text phoneme in the sequence of text.

8. The computer-implemented method of claim 1, wherein the synthesized reading is generated such that a first synthesized reading is different from a second synthesized reading, each of the first synthesized reading and the second synthesized reading being based on the sequence of text.

9. A method comprising: determining, from a plurality of audio samples comprising human speech, an alignment between a text of the plurality of audio samples, a duration of the plurality of audio samples, and at least one of a pitch of the audio samples or an energy of the audio samples; generating, based at least in part on the alignment, an alignment distribution; determining a soft alignment between a first sequence of text from the text and mel frames of the alignment distribution, the soft alignment normalizing a probability distribution for alignment across a duration of the plurality of audio samples; and determining a hard alignment between the first sequence of text and the mel frames of the alignment distribution, the hard alignment concentrating the probability distribution from the soft alignment onto a symbol for alignment across the duration of the plurality of audio samples; determining, based on the hard alignment, one or more vectors corresponding to one or more speaker characteristics; receiving a second sequence of text; and generating a synthetic audio clip corresponding to the second text sequence based at least in part on the second text sequence and the one or more vectors.

10. The method of claim 9, wherein the alignment is based at least in part on an alignment matrix having a beta-binomial distribution.

11. The method of claim 9, wherein an encoder and a decoder used to generate the synthetic audio clip operate in parallel.

12. The method of claim 9, further comprising: generating a second synthetic audio clip from the second text sequence, the second synthetic audio clip being different from the synthetic audio clip.

13. The method of claim 9, further comprising: generating at least one of a phone distribution, a pitch distribution, or an energy distribution; and sampling from at least one of the phone distribution, the pitch distribution, or the energy distribution.

14. A processor comprising: one or more processing units to: receive an audio clip representing human speech as a mel-spectrogram; determine an alignment matrix for the audio clip normalized as a probability distribution; apply a prior distribution to the alignment matrix to exclude pairs of phones and mel- frames in the audio clip from the alignment matrix; determine, from the alignment matrix, an alignment of a first text sequence within the audio clip and a plurality of mel-frames representing a duration of the audio clip; determine one or more distributions of pitch and energy associated with the audio clip; receive a second text sequence; and generate a second audio clip of the second text sequence based at least in part on the alignment and the one or more distributions.

15. The processor of claim 14, wherein the one or more processing units are further to implement an encoder and a decoder for generating the second audio clip, wherein the encoder and the decoder operate in parallel.

16. The processor of claim 14, wherein the one or more processing units are further to: fit a beta-binomial distribution on the alignment matrix.

17. The processor of claim 16, wherein the beta-binomial distribution excludes pairs of phones and mel-frames outside of a specified range.

18. The processor of claim 14, wherein the one or more processing units are further to generate a third audio clip from the second text sequence, the second audio clip being different from the third audio clip. ​

Images