Cross-lingual prosodic voice cloning in multiple languages

Abstract

A system and method for cross-lingual prosodic voice cloning in multiple languages. Cross-lingual prosodic voice cloning is the process of cloning the prosodic features of a source speaker's voice and transferring those prosodic features to the utterances of a target speaker in different languages. This involves capturing the rhythm, intonation, and other aspects of the source speaker's prosody and applying them to the synthesized voice of the target speaker. The method collects data from the speaker or language. Prosodic features, such as pitch envelope, rhythm, and intensity, are extracted from the source speaker's recording. Voice cloning generates a baseline voice for the target speaker. The prosodic features extracted from the source speaker are applied to the synthesized voice of the target speaker. Tacotron performs end-to-end speech synthesis and generates a spectrogram based on the input. A HIFI GAN vocoder generates high-quality audio waveforms and natural-sounding speech from the spectrogram.

Description

Technical Field

[0001] This application claims the benefit of Indian Patent Application No. 202311014863, filed on May 6, 2023, entitled "A SYSTEM FOR CROSS-LINGUAL PROSODIC VOICE AND MUSIC CLONING IN PLURALITY OF LANGUAGES", the content of which is incorporated herein by reference.

[0002] The present invention generally relates to methods and systems for cross-lingual cloning using prosody control, and more particularly, to methods and systems for integrating model training and model inference to obtain a new model using state-of-the-art infrastructure.

Background Art

[0003] In addition to being able to generate understandable and natural-sounding voices, a text-to-speech (TTS) system should be able to generate voices in a number of voices, styles, and ideally multiple languages. A multilingual text-to-speech system is a system that can process text input in multiple languages and generate voices from that text. There are several applications for a multilingual TTS system. In addition to being used within speech-to-speech language conversion systems and within interactive language tutoring systems, this functionality is necessary in order for a TTS system to be able to insert words in a foreign language that occasionally appear into a single-language voice otherwise, or even to switch languages in a manner that conforms to the syntax and phonology of each language, called code mixing.

[0004] Since the early 2000s, efforts have been made to generate cross-lingual synthesis for concatenative systems in order to produce phoneme mappings between source and target languages. While these methods have produced phonetically correct speech output, they are dependent on the source language due to the resulting intonation and have primarily been useful in generating some foreign languages ​​in the context of code mixing. Each spoken language has its own phonetic and prosodic characteristics. Phonetic inventories, phonotactic rules, stress patterns, and rhythms differ considerably between languages. Differences between the source and target languages ​​can result in non-native accents in the transcribed speech, which negatively impacts the intelligibility of the transcribed speech.

[0005] Because acoustic models for different languages ​​function independently of each other, cross-language services are impossible with current technology. This leads to scheduling problems for online music synthesis interfaces, making it impossible to guarantee consistency in the sound quality of synthesized songs, and potentially resulting in language versions being completely different. Since synthesized songs generated by different acoustic models will have completely different sound qualities, users will be confused.

[0006] Therefore, there is still work to be done in the field of computer-intelligent speech synthesis to understand how to accommodate a large number of speakers while maintaining controllable prosody, and how to organize the style of one speaker and apply it to others.

[0007] In light of the above, there is a clear need for systems and methods that simultaneously integrate cross-lingual cloning and prosodic control. [Overview of the project] [Means for solving the problem]

[0008] This specification provides a system for cross-lingual prosodic voice cloning in multiple languages. This system comprises an encoding unit equipped with a multilingual TTS text encoder configured to encode input strings; a multilingual TTS DAT speaker classifier configured to identify and classify speakers in a multilingual context; a capacitron VAE unit configured to learn speech prosody and perform control during inference, wherein the capacitron VAE unit is equipped with a capacitron reference encoder configured to process reference received information and incorporate that reference received information into the model's encoding and decoding process; a concat configured to combine multiple pieces of information from multiple inputs to merge inputs; a masking unit configured to handle variable sequences or missing data from concat data; a soft DTW configured to perform end-to-end training of a model with time-series data alignment, such as speech recognition or time-series prediction, and to improve audio generation quality, wherein the system has an architecture configured to enable simultaneous prosody control and cross-lingual cloning; and a HIFI configured to generate high-quality audio waveforms and natural-sounding speech from a spectrogram. Includes GAN vocoder.

[0009] Potentially, a multilingual TTS text encoder comprises a decoder or vocoder configured to produce speech in the desired language.

[0010] Potentially, multilingual TTS text encoders utilize transformer-based models for efficient encoding of input strings.

[0011] Potentially, a Tacotron2-like decoder could be configured to incorporate an attention mechanism for improving spectrogram generation.

[0012] Potentially, tacotron is known to be able to generate natural-sounding speech using relatively fewer parameters compared to conventional speech synthesis methods.

[0013] Potentially, concatenation is a fundamental operation when building complex neural network architectures, especially when information from different parts of the network needs to be combined or merged.

[0014] Furthermore, this specification provides a method for cross-lingual prosodic voice cloning in multiple languages, the method comprising the steps of: encoding an input string using a multilingual TTS text encoder; identifying and classifying speakers in a multilingual context using a multilingual TTS DAT speaker classifier; learning the prosody of speech and controlling it during inference using a capacitron VAE unit; processing reference received information using a capacitron reference encoder and incorporating the reference received information into the model's encoding and decoding process; combining multiple pieces of information from multiple inputs and merging the inputs using a concat; handling variable sequences or missing data from concat data using a masking unit; and performing end-to-end training of the model with time-series data alignment, such as speech recognition or time-series forecasting, using a soft DTW, and improving the audio generation quality, wherein the system has an architecture that enables simultaneous prosodic control and cross-lingual cloning, and the steps of performing end-to-end training of the model and improving the audio generation quality, and HIFI This includes steps such as using a GAN vocoder to generate high-quality audio waveforms and natural-sounding speech from a spectrogram.

[0015] In one embodiment, the soft DTW loss function is adapted to take into account the temporal alignment between the predicted spectrogram and the target spectrogram for audio generation.

[0016] Numerous additional features, examples, and benefits of the method and system of the present invention will be described below in the following embodiments for carrying out the invention.

[0017] The accompanying drawings illustrate various embodiments of the systems, methods, and other aspects of the present disclosure. Those skilled in the art will understand that the boundaries of elements shown in the drawings (e.g., boxes, groups of boxes, or other shapes) represent examples of such boundaries. In some examples, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some examples, an element shown as an internal component of one element may be implemented as an external component of another element, and an element implemented as an external component of one element may be shown as an internal component of another element. Furthermore, elements may not be drawn to a constant scale. [Brief explanation of the drawing]

[0018] [Figure 1] This diagram shows the architecture of a system for cross-lingual cloning using prosodic control by model training and model inference according to the present invention. [Figure 2] This diagram shows a flowchart for cross-lingual cloning using prosodic control by model training and model inference according to the present invention. [Modes for carrying out the invention]

[0019] Various embodiments will be described below with reference to the attached drawings, but the attached drawings are provided for illustrative purposes only and are not intended to limit the scope in any way, and similar designations refer to similar elements.

[0020] The subject matter is best understood by referring to the detailed figures and descriptions provided herein. Various embodiments will be described below with reference to the figures. However, since the method and system may extend beyond the embodiments described herein, those skilled in the art will readily understand that the detailed descriptions provided herein with respect to the figures are for illustrative purposes only. For example, the teachings and requirements presented in a particular application may lead to several alternative and suitable methods for implementing the functions of the details described herein. Therefore, any method may extend beyond the specific implementation options in the following description and the embodiments presented.

[0021] References to "one embodiment," "example," "at least one embodiment," "one example," "example," and "for example" indicate that the embodiment or example described in that way may include certain features, structures, properties, attributes, elements, or limitations, but not all embodiments or examples necessarily include certain features, structures, properties, attributes, elements, or limitations. Furthermore, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment.

[0022] Figure 1 shows an architectural diagram of a system 100 for cross-lingual cloning using prosodic control by model training and model inference according to the present invention. In particular, the present invention provides a system 100 for cross-lingual prosodic voice cloning in multiple languages. System 100 includes an encoding unit. The encoding unit further comprises a multilingual TTS text encoder 108 that encodes an input string 102. The multilingual text-to-speech (TTS) text encoder 108 is a component of a TTS system designed to process text in multiple languages ​​and generate language-independent embeddings that can be used by a TTS model to generate speech. The multilingual text encoder 108 is configured to process input text in any supported language and convert that text into fixed-size numerical representations (embeddings) that capture the semantic and linguistic content of the text. The encoder is trained on a multilingual corpus, which allows the encoder to learn to encode text from different languages ​​into a shared embedding space and allows the encoder to capture common linguistic characteristics across multiple languages. In some embodiments, the encoder may incorporate language-specific features or embeddings that help generate language-independent representations and preserve certain linguistic characteristics specific to each language.

[0023] The multilingual TTS DAT speaker classifier 112 is configured to identify and classify speakers in a multilingual context. The multilingual TTS DAT speaker classifier 112 combines elements of multilingual text-to-speech (TTS), denoising autoencoder transfer (DAT), and speaker classification. The multilingual TTS enables the system to synthesize speech in multiple languages. The DAT is a technique used to learn speaker embeddings from audio data. In some embodiments, the DAT involves training a neural network to reconstruct clean audio from corrupted audio using an internal representation of the network that acts as a speaker embedding. The learned representation can be transferred to other tasks such as speaker classification.

[0024] The capacitron VAE unit is configured to perform learning of the prosody of speech and to perform control during inference. The capacitron VAE unit includes a capacitron reference encoder 116, and the capacitron reference encoder 116 processes the reference received information 106 and incorporates the reference received information 106 into the encoding and decoding processes of the model. The capacitron VAE (Variational Autoencoder) unit is a type of neural network architecture designed for unsupervised learning of representations from sequential data, including sequential text or time-series text.

[0025] The concatenation unit 120 is configured to combine multiple pieces of information from multiple inputs and merge those inputs. The concatenation unit 120 connects or combines a series or a continuous set of items. In the context of neural networks and deep learning, concatenation generally refers to the operation of joining two or more tensors along a specified axis to create a new tensor. In some embodiments, in deep learning models, concatenation is often used to combine features from different parts of the network or different sources before passing them to the next layer, thereby allowing the model to learn from multiple representations simultaneously. The masking unit 122 is configured to handle variable sequences or missing data from concatenation data. In the context of neural networks, the masking unit 122 refers to a technique of ignoring certain elements in an input tensor or output tensor during computation. In some embodiments, masking 122 is used in sequences, such as in natural language processing (NLP) tasks like text generation or machine translation, where sequences typically have varying lengths.

[0026] This system further includes the use of Soft-DTW as a loss function to improve the quality of audio generation by considering the alignment between the predicted spectrogram and the target spectrogram. Soft-DTW performs end-to-end training of models involving time-series data alignment, speech recognition, or time-series prediction and is configured to improve audio generation quality. This system further comprises a unique architecture that enables simultaneous prosody control and cross-lingual cloning. Soft Dynamic Time Warping (DTW) is a variant of the traditional DTW algorithm that introduces a differentiable approximation, thereby making such a DTW algorithm suitable for use in deep learning models. DTW is a technique used to compare two sequences of different lengths by stretching and shrinking the time axis and find the optimal alignment between those sequences. Soft-DTW is particularly useful in sequence-to-sequence tasks in deep learning, such as speech recognition, where it is common to align sequences of different lengths, thereby enabling end-to-end training of models involving sequence alignment and making it easier to incorporate DTW-like operations into deep learning architectures.

[0027] The HIFI GAN vocoder 126 is configured to generate high-quality audio waveforms and natural-sounding speech from spectrograms. The spectrogram output is then fed into the HiFi-GAN vocoder 126 and converted into high-quality audio that captures the nuances of the speech. HiFi-GAN is an adversarial generative network (GAN) architecture specifically designed for high-fidelity speech synthesis. The HiFi-GAN vocoder 126 focuses particularly on improving the quality and realism of the synthesized speech with respect to capturing fine details and nuances.

[0028] In the context of HiFi-GAN126, the vocoder is part of the model responsible for generating the raw waveform of the synthesized speech. In speech synthesis, the vocoder takes a sequence of acoustic features as input and generates a waveform corresponding to the speech signal. HiFi-GAN's vocoder 126 is designed to generate high-quality waveforms that closely resemble natural speech.

[0029] Figure 2 shows a flowchart of the method 200 for cross-lingual cloning using prosodic control by model training and model inference according to the present invention. In particular, this specification provides a method for cross-lingual prosodic voice cloning in multiple languages. This method involves, in step 202, encoding the input string using a multilingual TTS text encoder; in step 204, identifying and classifying speakers in a multilingual context using a multilingual TTS DAT speaker classifier; in step 206, learning the speech prosody and controlling the inference process using a CapaCitron VAE unit; in step 208, processing reference received information using a CapaCitron reference encoder and incorporating that reference received information into the model's encoding and decoding process; in step 210, combining multiple pieces of information from multiple inputs using ConCut and merging those inputs; in step 212, handling variable sequences or missing data from ConCut data using a masking unit; in step 214, performing end-to-end training of the model with time-series data alignment, speech recognition, or time-series forecasting using Soft DTW to improve audio generation quality; and in step 216, HIFI This includes using a GAN vocoder to generate high-quality audio waveforms and natural-sounding speech from a spectrogram.

[0030] In a preferred embodiment, model training is performed using version 1.8. The model training method includes data acquisition, preprocessing, model initialization, fine-tuning, evaluation, and deployment. The model acquires parallel corpus or monolingual data in the target language, preprocesses the data to remove noise, prepares the data for training, and initializes a pre-trained language model using weights from a model pre-trained on a larger dataset. The model further fine-tunes the pre-trained model for the target language or domain data. The model learns to adapt to specific linguistic characteristics of the target language or domain. The method further includes evaluating the adapted model against a separate validation or test set to assess the performance of the adapted model in order to ensure that the adapted model is effective for the target language or domain. When the adapted model is satisfactorily trained and evaluated, it can be deployed for various natural language processing tasks in the target language or domain.

[0031] The mode obtains language and speaker embeddings from the dataset. The model features a CapaCitron VAE, which gives the user control over speech prosody and provides additional capability to learn speech prosody during inference. The CapaCitron VAE is used to learn representations of time-series data in different languages ​​or domains, and these representations can then be incorporated into the language model's adaptation process. The learned representations help capture the temporal aspects of language use or modification, which helps improve the language model's adaptation to new languages ​​or domains.

[0032] This model takes input as a string and a reference spectrogram, computes a target spectrogram, and then converts the target spectrogram to audio using a Hi-Fi GAN vocoder 126. The model further includes a soft DTW as a loss function to improve the quality of the generated spectrogram. The input string 102 is converted into some form of feature representation that can be fed into the neural network. The neural network architecture takes both the processed string 102 and the reference spectrogram as input. The input can consist of multiple layers, such as convolutional layers, regression layers, and possibly attention mechanisms, to match the text with the spectrogram. The output of the neural network is the target spectrogram, which is a 2D representation of audio frequencies (y-axis) over time (x-axis). The spectrogram is generated based on the input text and the reference spectrogram. The target spectrogram is converted to audio using a Hi-Fi GAN vocoder 126. A vocoder is a tool used to synthesize speech or other sounds by analyzing and replicating the spectral characteristics of an audio signal. Soft DTW is used as the loss function during training to compare the generated spectrogram to a target spectrogram. Soft DTW is a variation of Dynamic Time Warping (DTW) that allows for soft alignment between sequences, which can lead to higher quality output. The entire model is trained using a dataset of text and its corresponding audio spectrograms. The training process involves optimizing the model's parameters to minimize the difference between the generated spectrogram and the target spectrogram, as measured by the soft DTW loss function. Once the model is trained, it is used to generate spectrograms and convert those spectrograms into audio for new input text.

[0033] This method further includes training and model inference. In this method, learned spectrograms and speaker IDs are given as input, and the model is trained to generate spectrograms from the input text and speaker IDs. The spectrograms are then converted to generate audio using a Hi-Fi GAN vocoder. The spectrograms are learned by the model during training based on the input text and speaker ID pairs. The vocoder synthesizes high-quality audio from the spectrograms, thereby capturing the nuances of speech for a given speaker ID. The training process involves optimizing the model parameters to minimize the difference between the generated spectrograms and the target spectrograms.

[0034] In some exemplary embodiments, cross-lingual prosodic voice cloning across 25 language models may be used to perform music cloning. This method includes removing background music from vocals in a song using a state-of-the-art model such as demucs, fine-tuning existing voice cloning for singing, transcribing and translating the original script using OpenAI's Whisper application and language models, and generating a target voice for a given song sequence using the utterances of the original song and the translated strings. The last method is repeated for all sequences in the song. In some embodiments, the language model may be at least one of openNMT or M2M Large. In some embodiments, a candidate-based method may be used to select an ultra-accurate translation with the ability to manually correct it.

[0035] In some embodiments, the system can be model-trained and perform inference using version 1.7. The system takes an input string 102 as input and encodes the received data using a multilingual TTS text encoder 108. The encoded data then flows to a capacitron reference encoder, which interacts with other components. A multilingual TTS DAT speaker classifier is connected to the reference encoder. Data from the reference encoder flows through a VAE sampling process. The output from the VAE sampling is processed by a masking unit before reaching a Tacotron2-like decoder. The decoded output is processed for a HiFi-GAN vocoder to generate the final audio waveform. Version 1.7 further includes a direct connection from the text encoder to the decoder and a feedback loop from the capacitron beta optimizer to the reference encoder. Version 1.7 further includes an additional input from the speaker classification loss with gradient reversal to the reference encoder.

[0036] In some embodiments, existing methods rely on explicit duration prediction systems operating on mel-spectrograms. This method does not use explicit duration prediction; sound duration is implicitly learned in a language modeling task on the codec unit. This method infers the fundamental frequency and other aspects of prosody from a speaker embedded transformer network. This method operates purely on text sequences. Furthermore, speaker prosodic and tonal characteristics are inferred from reference utterances, rather than from a specified tone / pitch track. This method uses cross-entropy loss to predict discrete speech codec tokens.

[0037] This invention can be used for video post-production, AV post-production, AV editing, AV localization, AV internationalization, dubbing, voiceover, personal assistant, music repurposing, and content repurposing. This invention simultaneously achieves prosodic control and cross-lingual control using a soft DTW loss function implementation.

[0038] This system, along with the implementation of a proprietary software DTW loss function, offers advantages such as simultaneous prosodic control and cross-lingual cloning, though not limited to these. Examples of applications include video post-production, AV editing, dubbing, voiceover, music repurposing, and content localization. These steps highlight the complex processes within the TTS system architecture: language and speaker embedding learning, prosodic control, cross-lingual voice conversion, and music translation and repurposing.

[0039] The advanced text-to-speech system presented herein represents a major leap forward in speech synthesis technology. With its groundbreaking features, unique architecture, and superior performance, this system has the potential to revolutionize the way we interact with and consume audio content in diverse multimedia applications. This document provides a comprehensive description of the system, method, and dependent claims, highlighting the inventive features and functions of the proposed text-to-speech technology.

[0040] The above description of specific embodiments is intended to fully illustrate the general nature of the embodiments herein, that others may readily modify and / or adapt such specific embodiments to various applications without deviating from the general concept by applying their current knowledge, and that such adaptations and modifications should and should be understood within the meaning and scope of the equivalents of the disclosed embodiments. It should be understood that any expressions or terms used herein are for illustrative purposes only and not for limiting purposes. Therefore, while preferred embodiments have been described herein, those skilled in the art will recognize that the embodiments herein may be implemented with modifications within the spirit and scope of the appended claims.

[0041] Note that various connections between elements are described in this specification and in the drawings (whose contents are included in this disclosure by reference). Note that these connections may be direct or indirect unless otherwise specified, and this specification is not limited in this respect. In this regard, connections between entities may refer to either direct or indirect connections.

[0042] Various embodiments of the present invention have been disclosed. However, it should be apparent to those skilled in the art that modifications are possible in addition to those described herein, without departing from the inventive concept set forth herein. The embodiments are therefore not limiting except to the spirit of this disclosure. Furthermore, in interpreting this disclosure, all terms should be understood in the broadest possible form consistent with the context. In particular, the terms “equipped with” and “equipped with” should be interpreted in a non-exclusive manner as referring to an element, component, or step, indicating that the elements, components, or steps mentioned may be presented or used together with, or combined with, other elements, components, or steps not expressly mentioned.

[0043] The methods and systems disclosed herein, or any components thereof, may be implemented in the form of a computer system. Common examples of computer systems include general-purpose computers, programmed microprocessors, microcontrollers, peripheral integrated circuit elements, and other devices, or configurations of devices, that are capable of implementing the steps comprising the methods of this disclosure.

[0044] The computer system comprises a computer, input devices, a display unit, and the internet. The computer further comprises a microprocessor. The microprocessor is connected to a communication bus. The computer also includes memory. The memory may be random-access memory (RAM) or read-only memory (ROM). The computer system further comprises a storage device, which may be a hard disk drive or a removable storage drive such as a floppy disk drive or optical disk drive. The storage device may also be a means for loading computer programs or other instructions into the computer system. The computer system also includes a communication unit. The communication unit enables the computer to connect to other databases and the internet through input / output (I / O) interfaces, thereby enabling the transfer and reception of data from other sources. The communication unit may include a modem, an Ethernet card, or other similar device that enables the computer system to connect to databases and networks such as LANs, MANs, WANs, and the internet. The computer system facilitates user input through input devices accessible to the system via I / O interfaces.

[0045] To process input data, a computer system executes a set of instructions stored in one or more memory elements. These memory elements may also hold data or other information as desired. Memory elements may be in the form of information sources or physical memory elements present within the processing machine.

[0046] Programmable instructions or computer-readable instructions may include a variety of commands that instruct a processing machine to perform a particular task, such as the steps that constitute the methods of this disclosure. The systems and methods described may also be implemented using software programming alone, or using hardware alone, or by different combinations of the two techniques. This disclosure is independent of the programming languages ​​and operating systems used in computers. Instructions for this disclosure can be written in all programming languages, including, but not limited to, C, C#, C++, Embedded C, Visual C++, Java®, Python, and Visual Basic. Furthermore, software may take the form of a collection of separate programs, a program module containing a larger program, or a part of a program module, as described herein. Software may also include modular programming in the form of object-oriented programming. Processing of input data by a processing machine may be in response to user commands, as a result of previous processing, or as a result of a request made by another processing machine. This disclosure may also be implemented on a variety of operating systems and platforms, including, but not limited to, iOS, Mac, Unix®, DOS, Android, Symbian, and Linux®.

[0047] Programmable instructions can be stored on and transmitted on a computer-readable medium. The disclosure can also be implemented in a computer program product comprising a computer-readable medium, or in any product capable of implementing the above method and system, or in conjunction with a number of possible variations of such product.

[0048] Various implementations of the systems and techniques described herein can be realized in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, which include at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0049] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor.

[0050] To interact with the user, the systems and techniques described herein may be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can make input to the computer. Similarly, other types of devices may be used to interact with the user; for example, the feedback given to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, voice input, or tactile input.

[0051] Those skilled in the art will understand that this system, module, and submodule are illustrated and described to serve as examples and should not be considered limiting in any way. Furthermore, it will be understood that variations of the system elements, modules, and other features and functions disclosed above, or their substitutes, may be combined to create other different systems or applications.

[0052] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with the implementation of the systems and techniques described herein), or in any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet. The claims may encompass embodiments of hardware, software, or combinations thereof.

[0053] Although only a few implementations have been described in detail above, other modifications are possible. Furthermore, other mechanisms may be used to implement the systems and methods described in this book. Moreover, the logical flows depicted in the figures may not require a specific order or sequence to achieve the desired results. Other steps may be added to the described flows, or steps may be removed from the described flows; other components may be added to the described systems, or other components may be removed from the described systems. Therefore, other implementations fall within the scope of the following claims.

Claims

1. A system for cross-lingual prosodic voice cloning in multiple languages, wherein the system is A multilingual TTS text encoder configured to encode input strings. An encoding unit comprising, A multilingual TTS DAT speaker classifier configured to identify and classify speakers in a multilingual context, A capacitron VAE unit configured to perform speech prosody learning and control during inference, wherein the capacitron VAE unit is A capacitron reference encoder configured to process reference received information and incorporate said reference received information into the model's encoding and decoding process. A CapaCitron VAE unit equipped with, A concuter configured to combine multiple pieces of information from multiple inputs in order to merge the inputs, A masking unit configured to handle variable sequences or missing data from concatenated data, A soft DTW configured to perform end-to-end training of a model involving time-series data alignment, such as speech recognition or time-series forecasting, and to improve audio generation quality, wherein the system comprises an architecture configured to enable simultaneous prosodic control and cross-lingual cloning. A HIFI GAN vocoder configured to generate high-quality audio waveforms and natural-sounding voices from a spectrogram. A system equipped with these features.

2. The system according to claim 1, wherein the multilingual TTS text encoder comprises a decoder or vocoder configured to generate speech in a desired language.

3. The system according to claim 1, wherein the multilingual TTS text encoder utilizes a transformer-based model for efficient encoding of input strings.

4. The system according to claim 1, wherein a Tacotron 2-like decoder is configured to incorporate a attention mechanism for improving spectrogram generation.

5. The system according to claim 1, wherein tacotron is known to be able to generate natural-sounding speech using a relatively small number of parameters compared to conventional speech synthesis methods.

6. The system according to claim 1, wherein concatenation is a fundamental operation in constructing complex neural network architectures, particularly when information from different parts of a network needs to be combined or merged.

7. A method for cross-lingual prosodic voice cloning in multiple languages, wherein the method is Encoding the input string using a multilingual TTS text encoder, Using a multilingual TTS DAT speaker classifier, we identify and classify speakers in a multilingual context. Using the Capacitron VAE unit, we perform speech prosody learning and control during inference. Using a capacitron reference encoder, the reference received information is processed, and the said reference received information is incorporated into the model's encoding and decoding process. Using Concut, you can combine multiple pieces of information from multiple inputs and merge the inputs. Using a masking unit to handle variable sequences or missing data from concatenated data, Using soft DTW, the system performs end-to-end training of a model involving time-series data alignment, such as speech recognition or time-series forecasting, to improve audio generation quality, wherein the system has an architecture that enables simultaneous prosodic control and cross-lingual cloning. Using a HIFI GAN vocoder, it is possible to generate high-quality audio waveforms and natural-sounding speech from a spectrogram. Methods that include...

8. The method according to claim 7, wherein the soft DTW loss function is adapted to take into account the temporal alignment between the predicted spectrogram and the target spectrogram for audio generation.

Images