Text-to-speech synthesis without using parallel text-audio data

The unsupervised text-to-speech method generates a Mel spectrogram through dictionary and self-supervised acoustic models, solving the problem of existing systems' dependence on high-quality data, achieving flexible and controllable speech synthesis, reducing data collection costs, and being suitable for Diversified needs.

CN118985024APending Publication Date: 2024-11-19TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380031008.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-09-27
Filing Date
2023-03-23
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing supervised text-to-speech systems require a large amount of high-quality parallel voice and text data when generating high-quality voice. Collecting and labeling this data is time-consuming and expensive, limiting the AI ​​assistant, gaming and entertainment industries to custom voice. Common deployment of synthesizers.

Method used

Unsupervised text-to-speech method is used to decompose the input text into phoneme sequences of the target language through dictionary, predict the duration of speech, and use the self-supervised TTS acoustic model to generate a Mel spectrogram, which is finally converted into a waveform to achieve speech synthesis without Parallel speech and text data are required.

Benefits of technology

It realizes speech synthesis without relying on parallel speech and text data, provides a flexible and controllable TTS system, reduces the cost of data collection and labeling, and is suitable for diversified needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0005062814610000071
    Figure BDA0005062814610000071
  • Figure HDA0005062814620000011
    Figure HDA0005062814620000011
  • Figure HDA0005062814620000021
    Figure HDA0005062814620000021
Patent Text Reader

Abstract

An unsupervised text-to-speech system utilizes a dictionary to map input text to a sequence of phonemes that is extended to frame-level forced alignment by a speaker-related duration model. The alignment mapping module translates the forced alignment to an unsupervised alignment (UA). Then, a conditional de-entanglement sequential variational autoencoder (C-DSVAE) serving as a self-supervised TTS AM employs the predicted UA and target speaker embedding to generate a Mel-spectrum, which is finally converted to a waveform by a neural vocoder.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. patent application No. 17 / 953,851 filed on September 27, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0001] The present disclosure relates generally to text-to-speech, and more particularly to text-to-speech methods and apparatus for converting text to speech. Background Art

[0002] Text-to-speech (TTS) synthesis plays an important role in human-computer interaction. With the continuous development of neural-based TTS systems (e.g., Tacotron DurIAN, FastSpeech, or the recent Glow-TTS series), high-fidelity synthesized speech has narrowed the gap between machine-generated speech and human speech. This is especially true for languages ​​with rich resources (e.g., languages ​​with considerable high-quality parallel speech and text data). Typically, supervised TTS systems require tens of hours of single-speaker high-quality data to generate high-quality performances. However, collecting and labeling such data is a non-trivial task, time-consuming and expensive. Therefore, current supervised solutions still have their limitations in terms of the high requirements for the universal deployment of custom speech synthesizers in AI (Artificial Intelligence, AI) assistants, games or entertainment industries. When faced with these diverse needs, natural, flexible and controllable TTS approaches become more important. Summary of the invention

[0003] According to an embodiment, a system and method of an unsupervised text-to-speech method performed by at least one processor is provided, and the unsupervised text-to-speech method includes: receiving input text; generating an acoustic model, including: decomposing the input text into at least one synthetic sound of a target language via a dictionary; predicting the duration of speech generated based on the input text; aligning at least one synthetic sound to regularize the input text to follow the sound of the target language as an aligned output; automatically encoding the aligned output and the duration of the speech generated based on the target input text into an output waveform; and outputting the sound based on the output waveform.

[0004] In some implementations, predicting the duration of a speech comprises: sampling a speaker pool containing at least one speech; and calculating the duration of the speech by mapping a dictionary sound with the length of the input text and the speaker pool.

[0005] According to some embodiments, the dictionary contains at least one phoneme sequence.

[0006] According to some embodiments, an unsupervised alignment is predicted that aligns sounds of a target language with durations of speech; an input text is encoded; a priori content is encoded using the output of the predicted unsupervised alignment; a posterior content is encoded using the encoded input text; the prior content and the posterior content are decoded; a mel-spectrogram is generated based on the decoded prior content and the decoded posterior content; and the mel-spectrogram is processed by a neural vocoder to generate a waveform.

[0007] According to some embodiments, the target input text is selected from the group consisting of: a book, a text message, an email, a newspaper, a printed paper, and a logo.

[0008] According to some implementations, mapping the text to a forced alignment; and converting the forced alignment to an unsupervised alignment.

[0009] According to some embodiments the predicted duration is calculated in at least one logarithmic domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a systematic overview of implementations of unsupervised text-to-speech systems.

[0011] Figure 2 is a block diagram of an embodiment of the processing of an unsupervised text-to-speech system.

[0012] Figure 3 It is the implementation method of C-DSVAE (Conditional Disentangled Sequential Variational Auto-Encoder, C-DSVAE) system and training.

[0013] Figure 4 It is an implementation of speech conversion for unsupervised text-to-speech systems.

[0014] Figure 5 is an implementation of alignment-driven speech generation.

[0015] Figure 6 is a block diagram of an implementation of an unsupervised text-to-speech system. DETAILED DESCRIPTION

[0016] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0017] The foregoing disclosure provides explanation and description, but is not intended to be exhaustive or to limit implementation to disclosed exact form. According to the above disclosure, modifications and variations can be made, or modifications and variations can be obtained from the practice of implementation. In addition, one or more features or parts of an embodiment can be incorporated into another embodiment (or one or more features of another embodiment) or combined with another embodiment (or one or more features of another embodiment). In addition, in the flow chart and the description of the operation provided below, it should be understood that one or more operations can be omitted, one or more operations can be added, one or more operations can be performed (at least in part) simultaneously, and the order of one or more operations can be switched.

[0018] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes. It should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0019] Although specific combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly reference only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim group.

[0020] Unless explicitly described herein, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "a" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". When only one item is intended, the term "one" or similar language is used. In addition, as used herein, the terms "has", "have", "having", "including", "including", etc. are intended to be open terms. In addition, unless explicitly stated otherwise, the phrase "based on" is intended to mean "based at least in part". In addition, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0021] Embodiments of the present disclosure relate to unsupervised text-to-speech systems developed to overcome the problems discussed above. Embodiments of the present disclosure include an unsupervised text-to-speech (UTTS) framework that does not require text-audio pairs for TTS acoustic modeling (AM). In some embodiments, the UTTS can be a multi-speaker speech synthesizer developed from the perspective of disentangled speech representation learning. The method framework provides flexible selection of duration models, timbre features (identity) and content of speakers for TTS reasoning. Unsupervised text-to-speech systems can utilize the latest advances in speech synthesis front-end technology and self-supervised speech representation learning for system development.

[0022] Specifically, in some embodiments, the unsupervised text-to-speech system can utilize a dictionary to map the input text to a phoneme sequence, which is extended to a frame-level forced alignment (FA) through a speaker-dependent duration model. The input text can be any type of text, such as a book, a text message, an email, a newspaper, a printed paper, and a logo or any other letter or text representative pictogram. Next, the alignment mapping module can convert the FA into an unsupervised alignment (UA). Finally, a conditional disentangled sequential variational autoencoder (C-DSVAE) used as a self-supervised TTS AM can use the predicted UA and target speaker embedding to generate a mel-spectrogram, which is ultimately converted into a waveform by a neural vocoder. Unsupervised text-to-speech does not require parallel speech and text data to train the TTS acoustic model (AM). Therefore, the unsupervised text-to-speech system can perform speech synthesis without using a paired TTS corpus.

[0023] Figure 1 An exemplary system 100 for an embodiment using unsupervised text-to-speech is shown. The exemplary system 100 can be one of a variety of systems such as a personal computer, a mobile device, a computer cluster, a server, an embedded device, an ASIC, a microcontroller, or any other device capable of running code. A bus 110 connects the exemplary system 100 together so that all components can communicate with each other. The bus 110 connects the processor 120, the memory 130, the storage component 140, the input component 150, the output component 160, and the interface component.

[0024] The processor 120 can be a single processor, a processor with multiple processors inside, a processor cluster (more than one processor) and / or distributed processing. The processor executes instructions stored in both the memory 130 and the storage component 140. The processor 120 operates as a computing device, performing operations for unsupervised text-to-speech processing. The memory 130 is a fast storage device, and the retrieval of any one of the memory devices can be achieved by using a cache memory, which can be closely associated with one or more CPUs. The storage component 140 can be any long-term storage device such as one of HDD, SSD, tape or any other long-term storage format.

[0025] Input component 150 may be any file type or signal from a user interface component such as a camera or text capture device. Output component 160 outputs the processed information to communication interface 170. The communication interface may be a speaker or other communication device that can display information to a user or another observer such as another computing system.

[0026] Figure 2 The processing steps of an exemplary embodiment of unsupervised text-to-speech processing are described in detail. Processing can start at step S110, in which text is input into unsupervised text-to-speech (UTTS). The input text can be any text in any language in which there are spoken words. Processing proceeds to step S120, in which the text is then decomposed into phonemes that can correspond to the different sounds of each word and language. After the phonemes are determined, the length of the speech to be produced is determined. In the case of long text or words with a large amount of sounds and / or syllables, the length of the speech can be determined to be longer. In the case of short text or sounds and / or syllables, the length of the speech can be determined to be smaller.

[0027] Processing proceeds from step S130 to step S140, in which an unsupervised alignment is performed. As an example, forced alignment (FA) can refer to the process of aligning an orthographic transcription with an audio recording to automatically generate a phoneme-level segmentation. Unsupervised alignment (UA) can refer to the process of conditioning and regularizing the output of a text-to-speech system to follow the phonetic structure. Next, an Nn mapping step S150 converts the FA into an unsupervised alignment. Finally, a mel-spectrogram is generated S160 that can be played by a speaker that completes the text-to-speech processing.

[0028] Figure 3It is an implementation of a conditional disentangled sequential variational autoencoder (C-DSVAE) system and training. The backbone of the encoder can be a DSVAE (Disentangled Sequential Variational Auto-Encoder, DSVAE) architecture, which includes a shared encoder 305, a posterior speaker encoder 320, a posterior content encoder 325, a priori speaker encoder 315, a priori content encoder 330, a decoder 345, and a final synthesized Mel speech 350. The Mel spectrogram 300 can be passed to the shared encoder 305, and then passed to the posterior encoder 320 and the posterior content encoder 325, which encode the speaker posterior distribution q(z_s|X) and the content posterior distribution q(z_c|X). After the distribution is generated, the distribution is passed to the speaker embedding 335 and the content embedding 340 respectively. Next, the data processed by speaker embedding 335 and content embedding 340 are decoded by decoder 345 to generate synthesized mel speech 350.

[0029] For prior modeling, the prior speaker encoder 315 can encode the speaker prior p(z_s), and the posterior content encoder 325 can encode the content prior p(z_c). During the decoding / generation phase, the speaker embedding 335 (z_s) and the content embedding 340 (z_c) are sampled according to the posterior q(z_s|X) and q(z_c|X) or the prior p(z_s) and p(z_c), and their concatenation is passed to the decoder D to generate the synthesized speech.

[0030] In some embodiments, in order to generate a phonetically meaningful and continuous speech with a stable pronunciation, the unsupervised text-to-speech system can use acoustic alignment 310 (A_X) as a condition of the content prior distribution. In some embodiments, the unsupervised text-to-speech system can use two types of acoustic alignment: forced alignment (FA) and unsupervised alignment (UA). In this embodiment, A_X^FA can represent the forced alignment of speech X. Forced alignment can be a Montreal Forced Alignment (MFA) for obtaining a given audio-text pair for forced alignment. The unsupervised text-to-speech system uses the WavLM-Base model to extract acoustic features. In some embodiments, in order to capture robust and long-range time relationships on acoustic units and generate more continuous speech, the unsupervised text-to-speech system uses masked unit prediction (Masked Unit Prediction, MUP) when training the prior content encoder 330 (Ecp).

[0031] In computation, the variable It can represent a set of T masked indices for a specific condition A_X, where the masking configuration is consistent. The variable (AX)^ can represent a corrupted version of A_X, where if t∈M(A_X), A_Xt will be masked. The variable z_cp can represent a sample of the output of E_cp (i.e., z_cp~E_cp(A_X)). The negative log-likelihood loss (NLL) LMUP-C for conditional modeling is defined in Equation 1, where p(z_cpi|(A_Xi)^) is the softmax classification distribution. EAX represents the expectation for all A_X. The implementation of the masked prediction loss can be formulated as follows:

[0032] Formula 1:

[0033] In some embodiments, the C-DSVAE loss objective may be formulated as follows:

[0034] Formula 2:

[0035] Formula 3:

[0036] Figure 4 is an implementation of speech conversion for an unsupervised text-to-speech system. Speech conversion includes a target mel-spectrogram 400 and a source mel-spectrogram 405 that are fed to a shared encoder 410 / 415 (different according to a separate path). After being processed by the shared encoder 410, the first text 400 is then passed to the posterior speaker encoder 420, and then to the speaker embedding 430. The second text 405 is moved through the posterior content encoder 425 and then to the content embedding 435. Finally, these branches are fed together into a decoder 440, which ends with a synthesized mel speech 445.

[0037] Figure 5 is an implementation of alignment-driven speech generation. The speech conversion includes a target mel-spectrogram 500 and an acoustic alignment 505 that are fed into a shared encoder 510 / 515 (different according to the individual paths). After being processed by the shared encoder 510, the target mel-spectrogram 500 is then passed to the posterior speaker encoding 520, and then to the speaker embedding 530. The acoustic alignment 505 is moved through the posterior content encoder 525 and then to the content embedding 535. Finally, these branches are fed together into a decoder 540, which ends with a synthesized mel speech 545.

[0038] Figure 6600 is a block diagram of an embodiment of an unsupervised text-to-speech system. First, an input text 600 is fed into the system. The input text 600 is first decomposed to obtain a phoneme sequence of the text transcription using a dictionary 605. The dictionary 605 can be one of the Librispeech dictionary, CMU dictionary, Amazon Polly, or other defined dictionaries. The phoneme sequence is then converted into a list of token indices.

[0039] Meanwhile, duration predictor 620 takes the phoneme sequence (output of dictionary 605) and sampled 615 information from speaker pool 610 as input to predict the speaker perceived duration of each phoneme. Specifically, the phoneme sequence is first passed to a trainable lookup table to obtain phoneme embeddings. Then, a four-layer multi-head attention (Multi-Head Attention, MHA) module then extracts the potential phoneme representation. A two-layer conv-1D module is then used to obtain the sum of the potential phoneme representation and the speaker embedding sampled from the speaker pool. Finally, a linear layer is applied to generate the predicted duration in the logarithmic domain. The predicted duration output to speaker-aware duration prediction (SADP) 625 is used as the length of the predicted speech.

[0040] The phoneme sequence is passed to the speaker perceived duration prediction (SADP) 625 which delivers the predicted forced alignment (FA) along with the random speaker embedding. The Forced Alignment To Unsupervised Alignment (FA2UA) 630 module takes the predicted FA as input and predicts the unsupervised alignment 635 (UA). The UA 635 is fed into the conditional disentangled sequential variational autoencoder (C-DSVAE) 670 along with the input utterance 640 to generate a mel-spectrogram. The predicted unsupervised alignment 635 is fed to the prior content encoder 650 and then to the decoder 660. At the same time, the input utterance 640 is fed to the shared encoder 645 and then to the posterior speaker encoder 655, and the input utterance 640 finally meets the data of the prior content encoder 650 in the decoder 660 to generate a mel-spectrogram. The neural vocoder 665 is then applied to convert the mel-spectrogram into a waveform. It can be observed that the proposed UTTS system performs zero-shot voice cloning of the target utterance. The two modules comprising C-DSVAE 670 are trained separately. The detailed model architecture is presented in Table 1. Table 1

[0041] Table 1 shows the UTTS system in detail. For Conv1D, the configuration is (filter size, kernel size, padding and stride). For Multi-Head Attention (MHA), the configuration can be model dimension, key dimension, value dimension, number of heads. For LSTM / BiLTSM / RNN, the configuration can be hidden layer dimension, layers. For Dense layer, the configuration can be output dimension.

[0042] The UTTS system in Table 1 decomposes the architecture into other components, C-DSVAE 670, duration predictor and FA2UA. C-DSVAE 670 includes a shared encoder 645, a prior speaker encoder, a posterior speaker encoder 655, a posterior content encoder, a prior content encoder 650, a prior content decoder and a posterior content decoder. The shared encoder 645 consists of Conv1D, InstanceNorm2D and ReLU. The posterior speaker encoder includes BiLSTM, average pooling, Dense(64) to mean and Dense(64) to standard deviation. The prior speaker includes an identity mapping. The posterior content encoder includes BiLSTM, RNN, Dense(64) to mean and Dense(64) to standard deviation. The prior content encoder includes BiLSTM, Dense(64) to mean, Dense(64) to standard deviation and a linear classifier. The prior content decoder consists of LSTM(512,1) to LSTM(1024,2) to Dense(80). The posterior content decoder consists of Conc1D to tanh to InstanceNorm2D. In addition, the duration predictor consists of nn.Embedding to MHA to Conv1D to Dense(1). Finally, FA2UA consists of nn.Embedding to BiLSTM to linear classifier.

[0043] During training, MSE (Mean Squared Error) may be employed between the predicted duration 620 and the target duration. The target duration for the text is obtained from a forced alignment obtained by Montreal Forced Alignment (MFA) or other forced alignment methods. In some implementations, the target duration may be in the logarithmic domain. During inference, the duration may be rounded.

[0044] The FA2UA 630 module takes the forced alignment (FA) as input and predicts the corresponding unsupervised alignment (UA). Specifically, the FA is first passed into a learnable lookup table to obtain the FA embedding. Subsequently, a 3-layer Bi-LSTM module can be used to predict the UA embedding given the FA embedding. During training, a masked prediction training strategy is adopted to train the FA2UA module, because masked prediction is expected to capture long-range temporal dependencies across symbols well and encode more contextual information for each symbol.

[0045] Computationally, unsupervised text-to-speech will denoted as a set of T masked indices for a particular unsupervised alignment FA. Variable (FA) ~ can be a corrupted version of FA, where 〖FA〗_i can be masked if i∈M(FA). The variable 〖UA〗_i can correspond to the frame of FA. The negative log-likelihood loss (NLL) L FA2UA for masked prediction training can be defined as follows, where p(〖UA〗_i|(〖FA〗_i) ~ ) is a softmax categorical distribution. The variable E_((FA,UA)) represents the expectation for all (FA,UA) pairs. During inference, the symbol p(UA_i|〖FA_i) with the maximum probability is selected at each time step i to form the predicted UA sequence. Therefore, FA2UA 630 can be defined as follows:

[0046] Formula 4:

[0047] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed.Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0048] Some embodiments may involve systems, methods, and / or computer-readable media at any possible level of integrated technical detail. In addition, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform operations.

[0049] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (Random Access Memory, RAM), a read-only memory (Read-Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM or flash memory), a static random access memory (Static Random Access Memory, SRAM), a portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a digital versatile disk (Digital Versatile Disk, DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be construed as being themselves transient signals, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0050] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0051] The computer readable program code / instructions for performing operations may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc. and procedural programming languages ​​such as "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of being executed entirely on a remote computer or server, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit system including, for example, a programmable logic circuit system, a Field-Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA) can execute computer-readable program instructions by personalizing the electronic circuit system using state information of the computer-readable program instructions to perform various aspects or operations.

[0052] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device implement the operations specified in the flowchart and / or one or more block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct a computer, a programmable data processing device, and / or other equipment to operate in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the operations specified in the flowchart and / or one or more block diagram blocks.

[0053] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the operations specified in the flowchart and / or one or more block diagram blocks.

[0054] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of systems, methods and computer-readable media according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, a section or an instruction portion including one or more executable instructions for implementing a specified logical operation. Methods, computer systems and computer-readable media may include additional blocks, fewer blocks, different blocks or blocks of different arrangements compared to the blocks depicted in the figure. In some alternative implementations, the operations indicated in the blocks may not occur in the order indicated in the accompanying drawings. For example, two blocks shown in succession may actually be performed simultaneously or substantially simultaneously, or blocks may sometimes be performed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart illustration and the combination of blocks in the block diagram and / or flowchart illustration can be implemented by a system based on special hardware, which performs specific operations or actions or implements a combination of special hardware and computer instructions based on the system based on special hardware.

[0055] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. An unsupervised text-to-speech method performed by at least one processor, and comprising: Receive input text; Generate an acoustic model, including: decomposing the input text into at least one synthesized voice in a target language via a dictionary; predicting the duration of speech generated from the input text; aligning the at least one synthesized sound to regularize the input text to conform to the sounds of the target language as an aligned output; Automatically encoding the aligned output and the duration of the speech generated from the target input text into an output waveform; and The sound is output according to the output waveform.

2. The unsupervised text-to-speech method according to claim 1, wherein: Predicting the duration of the speech comprises: sampling a pool of speakers comprising at least one speech; and The duration of the speech is calculated by mapping dictionary sounds to the length of the input text and the speaker pool.

3. The unsupervised text-to-speech method according to claim 1, wherein: The dictionary contains at least one phoneme sequence.

4. The unsupervised text-to-speech method according to claim 1, wherein: Automatically encoding the alignment output further comprises: predicting an unsupervised alignment that aligns sounds of the target language with durations of the speech; Encoding the input text; Encoding the prior content using the predicted unsupervised aligned output; encoding the posterior content using the encoded input text; Decoding the a priori content and the a posteriori content; generating a mel-spectrogram according to the decoded prior content and the decoded posterior content; and The mel-spectrogram is processed by a neural vocoder to generate a waveform.

5. The unsupervised text-to-speech method according to claim 1, in, The target input text is selected from the group consisting of: a book, a text message, an email, a newspaper, a printed paper, and a logo.

6. The unsupervised text-to-speech method according to claim 1, wherein: The alignment further comprises: mapping the text to enforce alignment; and Convert the forced alignment to an unsupervised alignment.

7. The unsupervised text-to-speech method according to claim 2, wherein: The alignment further comprises: The predicted duration is calculated in at least one logarithmic domain.

8. An unsupervised text-to-speech device comprising: at least one memory configured to store computer program code; At least one processor, the at least one processor being configured to operate as instructed by the computer program code, the computer program code comprising: acoustic modeling code configured to cause the at least one processor to generate an acoustic model having at least one dictionary including sounds of a target language, the acoustic modeling code further comprising: duration predictor code configured to cause the at least one processor to predict a duration of speech generated from target input text; alignment code configured to cause the at least one processor to regularize the input text to conform to the sounds of the target language as an aligned output; and An autoencoder code is configured to cause the at least one processor to transform the aligned output and a duration of the speech generated from the target input text into an output waveform.

9. The unsupervised text-to-speech device of claim 8, wherein: The duration predictor code also includes a duration calculator code configured to cause the at least one processor to calculate the duration of the speech by mapping dictionary sounds to the length of input text, Wherein, the duration predictor code further causes the processor to predict the duration of the speech based on speaker pool data including at least one sampled speech.

10. The unsupervised text-to-speech device of claim 8, wherein: The dictionary includes at least one phoneme sequence.

11. The unsupervised text-to-speech device of claim 8, wherein: The autoencoder code is configured to also include: predictive unsupervised alignment code configured to cause the at least one processor to align sounds of the target language with durations of the speech; shared encoder code configured to cause the at least one processor to encode the input text; a priori content encoder code configured to cause the at least one processor to encode the predicted unsupervised aligned output posterior; a posteriori content encoder code configured to cause the at least one processor to encode an output of the shared encoder; decoder code configured to cause the at least one processor to combine outputs of the a posteriori content encoder and the a priori content encoder and generate a mel-spectrogram; and A neural vocoder is provided to generate a waveform according to the mel-spectrogram.

12. The unsupervised text-to-speech device according to claim 8, in, The target input text is selected from the group consisting of: a book, a text message, an email, a newspaper, a printed paper, and a logo.

13. The unsupervised text-to-speech device of claim 8, wherein: The alignment code is configured to cause the at least one processor to: mapping the text to be forced to align; as well as Convert the forced alignment to an unsupervised alignment.

14. The unsupervised text-to-speech device of claim 9, wherein: The duration predictor code is configured to cause the processor to predict the duration of the speech in at least one logarithmic domain.

15. A non-transitory computer readable medium having stored therein instructions which, when executed by a processor, cause the processor to: Receive input text; Generate an acoustic model, including: decomposing the input text into at least one synthesized voice in a target language via a dictionary; predicting the duration of speech generated from the input text; aligning the at least one synthesized sound to regularize the input text to conform to the sounds of the target language as an aligned output; Automatically encoding the aligned output and the duration of the speech generated from the input text into an output waveform; as well as The sound is output according to the output waveform.

16. The non-transitory computer readable medium of claim 15, in, Predicting said duration includes: sampling a pool of speakers comprising at least one speech; and The duration of the speech is calculated by mapping dictionary sounds to the length of the input text and the speaker pool.

17. The non-transitory computer readable medium of claim 15, in, The dictionary includes at least one phoneme sequence.

18. The non-transitory computer readable medium of claim 15, wherein: The instructions are configured to further cause the processor to: predicting an unsupervised alignment of the target language sounds to the duration of the speech; Encoding the input text; Encoding the prior content using the predicted unsupervised aligned output; encoding the posterior content using the encoded input text; Decoding the a priori content and the a posteriori content; generating a mel-spectrogram according to the decoded prior content and the decoded posterior content; as well as The mel-spectrogram is processed by a neural vocoder to generate a waveform.

19. The non-transitory computer readable medium of claim 15, in, The target input text is selected from the group consisting of: a book, a text message, an email, a newspaper, a printed paper, and a logo.

20. The non-transitory computer readable medium of claim 15, wherein: The instructions further cause the processor to: mapping the text to enforce alignment; and Convert the forced alignment to an unsupervised alignment.