Speech synthesis method and system, electronic equipment and readable storage medium

Abstract

The invention provides a speech synthesis method and system, electronic equipment and a computer readable storage medium. The speech synthesis method comprises the following steps: based on a plurality of different time resolution stages, performing multi-codebook quantization on an acoustic feature sequence of an input text to form a first quantization sequence; compressing the first quantized sequence into a second quantized sequence; and mapping the second quantized sequence into a voice waveform corresponding to the input text. According to the speech synthesis method and system provided by the invention, while the requirement for supervision data is reduced, the complexity of constructing the speech synthesis system is reduced, and high-quality speech synthesis is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech processing technology, and more specifically, to a speech synthesis method and system. The embodiments of the present application also relate to an electronic device and a computer-readable storage medium. Background Art

[0002] Text-to-Speech (TTS) aims to convert text into speech signals with correct pronunciation, natural rhythm and high sound quality. Currently, TTS has been widely used in various intelligent products, such as human-computer interaction (HCI), speech-to-speech translation (S2ST) and artificial intelligence generated content (AIGC).

[0003] As artificial intelligence technology spreads to various fields, in order to provide better services, TTS systems pay more and more attention to improving comprehensive performance in various aspects such as personalization and customization. However, the data sets required to improve the comprehensive performance of TTS often require enough high-quality voice audio and accurate transcriptions corresponding to these voice audios. Therefore, the number and cost of the above data sets seriously hinder the improvement of the comprehensive performance of TTS systems.

[0004] Specifically, the data sets required for TTS may include supervised data sets and unsupervised data sets (which can be understood as unlabeled speech data sets). Supervised data has both features and labels. After training, the TTS system can find the connection between features and labels. Unsupervised data has no clear annotations or labels, nor does it have corresponding transcripts. Compared with unsupervised data sets, supervised data sets are easier to improve the overall performance of the TTS system, but they are more difficult to collect and the collection cost is higher.

[0005] Therefore, the high cost and difficulty of creating supervised data sets seriously hinder the comprehensive development of TTS in terms of personalization and customization. Currently, there is a need for a TTS method and system that can reduce the demand for supervised data. Summary of the invention

[0006] The embodiments of the present application provide a speech synthesis method, system, electronic device and computer-readable storage medium that can at least partially solve the above-mentioned problems existing in the related art.

[0007] On the one hand, the present application provides a speech synthesis method, comprising: performing multi-stage multi-codebook quantization on an acoustic feature sequence of an input text to form a first quantization sequence; compressing the first quantization sequence into a second quantization sequence; and mapping the second quantization sequence into a speech waveform corresponding to the input text.

[0008] In one embodiment of the present application, multi-codebook quantization is performed on the acoustic feature sequence of the input text to form a first quantization sequence, including: converting the acoustic feature sequence into a multi-level sub-sequence with different time resolution stages; and using multiple predetermined codebooks to quantize each of the multi-level sub-sequences respectively to generate a multi-level quantization sub-sequence with different time resolution stages.

[0009] In one embodiment of the present application, at least one of the multiple predetermined codebooks is a multi-head codebook, and using the multiple predetermined codebooks to quantize each of the multi-level subsequences respectively to generate a multi-level quantized subsequence with the different time resolution stages includes: in at least one time resolution stage, using the multi-head codebook to perform multi-head vector quantization on the subsequence corresponding to the at least one time resolution stage to generate a quantized subsequence.

[0010] In one embodiment of the present application, the first quantization sequence includes a multi-level quantization subsequence with different time resolution stages, and compressing the first quantization sequence into a second quantization sequence includes: repeatedly sampling a part of the quantization subsequences in the multi-level quantization subsequences so that the multi-level quantization subsequences have the same length; and quantizing the multi-level quantization subsequences with the same length through a single codebook to form the second quantization sequence.

[0011] In one embodiment of the present application, the method also includes: extracting a speech-level embedding with global properties in the first quantization sequence, wherein mapping the second quantization sequence to a speech waveform corresponding to the input text includes: incorporating the speech-level embedding into the second quantization sequence; and mapping the incorporated second quantization sequence to the speech waveform.

[0012] In one embodiment of the present application, mapping the second quantization sequence into a speech waveform corresponding to the input text includes: decoding the second quantization sequence step by step according to a predetermined time resolution stage to form a multi-level prediction subsequence; and generating the speech waveform based on the multi-level prediction subsequence.

[0013] In one embodiment of the present application, within the same time resolution stage, at least one of the multi-level prediction sub-sequences corresponds to at least one of the multi-level sub-sequences of the first quantization sequence.

[0014] In one embodiment of the present application, the method is implemented by a speech synthesis system, and the method also includes training the speech synthesis system, and the training includes: based on multiple different time resolution stages, multi-codebook quantization of the acoustic feature sequence of the training set to form a first quantization sequence; mapping the first quantization sequence to a first waveform; compressing the first quantization sequence to a second quantization sequence; mapping the second quantization sequence to a second waveform; and adjusting the parameters of the speech synthesis system based on the first waveform and the second waveform to obtain the trained speech synthesis system.

[0015] In one embodiment of the present application, the training set includes a supervised data set and an unsupervised data set.

[0016] In one embodiment of the present application, the supervised dataset includes a low-resource language dataset.

[0017] In one embodiment of the present application, the training speech synthesis system further includes: pre-training the speech synthesis system using the unsupervised data set; and fine-tuning the speech synthesis system using the supervised data set.

[0018] In one embodiment of the present application, using the unsupervised dataset to pre-train the speech synthesis system includes: using the supervised dataset to train at least part of the unsupervised dataset to form a low-precision supervised dataset; and using the low-precision supervised dataset to pre-train the speech synthesis system.

[0019] On the other hand, the present application provides a speech synthesis system, which includes: a first learner, configured to perform multi-codebook quantization on an acoustic feature sequence of an input text based on multiple different time resolution stages to form a first quantization sequence; a second learner, configured to compress the first quantization sequence into a second quantization sequence; and a vocoder, configured to map the second quantization sequence into a speech waveform corresponding to the input text.

[0020] In one embodiment of the present application, the first learner is further configured to: convert the acoustic feature sequence into a multi-level sub-sequence with the different time resolution stages; and use multiple predetermined codebooks to quantize each of the multi-level sub-sequences respectively to generate a multi-level quantized sub-sequence with the different time resolution stages.

[0021] In one embodiment of the present application, at least one of the multiple predetermined codebooks is a multi-head codebook, and the first learner is further configured to: in at least one time resolution stage, use the multi-head codebook to perform multi-head vector quantization on a subsequence corresponding to the at least one time resolution stage to generate a quantized subsequence.

[0022] In one embodiment of the present application, the first quantization sequence includes a multi-level quantization subsequence with different time resolution stages, and the second learner is further configured to: repeatedly sample a part of the quantization subsequences in the multi-level quantization subsequences so that the multi-level quantization subsequences have the same length; and quantize the multi-level quantization subsequences with the same length through a single codebook to form the second quantization sequence.

[0023] In one embodiment of the present application, the second learner is further configured to: extract a utterance-level embedding with global properties in the first quantization sequence; incorporate the utterance-level embedding into the second quantization sequence; and map the incorporated second quantization sequence into the speech waveform.

[0024] In one embodiment of the present application, the vocoder is further configured to: decode the second quantized sequence level by level according to a predetermined time resolution stage to form a multi-level prediction subsequence; and generate the speech waveform based on the multi-level prediction subsequence.

[0025] In one embodiment of the present application, the vocoder is further configured to: within the same time resolution stage, at least one of the multi-level prediction subsequences corresponds to at least one of the multi-level subsequences of the first quantization sequence.

[0026] On the other hand, the present application provides an electronic device, which includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the speech synthesis methods provided in the first aspect of the present application.

[0027] In yet another aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any one of the speech synthesis methods provided in the first aspect of the present application.

[0028] According to the speech synthesis method, system, electronic device and computer-readable storage medium provided by at least one embodiment of the present application, based on multiple different time resolution stages, multi-codebook quantization is performed on the acoustic feature sequence of the input text to form a first quantization sequence, the first quantization sequence is compressed into a second quantization sequence, and the second quantization sequence is mapped to a speech waveform corresponding to the input text, while reducing the demand for supervision data and reducing the complexity of building a speech synthesis system, and achieving high-quality speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings, in which:

[0030] Figure 1 is a flow chart of a speech synthesis method according to one embodiment of the present application;

[0031] Figure 2 is a diagram of a TTS system according to one embodiment of the present application;

[0032] Figure 3 is a diagram of a TTS system according to one embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a first learner according to one embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a second learner according to one embodiment of the present application;

[0035] Figure 6 Amplitude specification diagram of samples synthesized by MSMC-TTS and QS-TTS according to one embodiment of the present application;

[0036] Figure 7 is a bar graph showing the impact of different codebook sizes on the second learner of Mandarin and Cantonese low-resource TTS according to one embodiment of the present application;

[0037] Figure 8 is a numerical performance curve diagram of FDs-ASRs and PERs of MSMC-TTS and QS-TTS using CSMSC data sets of different sizes according to one embodiment of the present application;

[0038] Fig. 9 is a schematic diagram of a speech synthesis system according to one embodiment of the present application; and

[0039] Fig.10 is a schematic diagram of an electronic device according to one embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] It should be noted that in this specification, the expressions first, second, third, etc. are only used to distinguish one feature from another feature region, but do not represent any limitation on the features, and especially do not represent any order of precedence.

[0042] In the drawings, the thickness, size and shape of the components have been slightly adjusted for ease of illustration. The drawings are for illustration only and are not drawn strictly to scale. As used herein, the terms "substantially," "approximately," and similar terms are used as terms of approximation, not as terms of degree, and are intended to account for the inherent deviations in measurements or calculations that would be recognized by one of ordinary skill in the art.

[0043] It should also be understood that expressions such as "include", "including", "have", "contain" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0044] Unless otherwise specified, all words (including engineering terms and scientific and technological terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that, unless clearly stated in this application, words defined in common dictionaries should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method described in the present application are not necessarily limited to the order described, but can be performed in any order or in parallel. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0046] In addition, in the present application, when “connected” or “coupled” is used, it may indicate that the corresponding components are in direct contact or indirect contact, unless otherwise clearly defined or can be inferred from the context.

[0047] Figure 1 It is a flowchart of a speech synthesis method 1000 according to one embodiment of the present application.

[0048] like Figure 1As shown, the present application provides a speech synthesis method 1000 including:

[0049] S1, based on multiple different time resolution stages, multi-codebook quantization is performed on the acoustic feature sequence of the input text to form a first quantization sequence.

[0050] S2, compressing the first quantized sequence into a second quantized sequence.

[0051] S3, mapping the second quantized sequence into a speech waveform corresponding to the input text.

[0052] The following will be combined Figures 2 to 9 The above-mentioned speech synthesis method 1000 is described in detail.

[0053] Creating TTS and improving its overall performance often requires a sufficient number of high-quality supervised datasets. However, when faced with a specific task, whether it is a corpus database or an image and video database, data with annotation information for the task itself usually only accounts for a small share of the total data volume, so self-supervised learning has great potential. In the field of speech synthesis, self-supervised learning refers to the process of learning meaningful speech representations from speech using self-supervised learning methods without any label information. Through self-supervised learning, TTS does not rely entirely on supervised datasets. For example, in TTS training, more unsupervised datasets can be used to compensate for insufficient supervised datasets.

[0054] In addition, for speech, there can be many labels, such as speakers, words, phonemes, etc. If only some of the labels, such as words, are used for learning, the expressiveness of the learned model will be insufficient. Self-supervised learning is not affected by the above, which makes the TTS system formed by self-supervised learning have very good generalization ability.

[0055] Figure 2 It is a TTS system diagram according to one embodiment of the present application.

[0056] Specifically, Figure 2 For example, a TTS system may include an analysis module, a prediction module, and a synthesis module, wherein the analysis module may extract acoustic features or speech representations from the waveform, which may be output by the prediction module and decoded into the waveform by the synthesis module. The speech representation of the derived data type has sufficient speech information and is easier to predict from the text. Figure 2In the TTS system shown, the unsupervised data set can first enhance the ability of the analysis module to provide speech representations of practical derived data types. In addition, the pre-trained model can also enhance the comprehensive module and prediction module of the TTS system by using the above unsupervised data set to complete related tasks in transfer learning. In this way, the present application uses the unsupervised data set to provide the TTS system with the expected features and suitable methods or tasks of the pre-trained model, and then establishes a semi-supervised TTS system by using more unsupervised data sets, and reduces the demand for supervised data sets for the semi-supervised TTS system.

[0057] Figure 3 It is a TTS system diagram according to one embodiment of the present application.

[0058] like Figure 2 and Figure 3 As shown, the steps of the speech synthesis method provided in this application, the specific structure of the speech synthesis system and the working principle of speech synthesis will be described in detail below, taking semi-supervised TTS (QS-TTS) based on vector quantized self-supervised speech representation learning (VQ-S3RL) as an example.

[0059] In the field of speech synthesis, supervised learning can be understood as knowing the relationship between input and output results based on existing data sets, and training to obtain the optimal TTS system based on the known relationship. Supervised learning can include regression and classification. In other words, in supervised learning, the training data has both features and labels. Through training, the TTS system can find the connection between features and labels, and can determine the labels when faced with data that only has features but no labels.

[0060] Compared with supervised learning, unsupervised learning means that the relationship between data and features in the data set is unknown. The training data of unsupervised learning only has features but no labels. The relationship between data can be obtained through clustering or models.

[0061] Self-supervised learning is a variant of unsupervised learning. It pre-trains the TTS system by solving a custom proxy task (pretexttask), and then migrates or fine-tunes the trained TTS system to solve specific tasks. In other words, self-supervised learning allows the TTS system to automatically use unlabeled data to improve learning performance without relying on external interactions.

[0062] In at least one embodiment of the present application, by means of VQ-S3RL that utilizes more unlabeled unsupervised data, QS-TTS can improve TTS quality with lower supervision data requirements.

[0063] Specifically, the QS-TTS system may include two learners composed of VQ-S3R (Vector-Quantized Self-Supervised Speech Representation), a first learner and a second learner. Optionally, the first learner may include a generative S3RL model and a contrastive S3RL (Contrastive Self-Supervised Speech Representation Learning) model. Optionally, the second learner may include a VQ (Vector-Quantized) encoder and a multi-stage decoder.

[0064] As an option, generating an S3RL model may include MSMC-VQ-GAN (Multi-Stage Multi-Codebook Vector-Quantized Generative Adversarial Network). In one embodiment of the present application, S3RL is used to establish a pre-trained model for a TTS system. S3RL can provide beneficial speech representations and pre-trained models for various supervised speech tasks. For example, in speech synthesis, it is possible to choose to generate S3RL based on vector quantization to establish a pre-trained model for a TTS system. Generating S3RL based on vector quantization can provide compact, discrete speech representations, which can achieve high-quality speech synthesis while reducing modeling complexity.

[0065] Generative S3RL aims to synthesize speech during training while imposing constraints on the latent space. Due to its overlapping goal of speech generation, it is more intuitively compatible with TTS. Generative S3RL is widely used in TTS tasks, such as autoregressive models, VAE (Variational Autoencoder) and VQ-VAE (Vector-quantizedVariational Autoencoder). Generative S3RL not only provides an effective speech representation, but also provides a good pre-trained model for TTS training.

[0066] The contrastive S3RL model is an encoder-based model trained with a contrastive loss function. The contrastive S3RL model is more robust to noisy data in training, so it can be used together with a large amount of low-quality speech data to learn a general speech representation to solve various downstream speech classification and synthesis tasks.

[0067] Furthermore, the comparison S3RL models may include Wav2Vec, HuBERT (Hidden-unit Bidirectional Encoder Representation from Transformers, Hidden-unit BERT), WavLM and other models. Among them, HuBERT can use clustering to provide labels for the loss used in BERT, and then use a mask operation loss similar to BERT to allow the model to learn the acoustic and language models in the data from continuous speech data.

[0068] Step S1

[0069] Figure 4 is a schematic diagram of a first learner according to one embodiment of the present application.

[0070] like Figure 3 and Figure 4 As shown, step S1 performs multi-codebook quantization on the acoustic feature sequence of the input text based on multiple different time resolution stages to form a first quantization sequence, which may for example include: converting the acoustic feature sequence into a multi-level sub-sequence with different time resolution stages; and using multiple predetermined codebooks to quantize each of the multi-level sub-sequences respectively to generate multi-level quantization sub-sequences with different time resolution stages.

[0071] Specifically, taking the example that the first learner may include a generated S3RL model and a compared S3RL model, and the generated S3RL model may include MSMC-VQ-GAN, the model structure of MSMC-VQ-GAN may include an MSMC-VQ encoder and a speech decoder. Figure 4 An implementation scheme including two stages with different time resolutions is shown, wherein the two stages with different time resolutions are the first stage and the second stage. MSMC-VQ-GAN is a model structure of a two-stage four-codebook VQ-GAN, wherein "MQ", They represent multi-head vector quantization, addition, and concatenation operations, respectively. The dotted arrows emphasize the process of using the trained first learner to convert the acoustic feature sequence into MSMCRs (Multi-Stage Multi-Codebook Representations) during inference, and converting MSMCRs into waveforms. Figure 4As shown, in the MSMC-VQ encoder, the input acoustic feature sequence z c First processed by the Transformer encoder, the processed acoustic feature sequence z c Quantization is performed in two subsequent stages to form a first quantized sequence. As an option, the Transformer encoder may include a linear layer and a feed-forward transformation block.

[0072] In the second stage, the processed acoustic feature sequence z c The second sampling sequence is obtained after being downsampled multiple times by the downsampling module along the time axis. Optionally, the downsampling module may include an average pooling layer and a feed-forward conversion block. The second sampling sequence can be Quantize to form the second-level quantization subsequence Furthermore, the second-level quantized subsequence It can also be processed by the upsampling block to help subsequent quantization and predict the next level of quantization sequence Optionally, the upsampling block may include two MLP (Multilayer Perceptron) layers, a Leaky ReLU (Rectified Line Unit) activation layer between the two MLP layers, a repeated operation layer for upsampling, and four residual convolutional layers.

[0073] In the first stage, a first predetermined codebook is used The lowest stage sequence with high-level information guidance after quantization can be obtained (first-level quantized subsequence), adding it to the high-order residual output, we can get the encoder's output first quantized sequence z P .

[0074] It should be noted that the first-level quantization subsequence and the second-level quantization subsequence are examples given for illustration and are not intended to limit actual situations. In actual situations, multiple levels of subsequences with a number exceeding 2 may be generated as required.

[0075] Optionally, the predetermined codebook in training, such as the first predetermined codebook and the second predetermined codebook, may be updated by an exponential moving average method.

[0076] In the above-mentioned embodiment, based on multiple different time resolution stages, the acoustic feature sequence is encoded into sub-quantization sequences of different time resolutions through multiple encoders and multiple codebooks, and the representation of this set of sequences is the MSMCRs proposed in this application. In other words, step S1 performs multi-codebook quantization on the acoustic feature sequence of the input text based on multiple different time resolution stages to form a first quantization sequence, which can be understood as obtaining a trained first learner through training, and using the trained first learner to convert the acoustic feature sequence of the input text into MSMCRs, so step S1 can also be understood as a step of performing multi-stage multi-codebook quantization on the acoustic feature sequence of the input text to form a first quantization sequence.

[0077] When the acoustic feature sequence is quantized, all kinds of information contained in the acoustic features are lost to varying degrees. This information is different in time granularity, such as coarse-grained timbre, pronunciation style, fine-grained pitch and pronunciation details. Excessive compression of information at any time scale may cause a certain degree of damage to the voice quality. In order to alleviate this problem, at least one embodiment of the present application proposes the above-mentioned multi-time scale speech representation MSMCRs.

[0078] Each subsequence is quantized using multiple predetermined codebooks. In QS-TTS, the above MSMCRs can be predicted by the acoustic model and inverted to the corresponding audio through the vocoder. By quantizing the speech sequence and converting it into a multi-level quantized subsequence, the information at different time resolution stages can be equally preserved, improving the compactness of the speech representation. In the corresponding QS-TTS system, the synthesized speech generated by the above compact speech representation can show better fidelity and more natural rhythm at both coarse and fine granularity.

[0079] Vector quantized representation learning, such as VQ-VAE and its enhanced VQ-GAN (Vector-Quantized Generative Adversarial Network) and VQ-Diffusion (Vector-Quantized Diffusion), is a generative self-supervised learning method that aims to compress the target into a compact discrete representation through vector quantization while maintaining high-quality reconstruction. Vector quantized representation learning has been widely used in speech generation, such as speech coding, voice conversion (VC), and TTS. In order to make the VQ speech representation both complete and compact, QS-TTS proposes a multi-stage multi-codebook representation, which includes multiple sequences with different time resolution stages and quantized separately using multiple predetermined codebooks. Multi-stage modeling can better predict from text and significantly improve TTS performance while reducing the system's demand for supervisory data.

[0080] In addition, in one embodiment of the present application, at least one of the multiple predetermined codebooks may be a multi-head codebook, and the QS-TTS method also includes: in at least one time resolution stage, using a multi-head codebook multi-head vector quantization (Multi-Head Vector Quantization, MHVQ) and a subsequence corresponding to at least one time resolution stage to generate a quantized subsequence.

[0081] Specifically, refer to Figure 4 In one embodiment of the present application, the second predetermined codebook is a multi-head codebook, the second sampling sequence Through MHVQ, the multi-head codebook is reserved Quantization, generate multi-head vector quantization sequence (Second level quantization subsequence). Predetermined multi-head codebook can be split into multiple sub-codebooks to quantize the input vectors that are divided in the same way. Optionally, the multi-head vector quantization sequence It is further processed by the upsampling block to help with subsequent quantization and predict the next level of quantization sequence.

[0082] Usually, the encoder processes the input acoustic feature sequence into a hidden sequence and quantizes it with a corresponding codebook, while the decoder reconstructs the quantized sequence into an acoustic feature sequence. This quantized sequence has good compactness as a discretized representation (which can be understood as a small number of feature parameters). The higher the degree of quantization, the smaller the codebook capacity, and the higher the feature compactness. However, this also forms information compression, which makes the feature completeness worse. In order to ensure that the discrete representation has sufficient completeness, a codebook with more codewords can be used. However, as the codebook capacity increases, the amount of data required for codebook updates and the number of training times will increase exponentially, which makes it difficult to effectively enhance the representation completeness by increasing the codebook. To solve this problem, at least one embodiment of the present application proposes MHVQ.

[0083] MHVQ divides a single codebook into multiple sub-codebooks (which can be understood as multi-head codebooks) in the feature dimension direction. During quantization, each input vector is equally cut into multiple sub-vectors, and each is quantized with the corresponding sub-codebook to generate a quantized subsequence. The output vector can be obtained by splicing the quantized subsequences. This can effectively improve the codebook utilization rate and representation capacity without increasing the codebook parameters. On this basis, the completeness of the discretized representation can be effectively adjusted.

[0084] Step S2

[0085] Figure 5 is a schematic diagram of a second learner according to one embodiment of the present application.

[0086] like Figure 3 and Figure 5 As shown, step S2 compresses the first quantization sequence into a second quantization sequence, which may for example include: repeatedly sampling a portion of the quantization subsequences in the multi-level quantization subsequences so that the multi-level quantization subsequences have the same length; and quantizing the multi-level quantization subsequences with the same length through a single codebook to form a second quantization sequence.

[0087] Specifically, the second learner aims to further abstract MSMCRs into more compact VQ sequences through a VQ-VAE based model.

[0088] Figure 5 The model structure of the two-stage four-codebook VQ-VAE is shown. As an option, the second learner may include a VQ encoder and a multi-stage decoder. In the second stage, the second-stage quantized subsequence z p (2) is downsampled, and the downsampling rate along the time axis can be 4. In the VQ encoder, all sequences are first upsampled to the lowest stage sequence z p (1) The same length, for example, repeating the second-level quantized subsequence z p (2) The aligned sequences are concatenated together, processed by the feedforward transform block, and quantized by a single codebook to obtain a highly compact VQ second quantized sequence z a .

[0089] The first learner provides MSMCRs through the combination of MSMC-VQ-GAN and contrast S3RL, and the second learner further abstracts MSMCRs into highly compact VQ representations through VQ-VAE. Therefore, the two generative VQS3R learners (the first learner and the second learner) provide useful speech representations and pre-training models for speech synthesis, significantly improving the quality of speech synthesis while placing lower demands on supervision data.

[0090] In addition, in one embodiment of the present application, the speech synthesis method also includes: extracting a speech-level embedding with global properties in the first quantization sequence; incorporating the speech-level embedding into the second quantization sequence; and mapping the incorporated second quantization sequence into a speech waveform.

[0091] Specifically, Figure 5 As shown, in order to make the second quantization sequence z aTo obtain more information related to speech, a global encoder based on, for example, ECAPA-TDNN (Emphasized Channel Attention Propagation and Aggregation in Time Delay Neural Networks) can be used to extract a speech-level embedding that represents a global attribute, such as speaker information from MSMCRs. The above embedding can be upsampled to the frame level through repeated operations and combined with the second quantized sequence z a Added for subsequent decoding.

[0092] Optionally, the second learner may update the predetermined codebook in training by using an exponential moving average method during training.

[0093] Step S3

[0094] like Figure 3 and Figure 5 As shown, step S3 maps the second quantization sequence to a speech waveform corresponding to the input text, which may for example include: decoding the second quantization sequence step by step according to a predetermined time resolution stage to form a multi-level prediction subsequence; and generating a speech waveform based on the multi-level prediction subsequence.

[0095] Specifically, QS-TTS synthesis aims to convert text into corresponding MSMCRs through an acoustic model, then convert MSMCRs into compact and abstract VQ sequences, and then predict MSMCRs based on VQ sequences through a vocoder and generate speech waveforms from MSMCRs. Optionally, the acoustic model can be MSMC-TTS based on the Fast Speech model.

[0096] Optionally, the acoustic model encodes the text sequence using a Transformer encoder, then repeatedly upsamples according to a predetermined duration and generates an MSMCR prediction sequence through a multi-stage decoder It includes a multi-level predictor sequence. During training, the acoustic model inherits the parameters of the pre-trained multi-level decoder in the associated VQ-S3R learner.

[0097] The multi-stage decoder aims to reconstruct the a The MSMCRs of the above. Optionally, the multi-stage decoder can predict the sequence from the high stage to the low stage in a cascaded manner. For example, for the above two-stage representation, the multi-stage decoder can first downsample the output of the encoder by 4 times and then send it to the decoding module, where the decoding module can include a feed-forward conversion block and a linear layer located thereafter. The output of the linear layer can be understood as the predicted second-stage quantized subsequence in training The output multi-stage multi-codebook representation sequence The multi-head codebook c of MSMC-VQ-GAN can be p (2) Similarly, in the first stage, the lowest stage sequence is predicted (The predicted first-level quantized subsequence) can be predicted from the output of a given encoder and the output of a higher stage.

[0098] Optionally, within the same time resolution stage, at least one of the multi-level prediction sub-sequences corresponds to at least one of the multi-level sub-sequences of the first quantization sequence.

[0099] In other words, in a multi-level decoder, by decoding the second quantized sequence step by step according to a predetermined time resolution stage, a multi-level predicted subsequence can be formed, and the predicted multi-level predicted subsequence can be restored to a multi-level subsequence with different time resolution stages formed after multi-stage multi-codebook quantization of the acoustic feature sequence of the input text. By adjusting the parameters of the QS-TTS system during training, the above two can be made close, so that the decoder can generate the speech waveform corresponding to the input text according to the compact and abstract second quantized sequence.

[0100] In addition, in some embodiments of the present application, the speech synthesis method is implemented by a speech synthesis system, and the speech synthesis method also includes training the speech synthesis system. The training of the speech synthesis system may include: performing multi-stage multi-codebook quantization on the acoustic feature sequence of the training set to form a first quantization sequence; mapping the first quantization sequence to a first waveform; compressing the first quantization sequence to a second quantization sequence; mapping the second quantization sequence to a second waveform; and adjusting the parameters of the speech synthesis system based on the first waveform and the second waveform to obtain a trained speech synthesis system.

[0101] As mentioned above, the training set may include supervised data sets and unsupervised data sets. Figure 3 and Figure 4 As shown, in the process of training the first learner, a training set including a supervised data set and an unsupervised data set can be used to train the first learner. Training the first learner based on the unsupervised data set can provide a more favorable speech representation and an effective pre-training model, wherein the unsupervised data set can include a large amount of low-quality speech data and high-quality speech data.

[0102] As an option, a large amount of low-quality speech data can be used to train the comparative S3RL model, and high-quality speech data can be used to train the MSMC-VQ encoder and speech decoder.

[0103] Based on the above selection, the comprehensive performance of QS-TTS can be improved while reducing its demand for supervision data. Specifically, in one embodiment of the present application, the first learner may include a comparison S3RL model and a generation S3RL model, a large amount of low-quality speech data is used to train the comparison S3RL model, and high-quality speech data is used to train the generation S3RL model. Optionally, when the generation S3RL model includes MSMC-VQ-GAN, MSMC-VQ-GAN includes an MSMC-VQ encoder and a speech decoder, and high-quality speech data can be used to train the MSMC-VQ encoder and the speech decoder.

[0104] The trained contrast model can extract effective and universal acoustic feature sequences z from the speech signal s. c Then, based on the generated VQ-S3RL model of MSMC-VQ-GAN, the acoustic feature sequence z c Convert to the first quantized sequence z P , while decoding it into high-quality audio

[0105] In one embodiment of the present application, the first quantized sequence z output by the encoder may be P Feed to the speech decoder to generate speech waveform The speech decoder may include a frame decoder and a waveform generator. First, the entire input sequence is processed using a frame decoder with a feedforward conversion block, and then the speech waveform is generated by a waveform generator based on the Hifi-GAN model.

[0106] Furthermore, as with MSMC-VQ-VAE, a mel-spectrogram is predicted from the output of the frame decoder using, for example, a mel-linear layer.

[0107] In addition, supervised datasets can also be used to train speech decoders, text encoders, and multi-stage decoders, etc. In other words, in the process of training a speech synthesis system, an unsupervised dataset can be used to pre-train the speech synthesis system, and a supervised dataset can be used to fine-tune the speech synthesis system.

[0108] For example, the speech decoder can be fine-tuned using the audio from the supervised dataset to better fit the target speaker. In addition, the same training configuration as MSMC-VQ-GAN can be used during fine-tuning, and only the parameters of the speech decoder are updated. It is fed into the MSMC-VQ encoder of the pre-trained MSMC-VQ-GAN to obtain the encoder output sequence, which is then passed through the fine-tuned speech decoder to synthesize the speech waveform.

[0109] Optionally, a quantized true value sequence z can be used in training p(2) Replace the MSMCR prediction sequence generated by the multi-stage decoder as the input to the speech decoder.

[0110] In VQ-S3RL, for example, the AIShell-3 speech dataset can be used as a training set. The AIShell-3 speech dataset includes 85 hours of high-quality training data from 218 native Mandarin Chinese speakers, covering rich speech and speaker information.

[0111] Optionally, HIA can use multiple supervised datasets to evaluate TTS systems in various scenarios during TTS training. For example, the multiple supervised datasets can include at least the CSMSC1 speech dataset. The CSMSC1 speech dataset includes 10 hours of high-quality supervised data from a single speaker whose native language is Mandarin Chinese, and can be widely used in the training of standard monolingual Mandarin TTS systems.

[0112] In addition, multiple supervised data sets may also include other supervised data sets corresponding to low resource scenarios. As a choice, low resource scenarios may be introduced by extracting a subset from the CSMSC speech data set. In addition, a test set without overlap with the training set may also be extracted from the CSMSC speech data set. Optionally, the test set may include 200 utterances.

[0113] As another option, a more challenging low-resource scenario can be constructed using only 10 minutes of Mandarin speech data from a five-year-old girl. Similarly, a test set with no overlap with the training set can be constructed using this supervised dataset, which can include 24 utterances.

[0114] In addition, in one embodiment of the present application, multiple supervised data sets may also include low-resource language data sets. The low-resource language data set may be, for example, a Cantonese data set, which has more pronunciations than the Mandarin data set, and thus may include pronunciations that supplement the above-mentioned training set. In addition, the low-resource language data set may include, for example, supervised data of 15 minutes in length. The low-resource language data set may evaluate the performance of the TTS system under low-resource conditions or low-resource scenarios.

[0115] In addition, in one embodiment of the present application, pre-training a speech synthesis system using an unsupervised dataset may include: training at least part of the unsupervised dataset with a supervised dataset to form a low-precision supervised dataset; and pre-training the speech synthesis system using the low-precision supervised dataset.

[0116] For example, an ASR (Automatic Speech Recognition) system can be trained for, for example, a few minutes, using a relatively small amount of supervised data, and then a large amount of unlabeled audio is decoded to generate a corresponding transcript, thereby creating a low-precision supervised dataset. QS-TTS is pre-trained directly using the above-mentioned low-precision supervised dataset and then fine-tuned using the target supervised data. In addition, pseudo-labels can be created, for example, using unsupervised acoustic units to form a pseudo-supervised dataset for QS-TTS pre-training. Ultimately, using more unlabeled speech unsupervised data in training can improve training efficiency and reduce the TTS system's demand for supervised data.

[0117] In one embodiment of the present application, adjusting the parameters of a speech synthesis system based on a first waveform and a second waveform to obtain a trained speech synthesis system may include: generating a system loss function based on the first waveform and the second waveform; and adjusting the parameters of the speech synthesis system based on the system loss function to obtain a trained speech synthesis system.

[0118] Specifically, the system loss function may include a loss function of the first learner, a loss function of the second learner, and an acoustic model loss function.

[0119] Optionally, the loss function of the first learner includes multiple loss terms. For example, considering that quantization is a non-differentiable operation, as an option, the vector quantization loss can be expressed as follows: Back-propagate gradients to the MSMC-VQ encoder.

[0120]

[0121] in, is the vector quantization loss; S is the number of stages; is the i-th sampling sequence in the i-th stage; and is the i-th level quantized subsequence in the i-th stage.

[0122] In order to enhance the role of multi-stage multi-codebook representation learning, the following formula (2) can also be used to enhance the multi-stage vector quantization loss: Enhance the relationship between two adjacent stages, where the multi-stage vector quantization loss This is conducive to the stable learning of effective representations at a high level.

[0123]

[0124] in, is the multi-stage vector quantization loss; S is the number of stages; Predicting the j-th level quantized subsequence in the j-th stage; and is the j-th level quantized subsequence in the j-th stage.

[0125] In order to synthesize high-quality speech audio, the speech synthesis system may include a discriminator, for example, a GAN training model with a UnivNet discriminator may be applied, and the UnivNet discriminator includes multiple sub-discriminators for multi-resolution spectrogram recognition and multi-period waveform recognition to capture more discriminative information in the frequency domain and the time domain. Discriminator loss It can be expressed by the following formula (3).

[0126]

[0127] in, is the discriminator loss; k is the number of sub-discriminators; s is the speech signal; and is the synthetic speech signal corresponding to the speech signal s.

[0128] In addition, the adversarial loss of the first learner’s generative S3RL model MSMC-VQ-GAN It can be expressed by the following formula (4).

[0129]

[0130] in, The adversarial loss of the generated S3RL model for the first learner; is a synthetic speech signal corresponding to the speech signal s; and k is the number of sub-discriminators.

[0131] In order to improve the quality of GAN training, the Mel-spectrogram loss and feature matching loss, which are widely used in GAN-based neural network vocoder training, are also used in MSMC-VQ-GAN training. is the L1 norm of the two waveforms in the Mel-scale frequency domain, and the Mel-spectrogram loss It can be expressed by the following formula (5), where the Mel spectrum loss This improves the perceptual quality of the generated audio.

[0132]

[0133] in, is the Mel-spectrogram loss; s is the speech signal; is the synthesized speech signal corresponding to the speech signal s; and φ is the Mel-spectrogram converted from the waveform.

[0134] Optionally, the speech synthesis system may include a discriminator, a feature matching loss The quality of GAN training can be further improved by reducing the difference between the real waveform and the generated waveform in the hidden feature space of the discriminator. Feature matching loss It can be expressed by the following formula (6).

[0135]

[0136] in, is the feature matching loss; k is the number of sub-discriminators; N k is the number of hidden layers of the k-th sub-discriminator; and is the output feature of the i-th hidden layer of the k-th sub-discriminator layer.

[0137] In addition, a frame-level reconstruction loss can be applied to avoid the negative impact of GAN on the unstable performance of VQ-S3RL in the early training. It can be the L2 norm between the predicted Mel-spectrogram and the true value, and is expressed by the following formula (7).

[0138]

[0139] in, is the frame-level reconstruction loss; and x and Mel-spectrograms of true value and prediction respectively.

[0140] Therefore, the loss function of the first learner is It may include at least one of the above losses. Optionally, the loss function of the first learner is It can be expressed by the following formula (8).

[0141]

[0142] in, is the loss function of the first learner; and λ fm , mel , vq , ms and λ frame are all weight coefficients.

[0143] Optionally, the loss function of the second learner is It can be expressed by the following formula (9).

[0144]

[0145]

[0146]

[0147] in, is the loss function of the second learner; S is the number of stages; is the quantized subsequence of level i in stage i Predict the i-th level quantized subsequence in the i-th stage; is the vector quantization loss of the VQ encoder; is the reconstruction loss between the true value and the reconstructed MSMCR; and λ rec is the weight coefficient.

[0148] Optionally, during training, the acoustic model inherits the parameters of the pre-trained multi-stage decoder in the second VQ-S3R learner. In addition, the acoustic model can also be trained by a supervised dataset. The training loss of the acoustic model can be expressed by the following formula (10).

[0149]

[0150]

[0151] in, is the loss of the speech synthesis system; d is the actual duration; To predict the duration; is the actual duration d and the predicted duration The time loss between dur is the weight coefficient.

[0152] In a specific embodiment of the training process, all mono audio is downsampled to a sampling frequency of 16kHz. Then, the Mel spectrograms of all data sets are extracted and obtained by the following method: pre-emphasize the audio with a pre-emphasis coefficient of 0.97; convert the pre-emphasized audio into a 1025-dimensional magnitude spectrum by short-time Fourier transform (SFFT), where the parameters of SFFT may include a window length of 50ms, a frame shift of 12.5ms, and an FFT of 2048; compress the 1025-dimensional magnitude spectrum into an 80-dimensional Mel spectrogram.

[0153] In addition, as an option, HuBERT, the S3RL model, and the WenetSpeech dataset can be used to extract general speech representations, where the WenetSpeech dataset includes more than 10,000 hours of Mandarin supervision data. HuBERT includes 1024 vectors with a 20ms frame shift. In order to align it with the Mel spectrogram, it can be upsampled by nearest neighbor interpolation to a 12.5ms frame shift.

[0154] In text processing, text can be converted into phonemes as input to the acoustic model. For the CSMSC speech dataset and its sub-datasets, the phonemes and their corresponding durations marked in the dataset for training can be used directly. For low-resource language datasets, such as the Cantonese dataset, phonemes can be obtained using, for example, the open source G2P tool, and the Montreal Forced Aligner (MFA) can be trained to obtain the phoneme level durations for training.

[0155] Optionally, VQ-S3RL is constructed using a two-stage MSMC-VQ-GAN, where the second stage has a downsampling with a sampling rate of 4 along the time axis. At each stage, a 4-head codebook can be used for vector quantization, where each head can include 64 codewords, each codeword has 64 dimensions. In the MSMC-VQ encoder, we use a 4-layer 256-dimensional feedforward transformation block, where the feedforward transformation block has a 2-head self-attention Transformer encoder and a downsampling block. The MLP block includes two 256-dimensional linear layers with Tanh activation function. The residual CNN block includes 4 1-dimensional residual convolution layers with a particle size of 5. In the speech decoder, the frame decoder may include a 4-layer 256-dimensional feedforward transformation block. The waveform generator may be, for example, a Hifi GAN V1 generator, which can upsample the input sequence, for example, 200 times, to form a 16kHz speech waveform through 4 CNN-based upsampling modules. The upsampling ratio may be, for example, [5, 5, 4, 2], and the kernel size may be, for example, [11, 11, 8, 4].

[0156] In the GAN training based on the UnivNet discriminator, three STFT parameter sets can be used to extract three amplitude spectra from the speech waveform for multi-resolution spectrogram discrimination. Among them, the FFT size can be, for example, [256, 512, 1024], the frame shift can be, for example, [40, 80, 160], and the frame length can be, for example, [120, 320, 640]. In addition, the one-dimensional speech waveform can be reshaped into five two-dimensional speech waveforms for multi-period waveform identification. Among them, the period of the multi-period is [2, 3, 5, 7, 11]. Optionally, MSMC-VQ-GAN can be trained using the AIShell-3 speech dataset. For example, the AdamW optimizer (parameter β1=0.8, parameter β2=0.99) is used for, for example, 400,000 iterations, with a batch size of 16 and a learning rate of 2×10 -4 , after 200,000 warm-up iterations, the exponential decay rate is Weight coefficient λ fm , mel , vq , ms and λframe They can be, for example, 2, 45, 10, 1, and 450, respectively. Warm-up training can be understood as a stable training process, in which there is no GAN training in the first 50,000 iterations.

[0157] The second VQ-S3R learner uses a single codebook to compress MSMCRs, where the single codebook may include 64 256-dimensional codewords. The VQ encoder may be, for example, a Transformer encoder, which has the same configuration as the MSMC-VQ-GAN. The global encoder may be, for example, an ECAPA TDNN (mainstream voiceprint model, Emphasized ChannelAttention, Propagation and Aggregation in Time Telay Neural Network based Speaker Verificaiton) model. The model has 128 channels for hidden layers. Optionally, the above model can be trained using the AIShell-3 speech dataset. For example, using the Adam optimizer (parameter β1=0.9, parameter β2=0.98) for, for example, 400,000 iterations,

[0158] The Adam optimizer (parameter β1 = 0.9, parameter β2 = 0.98) was used for 200,000 iterations, with a batch size of 64 and a learning rate of 2 × 10 -4 , after 200,000 warm-up iterations, the exponential decay rate is

[0159] Alternatively, the same training configuration can be used in TTS training to fine-tune the vocoder of MSMC-VQ-GAN using a supervised dataset. Alternatively, the acoustic model can also be trained based on the multi-stage decoder of the pre-trained second VQ-S3R learner, where λ dur =0.1.

[0160] In addition, for standard training sets with sufficient audio, fine-tuning can be achieved through 400,000 iterations of the vocoder and 200,000 iterations of the acoustic model; for low-resource datasets that lack sufficient audio, in order to avoid over-tuning, fine-tuning can be achieved through 100,000 iterations of the vocoder and 50,000 iterations of the acoustic model.

[0161] The QS-TTS provided by at least one embodiment of the present application has been comprehensively evaluated through subjective and objective tests in experiments in various scenarios, and the test results have strongly demonstrated the superior performance of QS-TTS, especially in resource-poor conditions, where it has achieved the highest mean opinion score (MOS) for audio naturalness compared to other supervised or semi-supervised baseline TTS methods. In addition, by comparing various speech representation and transfer learning methods in TTS, it is further verified that the application of VQ-S3RL in TTS significantly improves the overall performance of TTS, showing the best audio quality and comprehensibility indicators. As the supervised data gradually decreases, the overall quality of QS-TTS tends to decay slowly, which further highlights the low demand for supervised data by QS-TTS, indicating its great potential in low-resource conditions.

[0162] Specifically, the objective index for evaluating QS-TTS may include Frech'et Distances (FDs) in various embedding spaces. Audio quality may be evaluated by, for example, Frech'et Distances in various embedding spaces. Frech'et Distances may determine the distance between two sample sets by calculating the difference between the two sample sets. For details, refer to formula (11).

[0163]

[0164] in, are the normal distributions of the synthetic audio set and the target real audio set in the embedding space; μ s , μ t is the average vector of the synthetic audio set and the target real audio set; and Σ s ,Σ t is the covariance matrix of the synthetic audio set and the target real audio set.

[0165] As an option, we can extract embeddings using different audio classification models and compute their mean vector (μ s , μ t ) and the covariance matrix (Σ s ,Σ t ). For example, the following three embedding spaces can be used: acoustic, speaker, and speech. Fred's distance in the acoustic space, such as Fred's audio distance, can be used to evaluate the overall quality of the neural network vocoder. Specifically, the embedding amount relative to every 4 seconds of audio from the audio classification model can be extracted. Similarly, a speaker verification model based on ECAPA-TDNN can also be used to extract speech embedding at the utterance level by averaging its encoder output sequence into a vector.

[0166] Alternatively, the above three distances may be represented by FDs-AC, FDs-SV, and FDs-ASR. It is worth noting that in order to align Fred's distance in speaker space with other distances, Fred's distance in speaker space may be multiplied by 10.

[0167] In addition, in one embodiment of the present application, error rates (ERs), such as character error rate (CER) or phone number error rate (PER), may be used to evaluate audio clarity. Specifically, the G2P tool may be used to convert the transcribed text and the real text into phonemes to calculate the PER.

[0168] The subjective indicators for evaluating QS-TTS may include MOS testing to conduct a comprehensive evaluation of the TTS system based on subjective evaluation.

[0169] For example, in each MOS test, 10 native speakers can be selected to score each audio sample in 20 test cases, where each test case can contain multiple audio samples synthesized by different TTS systems or methods, and multiple audio samples come from the same real text. The rating range is 1 to 5, with an increment of 0.5, where a higher score indicates a better quality TTS system or method. In addition, the score of each method can be statistically calculated to obtain a MOS with a 95% confidence interval.

[0170] In one embodiment of the present application, the proposed semi-supervised QS-TTS system is compared with other supervised TTS systems, including Fast Speech, VITS (Very Deep Iterative Strategy) and MSMC-TTS (Multi-Stage Multi-Codebook TTS). Fast Speech is a mel-spectrogram-based TTS system that predicts mel-spectrograms from text and generates speech waveforms through a neural network vocoder. SOTA (Sate-Of-The-Art) is an end-to-end TTS system, VITS learns latent representations from audio via a VAE-GAN model and uses a Glow-based model to predict latent representations. MSMC-TTS can be regarded as a fully supervised version of QS-TTS that uses MSMC-VQ-GAN to learn MSMCR from mel-spectrograms and is trained only using supervised datasets.

[0171]

[0172] Table 1

[0173] Table 1 compares the speech synthesis effects of various TTS methods using the 10-hour CSMSC dataset for standard single-person Mandarin. As shown in Table 1, the modular TTS system based on Mel-spectrogram shows that Fast Speech has the worst synthesis quality among all TTS methods, and obtains the lowest MOS, which is 3.55; the highest FD-AC and FD-SV, whose values ​​are 0.79 and 1.19 respectively. VITS based on VAE can significantly improve the audio quality and has a higher MOS, which is 3.74. However, compared with Fast Speech, the clarity of VITS is seriously reduced with the increase of CER and PER. On the contrary, MSMC-TTS ensures higher clarity and synthesis quality, with a MOS value of 3.99, which shows the effectiveness of the VQ-based speech representation method. Similarly, QS-TTS inherits this idea and uses more unsupervised speech data by proposing VQ-S3RL.

[0174]

[0175] Table 2

[0176] Table 2 compares the speech synthesis effects of each TTS method using a low-resource dataset as a standard single-person training set. As shown in Table 2, all TTS methods implemented by supervised learning performed poorly in terms of audio, with very low subjective scores for quality and clarity. Although VITS and MSMC-TTS have better audio quality than Fast Speech, due to the lack of sufficient data to learn effective speech representations, the performance of VITS and MSMC-TTS in terms of intelligence is more severely degraded. QS-TTS has achieved significant improvements in all aspects by training with more unsupervised speech data. Its audio quality has been significantly improved, with an FDs-AC value of 0.25 and an FDs-SV value of 1.00; its clarity has also been significantly improved, with a CER value of 8.98 and a PER value of 1.46; in addition, the MOS value of 3.75 further verifies that QS-TTS greatly exceeds all TTS methods implemented by supervised learning in this low-resource situation.

[0177] Figure 6 Amplitude specification diagram of samples synthesized by MSMC-TTS and QS-TTS according to one embodiment of the present application.

[0178] like Figure 6 As shown, the upper figure is a visualized amplitude specification diagram of the sample synthesized by QS-TTS; the lower figure is a visualized amplitude specification diagram of the sample synthesized by MSMC-TTS. The horizontal axes of the lower and upper figures are time (Time), in seconds (Seconds), and the vertical axes are frequency (Frenquency), in Hertz (HZ).

[0179] The differences between the two can be further discovered by visualizing the amplitude specification diagrams of the samples synthesized by MSMC-TTS and QS-TTS. Figure 6 As shown in the figure above, high-quality audio usually produces clear and smooth harmonics in the mid- and low-frequency parts. Figure 6 As shown in the figure below, MSMC-TTS, which was trained with only 15 minutes of supervised data, cannot synthesize the expected harmonics. As shown in the black box, jittery and blurred mid-frequency harmonics are presented, which leads to a decrease in the quality and intelligibility of the audio synthesized by MSMC-TTS. In QS-TTS, these problems are significantly alleviated. The synthesized audio shows a clearer spectrogram with smoother harmonics, resulting in a higher MOS value in subjective tests.

[0180]

[0181] Table 3

[0182] Table 3 compares the speech synthesis effects of different semi-supervised TTS methods using a 10-minute (min) children's speech dataset. As shown in Table 3, different semi-supervised TTS methods show higher Fried space distances and error rates. MSMC-TTS, the SOTA supervised method, has the worst clarity, with CER and PER values ​​as high as 27.53% and 13.02%, and also performs poorly in audio quality, with the highest FDs-AC value. FastSpeech-SS is a semi-supervised version of FastSpeech, which achieves comparable performance to MSMC-TTS compared to the fully supervised version. In addition, VITS-SS, enhanced by semi-supervised training, significantly outperforms MSMC-TTS, achieving lower Fried distances and error rates. Although QS-TTS has a higher FDs-AC than VITS-SS, QS-TTS achieves the best performance in all other indicators, especially the CER value of 14.61% and the PER value of 5.35%, which are roughly lower than half of the relative values ​​of MSMC-TTS.

[0183] This demonstrates the effectiveness of QS-TTS in various extreme scenarios such as low-resource scenarios.

[0184]

[0185] Table 4

[0186] Table 4 compares the speech synthesis effects of various semi-supervised TTS methods using a 10-minute Cantonese dataset. As shown in Table 4, this dataset has a relatively low audio fidelity and low prosodic expressiveness, so the dataset itself (recording) only obtains a MOS value of 3.86. FastSpeech-SS has a significant improvement over MSMC-TTS in this task, obtaining a MOS value of 3.58 and having better audio quality and clarity. VITS-SS failed to maintain stable performance, and its clarity was severely degraded, with CER and PER values ​​as high as 42.35% and 19.92% respectively. The pseudo labels extracted from the Mandarin dataset cannot be well adapted to Cantonese, and seriously provide an inaccurate pre-training model. As a result, the lowest MOS value was obtained. QS-TTS still performed best, with the highest MOS value, the lowest CER and PER, and the Fried space distance value was lower than the corresponding values ​​of other methods, showing the quantitative audio quality.

[0187]

[0188] Table 5

[0189] Table 5 compares the performance of different speech representations in analysis synthesis and TTS synthesis. As shown in Table 5, the performance of different speech representations in analysis synthesis and TTS synthesis is compared, such as FastSpeech based on Mel-spectrogram, corresponding to speech representation without HuBERT and MSMC-VQ; FastSpeech-SS based on HuBERT corresponds to speech representation using only HuBERT; MSMC-TTS based on Mel-spectrogram corresponds to speech representation using only MSMC-VQ; QS-TTS uses the speech representation of the proposed VQ-S3R based on both HuBERT and MSMC-VQ.

[0190] In this experiment, all vocoders or autoencoders were pre-trained on the AIShell-3 dataset and fine-tuned for 15 minutes on the CSMSC dataset. In addition, for comparison purposes, QS-TTS did not use the associated VQ-S3R learner.

[0191] In terms of analysis and synthesis, Fast Speech with Mel-spectrogram performs best because Mel-spectrogram contains enough acoustic information required to complete speech synthesis. However, in low-resource scenarios, it is difficult for acoustic models to predict rich feature information, which will result in poor audio quality of the related TTS. MSMC-VQ compresses the Mel-spectrogram and converts it into a more compact representation with less information, thereby reducing the quality of analysis and synthesis. The features processed in this way are easier to predict and can improve the audio quality of the related TTS. However, the speech features processed in this way also lead to information loss, so the audio clarity is slightly reduced. Based on this, MSMC-VQ needs to balance audio quality and audio clarity. Therefore, it is not advisable to compress features too much in MSMC-TTS. HuBERT discards more acoustic information to learn the opposite speech representation from a large amount of speech audio, and thus shows lower audio quality in the analysis synthesizer.

[0192]

[0193] Table 6

[0194] Table 6 shows the performance of different transfer learning methods in Mandarin and Cantonese TTS. As shown in Table 6, in one embodiment of the present application, 15 minutes of Mandarin supervision data and Cantonese supervision data are used to evaluate the impact of transfer learning on QS-TTS.

[0195] For the above two languages, by using the trained MSMCRs and training the speech decoder and acoustic model from scratch, the TTS system without transfer learning can perform well in the fully supervised MSMC-TTS. This once again verifies the effectiveness of the VQ-S3R application proposed in this application in TTS. Specifically, by applying transfer learning to the vocoder and fine-tuning the speech decoder, the audio quality and clarity are significantly improved, with lower Fried space distance and error rate. In addition, by adding an additional VQ-S3R learner, such as the second learner, the system with transfer learning achieves the best performance, thus verifying that the VQ-S3RL model plays an important role in improving the quality of TTS as a pre-training model for transfer learning TTS.

[0196] Figure 7 It is a bar chart showing the influence of different codebook sizes on the second learner of Mandarin and Cantonese low-resource TTS according to one embodiment of the present application.

[0197] like Figure 7As shown in the figure, three VQ-VAE models with codebook sizes of 4, 64, and 1024 can be trained for Mandarin and Cantonese. The results show that the TTS system that uses Mandarin, the same language as the pre-trained model, is more suitable for smaller codebooks, while the Cantonese TTS system that uses a different language than the pre-trained model is more suitable for larger codebooks. This is because the highly compact VQ sequences extract speech information in the Mandarin training set, but due to the lack of generalization to cross-lingual representations, larger codebooks are beneficial for Cantonese TTS.

[0198] In one embodiment of the present application, a codebook with a codebook size of 64 may be selected, which can balance the degree of compactness and universality of the system setting parameters.

[0199] Figure 8 It is a numerical performance curve diagram of Fred's distance in language space (FD-ASR) and telephone number error rate (PER) of MSMC-TTS and QS-TTS using CSMSC data sets of different sizes according to one embodiment of the present application.

[0200] like Figure 8 As shown in the figure, with sufficient 10 hours (10h) of supervised data, fully supervised MSMC-TTS and semi-supervised QS-TTS achieve similar and good TTS performance. As the size of the supervised data decreases, the performance of MSMC-TTS decreases significantly, and its FD-ASR curve grows exponentially, which results in poor speech synthesis performance of MSMC-TTS in low-resource scenarios using only 15 minutes (15m) of supervised data. The FD-ASR curve of QS-TTS shows a slower linear decay, and as the supervised data decreases, the gap between it and the FD-ASR curve of MSMC-TTS gradually widens, indicating that QS-TTS has better performance in low-resource scenarios.

[0201] In addition, the two also have similar performances in terms of clarity. When the size of the supervision data is 10 hours (10h), 6 hours (6h) or 2 hours (2h), the FER curves of MSMC-TTS and QS-TTS are similar. However, as the size of the supervision data decreases, the difference between the FER curves of the two expands rapidly. The FER curve of MSMC-TS has serious pronunciation problems in low-resource scenarios, and QS-TTS with a lower PERs value still has good comprehensibility in low-resource scenarios. Therefore, QS-TTS can still achieve high-quality speech synthesis in low-resource scenarios and has lower requirements for supervision data.

[0202] Fig. 9 is a schematic diagram of a speech synthesis system 2000 according to an embodiment of the present application. Fig. 9As shown, another aspect of the present application also provides a speech synthesis system 2000. The speech synthesis system 2000 includes a first learner 2100, a second learner 2200 and a vocoder 2300. The first learner 2100 is configured to be configured to perform multi-codebook quantization on the acoustic feature sequence of the input text based on multiple different time resolution stages to form a first quantized sequence. The second learner 2200 is configured to compress the first quantized sequence into a second quantized sequence. The vocoder 2300 is configured to map the second quantized sequence to a speech waveform corresponding to the input text.

[0203] In one embodiment of the present application, the first learner 2100 is further configured to convert the acoustic feature sequence into a multi-level sub-sequence with different time resolution stages; and use multiple predetermined codebooks to quantize each of the multi-level sub-sequences respectively to generate a multi-level quantized sub-sequence with different time resolution stages.

[0204] In addition, at least one of the multiple predetermined codebooks is a multi-head codebook, and the first learner 2100 is further configured to: in at least one time resolution stage, use the multi-head codebook multi-head vector quantization to generate a quantized subsequence corresponding to at least one time resolution stage to generate a quantized subsequence.

[0205] In addition, as an option, the first quantization sequence may include a multi-level quantization subsequence with different time resolution stages, and the second learner 2200 is further configured to repeatedly sample a portion of the quantization subsequences in the multi-level quantization subsequences so that the multi-level quantization subsequences have the same length; and quantize the multi-level quantization subsequences with the same length through a single codebook to form a second quantization sequence.

[0206] Furthermore, the second learner 2200 is further configured to: extract an utterance-level embedding with global properties in the first quantization sequence; incorporate the utterance-level embedding into the second quantization sequence; and map the incorporated second quantization sequence into a speech waveform.

[0207] As an option, the vocoder 2300 is further configured to: decode the second quantized sequence level by level according to a predetermined time resolution stage to form a multi-level prediction subsequence; and generate a speech waveform based on the multi-level prediction subsequence.

[0208] In one embodiment of the present application, within the same time resolution stage, at least one of the multi-level prediction sub-sequences corresponds to at least one of the multi-level sub-sequences of the first quantization sequence.

[0209] The speech synthesis system provided in the present application has the same beneficial effects as the speech synthesis method provided in the present application because it adopts the speech synthesis method provided in the present application, and will not be described in detail here.

[0210] In another aspect, the present application provides an electronic device, which includes: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned speech synthesis method.

[0211] On the other hand, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned speech synthesis method.

[0212] Fig.10 is a schematic diagram of an electronic device 3000 according to one embodiment of the present application.

[0213] like Fig.10 As shown, electronic device 3000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0214] like Fig.10 The electronic device 3000 shown includes: one or more processors 3100, a memory 3200, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 3100 is taken as an example.

[0215] The memory 3200 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the speech synthesis method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the speech synthesis method provided in the present application.

[0216] The memory 3200 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 3100 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 3200, that is, implements the speech synthesis method in the above method implementation.

[0217] The memory 3200 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of an electronic device for quality control, etc. In addition, the memory 3200 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 3200 may include a memory remotely arranged relative to the processor 3100, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The electronic device 3000 may further include: an input device 3300 and an output device 3400. The processor 3100, the memory 3200, the input device 3300 and the output device 3400 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0219] The input device 3300 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for controlling quality, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 3400 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0220] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0221] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0223] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0224] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0225] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0226] The speech synthesis system, electronic device, computer-readable storage medium and computer program product provided in the present application have the same beneficial effects as the speech synthesis method provided in the present application, and are not described in detail here.

[0227] The above description is only an implementation method of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of protection involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A speech synthesis method, characterized in that: The speech synthesis method comprises: Based on a plurality of different time resolution stages, multi-codebook quantization is performed on the acoustic feature sequence of the input text to form a first quantization sequence; compressing the first quantized sequence into a second quantized sequence; and The second quantized sequence is mapped into a speech waveform corresponding to the input text.

2. The speech synthesis method according to claim 1, wherein: Performing multi-codebook quantization on the acoustic feature sequence of the input text to form a first quantization sequence includes: Converting the acoustic feature sequence into a multi-level subsequence with different time resolution stages; and A plurality of predetermined codebooks are used to quantize each of the multi-level subsequences respectively, so as to generate a multi-level quantized subsequence with the different time resolution stages.

3. The speech synthesis method according to claim 2, wherein: At least one of the multiple predetermined codebooks is a multi-head codebook, and using the multiple predetermined codebooks to quantize each of the multi-level subsequences respectively to generate the multi-level quantized subsequences with the different time resolution stages includes: In at least one time resolution stage, the multi-head codebook is used to perform multi-head vector quantization on a subsequence corresponding to the at least one time resolution stage to generate a quantized subsequence.

4. The speech synthesis method according to claim 1, wherein: The first quantization sequence includes a multi-level quantization subsequence with different time resolution stages, and compressing the first quantization sequence into a second quantization sequence includes: Repeating sampling of a part of the quantized subsequences in the multi-level quantized subsequences so that the multi-level quantized subsequences have the same length; and The multi-level quantization subsequences having the same length are quantized by a single codebook to form the second quantization sequence.

5. The speech synthesis method according to claim 1, wherein: The method further comprises: extracting utterance-level embeddings with global properties in the first quantized sequence, The step of mapping the second quantized sequence into a speech waveform corresponding to the input text includes: incorporating the utterance-level embedding into the second quantized sequence; and The incorporated second quantized sequence is mapped into the speech waveform.

6. The speech synthesis method according to claim 1, wherein: Mapping the second quantized sequence into a speech waveform corresponding to the input text comprises: Decoding the second quantized sequence level by level according to a predetermined time resolution stage to form a multi-level prediction subsequence; and The speech waveform is generated based on the multi-level predictor sequence.

7. The speech synthesis method according to claim 6, wherein: At least one of the multi-level prediction sub-sequences corresponds to at least one of the multi-level sub-sequences of the first quantization sequence within the same time resolution stage.

8. The speech synthesis method according to claim 1, wherein: The method is implemented by a speech synthesis system, and the method further comprises training the speech synthesis system, wherein the training comprises: Based on a plurality of different time resolution stages, multi-codebook quantization is performed on the acoustic feature sequence of the training set to form a first quantization sequence; Mapping the first quantized sequence into a first waveform; compressing the first quantized sequence into a second quantized sequence; mapping the second quantized sequence into a second waveform; and Based on the first waveform and the second waveform, the parameters of the speech synthesis system are adjusted to obtain the trained speech synthesis system.

9. The speech synthesis method according to claim 8, wherein: The training set includes a supervised data set and an unsupervised data set.

10. The speech synthesis method according to claim 9, wherein: The supervised dataset includes a low-resource language dataset.

11. The speech synthesis method according to claim 9, wherein: The training also includes: Pre-training the speech synthesis system using the unsupervised dataset; and The speech synthesis system is fine-tuned using the supervised dataset.

12. The speech synthesis method according to claim 11, wherein: Pre-training the speech synthesis system using the unsupervised dataset includes: Using the supervised dataset to train at least part of the unsupervised dataset to form a low-precision supervised dataset; and The speech synthesis system is pre-trained using the low-precision supervised dataset.

13. A speech synthesis system, characterized in that: The system comprises: A first learner is configured to perform multi-codebook quantization on an acoustic feature sequence of an input text based on a plurality of different time resolution stages to form a first quantized sequence; A second learner configured to compress the first quantized sequence into a second quantized sequence; and A vocoder is configured to map the second quantized sequence into a speech waveform corresponding to the input text.

14. The system according to claim 13, wherein: The first learner is further configured to: Converting the acoustic feature sequence into a multi-level subsequence with different time resolution stages; as well as A plurality of predetermined codebooks are used to quantize each of the multi-level subsequences respectively, so as to generate a multi-level quantized subsequence with the different time resolution stages.

15. The system of claim 14, wherein: At least one of the plurality of predetermined codebooks is a multi-head codebook, and the first learner is further configured to: In at least one time resolution stage, the multi-head codebook is used to perform multi-head vector quantization on a subsequence corresponding to the at least one time resolution stage to generate a quantized subsequence.

16. The system of claim 13, wherein: The first quantization sequence includes a multi-level quantization subsequence with different time resolution stages, and the second learner is further configured to: Repeatedly sampling a part of the quantized subsequences in the multi-level quantized subsequences so that the multi-level quantized subsequences have the same length; as well as The multi-level quantization subsequences having the same length are quantized by a single codebook to form the second quantization sequence.

17. The system of claim 13, wherein: The second learner is further configured to: extracting an utterance-level embedding having global properties from the first quantized sequence; incorporating the utterance-level embedding into the second quantized sequence; and The incorporated second quantized sequence is mapped into the speech waveform.

18. The system of claim 13, wherein: The vocoder is further configured to: Decoding the second quantized sequence level by level according to a predetermined time resolution stage to form a multi-level prediction subsequence; and The speech waveform is generated based on the multi-level predictor sequence.

19. The system of claim 18, wherein: The vocoder is further configured to: At least one of the multi-level prediction sub-sequences corresponds to at least one of the multi-level sub-sequences of the first quantization sequence within the same time resolution stage.

20. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech synthesis method according to any one of claims 1 to 12.

21. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the speech synthesis method according to any one of claims 1 to 12.

Images