Zero-sample speech synthesis method based on denoising non-parametric speaker adaptation

Abstract

The invention relates to a zero-sample speech synthesis method based on denoising non-parametric speaker adaptation, and belongs to the technical field of natural language processing. The method is used for a high-fidelity zero-sample speaker to adapt to speech without a speaker, and zero-sample speech synthesis faces the problems that a model is usually trained on a small data set, and the generalization ability is poor; when the condition depends on noise reference, unseen speaker identities are difficult to model; according to the zero-sample speech synthesis method based on denoising non-parametric speaker adaptation, speaker embedding is predicted by using a nearest neighbor classifier on a database of a large number of cache examples, similarity search is carried out by using representation of a speaker encoder, and the method does not need extra training and is high in robustness. Therefore, a highly robust model is formed, and the performance can be improved when the method depends on noise reference.

Description

Technical Field

[0001] The present invention relates to a zero-sample speech synthesis method based on denoising non-parametric speaker adaptation, and belongs to the technical field of natural language processing. Background Art

[0002] Zero-shot text-to-speech synthesis (ZS-TTS) aims to generate speech samples with unseen speaker identities from an acoustic reference. Compared to traditional TTS tasks, ZS-TTS presents two challenges: first, it requires the model to be robust enough to cope with diverse unseen conditions that differ from the source data. In particular, most TTS models are task-specific and trained on small datasets, and therefore do not generalize well. Second, it requires appropriate speaker modeling to reflect the unseen speaker identities in the reference speech. Previous studies have only encoded the speaker into a low-dimensional space and imposed an overly simplistic distribution on the speech given text and residual embeddings, which lacks the ability to model the unseen speaker identities under noisy reference conditions. These pose significant challenges to existing ZS-TTS methods. Summary of the Invention

[0003] The technical problem solved by the present invention is: the present invention provides a zero-shot speech synthesis method based on denoising non-parametric speaker adaptation to solve the problems of poor generalization and difficulty in modeling the identity of unseen speakers under noise reference conditions. The present invention can improve performance when relying on noise reference.

[0004] The technical solution of the present invention is: a zero-shot speech synthesis method based on denoising non-parametric speaker adaptation, the method comprising:

[0005] Step 1. Speaker database establishment: Establish a speaker database in an offline environment. The speaker database contains a series of key-value pairs.

[0006] Step 2: Non-parametric Speaker Adapter Generation: The non-parametric speaker adapter enhances the speaker encoder through a nearest neighbor retrieval mechanism. During testing, the average of the low-frequency discrete F0 representation of the most similar speech in the database is found and the corresponding unseen speaker embedding distribution is calculated. The unseen speaker embedding distribution is then interpolated with the output distribution of the speaker encoder, allowing the speaker encoder to directly access the speaker database of cached examples.

[0007] Step 3: The voice data is denoised and then denoised through a diffusion decoder to generate high-quality speech.

[0008] Furthermore, the Step 1 includes:

[0009] For a given input speech u from a known speaker s ;

[0010] Key K is the low-frequency discrete F0 representation of speech f(u s ), where f represents the mapping from the input speech to the low-frequency discrete F0 representation of the speech of the speaker encoder;

[0011] The high-dimensional speaker identity representation g(u s ), where g represents the mapping from input speech to high-dimensional speaker identity representation;

[0012] The low-frequency discrete F0 of speech is represented by f(u s ) and high-dimensional speaker identity representation g(u s ) is generated by making a single forward pass on each example; the complete speaker database is defined as follows:

[0013]

[0014] Among them, f(u si ) represents the low-frequency discrete F0 representation of the speech generated by the pre-trained speaker encoder, g(u si ) represents the high-dimensional speaker identity representation generated by the same speaker encoder, u si represents the i-th speech sample, belonging to the speaker's speech set; u s Represents the input speech, a speech sample from a registered speaker.

[0015] Furthermore, the Step 2 includes:

[0016] Step 2.1, extract speaker identity features from the acoustic reference of unseen speakers;

[0017] Step 2.2, search for the speaker embedding distribution of the nearest neighbor of the unseen speaker;

[0018] Step 2.3: Generate unseen speaker embedding distribution.

[0019] Furthermore, the Step 2.1 includes:

[0020] Let u uns For the input speech of an unseen speaker, extract the low-frequency discrete F0 representation and speaker embedding of the speech from the given reference in the constructed speaker database:

[0021] P unsj =f(u unsj );

[0022]

[0023] Among them, P unsj f(uunsj ), which represents the F0 key of the jth sample, u unsj is the jth speech sample of the unseen speaker, h unsj is g(u unsj ), which represents the speaker embedding of the jth sample, g(u unsj ) is a high-dimensional speaker embedding, a speaker representation generated by the same speaker encoder g.

[0024] Furthermore, the Step 2.2 includes:

[0025] Given the input speech u of the target unseen speaker uns , the pre-trained speaker encoder outputs the target unseen speaker’s input speech u uns The speaker embedding distribution h uns ;

[0026] The speaker encoder also outputs the low-frequency discrete F0 representation of the speech P uns , used according to the square L 2 , distance d queries the k nearest neighbor sets in the database, and converts the retrieved sets into speaker embedding distributions by applying temperature T to negative distances and aggregating multiple occurrences of the same F0 term; using a temperature T greater than 1 flattens the speaker embedding distribution and prevents overfitting to the most similar retrieval results

[0027] Among them, L 2 To calculate the similarity between F0, denote the nearest neighbor set N retrieved by F0.

[0028] Furthermore, the Step 2.3 includes:

[0029] Given the speaker embedding distribution h output by the speaker encoder uns and obtained by nearest neighbor search of the target unseen speaker Improve the unseen speaker embedding results by interpolating with the output distribution of the speaker encoder; the speaker embedding distribution h uns and the distribution obtained by nearest neighbor search of the target unseen speaker By adjusting the parameter λ for interpolation, the final unseen speaker embedding distribution is obtained as follows:

[0030]

[0031] in: is the embedding distribution of the unseen speaker obtained from the cached examples of the target unseen speaker via nearest neighbor search, and T is a temperature parameter used to adjust the scale or distribution of the nearest neighbor search embedding representation.

[0032] Furthermore, the Step 3 includes:

[0033] During the diffusion process, Gaussian noise is gradually added to the speech data;

[0034] The inverse process focuses on learning a denoising function to remove the added noise and restore the original speech data structure; the training process is defined as follows:

[0035]

[0036] Where t is the step index, α i is a constant parameter of e, e~N(0,I), x t is the noisy speech in step t, x0 is the original speech data, that is, the input speech signal without adding noise, x0′ is the original speech reconstruction value predicted by the model in the inverse process, and Decoder(,,) is the decoder;

[0037] Final unseen speaker embedding distribution Combined with a text encoder and a random-duration decoder, the diffusion decoder is guided to generate synthetic speech from unseen speakers.

[0038] The present invention also provides a zero-shot speech synthesis system based on denoising non-parametric speaker adaptation, which includes a module for executing the zero-shot speech synthesis method based on denoising non-parametric speaker adaptation.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the zero-sample speech synthesis method based on denoising and non-parametric speaker adaptation is implemented.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the zero-sample speech synthesis method based on denoising and non-parametric speaker adaptation is implemented.

[0041] The beneficial effects of the present invention are:

[0042] 1. This paper proposes a novel denoising ZS-TTS architecture that uses a speaker data repository to better synthesize speech with unseen speaker identities from acoustic references.

[0043] 2. This paper designs a non-parametric speaker adapter that predicts the speaker embedding of the reference speech through nearest neighbor retrieval. It can be added to any multi-speaker TTS model without further training and significantly improves the performance under in-domain, out-of-domain and noisy reference conditions.

[0044] 3. Experimental results show that the proposed method performs well on two benchmark datasets, surpassing the most advanced models. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a model structure diagram of the present invention;

[0046] Figure 2 t-SNE visualization of the utterance-level speaker embedding vectors of 8 unseen speakers;

[0047] Figure 3 Waveform visualization of the rhythmic diversity of two output audio files for the same input. DETAILED DESCRIPTION

[0048] Example 1: Figure 1-Figure 3 As shown, a zero-shot speech synthesis method based on denoising non-parametric speaker adaptation includes:

[0049] Step 1. Speaker database establishment: Establish a speaker database in an offline environment. The speaker database contains a series of key-value pairs.

[0050] Step 2: Non-parametric Speaker Adapter Generation: The non-parametric speaker adapter enhances the speaker encoder through a nearest neighbor retrieval mechanism. During testing, the average of the low-frequency discrete F0 representation of the most similar speech in the database is found and the corresponding unseen speaker embedding distribution is calculated. The unseen speaker embedding distribution is then interpolated with the output distribution of the speaker encoder, allowing the speaker encoder to directly access the speaker database of cached examples.

[0051] Step 3: The voice data is denoised and then denoised through a diffusion decoder to generate high-quality speech.

[0052] Furthermore, the Step 1 includes:

[0053] For a given input speech u from a known speaker s ;

[0054] Key K is the low-frequency discrete F0 representation of speech f(u s ), where f represents the mapping from the input speech to the low-frequency discrete F0 representation of the speech of the speaker encoder;

[0055] The high-dimensional speaker identity representation g(u s ), where g represents the mapping from input speech to high-dimensional speaker identity representation;

[0056] The low-frequency discrete F0 of speech is represented by f(u s ) and high-dimensional speaker identity representation g(u s ) is generated by making a single forward pass on each example; the complete speaker database is defined as follows:

[0057]

[0058] Among them, f(u si ) represents the low-frequency discrete F0 representation of the speech generated by the pre-trained speaker encoder, g(u si ) represents the high-dimensional speaker identity representation generated by the same speaker encoder, u si represents the i-th speech sample, belonging to the speaker's speech set; u s Represents the input speech, a speech sample from a registered speaker.

[0059] Furthermore, the Step 2 includes:

[0060] Step 2.1, extract speaker identity features from the acoustic reference of unseen speakers;

[0061] Step 2.2, search for the speaker embedding distribution of the nearest neighbor of the unseen speaker;

[0062] Step 2.3: Generate unseen speaker embedding distribution.

[0063] Furthermore, the Step 2.1 includes:

[0064] Let u uns For the input speech of an unseen speaker, extract the low-frequency discrete F0 representation and speaker embedding of the speech from the given reference in the constructed speaker database:

[0065] P unsj =f(u unsj );

[0066]

[0067] Among them, P unsj f(u unsj ), which represents the F0 key of the jth sample, u unsj is the jth speech sample of the unseen speaker, h unsj is g(u unsj ), which represents the speaker embedding of the jth sample, g(u unsj ) is a high-dimensional speaker embedding, a speaker representation generated by the same speaker encoder g.

[0068] Furthermore, the Step 2.2 includes:

[0069] Given the input speech u of the target unseen speaker uns , the pre-trained speaker encoder outputs the target unseen speaker’s input speech u uns The speaker embedding distribution h uns ;

[0070] The speaker encoder also outputs the low-frequency discrete F0 representation of the speech P uns , used according to the square L 2 , distance d queries the k nearest neighbor sets in the database, and converts the retrieved sets into speaker embedding distributions by applying temperature T to negative distances and aggregating multiple occurrences of the same F0 term; using a temperature T greater than 1 flattens the speaker embedding distribution and prevents overfitting to the most similar retrieval results

[0071] Among them, L 2 To calculate the similarity between F0, denote the nearest neighbor set N retrieved by F0.

[0072] Furthermore, the Step 2.3 includes:

[0073] Given the speaker embedding distribution h output by the speaker encoder uns and obtained by nearest neighbor search of the target unseen speaker The unseen speaker embedding results are improved by interpolating with the output distribution of the speaker encoder. This interpolation method can improve the robustness and stability of the unseen speaker embedding when there is a lack of high-similarity reference samples in the speaker database; the speaker embedding distribution h uns and the distribution obtained by nearest neighbor search of the target unseen speaker By adjusting the parameter λ for interpolation, the final unseen speaker embedding distribution is obtained as follows:

[0074]

[0075] in: is the embedding distribution of the unseen speaker obtained from the cached examples of the target unseen speaker via nearest neighbor search, and T is a temperature parameter used to adjust the scale or distribution of the nearest neighbor search embedding representation.

[0076] Furthermore, the Step 3 includes:

[0077] During the diffusion process, Gaussian noise is gradually added to the speech data;

[0078] The inverse process focuses on learning a denoising function to remove the added noise and restore the original speech data structure; the training process is defined as follows:

[0079]

[0080] Where t is the step index, α i is a constant parameter of e, e~N(0,I), x t is the noisy speech in step t, x0 is the original speech data, that is, the input speech signal without adding noise, x0′ is the original speech reconstruction value predicted by the model in the inverse process, and Decoder(,,) is the decoder;

[0081] Final unseen speaker embedding distribution Combined with a text encoder and a random-duration decoder, the diffusion decoder is guided to generate synthetic speech from unseen speakers.

[0082] like Figure 2 As shown in the figure, the present invention extracts embedding vectors for multiple unseen speakers from a non-parametric speaker adapter and performs dimensionality reduction using the t-SNE visualization method. It can be observed that the embedding features of different speakers show clear clustering in the low-dimensional space, demonstrating that the present method can accurately model the identity characteristics of unseen speakers.

[0083] like Figure 3 As shown in the figure, for the same text and reference speech input, the waveforms generated by the proposed method in two independent speech synthesis processes have significant differences in rhythm and speaking rate, demonstrating the rhythmic diversity of the model in speech generation. This diversity stems from the randomness of the diffusion decoder during the denoising sampling process, further enhancing the naturalness and personalization of the speech synthesis results.

[0084] Figure 2 and Figure 3 Together, they demonstrate the effectiveness and robustness of the method in achieving unseen speaker modeling and improving speech generation diversity.

[0085] The present invention also provides a zero-shot speech synthesis system based on denoising non-parametric speaker adaptation, which includes a module for executing the zero-shot speech synthesis method based on denoising non-parametric speaker adaptation.

[0086] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the zero-sample speech synthesis method based on denoising and non-parametric speaker adaptation is implemented.

[0087] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the zero-sample speech synthesis method based on denoising and non-parametric speaker adaptation is implemented.

[0088] In order to illustrate the effect of the present invention, the method of the present invention was carried out the following experiments:

[0089] (a) Comparison and ablation experiments. We also studied the speech quality using different data stores. Table 1 shows the MOS, VSS, MCD, and RMSE results of the related experiments.

[0090] Table 1 shows the scores of various evaluation indicators on the AISHELL3 and VCTK datasets

[0091]

[0092] Comparative Experimental Analysis: Table 1 summarizes the MCD and RMSE evaluation results on the VCTK and AISHELL3 datasets. It is observed that the proposed method consistently outperforms all baseline models in terms of MCD and RMSE. The above results indicate that the proposed method DNSA-ZSTTS* exhibits superior performance while maintaining language content. In addition, it is noted that the objective evaluation indicators of VALL-E are not satisfactory, which may be due to the accuracy issues of the character-to-character system, which affects its performance. In contrast, the proposed method DNSA-ZSTTS* is not affected by direct text input.

[0093] In the MOS and VSS tests, listeners were asked to evaluate the quality of the converted speech on a 5-point scale. The audio from the three systems was randomly shuffled before being presented to the listeners. Each set of audio corresponded to the same text content. The MOS and VSS results in Table 1 show that the proposed method DNSA-ZSTTS* consistently achieved the best or second-best quality in terms of MOS and VSS. On the VCTK dataset, the proposed method achieved the best MOS score and second-best speaker similarity. On the AISHELL3 dataset, the proposed method achieved second-best MOS score and speaker similarity. It is worth noting that the proposed method outperformed VALL-E in MOS scores and surpassed YourTTS in VSS. This shows that the proposed method DNSA-ZSTTS* achieves a better balance between speech expression accuracy and modeling the timbre of unseen speakers.

[0094] Ablation experiment analysis: As can be seen from the results in Table 1, the MCD performance degradation ranges from 0.52 on the AISHELL3 dataset to 0.75 on the VCTK dataset. This verifies that the BERT text encoder and diffusion decoder can provide the model with more powerful speech generation capabilities, while the non-parametric speaker adapter can improve the model's ability to model the identities of unseen speakers.

[0095] Speech Quality Analysis in Different Data Stores: Table 1 shows the in-domain performance of our method DNSA-ZSTTS* on the AISHELL3 and VCTK datasets, as well as its performance under out-of-domain and noisy conditions. For in-domain data, our method DNSA-ZSTTS* significantly outperforms VITS2, improving by 0.21 VSS on VCTK and 0.19 VSS on AISHELL3. For out-of-domain and noisy speech conditions, the speech recognition dataset LibriSpeech was used to construct the data store. In this case, the average VSS improved by 0.12 points, but the improvement was smaller than when using in-domain data. This suggests that adding domain-specific data to the data store is more valuable than adding large amounts of arbitrary data. We also measured the effectiveness of constructing a single multi-domain data store containing VCTK, AISHELL3, and LibriSpeech. The MCD and RMSE performance on VCTK and AISHELL3 improved by 0.48 and 0.37, and 0.40 and 0.52, respectively. This demonstrates that our method DNSA-ZSTTS* exhibits good robustness even in the presence of out-of-domain examples. Retrieving neighbor data from a data store containing a large amount of out-of-domain data does not degrade performance compared to using only in-domain data.

[0096] (b) Efficiency and running time

[0097] Table 2 shows the average time (T, in seconds) and real-time factor (RTF), where RTF represents the number of seconds required to generate one second of audio on an NVIDIA 3090 GPU.

[0098]

[0099]

[0100] Considering that the number of model parameters is similar to that of other non-autoregressive TTS benchmark models, the decoding time complexity of the proposed method DNSA-ZSTTS* is O(1). The converted speech is generated in parallel, and the decoding time is affected by the scale of the model parameters. To illustrate this, the present invention considers the case of a single converted speech sample. Table 2 shows the average runtime of each model over 10 runs on the two datasets. Overall, the proposed method DNSA-ZSTTS* achieves a balance between runtime and performance.

[0101] The present invention predicts speaker embeddings using a nearest neighbor classifier on a large data repository of cached examples and uses the representation of the speaker encoder for similarity search. This approach requires no additional training, resulting in a highly robust model that improves performance when relying on noisy references.

[0102] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A zero-shot speech synthesis method based on denoising and non-parametric speaker adaptation, characterized by: The method comprises: Step 1. Speaker database establishment: Establish a speaker database in an offline environment. The speaker database contains a series of key-value pairs. Step 2: Non-parametric Speaker Adapter Generation: The non-parametric speaker adapter enhances the speaker encoder through a nearest neighbor retrieval mechanism. During testing, the average of the low-frequency discrete F0 representation of the most similar speech in the database is found and the corresponding unseen speaker embedding distribution is calculated. The unseen speaker embedding distribution is then interpolated with the output distribution of the speaker encoder, allowing the speaker encoder to directly access the speaker database of cached examples. Step 3: The voice data is denoised and then denoised through a diffusion decoder to generate high-quality speech.

2. The zero-shot speech synthesis method based on denoising non-parametric speaker adaptation according to claim 1, characterized in that: Step 1 includes: For a given input speech u from a known speaker s ; Key K is the low-frequency discrete F0 representation of speech f(u s ), where f represents the mapping from the input speech to the low-frequency discrete F0 representation of the speech of the speaker encoder; The high-dimensional speaker identity representation g(u s ), where g represents the mapping from input speech to high-dimensional speaker identity representation; The low-frequency discrete F0 of speech is represented by f(u s ) and high-dimensional speaker identity representation g(u s ) is generated by making a single forward pass on each example; the complete speaker database is defined as follows: Among them, f(u si ) represents the low-frequency discrete F0 representation of the speech generated by the pre-trained speaker encoder, g(u si ) represents the high-dimensional speaker identity representation generated by the same speaker encoder, u si represents the i-th speech sample, belonging to the speaker's speech set; u s Represents the input speech, a speech sample from a registered speaker.

3. The zero-shot speech synthesis method based on denoising non-parametric speaker adaptation according to claim 1, characterized in that: Step 2 includes: Step 2.1, extract speaker identity features from the acoustic reference of unseen speakers; Step 2.2, search for the speaker embedding distribution of the nearest neighbor of the unseen speaker; Step 2.3: Generate unseen speaker embedding distribution.

4. The zero-shot speech synthesis method based on denoising and non-parametric speaker adaptation according to claim 3, characterized in that: Step 2.1 includes: Let u uns For the input speech of an unseen speaker, extract the low-frequency discrete F0 representation and speaker embedding of the speech from the given reference in the constructed speaker database: P unsj =f(u unsj ); Among them, P unsj f(u unsj ), which represents the F0 key of the jth sample, u unsj is the jth speech sample of the unseen speaker, h unsj is g(u unsj ), which represents the speaker embedding of the jth sample, g(u unsj ) is a high-dimensional speaker embedding, a speaker representation generated by the same speaker encoder g.

5. The zero-shot speech synthesis method based on denoising non-parametric speaker adaptation according to claim 3, characterized in that: Step 2.2 includes: Given the input speech u of the target unseen speaker uns , the pre-trained speaker encoder outputs the target unseen speaker’s input speech u uns The speaker embedding distribution h uns ; The speaker encoder also outputs the low-frequency discrete F0 representation of the speech P uns , used according to the square L 2 , distance d queries the k nearest neighbor sets in the database, and converts the retrieved sets into speaker embedding distributions by applying temperature T to negative distances and aggregating multiple occurrences of the same F0 term; using a temperature T greater than 1 flattens the speaker embedding distribution and prevents overfitting to the most similar retrieval results Among them, L 2 To calculate the similarity between F0, denote the nearest neighbor set N retrieved by F0.

6. The zero-shot speech synthesis method based on denoising non-parametric speaker adaptation according to claim 3, characterized in that: Step 2.3 includes: Given the speaker embedding distribution h output by the speaker encoder uns and obtained by nearest neighbor search of the target unseen speaker Improve the unseen speaker embedding results by interpolating with the output distribution of the speaker encoder; the speaker embedding distribution h uns and the distribution obtained by nearest neighbor search of the target unseen speaker By adjusting the parameter λ for interpolation, the final unseen speaker embedding distribution is obtained as follows: in: is the embedding distribution of the unseen speaker obtained from the cached examples of the target unseen speaker via nearest neighbor search, and T is a temperature parameter used to adjust the scale or distribution of the nearest neighbor search embedding representation.

7. The zero-shot speech synthesis method based on denoising non-parametric speaker adaptation according to claim 1, characterized in that: Step 3 includes: During the diffusion process, Gaussian noise is gradually added to the speech data; The inverse process focuses on learning a denoising function to remove the added noise and restore the original speech data structure; the training process is defined as follows: Where t is the step index, α i is a constant parameter of e, e~N(0,I), x t is the noisy speech in step t, x0 is the original speech data, that is, the input speech signal without adding noise, x0′ is the original speech reconstruction value predicted by the model in the inverse process, and Decoder(,,) is the decoder; Final unseen speaker embedding distribution Combined with a text encoder and a random-duration decoder, the diffusion decoder is guided to generate synthetic speech from unseen speakers.

8. A zero-shot speech synthesis system based on denoising non-parametric speaker adaptation, characterized by: The system comprises: a module for executing the zero-shot speech synthesis method based on denoising and non-parametric speaker adaptation according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the zero-sample speech synthesis method based on denoising and non-parametric speaker adaptation is implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the zero-shot speech synthesis method based on denoising and non-parametric speaker adaptation is implemented as claimed in any one of claims 1 to 7.

Images